Re: Double free or corruption error (fasttop)

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 4268 bytes --]

----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> Hello All,

> I am trying to instrument a pthread application using the provided pthread
> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
> )" error.

Please provide more information about the version of lttng-ust you are using, and how you setup 
your tracing session. 

Thanks, 

Mathieu 

> Here is a description of what I have tried and noticed:
> 1- The problem isn't consistent. It sometimes happen and sometimes works as
> expected.
> 2- From my experiments, the problem happens (more frequently at least) when
> adding performance counter contexts (I tried cycles, cpu _cycles and
> instructions).
> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
> PRELOAD to use the pthread wrapper as described in the LTTng documents.
> 4- Here is the backtrace printed after exiting:
> ======= Backtrace : =========
> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
> / usr /local/lib/ liblttng - ust .so.0( lttng
> _destroy_context+0x35)[0x7ffff7471575]
> / usr /local/lib/ liblttng - ust .so.0( lttng
> _session_destroy+0x21c)[0x7ffff747363c]
> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
> +0x9f)[0x7ffff746dccf]
> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
> +0x9f)[0x7ffff746dccf]
> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
> +0x9f)[0x7ffff746dccf]
> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
> _exit+0x68)[0x7ffff746ead8]
> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
> 5- Also, this is a backtrace I obtained from gdb :
> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
> field=<optimized out>) at lttng -context- perf -counters.c:418
> #5 0x00007ffff767a575 in lttng _destroy_context (
> ctx =0x7ffff0011090) at lttng -context.c:278
> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
> #7 lttng _session_destroy (session=0x7ffff0000900)
> at lttng -events.c:247
> #8 0x00007ffff7676906 in lttng _release_session (
> objd =<optimized out>) at lttng - ust - abi .c:601
> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
> is_owner=<optimized out>) at lttng - ust - abi .c:216
> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
> is_owner=<optimized out>) at lttng - ust - abi .c:216
> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
> #12 0x00007ffff7677ad8 in objd _table_destroy ()
> at lttng - ust - abi .c:235
> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
> at lttng - ust -comm.c:1807
> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
> at lttng - ust -comm.c:1874
> #16 0x00007ffff7dec85a in _ dl _ fini ()
> from /lib64/ ld - linux -x86-64.so.2
> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
> from /lib64/ libc .so.6
> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
> #19 0x0000000000401b51 in main ( argc =<optimized out>,
> argv =<optimized out>) at lu .c:368

> Any ideas, why this happens and how to fix it?

> Thanks,
> Shehab

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 12356 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 5431 bytes --]

Hi Mathieu,

Thank you very much for your reply.

I manually built lttng-ust from source (commit #:
8a208943e21700211beee3ea64180a5a534c7d2a).

This is how I set up the tracing session:
1- lttng create lu_ncb_8_native -o {path}
2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
    lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
    lttng enable-event --userspace
lttng_ust_pthread:pthread_mutex_lock_trylock
    lttng enable-event --userspace
lttng_ust_pthread:pthread_mutex_lock_unlock
3- lttng add-context -u -t procname
    lttng add-context -u -t vpid
    lttng add-context -u -t pthread_id
    lttng add-context -u -t vtid
    lttng add-context -u -t ip
    lttng add-context -u -t perf:thread:cpu-cycles
    lttng add-context -u -t perf:thread:cycles
    lttng add-context -u -t perf:thread:instructions
4- lttng start
5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8
-n8096 -b32
6- lttng stop
7- lttng destroy



Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

>
>
> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> Hello All,
>
> I am trying to instrument a pthread application using the provided pthread
> wrapper, but I sometimes run into a "Double free or corruption error (
> fasttop)" error.
>
>
> Please provide more information about the version of lttng-ust you are
> using, and how you setup
> your tracing session.
>
> Thanks,
>
> Mathieu
>
>
>
> Here is a description of what I have tried and noticed:
> 1- The problem isn't consistent. It sometimes happen and sometimes works
> as expected.
> 2- From my experiments, the problem happens (more frequently at least)
> when adding performance counter contexts (I tried cycles, cpu_cycles and
> instructions).
> 3- I am testing using lu_ncb from splash3 benchmark suite after setting LD
> _PRELOAD to use the pthread wrapper as described in the LTTng documents.
> 4- Here is the backtrace printed after exiting:
> ======= Backtrace: =========
> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
> 0x35)[0x7ffff7471575]
> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
> 0x21c)[0x7ffff747363c]
> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
> 0x9f)[0x7ffff746dccf]
> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
> 0x9f)[0x7ffff746dccf]
> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
> 0x9f)[0x7ffff746dccf]
> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x7ffff746ead8]
> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
> 5- Also, this is a backtrace I obtained from gdb:
> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>     field=<optimized out>) at lttng-context-perf-counters.c:418
> #5  0x00007ffff767a575 in lttng_destroy_context (
> ctx=0x7ffff0011090) at lttng-context.c:278
> #6  0x00007ffff767c63c in _lttng_channel_unmap (
> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
> #7  lttng_session_destroy (session=0x7ffff0000900)
>     at lttng-events.c:247
> #8  0x00007ffff7676906 in lttng_release_session (
> objd=<optimized out>) at lttng-ust-abi.c:601
> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>     is_owner=<optimized out>) at lttng-ust-abi.c:216
> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>     is_owner=<optimized out>) at lttng-ust-abi.c:216
> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>     at lttng-ust-abi.c:235
> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>     at lttng-ust-comm.c:1807
> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>     at lttng-ust-comm.c:1874
> #16 0x00007ffff7dec85a in _dl_fini ()
>    from /lib64/ld-linux-x86-64.so.2
> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>    from /lib64/libc.so.6
> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
> #19 0x0000000000401b51 in main (argc=<optimized out>,
> argv=<optimized out>) at lu.c:368
>
> Any ideas, why this happens and how to fix it?
>
> Thanks,
> Shehab
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 15326 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 6431 bytes --]

----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> Hi Mathieu,

> Thank you very much for your reply.

> I manually built lttng-ust from source (commit #:
> 8a208943e21700211beee3ea64180a5a534c7d2a).

> This is how I set up the tracing session:
> 1- lttng create lu_ncb_8_native -o {path}
> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
> 3- lttng add-context -u -t procname
> lttng add-context -u -t vpid
> lttng add-context -u -t pthread_id
> lttng add-context -u -t vtid
> lttng add-context -u -t ip
> lttng add-context -u -t perf:thread:cpu-cycles
> lttng add-context -u -t perf:thread:cycles
> lttng add-context -u -t perf:thread:instructions
> 4- lttng start
> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
> -b32
> 6- lttng stop
> 7- lttng destroy

Can you reproduce if you remove the following contexts ? 

lttng add-context -u -t perf:thread:cpu-cycles 
lttng add-context -u -t perf:thread:cycles 
lttng add-context -u -t perf:thread:instructions 

And if you only keep a single of those contexts ? 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>> Hello All,

>>> I am trying to instrument a pthread application using the provided pthread
>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>> )" error.

>> Please provide more information about the version of lttng-ust you are using,
>> and how you setup
>> your tracing session.

>> Thanks,

>> Mathieu

>>> Here is a description of what I have tried and noticed:
>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>> expected.
>>> 2- From my experiments, the problem happens (more frequently at least) when
>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>> instructions).
>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>> 4- Here is the backtrace printed after exiting:
>>> ======= Backtrace : =========
>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>> _destroy_context+0x35)[0x7ffff7471575]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>> _session_destroy+0x21c)[0x7ffff747363c]
>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>> +0x9f)[0x7ffff746dccf]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>> +0x9f)[0x7ffff746dccf]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>> +0x9f)[0x7ffff746dccf]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>> _exit+0x68)[0x7ffff746ead8]
>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>> 5- Also, this is a backtrace I obtained from gdb :
>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>> at lttng -events.c:247
>>> #8 0x00007ffff7676906 in lttng _release_session (
>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>> at lttng - ust - abi .c:235
>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>> at lttng - ust -comm.c:1807
>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>> at lttng - ust -comm.c:1874
>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>> from /lib64/ ld - linux -x86-64.so.2
>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>> from /lib64/ libc .so.6
>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>> argv =<optimized out>) at lu .c:368

>>> Any ideas, why this happens and how to fix it?

>>> Thanks,
>>> Shehab

>>> _______________________________________________
>>> lttng-dev mailing list
>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 17147 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 7608 bytes --]

----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> Yes, I tried with only one of those contexts and I still ran into the same
> problem.

What is the setting returned by 

cat /proc/sys/kernel/perf_event_paranoid 

on your system ? And do you run your test program as root or normal user ? 

Please CC the mailing list on your reply. 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>> Hi Mathieu,

>>> Thank you very much for your reply.

>>> I manually built lttng-ust from source (commit #:
>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>> This is how I set up the tracing session:
>>> 1- lttng create lu_ncb_8_native -o {path}
>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>> 3- lttng add-context -u -t procname
>>> lttng add-context -u -t vpid
>>> lttng add-context -u -t pthread_id
>>> lttng add-context -u -t vtid
>>> lttng add-context -u -t ip
>>> lttng add-context -u -t perf:thread:cpu-cycles
>>> lttng add-context -u -t perf:thread:cycles
>>> lttng add-context -u -t perf:thread:instructions
>>> 4- lttng start
>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>> -b32
>>> 6- lttng stop
>>> 7- lttng destroy

>> Can you reproduce if you remove the following contexts ?

>> lttng add-context -u -t perf:thread:cpu-cycles
>> lttng add-context -u -t perf:thread:cycles
>> lttng add-context -u -t perf:thread:instructions

>> And if you only keep a single of those contexts ?

>> Thanks,

>> Mathieu

>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>> shehabyomn@gmail.com ]

>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>> wrote:

>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>> Hello All,

>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>> )" error.

>>>> Please provide more information about the version of lttng-ust you are using,
>>>> and how you setup
>>>> your tracing session.

>>>> Thanks,

>>>> Mathieu

>>>>> Here is a description of what I have tried and noticed:
>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>> expected.
>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>> instructions).
>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>> 4- Here is the backtrace printed after exiting:
>>>>> ======= Backtrace : =========
>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>> at lttng -events.c:247
>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>> at lttng - ust - abi .c:235
>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>> at lttng - ust -comm.c:1807
>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>> at lttng - ust -comm.c:1874
>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>> from /lib64/ libc .so.6
>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>> argv =<optimized out>) at lu .c:368

>>>>> Any ideas, why this happens and how to fix it?

>>>>> Thanks,
>>>>> Shehab

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 22740 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 8014 bytes --]

---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:

>> Yes, I tried with only one of those contexts and I still ran into the same
>> problem.

> What is the setting returned by

> cat /proc/sys/kernel/perf_event_paranoid

> on your system ? And do you run your test program as root or normal user ?

> Please CC the mailing list on your reply.

I will also need to know what glibc version you have on your system. 

Thanks, 

Mathieu 

> Thanks,

> Mathieu

>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>> shehabyomn@gmail.com ]

>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>> wrote:

>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>> Hi Mathieu,

>>>> Thank you very much for your reply.

>>>> I manually built lttng-ust from source (commit #:
>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>> This is how I set up the tracing session:
>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>> 3- lttng add-context -u -t procname
>>>> lttng add-context -u -t vpid
>>>> lttng add-context -u -t pthread_id
>>>> lttng add-context -u -t vtid
>>>> lttng add-context -u -t ip
>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>> lttng add-context -u -t perf:thread:cycles
>>>> lttng add-context -u -t perf:thread:instructions
>>>> 4- lttng start
>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>> -b32
>>>> 6- lttng stop
>>>> 7- lttng destroy

>>> Can you reproduce if you remove the following contexts ?

>>> lttng add-context -u -t perf:thread:cpu-cycles
>>> lttng add-context -u -t perf:thread:cycles
>>> lttng add-context -u -t perf:thread:instructions

>>> And if you only keep a single of those contexts ?

>>> Thanks,

>>> Mathieu

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> Hello All,

>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>> )" error.

>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>> and how you setup
>>>>> your tracing session.

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Here is a description of what I have tried and noticed:
>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>> expected.
>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>> instructions).
>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>> ======= Backtrace : =========
>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>> at lttng -events.c:247
>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>> at lttng - ust - abi .c:235
>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>> at lttng - ust -comm.c:1807
>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>> at lttng - ust -comm.c:1874
>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>> from /lib64/ libc .so.6
>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>> argv =<optimized out>) at lu .c:368

>>>>>> Any ideas, why this happens and how to fix it?

>>>>>> Thanks,
>>>>>> Shehab

>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> [ http://www.efficios.com/ | http://www.efficios.com ]

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 23548 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 7836 bytes --]

cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

I run the program as a normal user

The glibc version I get by running "ldd --version" is 2.17

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> Yes, I tried with only one of those contexts and I still ran into the same
> problem.
>
> What is the setting returned by
>
> cat /proc/sys/kernel/perf_event_paranoid
>
> on your system ? And do you run your test program as root or normal user ?
>
> Please CC the mailing list on your reply.
>
>
> I will also need to know what glibc version you have on your system.
>
> Thanks,
>
> Mathieu
>
>
>
>
> Thanks,
>
> Mathieu
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> Hi Mathieu,
>>
>> Thank you very much for your reply.
>>
>> I manually built lttng-ust from source (commit #:
>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>
>> This is how I set up the tracing session:
>> 1- lttng create lu_ncb_8_native -o {path}
>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>> mutex_lock_req
>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>> mutex_lock_acq
>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>> mutex_lock_trylock
>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>> mutex_lock_unlock
>> 3- lttng add-context -u -t procname
>>     lttng add-context -u -t vpid
>>     lttng add-context -u -t pthread_id
>>     lttng add-context -u -t vtid
>>     lttng add-context -u -t ip
>>     lttng add-context -u -t perf:thread:cpu-cycles
>>     lttng add-context -u -t perf:thread:cycles
>>     lttng add-context -u -t perf:thread:instructions
>> 4- lttng start
>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb
>> -p8 -n8096 -b32
>> 6- lttng stop
>> 7- lttng destroy
>>
>>
>> Can you reproduce if you remove the following contexts ?
>>
>>     lttng add-context -u -t perf:thread:cpu-cycles
>>     lttng add-context -u -t perf:thread:cycles
>>     lttng add-context -u -t perf:thread:instructions
>>
>> And if you only keep a single of those contexts ?
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>>
>>>
>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> Hello All,
>>>
>>> I am trying to instrument a pthread application using the provided
>>> pthread wrapper, but I sometimes run into a "Double free or corruption
>>> error (fasttop)" error.
>>>
>>>
>>> Please provide more information about the version of lttng-ust you are
>>> using, and how you setup
>>> your tracing session.
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>> Here is a description of what I have tried and noticed:
>>> 1- The problem isn't consistent. It sometimes happen and sometimes works
>>> as expected.
>>> 2- From my experiments, the problem happens (more frequently at least)
>>> when adding performance counter contexts (I tried cycles, cpu_cycles
>>> and instructions).
>>> 3- I am testing using lu_ncb from splash3 benchmark suite after setting
>>> LD_PRELOAD to use the pthread wrapper as described in the LTTng
>>> documents.
>>> 4- Here is the backtrace printed after exiting:
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>> 0x35)[0x7ffff7471575]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>> 0x21c)[0x7ffff747363c]
>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>> 0x9f)[0x7ffff746dccf]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>> 0x9f)[0x7ffff746dccf]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>> 0x9f)[0x7ffff746dccf]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>> 0x7ffff746ead8]
>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>> 5- Also, this is a backtrace I obtained from gdb:
>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>     at lttng-events.c:247
>>> #8  0x00007ffff7676906 in lttng_release_session (
>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>     at lttng-ust-abi.c:235
>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>     at lttng-ust-comm.c:1807
>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>     at lttng-ust-comm.c:1874
>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>    from /lib64/ld-linux-x86-64.so.2
>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>    from /lib64/libc.so.6
>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>> argv=<optimized out>) at lu.c:368
>>>
>>> Any ideas, why this happens and how to fix it?
>>>
>>> Thanks,
>>> Shehab
>>>
>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 28034 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 9928 bytes --]

----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

> I run the program as a normal user

> The glibc version I get by running "ldd --version" is 2.17

Can you reproduce the issue after you do this as root ? 

echo "-1" > /proc/sys/kernel/perf_event_paranoid 

Based on this documentation of the Linux kernel: 

Documentation/sysctl/kernel.txt: 

perf_event_paranoid: 

Controls use of the performance events system by unprivileged 
users (without CAP_SYS_ADMIN). The default value is 2. 

-1: Allow use of (almost) all events by all users 
Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK 
>=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN 
Disallow raw tracepoint access by users without CAP_SYS_ADMIN 
>=1: Disallow CPU event access by users without CAP_SYS_ADMIN 
>=2: Disallow kernel profiling by users without CAP_SYS_ADMIN 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>> wrote:

>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>> problem.

>>> What is the setting returned by

>>> cat /proc/sys/kernel/perf_event_paranoid

>>> on your system ? And do you run your test program as root or normal user ?

>>> Please CC the mailing list on your reply.

>> I will also need to know what glibc version you have on your system.

>> Thanks,

>> Mathieu

>>> Thanks,

>>> Mathieu

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> Hi Mathieu,

>>>>>> Thank you very much for your reply.

>>>>>> I manually built lttng-ust from source (commit #:
>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>> This is how I set up the tracing session:
>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>> 3- lttng add-context -u -t procname
>>>>>> lttng add-context -u -t vpid
>>>>>> lttng add-context -u -t pthread_id
>>>>>> lttng add-context -u -t vtid
>>>>>> lttng add-context -u -t ip
>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>> 4- lttng start
>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>> -b32
>>>>>> 6- lttng stop
>>>>>> 7- lttng destroy

>>>>> Can you reproduce if you remove the following contexts ?

>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>> lttng add-context -u -t perf:thread:cycles
>>>>> lttng add-context -u -t perf:thread:instructions

>>>>> And if you only keep a single of those contexts ?

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>> shehabyomn@gmail.com ]

>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>> Hello All,

>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>> )" error.

>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>> and how you setup
>>>>>>> your tracing session.

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>> expected.
>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>> instructions).
>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>> ======= Backtrace : =========
>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>> at lttng -events.c:247
>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>> from /lib64/ libc .so.6
>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>> Thanks,
>>>>>>>> Shehab

>>>>>>>> _______________________________________________
>>>>>>>> lttng-dev mailing list
>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 30352 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 9568 bytes --]

I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made sure the
value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

It executed normally 2 times but then got the same error.

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

>
>
> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>
> I run the program as a normal user
>
> The glibc version I get by running "ldd --version" is 2.17
>
>
> Can you reproduce the issue after you do this as root ?
>
> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>
> Based on this documentation of the Linux kernel:
>
> Documentation/sysctl/kernel.txt:
>
> perf_event_paranoid:
>
> Controls use of the performance events system by unprivileged
> users (without CAP_SYS_ADMIN).  The default value is 2.
>
>  -1: Allow use of (almost) all events by all users
>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>
> Thanks,
>
> Mathieu
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> Yes, I tried with only one of those contexts and I still ran into the
>> same problem.
>>
>> What is the setting returned by
>>
>> cat /proc/sys/kernel/perf_event_paranoid
>>
>> on your system ? And do you run your test program as root or normal user ?
>>
>> Please CC the mailing list on your reply.
>>
>>
>> I will also need to know what glibc version you have on your system.
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> Hi Mathieu,
>>>
>>> Thank you very much for your reply.
>>>
>>> I manually built lttng-ust from source (commit #:
>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>
>>> This is how I set up the tracing session:
>>> 1- lttng create lu_ncb_8_native -o {path}
>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>>> mutex_lock_req
>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>> mutex_lock_acq
>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>> mutex_lock_trylock
>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>> mutex_lock_unlock
>>> 3- lttng add-context -u -t procname
>>>     lttng add-context -u -t vpid
>>>     lttng add-context -u -t pthread_id
>>>     lttng add-context -u -t vtid
>>>     lttng add-context -u -t ip
>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>     lttng add-context -u -t perf:thread:cycles
>>>     lttng add-context -u -t perf:thread:instructions
>>> 4- lttng start
>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb
>>> -p8 -n8096 -b32
>>> 6- lttng stop
>>> 7- lttng destroy
>>>
>>>
>>> Can you reproduce if you remove the following contexts ?
>>>
>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>     lttng add-context -u -t perf:thread:cycles
>>>     lttng add-context -u -t perf:thread:instructions
>>>
>>> And if you only keep a single of those contexts ?
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>>
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>>
>>>>
>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>>> wrote:
>>>>
>>>> Hello All,
>>>>
>>>> I am trying to instrument a pthread application using the provided
>>>> pthread wrapper, but I sometimes run into a "Double free or corruption
>>>> error (fasttop)" error.
>>>>
>>>>
>>>> Please provide more information about the version of lttng-ust you are
>>>> using, and how you setup
>>>> your tracing session.
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>> Here is a description of what I have tried and noticed:
>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>> works as expected.
>>>> 2- From my experiments, the problem happens (more frequently at least)
>>>> when adding performance counter contexts (I tried cycles, cpu_cycles
>>>> and instructions).
>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>> setting LD_PRELOAD to use the pthread wrapper as described in the LTTng
>>>> documents.
>>>> 4- Here is the backtrace printed after exiting:
>>>> ======= Backtrace: =========
>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>>> 0x35)[0x7ffff7471575]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>>> 0x21c)[0x7ffff747363c]
>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>> 0x9f)[0x7ffff746dccf]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>> 0x9f)[0x7ffff746dccf]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>> 0x9f)[0x7ffff746dccf]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>>> 0x7ffff746ead8]
>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>     at lttng-events.c:247
>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>     at lttng-ust-abi.c:235
>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>     at lttng-ust-comm.c:1807
>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>     at lttng-ust-comm.c:1874
>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>    from /lib64/ld-linux-x86-64.so.2
>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>    from /lib64/libc.so.6
>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>> argv=<optimized out>) at lu.c:368
>>>>
>>>> Any ideas, why this happens and how to fix it?
>>>>
>>>> Thanks,
>>>> Shehab
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev@lists.lttng.org
>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 34726 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 11309 bytes --]

----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

> It executed normally 2 times but then got the same error.

Can you provide the output when reproducing the issue with the 
LTTNG_UST_DEBUG=1 environment variable set when starting 
your test program ? 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>> I run the program as a normal user

>>> The glibc version I get by running "ldd --version" is 2.17

>> Can you reproduce the issue after you do this as root ?

>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>> Based on this documentation of the Linux kernel:

>> Documentation/sysctl/kernel.txt:

>> perf_event_paranoid:

>> Controls use of the performance events system by unprivileged
>> users (without CAP_SYS_ADMIN). The default value is 2.

>> -1: Allow use of (almost) all events by all users
>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>> Thanks,

>> Mathieu

>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>> shehabyomn@gmail.com ]

>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>> wrote:

>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>> problem.

>>>>> What is the setting returned by

>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>> Please CC the mailing list on your reply.

>>>> I will also need to know what glibc version you have on your system.

>>>> Thanks,

>>>> Mathieu

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>> shehabyomn@gmail.com ]

>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>> Hi Mathieu,

>>>>>>>> Thank you very much for your reply.

>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>> This is how I set up the tracing session:
>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>> lttng add-context -u -t vpid
>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>> lttng add-context -u -t vtid
>>>>>>>> lttng add-context -u -t ip
>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>> 4- lttng start
>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>> -b32
>>>>>>>> 6- lttng stop
>>>>>>>> 7- lttng destroy

>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>> And if you only keep a single of those contexts ?

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>> PhD Student
>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>> University of Toronto
>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>> wrote:

>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>> Hello All,

>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>> )" error.

>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>> and how you setup
>>>>>>>>> your tracing session.

>>>>>>>>> Thanks,

>>>>>>>>> Mathieu

>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>> expected.
>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>> instructions).
>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>> Thanks,
>>>>>>>>>> Shehab

>>>>>>>>>> _______________________________________________
>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>> --
>>>>>>>>> Mathieu Desnoyers
>>>>>>>>> EfficiOS Inc.
>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 36132 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 12426 bytes --]

----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:

>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>> It executed normally 2 times but then got the same error.

> Can you provide the output when reproducing the issue with the
> LTTNG_UST_DEBUG=1 environment variable set when starting
> your test program ?

I just noticed something that might explain what goes wrong here: 

lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread mutex 
in your case is instrumented with the pthread wrapper. This "add_thread_field" is invoked 
the first time the perf counter is hit by each given thread. When this happens, the 
instrumented pthread mutex will try to call into the instrumentation tracepoint again, 
which will call add_thread_field() (again), and so on until we reach the libringbuffer 
"lib_ring_buffer_nesting" threshold of 4 calls deep. 

I suspect this situation where we recursively call add_thread_field is unexpected, 
which may trigger your double-free here. 

Will investigate more. 

Thanks, 

Mathieu 

> Thanks,

> Mathieu

>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>> shehabyomn@gmail.com ]

>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>> wrote:

>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>> I run the program as a normal user

>>>> The glibc version I get by running "ldd --version" is 2.17

>>> Can you reproduce the issue after you do this as root ?

>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>> Based on this documentation of the Linux kernel:

>>> Documentation/sysctl/kernel.txt:

>>> perf_event_paranoid:

>>> Controls use of the performance events system by unprivileged
>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>> -1: Allow use of (almost) all events by all users
>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>> Thanks,

>>> Mathieu

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>> wrote:

>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>> problem.

>>>>>> What is the setting returned by

>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>> Please CC the mailing list on your reply.

>>>>> I will also need to know what glibc version you have on your system.

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Thanks,

>>>>>> Mathieu

>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>> PhD Student
>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>> University of Toronto
>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>> shehabyomn@gmail.com ]

>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>> wrote:

>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>> Hi Mathieu,

>>>>>>>>> Thank you very much for your reply.

>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>> 4- lttng start
>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>> -b32
>>>>>>>>> 6- lttng stop
>>>>>>>>> 7- lttng destroy

>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>> Thanks,

>>>>>>>> Mathieu

>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>> PhD Student
>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>> University of Toronto
>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>> wrote:

>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>> Hello All,

>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>> )" error.

>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>> and how you setup
>>>>>>>>>> your tracing session.

>>>>>>>>>> Thanks,

>>>>>>>>>> Mathieu

>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>> expected.
>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>> instructions).
>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>> Thanks,
>>>>>>>>>>> Shehab

>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>> --
>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> [ http://www.efficios.com/ | http://www.efficios.com ]

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 37995 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 12907 bytes --]

----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:

>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:

>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>>> It executed normally 2 times but then got the same error.

>> Can you provide the output when reproducing the issue with the
>> LTTNG_UST_DEBUG=1 environment variable set when starting
>> your test program ?

> I just noticed something that might explain what goes wrong here:

> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread
> mutex
> in your case is instrumented with the pthread wrapper. This "add_thread_field"
> is invoked
> the first time the perf counter is hit by each given thread. When this happens,
> the
> instrumented pthread mutex will try to call into the instrumentation tracepoint
> again,
> which will call add_thread_field() (again), and so on until we reach the
> libringbuffer
> "lib_ring_buffer_nesting" threshold of 4 calls deep.

> I suspect this situation where we recursively call add_thread_field is
> unexpected,
> which may trigger your double-free here.

> Will investigate more.

Can you try with the attached patch applied ? 

Thanks, 

Mathieu 

> Thanks,

> Mathieu

>> Thanks,

>> Mathieu

>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>> shehabyomn@gmail.com ]

>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>> wrote:

>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>>> I run the program as a normal user

>>>>> The glibc version I get by running "ldd --version" is 2.17

>>>> Can you reproduce the issue after you do this as root ?

>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>>> Based on this documentation of the Linux kernel:

>>>> Documentation/sysctl/kernel.txt:

>>>> perf_event_paranoid:

>>>> Controls use of the performance events system by unprivileged
>>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>>> -1: Allow use of (almost) all events by all users
>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>>> Thanks,

>>>> Mathieu

>>>>> Shehab Y. Elsayed, MSc.
>>>>> PhD Student
>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>> University of Toronto
>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>> shehabyomn@gmail.com ]

>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>> wrote:

>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>>> problem.

>>>>>>> What is the setting returned by

>>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>>> Please CC the mailing list on your reply.

>>>>>> I will also need to know what glibc version you have on your system.

>>>>>> Thanks,

>>>>>> Mathieu

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>> PhD Student
>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>> University of Toronto
>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>> wrote:

>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>> Hi Mathieu,

>>>>>>>>>> Thank you very much for your reply.

>>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>>> 4- lttng start
>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>>> -b32
>>>>>>>>>> 6- lttng stop
>>>>>>>>>> 7- lttng destroy

>>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>>> Thanks,

>>>>>>>>> Mathieu

>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>> PhD Student
>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>> University of Toronto
>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>> wrote:

>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>> Hello All,

>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>>> )" error.

>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>>> and how you setup
>>>>>>>>>>> your tracing session.

>>>>>>>>>>> Thanks,

>>>>>>>>>>> Mathieu

>>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>>> expected.
>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>>> instructions).
>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Shehab

>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>>> --
>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>> --
>>>>>>>>> Mathieu Desnoyers
>>>>>>>>> EfficiOS Inc.
>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 38754 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-perf-event-mutex-with-pthread-wrapper.patch --]
[-- Type: text/x-patch; name=0001-Fix-perf-event-mutex-with-pthread-wrapper.patch, Size: 5184 bytes --]

From 978e682321963f5b142dcb45e3d1dfe6b263c6ea Mon Sep 17 00:00:00 2001
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Tue, 20 Mar 2018 17:32:36 -0400
Subject: [RFC PATCH] Fix: perf event mutex with pthread wrapper

We do not want to recurse in the pthread mutex instrumentation when
setting up the perf counters for a given thread.

Introduce a "notrace" per-thread counter to inhibit tracing for the
current thread.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/lttng/ust-events.h                 |  3 +++
 liblttng-ust/lttng-context-perf-counters.c |  8 ++++++--
 liblttng-ust/lttng-ring-buffer-client.h    |  3 +++
 liblttng-ust/lttng-tracer-core.h           |  3 +++
 liblttng-ust/lttng-ust-comm.c              | 23 +++++++++++++++++++++++
 5 files changed, 38 insertions(+), 2 deletions(-)

diff --git a/include/lttng/ust-events.h b/include/lttng/ust-events.h
index 86733503..b33a804b 100644
--- a/include/lttng/ust-events.h
+++ b/include/lttng/ust-events.h
@@ -738,6 +738,9 @@ struct lttng_enum *lttng_ust_enum_get_from_desc(struct lttng_session *session,
 void lttng_ust_dl_update(void *ip);
 void lttng_ust_fixup_fd_tracker_tls(void);
 
+void lttng_ust_begin_notrace(void);
+void lttng_ust_end_notrace(void);
+
 /* For backward compatibility. Leave those exported symbols in place. */
 extern struct lttng_ctx *lttng_static_ctx;
 void lttng_context_init(void);
diff --git a/liblttng-ust/lttng-context-perf-counters.c b/liblttng-ust/lttng-context-perf-counters.c
index a15417cc..0ce89245 100644
--- a/liblttng-ust/lttng-context-perf-counters.c
+++ b/liblttng-ust/lttng-context-perf-counters.c
@@ -328,16 +328,20 @@ struct lttng_perf_counter_thread_field *
 	struct lttng_perf_counter_thread *perf_thread;
 	struct lttng_perf_counter_thread_field *thread_field;
 
+	lttng_ust_begin_notrace();
 	perf_thread = pthread_getspecific(perf_counter_key);
 	if (!perf_thread)
 		perf_thread = alloc_perf_counter_thread();
 	cds_list_for_each_entry_rcu(thread_field, &perf_thread->rcu_field_list,
 			rcu_field_node) {
 		if (thread_field->field == field)
-			return thread_field;
+			goto end;
 	}
 	/* perf_counter_thread_field not found, need to add one */
-	return add_thread_field(field, perf_thread);
+	thread_field = add_thread_field(field, perf_thread);
+end:
+	lttng_ust_end_notrace();
+	return thread_field;
 }
 
 static
diff --git a/liblttng-ust/lttng-ring-buffer-client.h b/liblttng-ust/lttng-ring-buffer-client.h
index 0fae8878..0960d4cb 100644
--- a/liblttng-ust/lttng-ring-buffer-client.h
+++ b/liblttng-ust/lttng-ring-buffer-client.h
@@ -724,6 +724,9 @@ int lttng_event_reserve(struct lttng_ust_lib_ring_buffer_ctx *ctx,
 	struct lttng_client_ctx client_ctx;
 	int ret, cpu;
 
+	if (caa_unlikely(URCU_TLS(lttng_ust_notrace_thread)) > 0)
+		return -EPERM;
+
 	/* Compute internal size of context structures. */
 
 	if (lttng_ctx) {
diff --git a/liblttng-ust/lttng-tracer-core.h b/liblttng-ust/lttng-tracer-core.h
index ba232f32..98d79be8 100644
--- a/liblttng-ust/lttng-tracer-core.h
+++ b/liblttng-ust/lttng-tracer-core.h
@@ -25,6 +25,7 @@
 #include <stddef.h>
 #include <urcu/arch.h>
 #include <urcu/list.h>
+#include <urcu/tls-compat.h>
 #include <lttng/ust-tracer.h>
 #include <lttng/bug.h>
 #include <lttng/ringbuffer-config.h>
@@ -37,6 +38,8 @@ struct lttng_ctx_field;
 struct lttng_ust_lib_ring_buffer_ctx;
 struct lttng_ctx_value;
 
+extern DECLARE_URCU_TLS(int, lttng_ust_notrace_thread);
+
 int ust_lock(void) __attribute__ ((warn_unused_result));
 void ust_lock_nocheck(void);
 void ust_unlock(void);
diff --git a/liblttng-ust/lttng-ust-comm.c b/liblttng-ust/lttng-ust-comm.c
index d4add1c0..9520f246 100644
--- a/liblttng-ust/lttng-ust-comm.c
+++ b/liblttng-ust/lttng-ust-comm.c
@@ -92,6 +92,9 @@ static pthread_mutex_t ust_mutex = PTHREAD_MUTEX_INITIALIZER;
 /* Allow nesting the ust_mutex within the same thread. */
 static DEFINE_URCU_TLS(int, ust_mutex_nest);
 
+/* Do not trace events for the current thread. */
+DEFINE_URCU_TLS(int, lttng_ust_notrace_thread) __attribute__((visibility("hidden")));
+
 /*
  * ust_exit_mutex protects thread_active variable wrt thread exit. It
  * cannot be done by ust_mutex because pthread_cancel(), which takes an
@@ -121,6 +124,19 @@ static int lttng_ust_comm_should_quit;
 int lttng_ust_loaded __attribute__((weak));
 
 /*
+ * Inhibit lttng-ust tracing for this thread.
+ */
+void lttng_ust_begin_notrace(void)
+{
+	URCU_TLS(lttng_ust_notrace_thread)++;
+}
+
+void lttng_ust_end_notrace(void)
+{
+	--URCU_TLS(lttng_ust_notrace_thread);
+}
+
+/*
  * Return 0 on success, -1 if should quit.
  * The lock is taken in both cases.
  * Signal-safe.
@@ -392,6 +408,12 @@ void lttng_fixup_ust_mutex_nest_tls(void)
 	asm volatile ("" : : "m" (URCU_TLS(ust_mutex_nest)));
 }
 
+static
+void lttng_fixup_lttng_ust_notrace_tls(void)
+{
+	asm volatile ("" : : "m" (URCU_TLS(lttng_ust_notrace_thread)));
+}
+
 /*
  * Fixup urcu bp TLS.
  */
@@ -410,6 +432,7 @@ void lttng_ust_fixup_tls(void)
 	lttng_fixup_nest_count_tls();
 	lttng_fixup_procname_tls();
 	lttng_fixup_ust_mutex_nest_tls();
+	lttng_fixup_lttng_ust_notrace_tls();
 	lttng_ust_fixup_fd_tracker_tls();
 }
 
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 13484 bytes --]

----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:

>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers
>> <mathieu.desnoyers@efficios.com> wrote:

>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:

>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>>>> It executed normally 2 times but then got the same error.

>>> Can you provide the output when reproducing the issue with the
>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>> your test program ?

>> I just noticed something that might explain what goes wrong here:

>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread
>> mutex
>> in your case is instrumented with the pthread wrapper. This "add_thread_field"
>> is invoked
>> the first time the perf counter is hit by each given thread. When this happens,
>> the
>> instrumented pthread mutex will try to call into the instrumentation tracepoint
>> again,
>> which will call add_thread_field() (again), and so on until we reach the
>> libringbuffer
>> "lib_ring_buffer_nesting" threshold of 4 calls deep.

>> I suspect this situation where we recursively call add_thread_field is
>> unexpected,
>> which may trigger your double-free here.

>> Will investigate more.

> Can you try with the attached patch applied ?

Here is an updated v2 of the patch which tests the notrace tls counter sooner (before evaluating 
filter). Please let me know how it goes. 

Thanks, 

Mathieu 

> Thanks,

> Mathieu

>> Thanks,

>> Mathieu

>>> Thanks,

>>> Mathieu

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>>>> I run the program as a normal user

>>>>>> The glibc version I get by running "ldd --version" is 2.17

>>>>> Can you reproduce the issue after you do this as root ?

>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>>>> Based on this documentation of the Linux kernel:

>>>>> Documentation/sysctl/kernel.txt:

>>>>> perf_event_paranoid:

>>>>> Controls use of the performance events system by unprivileged
>>>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>>>> -1: Allow use of (almost) all events by all users
>>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>> shehabyomn@gmail.com ]

>>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>> wrote:

>>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>>>> problem.

>>>>>>>> What is the setting returned by

>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>>>> Please CC the mailing list on your reply.

>>>>>>> I will also need to know what glibc version you have on your system.

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Thanks,

>>>>>>>> Mathieu

>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>> PhD Student
>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>> University of Toronto
>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>> wrote:

>>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>> Hi Mathieu,

>>>>>>>>>>> Thank you very much for your reply.

>>>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>>>> 4- lttng start
>>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>>>> -b32
>>>>>>>>>>> 6- lttng stop
>>>>>>>>>>> 7- lttng destroy

>>>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>>>> Thanks,

>>>>>>>>>> Mathieu

>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>> PhD Student
>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>> University of Toronto
>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>> wrote:

>>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>> Hello All,

>>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>>>> )" error.

>>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>>>> and how you setup
>>>>>>>>>>>> your tracing session.

>>>>>>>>>>>> Thanks,

>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>>>> expected.
>>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>>>> instructions).
>>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Shehab

>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>>>> --
>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>> --
>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 39888 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-perf-event-mutex-with-pthread-wrapper.patch --]
[-- Type: text/x-patch; name=0001-Fix-perf-event-mutex-with-pthread-wrapper.patch, Size: 6243 bytes --]

From 8671c6ae839615c0779041abada4703680fbc003 Mon Sep 17 00:00:00 2001
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Tue, 20 Mar 2018 17:32:36 -0400
Subject: [RFC PATCH v2] Fix: perf event mutex with pthread wrapper

We do not want to recurse in the pthread mutex instrumentation when
setting up the perf counters for a given thread.

Introduce a "notrace" per-thread counter to inhibit tracing for the
current thread.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 include/lttng/ust-events.h                 |  6 ++++++
 include/lttng/ust-tracepoint-event.h       |  7 ++-----
 liblttng-ust/lttng-context-perf-counters.c |  2 ++
 liblttng-ust/lttng-events.c                | 14 ++++++++++++++
 liblttng-ust/lttng-tracer-core.h           |  3 +++
 liblttng-ust/lttng-ust-comm.c              | 23 +++++++++++++++++++++++
 6 files changed, 50 insertions(+), 5 deletions(-)

diff --git a/include/lttng/ust-events.h b/include/lttng/ust-events.h
index 86733503..f8c130f9 100644
--- a/include/lttng/ust-events.h
+++ b/include/lttng/ust-events.h
@@ -738,6 +738,12 @@ struct lttng_enum *lttng_ust_enum_get_from_desc(struct lttng_session *session,
 void lttng_ust_dl_update(void *ip);
 void lttng_ust_fixup_fd_tracker_tls(void);
 
+void lttng_ust_begin_notrace(void);
+void lttng_ust_end_notrace(void);
+
+int lttng_ust_do_trace(struct lttng_session *session,
+		struct lttng_channel *chan, struct lttng_event *event);
+
 /* For backward compatibility. Leave those exported symbols in place. */
 extern struct lttng_ctx *lttng_static_ctx;
 void lttng_context_init(void);
diff --git a/include/lttng/ust-tracepoint-event.h b/include/lttng/ust-tracepoint-event.h
index ec292d24..0bc002f7 100644
--- a/include/lttng/ust-tracepoint-event.h
+++ b/include/lttng/ust-tracepoint-event.h
@@ -766,11 +766,8 @@ void __event_probe__##_provider##___##_name(_TP_ARGS_DATA_PROTO(_args))	      \
 		(void) __dynamic_len_idx;	/* don't warn if unused */    \
 	if (!_TP_SESSION_CHECK(session, __chan->session))		      \
 		return;							      \
-	if (caa_unlikely(!CMM_ACCESS_ONCE(__chan->session->active)))	      \
-		return;							      \
-	if (caa_unlikely(!CMM_ACCESS_ONCE(__chan->enabled)))		      \
-		return;							      \
-	if (caa_unlikely(!CMM_ACCESS_ONCE(__event->enabled)))		      \
+	if (caa_unlikely(!lttng_ust_do_trace(__chan->session, __chan,	      \
+					     __event)))			      \
 		return;							      \
 	if (caa_unlikely(!TP_RCU_LINK_TEST()))				      \
 		return;							      \
diff --git a/liblttng-ust/lttng-context-perf-counters.c b/liblttng-ust/lttng-context-perf-counters.c
index a15417cc..131aaa54 100644
--- a/liblttng-ust/lttng-context-perf-counters.c
+++ b/liblttng-ust/lttng-context-perf-counters.c
@@ -291,6 +291,7 @@ struct lttng_perf_counter_thread_field *
 	ret = pthread_sigmask(SIG_BLOCK, &newmask, &oldmask);
 	if (ret)
 		abort();
+	lttng_ust_begin_notrace();
 	/* Check again with signals disabled */
 	cds_list_for_each_entry_rcu(thread_field, &perf_thread->rcu_field_list,
 			rcu_field_node) {
@@ -315,6 +316,7 @@ struct lttng_perf_counter_thread_field *
 			&perf_field->thread_field_list);
 	ust_unlock();
 skip:
+	lttng_ust_end_notrace();
 	ret = pthread_sigmask(SIG_SETMASK, &oldmask, NULL);
 	if (ret)
 		abort();
diff --git a/liblttng-ust/lttng-events.c b/liblttng-ust/lttng-events.c
index 255c4b95..d8e4fd79 100644
--- a/liblttng-ust/lttng-events.c
+++ b/liblttng-ust/lttng-events.c
@@ -1284,3 +1284,17 @@ void lttng_ust_context_set_session_provider(const char *name,
 		}
 	}
 }
+
+int lttng_ust_do_trace(struct lttng_session *session,
+		struct lttng_channel *chan, struct lttng_event *event)
+{
+	if (caa_unlikely(!CMM_ACCESS_ONCE(session->active)))
+		return 0;
+	if (caa_unlikely(!CMM_ACCESS_ONCE(chan->enabled)))
+		return 0;
+	if (caa_unlikely(!CMM_ACCESS_ONCE(event->enabled)))
+		return 0;
+	if (caa_unlikely(URCU_TLS(lttng_ust_notrace_thread)) > 0)
+		return 0;
+	return 1;
+}
diff --git a/liblttng-ust/lttng-tracer-core.h b/liblttng-ust/lttng-tracer-core.h
index ba232f32..98d79be8 100644
--- a/liblttng-ust/lttng-tracer-core.h
+++ b/liblttng-ust/lttng-tracer-core.h
@@ -25,6 +25,7 @@
 #include <stddef.h>
 #include <urcu/arch.h>
 #include <urcu/list.h>
+#include <urcu/tls-compat.h>
 #include <lttng/ust-tracer.h>
 #include <lttng/bug.h>
 #include <lttng/ringbuffer-config.h>
@@ -37,6 +38,8 @@ struct lttng_ctx_field;
 struct lttng_ust_lib_ring_buffer_ctx;
 struct lttng_ctx_value;
 
+extern DECLARE_URCU_TLS(int, lttng_ust_notrace_thread);
+
 int ust_lock(void) __attribute__ ((warn_unused_result));
 void ust_lock_nocheck(void);
 void ust_unlock(void);
diff --git a/liblttng-ust/lttng-ust-comm.c b/liblttng-ust/lttng-ust-comm.c
index d4add1c0..9520f246 100644
--- a/liblttng-ust/lttng-ust-comm.c
+++ b/liblttng-ust/lttng-ust-comm.c
@@ -92,6 +92,9 @@ static pthread_mutex_t ust_mutex = PTHREAD_MUTEX_INITIALIZER;
 /* Allow nesting the ust_mutex within the same thread. */
 static DEFINE_URCU_TLS(int, ust_mutex_nest);
 
+/* Do not trace events for the current thread. */
+DEFINE_URCU_TLS(int, lttng_ust_notrace_thread) __attribute__((visibility("hidden")));
+
 /*
  * ust_exit_mutex protects thread_active variable wrt thread exit. It
  * cannot be done by ust_mutex because pthread_cancel(), which takes an
@@ -121,6 +124,19 @@ static int lttng_ust_comm_should_quit;
 int lttng_ust_loaded __attribute__((weak));
 
 /*
+ * Inhibit lttng-ust tracing for this thread.
+ */
+void lttng_ust_begin_notrace(void)
+{
+	URCU_TLS(lttng_ust_notrace_thread)++;
+}
+
+void lttng_ust_end_notrace(void)
+{
+	--URCU_TLS(lttng_ust_notrace_thread);
+}
+
+/*
  * Return 0 on success, -1 if should quit.
  * The lock is taken in both cases.
  * Signal-safe.
@@ -392,6 +408,12 @@ void lttng_fixup_ust_mutex_nest_tls(void)
 	asm volatile ("" : : "m" (URCU_TLS(ust_mutex_nest)));
 }
 
+static
+void lttng_fixup_lttng_ust_notrace_tls(void)
+{
+	asm volatile ("" : : "m" (URCU_TLS(lttng_ust_notrace_thread)));
+}
+
 /*
  * Fixup urcu bp TLS.
  */
@@ -410,6 +432,7 @@ void lttng_ust_fixup_tls(void)
 	lttng_fixup_nest_count_tls();
 	lttng_fixup_procname_tls();
 	lttng_fixup_ust_mutex_nest_tls();
+	lttng_fixup_lttng_ust_notrace_tls();
 	lttng_ust_fixup_fd_tracker_tls();
 }
 
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 12807 bytes --]

I am so sorry for the late replies.

I have tried the first patch you sent and the problem still happens
(although seemingly less frequently especially with debugging enabled!!!!).
I have attached the output I got from one of the erroneous runs.

I will try the updated patch and let you know how it goes.

Also, I am not sure if these points make any difference:
1- The error actually happens at the end of the application. It actually
finishes execution, but then something goes wrong.
2- I run into this problem only for some of the benchmarks (or at least the
problems happens much less frequently for others that I didn't run into it,
not yet)

Thanks you very much, and sorry again for the late replies.

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>
> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made sure
> the value was actually written by "cat /proc/sys/kernel/perf_event_
> paranoid"
>
> It executed normally 2 times but then got the same error.
>
>
> Can you provide the output when reproducing the issue with the
> LTTNG_UST_DEBUG=1 environment variable set when starting
> your test program ?
>
> I just noticed something that might explain what goes wrong here:
>
> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock().
> Pthread mutex
> in your case is instrumented with the pthread wrapper. This
> "add_thread_field" is invoked
> the first time the perf counter is hit by each given thread. When this
> happens, the
> instrumented pthread mutex will try to call into the instrumentation
> tracepoint again,
> which will call add_thread_field() (again), and so on until we reach the
> libringbuffer
> "lib_ring_buffer_nesting" threshold of 4 calls deep.
>
> I suspect this situation where we recursively call add_thread_field is
> unexpected,
> which may trigger your double-free here.
>
> Will investigate more.
>
> Can you try with the attached patch applied ?
>
> Here is an updated v2 of the patch which tests the notrace tls counter
> sooner (before evaluating
> filter). Please let me know how it goes.
>
> Thanks,
>
> Mathieu
>
>
>
>
>
> Thanks,
>
> Mathieu
>
>
>
>
>
> Thanks,
>
> Mathieu
>
>
>
>
> Thanks,
>
> Mathieu
>
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>>
>>
>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>>
>> I run the program as a normal user
>>
>> The glibc version I get by running "ldd --version" is 2.17
>>
>>
>> Can you reproduce the issue after you do this as root ?
>>
>> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>>
>> Based on this documentation of the Linux kernel:
>>
>> Documentation/sysctl/kernel.txt:
>>
>> perf_event_paranoid:
>>
>> Controls use of the performance events system by unprivileged
>> users (without CAP_SYS_ADMIN).  The default value is 2.
>>
>>  -1: Allow use of (almost) all events by all users
>>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> Yes, I tried with only one of those contexts and I still ran into the
>>> same problem.
>>>
>>> What is the setting returned by
>>>
>>> cat /proc/sys/kernel/perf_event_paranoid
>>>
>>> on your system ? And do you run your test program as root or normal user
>>> ?
>>>
>>> Please CC the mailing list on your reply.
>>>
>>>
>>> I will also need to know what glibc version you have on your system.
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <
>>>> shehabyomn@gmail.com> wrote:
>>>>
>>>> Hi Mathieu,
>>>>
>>>> Thank you very much for your reply.
>>>>
>>>> I manually built lttng-ust from source (commit #:
>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>>
>>>> This is how I set up the tracing session:
>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>> mutex_lock_req
>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>> mutex_lock_acq
>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>> mutex_lock_trylock
>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>> mutex_lock_unlock
>>>> 3- lttng add-context -u -t procname
>>>>     lttng add-context -u -t vpid
>>>>     lttng add-context -u -t pthread_id
>>>>     lttng add-context -u -t vtid
>>>>     lttng add-context -u -t ip
>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>     lttng add-context -u -t perf:thread:cycles
>>>>     lttng add-context -u -t perf:thread:instructions
>>>> 4- lttng start
>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb
>>>> -p8 -n8096 -b32
>>>> 6- lttng stop
>>>> 7- lttng destroy
>>>>
>>>>
>>>> Can you reproduce if you remove the following contexts ?
>>>>
>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>     lttng add-context -u -t perf:thread:cycles
>>>>     lttng add-context -u -t perf:thread:instructions
>>>>
>>>> And if you only keep a single of those contexts ?
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: shehabyomn@gmail.com
>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>
>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <
>>>>> shehabyomn@gmail.com> wrote:
>>>>>
>>>>> Hello All,
>>>>>
>>>>> I am trying to instrument a pthread application using the provided
>>>>> pthread wrapper, but I sometimes run into a "Double free or
>>>>> corruption error (fasttop)" error.
>>>>>
>>>>>
>>>>> Please provide more information about the version of lttng-ust you are
>>>>> using, and how you setup
>>>>> your tracing session.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>> Here is a description of what I have tried and noticed:
>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>>> works as expected.
>>>>> 2- From my experiments, the problem happens (more frequently at least)
>>>>> when adding performance counter contexts (I tried cycles, cpu_cycles
>>>>> and instructions).
>>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>>> setting LD_PRELOAD to use the pthread wrapper as described in the
>>>>> LTTng documents.
>>>>> 4- Here is the backtrace printed after exiting:
>>>>> ======= Backtrace: =========
>>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>>>> 0x35)[0x7ffff7471575]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>>>> 0x21c)[0x7ffff747363c]
>>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>> 0x9f)[0x7ffff746dccf]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>> 0x9f)[0x7ffff746dccf]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>> 0x9f)[0x7ffff746dccf]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>>>> 0x7ffff746ead8]
>>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
>>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>>     at lttng-events.c:247
>>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>>     at lttng-ust-abi.c:235
>>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>>     at lttng-ust-comm.c:1807
>>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>>     at lttng-ust-comm.c:1874
>>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>>    from /lib64/ld-linux-x86-64.so.2
>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>    from /lib64/libc.so.6
>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>>> argv=<optimized out>) at lu.c:368
>>>>>
>>>>> Any ideas, why this happens and how to fix it?
>>>>>
>>>>> Thanks,
>>>>> Shehab
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> lttng-dev@lists.lttng.org
>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 45702 bytes --]

[-- Attachment #2: out.dbg --]
[-- Type: application/octet-stream, Size: 57060 bytes --]

Session lu_ncb_8_native created.
Traces will be written in /aenao-99/elsayed9/LTTng/data/lttng-traces/double-free-fix/lu_ncb_8_native
UST channel my-channel enabled for session lu_ncb_8_native
UST event lttng_ust_pthread:pthread_mutex_lock_req created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_lock_acq created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_trylock created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_unlock created in channel my-channel
UST context procname added to channel my-channel
UST context vpid added to channel my-channel
UST context pthread_id added to channel my-channel
UST context vtid added to channel my-channel
UST context ip added to channel my-channel
UST context perf:thread:cpu-cycles added to channel my-channel
UST context perf:thread:cycles added to channel my-channel
Tracing started for session lu_ncb_8_native
Running: LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096 -b32
liblttng_ust_tracepoint[2897/2897]: Your compiler treats weak symbols with hidden visibility for integer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at tracepoint.c:916)
liblttng_ust_tracepoint[2897/2897]: Your compiler treats weak symbols with hidden visibility for pointer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at tracepoint.c:920)
liblttng_ust_tracepoint[2897/2897]: Your compiler treats weak symbols with hidden visibility for 24-byte structure objects as SAME address between compile units part of the same module. (in check_weak_hidden() at tracepoint.c:924)
liblttng_ust_tracepoint[2897/2897]: just registered a tracepoints section from 0x7f926b6f4240 and having 25 tracepoints (in tracepoint_register_lib() at tracepoint.c:867)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_statedump:end (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_statedump:debug_link (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_statedump:build_id (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_statedump:bin_info (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_statedump:start (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_lib:unload (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_lib:debug_link (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_lib:build_id (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_lib:load (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracef:event (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_LINE (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_UNIT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_MODULE (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROCESS (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_INFO (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_NOTICE (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_WARNING (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_ERR (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_CRIT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_ALERT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_tracelog:TRACE_EMERG (in tracepoint_register_lib() at tracepoint.c:872)
libust[2897/2897]: Provider "lttng_ust_statedump" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: adding probe lttng_ust_statedump containing 5 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: LTT : ltt ring buffer client "relay-metadata-mmap" init
 (in lttng_ring_buffer_metadata_client_init() at lttng-ring-buffer-metadata-client.h:352)
libust[2897/2897]: LTT : ltt ring buffer client "relay-overwrite-mmap" init
 (in lttng_ring_buffer_client_overwrite_init() at lttng-ring-buffer-client.h:871)
libust[2897/2897]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init
 (in lttng_ring_buffer_client_overwrite_rt_init() at lttng-ring-buffer-client.h:871)
libust[2897/2897]: LTT : ltt ring buffer client "relay-discard-mmap" init
 (in lttng_ring_buffer_client_discard_init() at lttng-ring-buffer-client.h:871)
libust[2897/2897]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init
 (in lttng_ring_buffer_client_discard_rt_init() at lttng-ring-buffer-client.h:871)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2899]: Message Received "Get Tracer Version" (65), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)                                                                                                                                                                                                                                                                                                                   [386/1809]
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Session" (64), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 1 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Channel" (81), Handle "session" (1) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: channel data received (in handle_message() at lttng-ust-comm.c:834)
libust[2897/2899]: Return value: 2 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 3 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Enable" (128), Handle "enabler" (3) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 4 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Enable" (128), Handle "enabler" (4) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 5 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Enable" (128), Handle "enabler" (5) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 6 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Enable" (128), Handle "enabler" (6) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Enable" (128), Handle "session" (1) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Sent register channel notification: chan_id 0, header_type 1
 (in ustcomm_register_channel() at lttng-ust-comm.c:1511)
libust[2897/2899]: just registered probe lttng_ust_statedump containing 5 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[2897/2899]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Wait for Quiescent State" (67), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2899]: Message Received "Registration Done" (68), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:453)
libust[2897/2899]: Return value: 0 (in handle_message() at lttng-ust-comm.c:995)
libust[2897/2897]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: adding probe lttng_ust_lib containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: just registered probe lttng_ust_lib containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
libust[2897/2897]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2899]: message successfully sent (in send_reply() at lttng-ust-comm.c:594)
libust[2897/2897]: adding probe lttng_ust_tracef containing 1 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: just registered probe lttng_ust_tracef containing 1 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
libust[2897/2897]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: adding probe lttng_ust_tracelog containing 15 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: just registered probe lttng_ust_tracelog containing 15 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
liblttng_ust_tracepoint[2897/2897]: just registered a tracepoints section from 0x7f926bc05128 and having 4 tracepoints (in tracepoint_register_lib() at tracepoint.c:867)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_pthread:pthread_mutex_unlock (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_pthread:pthread_mutex_trylock (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_pthread:pthread_mutex_lock_acq (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: lttng_ust_pthread:pthread_mutex_lock_req (in tracepoint_register_lib() at tracepoint.c:872)
libust[2897/2897]: Provider "lttng_ust_pthread" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: adding probe lttng_ust_pthread containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: just registered probe lttng_ust_pthread containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
libust[2897/2897]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_lock_req": ret_code 0, event_id 0
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[2897/2897]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_trylock": ret_code 0, event_id 1
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[2897/2897]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_unlock": ret_code 0, event_id 2
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[2897/2897]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_lock_acq": ret_code 0, event_id 3
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
liblttng_ust_tracepoint[2897/2897]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_lock_acq. Queuing release. (in __tracepoint_probe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[2897/2897]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_unlock. Queuing release. (in __tracepoint_probe_register_queue_release() at tracepoint.c:611)
libust[2897/2897]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
liblttng_ust_tracepoint[2897/2897]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_trylock. Queuing release. (in __tracepoint_probe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[2897/2897]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_lock_req. Queuing release. (in __tracepoint_probe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
libust[2897/2897]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
liblttng_ust_tracepoint[2897/2897]: just registered a tracepoints section from 0x6072f8 and having 4 tracepoints (in tracepoint_register_lib() at tracepoint.c:867)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: splash3:roi_begin (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: splash3:roi_end (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: splash3:thread_begin (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[2897/2897]: registered tracepoint: splash3:thread_end (in tracepoint_register_lib() at tracepoint.c:872)
libust[2897/2897]: Provider "splash3" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: adding probe splash3 containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[2897/2897]: just registered probe splash3 containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)

Blocked Dense LU Factorization
     8096 by 8096 Matrix
     8 Processors
     32 by 32 Element Blocks



libust[2897/2898]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2898]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1447)
libust[2897/2898]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1329)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2900]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2902]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_lock_nocheck() at lttng-ust-comm.c:186)
libust[2897/2904]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2906]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2905]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2903]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
libust[2897/2901]: Error: pthread_setcancelstate: unexpected oldstate (in ust_unlock() at lttng-ust-comm.c:225)
                            PROCESS STATISTICS
              Total      Diagonal     Perimeter      Interior       Barrier
 Proc         Time         Time         Time           Time          Time
    0            34             0             1            33             0

                            TIMING INFORMATION
Start time                        :       1521647935
Initialization finish time        :       1521647936
Overall finish time               :       1521647970
Total time with initialization    :               35
Total time without initialization :               34

libust[2897/2897]: Provider "splash3" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: just unregistered probe splash3 (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[2897/2897]: just unregistered a tracepoints section from 0x6072f8 (in tracepoint_unregister_lib() at tracepoint.c:897)
libust[2897/2897]: Provider "lttng_ust_pthread" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: just unregistered probe lttng_ust_pthread (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[2897/2897]: just unregistered a tracepoints section from 0x7f926bc05128 (in tracepoint_unregister_lib() at tracepoint.c:897)
libust[2897/2897]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: just unregistered probe lttng_ust_tracelog (in lttng_probe_unregister() at lttng-probes.c:250)
libust[2897/2897]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: just unregistered probe lttng_ust_tracef (in lttng_probe_unregister() at lttng-probes.c:250)
libust[2897/2897]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lttng-probes.c:175)
libust[2897/2897]: just unregistered probe lttng_ust_lib (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[2897/2897]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_lock_acq. Queuing release. (in __tracepoint_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[2897/2897]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_unlock. Queuing release. (in __tracepoint_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[2897/2897]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_trylock. Queuing release. (in __tracepoint_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[2897/2897]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_lock_req. Queuing release. (in __tracepoint_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[2897/2897]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c:702)
*** Error in `./lu_ncb': double free or corruption (fasttop): 0x00007f91e00009f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f926ad1c503]
/usr/local/lib/liblttng-ust.so.0(+0x32e55)[0x7f926b4b3e55]
/usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+0x35)[0x7f926b4a36a5]
/usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+0x21c)[0x7f926b4a576c]
/usr/local/lib/liblttng-ust.so.0(+0x1ea36)[0x7f926b49fa36]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f926b49fdff]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f926b49fdff]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f926b49fdff]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x7f926b4a0c08]
/usr/local/lib/liblttng-ust.so.0(+0x192b3)[0x7f926b49a2b3]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7f926b490e37]
/lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7f926bc1585a]
/lib64/libc.so.6(+0x38a49)[0x7f926acd8a49]
/lib64/libc.so.6(+0x38a95)[0x7f926acd8a95]
./lu_ncb[0x401b51]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f926acc1b35]
./lu_ncb[0x401c44]
======= Memory map: ========                                                                                                                                                                                                                                                                                                                                                                                             [37/1809]
00400000-00407000 r-xp 00000000 00:2e 233750214                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
00606000-00607000 r--p 00006000 00:2e 233750214                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
00607000-00608000 rw-p 00007000 00:2e 233750214                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
0224e000-02291000 rw-p 00000000 00:00 0                                  [heap]
7f91c8000000-7f91c8021000 rw-p 00000000 00:00 0 
7f91c8021000-7f91cc000000 ---p 00000000 00:00 0 
7f91d0000000-7f91d0021000 rw-p 00000000 00:00 0 
7f91d0021000-7f91d4000000 ---p 00000000 00:00 0 
7f91d4000000-7f91d4021000 rw-p 00000000 00:00 0 
7f91d4021000-7f91d8000000 ---p 00000000 00:00 0 
7f91d8000000-7f91d8021000 rw-p 00000000 00:00 0 
7f91d8021000-7f91dc000000 ---p 00000000 00:00 0 
7f91dc000000-7f91dc021000 rw-p 00000000 00:00 0 
7f91dc021000-7f91e0000000 ---p 00000000 00:00 0 
7f91e0000000-7f91e0021000 rw-p 00000000 00:00 0 
7f91e0021000-7f91e4000000 ---p 00000000 00:00 0 
7f91e4000000-7f91e4021000 rw-p 00000000 00:00 0 
7f91e4021000-7f91e8000000 ---p 00000000 00:00 0 
7f91ec000000-7f91ec021000 rw-p 00000000 00:00 0 
7f91ec021000-7f91f0000000 ---p 00000000 00:00 0 
7f91f3bab000-7f91f3bac000 ---p 00000000 00:00 0 
7f91f3bac000-7f91f43ac000 rw-p 00000000 00:00 0 
7f91f43ac000-7f91f43ad000 ---p 00000000 00:00 0 
7f91f43ad000-7f9213fc0000 rw-p 00000000 00:00 0 
7f9213fc0000-7f92167c2000 rw-s 00000000 00:12 4000908                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92167c2000-7f9218fc4000 rw-s 00000000 00:12 4000907                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9218fc4000-7f921b7c6000 rw-s 00000000 00:12 4000906                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f921b7c6000-7f921dfc8000 rw-s 00000000 00:12 4000905                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f921dfc8000-7f92207ca000 rw-s 00000000 00:12 4000904                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92207ca000-7f9222fcc000 rw-s 00000000 00:12 4000903                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9222fcc000-7f92257ce000 rw-s 00000000 00:12 4000902                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92257ce000-7f9227fd0000 rw-s 00000000 00:12 4000901                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9227fd0000-7f922a7d2000 rw-s 00000000 00:12 4000900                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f922a7d2000-7f922cfd4000 rw-s 00000000 00:12 4000899                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f922cfd4000-7f922f7d6000 rw-s 00000000 00:12 4000898                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f922f7d6000-7f9231fd8000 rw-s 00000000 00:12 4000897                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9231fd8000-7f92347da000 rw-s 00000000 00:12 4000896                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92347da000-7f9236fdc000 rw-s 00000000 00:12 4000895                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9236fdc000-7f92397de000 rw-s 00000000 00:12 4000894                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92397de000-7f923bfe0000 rw-s 00000000 00:12 4000893                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f923bfe0000-7f923e7e2000 rw-s 00000000 00:12 4000892                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f923e7e2000-7f9240fe4000 rw-s 00000000 00:12 4000891                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9240fe4000-7f92437e6000 rw-s 00000000 00:12 4000890                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92437e6000-7f9245fe8000 rw-s 00000000 00:12 4000889                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9245fe8000-7f92487ea000 rw-s 00000000 00:12 4000888                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92487ea000-7f924afec000 rw-s 00000000 00:12 4000887                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f924afec000-7f924d7ee000 rw-s 00000000 00:12 4000886                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f924d7ee000-7f924fff0000 rw-s 00000000 00:12 4000885                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f924fff0000-7f92527f2000 rw-s 00000000 00:12 4000884                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92527f2000-7f9254ff4000 rw-s 00000000 00:12 4000883                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9254ff4000-7f92577f6000 rw-s 00000000 00:12 4000882                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92577f6000-7f9259ff8000 rw-s 00000000 00:12 4000881                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9259ff8000-7f925c7fa000 rw-s 00000000 00:12 4000880                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f925c7fa000-7f925effc000 rw-s 00000000 00:12 4000879                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f925effc000-7f92617fe000 rw-s 00000000 00:12 4000878                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f92617fe000-7f9264000000 rw-s 00000000 00:12 4000877                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f9264000000-7f9264021000 rw-p 00000000 00:00 0 
7f9264021000-7f9268000000 ---p 00000000 00:00 0 
7f9268642000-7f9268643000 ---p 00000000 00:00 0 
7f9268643000-7f9268e43000 rw-p 00000000 00:00 0 
7f9268e43000-7f9268e44000 ---p 00000000 00:00 0 
7f9268e44000-7f9269644000 rw-p 00000000 00:00 0 
7f9269644000-7f9269645000 ---p 00000000 00:00 0 
7f9269645000-7f9269e45000 rw-p 00000000 00:00 0                          [stack:2898]
7f9269e45000-7f9269e5a000 r-xp 00000000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f9269e5a000-7f926a059000 ---p 00015000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f926a059000-7f926a05a000 r--p 00014000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f926a05a000-7f926a05b000 rw-p 00015000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f926a05b000-7f926a062000 r-xp 00000000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f926a062000-7f926a261000 ---p 00007000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f926a261000-7f926a262000 r--p 00006000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f926a262000-7f926a263000 rw-p 00007000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f926a263000-7f926a26a000 r-xp 00000000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f926a26a000-7f926a469000 ---p 00007000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f926a469000-7f926a46a000 r--p 00006000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f926a46a000-7f926a46b000 rw-p 00007000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f926a46b000-7f926a475000 r-xp 00000000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f926a475000-7f926a675000 ---p 0000a000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f926a675000-7f926a676000 r--p 0000a000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f926a676000-7f926a677000 rw-p 0000b000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f926a677000-7f926a67b000 r-xp 00000000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f926a67b000-7f926a87a000 ---p 00004000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f926a87a000-7f926a87b000 r--p 00003000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f926a87b000-7f926a87c000 rw-p 00004000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f926a87c000-7f926a883000 r-xp 00000000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f926a883000-7f926aa82000 ---p 00007000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f926aa82000-7f926aa83000 r--p 00006000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f926aa83000-7f926aa84000 rw-p 00007000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f926aa84000-7f926aa8f000 r-xp 00000000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f926aa8f000-7f926ac8e000 ---p 0000b000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f926ac8e000-7f926ac8f000 r--p 0000a000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f926ac8f000-7f926ac90000 rw-p 0000b000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f926ac90000-7f926aca0000 rw-p 00000000 00:00 0 
7f926aca0000-7f926ae56000 r-xp 00000000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f926ae56000-7f926b056000 ---p 001b6000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f926b056000-7f926b05a000 r--p 001b6000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f926b05a000-7f926b05c000 rw-p 001ba000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f926b05c000-7f926b061000 rw-p 00000000 00:00 0 
7f926b061000-7f926b078000 r-xp 00000000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f926b078000-7f926b277000 ---p 00017000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f926b277000-7f926b278000 r--p 00016000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f926b278000-7f926b279000 rw-p 00017000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f926b279000-7f926b27d000 rw-p 00000000 00:00 0 
7f926b27d000-7f926b27f000 r-xp 00000000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f926b27f000-7f926b47f000 ---p 00002000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f926b47f000-7f926b480000 r--p 00002000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f926b480000-7f926b481000 rw-p 00003000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f926b481000-7f926b4e9000 r-xp 00000000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f926b4e9000-7f926b6e8000 ---p 00068000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f926b6e8000-7f926b6ef000 r--p 00067000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f926b6ef000-7f926b6f5000 rw-p 0006e000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f926b6f5000-7f926b6fe000 rw-p 00000000 00:00 0 
7f926b6fe000-7f926b7fe000 r-xp 00000000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f926b7fe000-7f926b9fe000 ---p 00100000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f926b9fe000-7f926b9ff000 r--p 00100000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f926b9ff000-7f926ba00000 rw-p 00101000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f926ba00000-7f926ba04000 r-xp 00000000 fd:00 269953026                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f926ba04000-7f926bc03000 ---p 00004000 fd:00 269953026                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f926bc03000-7f926bc05000 r--p 00003000 fd:00 269953026                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f926bc05000-7f926bc06000 rw-p 00005000 fd:00 269953026                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f926bc06000-7f926bc26000 r-xp 00000000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f926bdfc000-7f926bdfd000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926bdfd000-7f926bdfe000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926be05000-7f926be06000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926be07000-7f926be08000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926be0c000-7f926be16000 rw-p 00000000 00:00 0 
7f926be1c000-7f926be1d000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926be1f000-7f926be20000 rw-p 00000000 00:00 0 
7f926be20000-7f926be21000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f926be21000-7f926be22000 rw-p 00000000 00:00 0 
7f926be22000-7f926be23000 r--s 00000000 00:12 139034                     /dev/shm/lttng-ust-wait-7
7f926be23000-7f926be25000 rw-p 00000000 00:00 0 
7f926be25000-7f926be26000 r--p 0001f000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f926be26000-7f926be27000 rw-p 00020000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f926be27000-7f926be28000 rw-p 00000000 00:00 0 
7ffc4787b000-7ffc4789c000 rw-p 00000000 00:00 0                          [stack]
7ffc47901000-7ffc47903000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
./run_benchmark.sh: line 88:  2897 Aborted                 (core dumped) LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096 -b32
Waiting for data availability.
Tracing stopped for session lu_ncb_8_native
Session lu_ncb_8_native destroyed 


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 15882 bytes --]

----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> I am so sorry for the late replies.

> I have tried the first patch you sent and the problem still happens (although
> seemingly less frequently especially with debugging enabled!!!!). I have
> attached the output I got from one of the erroneous runs.

> I will try the updated patch and let you know how it goes.

> Also, I am not sure if these points make any difference:
> 1- The error actually happens at the end of the application. It actually
> finishes execution, but then something goes wrong.
> 2- I run into this problem only for some of the benchmarks (or at least the
> problems happens much less frequently for others that I didn't run into it, not
> yet)

> Thanks you very much, and sorry again for the late replies.

No worries! Looking through your log, I notice that pthread set cancel state has problems when 
called from application threads. We do not restore the original thread's cancel state. Can you try 
with the attached patch applied on top of the previous one ? 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers < [
>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>> wrote:

>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>> wrote:

>>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>>>>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>>>>>> It executed normally 2 times but then got the same error.

>>>>> Can you provide the output when reproducing the issue with the
>>>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>>>> your test program ?

>>>> I just noticed something that might explain what goes wrong here:

>>>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread
>>>> mutex
>>>> in your case is instrumented with the pthread wrapper. This "add_thread_field"
>>>> is invoked
>>>> the first time the perf counter is hit by each given thread. When this happens,
>>>> the
>>>> instrumented pthread mutex will try to call into the instrumentation tracepoint
>>>> again,
>>>> which will call add_thread_field() (again), and so on until we reach the
>>>> libringbuffer
>>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.

>>>> I suspect this situation where we recursively call add_thread_field is
>>>> unexpected,
>>>> which may trigger your double-free here.

>>>> Will investigate more.

>>> Can you try with the attached patch applied ?

>> Here is an updated v2 of the patch which tests the notrace tls counter sooner
>> (before evaluating
>> filter). Please let me know how it goes.

>> Thanks,

>> Mathieu

>>> Thanks,

>>> Mathieu

>>>> Thanks,

>>>> Mathieu

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>> shehabyomn@gmail.com ]

>>>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>>>>>> I run the program as a normal user

>>>>>>>> The glibc version I get by running "ldd --version" is 2.17

>>>>>>> Can you reproduce the issue after you do this as root ?

>>>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>>>>>> Based on this documentation of the Linux kernel:

>>>>>>> Documentation/sysctl/kernel.txt:

>>>>>>> perf_event_paranoid:

>>>>>>> Controls use of the performance events system by unprivileged
>>>>>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>>>>>> -1: Allow use of (almost) all events by all users
>>>>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>> PhD Student
>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>> University of Toronto
>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>> wrote:

>>>>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>> wrote:

>>>>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>>>>>> problem.

>>>>>>>>>> What is the setting returned by

>>>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>>>>>> Please CC the mailing list on your reply.

>>>>>>>>> I will also need to know what glibc version you have on your system.

>>>>>>>>> Thanks,

>>>>>>>>> Mathieu

>>>>>>>>>> Thanks,

>>>>>>>>>> Mathieu

>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>> PhD Student
>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>> University of Toronto
>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>> wrote:

>>>>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>> Hi Mathieu,

>>>>>>>>>>>>> Thank you very much for your reply.

>>>>>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>>>>>> 4- lttng start
>>>>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>>>>>> -b32
>>>>>>>>>>>>> 6- lttng stop
>>>>>>>>>>>>> 7- lttng destroy

>>>>>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>>>>>> Thanks,

>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>> Hello All,

>>>>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>>>>>> )" error.

>>>>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>>>>>> and how you setup
>>>>>>>>>>>>>> your tracing session.

>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>>>>>> expected.
>>>>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>>>>>> instructions).
>>>>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Shehab

>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>> --
>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>> --
>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>> --
>>>>>>>>> Mathieu Desnoyers
>>>>>>>>> EfficiOS Inc.
>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 47259 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-restore-original-thread-cancel-state.patch --]
[-- Type: text/x-patch; name=0001-Fix-restore-original-thread-cancel-state.patch, Size: 3133 bytes --]

From ac5195f4bf95e24bd5b4fe1d9ccc7cccbe44dcdd Mon Sep 17 00:00:00 2001
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Wed, 21 Mar 2018 16:16:38 -0400
Subject: [RFC PATCH] Fix: restore original thread cancel state

Useful when ust_lock is used in application threads that have a
different cancel state (e.g. perf counters context).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 liblttng-ust/lttng-ust-comm.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/liblttng-ust/lttng-ust-comm.c b/liblttng-ust/lttng-ust-comm.c
index 9520f246..8c5985a9 100644
--- a/liblttng-ust/lttng-ust-comm.c
+++ b/liblttng-ust/lttng-ust-comm.c
@@ -92,6 +92,8 @@ static pthread_mutex_t ust_mutex = PTHREAD_MUTEX_INITIALIZER;
 /* Allow nesting the ust_mutex within the same thread. */
 static DEFINE_URCU_TLS(int, ust_mutex_nest);
 
+static DEFINE_URCU_TLS(int, pthread_cancel_oldstate);
+
 /* Do not trace events for the current thread. */
 DEFINE_URCU_TLS(int, lttng_ust_notrace_thread) __attribute__((visibility("hidden")));
 
@@ -144,15 +146,13 @@ void lttng_ust_end_notrace(void)
 int ust_lock(void)
 {
 	sigset_t sig_all_blocked, orig_mask;
-	int ret, oldstate;
+	int ret;
 
-	ret = pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate);
+	ret = pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,
+		&pthread_cancel_oldstate);
 	if (ret) {
 		ERR("pthread_setcancelstate: %s", strerror(ret));
 	}
-	if (oldstate != PTHREAD_CANCEL_ENABLE) {
-		ERR("pthread_setcancelstate: unexpected oldstate");
-	}
 	sigfillset(&sig_all_blocked);
 	ret = pthread_sigmask(SIG_SETMASK, &sig_all_blocked, &orig_mask);
 	if (ret) {
@@ -180,15 +180,13 @@ int ust_lock(void)
 void ust_lock_nocheck(void)
 {
 	sigset_t sig_all_blocked, orig_mask;
-	int ret, oldstate;
+	int ret;
 
-	ret = pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate);
+	ret = pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,
+		&pthread_cancel_oldstate);
 	if (ret) {
 		ERR("pthread_setcancelstate: %s", strerror(ret));
 	}
-	if (oldstate != PTHREAD_CANCEL_ENABLE) {
-		ERR("pthread_setcancelstate: unexpected oldstate");
-	}
 	sigfillset(&sig_all_blocked);
 	ret = pthread_sigmask(SIG_SETMASK, &sig_all_blocked, &orig_mask);
 	if (ret) {
@@ -221,7 +219,7 @@ void ust_unlock(void)
 	if (ret) {
 		ERR("pthread_sigmask: %s", strerror(ret));
 	}
-	ret = pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &oldstate);
+	ret = pthread_setcancelstate(pthread_cancel_oldstate, &oldstate);
 	if (ret) {
 		ERR("pthread_setcancelstate: %s", strerror(ret));
 	}
@@ -414,6 +412,12 @@ void lttng_fixup_lttng_ust_notrace_tls(void)
 	asm volatile ("" : : "m" (URCU_TLS(lttng_ust_notrace_thread)));
 }
 
+static
+void lttng_fixup_pthread_cancel_oldstate_tls(void)
+{
+	asm volatile ("" : : "m" (URCU_TLS(pthread_cancel_oldstate)));
+}
+
 /*
  * Fixup urcu bp TLS.
  */
@@ -433,6 +437,7 @@ void lttng_ust_fixup_tls(void)
 	lttng_fixup_procname_tls();
 	lttng_fixup_ust_mutex_nest_tls();
 	lttng_fixup_lttng_ust_notrace_tls();
+	lttng_fixup_pthread_cancel_oldstate_tls();
 	lttng_ust_fixup_fd_tracker_tls();
 }
 
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 14373 bytes --]

Still running into same problem. I attached the debug trace I got after
applying the 2 patches.

The benchmark I am running also includes some custom created tracepoints. I
am not adding the events being traced in the files I have provided. Do you
think this might be causing a problem when I have tracpoints from 2
different packages?

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> I am so sorry for the late replies.
>
> I have tried the first patch you sent and the problem still happens
> (although seemingly less frequently especially with debugging enabled!!!!).
> I have attached the output I got from one of the erroneous runs.
>
> I will try the updated patch and let you know how it goes.
>
> Also, I am not sure if these points make any difference:
> 1- The error actually happens at the end of the application. It actually
> finishes execution, but then something goes wrong.
> 2- I run into this problem only for some of the benchmarks (or at least
> the problems happens much less frequently for others that I didn't run into
> it, not yet)
>
> Thanks you very much, and sorry again for the late replies.
>
>
> No worries! Looking through your log, I notice that pthread set cancel
> state has problems when
> called from application threads. We do not restore the original thread's
> cancel state. Can you try
> with the attached patch applied on top of the previous one ?
>
> Thanks,
>
> Mathieu
>
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>
>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made sure
>> the value was actually written by "cat /proc/sys/kernel/perf_event_
>> paranoid"
>>
>> It executed normally 2 times but then got the same error.
>>
>>
>> Can you provide the output when reproducing the issue with the
>> LTTNG_UST_DEBUG=1 environment variable set when starting
>> your test program ?
>>
>> I just noticed something that might explain what goes wrong here:
>>
>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock().
>> Pthread mutex
>> in your case is instrumented with the pthread wrapper. This
>> "add_thread_field" is invoked
>> the first time the perf counter is hit by each given thread. When this
>> happens, the
>> instrumented pthread mutex will try to call into the instrumentation
>> tracepoint again,
>> which will call add_thread_field() (again), and so on until we reach the
>> libringbuffer
>> "lib_ring_buffer_nesting" threshold of 4 calls deep.
>>
>> I suspect this situation where we recursively call add_thread_field is
>> unexpected,
>> which may trigger your double-free here.
>>
>> Will investigate more.
>>
>> Can you try with the attached patch applied ?
>>
>> Here is an updated v2 of the patch which tests the notrace tls counter
>> sooner (before evaluating
>> filter). Please let me know how it goes.
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>>
>>>
>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>>>
>>> I run the program as a normal user
>>>
>>> The glibc version I get by running "ldd --version" is 2.17
>>>
>>>
>>> Can you reproduce the issue after you do this as root ?
>>>
>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>>>
>>> Based on this documentation of the Linux kernel:
>>>
>>> Documentation/sysctl/kernel.txt:
>>>
>>> perf_event_paranoid:
>>>
>>> Controls use of the performance events system by unprivileged
>>> users (without CAP_SYS_ADMIN).  The default value is 2.
>>>
>>>  -1: Allow use of (almost) all events by all users
>>>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>>> wrote:
>>>>
>>>> Yes, I tried with only one of those contexts and I still ran into the
>>>> same problem.
>>>>
>>>> What is the setting returned by
>>>>
>>>> cat /proc/sys/kernel/perf_event_paranoid
>>>>
>>>> on your system ? And do you run your test program as root or normal
>>>> user ?
>>>>
>>>> Please CC the mailing list on your reply.
>>>>
>>>>
>>>> I will also need to know what glibc version you have on your system.
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: shehabyomn@gmail.com
>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>
>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <
>>>>> shehabyomn@gmail.com> wrote:
>>>>>
>>>>> Hi Mathieu,
>>>>>
>>>>> Thank you very much for your reply.
>>>>>
>>>>> I manually built lttng-ust from source (commit #:
>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>>>
>>>>> This is how I set up the tracing session:
>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>> mutex_lock_req
>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>> mutex_lock_acq
>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>> mutex_lock_trylock
>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>> mutex_lock_unlock
>>>>> 3- lttng add-context -u -t procname
>>>>>     lttng add-context -u -t vpid
>>>>>     lttng add-context -u -t pthread_id
>>>>>     lttng add-context -u -t vtid
>>>>>     lttng add-context -u -t ip
>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>> 4- lttng start
>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb
>>>>> -p8 -n8096 -b32
>>>>> 6- lttng stop
>>>>> 7- lttng destroy
>>>>>
>>>>>
>>>>> Can you reproduce if you remove the following contexts ?
>>>>>
>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>
>>>>> And if you only keep a single of those contexts ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Shehab Y. Elsayed, MSc.
>>>>> PhD Student
>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>> University of Toronto
>>>>> E-mail: shehabyomn@gmail.com
>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <
>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>
>>>>>> Hello All,
>>>>>>
>>>>>> I am trying to instrument a pthread application using the provided
>>>>>> pthread wrapper, but I sometimes run into a "Double free or
>>>>>> corruption error (fasttop)" error.
>>>>>>
>>>>>>
>>>>>> Please provide more information about the version of lttng-ust you
>>>>>> are using, and how you setup
>>>>>> your tracing session.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is a description of what I have tried and noticed:
>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>>>> works as expected.
>>>>>> 2- From my experiments, the problem happens (more frequently at
>>>>>> least) when adding performance counter contexts (I tried cycles, cpu_cycles
>>>>>> and instructions).
>>>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>>>> setting LD_PRELOAD to use the pthread wrapper as described in the
>>>>>> LTTng documents.
>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>> ======= Backtrace: =========
>>>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>>>>> 0x35)[0x7ffff7471575]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>>>>> 0x21c)[0x7ffff747363c]
>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>>>>> 0x7ffff746ead8]
>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
>>>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>>>     at lttng-events.c:247
>>>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>>>     at lttng-ust-abi.c:235
>>>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>>>     at lttng-ust-comm.c:1807
>>>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>>>     at lttng-ust-comm.c:1874
>>>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>>>    from /lib64/ld-linux-x86-64.so.2
>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>    from /lib64/libc.so.6
>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>>>> argv=<optimized out>) at lu.c:368
>>>>>>
>>>>>> Any ideas, why this happens and how to fix it?
>>>>>>
>>>>>> Thanks,
>>>>>> Shehab
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> lttng-dev@lists.lttng.org
>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> http://www.efficios.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 52249 bytes --]

[-- Attachment #2: 2patches_out.dbg --]
[-- Type: application/octet-stream, Size: 49826 bytes --]

Session lu_ncb_8_native created.
Traces will be written in /aenao-99/elsayed9/LTTng/data/lttng-traces/double-free-fix/lu_ncb_8_native
UST channel my-channel enabled for session lu_ncb_8_native
UST event lttng_ust_pthread:pthread_mutex_lock_req created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_lock_acq created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_trylock created in channel my-channel
UST event lttng_ust_pthread:pthread_mutex_unlock created in channel my-channel
UST context procname added to channel my-channel
UST context vpid added to channel my-channel
UST context pthread_id added to channel my-channel
UST context vtid added to channel my-channel
UST context ip added to channel my-channel
UST context perf:thread:cpu-cycles added to channel my-channel
UST context perf:thread:cycles added to channel my-channel
Tracing started for session lu_ncb_8_native
Running: LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096 -b32
liblttng_ust_tracepoint[44998/44998]: Your compiler treats weak symbols with hidden visibility for integer objects as SAME address between compile 
units part of the same module. (in check_weak_hidden() at tracepoint.c:916)
liblttng_ust_tracepoint[44998/44998]: Your compiler treats weak symbols with hidden visibility for pointer objects as SAME address between compile 
units part of the same module. (in check_weak_hidden() at tracepoint.c:920)
liblttng_ust_tracepoint[44998/44998]: Your compiler treats weak symbols with hidden visibility for 24-byte structure objects as SAME address betwee
n compile units part of the same module. (in check_weak_hidden() at tracepoint.c:924)
liblttng_ust_tracepoint[44998/44998]: just registered a tracepoints section from 0x7f38e786f240 and having 25 tracepoints (in tracepoint_register_l
ib() at tracepoint.c:867)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_statedump:end (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_statedump:debug_link (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_statedump:build_id (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_statedump:bin_info (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_statedump:start (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_lib:unload (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_lib:debug_link (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_lib:build_id (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_lib:load (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracef:event (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_LINE (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (in tracepoint_register_lib() at tracepoint.c:
872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_UNIT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_MODULE (in tracepoint_register_lib() at tracepoint.c:87
2)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROCESS (in tracepoint_register_lib() at tracepoint.c:8
72)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (in tracepoint_register_lib() at tracepoint.c:8
72)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (in tracepoint_register_lib() at tracepoint.c:87
2)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_INFO (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_NOTICE (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_WARNING (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_ERR (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_CRIT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_ALERT (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_tracelog:TRACE_EMERG (in tracepoint_register_lib() at tracepoint.c:872)
libust[44998/44998]: Provider "lttng_ust_statedump" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_ver
sion() at lttng-probes.c:175)
libust[44998/44998]: adding probe lttng_ust_statedump containing 5 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:21
9)
libust[44998/44998]: LTT : ltt ring buffer client "relay-metadata-mmap" init
 (in lttng_ring_buffer_metadata_client_init() at lttng-ring-buffer-metadata-client.h:352)
libust[44998/44998]: LTT : ltt ring buffer client "relay-overwrite-mmap" init
 (in lttng_ring_buffer_client_overwrite_init() at lttng-ring-buffer-client.h:868)
libust[44998/44998]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init
 (in lttng_ring_buffer_client_overwrite_rt_init() at lttng-ring-buffer-client.h:868)
libust[44998/44998]: LTT : ltt ring buffer client "relay-discard-mmap" init
 (in lttng_ring_buffer_client_discard_init() at lttng-ring-buffer-client.h:868)
libust[44998/44998]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init
 (in lttng_ring_buffer_client_discard_rt_init() at lttng-ring-buffer-client.h:868)
libust[44998/45000]: Message Received "Get Tracer Version" (65), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Session" (64), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 1 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Channel" (81), Handle "session" (1) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: channel data received (in handle_message() at lttng-ust-comm.c:839)
libust[44998/45000]: Return value: 2 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)                 [258/937]
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Context" (112), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 3 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Enable" (128), Handle "enabler" (3) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 4 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Enable" (128), Handle "enabler" (4) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 5 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Enable" (128), Handle "enabler" (5) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 6 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Enable" (128), Handle "enabler" (6) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Enable" (128), Handle "session" (1) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Sent register channel notification: chan_id 0, header_type 1
 (in ustcomm_register_channel() at lttng-ust-comm.c:1511)
libust[44998/45000]: just registered probe lttng_ust_statedump containing 5 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[44998/45000]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Wait for Quiescent State" (67), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/45000]: Message Received "Registration Done" (68), Handle "root" (0) (in print_cmd() at lttng-ust-comm.c:458)
libust[44998/45000]: Return value: 0 (in handle_message() at lttng-ust-comm.c:1000)
libust[44998/44998]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version()
 at lttng-probes.c:175)
libust[44998/44998]: adding probe lttng_ust_lib containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[44998/44998]: just registered probe lttng_ust_lib containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
libust[44998/44998]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_versio
n() at lttng-probes.c:175)
libust[44998/45000]: message successfully sent (in send_reply() at lttng-ust-comm.c:599)
libust[44998/44998]: adding probe lttng_ust_tracef containing 1 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[44998/44998]: just registered probe lttng_ust_tracef containing 1 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
libust[44998/44998]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_vers
ion() at lttng-probes.c:175)
libust[44998/44998]: adding probe lttng_ust_tracelog containing 15 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:21
9)
libust[44998/44998]: just registered probe lttng_ust_tracelog containing 15 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
liblttng_ust_tracepoint[44998/44998]: just registered a tracepoints section from 0x7f38e7d80128 and having 4 tracepoints (in tracepoint_register_li
b() at tracepoint.c:867)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_pthread:pthread_mutex_unlock (in tracepoint_register_lib() at tracepoint.c:8
72)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_pthread:pthread_mutex_trylock (in tracepoint_register_lib() at tracepoint.c:
872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_pthread:pthread_mutex_lock_acq (in tracepoint_register_lib() at tracepoint.c
:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: lttng_ust_pthread:pthread_mutex_lock_req (in tracepoint_register_lib() at tracepoint.c
:872)
libust[44998/44998]: Provider "lttng_ust_pthread" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_versi
on() at lttng-probes.c:175)
libust[44998/44998]: adding probe lttng_ust_pthread containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[44998/44998]: just registered probe lttng_ust_pthread containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
libust[44998/44998]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_lock_req": ret_code 0, event_id 0
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[44998/44998]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_trylock": ret_code 0, event_id 1
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[44998/44998]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_unlock": ret_code 0, event_id 2
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
libust[44998/44998]: Sent register event notification for name "lttng_ust_pthread:pthread_mutex_lock_acq": ret_code 0, event_id 3
 (in ustcomm_register_event() at lttng-ust-comm.c:1293)
liblttng_ust_tracepoint[44998/44998]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_lock_acq. Queuing release. (in __tracepoint_p
robe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[44998/44998]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_unlock. Queuing release. (in __tracepoint_pro
be_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[44998/44998]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_trylock. Queuing release. (in __tracepoint_pr
obe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[44998/44998]: Registering probe to tracepoint lttng_ust_pthread:pthread_mutex_lock_req. Queuing release. (in __tracepoint_p
robe_register_queue_release() at tracepoint.c:611)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
liblttng_ust_tracepoint[44998/44998]: just registered a tracepoints section from 0x6072f8 and having 4 tracepoints (in tracepoint_register_lib() at
 tracepoint.c:867)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: splash3:roi_begin (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: splash3:roi_end (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: splash3:thread_begin (in tracepoint_register_lib() at tracepoint.c:872)
liblttng_ust_tracepoint[44998/44998]: registered tracepoint: splash3:thread_end (in tracepoint_register_lib() at tracepoint.c:872)
libust[44998/44998]: Provider "splash3" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lt
tng-probes.c:175)
libust[44998/44998]: adding probe splash3 containing 4 events to lazy registration list (in lttng_probe_register() at lttng-probes.c:219)
libust[44998/44998]: just registered probe splash3 containing 4 events (in lttng_lazy_probe_register() at lttng-probes.c:116)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)

Blocked Dense LU Factorization
     8096 by 8096 Matrix
     8 Processors
     32 by 32 Element Blocks


libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
libust[44998/44999]: Info: sessiond not accepting connections to global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1452)
libust[44998/44999]: Waiting for global apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1334)
                            PROCESS STATISTICS
              Total      Diagonal     Perimeter      Interior       Barrier
 Proc         Time         Time         Time           Time          Time
    0            34             0             1            33             0

                            TIMING INFORMATION
Start time                        :       1521674617
Initialization finish time        :       1521674618
Overall finish time               :       1521674652
Total time with initialization    :               35
Total time without initialization :               34

libust[44998/44998]: Provider "splash3" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at lt
tng-probes.c:175)
libust[44998/44998]: just unregistered probe splash3 (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[44998/44998]: just unregistered a tracepoints section from 0x6072f8 (in tracepoint_unregister_lib() at tracepoint.c:897)
libust[44998/44998]: Provider "lttng_ust_pthread" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_versi
on() at lttng-probes.c:175)
libust[44998/44998]: just unregistered probe lttng_ust_pthread (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[44998/44998]: just unregistered a tracepoints section from 0x7f38e7d80128 (in tracepoint_unregister_lib() at tracepoint.c:8
97)
libust[44998/44998]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_vers
ion() at lttng-probes.c:175)
libust[44998/44998]: just unregistered probe lttng_ust_tracelog (in lttng_probe_unregister() at lttng-probes.c:250)
libust[44998/44998]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_versio
n() at lttng-probes.c:175)
libust[44998/44998]: just unregistered probe lttng_ust_tracef (in lttng_probe_unregister() at lttng-probes.c:250)
libust[44998/44998]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version()
 at lttng-probes.c:175)
libust[44998/44998]: just unregistered probe lttng_ust_lib (in lttng_probe_unregister() at lttng-probes.c:250)
liblttng_ust_tracepoint[44998/44998]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_lock_acq. Queuing release. (in __tracepo
int_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[44998/44998]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_unlock. Queuing release. (in __tracepoin
t_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[44998/44998]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_trylock. Queuing release. (in __tracepoi
nt_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[44998/44998]: Un-registering probe from tracepoint lttng_ust_pthread:pthread_mutex_lock_req. Queuing release. (in __tracepo
int_probe_unregister_queue_release() at tracepoint.c:682)
liblttng_ust_tracepoint[44998/44998]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at tracepoint.c
:702)
*** Error in `./lu_ncb': double free or corruption (fasttop): 0x00007f3858000920 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f38e6e97503]
/usr/local/lib/liblttng-ust.so.0(+0x32d05)[0x7f38e762ed05]
/usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+0x35)[0x7f38e761e565]
/usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+0x21c)[0x7f38e762062c]
/usr/local/lib/liblttng-ust.so.0(+0x1e8f6)[0x7f38e761a8f6]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f38e761acbf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f38e761acbf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7f38e761acbf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x7f38e761bac8]
/usr/local/lib/liblttng-ust.so.0(+0x19303)[0x7f38e7615303]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7f38e760be87]
/lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7f38e7d9085a]
/lib64/libc.so.6(+0x38a49)[0x7f38e6e53a49]
/lib64/libc.so.6(+0x38a95)[0x7f38e6e53a95]
./lu_ncb[0x401b51]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f38e6e3cb35]
./lu_ncb[0x401c44]
======= Memory map: ========                                                                                                                [0/937]
00400000-00407000 r-xp 00000000 00:2e 233750343                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
00606000-00607000 r--p 00006000 00:2e 233750343                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
00607000-00608000 rw-p 00007000 00:2e 233750343                          /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb
0249a000-024dd000 rw-p 00000000 00:00 0                                  [heap]
7f3848000000-7f3848021000 rw-p 00000000 00:00 0 
7f3848021000-7f384c000000 ---p 00000000 00:00 0 
7f3850000000-7f3850021000 rw-p 00000000 00:00 0 
7f3850021000-7f3854000000 ---p 00000000 00:00 0 
7f3854000000-7f3854021000 rw-p 00000000 00:00 0 
7f3854021000-7f3858000000 ---p 00000000 00:00 0 
7f3858000000-7f3858021000 rw-p 00000000 00:00 0 
7f3858021000-7f385c000000 ---p 00000000 00:00 0 
7f385c000000-7f385c021000 rw-p 00000000 00:00 0 
7f385c021000-7f3860000000 ---p 00000000 00:00 0 
7f3860000000-7f3860021000 rw-p 00000000 00:00 0 
7f3860021000-7f3864000000 ---p 00000000 00:00 0 
7f3864000000-7f3864021000 rw-p 00000000 00:00 0 
7f3864021000-7f3868000000 ---p 00000000 00:00 0 
7f3868000000-7f3868021000 rw-p 00000000 00:00 0 
7f3868021000-7f386c000000 ---p 00000000 00:00 0 
7f386fbab000-7f386fbac000 ---p 00000000 00:00 0 
7f386fbac000-7f38703ac000 rw-p 00000000 00:00 0 
7f38703ac000-7f38703ad000 ---p 00000000 00:00 0 
7f38703ad000-7f388ffc0000 rw-p 00000000 00:00 0 
7f388ffc0000-7f38927c2000 rw-s 00000000 00:12 4043755                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38927c2000-7f3894fc4000 rw-s 00000000 00:12 4043754                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f3894fc4000-7f38977c6000 rw-s 00000000 00:12 4043753                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38977c6000-7f3899fc8000 rw-s 00000000 00:12 4043752                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f3899fc8000-7f389c7ca000 rw-s 00000000 00:12 4043751                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f389c7ca000-7f389efcc000 rw-s 00000000 00:12 4043750                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f389efcc000-7f38a17ce000 rw-s 00000000 00:12 4043749                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38a17ce000-7f38a3fd0000 rw-s 00000000 00:12 4043748                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38a3fd0000-7f38a67d2000 rw-s 00000000 00:12 4043747                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38a67d2000-7f38a8fd4000 rw-s 00000000 00:12 4043746                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38a8fd4000-7f38ab7d6000 rw-s 00000000 00:12 4043745                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38ab7d6000-7f38adfd8000 rw-s 00000000 00:12 4043744                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38adfd8000-7f38b07da000 rw-s 00000000 00:12 4043743                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38b07da000-7f38b2fdc000 rw-s 00000000 00:12 4043742                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38b2fdc000-7f38b57de000 rw-s 00000000 00:12 4043741                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38b57de000-7f38b7fe0000 rw-s 00000000 00:12 4043740                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38b7fe0000-7f38ba7e2000 rw-s 00000000 00:12 4043739                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38ba7e2000-7f38bcfe4000 rw-s 00000000 00:12 4043738                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38bcfe4000-7f38bf7e6000 rw-s 00000000 00:12 4043737                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38bf7e6000-7f38c1fe8000 rw-s 00000000 00:12 4043736                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38c1fe8000-7f38c47ea000 rw-s 00000000 00:12 4043735                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38c47ea000-7f38c6fec000 rw-s 00000000 00:12 4043734                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38c6fec000-7f38c97ee000 rw-s 00000000 00:12 4043733                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38c97ee000-7f38cbff0000 rw-s 00000000 00:12 4043732                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38cbff0000-7f38ce7f2000 rw-s 00000000 00:12 4043731                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38ce7f2000-7f38d0ff4000 rw-s 00000000 00:12 4043730                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38d0ff4000-7f38d37f6000 rw-s 00000000 00:12 4043729                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38d37f6000-7f38d5ff8000 rw-s 00000000 00:12 4043728                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38d5ff8000-7f38d87fa000 rw-s 00000000 00:12 4043727                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38d87fa000-7f38daffc000 rw-s 00000000 00:12 4043726                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38daffc000-7f38dd7fe000 rw-s 00000000 00:12 4043725                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38dd7fe000-7f38e0000000 rw-s 00000000 00:12 4043724                    /dev/shm/ust-shm-consumer-29340 (deleted)
7f38e0000000-7f38e0021000 rw-p 00000000 00:00 0 
7f38e0021000-7f38e4000000 ---p 00000000 00:00 0 
7f38e47bd000-7f38e47be000 ---p 00000000 00:00 0 
7f38e47be000-7f38e4fbe000 rw-p 00000000 00:00 0 
7f38e4fbe000-7f38e4fbf000 ---p 00000000 00:00 0 
7f38e4fbf000-7f38e57bf000 rw-p 00000000 00:00 0 
7f38e57bf000-7f38e57c0000 ---p 00000000 00:00 0 
7f38e57c0000-7f38e5fc0000 rw-p 00000000 00:00 0                          [stack:44999]
7f38e5fc0000-7f38e5fd5000 r-xp 00000000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f38e5fd5000-7f38e61d4000 ---p 00015000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f38e61d4000-7f38e61d5000 r--p 00014000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f38e61d5000-7f38e61d6000 rw-p 00015000 fd:00 806796360                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f38e61d6000-7f38e61dd000 r-xp 00000000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f38e61dd000-7f38e63dc000 ---p 00007000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f38e63dc000-7f38e63dd000 r--p 00006000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f38e63dd000-7f38e63de000 rw-p 00007000 fd:00 269855514                  /usr/local/lib/liburcu-cds.so.4.1.0
7f38e63de000-7f38e63e5000 r-xp 00000000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f38e63e5000-7f38e65e4000 ---p 00007000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f38e65e4000-7f38e65e5000 r--p 00006000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f38e65e5000-7f38e65e6000 rw-p 00007000 fd:00 269855510                  /usr/local/lib/liburcu-bp.so.4.1.0
7f38e65e6000-7f38e65f0000 r-xp 00000000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f38e65f0000-7f38e67f0000 ---p 0000a000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f38e67f0000-7f38e67f1000 r--p 0000a000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f38e67f1000-7f38e67f2000 rw-p 0000b000 fd:00 807171531                  /usr/lib64/libnuma.so.1
7f38e67f2000-7f38e67f6000 r-xp 00000000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f38e67f6000-7f38e69f5000 ---p 00004000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f38e69f5000-7f38e69f6000 r--p 00003000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f38e69f6000-7f38e69f7000 rw-p 00004000 fd:00 269548485                  /usr/local/lib/liburcu-common.so.4.1.0
7f38e69f7000-7f38e69fe000 r-xp 00000000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f38e69fe000-7f38e6bfd000 ---p 00007000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f38e6bfd000-7f38e6bfe000 r--p 00006000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f38e6bfe000-7f38e6bff000 rw-p 00007000 fd:00 805308584                  /usr/lib64/librt-2.17.so
7f38e6bff000-7f38e6c0a000 r-xp 00000000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f38e6c0a000-7f38e6e09000 ---p 0000b000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f38e6e09000-7f38e6e0a000 r--p 0000a000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f38e6e0a000-7f38e6e0b000 rw-p 0000b000 fd:00 268585988                  /usr/local/lib/liblttng-ust-tracepoint.so.0.0.0
7f38e6e0b000-7f38e6e1b000 rw-p 00000000 00:00 0 
7f38e6e1b000-7f38e6fd1000 r-xp 00000000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f38e6fd1000-7f38e71d1000 ---p 001b6000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f38e71d1000-7f38e71d5000 r--p 001b6000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f38e71d5000-7f38e71d7000 rw-p 001ba000 fd:00 805308548                  /usr/lib64/libc-2.17.so
7f38e71d7000-7f38e71dc000 rw-p 00000000 00:00 0 
7f38e71dc000-7f38e71f3000 r-xp 00000000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f38e71f3000-7f38e73f2000 ---p 00017000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f38e73f2000-7f38e73f3000 r--p 00016000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f38e73f3000-7f38e73f4000 rw-p 00017000 fd:00 805308579                  /usr/lib64/libpthread-2.17.so
7f38e73f4000-7f38e73f8000 rw-p 00000000 00:00 0 
7f38e73f8000-7f38e73fa000 r-xp 00000000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f38e73fa000-7f38e75fa000 ---p 00002000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f38e75fa000-7f38e75fb000 r--p 00002000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f38e75fb000-7f38e75fc000 rw-p 00003000 fd:00 805308554                  /usr/lib64/libdl-2.17.so
7f38e75fc000-7f38e7663000 r-xp 00000000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f38e7663000-7f38e7863000 ---p 00067000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f38e7863000-7f38e786a000 r--p 00067000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f38e786a000-7f38e7870000 rw-p 0006e000 fd:00 268585961                  /usr/local/lib/liblttng-ust.so.0.0.0
7f38e7870000-7f38e7879000 rw-p 00000000 00:00 0 
7f38e7879000-7f38e7979000 r-xp 00000000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f38e7979000-7f38e7b79000 ---p 00100000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f38e7b79000-7f38e7b7a000 r--p 00100000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f38e7b7a000-7f38e7b7b000 rw-p 00101000 fd:00 805308556                  /usr/lib64/libm-2.17.so
7f38e7b7b000-7f38e7b7f000 r-xp 00000000 fd:00 269953025                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f38e7b7f000-7f38e7d7e000 ---p 00004000 fd:00 269953025                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f38e7d7e000-7f38e7d80000 r--p 00003000 fd:00 269953025                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f38e7d80000-7f38e7d81000 rw-p 00005000 fd:00 269953025                  /usr/local/lib/liblttng-ust-pthread-wrapper.so.0.0.0
7f38e7d81000-7f38e7da1000 r-xp 00000000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f38e7f7b000-7f38e7f7c000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f38e7f80000-7f38e7f81000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f38e7f85000-7f38e7f86000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f38e7f87000-7f38e7f8f000 rw-p 00000000 00:00 0 
7f38e7f92000-7f38e7f94000 rw-p 00000000 00:00 0 
7f38e7f9a000-7f38e7f9b000 rw-p 00000000 00:00 0 
7f38e7f9b000-7f38e7f9c000 r--s 00000000 00:09 7899                       anon_inode:[perf_event]
7f38e7f9c000-7f38e7f9d000 rw-p 00000000 00:00 0 
7f38e7f9d000-7f38e7f9e000 r--s 00000000 00:12 139034                     /dev/shm/lttng-ust-wait-7
7f38e7f9e000-7f38e7fa0000 rw-p 00000000 00:00 0 
7f38e7fa0000-7f38e7fa1000 r--p 0001f000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f38e7fa1000-7f38e7fa2000 rw-p 00020000 fd:00 806796390                  /usr/lib64/ld-2.17.so
7f38e7fa2000-7f38e7fa3000 rw-p 00000000 00:00 0 
7ffcc91e0000-7ffcc9201000 rw-p 00000000 00:00 0                          [stack]
7ffcc92de000-7ffcc92e0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
./run_benchmark.sh: line 89: 44998 Aborted                 (core dumped) LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8
096 -b32
Waiting for data availability.
Tracing stopped for session lu_ncb_8_native
Session lu_ncb_8_native destroyed

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 15821 bytes --]

Just to clarify more what I meant by custom events; I have new tracepoints
added in the source code of the benchmark. However, I don't enable the
corresponding events when I do the actual tracing.

This is the sequence followed in building the benchmark:
gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112
-std=c11 -g -fno-strict-aliasing -DLTTNG_INST lu.c
gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11
-g -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o
../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl

LTTNG_INST is just a preprocessor flag I have and tp.o is my custom
tracepoints

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
wrote:

> Still running into same problem. I attached the debug trace I got after
> applying the 2 patches.
>
> The benchmark I am running also includes some custom created tracepoints.
> I am not adding the events being traced in the files I have provided. Do
> you think this might be causing a problem when I have tracpoints from 2
> different packages?
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> I am so sorry for the late replies.
>>
>> I have tried the first patch you sent and the problem still happens
>> (although seemingly less frequently especially with debugging enabled!!!!).
>> I have attached the output I got from one of the erroneous runs.
>>
>> I will try the updated patch and let you know how it goes.
>>
>> Also, I am not sure if these points make any difference:
>> 1- The error actually happens at the end of the application. It actually
>> finishes execution, but then something goes wrong.
>> 2- I run into this problem only for some of the benchmarks (or at least
>> the problems happens much less frequently for others that I didn't run into
>> it, not yet)
>>
>> Thanks you very much, and sorry again for the late replies.
>>
>>
>> No worries! Looking through your log, I notice that pthread set cancel
>> state has problems when
>> called from application threads. We do not restore the original thread's
>> cancel state. Can you try
>> with the attached patch applied on top of the previous one ?
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>
>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made sure
>>> the value was actually written by "cat /proc/sys/kernel/perf_event_pa
>>> ranoid"
>>>
>>> It executed normally 2 times but then got the same error.
>>>
>>>
>>> Can you provide the output when reproducing the issue with the
>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>> your test program ?
>>>
>>> I just noticed something that might explain what goes wrong here:
>>>
>>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock().
>>> Pthread mutex
>>> in your case is instrumented with the pthread wrapper. This
>>> "add_thread_field" is invoked
>>> the first time the perf counter is hit by each given thread. When this
>>> happens, the
>>> instrumented pthread mutex will try to call into the instrumentation
>>> tracepoint again,
>>> which will call add_thread_field() (again), and so on until we reach the
>>> libringbuffer
>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.
>>>
>>> I suspect this situation where we recursively call add_thread_field is
>>> unexpected,
>>> which may trigger your double-free here.
>>>
>>> Will investigate more.
>>>
>>> Can you try with the attached patch applied ?
>>>
>>> Here is an updated v2 of the patch which tests the notrace tls counter
>>> sooner (before evaluating
>>> filter). Please let me know how it goes.
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>>
>>>>
>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>>> wrote:
>>>>
>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>>>>
>>>> I run the program as a normal user
>>>>
>>>> The glibc version I get by running "ldd --version" is 2.17
>>>>
>>>>
>>>> Can you reproduce the issue after you do this as root ?
>>>>
>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>>>>
>>>> Based on this documentation of the Linux kernel:
>>>>
>>>> Documentation/sysctl/kernel.txt:
>>>>
>>>> perf_event_paranoid:
>>>>
>>>> Controls use of the performance events system by unprivileged
>>>> users (without CAP_SYS_ADMIN).  The default value is 2.
>>>>
>>>>  -1: Allow use of (almost) all events by all users
>>>>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: shehabyomn@gmail.com
>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>
>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <
>>>>> shehabyomn@gmail.com> wrote:
>>>>>
>>>>> Yes, I tried with only one of those contexts and I still ran into the
>>>>> same problem.
>>>>>
>>>>> What is the setting returned by
>>>>>
>>>>> cat /proc/sys/kernel/perf_event_paranoid
>>>>>
>>>>> on your system ? And do you run your test program as root or normal
>>>>> user ?
>>>>>
>>>>> Please CC the mailing list on your reply.
>>>>>
>>>>>
>>>>> I will also need to know what glibc version you have on your system.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>> Shehab Y. Elsayed, MSc.
>>>>> PhD Student
>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>> University of Toronto
>>>>> E-mail: shehabyomn@gmail.com
>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <
>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>
>>>>>> Hi Mathieu,
>>>>>>
>>>>>> Thank you very much for your reply.
>>>>>>
>>>>>> I manually built lttng-ust from source (commit #:
>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>>>>
>>>>>> This is how I set up the tracing session:
>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mute
>>>>>> x_lock_req
>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_mute
>>>>>> x_lock_acq
>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_mute
>>>>>> x_lock_trylock
>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_mute
>>>>>> x_lock_unlock
>>>>>> 3- lttng add-context -u -t procname
>>>>>>     lttng add-context -u -t vpid
>>>>>>     lttng add-context -u -t pthread_id
>>>>>>     lttng add-context -u -t vtid
>>>>>>     lttng add-context -u -t ip
>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>> 4- lttng start
>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so
>>>>>> ./lu_ncb -p8 -n8096 -b32
>>>>>> 6- lttng stop
>>>>>> 7- lttng destroy
>>>>>>
>>>>>>
>>>>>> Can you reproduce if you remove the following contexts ?
>>>>>>
>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>>
>>>>>> And if you only keep a single of those contexts ?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: shehabyomn@gmail.com
>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <
>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello All,
>>>>>>>
>>>>>>> I am trying to instrument a pthread application using the provided
>>>>>>> pthread wrapper, but I sometimes run into a "Double free or
>>>>>>> corruption error (fasttop)" error.
>>>>>>>
>>>>>>>
>>>>>>> Please provide more information about the version of lttng-ust you
>>>>>>> are using, and how you setup
>>>>>>> your tracing session.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mathieu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>>>>> works as expected.
>>>>>>> 2- From my experiments, the problem happens (more frequently at
>>>>>>> least) when adding performance counter contexts (I tried cycles, cpu_cycles
>>>>>>> and instructions).
>>>>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>>>>> setting LD_PRELOAD to use the pthread wrapper as described in the
>>>>>>> LTTng documents.
>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>> ======= Backtrace: =========
>>>>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+0x35)
>>>>>>> [0x7ffff7471575]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+0x21c
>>>>>>> )[0x7ffff747363c]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[
>>>>>>> 0x7ffff746dccf]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[
>>>>>>> 0x7ffff746dccf]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[
>>>>>>> 0x7ffff746dccf]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x
>>>>>>> 7ffff746ead8]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7fff
>>>>>>> f745ed57]
>>>>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>>>>     at lttng-events.c:247
>>>>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>>>>     at lttng-ust-abi.c:235
>>>>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>>>>     at lttng-ust-comm.c:1807
>>>>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>>>>     at lttng-ust-comm.c:1874
>>>>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>>>>    from /lib64/ld-linux-x86-64.so.2
>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>    from /lib64/libc.so.6
>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>>>>> argv=<optimized out>) at lu.c:368
>>>>>>>
>>>>>>> Any ideas, why this happens and how to fix it?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shehab
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> lttng-dev mailing list
>>>>>>> lttng-dev@lists.lttng.org
>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> http://www.efficios.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> http://www.efficios.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 57692 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 18975 bytes --]

----- On Mar 21, 2018, at 8:01 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> Just to clarify more what I meant by custom events; I have new tracepoints added
> in the source code of the benchmark. However, I don't enable the corresponding
> events when I do the actual tracing.

> This is the sequence followed in building the benchmark:
> gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g
> -fno-strict-aliasing -DLTTNG_INST lu.c
> gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g
> -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o
> ../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl

> LTTNG_INST is just a preprocessor flag I have and tp.o is my custom tracepoints

Could you share a repository with your custom instrumentation on github, so I could 
try it out ? 

My current problem is that I cannot reproduce your issue on my end. 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed < [ mailto:shehabyomn@gmail.com
> | shehabyomn@gmail.com ] > wrote:

>> Still running into same problem. I attached the debug trace I got after applying
>> the 2 patches.

>> The benchmark I am running also includes some custom created tracepoints. I am
>> not adding the events being traced in the files I have provided. Do you think
>> this might be causing a problem when I have tracpoints from 2 different
>> packages?

>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>> shehabyomn@gmail.com ]

>> On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers < [
>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>> wrote:

>>> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed < [
>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>> I am so sorry for the late replies.

>>>> I have tried the first patch you sent and the problem still happens (although
>>>> seemingly less frequently especially with debugging enabled!!!!). I have
>>>> attached the output I got from one of the erroneous runs.

>>>> I will try the updated patch and let you know how it goes.

>>>> Also, I am not sure if these points make any difference:
>>>> 1- The error actually happens at the end of the application. It actually
>>>> finishes execution, but then something goes wrong.
>>>> 2- I run into this problem only for some of the benchmarks (or at least the
>>>> problems happens much less frequently for others that I didn't run into it, not
>>>> yet)

>>>> Thanks you very much, and sorry again for the late replies.

>>> No worries! Looking through your log, I notice that pthread set cancel state has
>>> problems when
>>> called from application threads. We do not restore the original thread's cancel
>>> state. Can you try
>>> with the attached patch applied on top of the previous one ?

>>> Thanks,

>>> Mathieu

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers < [
>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>> wrote:

>>>>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers < [
>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>> wrote:

>>>>>>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed < [
>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>>>>>>>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>>>>>>>>> It executed normally 2 times but then got the same error.

>>>>>>>> Can you provide the output when reproducing the issue with the
>>>>>>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>>>>>>> your test program ?

>>>>>>> I just noticed something that might explain what goes wrong here:

>>>>>>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread
>>>>>>> mutex
>>>>>>> in your case is instrumented with the pthread wrapper. This "add_thread_field"
>>>>>>> is invoked
>>>>>>> the first time the perf counter is hit by each given thread. When this happens,
>>>>>>> the
>>>>>>> instrumented pthread mutex will try to call into the instrumentation tracepoint
>>>>>>> again,
>>>>>>> which will call add_thread_field() (again), and so on until we reach the
>>>>>>> libringbuffer
>>>>>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.

>>>>>>> I suspect this situation where we recursively call add_thread_field is
>>>>>>> unexpected,
>>>>>>> which may trigger your double-free here.

>>>>>>> Will investigate more.

>>>>>> Can you try with the attached patch applied ?

>>>>> Here is an updated v2 of the patch which tests the notrace tls counter sooner
>>>>> (before evaluating
>>>>> filter). Please let me know how it goes.

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Thanks,

>>>>>> Mathieu

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Thanks,

>>>>>>>> Mathieu

>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>> PhD Student
>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>> University of Toronto
>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>> wrote:

>>>>>>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>>>>>>>>> I run the program as a normal user

>>>>>>>>>>> The glibc version I get by running "ldd --version" is 2.17

>>>>>>>>>> Can you reproduce the issue after you do this as root ?

>>>>>>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>>>>>>>>> Based on this documentation of the Linux kernel:

>>>>>>>>>> Documentation/sysctl/kernel.txt:

>>>>>>>>>> perf_event_paranoid:

>>>>>>>>>> Controls use of the performance events system by unprivileged
>>>>>>>>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>>>>>>>>> -1: Allow use of (almost) all events by all users
>>>>>>>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>>>>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>>>>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>>>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>>>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>>>>>>>>> Thanks,

>>>>>>>>>> Mathieu

>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>> PhD Student
>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>> University of Toronto
>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>> wrote:

>>>>>>>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>> wrote:

>>>>>>>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>>>>>>>>> problem.

>>>>>>>>>>>>> What is the setting returned by

>>>>>>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>>>>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>>>>>>>>> Please CC the mailing list on your reply.

>>>>>>>>>>>> I will also need to know what glibc version you have on your system.

>>>>>>>>>>>> Thanks,

>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>>> Hi Mathieu,

>>>>>>>>>>>>>>>> Thank you very much for your reply.

>>>>>>>>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>>>>>>>>> 4- lttng start
>>>>>>>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>>>>>>>>> -b32
>>>>>>>>>>>>>>>> 6- lttng stop
>>>>>>>>>>>>>>>> 7- lttng destroy

>>>>>>>>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>>>>> Hello All,

>>>>>>>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>>>>>>>>> )" error.

>>>>>>>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>>>>>>>>> and how you setup
>>>>>>>>>>>>>>>>> your tracing session.

>>>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>>>>>>>>> expected.
>>>>>>>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>>>>>>>>> instructions).
>>>>>>>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Shehab

>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>>> --
>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>> --
>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>> --
>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 59130 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 17190 bytes --]

Actually, i am not sure if this would help. I have been trying to reproduce
the problem on a different machine, but I can't. Even without the patches
!!!!!

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Wed, Mar 21, 2018 at 8:13 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> ----- On Mar 21, 2018, at 8:01 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> Just to clarify more what I meant by custom events; I have new tracepoints
> added in the source code of the benchmark. However, I don't enable the
> corresponding events when I do the actual tracing.
>
> This is the sequence followed in building the benchmark:
> gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112
> -std=c11 -g -fno-strict-aliasing -DLTTNG_INST lu.c
> gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11
> -g -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o
> ../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl
>
> LTTNG_INST is just a preprocessor flag I have and tp.o is my custom
> tracepoints
>
> Could you share a repository with your custom instrumentation on github,
> so I could
> try it out ?
>
> My current problem is that I cannot reproduce your issue on my end.
>
> Thanks,
>
> Mathieu
>
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
>> Still running into same problem. I attached the debug trace I got after
>> applying the 2 patches.
>>
>> The benchmark I am running also includes some custom created tracepoints.
>> I am not adding the events being traced in the files I have provided. Do
>> you think this might be causing a problem when I have tracpoints from 2
>> different packages?
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers <
>> mathieu.desnoyers@efficios.com> wrote:
>>
>>> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>> wrote:
>>>
>>> I am so sorry for the late replies.
>>>
>>> I have tried the first patch you sent and the problem still happens
>>> (although seemingly less frequently especially with debugging enabled!!!!).
>>> I have attached the output I got from one of the erroneous runs.
>>>
>>> I will try the updated patch and let you know how it goes.
>>>
>>> Also, I am not sure if these points make any difference:
>>> 1- The error actually happens at the end of the application. It actually
>>> finishes execution, but then something goes wrong.
>>> 2- I run into this problem only for some of the benchmarks (or at least
>>> the problems happens much less frequently for others that I didn't run into
>>> it, not yet)
>>>
>>> Thanks you very much, and sorry again for the late replies.
>>>
>>>
>>> No worries! Looking through your log, I notice that pthread set cancel
>>> state has problems when
>>> called from application threads. We do not restore the original thread's
>>> cancel state. Can you try
>>> with the attached patch applied on top of the previous one ?
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>
>>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>>> wrote:
>>>>
>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made sure
>>>> the value was actually written by "cat /proc/sys/kernel/perf_event_
>>>> paranoid"
>>>>
>>>> It executed normally 2 times but then got the same error.
>>>>
>>>>
>>>> Can you provide the output when reproducing the issue with the
>>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>>> your test program ?
>>>>
>>>> I just noticed something that might explain what goes wrong here:
>>>>
>>>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock().
>>>> Pthread mutex
>>>> in your case is instrumented with the pthread wrapper. This
>>>> "add_thread_field" is invoked
>>>> the first time the perf counter is hit by each given thread. When this
>>>> happens, the
>>>> instrumented pthread mutex will try to call into the instrumentation
>>>> tracepoint again,
>>>> which will call add_thread_field() (again), and so on until we reach
>>>> the libringbuffer
>>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.
>>>>
>>>> I suspect this situation where we recursively call add_thread_field is
>>>> unexpected,
>>>> which may trigger your double-free here.
>>>>
>>>> Will investigate more.
>>>>
>>>> Can you try with the attached patch applied ?
>>>>
>>>> Here is an updated v2 of the patch which tests the notrace tls counter
>>>> sooner (before evaluating
>>>> filter). Please let me know how it goes.
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: shehabyomn@gmail.com
>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>
>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <
>>>>> shehabyomn@gmail.com> wrote:
>>>>>
>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>>>>>
>>>>> I run the program as a normal user
>>>>>
>>>>> The glibc version I get by running "ldd --version" is 2.17
>>>>>
>>>>>
>>>>> Can you reproduce the issue after you do this as root ?
>>>>>
>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>>>>>
>>>>> Based on this documentation of the Linux kernel:
>>>>>
>>>>> Documentation/sysctl/kernel.txt:
>>>>>
>>>>> perf_event_paranoid:
>>>>>
>>>>> Controls use of the performance events system by unprivileged
>>>>> users (without CAP_SYS_ADMIN).  The default value is 2.
>>>>>
>>>>>  -1: Allow use of (almost) all events by all users
>>>>>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>> Shehab Y. Elsayed, MSc.
>>>>> PhD Student
>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>> University of Toronto
>>>>> E-mail: shehabyomn@gmail.com
>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>
>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <
>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>
>>>>>> Yes, I tried with only one of those contexts and I still ran into the
>>>>>> same problem.
>>>>>>
>>>>>> What is the setting returned by
>>>>>>
>>>>>> cat /proc/sys/kernel/perf_event_paranoid
>>>>>>
>>>>>> on your system ? And do you run your test program as root or normal
>>>>>> user ?
>>>>>>
>>>>>> Please CC the mailing list on your reply.
>>>>>>
>>>>>>
>>>>>> I will also need to know what glibc version you have on your system.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>
>>>>>>
>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: shehabyomn@gmail.com
>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>
>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <
>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Mathieu,
>>>>>>>
>>>>>>> Thank you very much for your reply.
>>>>>>>
>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>>>>>
>>>>>>> This is how I set up the tracing session:
>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>> mutex_lock_req
>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>> mutex_lock_acq
>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>> mutex_lock_trylock
>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>> mutex_lock_unlock
>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>     lttng add-context -u -t vpid
>>>>>>>     lttng add-context -u -t pthread_id
>>>>>>>     lttng add-context -u -t vtid
>>>>>>>     lttng add-context -u -t ip
>>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>>> 4- lttng start
>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so
>>>>>>> ./lu_ncb -p8 -n8096 -b32
>>>>>>> 6- lttng stop
>>>>>>> 7- lttng destroy
>>>>>>>
>>>>>>>
>>>>>>> Can you reproduce if you remove the following contexts ?
>>>>>>>
>>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>>>
>>>>>>> And if you only keep a single of those contexts ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mathieu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>> PhD Student
>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer
>>>>>>> Engineering
>>>>>>> University of Toronto
>>>>>>> E-mail: shehabyomn@gmail.com
>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <
>>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hello All,
>>>>>>>>
>>>>>>>> I am trying to instrument a pthread application using the provided
>>>>>>>> pthread wrapper, but I sometimes run into a "Double free or
>>>>>>>> corruption error (fasttop)" error.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please provide more information about the version of lttng-ust you
>>>>>>>> are using, and how you setup
>>>>>>>> your tracing session.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Mathieu
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>>>>>> works as expected.
>>>>>>>> 2- From my experiments, the problem happens (more frequently at
>>>>>>>> least) when adding performance counter contexts (I tried cycles,
>>>>>>>> cpu_cycles and instructions).
>>>>>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>>>>>> setting LD_PRELOAD to use the pthread wrapper as described in the
>>>>>>>> LTTng documents.
>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>> ======= Backtrace: =========
>>>>>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>>>>>>> 0x35)[0x7ffff7471575]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>>>>>>> 0x21c)[0x7ffff747363c]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>>>>>>> 0x7ffff746ead8]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[
>>>>>>>> 0x7ffff745ed57]
>>>>>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>>>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>>>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>>>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>>>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>>>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>>>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>>>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>>>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>>>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>>>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>>>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>>>>>     at lttng-events.c:247
>>>>>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>>>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>>>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>>>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>>>>>     at lttng-ust-abi.c:235
>>>>>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>>>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>>>>>     at lttng-ust-comm.c:1807
>>>>>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>>>>>     at lttng-ust-comm.c:1874
>>>>>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>>>>>    from /lib64/ld-linux-x86-64.so.2
>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>    from /lib64/libc.so.6
>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>>>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>>>>>> argv=<optimized out>) at lu.c:368
>>>>>>>>
>>>>>>>> Any ideas, why this happens and how to fix it?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Shehab
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> lttng-dev mailing list
>>>>>>>> lttng-dev@lists.lttng.org
>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> http://www.efficios.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> http://www.efficios.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> http://www.efficios.com
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> http://www.efficios.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>>>
>>
>>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 63834 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 20498 bytes --]

----- On Mar 22, 2018, at 12:24 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote: 

> Actually, i am not sure if this would help. I have been trying to reproduce the
> problem on a different machine, but I can't. Even without the patches !!!!!

Does it have the same glibc version ? 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn@gmail.com ]

> On Wed, Mar 21, 2018 at 8:13 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
> wrote:

>> ----- On Mar 21, 2018, at 8:01 PM, Shehab Elsayed < [
>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>> Just to clarify more what I meant by custom events; I have new tracepoints added
>>> in the source code of the benchmark. However, I don't enable the corresponding
>>> events when I do the actual tracing.

>>> This is the sequence followed in building the benchmark:
>>> gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g
>>> -fno-strict-aliasing -DLTTNG_INST lu.c
>>> gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g
>>> -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o
>>> ../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl

>>> LTTNG_INST is just a preprocessor flag I have and tp.o is my custom tracepoints

>> Could you share a repository with your custom instrumentation on github, so I
>> could
>> try it out ?

>> My current problem is that I cannot reproduce your issue on my end.

>> Thanks,

>> Mathieu

>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>> shehabyomn@gmail.com ]

>>> On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed < [ mailto:shehabyomn@gmail.com
>>> | shehabyomn@gmail.com ] > wrote:

>>>> Still running into same problem. I attached the debug trace I got after applying
>>>> the 2 patches.

>>>> The benchmark I am running also includes some custom created tracepoints. I am
>>>> not adding the events being traced in the files I have provided. Do you think
>>>> this might be causing a problem when I have tracpoints from 2 different
>>>> packages?

>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>> shehabyomn@gmail.com ]

>>>> On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers < [
>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>> wrote:

>>>>> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed < [
>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>> I am so sorry for the late replies.

>>>>>> I have tried the first patch you sent and the problem still happens (although
>>>>>> seemingly less frequently especially with debugging enabled!!!!). I have
>>>>>> attached the output I got from one of the erroneous runs.

>>>>>> I will try the updated patch and let you know how it goes.

>>>>>> Also, I am not sure if these points make any difference:
>>>>>> 1- The error actually happens at the end of the application. It actually
>>>>>> finishes execution, but then something goes wrong.
>>>>>> 2- I run into this problem only for some of the benchmarks (or at least the
>>>>>> problems happens much less frequently for others that I didn't run into it, not
>>>>>> yet)

>>>>>> Thanks you very much, and sorry again for the late replies.

>>>>> No worries! Looking through your log, I notice that pthread set cancel state has
>>>>> problems when
>>>>> called from application threads. We do not restore the original thread's cancel
>>>>> state. Can you try
>>>>> with the attached patch applied on top of the previous one ?

>>>>> Thanks,

>>>>> Mathieu

>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>> shehabyomn@gmail.com ]

>>>>>> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers < [
>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>> wrote:

>>>>>>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers < [
>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>> wrote:

>>>>>>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers < [
>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>> wrote:

>>>>>>>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers < [
>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>> wrote:

>>>>>>>>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed < [
>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the
>>>>>>>>>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"

>>>>>>>>>>> It executed normally 2 times but then got the same error.

>>>>>>>>>> Can you provide the output when reproducing the issue with the
>>>>>>>>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>>>>>>>>> your test program ?

>>>>>>>>> I just noticed something that might explain what goes wrong here:

>>>>>>>>> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread
>>>>>>>>> mutex
>>>>>>>>> in your case is instrumented with the pthread wrapper. This "add_thread_field"
>>>>>>>>> is invoked
>>>>>>>>> the first time the perf counter is hit by each given thread. When this happens,
>>>>>>>>> the
>>>>>>>>> instrumented pthread mutex will try to call into the instrumentation tracepoint
>>>>>>>>> again,
>>>>>>>>> which will call add_thread_field() (again), and so on until we reach the
>>>>>>>>> libringbuffer
>>>>>>>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.

>>>>>>>>> I suspect this situation where we recursively call add_thread_field is
>>>>>>>>> unexpected,
>>>>>>>>> which may trigger your double-free here.

>>>>>>>>> Will investigate more.

>>>>>>>> Can you try with the attached patch applied ?

>>>>>>> Here is an updated v2 of the patch which tests the notrace tls counter sooner
>>>>>>> (before evaluating
>>>>>>> filter). Please let me know how it goes.

>>>>>>> Thanks,

>>>>>>> Mathieu

>>>>>>>> Thanks,

>>>>>>>> Mathieu

>>>>>>>>> Thanks,

>>>>>>>>> Mathieu

>>>>>>>>>> Thanks,

>>>>>>>>>> Mathieu

>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>> PhD Student
>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>> University of Toronto
>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [
>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>> wrote:

>>>>>>>>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [
>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1

>>>>>>>>>>>>> I run the program as a normal user

>>>>>>>>>>>>> The glibc version I get by running "ldd --version" is 2.17

>>>>>>>>>>>> Can you reproduce the issue after you do this as root ?

>>>>>>>>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid

>>>>>>>>>>>> Based on this documentation of the Linux kernel:

>>>>>>>>>>>> Documentation/sysctl/kernel.txt:

>>>>>>>>>>>> perf_event_paranoid:

>>>>>>>>>>>> Controls use of the performance events system by unprivileged
>>>>>>>>>>>> users (without CAP_SYS_ADMIN). The default value is 2.

>>>>>>>>>>>> -1: Allow use of (almost) all events by all users
>>>>>>>>>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>>>>>>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
>>>>>>>>>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>>>>>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>>>>>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN

>>>>>>>>>>>> Thanks,

>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [
>>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same
>>>>>>>>>>>>>>>> problem.

>>>>>>>>>>>>>>> What is the setting returned by

>>>>>>>>>>>>>>> cat /proc/sys/kernel/perf_event_paranoid

>>>>>>>>>>>>>>> on your system ? And do you run your test program as root or normal user ?

>>>>>>>>>>>>>>> Please CC the mailing list on your reply.

>>>>>>>>>>>>>> I will also need to know what glibc version you have on your system.

>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>>>>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>>>>> Hi Mathieu,

>>>>>>>>>>>>>>>>>> Thank you very much for your reply.

>>>>>>>>>>>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>>>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>>>>>>>>>>>>>>>>> This is how I set up the tracing session:
>>>>>>>>>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>>>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>>>>>>>>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>>>>>>>>>>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>>>>>>>>>>> lttng add-context -u -t vpid
>>>>>>>>>>>>>>>>>> lttng add-context -u -t pthread_id
>>>>>>>>>>>>>>>>>> lttng add-context -u -t vtid
>>>>>>>>>>>>>>>>>> lttng add-context -u -t ip
>>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions
>>>>>>>>>>>>>>>>>> 4- lttng start
>>>>>>>>>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>>>>>>>>>>>>>>>>> -b32
>>>>>>>>>>>>>>>>>> 6- lttng stop
>>>>>>>>>>>>>>>>>> 7- lttng destroy

>>>>>>>>>>>>>>>>> Can you reproduce if you remove the following contexts ?

>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:cycles
>>>>>>>>>>>>>>>>> lttng add-context -u -t perf:thread:instructions

>>>>>>>>>>>>>>>>> And if you only keep a single of those contexts ?

>>>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>>>>>>>>>>>>>> University of Toronto
>>>>>>>>>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>>>>>>>>>>>>>>>>> shehabyomn@gmail.com ]

>>>>>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>>>>>>>>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] >
>>>>>>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>>>>>>>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote:

>>>>>>>>>>>>>>>>>>>> Hello All,

>>>>>>>>>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>>>>>>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>>>>>>>>>>>>>>>>> )" error.

>>>>>>>>>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using,
>>>>>>>>>>>>>>>>>>> and how you setup
>>>>>>>>>>>>>>>>>>> your tracing session.

>>>>>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>>>>>> Mathieu

>>>>>>>>>>>>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>>>>>>>>>>>>>>>>> expected.
>>>>>>>>>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>>>>>>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>>>>>>>>>>>>>>>>> instructions).
>>>>>>>>>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>>>>>>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>>>>>>>>>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>>>>>>>>>>>>> ======= Backtrace : =========
>>>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>>>>>>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>>>>>>>>>>>>>>>>> +0x9f)[0x7ffff746dccf]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>>>>>>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>>>>>>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>>>>>>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>>>>>>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>>>>>>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>>>>>>>>>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>>>>>>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>>>>>>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>>>>>>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>>>>>>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>>>>>>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>>>>>>>>>>>>>>>>> at lttng -events.c:247
>>>>>>>>>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>>>>>>>>>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>>>>>>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>>>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>>>>>>>>>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18,
>>>>>>>>>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216
>>>>>>>>>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>>>>>>>>>>>>>>>>> at lttng - ust - abi .c:235
>>>>>>>>>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>>>>>>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>>>>>>>>>>>>>>>>> at lttng - ust -comm.c:1807
>>>>>>>>>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>>>>>>>>>>>>>>>>> at lttng - ust -comm.c:1874
>>>>>>>>>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>>>>>>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>>>>>>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>>>>>>>>>>>> from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>>>>>>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>>>>>>>>>>>>>>>>> argv =<optimized out>) at lu .c:368

>>>>>>>>>>>>>>>>>>>> Any ideas, why this happens and how to fix it?

>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Shehab

>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> lttng-dev mailing list
>>>>>>>>>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>>>>>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>>>>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>>>> --
>>>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>>> --
>>>>>>>>>> Mathieu Desnoyers
>>>>>>>>>> EfficiOS Inc.
>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>>> --
>>>>>>>>> Mathieu Desnoyers
>>>>>>>>> EfficiOS Inc.
>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 64983 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 18373 bytes --]

It is v2.17 (for the problematic system) and 2.23 for the other

Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto
E-mail: shehabyomn@gmail.com
<https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>

On Thu, Mar 22, 2018 at 1:00 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> ----- On Mar 22, 2018, at 12:24 PM, Shehab Elsayed <shehabyomn@gmail.com>
> wrote:
>
> Actually, i am not sure if this would help. I have been trying to
> reproduce the problem on a different machine, but I can't. Even without the
> patches !!!!!
>
>
> Does it have the same glibc version ?
>
> Thanks,
>
> Mathieu
>
>
>
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn@gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Wed, Mar 21, 2018 at 8:13 PM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com> wrote:
>
>> ----- On Mar 21, 2018, at 8:01 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>> Just to clarify more what I meant by custom events; I have new
>> tracepoints added in the source code of the benchmark. However, I don't
>> enable the corresponding events when I do the actual tracing.
>>
>> This is the sequence followed in building the benchmark:
>> gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112
>> -std=c11 -g -fno-strict-aliasing -DLTTNG_INST lu.c
>> gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112
>> -std=c11 -g -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o
>> ../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl
>>
>> LTTNG_INST is just a preprocessor flag I have and tp.o is my custom
>> tracepoints
>>
>> Could you share a repository with your custom instrumentation on github,
>> so I could
>> try it out ?
>>
>> My current problem is that I cannot reproduce your issue on my end.
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn@gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
>> wrote:
>>
>>> Still running into same problem. I attached the debug trace I got after
>>> applying the 2 patches.
>>>
>>> The benchmark I am running also includes some custom created
>>> tracepoints. I am not adding the events being traced in the files I have
>>> provided. Do you think this might be causing a problem when I have
>>> tracpoints from 2 different packages?
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn@gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers <
>>> mathieu.desnoyers@efficios.com> wrote:
>>>
>>>> ----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <shehabyomn@gmail.com>
>>>> wrote:
>>>>
>>>> I am so sorry for the late replies.
>>>>
>>>> I have tried the first patch you sent and the problem still happens
>>>> (although seemingly less frequently especially with debugging enabled!!!!).
>>>> I have attached the output I got from one of the erroneous runs.
>>>>
>>>> I will try the updated patch and let you know how it goes.
>>>>
>>>> Also, I am not sure if these points make any difference:
>>>> 1- The error actually happens at the end of the application. It
>>>> actually finishes execution, but then something goes wrong.
>>>> 2- I run into this problem only for some of the benchmarks (or at least
>>>> the problems happens much less frequently for others that I didn't run into
>>>> it, not yet)
>>>>
>>>> Thanks you very much, and sorry again for the late replies.
>>>>
>>>>
>>>> No worries! Looking through your log, I notice that pthread set cancel
>>>> state has problems when
>>>> called from application threads. We do not restore the original
>>>> thread's cancel state. Can you try
>>>> with the attached patch applied on top of the previous one ?
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>> Shehab Y. Elsayed, MSc.
>>>> PhD Student
>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>> University of Toronto
>>>> E-mail: shehabyomn@gmail.com
>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>
>>>> On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>> ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>> ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>>
>>>>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <
>>>>> shehabyomn@gmail.com> wrote:
>>>>>
>>>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid" and made
>>>>> sure the value was actually written by "cat /proc/sys/kernel/perf_event_
>>>>> paranoid"
>>>>>
>>>>> It executed normally 2 times but then got the same error.
>>>>>
>>>>>
>>>>> Can you provide the output when reproducing the issue with the
>>>>> LTTNG_UST_DEBUG=1 environment variable set when starting
>>>>> your test program ?
>>>>>
>>>>> I just noticed something that might explain what goes wrong here:
>>>>>
>>>>> lttng-context-perf-counters.c: add_thread_field() grabs the
>>>>> ust_lock(). Pthread mutex
>>>>> in your case is instrumented with the pthread wrapper. This
>>>>> "add_thread_field" is invoked
>>>>> the first time the perf counter is hit by each given thread. When this
>>>>> happens, the
>>>>> instrumented pthread mutex will try to call into the instrumentation
>>>>> tracepoint again,
>>>>> which will call add_thread_field() (again), and so on until we reach
>>>>> the libringbuffer
>>>>> "lib_ring_buffer_nesting" threshold of 4 calls deep.
>>>>>
>>>>> I suspect this situation where we recursively call add_thread_field is
>>>>> unexpected,
>>>>> which may trigger your double-free here.
>>>>>
>>>>> Will investigate more.
>>>>>
>>>>> Can you try with the attached patch applied ?
>>>>>
>>>>> Here is an updated v2 of the patch which tests the notrace tls counter
>>>>> sooner (before evaluating
>>>>> filter). Please let me know how it goes.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Shehab Y. Elsayed, MSc.
>>>>> PhD Student
>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>> University of Toronto
>>>>> E-mail: shehabyomn@gmail.com
>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <
>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <
>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>
>>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1
>>>>>>
>>>>>> I run the program as a normal user
>>>>>>
>>>>>> The glibc version I get by running "ldd --version" is 2.17
>>>>>>
>>>>>>
>>>>>> Can you reproduce the issue after you do this as root ?
>>>>>>
>>>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid
>>>>>>
>>>>>> Based on this documentation of the Linux kernel:
>>>>>>
>>>>>> Documentation/sysctl/kernel.txt:
>>>>>>
>>>>>> perf_event_paranoid:
>>>>>>
>>>>>> Controls use of the performance events system by unprivileged
>>>>>> users (without CAP_SYS_ADMIN).  The default value is 2.
>>>>>>
>>>>>>  -1: Allow use of (almost) all events by all users
>>>>>>      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>>>>>> >=0: Disallow ftrace function tracepoint by users without
>>>>>> CAP_SYS_ADMIN
>>>>>>      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>>>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN
>>>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>
>>>>>>
>>>>>> Shehab Y. Elsayed, MSc.
>>>>>> PhD Student
>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>>>>> University of Toronto
>>>>>> E-mail: shehabyomn@gmail.com
>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <
>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>
>>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <
>>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>>
>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <
>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>
>>>>>>> Yes, I tried with only one of those contexts and I still ran into
>>>>>>> the same problem.
>>>>>>>
>>>>>>> What is the setting returned by
>>>>>>>
>>>>>>> cat /proc/sys/kernel/perf_event_paranoid
>>>>>>>
>>>>>>> on your system ? And do you run your test program as root or normal
>>>>>>> user ?
>>>>>>>
>>>>>>> Please CC the mailing list on your reply.
>>>>>>>
>>>>>>>
>>>>>>> I will also need to know what glibc version you have on your system.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mathieu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mathieu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>> PhD Student
>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer
>>>>>>> Engineering
>>>>>>> University of Toronto
>>>>>>> E-mail: shehabyomn@gmail.com
>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <
>>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>>
>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <
>>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Mathieu,
>>>>>>>>
>>>>>>>> Thank you very much for your reply.
>>>>>>>>
>>>>>>>> I manually built lttng-ust from source (commit #:
>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a).
>>>>>>>>
>>>>>>>> This is how I set up the tracing session:
>>>>>>>> 1- lttng create lu_ncb_8_native -o {path}
>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>>> mutex_lock_req
>>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>>> mutex_lock_acq
>>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>>> mutex_lock_trylock
>>>>>>>>     lttng enable-event --userspace lttng_ust_pthread:pthread_
>>>>>>>> mutex_lock_unlock
>>>>>>>> 3- lttng add-context -u -t procname
>>>>>>>>     lttng add-context -u -t vpid
>>>>>>>>     lttng add-context -u -t pthread_id
>>>>>>>>     lttng add-context -u -t vtid
>>>>>>>>     lttng add-context -u -t ip
>>>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>>>> 4- lttng start
>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so
>>>>>>>> ./lu_ncb -p8 -n8096 -b32
>>>>>>>> 6- lttng stop
>>>>>>>> 7- lttng destroy
>>>>>>>>
>>>>>>>>
>>>>>>>> Can you reproduce if you remove the following contexts ?
>>>>>>>>
>>>>>>>>     lttng add-context -u -t perf:thread:cpu-cycles
>>>>>>>>     lttng add-context -u -t perf:thread:cycles
>>>>>>>>     lttng add-context -u -t perf:thread:instructions
>>>>>>>>
>>>>>>>> And if you only keep a single of those contexts ?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Mathieu
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Shehab Y. Elsayed, MSc.
>>>>>>>> PhD Student
>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer
>>>>>>>> Engineering
>>>>>>>> University of Toronto
>>>>>>>> E-mail: shehabyomn@gmail.com
>>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <
>>>>>>>> mathieu.desnoyers@efficios.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <
>>>>>>>>> shehabyomn@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello All,
>>>>>>>>>
>>>>>>>>> I am trying to instrument a pthread application using the
>>>>>>>>> provided pthread wrapper, but I sometimes run into a "Double free
>>>>>>>>> or corruption error (fasttop)" error.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please provide more information about the version of lttng-ust you
>>>>>>>>> are using, and how you setup
>>>>>>>>> your tracing session.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Mathieu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is a description of what I have tried and noticed:
>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes
>>>>>>>>> works as expected.
>>>>>>>>> 2- From my experiments, the problem happens (more frequently at
>>>>>>>>> least) when adding performance counter contexts (I tried cycles,
>>>>>>>>> cpu_cycles and instructions).
>>>>>>>>> 3- I am testing using lu_ncb from splash3 benchmark suite after
>>>>>>>>> setting LD_PRELOAD to use the pthread wrapper as described in the
>>>>>>>>> LTTng documents.
>>>>>>>>> 4- Here is the backtrace printed after exiting:
>>>>>>>>> ======= Backtrace: =========
>>>>>>>>> /lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+
>>>>>>>>> 0x35)[0x7ffff7471575]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+
>>>>>>>>> 0x21c)[0x7ffff747363c]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+
>>>>>>>>> 0x9f)[0x7ffff746dccf]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[
>>>>>>>>> 0x7ffff746ead8]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
>>>>>>>>> /usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[
>>>>>>>>> 0x7ffff745ed57]
>>>>>>>>> /lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>>>>>> /lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>>>>>> /lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
>>>>>>>>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
>>>>>>>>> /aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb:
>>>>>>>>> #0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
>>>>>>>>> #1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
>>>>>>>>> #2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
>>>>>>>>> #3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
>>>>>>>>> #4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
>>>>>>>>>     field=<optimized out>) at lttng-context-perf-counters.c:418
>>>>>>>>> #5  0x00007ffff767a575 in lttng_destroy_context (
>>>>>>>>> ctx=0x7ffff0011090) at lttng-context.c:278
>>>>>>>>> #6  0x00007ffff767c63c in _lttng_channel_unmap (
>>>>>>>>> lttng_chan=0x7ffff0010f40) at lttng-events.c:172
>>>>>>>>> #7  lttng_session_destroy (session=0x7ffff0000900)
>>>>>>>>>     at lttng-events.c:247
>>>>>>>>> #8  0x00007ffff7676906 in lttng_release_session (
>>>>>>>>> objd=<optimized out>) at lttng-ust-abi.c:601
>>>>>>>>> #9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
>>>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>>>> #10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
>>>>>>>>>     is_owner=<optimized out>) at lttng-ust-abi.c:216
>>>>>>>>> #11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
>>>>>>>>>     is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
>>>>>>>>> #12 0x00007ffff7677ad8 in objd_table_destroy ()
>>>>>>>>>     at lttng-ust-abi.c:235
>>>>>>>>> #13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
>>>>>>>>> #14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
>>>>>>>>>     at lttng-ust-comm.c:1807
>>>>>>>>> #15 0x00007ffff7667d57 in lttng_ust_exit ()
>>>>>>>>>     at lttng-ust-comm.c:1874
>>>>>>>>> #16 0x00007ffff7dec85a in _dl_fini ()
>>>>>>>>>    from /lib64/ld-linux-x86-64.so.2
>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>>>>>>    from /lib64/libc.so.6
>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
>>>>>>>>> #19 0x0000000000401b51 in main (argc=<optimized out>,
>>>>>>>>> argv=<optimized out>) at lu.c:368
>>>>>>>>>
>>>>>>>>> Any ideas, why this happens and how to fix it?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Shehab
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> lttng-dev mailing list
>>>>>>>>> lttng-dev@lists.lttng.org
>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Mathieu Desnoyers
>>>>>>>>> EfficiOS Inc.
>>>>>>>>> http://www.efficios.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mathieu Desnoyers
>>>>>>>> EfficiOS Inc.
>>>>>>>> http://www.efficios.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> http://www.efficios.com
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mathieu Desnoyers
>>>>>>> EfficiOS Inc.
>>>>>>> http://www.efficios.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mathieu Desnoyers
>>>>>> EfficiOS Inc.
>>>>>> http://www.efficios.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>>
>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com
>>>>
>>>
>>>
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 69802 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Double free or corruption error (fasttop)

[Shehab Elsayed]

[-- Attachment #1.1: Type: text/plain, Size: 3488 bytes --]

Hello All,

I am trying to instrument a pthread application using the provided pthread
wrapper, but I sometimes run into a "Double free or corruption error (
fasttop)" error.

Here is a description of what I have tried and noticed:
1- The problem isn't consistent. It sometimes happen and sometimes works as
expected.
2- From my experiments, the problem happens (more frequently at least) when
adding performance counter contexts (I tried cycles, cpu_cycles and
instructions).
3- I am testing using lu_ncb from splash3 benchmark suite after setting LD_
PRELOAD to use the pthread wrapper as described in the LTTng documents.
4- Here is the backtrace printed after exiting:
======= Backtrace: =========
/lib64/libc.so.6([Thread 0x7ffff5611700 (LWP 97229) exited]
/usr/local/lib/liblttng-ust.so.0(lttng_destroy_context+0x35)[0x7ffff7471575]
/usr/local/lib/liblttng-ust.so.0(lttng
_session_destroy+0x21c)[0x7ffff747363c]
/usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7ffff746dccf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7ffff746dccf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_objd_unref+0x9f)[0x7ffff746dccf]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x7ffff746ead8]
/usr/local/lib/liblttng-ust.so.0(+0x191d3)[0x7ffff74681d3]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
/lib64/ld-linux-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
/lib64/libc.so.6(+0x38a49)[0x7ffff6ca6a49]
/lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
/aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401b51]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]
/aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
5- Also, this is a backtrace I obtained from gdb:
#0  0x00007ffff6eac1d7 in raise () from /lib64/libc.so.6
#1  0x00007ffff6ead8c8 in abort () from /lib64/libc.so.6
#2  0x00007ffff6eebf07 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
#4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
    field=<optimized out>) at lttng-context-perf-counters.c:418
#5  0x00007ffff767a575 in lttng_destroy_context (
    ctx=0x7ffff0011090) at lttng-context.c:278
#6  0x00007ffff767c63c in _lttng_channel_unmap (
    lttng_chan=0x7ffff0010f40) at lttng-events.c:172
#7  lttng_session_destroy (session=0x7ffff0000900)
    at lttng-events.c:247
#8  0x00007ffff7676906 in lttng_release_session (
    objd=<optimized out>) at lttng-ust-abi.c:601
#9  0x00007ffff7676ccf in lttng_ust_objd_unref (id=1,
    is_owner=<optimized out>) at lttng-ust-abi.c:216
#10 0x00007ffff7676ccf in lttng_ust_objd_unref (id=2,
    is_owner=<optimized out>) at lttng-ust-abi.c:216
#11 0x00007ffff7676ccf in lttng_ust_objd_unref (id=id@entry=18,
    is_owner=is_owner@entry=1) at lttng-ust-abi.c:216
#12 0x00007ffff7677ad8 in objd_table_destroy ()
    at lttng-ust-abi.c:235
#13 lttng_ust_abi_exit () at lttng-ust-abi.c:1002
#14 0x00007ffff76711d3 in lttng_ust_cleanup (exiting=1)
    at lttng-ust-comm.c:1807
#15 0x00007ffff7667d57 in lttng_ust_exit ()
    at lttng-ust-comm.c:1874
#16 0x00007ffff7dec85a in _dl_fini ()
   from /lib64/ld-linux-x86-64.so.2
#17 0x00007ffff6eafa49 in __run_exit_handlers ()
   from /lib64/libc.so.6
#18 0x00007ffff6eafa95 in exit () from /lib64/libc.so.6
#19 0x0000000000401b51 in main (argc=<optimized out>,
    argv=<optimized out>) at lu.c:368

Any ideas, why this happens and how to fix it?

Thanks,
Shehab

[-- Attachment #1.2: Type: text/html, Size: 12774 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev