Re: segfault when exiting process with background thread

Re: segfault when exiting process with background thread

[Jonathan Rajotte Julien]

[-- Attachment #1.1: Type: text/plain, Size: 2501 bytes --]

Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules 
(if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:
>
> Hi LTTng community:
>
>
> We are using LTTng for our production environment. We have been 
> noticing segfault problem when our process exit. We think the problem 
> is due to the background thread continue to write traces, while LTTng 
> cleanup its memory. We do not know how to fix this issue without 
> changing LTTng code base. I have written a very simple app that could 
> repro the problem. Most of my codes are copy pasted from the LTTng doc 
> sample. Is there any fix that the LTTng side could do?
>
>
>
> *_hello.c_*
>
> #include <stdio.h>
> #include "hello-tp.h"
> #include <pthread.h>
>
>
> void* doSomeThing(void *arg)
> {
>     int x;
>     for (x = 0; x < 100000; ++x) {
>         tracepoint(hello_world, my_first_tracepoint, x, "test");
>     }
> }
>
> int main(int argc, char *argv[])
> {
>     int x;
>
>     getchar();
>
>     pthread_t inc_x_thread;
>
>     if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {
>
>         fprintf(stderr, "Error creating thread\n");
>         return 1;
>     }
>
>     tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
>
>     return 0;
> }
>
>
> *_hello-tp.c_*
>
> #define TRACEPOINT_CREATE_PROBES
> #define TRACEPOINT_DEFINE
>
> #include "hello-tp.h"
>
>
> *_hello-tp.h_*
>
> #undef TRACEPOINT_PROVIDER
> #define TRACEPOINT_PROVIDER hello_world
>
> #undef TRACEPOINT_INCLUDE
> #define TRACEPOINT_INCLUDE "./hello-tp.h"
>
> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
> #define _HELLO_TP_H
>
> #include <lttng/tracepoint.h>
>
> TRACEPOINT_EVENT(
>     hello_world,
>     my_first_tracepoint,
>     TP_ARGS(
>         int, my_integer_arg,
>         char*, my_string_arg
>     ),
>     TP_FIELDS(
>         ctf_string(my_string_field, my_string_arg)
>         ctf_integer(int, my_integer_field, my_integer_arg)
>     )
> )
>
> #endif /* _HELLO_TP_H */
>
> #include <lttng/tracepoint-event.h>
>
>
> *_Compile_*
>
> gcc -c -I. hello-tp.c
> gcc -c hello.c
> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread
>
>
>
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Jonathan R. Julien
Efficios


[-- Attachment #1.2: Type: text/html, Size: 5192 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 2949 bytes --]

Reproduced it on master (ust and tools). 

Repro with: 

lttng create 
lttng enable-event -u -a 
lttng start 

then run hello a few times, it eventually segfaults in the spawned thread. 

Looking into it. 

Thanks, 

Mathieu 

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote: 

> Hi Jeffrey,

> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
> installed)?

> Were you able to reproduce the issue on master (git) ?

> What are the exact order of command to reproduce the problem ?

> Thanks

> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>> Hi LTTng community:

>> We are using LTTng for our production environment. We have been noticing
>> segfault problem when our process exit. We think the problem is due to the
>> background thread continue to write traces, while LTTng cleanup its memory. We
>> do not know how to fix this issue without changing LTTng code base. I have
>> written a very simple app that could repro the problem. Most of my codes are
>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>> could do?

>> hello.c

>> #include <stdio.h>
>> #include "hello-tp.h"
>> #include <pthread.h>

>> void* doSomeThing(void *arg)
>> {
>> int x;
>> for (x = 0; x < 100000; ++x) {
>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>> }
>> }

>> int main(int argc, char *argv[])
>> {
>> int x;

>> getchar();

>> pthread_t inc_x_thread;

>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>> fprintf(stderr, "Error creating thread\n");
>> return 1;
>> }

>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>> return 0;
>> }

>> hello-tp.c

>> #define TRACEPOINT_CREATE_PROBES
>> #define TRACEPOINT_DEFINE

>> #include "hello-tp.h"

>> hello-tp.h

>> #undef TRACEPOINT_PROVIDER
>> #define TRACEPOINT_PROVIDER hello_world

>> #undef TRACEPOINT_INCLUDE
>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>> #define _HELLO_TP_H

>> #include <lttng/tracepoint.h>

>> TRACEPOINT_EVENT(
>> hello_world,
>> my_first_tracepoint,
>> TP_ARGS(
>> int, my_integer_arg,
>> char*, my_string_arg
>> ),
>> TP_FIELDS(
>> ctf_string(my_string_field, my_string_arg)
>> ctf_integer(int, my_integer_field, my_integer_arg)
>> )
>> )

>> #endif /* _HELLO_TP_H */

>> #include <lttng/tracepoint-event.h>

>> Compile

>> gcc -c -I. hello-tp.c
>> gcc -c hello.c
>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>> _______________________________________________
>> lttng-dev mailing list lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

> --
> Jonathan R. Julien
> Efficios

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 7238 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 3695 bytes --]

The attached patch works around the issue. The problem is 
the tracepoint destructors that run in parallel with the thread 
that still exists. 

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared 
object on dlclose of the instrumented code. Not sure if we should 
care though... 

Thoughts ? 

Thanks, 

Mathieu 

----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: 

> Reproduced it on master (ust and tools).

> Repro with:

> lttng create
> lttng enable-event -u -a
> lttng start

> then run hello a few times, it eventually segfaults in the spawned thread.

> Looking into it.

> Thanks,

> Mathieu

> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
> <Jonathan.rajotte-julien@efficios.com> wrote:

>> Hi Jeffrey,

>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>> installed)?

>> Were you able to reproduce the issue on master (git) ?

>> What are the exact order of command to reproduce the problem ?

>> Thanks

>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>> Hi LTTng community:

>>> We are using LTTng for our production environment. We have been noticing
>>> segfault problem when our process exit. We think the problem is due to the
>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>> do not know how to fix this issue without changing LTTng code base. I have
>>> written a very simple app that could repro the problem. Most of my codes are
>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>> could do?

>>> hello.c

>>> #include <stdio.h>
>>> #include "hello-tp.h"
>>> #include <pthread.h>

>>> void* doSomeThing(void *arg)
>>> {
>>> int x;
>>> for (x = 0; x < 100000; ++x) {
>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>> }
>>> }

>>> int main(int argc, char *argv[])
>>> {
>>> int x;

>>> getchar();

>>> pthread_t inc_x_thread;

>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>> fprintf(stderr, "Error creating thread\n");
>>> return 1;
>>> }

>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>> return 0;
>>> }

>>> hello-tp.c

>>> #define TRACEPOINT_CREATE_PROBES
>>> #define TRACEPOINT_DEFINE

>>> #include "hello-tp.h"

>>> hello-tp.h

>>> #undef TRACEPOINT_PROVIDER
>>> #define TRACEPOINT_PROVIDER hello_world

>>> #undef TRACEPOINT_INCLUDE
>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>> #define _HELLO_TP_H

>>> #include <lttng/tracepoint.h>

>>> TRACEPOINT_EVENT(
>>> hello_world,
>>> my_first_tracepoint,
>>> TP_ARGS(
>>> int, my_integer_arg,
>>> char*, my_string_arg
>>> ),
>>> TP_FIELDS(
>>> ctf_string(my_string_field, my_string_arg)
>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>> )
>>> )

>>> #endif /* _HELLO_TP_H */

>>> #include <lttng/tracepoint-event.h>

>>> Compile

>>> gcc -c -I. hello-tp.c
>>> gcc -c hello.c
>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>> _______________________________________________
>>> lttng-dev mailing list lttng-dev@lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Jonathan R. Julien
>> Efficios

>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 8170 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: workaround-destructor-vs-threads.patch --]
[-- Type: text/x-patch; name=workaround-destructor-vs-threads.patch, Size: 1199 bytes --]

commit fcf3b4b09fa4558014328f34b0a1c396a03b6515
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Mar 8 21:05:32 2016 -0500

    Workaround

diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 16348b8..d9e2003 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -301,6 +301,7 @@ __tracepoints__destroy(void)
 
 	if (--__tracepoint_registered)
 		return;
+#if 0
 	if (tracepoint_dlopen.liblttngust_handle && !__tracepoint_ptrs_registered) {
 		ret = dlclose(tracepoint_dlopen.liblttngust_handle);
 		if (ret) {
@@ -309,6 +310,7 @@ __tracepoints__destroy(void)
 		}
 		memset(&tracepoint_dlopen, 0, sizeof(tracepoint_dlopen));
 	}
+#endif
 }
 
 #endif
@@ -404,6 +406,7 @@ __tracepoints__ptrs_destroy(void)
 
 	if (--__tracepoint_ptrs_registered)
 		return;
+#if 0
 	if (tracepoint_dlopen.tracepoint_unregister_lib)
 		tracepoint_dlopen.tracepoint_unregister_lib(__start___tracepoints_ptrs);
 	if (tracepoint_dlopen.liblttngust_handle && !__tracepoint_registered) {
@@ -414,6 +417,7 @@ __tracepoints__ptrs_destroy(void)
 		}
 		memset(&tracepoint_dlopen, 0, sizeof(tracepoint_dlopen));
 	}
+#endif
 }
 
 #else /* TRACEPOINT_DEFINE */

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 4151 bytes --]

Thanks Mathieu.

Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 8, 2016 6:09 PM
To: Jonathan Rajotte Julien
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread

The attached patch works around the issue. The problem is
the tracepoint destructors that run in parallel with the thread
that still exists.

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
object on dlclose of the instrumented code. Not sure if we should
care though...

Thoughts ?

Thanks,

Mathieu


----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
Reproduced it on master (ust and tools).

Repro with:

lttng create
lttng enable-event -u -a
lttng start

then run hello a few times, it eventually segfaults in the spawned thread.

Looking into it.

Thanks,

Mathieu

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:
Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread







_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev



--
Jonathan R. Julien
Efficios

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 7958 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 4616 bytes --]

Hi Jeffrey, 

I CC'd your @live.com address on the RFC patch. Please let me know if the approach 
is OK with you (and try it out). 

See: 
https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html 

Thanks, 

Mathieu 

----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote: 

> Thanks Mathieu.

> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
> fix. If not, we will have to workaround the problem for now (probably by
> applying your fix ourselves). Thanks.

> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Sent: Tuesday, March 8, 2016 6:09 PM
> To: Jonathan Rajotte Julien
> Cc: lttng-dev
> Subject: Re: [lttng-dev] segfault when exiting process with background thread
> The attached patch works around the issue. The problem is
> the tracepoint destructors that run in parallel with the thread
> that still exists.

> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
> object on dlclose of the instrumented code. Not sure if we should
> care though...

> Thoughts ?

> Thanks,

> Mathieu

> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:

>> Reproduced it on master (ust and tools).

>> Repro with:

>> lttng create
>> lttng enable-event -u -a
>> lttng start

>> then run hello a few times, it eventually segfaults in the spawned thread.

>> Looking into it.

>> Thanks,

>> Mathieu

>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>> <Jonathan.rajotte-julien@efficios.com> wrote:

>>> Hi Jeffrey,

>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>> installed)?

>>> Were you able to reproduce the issue on master (git) ?

>>> What are the exact order of command to reproduce the problem ?

>>> Thanks

>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>> Hi LTTng community:

>>>> We are using LTTng for our production environment. We have been noticing
>>>> segfault problem when our process exit. We think the problem is due to the
>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>> written a very simple app that could repro the problem. Most of my codes are
>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>> could do?

>>>> hello.c

>>>> #include <stdio.h>
>>>> #include "hello-tp.h"
>>>> #include <pthread.h>

>>>> void* doSomeThing(void *arg)
>>>> {
>>>> int x;
>>>> for (x = 0; x < 100000; ++x) {
>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>> }
>>>> }

>>>> int main(int argc, char *argv[])
>>>> {
>>>> int x;

>>>> getchar();

>>>> pthread_t inc_x_thread;

>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>> fprintf(stderr, "Error creating thread\n");
>>>> return 1;
>>>> }

>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>> return 0;
>>>> }

>>>> hello-tp.c

>>>> #define TRACEPOINT_CREATE_PROBES
>>>> #define TRACEPOINT_DEFINE

>>>> #include "hello-tp.h"

>>>> hello-tp.h

>>>> #undef TRACEPOINT_PROVIDER
>>>> #define TRACEPOINT_PROVIDER hello_world

>>>> #undef TRACEPOINT_INCLUDE
>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>> #define _HELLO_TP_H

>>>> #include <lttng/tracepoint.h>

>>>> TRACEPOINT_EVENT(
>>>> hello_world,
>>>> my_first_tracepoint,
>>>> TP_ARGS(
>>>> int, my_integer_arg,
>>>> char*, my_string_arg
>>>> ),
>>>> TP_FIELDS(
>>>> ctf_string(my_string_field, my_string_arg)
>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>> )
>>>> )

>>>> #endif /* _HELLO_TP_H */

>>>> #include <lttng/tracepoint-event.h>

>>>> Compile

>>>> gcc -c -I. hello-tp.c
>>>> gcc -c hello.c
>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>> _______________________________________________
>>>> lttng-dev mailing list lttng-dev@lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> --
>>> Jonathan R. Julien
>>> Efficios

>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 9181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 5252 bytes --]

Thanks, Mathieu.


Yes, the fix works as a workaround. This is also the same workaround we have now.

It seems the problem is due to a race condition, that it checked whether "tracepoint_dlopen.rcu_read_lock_sym_bp" is not NULL. If not NULL, it calls to that. But, at this moment, memset cleared the memory.

This fix is okay with us, but I am not certain if it would break other component, since I do not understand LTTng enough to tell. Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 10:52 AM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread

Hi Jeffrey,

I CC'd your @live.com address on the RFC patch. Please let me know if the approach
is OK with you (and try it out).

See:
https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

Thanks,

Mathieu


----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks Mathieu.

Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 8, 2016 6:09 PM
To: Jonathan Rajotte Julien
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread

The attached patch works around the issue. The problem is
the tracepoint destructors that run in parallel with the thread
that still exists.

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
object on dlclose of the instrumented code. Not sure if we should
care though...

Thoughts ?

Thanks,

Mathieu


----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
Reproduced it on master (ust and tools).

Repro with:

lttng create
lttng enable-event -u -a
lttng start

then run hello a few times, it eventually segfaults in the spawned thread.

Looking into it.

Thanks,

Mathieu

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:
Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread







_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev



--
Jonathan R. Julien
Efficios

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 10446 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 5881 bytes --]

----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote: 

> Thanks, Mathieu.

> Yes, the fix works as a workaround. This is also the same workaround we have
> now.

> It seems the problem is due to a race condition, that it checked whether "
> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls to
> that. But, at this moment, memset cleared the memory.

> This fix is okay with us, but I am not certain if it would break other
> component, since I do not understand LTTng enough to tell. Thanks.

In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint 
shared object will clear the memory containing the called functions. Therefore, 
we need to skip both steps of the destructor if we want threads to survive after 
this destructor execution. 

Thanks, 

Mathieu 

> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Sent: Thursday, March 10, 2016 10:52 AM
> To: Jeffrey Chen
> Cc: Jonathan Rajotte Julien; lttng-dev
> Subject: Re: segfault when exiting process with background thread
> Hi Jeffrey,

> I CC'd your @live.com address on the RFC patch. Please let me know if the
> approach
> is OK with you (and try it out).

> See:
> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

> Thanks,

> Mathieu

> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>> Thanks Mathieu.

>> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
>> fix. If not, we will have to workaround the problem for now (probably by
>> applying your fix ourselves). Thanks.

>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Sent: Tuesday, March 8, 2016 6:09 PM
>> To: Jonathan Rajotte Julien
>> Cc: lttng-dev
>> Subject: Re: [lttng-dev] segfault when exiting process with background thread
>> The attached patch works around the issue. The problem is
>> the tracepoint destructors that run in parallel with the thread
>> that still exists.

>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
>> object on dlclose of the instrumented code. Not sure if we should
>> care though...

>> Thoughts ?

>> Thanks,

>> Mathieu

>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
>> <mathieu.desnoyers@efficios.com> wrote:

>>> Reproduced it on master (ust and tools).

>>> Repro with:

>>> lttng create
>>> lttng enable-event -u -a
>>> lttng start

>>> then run hello a few times, it eventually segfaults in the spawned thread.

>>> Looking into it.

>>> Thanks,

>>> Mathieu

>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>>> <Jonathan.rajotte-julien@efficios.com> wrote:

>>>> Hi Jeffrey,

>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>>> installed)?

>>>> Were you able to reproduce the issue on master (git) ?

>>>> What are the exact order of command to reproduce the problem ?

>>>> Thanks

>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>>> Hi LTTng community:

>>>>> We are using LTTng for our production environment. We have been noticing
>>>>> segfault problem when our process exit. We think the problem is due to the
>>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>>> written a very simple app that could repro the problem. Most of my codes are
>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>>> could do?

>>>>> hello.c

>>>>> #include <stdio.h>
>>>>> #include "hello-tp.h"
>>>>> #include <pthread.h>

>>>>> void* doSomeThing(void *arg)
>>>>> {
>>>>> int x;
>>>>> for (x = 0; x < 100000; ++x) {
>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>>> }
>>>>> }

>>>>> int main(int argc, char *argv[])
>>>>> {
>>>>> int x;

>>>>> getchar();

>>>>> pthread_t inc_x_thread;

>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>>> fprintf(stderr, "Error creating thread\n");
>>>>> return 1;
>>>>> }

>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>>> return 0;
>>>>> }

>>>>> hello-tp.c

>>>>> #define TRACEPOINT_CREATE_PROBES
>>>>> #define TRACEPOINT_DEFINE

>>>>> #include "hello-tp.h"

>>>>> hello-tp.h

>>>>> #undef TRACEPOINT_PROVIDER
>>>>> #define TRACEPOINT_PROVIDER hello_world

>>>>> #undef TRACEPOINT_INCLUDE
>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>>> #define _HELLO_TP_H

>>>>> #include <lttng/tracepoint.h>

>>>>> TRACEPOINT_EVENT(
>>>>> hello_world,
>>>>> my_first_tracepoint,
>>>>> TP_ARGS(
>>>>> int, my_integer_arg,
>>>>> char*, my_string_arg
>>>>> ),
>>>>> TP_FIELDS(
>>>>> ctf_string(my_string_field, my_string_arg)
>>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>>> )
>>>>> )

>>>>> #endif /* _HELLO_TP_H */

>>>>> #include <lttng/tracepoint-event.h>

>>>>> Compile

>>>>> gcc -c -I. hello-tp.c
>>>>> gcc -c hello.c
>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>>> _______________________________________________
>>>>> lttng-dev mailing list lttng-dev@lists.lttng.org
>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>> --
>>>> Jonathan R. Julien
>>>> Efficios

>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev@lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 11385 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 6015 bytes --]

Hi Mathieu:


Will this fix be checked-in? Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 3:27 PM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread



----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks, Mathieu.


Yes, the fix works as a workaround. This is also the same workaround we have now.

It seems the problem is due to a race condition, that it checked whether "tracepoint_dlopen.rcu_read_lock_sym_bp" is not NULL. If not NULL, it calls to that. But, at this moment, memset cleared the memory.

This fix is okay with us, but I am not certain if it would break other component, since I do not understand LTTng enough to tell. Thanks.

In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint
shared object will clear the memory containing the called functions. Therefore,
we need to skip both steps of the destructor if we want threads to survive after
this destructor execution.

Thanks,

Mathieu





________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 10:52 AM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread
Hi Jeffrey,

I CC'd your @live.com address on the RFC patch. Please let me know if the approach
is OK with you (and try it out).

See:
https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

Thanks,

Mathieu


----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks Mathieu.

Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 8, 2016 6:09 PM
To: Jonathan Rajotte Julien
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread
The attached patch works around the issue. The problem is
the tracepoint destructors that run in parallel with the thread
that still exists.

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
object on dlclose of the instrumented code. Not sure if we should
care though...

Thoughts ?

Thanks,

Mathieu


----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
Reproduced it on master (ust and tools).

Repro with:

lttng create
lttng enable-event -u -a
lttng start

then run hello a few times, it eventually segfaults in the spawned thread.

Looking into it.

Thanks,

Mathieu

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:
Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread







_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


--
Jonathan R. Julien
Efficios

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 12578 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 6531 bytes --]

Commit merged into master, thanks for your feedback! 

Mathieu 

----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk@hotmail.com> wrote: 

> Hi Mathieu:

> Will this fix be checked-in? Thanks.

> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Sent: Thursday, March 10, 2016 3:27 PM
> To: Jeffrey Chen
> Cc: Jonathan Rajotte Julien; lttng-dev
> Subject: Re: segfault when exiting process with background thread

> ----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>> Thanks, Mathieu.

>> Yes, the fix works as a workaround. This is also the same workaround we have
>> now.

>> It seems the problem is due to a race condition, that it checked whether "
>> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls to
>> that. But, at this moment, memset cleared the memory.

>> This fix is okay with us, but I am not certain if it would break other
>> component, since I do not understand LTTng enough to tell. Thanks.

> In addition to memset clearing that memory, dlclose() of the
> lttng-ust-tracepoint
> shared object will clear the memory containing the called functions. Therefore,
> we need to skip both steps of the destructor if we want threads to survive after
> this destructor execution.

> Thanks,

> Mathieu

>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Sent: Thursday, March 10, 2016 10:52 AM
>> To: Jeffrey Chen
>> Cc: Jonathan Rajotte Julien; lttng-dev
>> Subject: Re: segfault when exiting process with background thread
>> Hi Jeffrey,

>> I CC'd your @live.com address on the RFC patch. Please let me know if the
>> approach
>> is OK with you (and try it out).

>> See:
>> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

>> Thanks,

>> Mathieu

>> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>>> Thanks Mathieu.

>>> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
>>> fix. If not, we will have to workaround the problem for now (probably by
>>> applying your fix ourselves). Thanks.

>>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>> Sent: Tuesday, March 8, 2016 6:09 PM
>>> To: Jonathan Rajotte Julien
>>> Cc: lttng-dev
>>> Subject: Re: [lttng-dev] segfault when exiting process with background thread
>>> The attached patch works around the issue. The problem is
>>> the tracepoint destructors that run in parallel with the thread
>>> that still exists.

>>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
>>> object on dlclose of the instrumented code. Not sure if we should
>>> care though...

>>> Thoughts ?

>>> Thanks,

>>> Mathieu

>>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
>>> <mathieu.desnoyers@efficios.com> wrote:

>>>> Reproduced it on master (ust and tools).

>>>> Repro with:

>>>> lttng create
>>>> lttng enable-event -u -a
>>>> lttng start

>>>> then run hello a few times, it eventually segfaults in the spawned thread.

>>>> Looking into it.

>>>> Thanks,

>>>> Mathieu

>>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>>>> <Jonathan.rajotte-julien@efficios.com> wrote:

>>>>> Hi Jeffrey,

>>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>>>> installed)?

>>>>> Were you able to reproduce the issue on master (git) ?

>>>>> What are the exact order of command to reproduce the problem ?

>>>>> Thanks

>>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>>>> Hi LTTng community:

>>>>>> We are using LTTng for our production environment. We have been noticing
>>>>>> segfault problem when our process exit. We think the problem is due to the
>>>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>>>> written a very simple app that could repro the problem. Most of my codes are
>>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>>>> could do?

>>>>>> hello.c

>>>>>> #include <stdio.h>
>>>>>> #include "hello-tp.h"
>>>>>> #include <pthread.h>

>>>>>> void* doSomeThing(void *arg)
>>>>>> {
>>>>>> int x;
>>>>>> for (x = 0; x < 100000; ++x) {
>>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>>>> }
>>>>>> }

>>>>>> int main(int argc, char *argv[])
>>>>>> {
>>>>>> int x;

>>>>>> getchar();

>>>>>> pthread_t inc_x_thread;

>>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>>>> fprintf(stderr, "Error creating thread\n");
>>>>>> return 1;
>>>>>> }

>>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>>>> return 0;
>>>>>> }

>>>>>> hello-tp.c

>>>>>> #define TRACEPOINT_CREATE_PROBES
>>>>>> #define TRACEPOINT_DEFINE

>>>>>> #include "hello-tp.h"

>>>>>> hello-tp.h

>>>>>> #undef TRACEPOINT_PROVIDER
>>>>>> #define TRACEPOINT_PROVIDER hello_world

>>>>>> #undef TRACEPOINT_INCLUDE
>>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>>>> #define _HELLO_TP_H

>>>>>> #include <lttng/tracepoint.h>

>>>>>> TRACEPOINT_EVENT(
>>>>>> hello_world,
>>>>>> my_first_tracepoint,
>>>>>> TP_ARGS(
>>>>>> int, my_integer_arg,
>>>>>> char*, my_string_arg
>>>>>> ),
>>>>>> TP_FIELDS(
>>>>>> ctf_string(my_string_field, my_string_arg)
>>>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>>>> )
>>>>>> )

>>>>>> #endif /* _HELLO_TP_H */

>>>>>> #include <lttng/tracepoint-event.h>

>>>>>> Compile

>>>>>> gcc -c -I. hello-tp.c
>>>>>> gcc -c hello.c
>>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list lttng-dev@lists.lttng.org
>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>>> --
>>>>> Jonathan R. Julien
>>>>> Efficios

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> lttng-dev@lists.lttng.org
>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com

>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev@lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 12880 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 7006 bytes --]

It seems that some changes went in recently that caused this workaround to not work again. Originally, the __tracepoints__disable_destructors variable should be checked around the beginning of the destructor functions (__tracepoints__destroy and __tracepoints__ptrs_destroy), and return the function when the variable is true. It is not in the beginning any more. So, the segfault problem could happen. Is it possible to fix it again at your convenience? Thanks a lot.


________________________________
From: lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 15, 2016 12:00 PM
To: Jeffrey Chen
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread

Commit merged into master, thanks for your feedback!

Mathieu

----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Hi Mathieu:


Will this fix be checked-in? Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 3:27 PM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread



----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks, Mathieu.


Yes, the fix works as a workaround. This is also the same workaround we have now.

It seems the problem is due to a race condition, that it checked whether "tracepoint_dlopen.rcu_read_lock_sym_bp" is not NULL. If not NULL, it calls to that. But, at this moment, memset cleared the memory.

This fix is okay with us, but I am not certain if it would break other component, since I do not understand LTTng enough to tell. Thanks.

In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint
shared object will clear the memory containing the called functions. Therefore,
we need to skip both steps of the destructor if we want threads to survive after
this destructor execution.

Thanks,

Mathieu





________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 10:52 AM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread
Hi Jeffrey,

I CC'd your @live.com address on the RFC patch. Please let me know if the approach
is OK with you (and try it out).

See:
https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

Thanks,

Mathieu


----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks Mathieu.

Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 8, 2016 6:09 PM
To: Jonathan Rajotte Julien
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread
The attached patch works around the issue. The problem is
the tracepoint destructors that run in parallel with the thread
that still exists.

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
object on dlclose of the instrumented code. Not sure if we should
care though...

Thoughts ?

Thanks,

Mathieu


----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
Reproduced it on master (ust and tools).

Repro with:

lttng create
lttng enable-event -u -a
lttng start

then run hello a few times, it eventually segfaults in the spawned thread.

Looking into it.

Thanks,

Mathieu

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:
Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread







_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


--
Jonathan R. Julien
Efficios

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 14549 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 8795 bytes --]

I don't see anything relevant that changed since commit: 

commit 664ccf245fcb343e1ea10e145bab3749423e9d0f 
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> 
Date: Thu Mar 10 09:32:59 2016 -0500 

Add tracepoint_disable_destructors() 

Calling this function from an instrumented program allows disabling 
tracepoint destructors. This allows threads to continue calling 
tracepoint code even after the tracepoint destructors have run. This is 
needed for applications that exit without joining all their threads. 

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> 
CC: Jeffrey Chen <cpthk@live.com> 

Can you point it out ? And let me know what is segfaulting, with a backtrace. 

I see that tracepoint_dlopen_ptr->tracepoint_unregister_lib(__start___tracepoints_ptrs) 
is still invoked from __tracepoints__ptrs_destroy() even if the disable destructors is set, 
but this only unregisters the library from the list of available events. I don't see how it would 
trigger a crash of threads still actively performing tracing. 

Thanks, 

Mathieu 

----- On Sep 21, 2016, at 4:00 PM, Jeffrey Chen <cpthk@live.com> wrote: 

> It seems that some changes went in recently that caused this workaround to not
> work again. Originally, the __tracepoints__disable_destructors variable should
> be checked around the beginning of the destructor functions (
> __tracepoints__destroy and __tracepoints__ptrs_destroy ), and return the
> function when the variable is true. It is not in the beginning any more. So,
> the segfault problem could happen. Is it possible to fix it again at your
> convenience? Thanks a lot.

> From: lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Mathieu
> Desnoyers <mathieu.desnoyers@efficios.com>
> Sent: Tuesday, March 15, 2016 12:00 PM
> To: Jeffrey Chen
> Cc: lttng-dev
> Subject: Re: [lttng-dev] segfault when exiting process with background thread
> Commit merged into master, thanks for your feedback!

> Mathieu

> ----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>> Hi Mathieu:

>> Will this fix be checked-in? Thanks.

>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Sent: Thursday, March 10, 2016 3:27 PM
>> To: Jeffrey Chen
>> Cc: Jonathan Rajotte Julien; lttng-dev
>> Subject: Re: segfault when exiting process with background thread

>> ----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>>> Thanks, Mathieu.

>>> Yes, the fix works as a workaround. This is also the same workaround we have
>>> now.

>>> It seems the problem is due to a race condition, that it checked whether "
>>> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls to
>>> that. But, at this moment, memset cleared the memory.

>>> This fix is okay with us, but I am not certain if it would break other
>>> component, since I do not understand LTTng enough to tell. Thanks.

>> In addition to memset clearing that memory, dlclose() of the
>> lttng-ust-tracepoint
>> shared object will clear the memory containing the called functions. Therefore,
>> we need to skip both steps of the destructor if we want threads to survive after
>> this destructor execution.

>> Thanks,

>> Mathieu

>>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>> Sent: Thursday, March 10, 2016 10:52 AM
>>> To: Jeffrey Chen
>>> Cc: Jonathan Rajotte Julien; lttng-dev
>>> Subject: Re: segfault when exiting process with background thread
>>> Hi Jeffrey,

>>> I CC'd your @live.com address on the RFC patch. Please let me know if the
>>> approach
>>> is OK with you (and try it out).

>>> See:
>>> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

>>> Thanks,

>>> Mathieu

>>> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

>>>> Thanks Mathieu.

>>>> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
>>>> fix. If not, we will have to workaround the problem for now (probably by
>>>> applying your fix ourselves). Thanks.

>>>> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>>> Sent: Tuesday, March 8, 2016 6:09 PM
>>>> To: Jonathan Rajotte Julien
>>>> Cc: lttng-dev
>>>> Subject: Re: [lttng-dev] segfault when exiting process with background thread
>>>> The attached patch works around the issue. The problem is
>>>> the tracepoint destructors that run in parallel with the thread
>>>> that still exists.

>>>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
>>>> object on dlclose of the instrumented code. Not sure if we should
>>>> care though...

>>>> Thoughts ?

>>>> Thanks,

>>>> Mathieu

>>>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
>>>> <mathieu.desnoyers@efficios.com> wrote:

>>>>> Reproduced it on master (ust and tools).

>>>>> Repro with:

>>>>> lttng create
>>>>> lttng enable-event -u -a
>>>>> lttng start

>>>>> then run hello a few times, it eventually segfaults in the spawned thread.

>>>>> Looking into it.

>>>>> Thanks,

>>>>> Mathieu

>>>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>>>>> <Jonathan.rajotte-julien@efficios.com> wrote:

>>>>>> Hi Jeffrey,

>>>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>>>>> installed)?

>>>>>> Were you able to reproduce the issue on master (git) ?

>>>>>> What are the exact order of command to reproduce the problem ?

>>>>>> Thanks

>>>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>>>>> Hi LTTng community:

>>>>>>> We are using LTTng for our production environment. We have been noticing
>>>>>>> segfault problem when our process exit. We think the problem is due to the
>>>>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>>>>> written a very simple app that could repro the problem. Most of my codes are
>>>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>>>>> could do?

>>>>>>> hello.c

>>>>>>> #include <stdio.h>
>>>>>>> #include "hello-tp.h"
>>>>>>> #include <pthread.h>

>>>>>>> void* doSomeThing(void *arg)
>>>>>>> {
>>>>>>> int x;
>>>>>>> for (x = 0; x < 100000; ++x) {
>>>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>>>>> }
>>>>>>> }

>>>>>>> int main(int argc, char *argv[])
>>>>>>> {
>>>>>>> int x;

>>>>>>> getchar();

>>>>>>> pthread_t inc_x_thread;

>>>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>>>>> fprintf(stderr, "Error creating thread\n");
>>>>>>> return 1;
>>>>>>> }

>>>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>>>>> return 0;
>>>>>>> }

>>>>>>> hello-tp.c

>>>>>>> #define TRACEPOINT_CREATE_PROBES
>>>>>>> #define TRACEPOINT_DEFINE

>>>>>>> #include "hello-tp.h"

>>>>>>> hello-tp.h

>>>>>>> #undef TRACEPOINT_PROVIDER
>>>>>>> #define TRACEPOINT_PROVIDER hello_world

>>>>>>> #undef TRACEPOINT_INCLUDE
>>>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>>>>> #define _HELLO_TP_H

>>>>>>> #include <lttng/tracepoint.h>

>>>>>>> TRACEPOINT_EVENT(
>>>>>>> hello_world,
>>>>>>> my_first_tracepoint,
>>>>>>> TP_ARGS(
>>>>>>> int, my_integer_arg,
>>>>>>> char*, my_string_arg
>>>>>>> ),
>>>>>>> TP_FIELDS(
>>>>>>> ctf_string(my_string_field, my_string_arg)
>>>>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>>>>> )
>>>>>>> )

>>>>>>> #endif /* _HELLO_TP_H */

>>>>>>> #include <lttng/tracepoint-event.h>

>>>>>>> Compile

>>>>>>> gcc -c -I. hello-tp.c
>>>>>>> gcc -c hello.c
>>>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>>>>> _______________________________________________
>>>>>>> lttng-dev mailing list [ mailto:lttng-dev@lists.lttng.org |
>>>>>>> lttng-dev@lists.lttng.org ] [
>>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>>>> --
>>>>>> Jonathan R. Julien
>>>>>> Efficios

>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> lttng-dev@lists.lttng.org
>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> lttng-dev@lists.lttng.org
>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 16767 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 8754 bytes --]

Yes, tracepoint_unregister_lib is the function I was referring. If the function is okay to call with background thread still writing trace, I am definitely fine. I just want to double check it, if there could be a potential bug. Thanks.


________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Wednesday, September 21, 2016 2:05 PM
To: Jeffrey Chen
Cc: Jeffrey Chen; lttng-dev
Subject: Re: segfault when exiting process with background thread

I don't see anything relevant that changed since commit:

commit 664ccf245fcb343e1ea10e145bab3749423e9d0f
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Thu Mar 10 09:32:59 2016 -0500

    Add tracepoint_disable_destructors()

    Calling this function from an instrumented program allows disabling
    tracepoint destructors. This allows threads to continue calling
    tracepoint code even after the tracepoint destructors have run. This is
    needed for applications that exit without joining all their threads.

    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
    CC: Jeffrey Chen <cpthk@live.com>

Can you point it out ? And let me know what is segfaulting, with a backtrace.

I see that tracepoint_dlopen_ptr->tracepoint_unregister_lib(__start___tracepoints_ptrs)
is still invoked from __tracepoints__ptrs_destroy() even if the disable destructors is set,
but this only unregisters the library from the list of available events. I don't see how it would
trigger a crash of threads still actively performing tracing.

Thanks,

Mathieu

----- On Sep 21, 2016, at 4:00 PM, Jeffrey Chen <cpthk@live.com> wrote:

It seems that some changes went in recently that caused this workaround to not work again. Originally, the __tracepoints__disable_destructors variable should be checked around the beginning of the destructor functions (__tracepoints__destroy and __tracepoints__ptrs_destroy), and return the function when the variable is true. It is not in the beginning any more. So, the segfault problem could happen. Is it possible to fix it again at your convenience? Thanks a lot.


________________________________
From: lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 15, 2016 12:00 PM
To: Jeffrey Chen
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread

Commit merged into master, thanks for your feedback!

Mathieu

----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Hi Mathieu:


Will this fix be checked-in? Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 3:27 PM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread



----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks, Mathieu.


Yes, the fix works as a workaround. This is also the same workaround we have now.

It seems the problem is due to a race condition, that it checked whether "tracepoint_dlopen.rcu_read_lock_sym_bp" is not NULL. If not NULL, it calls to that. But, at this moment, memset cleared the memory.

This fix is okay with us, but I am not certain if it would break other component, since I do not understand LTTng enough to tell. Thanks.

In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint
shared object will clear the memory containing the called functions. Therefore,
we need to skip both steps of the destructor if we want threads to survive after
this destructor execution.

Thanks,

Mathieu





________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Thursday, March 10, 2016 10:52 AM
To: Jeffrey Chen
Cc: Jonathan Rajotte Julien; lttng-dev
Subject: Re: segfault when exiting process with background thread
Hi Jeffrey,

I CC'd your @live.com address on the RFC patch. Please let me know if the approach
is OK with you (and try it out).

See:
https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

Thanks,

Mathieu


----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:

Thanks Mathieu.

Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.



________________________________
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Tuesday, March 8, 2016 6:09 PM
To: Jonathan Rajotte Julien
Cc: lttng-dev
Subject: Re: [lttng-dev] segfault when exiting process with background thread
The attached patch works around the issue. The problem is
the tracepoint destructors that run in parallel with the thread
that still exists.

If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
object on dlclose of the instrumented code. Not sure if we should
care though...

Thoughts ?

Thanks,

Mathieu


----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
Reproduced it on master (ust and tools).

Repro with:

lttng create
lttng enable-event -u -a
lttng start

then run hello a few times, it eventually segfaults in the spawned thread.

Looking into it.

Thanks,

Mathieu

----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:
Hi Jeffrey,

Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?

Were you able to reproduce the issue on master (git) ?

What are the exact order of command to reproduce the problem ?

Thanks

On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread







_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org<mailto:lttng-dev@lists.lttng.org>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


--
Jonathan R. Julien
Efficios

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 17637 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

segfault when exiting process with background thread

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 1934 bytes --]

Hi LTTng community:


We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?



hello.c

#include <stdio.h>
#include "hello-tp.h"
#include <pthread.h>


void* doSomeThing(void *arg)
{
    int x;
    for (x = 0; x < 100000; ++x) {
        tracepoint(hello_world, my_first_tracepoint, x, "test");
    }
}

int main(int argc, char *argv[])
{
    int x;

    getchar();

    pthread_t inc_x_thread;

    if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;
    }

    tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

    return 0;
}


hello-tp.c

#define TRACEPOINT_CREATE_PROBES
#define TRACEPOINT_DEFINE

#include "hello-tp.h"


hello-tp.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER hello_world

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    hello_world,
    my_first_tracepoint,
    TP_ARGS(
        int, my_integer_arg,
        char*, my_string_arg
    ),
    TP_FIELDS(
        ctf_string(my_string_field, my_string_arg)
        ctf_integer(int, my_integer_field, my_integer_arg)
    )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>


Compile

gcc -c -I. hello-tp.c
gcc -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread





[-- Attachment #1.2: Type: text/html, Size: 3425 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev