Re: Tracepoints firing without being enabled in LTTng

Re: Tracepoints firing without being enabled in LTTng

[Francis Deslauriers]

Hi Mosleh,
It's expected to not see any event when running lttng list -u if no
lttng-ust instrumented applications are running.

What are the exact lttng commands you are running to see those
unexpected events?

Thank you,
Francis


Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>
> Hello,
>
> I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>
> The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>
> Thanks for any assistance,
> Mosleh
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Francis Deslauriers]

Adding back lttng-dev mailing list.

So you don't even run a lttng-enable-event command?
That's odd.

Can you send us the output of the following commands:
Is it possible that there are other lttng-sessiond daemons running on
the machine?
> ps aux | grep "lttng-sessiond"
Verify if there are other active tracing sessions:
> lttng list
Check the details of the current session:
> lttng status

Thank you,
Francis

Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>
> Hi Francis,
>
> Thank you for the quick response.
> Just to clear things up, my application is running when I run that command.
>
> The commands I am running are:
> > lttng-sessiond -d
> > lttng create my-sess --live
> After this command I start my application. If I then do the command:
> > lttng view
> I will see the tracepoints in the application in the terminal even though I have not done: lttng start.
>
> I thought it was strange that I cannot find my application in the: lttng list command under any domain.
> Also the reason I start my application after creating a session is because the system automatically starts the application using systemd,
> in order to get tracing data at all I have to manually stop the application and start it again after creating a session.
>
> Thanks again for any help,
> Mosleh
>
> On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>
>> Hi Mosleh,
>> It's expected to not see any event when running lttng list -u if no
>> lttng-ust instrumented applications are running.
>>
>> What are the exact lttng commands you are running to see those
>> unexpected events?
>>
>> Thank you,
>> Francis
>>
>>
>> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>> >
>> > Hello,
>> >
>> > I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>> >
>> > The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>> >
>> > Thanks for any assistance,
>> > Mosleh
>> > _______________________________________________
>> > lttng-dev mailing list
>> > lttng-dev@lists.lttng.org
>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.



-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1.1: Type: text/plain, Size: 3937 bytes --]

That's correct, I don't enable any events or do: lttng start.
Here is some snapshots of starting everything up, followed by starting my
application.
[image: image.png]

Here are the outputs for the commands:

[image: image.png]

[image: image.png]

[image: image.png]

Note that even though the session is inactive, when a trace event occurs in
my application it is being recorded in the appropriate directory:

[image: image.png]

Also maybe important, the system has root only user. Would having the
application belonging to a different user and trying to run as root cause
these kinds of complications? I did notice when testing on a raspberry pi
that running an application as a root user would mean running sessions as
root user as well.

Thanks,
Mosleh

On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
francis.deslauriers@efficios.com> wrote:

> Adding back lttng-dev mailing list.
>
> So you don't even run a lttng-enable-event command?
> That's odd.
>
> Can you send us the output of the following commands:
> Is it possible that there are other lttng-sessiond daemons running on
> the machine?
> > ps aux | grep "lttng-sessiond"
> Verify if there are other active tracing sessions:
> > lttng list
> Check the details of the current session:
> > lttng status
>
> Thank you,
> Francis
>
> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a
> écrit :
> >
> > Hi Francis,
> >
> > Thank you for the quick response.
> > Just to clear things up, my application is running when I run that
> command.
> >
> > The commands I am running are:
> > > lttng-sessiond -d
> > > lttng create my-sess --live
> > After this command I start my application. If I then do the command:
> > > lttng view
> > I will see the tracepoints in the application in the terminal even
> though I have not done: lttng start.
> >
> > I thought it was strange that I cannot find my application in the: lttng
> list command under any domain.
> > Also the reason I start my application after creating a session is
> because the system automatically starts the application using systemd,
> > in order to get tracing data at all I have to manually stop the
> application and start it again after creating a session.
> >
> > Thanks again for any help,
> > Mosleh
> >
> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
> >>
> >> Hi Mosleh,
> >> It's expected to not see any event when running lttng list -u if no
> >> lttng-ust instrumented applications are running.
> >>
> >> What are the exact lttng commands you are running to see those
> >> unexpected events?
> >>
> >> Thank you,
> >> Francis
> >>
> >>
> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com>
> a écrit :
> >> >
> >> > Hello,
> >> >
> >> > I am having a strange issue with my current application. For some
> context, I have integrated LTTng into a Yocto build and put custom trace
> points in my embedded system application. Once I load the built image onto
> my system and start a session daemon, I try to list all available userspace
> events (using lttng list -u), but no events show up.
> >> >
> >> > The strange occurrence is when I create a session (lltng create
> my-sess) and not start tracing the trace events will display on the screen
> regardless (when starting session in live mode with lttng view). I was
> wondering if anyone has any insight on this issue I am having.
> >> >
> >> > Thanks for any assistance,
> >> > Mosleh
> >> > _______________________________________________
> >> > lttng-dev mailing list
> >> > lttng-dev@lists.lttng.org
> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >>
> >> --
> >> Francis Deslauriers
> >> Computer Engineer
> >> EfficiOS inc.
>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>

[-- Attachment #1.1.2: Type: text/html, Size: 5602 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 9125 bytes --]

[-- Attachment #1.3: image.png --]
[-- Type: image/png, Size: 13006 bytes --]

[-- Attachment #1.4: image.png --]
[-- Type: image/png, Size: 12073 bytes --]

[-- Attachment #1.5: image.png --]
[-- Type: image/png, Size: 6026 bytes --]

[-- Attachment #1.6: image.png --]
[-- Type: image/png, Size: 3036 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Francis Deslauriers]

[-- Attachment #1.1.1: Type: text/plain, Size: 4706 bytes --]

Can you start the lttng-sessiond in verbose mode using the following
command:
> lttng-sessiond -vvv
and then do the lttng create, start of application and lttng view on
another console.

Then copy the output of the sessiond on https://pastebin.com/ and send us
the link.

Thank you,
Francis

Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com> a
écrit :

> That's correct, I don't enable any events or do: lttng start.
> Here is some snapshots of starting everything up, followed by starting my
> application.
> [image: image.png]
>
> Here are the outputs for the commands:
>
> [image: image.png]
>
> [image: image.png]
>
> [image: image.png]
>
> Note that even though the session is inactive, when a trace event occurs
> in my application it is being recorded in the appropriate directory:
>
> [image: image.png]
>
> Also maybe important, the system has root only user. Would having the
> application belonging to a different user and trying to run as root cause
> these kinds of complications? I did notice when testing on a raspberry pi
> that running an application as a root user would mean running sessions as
> root user as well.
>
> Thanks,
> Mosleh
>
> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
>
>> Adding back lttng-dev mailing list.
>>
>> So you don't even run a lttng-enable-event command?
>> That's odd.
>>
>> Can you send us the output of the following commands:
>> Is it possible that there are other lttng-sessiond daemons running on
>> the machine?
>> > ps aux | grep "lttng-sessiond"
>> Verify if there are other active tracing sessions:
>> > lttng list
>> Check the details of the current session:
>> > lttng status
>>
>> Thank you,
>> Francis
>>
>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a
>> écrit :
>> >
>> > Hi Francis,
>> >
>> > Thank you for the quick response.
>> > Just to clear things up, my application is running when I run that
>> command.
>> >
>> > The commands I am running are:
>> > > lttng-sessiond -d
>> > > lttng create my-sess --live
>> > After this command I start my application. If I then do the command:
>> > > lttng view
>> > I will see the tracepoints in the application in the terminal even
>> though I have not done: lttng start.
>> >
>> > I thought it was strange that I cannot find my application in the:
>> lttng list command under any domain.
>> > Also the reason I start my application after creating a session is
>> because the system automatically starts the application using systemd,
>> > in order to get tracing data at all I have to manually stop the
>> application and start it again after creating a session.
>> >
>> > Thanks again for any help,
>> > Mosleh
>> >
>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
>> francis.deslauriers@efficios.com> wrote:
>> >>
>> >> Hi Mosleh,
>> >> It's expected to not see any event when running lttng list -u if no
>> >> lttng-ust instrumented applications are running.
>> >>
>> >> What are the exact lttng commands you are running to see those
>> >> unexpected events?
>> >>
>> >> Thank you,
>> >> Francis
>> >>
>> >>
>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com>
>> a écrit :
>> >> >
>> >> > Hello,
>> >> >
>> >> > I am having a strange issue with my current application. For some
>> context, I have integrated LTTng into a Yocto build and put custom trace
>> points in my embedded system application. Once I load the built image onto
>> my system and start a session daemon, I try to list all available userspace
>> events (using lttng list -u), but no events show up.
>> >> >
>> >> > The strange occurrence is when I create a session (lltng create
>> my-sess) and not start tracing the trace events will display on the screen
>> regardless (when starting session in live mode with lttng view). I was
>> wondering if anyone has any insight on this issue I am having.
>> >> >
>> >> > Thanks for any assistance,
>> >> > Mosleh
>> >> > _______________________________________________
>> >> > lttng-dev mailing list
>> >> > lttng-dev@lists.lttng.org
>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>
>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.
>>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>


-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.

[-- Attachment #1.1.2: Type: text/html, Size: 7002 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 9125 bytes --]

[-- Attachment #1.3: image.png --]
[-- Type: image/png, Size: 13006 bytes --]

[-- Attachment #1.4: image.png --]
[-- Type: image/png, Size: 12073 bytes --]

[-- Attachment #1.5: image.png --]
[-- Type: image/png, Size: 6026 bytes --]

[-- Attachment #1.6: image.png --]
[-- Type: image/png, Size: 3036 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1.1: Type: text/plain, Size: 5477 bytes --]

Hello,

Sorry for the late reply. I was able to figure out the weird behavior from
the application. Turns out if I run my application as a service,  lttng no
longer can see any of the UST tracepoints and causes the behavior I was
explaining.
When I execute the application normally (./app_name), the functionality is
as expected. Is this the expected behavior? Is there some way I can trace
the userspace application while running it as a service simultaneously?

Thanks for any info,
Mosleh

On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <
francis.deslauriers@efficios.com> wrote:

> Can you start the lttng-sessiond in verbose mode using the following
> command:
> > lttng-sessiond -vvv
> and then do the lttng create, start of application and lttng view on
> another console.
>
> Then copy the output of the sessiond on https://pastebin.com/ and send us
> the link.
>
> Thank you,
> Francis
>
> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com> a
> écrit :
>
>> That's correct, I don't enable any events or do: lttng start.
>> Here is some snapshots of starting everything up, followed by starting my
>> application.
>> [image: image.png]
>>
>> Here are the outputs for the commands:
>>
>> [image: image.png]
>>
>> [image: image.png]
>>
>> [image: image.png]
>>
>> Note that even though the session is inactive, when a trace event occurs
>> in my application it is being recorded in the appropriate directory:
>>
>> [image: image.png]
>>
>> Also maybe important, the system has root only user. Would having the
>> application belonging to a different user and trying to run as root cause
>> these kinds of complications? I did notice when testing on a raspberry pi
>> that running an application as a root user would mean running sessions as
>> root user as well.
>>
>> Thanks,
>> Mosleh
>>
>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
>> francis.deslauriers@efficios.com> wrote:
>>
>>> Adding back lttng-dev mailing list.
>>>
>>> So you don't even run a lttng-enable-event command?
>>> That's odd.
>>>
>>> Can you send us the output of the following commands:
>>> Is it possible that there are other lttng-sessiond daemons running on
>>> the machine?
>>> > ps aux | grep "lttng-sessiond"
>>> Verify if there are other active tracing sessions:
>>> > lttng list
>>> Check the details of the current session:
>>> > lttng status
>>>
>>> Thank you,
>>> Francis
>>>
>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a
>>> écrit :
>>> >
>>> > Hi Francis,
>>> >
>>> > Thank you for the quick response.
>>> > Just to clear things up, my application is running when I run that
>>> command.
>>> >
>>> > The commands I am running are:
>>> > > lttng-sessiond -d
>>> > > lttng create my-sess --live
>>> > After this command I start my application. If I then do the command:
>>> > > lttng view
>>> > I will see the tracepoints in the application in the terminal even
>>> though I have not done: lttng start.
>>> >
>>> > I thought it was strange that I cannot find my application in the:
>>> lttng list command under any domain.
>>> > Also the reason I start my application after creating a session is
>>> because the system automatically starts the application using systemd,
>>> > in order to get tracing data at all I have to manually stop the
>>> application and start it again after creating a session.
>>> >
>>> > Thanks again for any help,
>>> > Mosleh
>>> >
>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
>>> francis.deslauriers@efficios.com> wrote:
>>> >>
>>> >> Hi Mosleh,
>>> >> It's expected to not see any event when running lttng list -u if no
>>> >> lttng-ust instrumented applications are running.
>>> >>
>>> >> What are the exact lttng commands you are running to see those
>>> >> unexpected events?
>>> >>
>>> >> Thank you,
>>> >> Francis
>>> >>
>>> >>
>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com>
>>> a écrit :
>>> >> >
>>> >> > Hello,
>>> >> >
>>> >> > I am having a strange issue with my current application. For some
>>> context, I have integrated LTTng into a Yocto build and put custom trace
>>> points in my embedded system application. Once I load the built image onto
>>> my system and start a session daemon, I try to list all available userspace
>>> events (using lttng list -u), but no events show up.
>>> >> >
>>> >> > The strange occurrence is when I create a session (lltng create
>>> my-sess) and not start tracing the trace events will display on the screen
>>> regardless (when starting session in live mode with lttng view). I was
>>> wondering if anyone has any insight on this issue I am having.
>>> >> >
>>> >> > Thanks for any assistance,
>>> >> > Mosleh
>>> >> > _______________________________________________
>>> >> > lttng-dev mailing list
>>> >> > lttng-dev@lists.lttng.org
>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>> >>
>>> >> --
>>> >> Francis Deslauriers
>>> >> Computer Engineer
>>> >> EfficiOS inc.
>>>
>>>
>>>
>>> --
>>> Francis Deslauriers
>>> Computer Engineer
>>> EfficiOS inc.
>>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>

[-- Attachment #1.1.2: Type: text/html, Size: 8012 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 9125 bytes --]

[-- Attachment #1.3: image.png --]
[-- Type: image/png, Size: 13006 bytes --]

[-- Attachment #1.4: image.png --]
[-- Type: image/png, Size: 12073 bytes --]

[-- Attachment #1.5: image.png --]
[-- Type: image/png, Size: 6026 bytes --]

[-- Attachment #1.6: image.png --]
[-- Type: image/png, Size: 3036 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Francis Deslauriers]

Hi Mosleh,
Just to be clear:
When you run your app as a service (background), lttng list -u doesn't
display any events AND you see events in the live session without
having run enable-event and start commands.
When you run your app normally, lttng list -u displays events AND you
only see events after running enable-event and start commands.
Is that correct?

Please provide the output of the lttng-sessiond in verbose mode using
the following command:
> lttng-sessiond -vvv
please provide the output of your application as a service with this
environment variable:  LTTNG_UST_DEBUG=1
and please provide us with the service file describing how your
service is started.
Put all of this on a pastebin.com

Is your application mass-closing file descriptors when it's
daemonizing? how about using fork() syscall without an exec() after
it? If so, have a look at this part of the UST man page:
https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons

Thank you,
Francis


Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>
> Hello,
>
> Sorry for the late reply. I was able to figure out the weird behavior from the application. Turns out if I run my application as a service,  lttng no longer can see any of the UST tracepoints and causes the behavior I was explaining.
> When I execute the application normally (./app_name), the functionality is as expected. Is this the expected behavior? Is there some way I can trace the userspace application while running it as a service simultaneously?
>
> Thanks for any info,
> Mosleh
>
> On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>
>> Can you start the lttng-sessiond in verbose mode using the following command:
>> > lttng-sessiond -vvv
>> and then do the lttng create, start of application and lttng view on another console.
>>
>> Then copy the output of the sessiond on https://pastebin.com/ and send us the link.
>>
>> Thank you,
>> Francis
>>
>> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>>>
>>> That's correct, I don't enable any events or do: lttng start.
>>> Here is some snapshots of starting everything up, followed by starting my application.
>>>
>>>
>>> Here are the outputs for the commands:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Note that even though the session is inactive, when a trace event occurs in my application it is being recorded in the appropriate directory:
>>>
>>>
>>>
>>> Also maybe important, the system has root only user. Would having the application belonging to a different user and trying to run as root cause these kinds of complications? I did notice when testing on a raspberry pi that running an application as a root user would mean running sessions as root user as well.
>>>
>>> Thanks,
>>> Mosleh
>>>
>>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>>>
>>>> Adding back lttng-dev mailing list.
>>>>
>>>> So you don't even run a lttng-enable-event command?
>>>> That's odd.
>>>>
>>>> Can you send us the output of the following commands:
>>>> Is it possible that there are other lttng-sessiond daemons running on
>>>> the machine?
>>>> > ps aux | grep "lttng-sessiond"
>>>> Verify if there are other active tracing sessions:
>>>> > lttng list
>>>> Check the details of the current session:
>>>> > lttng status
>>>>
>>>> Thank you,
>>>> Francis
>>>>
>>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>>>> >
>>>> > Hi Francis,
>>>> >
>>>> > Thank you for the quick response.
>>>> > Just to clear things up, my application is running when I run that command.
>>>> >
>>>> > The commands I am running are:
>>>> > > lttng-sessiond -d
>>>> > > lttng create my-sess --live
>>>> > After this command I start my application. If I then do the command:
>>>> > > lttng view
>>>> > I will see the tracepoints in the application in the terminal even though I have not done: lttng start.
>>>> >
>>>> > I thought it was strange that I cannot find my application in the: lttng list command under any domain.
>>>> > Also the reason I start my application after creating a session is because the system automatically starts the application using systemd,
>>>> > in order to get tracing data at all I have to manually stop the application and start it again after creating a session.
>>>> >
>>>> > Thanks again for any help,
>>>> > Mosleh
>>>> >
>>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>>> >>
>>>> >> Hi Mosleh,
>>>> >> It's expected to not see any event when running lttng list -u if no
>>>> >> lttng-ust instrumented applications are running.
>>>> >>
>>>> >> What are the exact lttng commands you are running to see those
>>>> >> unexpected events?
>>>> >>
>>>> >> Thank you,
>>>> >> Francis
>>>> >>
>>>> >>
>>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>>>> >> >
>>>> >> > Hello,
>>>> >> >
>>>> >> > I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>>>> >> >
>>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>>>> >> >
>>>> >> > Thanks for any assistance,
>>>> >> > Mosleh
>>>> >> > _______________________________________________
>>>> >> > lttng-dev mailing list
>>>> >> > lttng-dev@lists.lttng.org
>>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>> >>
>>>> >> --
>>>> >> Francis Deslauriers
>>>> >> Computer Engineer
>>>> >> EfficiOS inc.
>>>>
>>>>
>>>>
>>>> --
>>>> Francis Deslauriers
>>>> Computer Engineer
>>>> EfficiOS inc.
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.



-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1: Type: text/plain, Size: 8092 bytes --]

Hello,

Here is what I do to get the consistent behavior:
When the app is running as a service, lttng list -u does not display any
events, initially I do have to lttng enable-events and lttng start to see
the tracepoints in lttng view. However, after lttng start, I can disable
all events and I can still see the events coming through in lttng view.
When I run the app normally, lttng list -u displays the tracepoints I place
in the app, and all functionality of lttng works as expected.

I apologize, I am in the process of trying to get you the outputs you are
asking for, however due to confidentiality with the app the outputs of
running the commands need to be reviewed by others. I will try to get this
to you as soon as I can.

I can say the service is being kick started by systemd. Also I believe the
app is mass-closing file descriptors when it is daemonizing. I tried to
dynamically load the liblttng-ust-fd.so but the behavior did not change.

Thanks,
Mosleh

On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers <
francis.deslauriers@efficios.com> wrote:

> Hi Mosleh,
> Just to be clear:
> When you run your app as a service (background), lttng list -u doesn't
> display any events AND you see events in the live session without
> having run enable-event and start commands.
> When you run your app normally, lttng list -u displays events AND you
> only see events after running enable-event and start commands.
> Is that correct?
>
> Please provide the output of the lttng-sessiond in verbose mode using
> the following command:
> > lttng-sessiond -vvv
> please provide the output of your application as a service with this
> environment variable:  LTTNG_UST_DEBUG=1
> and please provide us with the service file describing how your
> service is started.
> Put all of this on a pastebin.com
>
> Is your application mass-closing file descriptors when it's
> daemonizing? how about using fork() syscall without an exec() after
> it? If so, have a look at this part of the UST man page:
> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons
>
> Thank you,
> Francis
>
>
> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235@gmail.com> a
> écrit :
> >
> > Hello,
> >
> > Sorry for the late reply. I was able to figure out the weird behavior
> from the application. Turns out if I run my application as a service,
> lttng no longer can see any of the UST tracepoints and causes the behavior
> I was explaining.
> > When I execute the application normally (./app_name), the functionality
> is as expected. Is this the expected behavior? Is there some way I can
> trace the userspace application while running it as a service
> simultaneously?
> >
> > Thanks for any info,
> > Mosleh
> >
> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
> >>
> >> Can you start the lttng-sessiond in verbose mode using the following
> command:
> >> > lttng-sessiond -vvv
> >> and then do the lttng create, start of application and lttng view on
> another console.
> >>
> >> Then copy the output of the sessiond on https://pastebin.com/ and send
> us the link.
> >>
> >> Thank you,
> >> Francis
> >>
> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com>
> a écrit :
> >>>
> >>> That's correct, I don't enable any events or do: lttng start.
> >>> Here is some snapshots of starting everything up, followed by starting
> my application.
> >>>
> >>>
> >>> Here are the outputs for the commands:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Note that even though the session is inactive, when a trace event
> occurs in my application it is being recorded in the appropriate directory:
> >>>
> >>>
> >>>
> >>> Also maybe important, the system has root only user. Would having the
> application belonging to a different user and trying to run as root cause
> these kinds of complications? I did notice when testing on a raspberry pi
> that running an application as a root user would mean running sessions as
> root user as well.
> >>>
> >>> Thanks,
> >>> Mosleh
> >>>
> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
> >>>>
> >>>> Adding back lttng-dev mailing list.
> >>>>
> >>>> So you don't even run a lttng-enable-event command?
> >>>> That's odd.
> >>>>
> >>>> Can you send us the output of the following commands:
> >>>> Is it possible that there are other lttng-sessiond daemons running on
> >>>> the machine?
> >>>> > ps aux | grep "lttng-sessiond"
> >>>> Verify if there are other active tracing sessions:
> >>>> > lttng list
> >>>> Check the details of the current session:
> >>>> > lttng status
> >>>>
> >>>> Thank you,
> >>>> Francis
> >>>>
> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com>
> a écrit :
> >>>> >
> >>>> > Hi Francis,
> >>>> >
> >>>> > Thank you for the quick response.
> >>>> > Just to clear things up, my application is running when I run that
> command.
> >>>> >
> >>>> > The commands I am running are:
> >>>> > > lttng-sessiond -d
> >>>> > > lttng create my-sess --live
> >>>> > After this command I start my application. If I then do the command:
> >>>> > > lttng view
> >>>> > I will see the tracepoints in the application in the terminal even
> though I have not done: lttng start.
> >>>> >
> >>>> > I thought it was strange that I cannot find my application in the:
> lttng list command under any domain.
> >>>> > Also the reason I start my application after creating a session is
> because the system automatically starts the application using systemd,
> >>>> > in order to get tracing data at all I have to manually stop the
> application and start it again after creating a session.
> >>>> >
> >>>> > Thanks again for any help,
> >>>> > Mosleh
> >>>> >
> >>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
> >>>> >>
> >>>> >> Hi Mosleh,
> >>>> >> It's expected to not see any event when running lttng list -u if no
> >>>> >> lttng-ust instrumented applications are running.
> >>>> >>
> >>>> >> What are the exact lttng commands you are running to see those
> >>>> >> unexpected events?
> >>>> >>
> >>>> >> Thank you,
> >>>> >> Francis
> >>>> >>
> >>>> >>
> >>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <
> mosleh1235@gmail.com> a écrit :
> >>>> >> >
> >>>> >> > Hello,
> >>>> >> >
> >>>> >> > I am having a strange issue with my current application. For
> some context, I have integrated LTTng into a Yocto build and put custom
> trace points in my embedded system application. Once I load the built image
> onto my system and start a session daemon, I try to list all available
> userspace events (using lttng list -u), but no events show up.
> >>>> >> >
> >>>> >> > The strange occurrence is when I create a session (lltng create
> my-sess) and not start tracing the trace events will display on the screen
> regardless (when starting session in live mode with lttng view). I was
> wondering if anyone has any insight on this issue I am having.
> >>>> >> >
> >>>> >> > Thanks for any assistance,
> >>>> >> > Mosleh
> >>>> >> > _______________________________________________
> >>>> >> > lttng-dev mailing list
> >>>> >> > lttng-dev@lists.lttng.org
> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >>>> >>
> >>>> >> --
> >>>> >> Francis Deslauriers
> >>>> >> Computer Engineer
> >>>> >> EfficiOS inc.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Francis Deslauriers
> >>>> Computer Engineer
> >>>> EfficiOS inc.
> >>>
> >>> _______________________________________________
> >>> lttng-dev mailing list
> >>> lttng-dev@lists.lttng.org
> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >>
> >>
> >>
> >> --
> >> Francis Deslauriers
> >> Computer Engineer
> >> EfficiOS inc.
>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>

[-- Attachment #1.2: Type: text/html, Size: 11473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Francis Deslauriers]

Le mer. 20 févr. 2019, à 09 h 10, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>
> Hello,
>
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any events, initially I do have to lttng enable-events and lttng start to see the tracepoints in lttng view. However, after lttng start, I can disable all events and I can still see the events coming through in lttng view. When I run the app normally, lttng list -u displays the tracepoints I place in the app, and all functionality of lttng works as expected.
>
> I apologize, I am in the process of trying to get you the outputs you are asking for, however due to confidentiality with the app the outputs of running the commands need to be reviewed by others. I will try to get this to you as soon as I can.
I understand.

Thanks,
Francis
>
> I can say the service is being kick started by systemd. Also I believe the app is mass-closing file descriptors when it is daemonizing. I tried to dynamically load the liblttng-ust-fd.so but the behavior did not change.
>
> Thanks,
> Mosleh
>
> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>
>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?
>>
>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable:  LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a pastebin.com
>>
>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons
>>
>> Thank you,
>> Francis
>>
>>
>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>> >
>> > Hello,
>> >
>> > Sorry for the late reply. I was able to figure out the weird behavior from the application. Turns out if I run my application as a service,  lttng no longer can see any of the UST tracepoints and causes the behavior I was explaining.
>> > When I execute the application normally (./app_name), the functionality is as expected. Is this the expected behavior? Is there some way I can trace the userspace application while running it as a service simultaneously?
>> >
>> > Thanks for any info,
>> > Mosleh
>> >
>> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>> >>
>> >> Can you start the lttng-sessiond in verbose mode using the following command:
>> >> > lttng-sessiond -vvv
>> >> and then do the lttng create, start of application and lttng view on another console.
>> >>
>> >> Then copy the output of the sessiond on https://pastebin.com/ and send us the link.
>> >>
>> >> Thank you,
>> >> Francis
>> >>
>> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>> >>>
>> >>> That's correct, I don't enable any events or do: lttng start.
>> >>> Here is some snapshots of starting everything up, followed by starting my application.
>> >>>
>> >>>
>> >>> Here are the outputs for the commands:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Note that even though the session is inactive, when a trace event occurs in my application it is being recorded in the appropriate directory:
>> >>>
>> >>>
>> >>>
>> >>> Also maybe important, the system has root only user. Would having the application belonging to a different user and trying to run as root cause these kinds of complications? I did notice when testing on a raspberry pi that running an application as a root user would mean running sessions as root user as well.
>> >>>
>> >>> Thanks,
>> >>> Mosleh
>> >>>
>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>> >>>>
>> >>>> Adding back lttng-dev mailing list.
>> >>>>
>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.
>> >>>>
>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status
>> >>>>
>> >>>> Thank you,
>> >>>> Francis
>> >>>>
>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>> >>>> >
>> >>>> > Hi Francis,
>> >>>> >
>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that command.
>> >>>> >
>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the command:
>> >>>> > > lttng view
>> >>>> > I will see the tracepoints in the application in the terminal even though I have not done: lttng start.
>> >>>> >
>> >>>> > I thought it was strange that I cannot find my application in the: lttng list command under any domain.
>> >>>> > Also the reason I start my application after creating a session is because the system automatically starts the application using systemd,
>> >>>> > in order to get tracing data at all I have to manually stop the application and start it again after creating a session.
>> >>>> >
>> >>>> > Thanks again for any help,
>> >>>> > Mosleh
>> >>>> >
>> >>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>> >>>> >>
>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if no
>> >>>> >> lttng-ust instrumented applications are running.
>> >>>> >>
>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?
>> >>>> >>
>> >>>> >> Thank you,
>> >>>> >> Francis
>> >>>> >>
>> >>>> >>
>> >>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <mosleh1235@gmail.com> a écrit :
>> >>>> >> >
>> >>>> >> > Hello,
>> >>>> >> >
>> >>>> >> > I am having a strange issue with my current application. For some context, I have integrated LTTng into a Yocto build and put custom trace points in my embedded system application. Once I load the built image onto my system and start a session daemon, I try to list all available userspace events (using lttng list -u), but no events show up.
>> >>>> >> >
>> >>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not start tracing the trace events will display on the screen regardless (when starting session in live mode with lttng view). I was wondering if anyone has any insight on this issue I am having.
>> >>>> >> >
>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > lttng-dev@lists.lttng.org
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>>> >>
>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.
>> >>>
>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> lttng-dev@lists.lttng.org
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>
>> >>
>> >>
>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.



-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 9308 bytes --]

----- On Feb 20, 2019, at 9:10 AM, Mosleh Uddin <mosleh1235@gmail.com> wrote: 

> Hello,
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any events,
> initially I do have to lttng enable-events and lttng start to see the
> tracepoints in lttng view. However, after lttng start, I can disable all events
> and I can still see the events coming through in lttng view. When I run the app
> normally, lttng list -u displays the tracepoints I place in the app, and all
> functionality of lttng works as expected.

> I apologize, I am in the process of trying to get you the outputs you are asking
> for, however due to confidentiality with the app the outputs of running the
> commands need to be reviewed by others. I will try to get this to you as soon
> as I can.

> I can say the service is being kick started by systemd. Also I believe the app
> is mass-closing file descriptors when it is daemonizing. I tried to dynamically
> load the liblttng-ust-fd.so but the behavior did not change.

Did you load both liblttng-ust-fork.so and liblttng-ust-fd.so with LD_PRELOAD when launching your application ? 

If not using LD_PRELOAD env. var., how did you load them ? 

Thanks, 

Mathieu 

> Thanks,
> Mosleh

> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers < [
> mailto:francis.deslauriers@efficios.com | francis.deslauriers@efficios.com ] >
> wrote:

>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?

>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable: LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a [ http://pastebin.com/ | pastebin.com ]

>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> [ https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons |
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons ]

>> Thank you,
>> Francis

>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin < [ mailto:mosleh1235@gmail.com |
>> mosleh1235@gmail.com ] > a écrit :

>> > Hello,

>>> Sorry for the late reply. I was able to figure out the weird behavior from the
>>> application. Turns out if I run my application as a service, lttng no longer
>> > can see any of the UST tracepoints and causes the behavior I was explaining.
>>> When I execute the application normally (./app_name), the functionality is as
>>> expected. Is this the expected behavior? Is there some way I can trace the
>> > userspace application while running it as a service simultaneously?

>> > Thanks for any info,
>> > Mosleh

>>> On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers < [
>>> mailto:francis.deslauriers@efficios.com | francis.deslauriers@efficios.com ] >
>> > wrote:

>> >> Can you start the lttng-sessiond in verbose mode using the following command:
>> >> > lttng-sessiond -vvv
>>>> and then do the lttng create, start of application and lttng view on another
>> >> console.

>>>> Then copy the output of the sessiond on [ https://pastebin.com/ |
>> >> https://pastebin.com/ ] and send us the link.

>> >> Thank you,
>> >> Francis

>>>> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin < [ mailto:mosleh1235@gmail.com |
>> >> mosleh1235@gmail.com ] > a écrit :

>> >>> That's correct, I don't enable any events or do: lttng start.
>>>>> Here is some snapshots of starting everything up, followed by starting my
>> >>> application.


>> >>> Here are the outputs for the commands:







>>>>> Note that even though the session is inactive, when a trace event occurs in my
>> >>> application it is being recorded in the appropriate directory:



>>>>> Also maybe important, the system has root only user. Would having the
>>>>> application belonging to a different user and trying to run as root cause these
>>>>> kinds of complications? I did notice when testing on a raspberry pi that
>>>>> running an application as a root user would mean running sessions as root user
>> >>> as well.

>> >>> Thanks,
>> >>> Mosleh

>>>>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers < [
>>>>> mailto:francis.deslauriers@efficios.com | francis.deslauriers@efficios.com ] >
>> >>> wrote:

>> >>>> Adding back lttng-dev mailing list.

>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.

>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status

>> >>>> Thank you,
>> >>>> Francis

>>>>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin < [ mailto:mosleh1235@gmail.com |
>> >>>> mosleh1235@gmail.com ] > a écrit :

>> >>>> > Hi Francis,

>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that command.

>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the command:
>> >>>> > > lttng view
>>>>>> > I will see the tracepoints in the application in the terminal even though I have
>> >>>> > not done: lttng start.

>>>>>> > I thought it was strange that I cannot find my application in the: lttng list
>> >>>> > command under any domain.
>>>>>> > Also the reason I start my application after creating a session is because the
>> >>>> > system automatically starts the application using systemd,
>>>>>> > in order to get tracing data at all I have to manually stop the application and
>> >>>> > start it again after creating a session.

>> >>>> > Thanks again for any help,
>> >>>> > Mosleh

>>>>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers < [
>>>>>> > mailto:francis.deslauriers@efficios.com | francis.deslauriers@efficios.com ] >
>> >>>> > wrote:

>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if no
>> >>>> >> lttng-ust instrumented applications are running.

>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?

>> >>>> >> Thank you,
>> >>>> >> Francis


>>>>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin < [ mailto:mosleh1235@gmail.com |
>> >>>> >> mosleh1235@gmail.com ] > a écrit :

>> >>>> >> > Hello,

>>>>>> >> > I am having a strange issue with my current application. For some context, I
>>>>>> >> > have integrated LTTng into a Yocto build and put custom trace points in my
>>>>>> >> > embedded system application. Once I load the built image onto my system and
>>>>>> >> > start a session daemon, I try to list all available userspace events (using
>> >>>> >> > lttng list -u), but no events show up.

>>>>>> >> > The strange occurrence is when I create a session (lltng create my-sess) and not
>>>>>> >> > start tracing the trace events will display on the screen regardless (when
>>>>>> >> > starting session in live mode with lttng view). I was wondering if anyone has
>> >>>> >> > any insight on this issue I am having.

>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>>> >> > [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.



>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.

>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ]
>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]



>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.

>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 12993 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1: Type: text/plain, Size: 9488 bytes --]

Hello,

I see, I was loading the liblttng-ust-fd but not fork. Once loading both of
these with LD_PRELOAD everything seems to be working great. Thank you for
all your assistance.

I was also wondering, it may be impractical to always load these shared
objects because the systems starts the app service on boot. Is there an
lttng alternative to integrate this into the build?  I am not too familiar
with dynamic linking so sorry if this sounds trivial.

Thanks,
Mosleh

On Wed, Feb 20, 2019 at 9:46 AM Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

>
>
> ----- On Feb 20, 2019, at 9:10 AM, Mosleh Uddin <mosleh1235@gmail.com>
> wrote:
>
> Hello,
> Here is what I do to get the consistent behavior:
> When the app is running as a service, lttng list -u does not display any
> events, initially I do have to lttng enable-events and lttng start to see
> the tracepoints in lttng view. However, after lttng start, I can disable
> all events and I can still see the events coming through in lttng view.
> When I run the app normally, lttng list -u displays the tracepoints I place
> in the app, and all functionality of lttng works as expected.
>
> I apologize, I am in the process of trying to get you the outputs you are
> asking for, however due to confidentiality with the app the outputs of
> running the commands need to be reviewed by others. I will try to get this
> to you as soon as I can.
>
> I can say the service is being kick started by systemd. Also I believe the
> app is mass-closing file descriptors when it is daemonizing. I tried to
> dynamically load the liblttng-ust-fd.so but the behavior did not change.
>
>
> Did you load both liblttng-ust-fork.so and liblttng-ust-fd.so with
> LD_PRELOAD when launching your application ?
>
> If not using LD_PRELOAD env. var., how did you load them ?
>
> Thanks,
>
> Mathieu
>
>
> Thanks,
> Mosleh
>
> On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
>
>> Hi Mosleh,
>> Just to be clear:
>> When you run your app as a service (background), lttng list -u doesn't
>> display any events AND you see events in the live session without
>> having run enable-event and start commands.
>> When you run your app normally, lttng list -u displays events AND you
>> only see events after running enable-event and start commands.
>> Is that correct?
>>
>> Please provide the output of the lttng-sessiond in verbose mode using
>> the following command:
>> > lttng-sessiond -vvv
>> please provide the output of your application as a service with this
>> environment variable:  LTTNG_UST_DEBUG=1
>> and please provide us with the service file describing how your
>> service is started.
>> Put all of this on a pastebin.com
>>
>> Is your application mass-closing file descriptors when it's
>> daemonizing? how about using fork() syscall without an exec() after
>> it? If so, have a look at this part of the UST man page:
>> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons
>>
>> Thank you,
>> Francis
>>
>>
>> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin <mosleh1235@gmail.com> a
>> écrit :
>> >
>> > Hello,
>> >
>> > Sorry for the late reply. I was able to figure out the weird behavior
>> from the application. Turns out if I run my application as a service,
>> lttng no longer can see any of the UST tracepoints and causes the behavior
>> I was explaining.
>> > When I execute the application normally (./app_name), the functionality
>> is as expected. Is this the expected behavior? Is there some way I can
>> trace the userspace application while running it as a service
>> simultaneously?
>> >
>> > Thanks for any info,
>> > Mosleh
>> >
>> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers <
>> francis.deslauriers@efficios.com> wrote:
>> >>
>> >> Can you start the lttng-sessiond in verbose mode using the following
>> command:
>> >> > lttng-sessiond -vvv
>> >> and then do the lttng create, start of application and lttng view on
>> another console.
>> >>
>> >> Then copy the output of the sessiond on https://pastebin.com/ and
>> send us the link.
>> >>
>> >> Thank you,
>> >> Francis
>> >>
>> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin <mosleh1235@gmail.com>
>> a écrit :
>> >>>
>> >>> That's correct, I don't enable any events or do: lttng start.
>> >>> Here is some snapshots of starting everything up, followed by
>> starting my application.
>> >>>
>> >>>
>> >>> Here are the outputs for the commands:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Note that even though the session is inactive, when a trace event
>> occurs in my application it is being recorded in the appropriate directory:
>> >>>
>> >>>
>> >>>
>> >>> Also maybe important, the system has root only user. Would having the
>> application belonging to a different user and trying to run as root cause
>> these kinds of complications? I did notice when testing on a raspberry pi
>> that running an application as a root user would mean running sessions as
>> root user as well.
>> >>>
>> >>> Thanks,
>> >>> Mosleh
>> >>>
>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers <
>> francis.deslauriers@efficios.com> wrote:
>> >>>>
>> >>>> Adding back lttng-dev mailing list.
>> >>>>
>> >>>> So you don't even run a lttng-enable-event command?
>> >>>> That's odd.
>> >>>>
>> >>>> Can you send us the output of the following commands:
>> >>>> Is it possible that there are other lttng-sessiond daemons running on
>> >>>> the machine?
>> >>>> > ps aux | grep "lttng-sessiond"
>> >>>> Verify if there are other active tracing sessions:
>> >>>> > lttng list
>> >>>> Check the details of the current session:
>> >>>> > lttng status
>> >>>>
>> >>>> Thank you,
>> >>>> Francis
>> >>>>
>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin <mosleh1235@gmail.com>
>> a écrit :
>> >>>> >
>> >>>> > Hi Francis,
>> >>>> >
>> >>>> > Thank you for the quick response.
>> >>>> > Just to clear things up, my application is running when I run that
>> command.
>> >>>> >
>> >>>> > The commands I am running are:
>> >>>> > > lttng-sessiond -d
>> >>>> > > lttng create my-sess --live
>> >>>> > After this command I start my application. If I then do the
>> command:
>> >>>> > > lttng view
>> >>>> > I will see the tracepoints in the application in the terminal even
>> though I have not done: lttng start.
>> >>>> >
>> >>>> > I thought it was strange that I cannot find my application in the:
>> lttng list command under any domain.
>> >>>> > Also the reason I start my application after creating a session is
>> because the system automatically starts the application using systemd,
>> >>>> > in order to get tracing data at all I have to manually stop the
>> application and start it again after creating a session.
>> >>>> >
>> >>>> > Thanks again for any help,
>> >>>> > Mosleh
>> >>>> >
>> >>>> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers <
>> francis.deslauriers@efficios.com> wrote:
>> >>>> >>
>> >>>> >> Hi Mosleh,
>> >>>> >> It's expected to not see any event when running lttng list -u if
>> no
>> >>>> >> lttng-ust instrumented applications are running.
>> >>>> >>
>> >>>> >> What are the exact lttng commands you are running to see those
>> >>>> >> unexpected events?
>> >>>> >>
>> >>>> >> Thank you,
>> >>>> >> Francis
>> >>>> >>
>> >>>> >>
>> >>>> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin <
>> mosleh1235@gmail.com> a écrit :
>> >>>> >> >
>> >>>> >> > Hello,
>> >>>> >> >
>> >>>> >> > I am having a strange issue with my current application. For
>> some context, I have integrated LTTng into a Yocto build and put custom
>> trace points in my embedded system application. Once I load the built image
>> onto my system and start a session daemon, I try to list all available
>> userspace events (using lttng list -u), but no events show up.
>> >>>> >> >
>> >>>> >> > The strange occurrence is when I create a session (lltng create
>> my-sess) and not start tracing the trace events will display on the screen
>> regardless (when starting session in live mode with lttng view). I was
>> wondering if anyone has any insight on this issue I am having.
>> >>>> >> >
>> >>>> >> > Thanks for any assistance,
>> >>>> >> > Mosleh
>> >>>> >> > _______________________________________________
>> >>>> >> > lttng-dev mailing list
>> >>>> >> > lttng-dev@lists.lttng.org
>> >>>> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>>> >>
>> >>>> >> --
>> >>>> >> Francis Deslauriers
>> >>>> >> Computer Engineer
>> >>>> >> EfficiOS inc.
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Francis Deslauriers
>> >>>> Computer Engineer
>> >>>> EfficiOS inc.
>> >>>
>> >>> _______________________________________________
>> >>> lttng-dev mailing list
>> >>> lttng-dev@lists.lttng.org
>> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> >>
>> >>
>> >>
>> >> --
>> >> Francis Deslauriers
>> >> Computer Engineer
>> >> EfficiOS inc.
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.
>>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 14017 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracepoints firing without being enabled in LTTng

[Jonathan Rajotte-Julien]

On Wed, Feb 20, 2019 at 01:43:00PM -0500, Mosleh Uddin wrote:
> Hello,
> 
> I see, I was loading the liblttng-ust-fd but not fork. Once loading both of
> these with LD_PRELOAD everything seems to be working great. Thank you for
> all your assistance.
> 
> I was also wondering, it may be impractical to always load these shared
> objects because the systems starts the app service on boot. Is there an

How is it not practical? I fail to see the problem with launching the
application with LD_PRELOAD set inside the service.

> lttng alternative to integrate this into the build?  I am not too familiar
> with dynamic linking so sorry if this sounds trivial.
> 
> Thanks
-- 
Jonathan Rajotte-Julien
EfficiOS

Re: Tracepoints firing without being enabled in LTTng

[Trent Piepho via lttng-dev]

On Wed, 2019-02-20 at 13:53 -0500, Jonathan Rajotte-Julien wrote:
> On Wed, Feb 20, 2019 at 01:43:00PM -0500, Mosleh Uddin wrote:
> > Hello,
> > 
> > I see, I was loading the liblttng-ust-fd but not fork. Once loading
> > both of
> > these with LD_PRELOAD everything seems to be working great. Thank
> > you for
> > all your assistance.
> > 
> > I was also wondering, it may be impractical to always load these
> > shared
> > objects because the systems starts the app service on boot. Is
> > there an
> 
> How is it not practical? I fail to see the problem with launching the
> application with LD_PRELOAD set inside the service.

One should be able to add LD_PRELOAD to an Environment line in a
systemd unit file.  But beware, from ld.so docs:

In secure-execution mode, preload pathnames containing slashes are
ignored.  Furthermore, shared objects are preloaded only from the
standard search directories and only if they have set-user-ID mode bit
enabled (which is not typical).

Another way that setting ld_preload can be hard is when the program
that needs it is started by another process that does not need it. 
Like say "mount" will fork and exec "losetup".  How do you set
LD_PRELOAD for just losetup without getting mount too?  Modify the code
of mount to inject an environment variable?  Make a wrapper script
named losetup that sets LD_PRELOAD and runs the real losetup binary?

Re: Tracepoints firing without being enabled in LTTng

[Jonathan Rajotte-Julien]

> One should be able to add LD_PRELOAD to an Environment line in a
> systemd unit file.  But beware, from ld.so docs:
> 
> In secure-execution mode, preload pathnames containing slashes are
> ignored.  Furthermore, shared objects are preloaded only from the
> standard search directories and only if they have set-user-ID mode bit
> enabled (which is not typical).

Those are all valid constraints and if I'm not mistaken can be controlled by a
system administrator. Thanks for bringing it up.

> 
> Another way that setting ld_preload can be hard is when the program
> that needs it is started by another process that does not need it. 
> Like say "mount" will fork and exec "losetup".  How do you set
> LD_PRELOAD for just losetup without getting mount too?  Modify the code
> of mount to inject an environment variable?

Unfortunately, for now, yes, if you know that the application that you exec is
instrumented with lttng and will perform either a fork call not followed by an
exec (family) call and/or close all FDs.

> Make a wrapper script
> named losetup that sets LD_PRELOAD and runs the real losetup binary?

This is valid option.

We do have a potential solution to eliminate the need of preloading the fork
wrapper but we did not yet have time to work on it.

As for the FD lib utility (liblttng-ust-fd) we are open to opinion on how to
solve the problem and make liblttng-ust-fd obsolete.

Cheers
-- 
Jonathan Rajotte-Julien
EfficiOS

Re: Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1: Type: text/plain, Size: 1835 bytes --]

Hello,

Thank you everyone for the assistance and quick response. Thankfully adding
the Environment with LD_PRELOAD in the systemd service file was enough to
get everything running smoothly.

Regards,
Mosleh

On Wed, Feb 20, 2019 at 3:06 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> > One should be able to add LD_PRELOAD to an Environment line in a
> > systemd unit file.  But beware, from ld.so docs:
> >
> > In secure-execution mode, preload pathnames containing slashes are
> > ignored.  Furthermore, shared objects are preloaded only from the
> > standard search directories and only if they have set-user-ID mode bit
> > enabled (which is not typical).
>
> Those are all valid constraints and if I'm not mistaken can be controlled
> by a
> system administrator. Thanks for bringing it up.
>
> >
> > Another way that setting ld_preload can be hard is when the program
> > that needs it is started by another process that does not need it.
> > Like say "mount" will fork and exec "losetup".  How do you set
> > LD_PRELOAD for just losetup without getting mount too?  Modify the code
> > of mount to inject an environment variable?
>
> Unfortunately, for now, yes, if you know that the application that you
> exec is
> instrumented with lttng and will perform either a fork call not followed
> by an
> exec (family) call and/or close all FDs.
>
> > Make a wrapper script
> > named losetup that sets LD_PRELOAD and runs the real losetup binary?
>
> This is valid option.
>
> We do have a potential solution to eliminate the need of preloading the
> fork
> wrapper but we did not yet have time to work on it.
>
> As for the FD lib utility (liblttng-ust-fd) we are open to opinion on how
> to
> solve the problem and make liblttng-ust-fd obsolete.
>
> Cheers
> --
> Jonathan Rajotte-Julien
> EfficiOS
>

[-- Attachment #1.2: Type: text/html, Size: 2389 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Tracepoints firing without being enabled in LTTng

[Mosleh Uddin]

[-- Attachment #1.1: Type: text/plain, Size: 660 bytes --]

Hello,

I am having a strange issue with my current application. For some context,
I have integrated LTTng into a Yocto build and put custom trace points in
my embedded system application. Once I load the built image onto my system
and start a session daemon, I try to list all available userspace events
(using lttng list -u), but no events show up.

The strange occurrence is when I create a session (lltng create my-sess)
and not start tracing the trace events will display on the screen
regardless (when starting session in live mode with lttng view). I was
wondering if anyone has any insight on this issue I am having.

Thanks for any assistance,
Mosleh

[-- Attachment #1.2: Type: text/html, Size: 764 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev