Re: lttng-track on userspace channel

Re: lttng-track on userspace channel

[Mathieu Desnoyers]

----- On May 31, 2017, at 7:28 PM, Deneau, Tom tom.deneau@amd.com wrote:

> I am using lttng 2.9.4.
> 
> I have a server database daemon with many threads and I would like to track
> events on a smaller subset of those threads.
> I noticed that on the kernel channel I can use lttng track --pid=123,124,125
> 
> and this will trace kernel events for just those TIDS (in spite of the
> documentation saying it will track whole pids).
> But I am happy because tid tracking is what I want.

Argh.. this is an unfortunate discrepancy! It should have been using
the kernel "tgid" at kernel-level to match what we do for userspace.
I'll prepare a patch fixing that.

LTTng really try hard to stick to "pid means process ID" and "tid means
thread ID". Internally in lttng-modules we do all that is needed to map
the pid/tgid notions back to tid/pid.

> 
> However, is there an equivalent way to track certain tids on the userspace
> channel?  I find that if I use
>    lttng track -u --pid=xxx
> it just tracks all the tids under that pid.

You can use e.g.

lttng enable-event -k -a --filter '$ctx.tid == 123'

or if you want it container-aware:

lttng enable-event -k -a --filter '$ctx.vtid == 123'

Thanks,

Mathieu

> 
> Here is my script (where 79653 is a single tid).  In this case it traces only
> kernel events on that tid.
> -----------
> lttng create
> lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
> lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
> lttng add-context -k -c k -t tid -t pid
> lttng add-context -u -c u -t vtid -t vpid
> lttng enable-event -u -c u -a
> lttng enable-event -k -c k --syscall futex
> lttng track -k --pid=79653
> lttng track -u --pid=79653
> lttng start
> sleep 1.0
> lttng stop
> lttng destroy
> 
> -- Tom Deneau
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-track on userspace channel

[Mathieu Desnoyers]

----- On Jun 1, 2017, at 2:24 PM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:

> ----- On May 31, 2017, at 7:28 PM, Deneau, Tom tom.deneau@amd.com wrote:
> 
>> I am using lttng 2.9.4.
>> 
>> I have a server database daemon with many threads and I would like to track
>> events on a smaller subset of those threads.
>> I noticed that on the kernel channel I can use lttng track --pid=123,124,125
>> 
>> and this will trace kernel events for just those TIDS (in spite of the
>> documentation saying it will track whole pids).
>> But I am happy because tid tracking is what I want.
> 
> Argh.. this is an unfortunate discrepancy! It should have been using
> the kernel "tgid" at kernel-level to match what we do for userspace.
> I'll prepare a patch fixing that.
> 
> LTTng really try hard to stick to "pid means process ID" and "tid means
> thread ID". Internally in lttng-modules we do all that is needed to map
> the pid/tgid notions back to tid/pid.

Here is the upstream lttng-modules commit that fixes the issue, backported
to 2.10, 2.9, 2.8:

commit 12d4702757b86de457904f6e344b4f412cef611a
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Thu Jun 1 14:24:11 2017 -0400

    Fix: pid tracker should track "pgid"
    
    The "pid" notion exposed by LTTng translates to the "pgid" notion in the
    Linux kernel. Therefore using "current->pid" as argument to the PID
    tracker actually ends up behaving as a "tid" tracker, which does not
    match the intent nor the user-space tracer behavior.
    
    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Thanks for letting us know about this problem.

Mathieu

> 
>> 
>> However, is there an equivalent way to track certain tids on the userspace
>> channel?  I find that if I use
>>    lttng track -u --pid=xxx
>> it just tracks all the tids under that pid.
> 
> You can use e.g.
> 
> lttng enable-event -k -a --filter '$ctx.tid == 123'
> 
> or if you want it container-aware:
> 
> lttng enable-event -k -a --filter '$ctx.vtid == 123'
> 
> Thanks,
> 
> Mathieu
> 
>> 
>> Here is my script (where 79653 is a single tid).  In this case it traces only
>> kernel events on that tid.
>> -----------
>> lttng create
>> lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
>> lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
>> lttng add-context -k -c k -t tid -t pid
>> lttng add-context -u -c u -t vtid -t vpid
>> lttng enable-event -u -c u -a
>> lttng enable-event -k -c k --syscall futex
>> lttng track -k --pid=79653
>> lttng track -u --pid=79653
>> lttng start
>> sleep 1.0
>> lttng stop
>> lttng destroy
>> 
>> -- Tom Deneau
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-track on userspace channel

[Deneau, Tom]

Mathieu --

So just to confirm, once fixed the lttng-track command in both kernel and user
will track process ids, not thread ids, and if one does want to track thread ids,
one should use --filter with $ctx.tid.

A related question, I got errors when trying to do the following:
   lttng add-context -u -c u -t tid -t pid
so had to change to vtid and vpid.

So in user channel to filter to a tid level I would do something like:
   lttng add-context -u -c u -t vtid -t vpid
   lttng enable-event -u -c u -a -filter '$ctx.vtid == 123'

Is that correct?

-- Tom

> -----Original Message-----
> From: Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com]
> Sent: Thursday, June 01, 2017 1:25 PM
> To: Deneau, Tom <tom.deneau@amd.com>
> Cc: lttng-dev <lttng-dev@lists.lttng.org>
> Subject: Re: [lttng-dev] lttng-track on userspace channel
> 
> ----- On May 31, 2017, at 7:28 PM, Deneau, Tom tom.deneau@amd.com wrote:
> 
> > I am using lttng 2.9.4.
> >
> > I have a server database daemon with many threads and I would like to
> track
> > events on a smaller subset of those threads.
> > I noticed that on the kernel channel I can use lttng track --
> pid=123,124,125
> >
> > and this will trace kernel events for just those TIDS (in spite of the
> > documentation saying it will track whole pids).
> > But I am happy because tid tracking is what I want.
> 
> Argh.. this is an unfortunate discrepancy! It should have been using
> the kernel "tgid" at kernel-level to match what we do for userspace.
> I'll prepare a patch fixing that.
> 
> LTTng really try hard to stick to "pid means process ID" and "tid means
> thread ID". Internally in lttng-modules we do all that is needed to map
> the pid/tgid notions back to tid/pid.
> 
> >
> > However, is there an equivalent way to track certain tids on the
> userspace
> > channel?  I find that if I use
> >    lttng track -u --pid=xxx
> > it just tracks all the tids under that pid.
> 
> You can use e.g.
> 
> lttng enable-event -k -a --filter '$ctx.tid == 123'
> 
> or if you want it container-aware:
> 
> lttng enable-event -k -a --filter '$ctx.vtid == 123'
> 
> Thanks,
> 
> Mathieu
> 
> >
> > Here is my script (where 79653 is a single tid).  In this case it traces
> only
> > kernel events on that tid.
> > -----------
> > lttng create
> > lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
> > lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
> > lttng add-context -k -c k -t tid -t pid
> > lttng add-context -u -c u -t vtid -t vpid
> > lttng enable-event -u -c u -a
> > lttng enable-event -k -c k --syscall futex
> > lttng track -k --pid=79653
> > lttng track -u --pid=79653
> > lttng start
> > sleep 1.0
> > lttng stop
> > lttng destroy
> >
> > -- Tom Deneau
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-track on userspace channel

[Mathieu Desnoyers]

----- On Jun 1, 2017, at 2:49 PM, Tom Deneau tom.deneau@amd.com wrote:

> Mathieu --
> 
> So just to confirm, once fixed the lttng-track command in both kernel and user
> will track process ids, not thread ids, and if one does want to track thread
> ids,
> one should use --filter with $ctx.tid.

Yes, ideally with single-quotes so the shell does not expand $ctx ;)

> 
> A related question, I got errors when trying to do the following:
>   lttng add-context -u -c u -t tid -t pid
> so had to change to vtid and vpid.
> 
> So in user channel to filter to a tid level I would do something like:
>   lttng add-context -u -c u -t vtid -t vpid
>   lttng enable-event -u -c u -a -filter '$ctx.vtid == 123'
> 
> Is that correct?

Yes, this is correct. It's because lttng-ust, from an application
perspective, only knows about the vtid and vpid (the containerized
values of its thread and process ID).

Thanks,

Mathieu

> 
> -- Tom
> 
>> -----Original Message-----
>> From: Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com]
>> Sent: Thursday, June 01, 2017 1:25 PM
>> To: Deneau, Tom <tom.deneau@amd.com>
>> Cc: lttng-dev <lttng-dev@lists.lttng.org>
>> Subject: Re: [lttng-dev] lttng-track on userspace channel
>> 
>> ----- On May 31, 2017, at 7:28 PM, Deneau, Tom tom.deneau@amd.com wrote:
>> 
>> > I am using lttng 2.9.4.
>> >
>> > I have a server database daemon with many threads and I would like to
>> track
>> > events on a smaller subset of those threads.
>> > I noticed that on the kernel channel I can use lttng track --
>> pid=123,124,125
>> >
>> > and this will trace kernel events for just those TIDS (in spite of the
>> > documentation saying it will track whole pids).
>> > But I am happy because tid tracking is what I want.
>> 
>> Argh.. this is an unfortunate discrepancy! It should have been using
>> the kernel "tgid" at kernel-level to match what we do for userspace.
>> I'll prepare a patch fixing that.
>> 
>> LTTng really try hard to stick to "pid means process ID" and "tid means
>> thread ID". Internally in lttng-modules we do all that is needed to map
>> the pid/tgid notions back to tid/pid.
>> 
>> >
>> > However, is there an equivalent way to track certain tids on the
>> userspace
>> > channel?  I find that if I use
>> >    lttng track -u --pid=xxx
>> > it just tracks all the tids under that pid.
>> 
>> You can use e.g.
>> 
>> lttng enable-event -k -a --filter '$ctx.tid == 123'
>> 
>> or if you want it container-aware:
>> 
>> lttng enable-event -k -a --filter '$ctx.vtid == 123'
>> 
>> Thanks,
>> 
>> Mathieu
>> 
>> >
>> > Here is my script (where 79653 is a single tid).  In this case it traces
>> only
>> > kernel events on that tid.
>> > -----------
>> > lttng create
>> > lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
>> > lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
>> > lttng add-context -k -c k -t tid -t pid
>> > lttng add-context -u -c u -t vtid -t vpid
>> > lttng enable-event -u -c u -a
>> > lttng enable-event -k -c k --syscall futex
>> > lttng track -k --pid=79653
>> > lttng track -u --pid=79653
>> > lttng start
>> > sleep 1.0
>> > lttng stop
>> > lttng destroy
>> >
>> > -- Tom Deneau
>> > _______________________________________________
>> > lttng-dev mailing list
>> > lttng-dev@lists.lttng.org
>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> 
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
> > http://www.efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

lttng-track on userspace channel

[Deneau, Tom]

I am using lttng 2.9.4.

I have a server database daemon with many threads and I would like to track events on a smaller subset of those threads.
I noticed that on the kernel channel I can use lttng track --pid=123,124,125

and this will trace kernel events for just those TIDS (in spite of the documentation saying it will track whole pids).
But I am happy because tid tracking is what I want.

However, is there an equivalent way to track certain tids on the userspace channel?  I find that if I use
    lttng track -u --pid=xxx
it just tracks all the tids under that pid.

Here is my script (where 79653 is a single tid).  In this case it traces only kernel events on that tid.
-----------
lttng create
lttng enable-channel u -u --num-subbuf 64 --subbuf-size 512k
lttng enable-channel k -k --num-subbuf 64 --subbuf-size 512k
lttng add-context -k -c k -t tid -t pid
lttng add-context -u -c u -t vtid -t vpid
lttng enable-event -u -c u -a
lttng enable-event -k -c k --syscall futex
lttng track -k --pid=79653
lttng track -u --pid=79653
lttng start
sleep 1.0
lttng stop
lttng destroy

-- Tom Deneau
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev