Re: Can lttng support kernel function tracer

Re: Can lttng support kernel function tracer

[Francis Deslauriers]

Hi Zhenyu,

If I understandd your question correctly, I can think of two ways to
trace an arbitrary kernel function. First, you can hook the tracer on
a dynamic probe[1] using something like this:
"lttng enable-event customEventName -k --probe kernelFunctionName".
You can also hook the tracer on both entry and exit of a function
using "--function" instead of "--probe".
To list your kernel symbols use "cat /proc/kallsyms"
Second, you can create custom kernel tracepoints and compile them in
you kernel. Have a look at the online documentation [2].

Don't hesitate to explain further if this doesn't answer your question.

Cheers!
Francis

[1] http://git.lttng.org/?p=lttng-tools.git;a=blob;f=doc/quickstart.txt;h=018c27b2b939ef7cd075255fd01bcfd8b2b21b1e;hb=HEAD#l93
[2] http://lttng.org/docs/#doc-instrumenting-linux-kernel-itself

On Thu, Feb 12, 2015 at 7:49 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
> Hi,
>
>    It's well known that lttng can make use of kernel tracepoints to do block tracing ,system call tracing etc.Is it possible that  lttng can produce anything like kernel function tracer does?
>
> Thanks
> zhenyu.ren
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: 答复： Can lttng support kernel function tracer

[Francis Deslauriers]

CCing lttng-dev back in the thread.

I might be wrong but I think that might be a usecase for a profiler
rather than a usecase for a tracer. Can someone else confirm?
To do that kind of work, you could compile your kernel with gprof support.

Thanks,
Francis


On Tue, Feb 17, 2015 at 3:38 AM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
> Hi,Francis
>    In fact, I want to trace all kernel functions related to some
> behavior.For example,I write()  and it returns 100ms later ,so  I want to
> know which kernel  function is too slow. In this case,I can use function
> tracer to trace "all" kernel function in the write() time window.However,I
> want to know is it possible that lttng can be useful in this case.
>
> Thanks
> zhenyu.ren
>
>
> ------------------------------------------------------------------
> 发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
> 发送时间：2015年2月17日(星期二) 00:53
> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
> 抄　送：lttng-dev <lttng-dev@lists.lttng.org>
> 主　题：Re: [lttng-dev] Can lttng support kernel function tracer
>
> Hi Zhenyu,
>
> If I understandd your question correctly, I can think of two ways to
> trace an arbitrary kernel function. First, you can hook the tracer on
> a dynamic probe[1] using something like this:
> "lttng enable-event customEventName -k --probe kernelFunctionName".
> You can also hook the tracer on both entry and exit of a function
> using "--function" instead of "--probe".
> To list your kernel symbols use "cat /proc/kallsyms"
> Second, you can create custom kernel tracepoints and compile them in
> you kernel. Have a look at the online documentation [2].
>
> Don't hesitate to explain further if this doesn't answer your question.
>
> Cheers!
> Francis
>
> [1]
> http://git.lttng.org/?p=lttng-tools.git;a=blob;f=doc/quickstart.txt;h=018c27b2b939ef7cd075255fd01bcfd8b2b21b1e;hb=HEAD#l93
> [2] http://lttng.org/docs/#doc-instrumenting-linux-kernel-itself
>
> On Thu, Feb 12, 2015 at 7:49 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>> Hi,
>>
>> It's well known that lttng can make use of kernel tracepoints to do block
>> tracing ,system call tracing etc.Is it possible that lttng can produce
>> anything like kernel function tracer does?
>>
>> Thanks
>> zhenyu.ren
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

答复：答复： Can lttng support kernel function tracer

[zhenyu.ren]

> I think that might be a usecase for a profiler rather than a usecase for a tracer

I am sorry that I have misled you.Not all write instances(a chance in a million) take 100ms+ to complete.So I have to trace all write instances to capture the deviant ones.

Thanks
zhenyu.ren






------------------------------------------------------------------
发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>


发送时间：2015年2月17日(星期二) 23:09
收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
抄　送：lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
主　题：Re: 答复：[lttng-dev] Can lttng support kernel function tracer


CCing lttng-dev back in the thread.

I might be wrong but I think that might be a usecase for a profiler
rather than a usecase for a tracer. Can someone else confirm?
To do that kind of work, you could compile your kernel with gprof support.

Thanks,
Francis


On Tue, Feb 17, 2015 at 3:38 AM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
> Hi,Francis
>    In fact, I want to trace all kernel functions related to some
> behavior.For example,I write()  and it returns 100ms later ,so  I want to
> know which kernel  function is too slow. In this case,I can use function
> tracer to trace "all" kernel function in the write() time window.However,I
> want to know is it possible that lttng can be useful in this case.
>
> Thanks
> zhenyu.ren
>
>
> ------------------------------------------------------------------
> 发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
> 发送时间：2015年2月17日(星期二) 00:53
> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
> 抄　送：lttng-dev <lttng-dev@lists.lttng.org>
> 主　题：Re: [lttng-dev] Can lttng support kernel function tracer
>
> Hi Zhenyu,
>
> If I understandd your question correctly, I can think of two ways to
> trace an arbitrary kernel function. First, you can hook the tracer on
> a dynamic probe[1] using something like this:
> "lttng enable-event customEventName -k --probe kernelFunctionName".
> You can also hook the tracer on both entry and exit of a function
> using "--function" instead of "--probe".
> To list your kernel symbols use "cat /proc/kallsyms"
> Second, you can create custom kernel tracepoints and compile them in
> you kernel. Have a look at the online documentation [2].
>
> Don't hesitate to explain further if this doesn't answer your question.
>
> Cheers!
> Francis
>
> [1]
> http://git.lttng.org/?p=lttng-tools.git;a=blob;f=doc/quickstart.txt;h=018c27b2b939ef7cd075255fd01bcfd8b2b21b1e;hb=HEAD#l93
> [2] http://lttng.org/docs/#doc-instrumenting-linux-kernel-itself
>
> On Thu, Feb 12, 2015 at 7:49 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>> Hi,
>>
>> It's well known that lttng can make use of kernel tracepoints to do block
>> tracing ,system call tracing etc.Is it possible that lttng can produce
>> anything like kernel function tracer does?
>>
>> Thanks
>> zhenyu.ren
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: 答复：答复： Can lttng support kernel function tracer

[Francis Deslauriers]

You can use the snapshot feature to start recording whenever you
witness a significant latency[1].
You can use the system() function to trigger a record from your app.
e.g. system("lttng snapshot record");

[1] http://lttng.org/docs/#doc-taking-a-snapshot

I hope it helps.

Francis

On Wed, Feb 18, 2015 at 1:29 AM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>> I think that might be a usecase for a profiler rather than a usecase for a tracer
>
> I am sorry that I have misled you.Not all write instances(a chance in a million) take 100ms+ to complete.So I have to trace all write instances to capture the deviant ones.
>
> Thanks
> zhenyu.ren
>
>
>
>
>
>
> ------------------------------------------------------------------
> 发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
>
>
> 发送时间：2015年2月17日(星期二) 23:09
> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
> 抄　送：lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
> 主　题：Re: 答复：[lttng-dev] Can lttng support kernel function tracer
>
>
> CCing lttng-dev back in the thread.
>
> I might be wrong but I think that might be a usecase for a profiler
> rather than a usecase for a tracer. Can someone else confirm?
> To do that kind of work, you could compile your kernel with gprof support.
>
> Thanks,
> Francis
>
>
> On Tue, Feb 17, 2015 at 3:38 AM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>> Hi,Francis
>>    In fact, I want to trace all kernel functions related to some
>> behavior.For example,I write()  and it returns 100ms later ,so  I want to
>> know which kernel  function is too slow. In this case,I can use function
>> tracer to trace "all" kernel function in the write() time window.However,I
>> want to know is it possible that lttng can be useful in this case.
>>
>> Thanks
>> zhenyu.ren
>>
>>
>> ------------------------------------------------------------------
>> 发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
>> 发送时间：2015年2月17日(星期二) 00:53
>> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
>> 抄　送：lttng-dev <lttng-dev@lists.lttng.org>
>> 主　题：Re: [lttng-dev] Can lttng support kernel function tracer
>>
>> Hi Zhenyu,
>>
>> If I understandd your question correctly, I can think of two ways to
>> trace an arbitrary kernel function. First, you can hook the tracer on
>> a dynamic probe[1] using something like this:
>> "lttng enable-event customEventName -k --probe kernelFunctionName".
>> You can also hook the tracer on both entry and exit of a function
>> using "--function" instead of "--probe".
>> To list your kernel symbols use "cat /proc/kallsyms"
>> Second, you can create custom kernel tracepoints and compile them in
>> you kernel. Have a look at the online documentation [2].
>>
>> Don't hesitate to explain further if this doesn't answer your question.
>>
>> Cheers!
>> Francis
>>
>> [1]
>> http://git.lttng.org/?p=lttng-tools.git;a=blob;f=doc/quickstart.txt;h=018c27b2b939ef7cd075255fd01bcfd8b2b21b1e;hb=HEAD#l93
>> [2] http://lttng.org/docs/#doc-instrumenting-linux-kernel-itself
>>
>> On Thu, Feb 12, 2015 at 7:49 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>>> Hi,
>>>
>>> It's well known that lttng can make use of kernel tracepoints to do block
>>> tracing ,system call tracing etc.Is it possible that lttng can produce
>>> anything like kernel function tracer does?
>>>
>>> Thanks
>>> zhenyu.ren
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

答复：答复： Can lttng support kernel function tracer

[zhenyu.ren]

Oh,What I want is something like "lttng enable-event -a -k --function " to enable ALL function trace.

Thanks
zhenyu.ren






------------------------------------------------------------------
发件人：zhenyu.ren <zhenyu.ren@aliyun.com>
发送时间：2015年2月18日(星期三) 14:29
收件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
抄　送：lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
主　题：答复：答复：[lttng-dev] Can lttng support kernel function tracer


> I think that might be a usecase for a profiler rather than a usecase for a tracer

I am sorry that I have misled you.Not all write instances(a chance in a million) take 100ms+ to complete.So I have to trace all write instances to capture the deviant ones.

Thanks
zhenyu.ren






------------------------------------------------------------------
发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>


发送时间：2015年2月17日(星期二) 23:09
收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
抄　送：lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
主　题：Re: 答复：[lttng-dev] Can lttng support kernel function tracer


CCing lttng-dev back in the thread.

I might be wrong but I think that might be a usecase for a profiler
rather than a usecase for a tracer. Can someone else confirm?
To do that kind of work, you could compile your kernel with gprof support.

Thanks,
Francis


On Tue, Feb 17, 2015 at 3:38 AM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
> Hi,Francis
>    In fact, I want to trace all kernel functions related to some
> behavior.For example,I write()  and it returns 100ms later ,so  I want to
> know which kernel  function is too slow. In this case,I can use function
> tracer to trace "all" kernel function in the write() time window.However,I
> want to know is it possible that lttng can be useful in this case.
>
> Thanks
> zhenyu.ren
>
>
> ------------------------------------------------------------------
> 发件人：Francis Deslauriers <francis.deslauriers@mail.utoronto.ca>
> 发送时间：2015年2月17日(星期二) 00:53
> 收件人：zhenyu.ren <zhenyu.ren@aliyun.com>
> 抄　送：lttng-dev <lttng-dev@lists.lttng.org>
> 主　题：Re: [lttng-dev] Can lttng support kernel function tracer
>
> Hi Zhenyu,
>
> If I understandd your question correctly, I can think of two ways to
> trace an arbitrary kernel function. First, you can hook the tracer on
> a dynamic probe[1] using something like this:
> "lttng enable-event customEventName -k --probe kernelFunctionName".
> You can also hook the tracer on both entry and exit of a function
> using "--function" instead of "--probe".
> To list your kernel symbols use "cat /proc/kallsyms"
> Second, you can create custom kernel tracepoints and compile them in
> you kernel. Have a look at the online documentation [2].
>
> Don't hesitate to explain further if this doesn't answer your question.
>
> Cheers!
> Francis
>
> [1]
> http://git.lttng.org/?p=lttng-tools.git;a=blob;f=doc/quickstart.txt;h=018c27b2b939ef7cd075255fd01bcfd8b2b21b1e;hb=HEAD#l93
> [2] http://lttng.org/docs/#doc-instrumenting-linux-kernel-itself
>
> On Thu, Feb 12, 2015 at 7:49 PM, zhenyu.ren <zhenyu.ren@aliyun.com> wrote:
>> Hi,
>>
>> It's well known that lttng can make use of kernel tracepoints to do block
>> tracing ,system call tracing etc.Is it possible that lttng can produce
>> anything like kernel function tracer does?
>>
>> Thanks
>> zhenyu.ren
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Can lttng support kernel function tracer

[zhenyu.ren]

Hi,

   It's well known that lttng can make use of kernel tracepoints to do block tracing ,system call tracing etc.Is it possible that  lttng can produce anything like kernel function tracer does?

Thanks
zhenyu.ren