Re: [ftrace] possible to implement user-space tracers?

Re: [ftrace] possible to implement user-space tracers?

[Scotty Bauer]

It is possible to trace from userland, Android does it.

Essentially you need to write your data into 
/sys/kernel/debug/tracing/trace_marker

then read it out of /sys/kernel/debug/tracing/trace


If you care how the implementation works you can read it in 
/kernel/tracing/trace.c 
(http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for 
tracing_mark_fops and tracing_fops.


In Android here are the relevant files you will need to see how it's 
used in userland:
Atrace (for reading data out + setting things up):
https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
https://android.googlesource.com/platform/external/chromium-trace/+/master 
(host side setup of the device)

trace-dev (for writing data in):
https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c

Once we have all the data we usually run it through trace-viewer, 
(https://github.com/catapult-project/catapult)

Cheers.


> On 2015-08-11 06:53, Kun Huang wrote:
>> Hi ftrace developers
>> 
>> I'm developing a huge python based programs and facing performance
>> issue everyday. I like the ftrace system and hope there could be a
>> tracer to trace&report my python codes. Is it possible or is it
> worthy
>> to do this?
>> 
>> ---
>> 
>> Kun
>> --
>> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> [6]
>> Please read the FAQ at  http://www.tux.org/lkml/ [7]
> 
> Links:
> ------
> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
> [2]
> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
> [3] 
> https://android.googlesource.com/platform/external/chromium-trace/+/master
> [4]
> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
> [5] https://github.com/catapult-project/catapult
> [6] http://vger.kernel.org/majordomo-info.html
> [7] http://www.tux.org/lkml/

Re: [ftrace] possible to implement user-space tracers?

[Kun Huang]

Thank you for your help Scotty :) I'm reading it.

On Wed, Aug 12, 2015 at 2:34 AM, Scotty Bauer <sbauer@eng.utah.edu> wrote:
> It is possible to trace from userland, Android does it.
>
> Essentially you need to write your data into
> /sys/kernel/debug/tracing/trace_marker
>
> then read it out of /sys/kernel/debug/tracing/trace
>
>
> If you care how the implementation works you can read it in
> /kernel/tracing/trace.c
> (http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for
> tracing_mark_fops and tracing_fops.
>
>
> In Android here are the relevant files you will need to see how it's used in
> userland:
> Atrace (for reading data out + setting things up):
> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
> https://android.googlesource.com/platform/external/chromium-trace/+/master
> (host side setup of the device)
>
> trace-dev (for writing data in):
> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>
> Once we have all the data we usually run it through trace-viewer,
> (https://github.com/catapult-project/catapult)
>
> Cheers.
>
>
>> On 2015-08-11 06:53, Kun Huang wrote:
>>>
>>> Hi ftrace developers
>>>
>>> I'm developing a huge python based programs and facing performance
>>> issue everyday. I like the ftrace system and hope there could be a
>>> tracer to trace&report my python codes. Is it possible or is it
>>
>> worthy
>>>
>>> to do this?
>>>
>>> ---
>>>
>>> Kun
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>
>> linux-kernel" in
>>>
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> [6]
>>>
>>> Please read the FAQ at  http://www.tux.org/lkml/ [7]
>>
>>
>> Links:
>> ------
>> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
>> [2]
>>
>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>> [3]
>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>> [4]
>>
>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>> [5] https://github.com/catapult-project/catapult
>> [6] http://vger.kernel.org/majordomo-info.html
>> [7] http://www.tux.org/lkml/

Re: [ftrace] possible to implement user-space tracers?

[Kun Huang]

Hi Scotty

I have read your links. I found the read from 'trace' and the write to
'trace_marker' are normal read&write which works like proc system.

I have questions that what's more difference between ftrace and proc?
I just know I could read data as pipeline from ftrace.

On Wed, Aug 12, 2015 at 10:25 AM, Kun Huang <gareth@openstacker.org> wrote:
> Thank you for your help Scotty :) I'm reading it.
>
> On Wed, Aug 12, 2015 at 2:34 AM, Scotty Bauer <sbauer@eng.utah.edu> wrote:
>> It is possible to trace from userland, Android does it.
>>
>> Essentially you need to write your data into
>> /sys/kernel/debug/tracing/trace_marker
>>
>> then read it out of /sys/kernel/debug/tracing/trace
>>
>>
>> If you care how the implementation works you can read it in
>> /kernel/tracing/trace.c
>> (http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for
>> tracing_mark_fops and tracing_fops.
>>
>>
>> In Android here are the relevant files you will need to see how it's used in
>> userland:
>> Atrace (for reading data out + setting things up):
>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>> (host side setup of the device)
>>
>> trace-dev (for writing data in):
>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>
>> Once we have all the data we usually run it through trace-viewer,
>> (https://github.com/catapult-project/catapult)
>>
>> Cheers.
>>
>>
>>> On 2015-08-11 06:53, Kun Huang wrote:
>>>>
>>>> Hi ftrace developers
>>>>
>>>> I'm developing a huge python based programs and facing performance
>>>> issue everyday. I like the ftrace system and hope there could be a
>>>> tracer to trace&report my python codes. Is it possible or is it
>>>
>>> worthy
>>>>
>>>> to do this?
>>>>
>>>> ---
>>>>
>>>> Kun
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>
>>> linux-kernel" in
>>>>
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> [6]
>>>>
>>>> Please read the FAQ at  http://www.tux.org/lkml/ [7]
>>>
>>>
>>> Links:
>>> ------
>>> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
>>> [2]
>>>
>>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>>> [3]
>>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>>> [4]
>>>
>>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>> [5] https://github.com/catapult-project/catapult
>>> [6] http://vger.kernel.org/majordomo-info.html
>>> [7] http://www.tux.org/lkml/

Re: [ftrace] possible to implement user-space tracers?

[Scotty]

Hi Kun,



On 08/11/2015 08:42 PM, Kun Huang wrote:
> Hi Scotty
> 
> I have read your links. I found the read from 'trace' and the write to
> 'trace_marker' are normal read&write which works like proc system.
> 
> I have questions that what's more difference between ftrace and proc?
> I just know I could read data as pipeline from ftrace.

Well, in kernel land you can trace your functions, with very little over head. For example, in the binder driver (http://lxr.free-electrons.com/source/drivers/android/binder.c#L2423) there are trace events everywhere throughout the code. The thing is you need to build the traces into your code, if you're going to use it.

When you're in userland one way to trace your application flow is by making a wrapper around the debugfs entry (like android does) and manually write()'ing the data into the kernel buffer -- This is what android does. 

Why ftrace is in debugfs and not procfs? I'm not sure you'll have to ask the original developers, I'm sure there is a good reason.

Perhaps I didn't understand your use case. Did you want to see what was happening in kernel land while you were running your python project? Or do you want to be able to trace your python functions like the way ftrace can trace kernel functions during execution?




> On Wed, Aug 12, 2015 at 10:25 AM, Kun Huang <gareth@openstacker.org> wrote:
>> Thank you for your help Scotty :) I'm reading it.
>>
>> On Wed, Aug 12, 2015 at 2:34 AM, Scotty Bauer <sbauer@eng.utah.edu> wrote:
>>> It is possible to trace from userland, Android does it.
>>>
>>> Essentially you need to write your data into
>>> /sys/kernel/debug/tracing/trace_marker
>>>
>>> then read it out of /sys/kernel/debug/tracing/trace
>>>
>>>
>>> If you care how the implementation works you can read it in
>>> /kernel/tracing/trace.c
>>> (http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for
>>> tracing_mark_fops and tracing_fops.
>>>
>>>
>>> In Android here are the relevant files you will need to see how it's used in
>>> userland:
>>> Atrace (for reading data out + setting things up):
>>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>>> (host side setup of the device)
>>>
>>> trace-dev (for writing data in):
>>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>>
>>> Once we have all the data we usually run it through trace-viewer,
>>> (https://github.com/catapult-project/catapult)
>>>
>>> Cheers.
>>>
>>>
>>>> On 2015-08-11 06:53, Kun Huang wrote:
>>>>>
>>>>> Hi ftrace developers
>>>>>
>>>>> I'm developing a huge python based programs and facing performance
>>>>> issue everyday. I like the ftrace system and hope there could be a
>>>>> tracer to trace&report my python codes. Is it possible or is it
>>>>
>>>> worthy
>>>>>
>>>>> to do this?
>>>>>
>>>>> ---
>>>>>
>>>>> Kun
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>
>>>> linux-kernel" in
>>>>>
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> [6]
>>>>>
>>>>> Please read the FAQ at  http://www.tux.org/lkml/ [7]
>>>>
>>>>
>>>> Links:
>>>> ------
>>>> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
>>>> [2]
>>>>
>>>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>>>> [3]
>>>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>>>> [4]
>>>>
>>>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>>> [5] https://github.com/catapult-project/catapult
>>>> [6] http://vger.kernel.org/majordomo-info.html
>>>> [7] http://www.tux.org/lkml/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [ftrace] possible to implement user-space tracers?

[Kun Huang]

I need trace python projects, with low over head and a little flexibility.

There are some pure python based trace tools, but most of them make
codes slower. I also tried systemtap, but it still become much slower
in complicated python codes. I'm now finding a better tool or some
inspirations.

Flexibility for me means that operators could write some scripts to
choose what to trace. Someone traces function A, and someone traces B
for other purposes. Using the android case as example, I think I could
control the format of input data to manage trace objects. However I'm
not confident with this idea.


On Wed, Aug 12, 2015 at 11:58 AM, Scotty <sbauer@eng.utah.edu> wrote:
> Hi Kun,
>
>
>
> On 08/11/2015 08:42 PM, Kun Huang wrote:
>> Hi Scotty
>>
>> I have read your links. I found the read from 'trace' and the write to
>> 'trace_marker' are normal read&write which works like proc system.
>>
>> I have questions that what's more difference between ftrace and proc?
>> I just know I could read data as pipeline from ftrace.
>
> Well, in kernel land you can trace your functions, with very little over head. For example, in the binder driver (http://lxr.free-electrons.com/source/drivers/android/binder.c#L2423) there are trace events everywhere throughout the code. The thing is you need to build the traces into your code, if you're going to use it.
>
> When you're in userland one way to trace your application flow is by making a wrapper around the debugfs entry (like android does) and manually write()'ing the data into the kernel buffer -- This is what android does.
>
> Why ftrace is in debugfs and not procfs? I'm not sure you'll have to ask the original developers, I'm sure there is a good reason.
>
> Perhaps I didn't understand your use case. Did you want to see what was happening in kernel land while you were running your python project? Or do you want to be able to trace your python functions like the way ftrace can trace kernel functions during execution?
>
>
>
>
>> On Wed, Aug 12, 2015 at 10:25 AM, Kun Huang <gareth@openstacker.org> wrote:
>>> Thank you for your help Scotty :) I'm reading it.
>>>
>>> On Wed, Aug 12, 2015 at 2:34 AM, Scotty Bauer <sbauer@eng.utah.edu> wrote:
>>>> It is possible to trace from userland, Android does it.
>>>>
>>>> Essentially you need to write your data into
>>>> /sys/kernel/debug/tracing/trace_marker
>>>>
>>>> then read it out of /sys/kernel/debug/tracing/trace
>>>>
>>>>
>>>> If you care how the implementation works you can read it in
>>>> /kernel/tracing/trace.c
>>>> (http://lxr.free-electrons.com/source/kernel/trace/trace.c) search for
>>>> tracing_mark_fops and tracing_fops.
>>>>
>>>>
>>>> In Android here are the relevant files you will need to see how it's used in
>>>> userland:
>>>> Atrace (for reading data out + setting things up):
>>>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>>>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>>>> (host side setup of the device)
>>>>
>>>> trace-dev (for writing data in):
>>>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>>>
>>>> Once we have all the data we usually run it through trace-viewer,
>>>> (https://github.com/catapult-project/catapult)
>>>>
>>>> Cheers.
>>>>
>>>>
>>>>> On 2015-08-11 06:53, Kun Huang wrote:
>>>>>>
>>>>>> Hi ftrace developers
>>>>>>
>>>>>> I'm developing a huge python based programs and facing performance
>>>>>> issue everyday. I like the ftrace system and hope there could be a
>>>>>> tracer to trace&report my python codes. Is it possible or is it
>>>>>
>>>>> worthy
>>>>>>
>>>>>> to do this?
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> Kun
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>
>>>>> linux-kernel" in
>>>>>>
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> [6]
>>>>>>
>>>>>> Please read the FAQ at  http://www.tux.org/lkml/ [7]
>>>>>
>>>>>
>>>>> Links:
>>>>> ------
>>>>> [1] http://lxr.free-electrons.com/source/kernel/trace/trace.c
>>>>> [2]
>>>>>
>>>>> https://android.googlesource.com/platform/frameworks/native/+/master/cmds/atrace/atrace.cpp
>>>>> [3]
>>>>> https://android.googlesource.com/platform/external/chromium-trace/+/master
>>>>> [4]
>>>>>
>>>>> https://android.googlesource.com/platform/system/core/+/master/libcutils/trace-dev.c
>>>>> [5] https://github.com/catapult-project/catapult
>>>>> [6] http://vger.kernel.org/majordomo-info.html
>>>>> [7] http://www.tux.org/lkml/
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>

[ftrace] possible to implement user-space tracers?

[Kun Huang]

Hi ftrace developers

I'm developing a huge python based programs and facing performance
issue everyday. I like the ftrace system and hope there could be a
tracer to trace&report my python codes. Is it possible or is it worthy
to do this?

---

Kun