Re: [linuxtools-dev] Standard protocols/interfaces/formats for performance tools (TCF, LTTng, ...)

Re: [linuxtools-dev] Standard protocols/interfaces/formats for performance tools (TCF, LTTng, ...)

[Mathieu Desnoyers]

* Spear, Aaron (aaron_spear@mentor.com) wrote:
> Hello everyone,
> 
> As some of you know, at Mentor Graphics we are currently working on
> multi-core profiling products for embedded systems.  The focus currently
> is using real-time trace data as the event source for analysis.  In the
> future this will expand as we desire to be able to correlate analysis of
> heterogeneous systems, e.g. embedded Linux + LTTng events on one
> machine, correlated with real-time trace data collected from a bunch of
> DSP's. 
> 
> Like the rest of you, we have spent much time in the past inventing
> proprietary data collection frameworks, mechanisms and formats only to
> eventually throw them away because a standard eventually emerges.  We
> would like to stop the insanity.
> 
> So, as an FYI, I am planning to participate in a new tools
> infrastructure working group under the auspices of the Multi-core
> association (http://www.multicore-association.org).  The working group
> aims to:
> 
> 1.  Identify common needs, functionality, and opportunities for
> information sharing between performance analysis tools.
> 2.  Discussion on identifying sharable components between performance
> analysis tools.
> 3.  Discussion on metadata dimensions of interest for standardization
> (e.g., code, space, metric, time, state)
> 
> Along those lines, we (Mentor) have a need for a protocol to connect to
> remote trace collectors and configure trace triggering/collection, and
> then efficiently download lots of binary trace data.  Sound familiar?  
> 
> TCF is an obvious choice for this as various companies are already using
> it for this purpose from what I have observed.
> 
> So, to my point:
> -What protocols are currently in use that we might consider as a
> starting point?  I see that the linuxtools project apparently has one
> for transferring LTTng event data.  Are there any docs for this
> protocol?
> 
> -Is there any other company with a proprietary protocol that would
> consider donating it to a standardization effort? (some one else who
> also desires to end the insanity :-)
> 
> -file formats: event log file formats is another obvious candidate for
> standardization.  Mentor has a file format we use that was inspired by
> LTTng's format but is optimized for extremely large real-time trace
> logs.  I intend to throw this into the mix.  Any others we should think
> about? (The LTTng format obviously...) 

Hi Aaron,

I would be glad to provide insight into the LTTng file format as needed.

It would be good to ask if the Ftrace team is interested to participate
in this standardization effort. Proposing modifications to the Ftrace
file format is on my roadmap.

I'm curious.. which version of the LTTng trace file format have you
derived your own format from ?

Thanks,

Mathieu

> 
> Best regards,
> Aaron
> 
> --
> Aaron Spear      
> Debug Tools Architect/Staff Engineer
> Embedded Systems Division
> Mentor Graphics Corporation
> Office: 303-679-8457
> 
> 
> _______________________________________________
> linuxtools-dev mailing list
> linuxtools-dev@eclipse.org
> https://dev.eclipse.org/mailman/listinfo/linuxtools-dev
> 

-- 
Mathieu Desnoyers
Operating System Efficiency Consultant
EfficiOS Inc.
http://www.efficios.com

RE: [linuxtools-dev] Standard protocols/interfaces/formats forperformance tools (TCF, LTTng, ...)

[Spear, Aaron]

Hi Mathieu, 

More dialogue below: 

> -----Original Message-----
> From: linuxtools-dev-bounces@eclipse.org 
> [mailto:linuxtools-dev-bounces@eclipse.org] On Behalf Of 
> Mathieu Desnoyers
> Sent: Wednesday, February 24, 2010 8:40 AM
> To: Linux Tools developer discussions
> Cc: dsdp-tcf-dev@eclipse.org; ltt-dev@lists.casi.polymtl.ca; 
> linux-kernel@vger.kernel.org; Steven Rostedt
> Subject: Re: [linuxtools-dev] Standard 
> protocols/interfaces/formats forperformance tools (TCF, LTTng, ...)
> 
> * Spear, Aaron (aaron_spear@mentor.com) wrote:
> > Hello everyone,
> > 
> > As some of you know, at Mentor Graphics we are currently working on 
> > multi-core profiling products for embedded systems.  The focus 
> > currently is using real-time trace data as the event source for 
> > analysis.  In the future this will expand as we desire to 
> be able to 
> > correlate analysis of heterogeneous systems, e.g. embedded Linux + 
> > LTTng events on one machine, correlated with real-time trace data 
> > collected from a bunch of DSP's.
> > 
> > Like the rest of you, we have spent much time in the past inventing 
> > proprietary data collection frameworks, mechanisms and 
> formats only to 
> > eventually throw them away because a standard eventually 
> emerges.  We 
> > would like to stop the insanity.
> > 
> > So, as an FYI, I am planning to participate in a new tools 
> > infrastructure working group under the auspices of the Multi-core 
> > association (http://www.multicore-association.org).  The 
> working group 
> > aims to:
> > 
> > 1.  Identify common needs, functionality, and opportunities for 
> > information sharing between performance analysis tools.
> > 2.  Discussion on identifying sharable components between 
> performance 
> > analysis tools.
> > 3.  Discussion on metadata dimensions of interest for 
> standardization 
> > (e.g., code, space, metric, time, state)
> > 
> > Along those lines, we (Mentor) have a need for a protocol 
> to connect 
> > to remote trace collectors and configure trace 
> triggering/collection, 
> > and then efficiently download lots of binary trace data.  
> Sound familiar?
> > 
> > TCF is an obvious choice for this as various companies are already 
> > using it for this purpose from what I have observed.
> > 
> > So, to my point:
> > -What protocols are currently in use that we might consider as a 
> > starting point?  I see that the linuxtools project 
> apparently has one 
> > for transferring LTTng event data.  Are there any docs for this 
> > protocol?
> > 
> > -Is there any other company with a proprietary protocol that would 
> > consider donating it to a standardization effort? (some one 
> else who 
> > also desires to end the insanity :-)
> > 
> > -file formats: event log file formats is another obvious 
> candidate for 
> > standardization.  Mentor has a file format we use that was 
> inspired by 
> > LTTng's format but is optimized for extremely large real-time trace 
> > logs.  I intend to throw this into the mix.  Any others we should 
> > think about? (The LTTng format obviously...)
> 
> Hi Aaron,
> 
> I would be glad to provide insight into the LTTng file format 
> as needed.

Great! Insight and experience gleaned from your work is certainly
desired.

> 
> It would be good to ask if the Ftrace team is interested to 
> participate in this standardization effort. Proposing 
> modifications to the Ftrace file format is on my roadmap.

I must confess that I know nothing about Ftrace.  That said, any prior
art in the space of file formats and protocols for exchanging profiling
and trace data should be considered, and input from existing communities
is warmly welcomed.  The charter of this multi-core working group is to
help forge some interoperability standards between different tools as a
start.  We believe that the future will be heavily multi-core, and it is
a difficult problem to solve figuring out graceful ways to partition a
complex "application" across these cores effectively.  E.g. a system
with SMP Linux on a couple of cores, a low level RTOS on another core,
and then some DSP's as well.  Today you often use totally different
tools for all of those cores.  How do you understand what the heck is
happening in this system, never mind figuring out how to optimize the
system as a whole...   I think a good first step is some level of
interoperability in data formats so that event data collected from
different sources and technologies (e.g. LTTng for Linux and real-time
trace for the DSP's) can be correlated and analyzed side by side. 

Note btw that this is just me being proactive and taking some liberty at
this point to get discussion started.  I don't (yet) have any official
sanction from the MC working group, though I know these thoughts
resonate with various initial participants (some of whom I know are on
these lists)
 
> I'm curious.. which version of the LTTng trace file format 
> have you derived your own format from ?

I just tried to look and cannot remember the exact version of LTTng that
I initially played with and studied now.  Not sure if it tells you, but
lttctl says it is version 0.70-18082009 on a 2.6.30 kernel.  I patched
and rebuilt the kernel on top of an Ubuntu 9.04 about 8 months ago or
so.  I will be working on polishing my spec a bit and will circulate for
people to look at as soon as I can.  I have a non-related deadline I am
fighting right now, so it may be a couple of weeks.

Best regards,
Aaron

> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Best regards,
> > Aaron
> > 
> > --
> > Aaron Spear      
> > Debug Tools Architect/Staff Engineer
> > Embedded Systems Division
> > Mentor Graphics Corporation
> > Office: 303-679-8457
> > 
> > 
> > _______________________________________________
> > linuxtools-dev mailing list
> > linuxtools-dev@eclipse.org
> > https://dev.eclipse.org/mailman/listinfo/linuxtools-dev
> > 
> 
> --
> Mathieu Desnoyers
> Operating System Efficiency Consultant
> EfficiOS Inc.
> http://www.efficios.com
> _______________________________________________
> linuxtools-dev mailing list
> linuxtools-dev@eclipse.org
> https://dev.eclipse.org/mailman/listinfo/linuxtools-dev
> 

RE: [linuxtools-dev] Standard protocols/interfaces/formats forperformance tools (TCF, LTTng, ...)

[Steven Rostedt]

On Wed, 2010-02-24 at 14:47 -0800, Spear, Aaron wrote:
> Hi Mathieu, 
> 

Mathieu, thanks for Cc'ing me.

> > > So, as an FYI, I am planning to participate in a new tools 
> > > infrastructure working group under the auspices of the Multi-core 
> > > association (http://www.multicore-association.org).  The 
> > working group 
> > > aims to:
> > > 
> > > 1.  Identify common needs, functionality, and opportunities for 
> > > information sharing between performance analysis tools.
> > > 2.  Discussion on identifying sharable components between 
> > performance 
> > > analysis tools.
> > > 3.  Discussion on metadata dimensions of interest for 
> > standardization 
> > > (e.g., code, space, metric, time, state)

I would most definitely like to be apart of this.

> > > 
> > > Along those lines, we (Mentor) have a need for a protocol 
> > to connect 
> > > to remote trace collectors and configure trace 
> > triggering/collection, 
> > > and then efficiently download lots of binary trace data.  
> > Sound familiar?
> > > 
> > > TCF is an obvious choice for this as various companies are already 
> > > using it for this purpose from what I have observed.
> > > 
> > > So, to my point:
> > > -What protocols are currently in use that we might consider as a 
> > > starting point?  I see that the linuxtools project 
> > apparently has one 
> > > for transferring LTTng event data.  Are there any docs for this 
> > > protocol?
> > > 
> > > -Is there any other company with a proprietary protocol that would 
> > > consider donating it to a standardization effort? (some one 
> > else who 
> > > also desires to end the insanity :-)
> > > 
> > > -file formats: event log file formats is another obvious 
> > candidate for 
> > > standardization.  Mentor has a file format we use that was 
> > inspired by 
> > > LTTng's format but is optimized for extremely large real-time trace 
> > > logs.  I intend to throw this into the mix.  Any others we should 
> > > think about? (The LTTng format obviously...)
> > 
> > Hi Aaron,
> > 
> > I would be glad to provide insight into the LTTng file format 
> > as needed.
> 
> Great! Insight and experience gleaned from your work is certainly
> desired.
> 
> > 
> > It would be good to ask if the Ftrace team is interested to 
> > participate in this standardization effort. Proposing 
> > modifications to the Ftrace file format is on my roadmap.
> 
> I must confess that I know nothing about Ftrace.

Here's a bunch of quick pointers:

http://people.redhat.com/srostedt/ftrace-tutorial-linux-con-2009.odp
http://lwn.net/Articles/365835/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/370423/
http://lwn.net/Articles/343766/

>   That said, any prior
> art in the space of file formats and protocols for exchanging profiling
> and trace data should be considered, and input from existing communities
> is warmly welcomed.  The charter of this multi-core working group is to
> help forge some interoperability standards between different tools as a
> start.  We believe that the future will be heavily multi-core, and it is
> a difficult problem to solve figuring out graceful ways to partition a
> complex "application" across these cores effectively.  E.g. a system
> with SMP Linux on a couple of cores, a low level RTOS on another core,
> and then some DSP's as well.  Today you often use totally different
> tools for all of those cores.  How do you understand what the heck is
> happening in this system, never mind figuring out how to optimize the
> system as a whole...   I think a good first step is some level of
> interoperability in data formats so that event data collected from
> different sources and technologies (e.g. LTTng for Linux and real-time
> trace for the DSP's) can be correlated and analyzed side by side. 

Aaron,

Let me introduce myself. I'm the author and current maintainer of
Ftrace. Ftrace was accepted in the mainline kernel in 2.6.27 and has
grown tremendously. It has a rich array of features to trace the inner
workings of the kernel.

Lately I've been working hard on a userspace tool to read from ftrace.
It uses the linux syscall splice, that allows recording of the trace
buffer into disk storage or the network with zero copy overhead (splice
lets userspace tell the kernel to move one page from one file descriptor
to another).

I'll be debuting this tool at the CELF conference this April in San
Francisco. It's called trace-cmd and you can down load the latest code
from:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

I also have a gui tool that I will be debuting at the Linux
Collaboration Conference that takes place immediately after CELF.

-- Steve

RE: [linuxtools-dev] Standard protocols/interfaces/formats forperformance tools (TCF, LTTng, ...)

[Tasneem Brutch - SISA]

Hello,
I proposed, and currently chair the newly formed Multicore Association,
Tool Infrastructure work group (TIWG).  The work group welcomes
opportunities to better understand other efforts, that TIWG can
leverage, and learn from.  I will be at the Multicore Expo, where I am
presenting, and I also plan on attending the EclipseCon.  
I would very much appreciate an opportunity to talk face to face, and to
get a better understanding of other efforts.

Regards,
Tasneem Brutch, Ph.D.
Samsung Electronics US R&D Center
75 W. Plumeria Drive
San Jose, CA. 95134
Work: 408-544-5626
Cell: 408-712-7858

-----Original Message-----
From: Steven Rostedt [mailto:rostedt@goodmis.org] 
Sent: Wednesday, February 24, 2010 8:32 PM
To: Spear, Aaron
Cc: Linux Tools developer discussions; dsdp-tcf-dev@eclipse.org;
ltt-dev@lists.casi.polymtl.ca; linux-kernel@vger.kernel.org; Tasneem
Brutch - SISA
Subject: RE: [linuxtools-dev] Standard protocols/interfaces/formats
forperformance tools (TCF, LTTng, ...)

On Wed, 2010-02-24 at 14:47 -0800, Spear, Aaron wrote:
> Hi Mathieu, 
> 

Mathieu, thanks for Cc'ing me.

> > > So, as an FYI, I am planning to participate in a new tools 
> > > infrastructure working group under the auspices of the Multi-core 
> > > association (http://www.multicore-association.org).  The 
> > working group 
> > > aims to:
> > > 
> > > 1.  Identify common needs, functionality, and opportunities for 
> > > information sharing between performance analysis tools.
> > > 2.  Discussion on identifying sharable components between 
> > performance 
> > > analysis tools.
> > > 3.  Discussion on metadata dimensions of interest for 
> > standardization 
> > > (e.g., code, space, metric, time, state)

I would most definitely like to be apart of this.

> > > 
> > > Along those lines, we (Mentor) have a need for a protocol 
> > to connect 
> > > to remote trace collectors and configure trace 
> > triggering/collection, 
> > > and then efficiently download lots of binary trace data.  
> > Sound familiar?
> > > 
> > > TCF is an obvious choice for this as various companies are already

> > > using it for this purpose from what I have observed.
> > > 
> > > So, to my point:
> > > -What protocols are currently in use that we might consider as a 
> > > starting point?  I see that the linuxtools project 
> > apparently has one 
> > > for transferring LTTng event data.  Are there any docs for this 
> > > protocol?
> > > 
> > > -Is there any other company with a proprietary protocol that would

> > > consider donating it to a standardization effort? (some one 
> > else who 
> > > also desires to end the insanity :-)
> > > 
> > > -file formats: event log file formats is another obvious 
> > candidate for 
> > > standardization.  Mentor has a file format we use that was 
> > inspired by 
> > > LTTng's format but is optimized for extremely large real-time
trace 
> > > logs.  I intend to throw this into the mix.  Any others we should 
> > > think about? (The LTTng format obviously...)
> > 
> > Hi Aaron,
> > 
> > I would be glad to provide insight into the LTTng file format 
> > as needed.
> 
> Great! Insight and experience gleaned from your work is certainly
> desired.
> 
> > 
> > It would be good to ask if the Ftrace team is interested to 
> > participate in this standardization effort. Proposing 
> > modifications to the Ftrace file format is on my roadmap.
> 
> I must confess that I know nothing about Ftrace.

Here's a bunch of quick pointers:

http://people.redhat.com/srostedt/ftrace-tutorial-linux-con-2009.odp
http://lwn.net/Articles/365835/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/370423/
http://lwn.net/Articles/343766/

>   That said, any prior
> art in the space of file formats and protocols for exchanging
profiling
> and trace data should be considered, and input from existing
communities
> is warmly welcomed.  The charter of this multi-core working group is
to
> help forge some interoperability standards between different tools as
a
> start.  We believe that the future will be heavily multi-core, and it
is
> a difficult problem to solve figuring out graceful ways to partition a
> complex "application" across these cores effectively.  E.g. a system
> with SMP Linux on a couple of cores, a low level RTOS on another core,
> and then some DSP's as well.  Today you often use totally different
> tools for all of those cores.  How do you understand what the heck is
> happening in this system, never mind figuring out how to optimize the
> system as a whole...   I think a good first step is some level of
> interoperability in data formats so that event data collected from
> different sources and technologies (e.g. LTTng for Linux and real-time
> trace for the DSP's) can be correlated and analyzed side by side. 

Aaron,

Let me introduce myself. I'm the author and current maintainer of
Ftrace. Ftrace was accepted in the mainline kernel in 2.6.27 and has
grown tremendously. It has a rich array of features to trace the inner
workings of the kernel.

Lately I've been working hard on a userspace tool to read from ftrace.
It uses the linux syscall splice, that allows recording of the trace
buffer into disk storage or the network with zero copy overhead (splice
lets userspace tell the kernel to move one page from one file descriptor
to another).

I'll be debuting this tool at the CELF conference this April in San
Francisco. It's called trace-cmd and you can down load the latest code
from:

$ git clone
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

I also have a gui tool that I will be debuting at the Linux
Collaboration Conference that takes place immediately after CELF.

-- Steve

Re: [ltt-dev] [linuxtools-dev] Standard protocols/interfaces/formats forperformance tools (TCF, LTTng, ...)

[Michel Dagenais]

> I proposed, and currently chair the newly formed Multicore Association,
> Tool Infrastructure work group (TIWG).  The work group welcomes
> opportunities to better understand other efforts, that TIWG can
> leverage, and learn from.  I will be at the Multicore Expo, where I am
> presenting, and I also plan on attending the EclipseCon.  

Great! it may be a good idea to start accumulating pointers, identified 
shortcomings, ideas... in preparation for this and LinuxCon.

>>>> Along those lines, we (Mentor) have a need for a protocol 
>>> to connect to remote trace collectors and configure trace 
>>> triggering/collection, and then efficiently download lots of binary trace data.  
>>> Sound familiar?
...
>>>> Mentor has a file format we use that was 
>>> inspired by LTTng's format but is optimized for extremely large real-time trace 
>>>> logs.  I intend to throw this into the mix.
...
>>> It would be good to ask if the Ftrace team is interested to 
>>> participate in this standardization effort. Proposing 
>>> modifications to the Ftrace file format is on my roadmap.

This is indeed the problem I currently see with Ftrace, suitability for 
huge live/realtime traces. For this you need an extremely compact format 
and a good way to pass and update metadata along with the trace. 
Otherwise, Ftrace and Perf offer a large number of exciting features.

In LTTng, following some feedback from Google among others, quite a bit 
of information is implicit: per cpu files and scheduling events obviate 
the need for pid and cpu id; event ids implicitly tells the event size 
and format... Similarly, event ids are scoped by channel using little 
space, and timestamps do not store all the most significant bits. Since 
new modules may be loaded at any time with new event types, the dynamic 
allocation of event ids and update of associated metadata is something 
which must be handled properly.

Other approaches are possible to achieve the same result. Aaron Spear 
mentioned "contexts" to qualify node/cpu/pid, I am eager to learn more 
about that... You could have "define context" events, where a context id 
would be associated with a number of attributes (CPU, pid, event 
name...) and could be reused at any time simply by issuing another 
"define context" event with the same id but different attributes. The 
important part is that each event should use little more than its 
specific payload (typical event has a payload of 4 bytes and occupies a 
total of 8 to 12 bytes on LTTng). Ftrace currently has a large number of 
common fields and was thus not optimised for this; this rapidly turns a 
10GB trace into a 30GB one.

The second important missing feature is dynamic updates of the metadata 
as new event types are added when modules are loaded. In LTTng, metadata 
is received as events of a predefined type in a dedicated channel. I am 
sure that something similar could be possible for Ftrace.

>> We believe that the future will be heavily multi-core, and it is
>> a difficult problem to solve figuring out graceful ways to partition a
>> complex "application" across these cores effectively.  E.g. a system
>> with SMP Linux on a couple of cores, a low level RTOS on another core,
>> and then some DSP's as well.  Today you often use totally different
>> tools for all of those cores.  How do you understand what the heck is
>> happening in this system, never mind figuring out how to optimize the
>> system as a whole...   I think a good first step is some level of
>> interoperability in data formats so that event data collected from
>> different sources and technologies (e.g. LTTng for Linux and real-time
>> trace for the DSP's) can be correlated and analyzed side by side. 

We have some neat and fairly sophisticated tools in LTTV now to 
correlate traces taken on distributed systems with non synchronized 
clocks simply by looking at messages exchanges.