Re: [lttng-dev] tracing of non-JNI java in IBM jvm

Re: [lttng-dev] tracing of non-JNI java in IBM jvm

[Aaron Spear]

[-- Attachment #1: Type: text/plain, Size: 8476 bytes --]

Hi Erik,

I am happy to report that I already implemented what I mentioned before as a *prototype*.  My solution has the following:

-JNI library and classes that use UST to log function entries-exits.  So I end up with a CTF log that has function entries and exits.
-extension to in-trace agent library to use this library for function entries and exits.  So, you can select a subset of classes to instrument dynamically.
-clone of the Control Flow View that understands this particular function entry and exit event schema
-I have threads as the parent, and then whatever functions the thread calls as children.  If you collapse the thread, you still see intervals over which it is running which is nice.

The net result is that you can see Gantt charts of function calls in *any* java application.  All you have to do is
1) launch the java app injecting the custom in-trace agent
2) start lttng ust tracing
3) connect to the java app's injected agent and select what classes you want instrumented

See the attached screenshot that shows a very simple java application that spawns 5 threads, and each thread calls 4 functions and then exits.

I think that the idea is pretty cool and have the intent to contribute it if there is interest.  This implementation is not yet ready to show anyone as I hacked it together for a paper that I am writing.  Here are a few of the caveats about it:

1) the current implementation is quite intrusive on the java side.  Sloppy use of strings in particular.  It needs some careful architecting.  I have a number of ideas on this front, but have not yet had bandwidth to implement them.
2) There are lots of details to be worked out with the in-trace integration.  The current collection UI really needs some help.  I tried briefly to get the eclipse plugin to work with Eclipse 4, but only the RCP app seems to work for me.  I think that the "Connections" view needs to have different collection agents plugged in and you should be able to connect to a java app and dynamically select classes to instrument and/or select from sets of classes/methods.
3) My vision is to create a viewer that could do arbitrary state over time, and use CTF extensions to feed this so that the implementation is entirely data driven.  Basically any hierarchy of parent-child entries with state under them.  This particular one is, like the LTTng Control Flow View, hard coded to the event schema right now, but I can see how we can make it more general.   I should say that the guys that wrote the framework did a good job of making it useable for different things (nice work guys!) 

Note that you could in fact use it for instrumenting native applications as well of course.  The "DynamoRIO" project in particular does the same sort of thing as In Trace does (only more efficiently) so I think this is quite viable as a collection mechanism.

cheers!
Aaron Spear

----- Original Message -----
> From: "Erik Ostermueller" <Erik.Ostermueller-/Lv4ykeZ0xsS+FvcfC7Uqw@public.gmane.org>
> To: "Aaron Spear" <aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Cc: lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org, "Alexandre Montplaisir" <alexmonthy-GLOHREzkiE4oS2+YlJWNIQ@public.gmane.org>
> Sent: Wednesday, January 23, 2013 11:03:15 AM
> Subject: RE: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> Aaron,
> 
> Sorry to have disappeared from this discussion.
> What if we went is a slightly different direction.
> 
> The InTrace tool (http://mchr3k.github.com/org.intrace/) will log
> start/stop times of dynamically instrumented methods to a text file
> (proprietary format).
> What if we simply wrote a process to convert that log to CTF, and
> view the trace with the lttng eclipse plug-in, unmodified.
> 
> The idea would be to, at first, just support these two views:
> http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Control_Flow_View
> http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Statistics_View
> 
> --Erik
> 
> 
> 
> -----Original Message-----
> From: Aaron Spear [mailto:aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> Sent: Friday, January 11, 2013 12:56 PM
> To: Ostermueller, Erik
> Cc: lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org; Alexandre Montplaisir
> Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> 
> ----- Original Message -----
> > From: "Erik Ostermueller" <Erik.Ostermueller-/Lv4ykeZ0xsS+FvcfC7Uqw@public.gmane.org>
> > To: "Alexandre Montplaisir" <alexmonthy-GLOHREzkiE4oS2+YlJWNIQ@public.gmane.org>
> > Cc: lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> > Sent: Friday, January 11, 2013 9:26:51 AM
> > Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> > 
> > Thanks Alex for the quick reply....now I get it.
> > I'll jump in on that other thread you mentioned.
> > 
> > There is a tool called InTrace that does some of this work.
> > Perhaps some design ideas or even code could be stolen from here:
> >  http://mchr3k.github.com/org.intrace/
> > This project:
> > * implements the java.lang.instrumentation api, which can be used
> > to
> > plug a tracing tool into an already-running JVM.
> > * allows you to instrument/de-instrument various classes without
> > restarting the JVM.
> > * Uses http://asm.ow2.org/ to inject byte code for entry/exit
> > tracing
> > methods.
> > * stores thread id (good) but does nothing to sort/organize
> > activity
> > by thread id into a session(bad)
> > * no graphing/aggregating of performance metrics...just displays
> > entry/exit times, method parameters, etc...
> 
> Hi Erik,
> 
> I think that what you are mentioning is certainly one thing that is
> needed. To me I see a couple of different components here:
> 1) implementation of java.util.logging interfaces as well as SLF4J
> interfaces, as well as a dedicated LTTng java API for registering
> custom event typess.  This would be used for static, compile time
> instrumentation of java apps.
> 2) an implementation that uses the tooling you mention above for
> dynamic instrumentation of any running java app.  You basically
> would define trace and install trace points on the fly.  This would
> be really, really powerful.
> 
> The implementations of the above would have to be quite different
> from the way that things are done currently with UST.  Currently UST
> uses macros in C files to define the various tracepoints statically.
>  We would need to have API's that could be called dynamically to
> define tracepoints, so that the java code could do this.  I have not
> actually looked under the hood with the current UST implementation
> to see what the implications are for this.
> 
> It seems to me that we would want to have a Java API usage similar to
> what you can do with SLF4J:
> 
> class MyClass {
>       private static final Tracer tracer =
>       LTTngFactory.getTracer("MyComponentName");
>       private static final String MY_TRACEPOINT =
>       "MyTracepointWithIntegerStringAndLong";
> 
>       static {
>           // register a tracepoint.  this goes through JNI and
>           creates a tracepoint
>           // in UST that has the given name and the specified objects
>           tracer.registerTracepoint(MY_TRACEPOINT,
>                                      new
>                                      Object[]{Integer.class,String.class,Long.class});
>       }
> 
>       public void someMethod() {
>           Integer someInteger = 123456;
>           String  someString = "this is the string";
>           Long  someLong = 56789;
> 
>           tracer.trace(MY_TRACEPOINT,someInteger,someString,someLong);
>       }
> }
> 
> Then over the top of this you have implementations of SLF4J,
> java.util.logging, whatever.  The infrastructure required in the JNI
> side for this would be the same implementation needed by the binding
> to the InTrace technology to dynamically create tracepoints.
> 
> I need all of this right now, and am willing to spend time in the
> next few months to implement it.
> 
> cheers,
> Aaron Spear, Staff Engineer, VMware Inc
> 
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i)
> delete the message and all copies; (ii) do not disclose, distribute
> or use the message in any manner; and (iii) notify the sender
> immediately. In addition, please be aware that any message addressed
> to our domain is subject to archiving and review by persons other
> than the intended recipient. Thank you
> 

[-- Attachment #2: execution-flow-view.png --]
[-- Type: image/png, Size: 197827 bytes --]

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: tracing of non-JNI java in IBM jvm

[Ostermueller, Erik]

Aaron wrote:
>> not yet ready to show anyone as I hacked it together
Hacked, shmacked. Pour that code into a jar file ship it! ...just kidding.
Seriously, though, I'd love to test out your prototype to get a feel for all the pieces involved -- I'm new to lttng.
If the prototype grows into something more release-able, I might have a small amount of time to contribute some code....if nothing else, I'd be happy to some load and do some profiling of the solution.

Would also be nice to optionally capture some "state"  to attach to a single gantt trace.  For example, SOA developers (they're so picky) would want the XML input that led to a performance problem that has been traced.  The uses would be responsible for identifying which method return val or parm contained input XML.  In Trace already knows how to capture that.  Would then want the new solution to dump the XML onto thread local storage (or similar) so it could be included (somehow) in the trace output.  Does CTF support storage of additional data like this?

I'm the kernel folks are rolling their eyes... sounds like a lot of overhead....but CompuWare's DynaTrace already does this with impressively low overhead.

--Erik

-----Original Message-----
From: Aaron Spear [mailto:aspear@vmware.com]
Sent: Wednesday, January 23, 2013 1:45 PM
To: Ostermueller, Erik
Cc: lttng-dev@lists.lttng.org; Alexandre Montplaisir; linuxtools-dev@eclipse.org
Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm

Hi Erik,

I am happy to report that I already implemented what I mentioned before as a *prototype*.  My solution has the following:

-JNI library and classes that use UST to log function entries-exits.  So I end up with a CTF log that has function entries and exits.
-extension to in-trace agent library to use this library for function entries and exits.  So, you can select a subset of classes to instrument dynamically.
-clone of the Control Flow View that understands this particular function entry and exit event schema -I have threads as the parent, and then whatever functions the thread calls as children.  If you collapse the thread, you still see intervals over which it is running which is nice.

The net result is that you can see Gantt charts of function calls in *any* java application.  All you have to do is
1) launch the java app injecting the custom in-trace agent
2) start lttng ust tracing
3) connect to the java app's injected agent and select what classes you want instrumented

See the attached screenshot that shows a very simple java application that spawns 5 threads, and each thread calls 4 functions and then exits.

I think that the idea is pretty cool and have the intent to contribute it if there is interest.  This implementation is not yet ready to show anyone as I hacked it together for a paper that I am writing.  Here are a few of the caveats about it:

1) the current implementation is quite intrusive on the java side.  Sloppy use of strings in particular.  It needs some careful architecting.  I have a number of ideas on this front, but have not yet had bandwidth to implement them.
2) There are lots of details to be worked out with the in-trace integration.  The current collection UI really needs some help.  I tried briefly to get the eclipse plugin to work with Eclipse 4, but only the RCP app seems to work for me.  I think that the "Connections" view needs to have different collection agents plugged in and you should be able to connect to a java app and dynamically select classes to instrument and/or select from sets of classes/methods.
3) My vision is to create a viewer that could do arbitrary state over time, and use CTF extensions to feed this so that the implementation is entirely data driven.  Basically any hierarchy of parent-child entries with state under them.  This particular one is, like the LTTng Control Flow View, hard coded to the event schema right now, but I can see how we can make it more general.   I should say that the guys that wrote the framework did a good job of making it useable for different things (nice work guys!) 

Note that you could in fact use it for instrumenting native applications as well of course.  The "DynamoRIO" project in particular does the same sort of thing as In Trace does (only more efficiently) so I think this is quite viable as a collection mechanism.

cheers!
Aaron Spear

----- Original Message -----
> From: "Erik Ostermueller" <Erik.Ostermueller@fisglobal.com>
> To: "Aaron Spear" <aspear@vmware.com>
> Cc: lttng-dev@lists.lttng.org, "Alexandre Montplaisir" 
> <alexmonthy@voxpopuli.im>
> Sent: Wednesday, January 23, 2013 11:03:15 AM
> Subject: RE: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> Aaron,
> 
> Sorry to have disappeared from this discussion.
> What if we went is a slightly different direction.
> 
> The InTrace tool
> (https://urldefense.proofpoint.com/v1/url?u=http://mchr3k.github.com/o
> rg.intrace/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=k0mtagpeWG6phJwAIzKq%2B4lTFA1JCM2OiWhVYmwwWeM%3D%0A&m=lFXy7dQ3u8Zg1jElCpolYCb7TruJrJgRwBFE22Y6%2BmY%3D%0A&s=96ea6ae0f26bd65195aad3b74892a253c5f364ba26e802b78e04d40b04f6ea10) will log start/stop times of dynamically instrumented methods to a text file (proprietary format).
> What if we simply wrote a process to convert that log to CTF, and view 
> the trace with the lttng eclipse plug-in, unmodified.
> 
> The idea would be to, at first, just support these two views:
> https://urldefense.proofpoint.com/v1/url?u=http://wiki.eclipse.org/Lin
> ux_Tools_Project/LTTng/User_Guide%23Control_Flow_View&k=%2FbkpAUdJWZui
> TILCq%2FFnQg%3D%3D%0A&r=k0mtagpeWG6phJwAIzKq%2B4lTFA1JCM2OiWhVYmwwWeM%
> 3D%0A&m=lFXy7dQ3u8Zg1jElCpolYCb7TruJrJgRwBFE22Y6%2BmY%3D%0A&s=912d647a
> 2e19035e7ceb23bfbb7ca8b2e4217d47037a04c3e4ec65862ceae233
> https://urldefense.proofpoint.com/v1/url?u=http://wiki.eclipse.org/Lin
> ux_Tools_Project/LTTng/User_Guide%23Statistics_View&k=%2FbkpAUdJWZuiTI
> LCq%2FFnQg%3D%3D%0A&r=k0mtagpeWG6phJwAIzKq%2B4lTFA1JCM2OiWhVYmwwWeM%3D
> %0A&m=lFXy7dQ3u8Zg1jElCpolYCb7TruJrJgRwBFE22Y6%2BmY%3D%0A&s=b8aa3bbc7e
> 1bd3611941f0be18ced0000cd89295515ff32346c71549cc49ea26
> 
> --Erik
> 
> 
> 
> -----Original Message-----
> From: Aaron Spear [mailto:aspear@vmware.com]
> Sent: Friday, January 11, 2013 12:56 PM
> To: Ostermueller, Erik
> Cc: lttng-dev@lists.lttng.org; Alexandre Montplaisir
> Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> 
> ----- Original Message -----
> > From: "Erik Ostermueller" <Erik.Ostermueller@fisglobal.com>
> > To: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> > Cc: lttng-dev@lists.lttng.org
> > Sent: Friday, January 11, 2013 9:26:51 AM
> > Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> > 
> > Thanks Alex for the quick reply....now I get it.
> > I'll jump in on that other thread you mentioned.
> > 
> > There is a tool called InTrace that does some of this work.
> > Perhaps some design ideas or even code could be stolen from here:
> >  
> > https://urldefense.proofpoint.com/v1/url?u=http://mchr3k.github.com/
> > org.intrace/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=k0mtagpeWG6phJw
> > AIzKq%2B4lTFA1JCM2OiWhVYmwwWeM%3D%0A&m=lFXy7dQ3u8Zg1jElCpolYCb7TruJr
> > JgRwBFE22Y6%2BmY%3D%0A&s=96ea6ae0f26bd65195aad3b74892a253c5f364ba26e
> > 802b78e04d40b04f6ea10
> > This project:
> > * implements the java.lang.instrumentation api, which can be used to 
> > plug a tracing tool into an already-running JVM.
> > * allows you to instrument/de-instrument various classes without 
> > restarting the JVM.
> > * Uses
> > https://urldefense.proofpoint.com/v1/url?u=http://asm.ow2.org/&k=%2F
> > bkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=k0mtagpeWG6phJwAIzKq%2B4lTFA1JCM2OiWhVYmwwWeM%3D%0A&m=lFXy7dQ3u8Zg1jElCpolYCb7TruJrJgRwBFE22Y6%2BmY%3D%0A&s=e3bd826c82ea04de7f36936be65be723da73938e69695a85c5dcc17b05bc8caf to inject byte code for entry/exit tracing methods.
> > * stores thread id (good) but does nothing to sort/organize activity 
> > by thread id into a session(bad)
> > * no graphing/aggregating of performance metrics...just displays 
> > entry/exit times, method parameters, etc...
> 
> Hi Erik,
> 
> I think that what you are mentioning is certainly one thing that is 
> needed. To me I see a couple of different components here:
> 1) implementation of java.util.logging interfaces as well as SLF4J 
> interfaces, as well as a dedicated LTTng java API for registering 
> custom event typess.  This would be used for static, compile time 
> instrumentation of java apps.
> 2) an implementation that uses the tooling you mention above for 
> dynamic instrumentation of any running java app.  You basically would 
> define trace and install trace points on the fly.  This would be 
> really, really powerful.
> 
> The implementations of the above would have to be quite different from 
> the way that things are done currently with UST.  Currently UST uses 
> macros in C files to define the various tracepoints statically.
>  We would need to have API's that could be called dynamically to 
> define tracepoints, so that the java code could do this.  I have not 
> actually looked under the hood with the current UST implementation to 
> see what the implications are for this.
> 
> It seems to me that we would want to have a Java API usage similar to 
> what you can do with SLF4J:
> 
> class MyClass {
>       private static final Tracer tracer =
>       LTTngFactory.getTracer("MyComponentName");
>       private static final String MY_TRACEPOINT =
>       "MyTracepointWithIntegerStringAndLong";
> 
>       static {
>           // register a tracepoint.  this goes through JNI and
>           creates a tracepoint
>           // in UST that has the given name and the specified objects
>           tracer.registerTracepoint(MY_TRACEPOINT,
>                                      new
>                                      Object[]{Integer.class,String.class,Long.class});
>       }
> 
>       public void someMethod() {
>           Integer someInteger = 123456;
>           String  someString = "this is the string";
>           Long  someLong = 56789;
> 
>           tracer.trace(MY_TRACEPOINT,someInteger,someString,someLong);
>       }
> }
> 
> Then over the top of this you have implementations of SLF4J, 
> java.util.logging, whatever.  The infrastructure required in the JNI 
> side for this would be the same implementation needed by the binding 
> to the InTrace technology to dynamically create tracepoints.
> 
> I need all of this right now, and am willing to spend time in the next 
> few months to implement it.
> 
> cheers,
> Aaron Spear, Staff Engineer, VMware Inc
> 
> _____________
> The information contained in this message is proprietary and/or 
> confidential. If you are not the intended recipient, please: (i) 
> delete the message and all copies; (ii) do not disclose, distribute or 
> use the message in any manner; and (iii) notify the sender 
> immediately. In addition, please be aware that any message addressed 
> to our domain is subject to archiving and review by persons other than 
> the intended recipient. Thank you
> 

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: tracing of non-JNI java in IBM jvm

[Aaron Spear]

----- Original Message -----
> From: "Erik Ostermueller" <Erik.Ostermueller@fisglobal.com>
> To: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> Cc: lttng-dev@lists.lttng.org
> Sent: Friday, January 11, 2013 9:26:51 AM
> Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> Thanks Alex for the quick reply....now I get it.
> I'll jump in on that other thread you mentioned.
> 
> There is a tool called InTrace that does some of this work.
> Perhaps some design ideas or even code could be stolen from here:
>  http://mchr3k.github.com/org.intrace/
> This project:
> * implements the java.lang.instrumentation api, which can be used to
> plug a tracing tool into an already-running JVM.
> * allows you to instrument/de-instrument various classes without
> restarting the JVM.
> * Uses http://asm.ow2.org/ to inject byte code for entry/exit tracing
> methods.
> * stores thread id (good) but does nothing to sort/organize activity
> by thread id into a session(bad)
> * no graphing/aggregating of performance metrics...just displays
> entry/exit times, method parameters, etc...

Hi Erik,

I think that what you are mentioning is certainly one thing that is needed. To me I see a couple of different components here:
1) implementation of java.util.logging interfaces as well as SLF4J interfaces, as well as a dedicated LTTng java API for registering
custom event typess.  This would be used for static, compile time instrumentation of java apps.
2) an implementation that uses the tooling you mention above for dynamic instrumentation of any running java app.  You basically would define trace and install trace points on the fly.  This would be really, really powerful.

The implementations of the above would have to be quite different from the way that things are done currently with UST.  Currently UST uses macros in C files to define the various tracepoints statically.  We would need to have API's that could be called dynamically to define tracepoints, so that the java code could do this.  I have not actually looked under the hood with the current UST implementation to see what the implications are for this.  

It seems to me that we would want to have a Java API usage similar to what you can do with SLF4J:

class MyClass {
      private static final Tracer tracer = LTTngFactory.getTracer("MyComponentName");
      private static final String MY_TRACEPOINT = "MyTracepointWithIntegerStringAndLong";

      static {
          // register a tracepoint.  this goes through JNI and creates a tracepoint
          // in UST that has the given name and the specified objects
          tracer.registerTracepoint(MY_TRACEPOINT,
                                     new Object[]{Integer.class,String.class,Long.class});
      }

      public void someMethod() {
          Integer someInteger = 123456;
          String  someString = "this is the string";
          Long  someLong = 56789;

          tracer.trace(MY_TRACEPOINT,someInteger,someString,someLong);
      }
}

Then over the top of this you have implementations of SLF4J, java.util.logging, whatever.  The infrastructure required in the JNI side for this would be the same implementation needed by the binding to the InTrace technology to dynamically create tracepoints.

I need all of this right now, and am willing to spend time in the next few months to implement it.

cheers,
Aaron Spear, Staff Engineer, VMware Inc

Re: tracing of non-JNI java in IBM jvm

[Ostermueller, Erik]

Aaron,

Sorry to have disappeared from this discussion.
What if we went is a slightly different direction.

The InTrace tool (http://mchr3k.github.com/org.intrace/) will log start/stop times of dynamically instrumented methods to a text file (proprietary format).
What if we simply wrote a process to convert that log to CTF, and view the trace with the lttng eclipse plug-in, unmodified. 

The idea would be to, at first, just support these two views:
http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Control_Flow_View
http://wiki.eclipse.org/Linux_Tools_Project/LTTng/User_Guide#Statistics_View

--Erik



-----Original Message-----
From: Aaron Spear [mailto:aspear@vmware.com] 
Sent: Friday, January 11, 2013 12:56 PM
To: Ostermueller, Erik
Cc: lttng-dev@lists.lttng.org; Alexandre Montplaisir
Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm


----- Original Message -----
> From: "Erik Ostermueller" <Erik.Ostermueller@fisglobal.com>
> To: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> Cc: lttng-dev@lists.lttng.org
> Sent: Friday, January 11, 2013 9:26:51 AM
> Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm
> 
> Thanks Alex for the quick reply....now I get it.
> I'll jump in on that other thread you mentioned.
> 
> There is a tool called InTrace that does some of this work.
> Perhaps some design ideas or even code could be stolen from here:
>  http://mchr3k.github.com/org.intrace/
> This project:
> * implements the java.lang.instrumentation api, which can be used to 
> plug a tracing tool into an already-running JVM.
> * allows you to instrument/de-instrument various classes without 
> restarting the JVM.
> * Uses http://asm.ow2.org/ to inject byte code for entry/exit tracing 
> methods.
> * stores thread id (good) but does nothing to sort/organize activity 
> by thread id into a session(bad)
> * no graphing/aggregating of performance metrics...just displays 
> entry/exit times, method parameters, etc...

Hi Erik,

I think that what you are mentioning is certainly one thing that is needed. To me I see a couple of different components here:
1) implementation of java.util.logging interfaces as well as SLF4J interfaces, as well as a dedicated LTTng java API for registering custom event typess.  This would be used for static, compile time instrumentation of java apps.
2) an implementation that uses the tooling you mention above for dynamic instrumentation of any running java app.  You basically would define trace and install trace points on the fly.  This would be really, really powerful.

The implementations of the above would have to be quite different from the way that things are done currently with UST.  Currently UST uses macros in C files to define the various tracepoints statically.  We would need to have API's that could be called dynamically to define tracepoints, so that the java code could do this.  I have not actually looked under the hood with the current UST implementation to see what the implications are for this.  

It seems to me that we would want to have a Java API usage similar to what you can do with SLF4J:

class MyClass {
      private static final Tracer tracer = LTTngFactory.getTracer("MyComponentName");
      private static final String MY_TRACEPOINT = "MyTracepointWithIntegerStringAndLong";

      static {
          // register a tracepoint.  this goes through JNI and creates a tracepoint
          // in UST that has the given name and the specified objects
          tracer.registerTracepoint(MY_TRACEPOINT,
                                     new Object[]{Integer.class,String.class,Long.class});
      }

      public void someMethod() {
          Integer someInteger = 123456;
          String  someString = "this is the string";
          Long  someLong = 56789;

          tracer.trace(MY_TRACEPOINT,someInteger,someString,someLong);
      }
}

Then over the top of this you have implementations of SLF4J, java.util.logging, whatever.  The infrastructure required in the JNI side for this would be the same implementation needed by the binding to the InTrace technology to dynamically create tracepoints.

I need all of this right now, and am willing to spend time in the next few months to implement it.

cheers,
Aaron Spear, Staff Engineer, VMware Inc

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: tracing of non-JNI java in IBM jvm

[Alexandre Montplaisir]

Hi Erik,

I'm not sure what you mean by "non-JNI java calls". LTTng-UST is a
tracer implemented in C. There are Java bindings available (if you
configure UST with --with-jni-interface), but if you want to use it to
trace Java applications, it will have to go through JNI.

If this is what you want to do, I have an example at
http://git.dorsal.polymtl.ca/~alexmont?p=ust-jni-tests.git;a=summary .
It's a small Java application instrumented with UST JNI tracepoints.

Finally, there is an ongoing project about hooking UST into a JUL
(java.util.logging) handler, and moving more logic like session control
to the Java side of things. It's still at the design stage afaik, but
you might want to monitor this thread:
http://lists.lttng.org/pipermail/lttng-dev/2013-January/019371.html

One more comment below,

On 13-01-10 05:52 PM, Ostermueller, Erik wrote:
> Hi all,
>
> Would like to use LTTng to trace some pure, non-JNI java method calls.
> Seems like I need to
>
> *         install 2.1.0-rc2
>
> *         install the TMF/Eclipse LTTng plugin

The Eclipse plugin does no tracing by itself. It's mainly only a viewer
for LTTng/UST traces. It also has a control feature that allows
controlling a LTTng session daemon from within Eclipse, but it still
requires the lttng tools to be installed on the traced machine.


Cheers,
Alex

>
> *         how do I indicate which classes/methods to be traced?
>
> I started with the "User-space" tracing section this doc:  https://bugs.lttng.org/projects/lttng-tools/wiki
> ...but then I stop understanding things when I read "So, after instrumenting you applications with LTTng-ust 2.0".
> That was followed by a link to http://lttng.org/lttng2.0, but that didn't help.
>
> Would love to be an alpha/beta tester if this isn't ready yet.
>
> Thanks a lot,
> --Erik
>
> ###   some info about my environment:  ####################################
>
>
> =>uname -a
> Linux vlxesngperf01.atldev.com 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
>
> =>cat /etc/issue
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
> Kernel \r on an \m
>
>
> tp@vlxesngperf01 /usr/tp/app/tp/nmon/bin$=>java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build pxa6460sr9fp2ifix-20111111_05(SR9 FP2+IV03622+IV02378+IZ99243+IZ97310+IV00707))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20111111_94827 (JIT enabled, AOT enabled)
> J9VM - 20111111_094827
> JIT  - r9_20101028_17488ifx45
> GC   - 20101027_AA)
> JCL  - 20110727_04
>
>
> ________________________________
> Erik Ostermueller
> Xpress Architecture
>
> T: 501.220.7753
> C:501.213.8309
> erik.ostermueller@fisglobal.com<mailto:erik.ostermueller@fisglobal.com>
> [Description: Description: Description: Description: cid:image001.png@01CD8AB3.84921E40]
> [Description: Description: Description: Description: cid:image003.png@01CD8AB3.84921E40]
>
> _____________
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
>
>
>

Re: tracing of non-JNI java in IBM jvm

[Ostermueller, Erik]

Thanks Alex for the quick reply....now I get it.
I'll jump in on that other thread you mentioned.

There is a tool called InTrace that does some of this work.
Perhaps some design ideas or even code could be stolen from here:  http://mchr3k.github.com/org.intrace/
This project:
* implements the java.lang.instrumentation api, which can be used to plug a tracing tool into an already-running JVM.
* allows you to instrument/de-instrument various classes without restarting the JVM.
* Uses http://asm.ow2.org/ to inject byte code for entry/exit tracing methods.
* stores thread id (good) but does nothing to sort/organize activity by thread id into a session(bad)
* no graphing/aggregating of performance metrics...just displays entry/exit times, method parameters, etc...

--Erik




-----Original Message-----
From: Alexandre Montplaisir [mailto:alexmonthy@voxpopuli.im] 
Sent: Friday, January 11, 2013 9:52 AM
To: Ostermueller, Erik
Cc: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] tracing of non-JNI java in IBM jvm

Hi Erik,

I'm not sure what you mean by "non-JNI java calls". LTTng-UST is a tracer implemented in C. There are Java bindings available (if you configure UST with --with-jni-interface), but if you want to use it to trace Java applications, it will have to go through JNI.

If this is what you want to do, I have an example at http://git.dorsal.polymtl.ca/~alexmont?p=ust-jni-tests.git;a=summary .
It's a small Java application instrumented with UST JNI tracepoints.

Finally, there is an ongoing project about hooking UST into a JUL
(java.util.logging) handler, and moving more logic like session control to the Java side of things. It's still at the design stage afaik, but you might want to monitor this thread:
http://lists.lttng.org/pipermail/lttng-dev/2013-January/019371.html

One more comment below,

On 13-01-10 05:52 PM, Ostermueller, Erik wrote:
> Hi all,
>
> Would like to use LTTng to trace some pure, non-JNI java method calls.
> Seems like I need to
>
> *         install 2.1.0-rc2
>
> *         install the TMF/Eclipse LTTng plugin

The Eclipse plugin does no tracing by itself. It's mainly only a viewer for LTTng/UST traces. It also has a control feature that allows controlling a LTTng session daemon from within Eclipse, but it still requires the lttng tools to be installed on the traced machine.


Cheers,
Alex

>
> *         how do I indicate which classes/methods to be traced?
>
> I started with the "User-space" tracing section this doc:  
> https://bugs.lttng.org/projects/lttng-tools/wiki
> ...but then I stop understanding things when I read "So, after instrumenting you applications with LTTng-ust 2.0".
> That was followed by a link to http://lttng.org/lttng2.0, but that didn't help.
>
> Would love to be an alpha/beta tester if this isn't ready yet.
>
> Thanks a lot,
> --Erik
>
> ###   some info about my environment:  ####################################
>
>
> =>uname -a
> Linux vlxesngperf01.atldev.com 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 
> 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
>
> =>cat /etc/issue
> Red Hat Enterprise Linux Server release 5.6 (Tikanga) Kernel \r on an 
> \m
>
>
> tp@vlxesngperf01 /usr/tp/app/tp/nmon/bin$=>java -version java version 
> "1.6.0"
> Java(TM) SE Runtime Environment (build 
> pxa6460sr9fp2ifix-20111111_05(SR9 
> FP2+IV03622+IV02378+IZ99243+IZ97310+IV00707))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 
> jvmxa6460sr9-20111111_94827 (JIT enabled, AOT enabled) J9VM - 
> 20111111_094827 JIT  - r9_20101028_17488ifx45
> GC   - 20101027_AA)
> JCL  - 20110727_04
>
>
> ________________________________
> Erik Ostermueller
> Xpress Architecture
>
> T: 501.220.7753
> C:501.213.8309
> erik.ostermueller@fisglobal.com<mailto:erik.ostermueller@fisglobal.com
> >
> [Description: Description: Description: Description: 
> cid:image001.png@01CD8AB3.84921E40]
> [Description: Description: Description: Description: 
> cid:image003.png@01CD8AB3.84921E40]
>
> _____________
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
>
>
>

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

tracing of non-JNI java in IBM jvm

[Ostermueller, Erik]

[-- Attachment #1.1.1: Type: text/plain, Size: 2192 bytes --]

Hi all,

Would like to use LTTng to trace some pure, non-JNI java method calls.
Seems like I need to

*         install 2.1.0-rc2

*         install the TMF/Eclipse LTTng plugin

*         how do I indicate which classes/methods to be traced?

I started with the "User-space" tracing section this doc:  https://bugs.lttng.org/projects/lttng-tools/wiki
...but then I stop understanding things when I read "So, after instrumenting you applications with LTTng-ust 2.0".
That was followed by a link to http://lttng.org/lttng2.0, but that didn't help.

Would love to be an alpha/beta tester if this isn't ready yet.

Thanks a lot,
--Erik

###   some info about my environment:  ####################################


=>uname -a
Linux vlxesngperf01.atldev.com 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

=>cat /etc/issue
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Kernel \r on an \m


tp@vlxesngperf01 /usr/tp/app/tp/nmon/bin$=>java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460sr9fp2ifix-20111111_05(SR9 FP2+IV03622+IV02378+IZ99243+IZ97310+IV00707))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20111111_94827 (JIT enabled, AOT enabled)
J9VM - 20111111_094827
JIT  - r9_20101028_17488ifx45
GC   - 20101027_AA)
JCL  - 20110727_04


________________________________
Erik Ostermueller
Xpress Architecture

T: 501.220.7753
C:501.213.8309
erik.ostermueller@fisglobal.com<mailto:erik.ostermueller@fisglobal.com>
[Description: Description: Description: Description: cid:image001.png@01CD8AB3.84921E40]
[Description: Description: Description: Description: cid:image003.png@01CD8AB3.84921E40]

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

[-- Attachment #1.1.2: Type: text/html, Size: 10421 bytes --]

[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 2842 bytes --]

[-- Attachment #1.3: image002.png --]
[-- Type: image/png, Size: 5509 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev