Re: "Hands-free" tracepoints using LD_PRELOAD

Re: "Hands-free" tracepoints using LD_PRELOAD

[Francis Deslauriers]

Hi Brian,
If I understand correctly, you want developers to provide a whitelist
of functions that they want to instrument in an arbitrary shared
library, and then have a system that instruments those functions
without having to recompile the library with LTTng-UST tracepoints.
Can you give us an example of annotation that the shim developer would
provide?
Do you need to support the extraction of function arguments? If you
don't, here are two straightforward and tested approaches to solve
your problem:

First, overriding target symbols using LD_PRELOAD (like you suggested above):
1. Create boilerplate script #1 that generates tracepoints definitions
[1] based on function name,
2. Create boilerplate script #2 that generates a function calling the
right tracepoint and the original target function [2],
3. For each target symbol "bar", using script #1, create a tracepoint
definition "libfoo:bar_tp", and compile them into a tracepoints.o.
object file,
4. For each target symbol "bar", using script #2, create a function
"bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo,
bar_tp) ) and then calls the original bar function in libfoo.so (using
dlsym(3)), and compile those callsites into a callsites.o object file,
5. Create a shared library libshim.so using tracepoints.o and
callsites.o object files and link it to lttng-ust lib and the dynamic
loader lib using :"-llttng-ust -ldl",
Those steps results in the creation of a libshim.so file that contains
the "bar" overriding function (containing the tracepoint callsite)
that will be called instead of "bar" function in the libfoo.so library
when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is
the Makefile I used to test this [3] to give you a sense of the
pipeline.

Second, using the newly added --userspace-probe instrumentation:
This option allows you to instrument any ELF symbol without
recompiling or restarting the target process.
For example, to instrument the "bar" function in the /tmp/libfoo.so
file you could run to following command:
    > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar"
$event_name
You could write a script that takes the list of target functions and
adds userspace-probes on each of them.
This approach has the advantage that you can also instrument functions
in the binary and not only functions in shared libraries.
This instrumentation type uses the kernel tracer and thus requires
using the lttng-modules.

If you do need to save the function arguments, I believe it's possible
using the first approach, by tweaking the two boilerplate scripts
depending on what are the annotations you want to provide.
Supporting argument extraction with userspace-probes is part of our
long term goal and would be a really powerful feature but will require
a significant engineering effort to accomplish.

[1] : https://lttng.org/docs/v2.10/#doc-tpp-header
[2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code
[3]: http://paste.ubuntu.com/p/5fBSgRqXsB/

I hope this make sense. Please don't hesitate if you want more details.

Thank you,
Francis

Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa <br@f0cal.com> a écrit :
>
> Hello,
>
> I have a custom tracing solution that I'm interested in porting to LTTng. It works as follows:
>
> Input is a whitelist of mangled names taken from a target library, libfoo.so.
> For each name, generate a logging shim (example) for the corresponding call.
> Compile all the logging shims into a shared library, libshim.so.
> Trace any application using libfoo by running it as LD_PRELOAD=libshim.so ./fooapp
>
> There are two nice things about this approach that I would like to preserve:
>
> The shim developer only needs to provide the whitelist and a bit of extra annotation. From there, the shim can be be authored using a boilerplate generator.
> The app developer doesn't have to do anything other than pass the LD_PRELOAD flag.
>
> The downside is that the only tracepoints are those corresponding to the whitelist, but I'm fine with that.
>
> Can this kind of "hands-free" developer experience be supported by LTTng?
>
> Thanks!
> ~br
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev



--
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

[-- Attachment #1.1: Type: text/plain, Size: 6957 bytes --]

Francis,

Thanks for your detailed, helpful reply. I was especially excited to learn
about the progress on `--userspace-probe`, but unfortunately I *do* need to
extract the function arguments. It would seem, then, that interposition
through LD_PRELOAD is really the only option. The decision to port to LTTng
or not would just come down to how easy it was to generate all the required
boilerplate.

Here's a little more about my application:

   1. The whitelist is composed of mangled names that the user can extract
   from the target object with the standard `nm` workflow.
   2. Each mangled name in the whitelist gets demangled and parsed into
   `prefix`, `callable_name`, `argument_types`, etc.
   3. The list of `argument_types` is compared to the "extra annotation" I
   alluded to. This extra annotation is just a one-to-one map between argument
   type names and the names of structs that corresponding variables should be
   wrapped with before passing them to the logger. These structs just provide
   custom ostream operators for types that don't log nicely otherwise.
   4. Boilerplate that does the typical `log(...); auto return_val =
   dlsym(...); log(...); return return_val;` gets generated.
   5. `log(...)` is a thin interface to spdlog
   <https://github.com/gabime/spdlog> that handles `__attribute__`-based
   setup and teardown of a logger.

So at the end of the day, the shim developer provides:

   - The whitelist of mangled names
   - Implementations of struct "wrappers" that provide custom ostream
   operators
   - A map between type names and wrapper names

The machinery here seems fairly general-purpose, but I don't presume to be
an expert. My implementation is somewhat error-prone, and my main hope in
reaching out to the mailing list was that LTTng already had some of these
steps better-implemented. Step #2 is particularly problematic due to
ambiguities in the mangling grammar, and will need support going forward to
generalize well.

I would be happy to contribute some or all of my implementation if it's
something that the LTTng community would be interested in supporting and
extending.

Cheers!
~br

On Tue, Jan 22, 2019 at 12:16 PM Francis Deslauriers <
francis.deslauriers@efficios.com> wrote:

> Hi Brian,
> If I understand correctly, you want developers to provide a whitelist
> of functions that they want to instrument in an arbitrary shared
> library, and then have a system that instruments those functions
> without having to recompile the library with LTTng-UST tracepoints.
> Can you give us an example of annotation that the shim developer would
> provide?
> Do you need to support the extraction of function arguments? If you
> don't, here are two straightforward and tested approaches to solve
> your problem:
>
> First, overriding target symbols using LD_PRELOAD (like you suggested
> above):
> 1. Create boilerplate script #1 that generates tracepoints definitions
> [1] based on function name,
> 2. Create boilerplate script #2 that generates a function calling the
> right tracepoint and the original target function [2],
> 3. For each target symbol "bar", using script #1, create a tracepoint
> definition "libfoo:bar_tp", and compile them into a tracepoints.o.
> object file,
> 4. For each target symbol "bar", using script #2, create a function
> "bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo,
> bar_tp) ) and then calls the original bar function in libfoo.so (using
> dlsym(3)), and compile those callsites into a callsites.o object file,
> 5. Create a shared library libshim.so using tracepoints.o and
> callsites.o object files and link it to lttng-ust lib and the dynamic
> loader lib using :"-llttng-ust -ldl",
> Those steps results in the creation of a libshim.so file that contains
> the "bar" overriding function (containing the tracepoint callsite)
> that will be called instead of "bar" function in the libfoo.so library
> when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is
> the Makefile I used to test this [3] to give you a sense of the
> pipeline.
>
> Second, using the newly added --userspace-probe instrumentation:
> This option allows you to instrument any ELF symbol without
> recompiling or restarting the target process.
> For example, to instrument the "bar" function in the /tmp/libfoo.so
> file you could run to following command:
>     > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar"
> $event_name
> You could write a script that takes the list of target functions and
> adds userspace-probes on each of them.
> This approach has the advantage that you can also instrument functions
> in the binary and not only functions in shared libraries.
> This instrumentation type uses the kernel tracer and thus requires
> using the lttng-modules.
>
> If you do need to save the function arguments, I believe it's possible
> using the first approach, by tweaking the two boilerplate scripts
> depending on what are the annotations you want to provide.
> Supporting argument extraction with userspace-probes is part of our
> long term goal and would be a really powerful feature but will require
> a significant engineering effort to accomplish.
>
> [1] : https://lttng.org/docs/v2.10/#doc-tpp-header
> [2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code
> [3]: http://paste.ubuntu.com/p/5fBSgRqXsB/
>
> I hope this make sense. Please don't hesitate if you want more details.
>
> Thank you,
> Francis
>
> Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa <br@f0cal.com> a écrit :
> >
> > Hello,
> >
> > I have a custom tracing solution that I'm interested in porting to
> LTTng. It works as follows:
> >
> > Input is a whitelist of mangled names taken from a target library,
> libfoo.so.
> > For each name, generate a logging shim (example) for the corresponding
> call.
> > Compile all the logging shims into a shared library, libshim.so.
> > Trace any application using libfoo by running it as
> LD_PRELOAD=libshim.so ./fooapp
> >
> > There are two nice things about this approach that I would like to
> preserve:
> >
> > The shim developer only needs to provide the whitelist and a bit of
> extra annotation. From there, the shim can be be authored using a
> boilerplate generator.
> > The app developer doesn't have to do anything other than pass the
> LD_PRELOAD flag.
> >
> > The downside is that the only tracepoints are those corresponding to the
> whitelist, but I'm fine with that.
> >
> > Can this kind of "hands-free" developer experience be supported by LTTng?
> >
> > Thanks!
> > ~br
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>

[-- Attachment #1.2: Type: text/html, Size: 8299 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Jonathan Rajotte-Julien]

Hi Brian,

On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
>    4. Boilerplate that does the typical `log(...); auto return_val =
>    dlsym(...); log(...); return return_val;` gets generated.

As proposed by Francis, this is when you need to "generate" a corresponding
tracepoint definition and call the tracepoint() call with the appropriate
arguments.

As Francis demonstrated we do not see any reason for lttng-ust not to work here
given that you compile the libshim object correctly.

>    5. `log(...)` is a thin interface to spdlog
>    <https://github.com/gabime/spdlog> that handles `__attribute__`-based
>    setup and teardown of a logger.
> 
> So at the end of the day, the shim developer provides:
> 
>    - The whitelist of mangled names
>    - Implementations of struct "wrappers" that provide custom ostream
>    operators
>    - A map between type names and wrapper names
> 
> The machinery here seems fairly general-purpose, but I don't presume to be
> an expert. My implementation is somewhat error-prone, and my main hope in
> reaching out to the mailing list was that LTTng already had some of these
> steps better-implemented.

AFAIK, lttng does not have an equivalent.

> Step #2 is particularly problematic due to
> ambiguities in the mangling grammar, and will need support going forward to
> generalize well.

What is the status of this step in your project?

What are the problems that make your implementation "error-pone"?

Would you mind linking us to said project so we can have a look?

I would be interested in seeing at first lttng tracepoint used as Francis
demonstrated and see from there were this project can go.

> 
> I would be happy to contribute some or all of my implementation if it's
> something that the LTTng community would be interested in supporting and
> extending.

We are clearly open for discussion and helping you improve the project. I am not
so sure on supporting and extending it. Others might have a different opinion.


Cheers

--
Jonathan Rajotte-Julien
EfficiOS

Re: "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

[-- Attachment #1.1: Type: text/plain, Size: 3936 bytes --]

Jonathan,

Responses below:

AFAIK, lttng does not have an equivalent.
>

I believe my code could significantly reduce the need for hand-writing the
tracepoints. But I won't likely take on a port to LTTng immediately, as a
"vanilla" interposition approach seems to be meeting my requirements.


> > Step #2 is particularly problematic due to
> > ambiguities in the mangling grammar, and will need support going forward
> to
> > generalize well.
>
> What is the status of this step in your project?
>

I demangle the symbol name using c++filt and then use regex to extract the
list of argument types. Using something like ItaniumPartialDemangler would
be better.


> What are the problems that make your implementation "error-pone"?
>

Use of regex as above.


> Would you mind linking us to said project so we can have a look?
>

It's a project for my employer, and I would need to keep the source closed
until I can get an OK. A signal of interest from LTTng would be helpful
there. In the meantime, I think it would be fine to share as read-only with
the LTTng maintainers if someone wants to send me a Github username.


> I would be interested in seeing at first lttng tracepoint used as Francis
> demonstrated and see from there were this project can go.
>
...

> > I would be happy to contribute some or all of my implementation if it's
> > something that the LTTng community would be interested in supporting and
> > extending.
>
> We are clearly open for discussion and helping you improve the project. I
> am not
> so sure on supporting and extending it. Others might have a different
> opinion.
>

Sounds good. Looking forward to connecting via personal email.

Cheers!
~br


On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi Brian,
>
> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
> >    4. Boilerplate that does the typical `log(...); auto return_val =
> >    dlsym(...); log(...); return return_val;` gets generated.
>
> As proposed by Francis, this is when you need to "generate" a corresponding
> tracepoint definition and call the tracepoint() call with the appropriate
> arguments.
>
> As Francis demonstrated we do not see any reason for lttng-ust not to work
> here
> given that you compile the libshim object correctly.
>
> >    5. `log(...)` is a thin interface to spdlog
> >    <https://github.com/gabime/spdlog> that handles `__attribute__`-based
> >    setup and teardown of a logger.
> >
> > So at the end of the day, the shim developer provides:
> >
> >    - The whitelist of mangled names
> >    - Implementations of struct "wrappers" that provide custom ostream
> >    operators
> >    - A map between type names and wrapper names
> >
> > The machinery here seems fairly general-purpose, but I don't presume to
> be
> > an expert. My implementation is somewhat error-prone, and my main hope in
> > reaching out to the mailing list was that LTTng already had some of these
> > steps better-implemented.
>
> AFAIK, lttng does not have an equivalent.
>
> > Step #2 is particularly problematic due to
> > ambiguities in the mangling grammar, and will need support going forward
> to
> > generalize well.
>
> What is the status of this step in your project?
>
> What are the problems that make your implementation "error-pone"?
>
> Would you mind linking us to said project so we can have a look?
>
> I would be interested in seeing at first lttng tracepoint used as Francis
> demonstrated and see from there were this project can go.
>
> >
> > I would be happy to contribute some or all of my implementation if it's
> > something that the LTTng community would be interested in supporting and
> > extending.
>
> We are clearly open for discussion and helping you improve the project. I
> am not
> so sure on supporting and extending it. Others might have a different
> opinion.
>
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>

[-- Attachment #1.2: Type: text/html, Size: 5860 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Francis Deslauriers]

"

Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br@f0cal.com> a écrit :
>
> Jonathan,
>
> Responses below:
>
>> AFAIK, lttng does not have an equivalent.
>
>
> I believe my code could significantly reduce the need for hand-writing the tracepoints. But I won't likely take on a port to LTTng immediately, as a "vanilla" interposition approach seems to be meeting my requirements.
>
>>
>> > Step #2 is particularly problematic due to
>> > ambiguities in the mangling grammar, and will need support going forward to
>> > generalize well.
>>
>> What is the status of this step in your project?
>
>
> I demangle the symbol name using c++filt and then use regex to extract the list of argument types. Using something like ItaniumPartialDemangler would be better.

Hi Brian,

If you have a list of the argument types, you can programmatically
generate the tracepoint descriptions and callsites accordingly. The
main blocker I see here is tracing arguments that are pointers to
classes or containers. We need to be able to map each argument with a
CTF type [1]. It's easy enough for int, float and char * but it's
harder for complex structs and data structures.

You mentioned that you wrap structures with ostreams to output them in
text format. Can you explain this a bit more?

Here are a few ideas:

#First idea
If you are already defining the printing format and order of each of
the fields of each structures in your libfoo.so maybe you could do the
same but in LTTng-UST format. See "my-custom-structure.h" example [2].

#Second idea:
If you prefer not convert those ostreams wrappers to ctf wrapper, you
could reuse them to generate CTF_STRINGs.
1. Simple data types (int, float, char*) are mapped directly to CTF types,
2. Complex data types are wrapped with ostream function,
3. Complex data types are saved in the trace as CTF_STRING using the ostream.
All this could be done by the boilerplate scripts I mentioned earlier.
By using string format for some argument, you don't get the full power
of LTTng but it will still be faster that saving everything in text.

#Third idea:
Do you know of tracef() [3] ? Using it, you can save any string to a
UST trace. As a first step, you could directly replace your calls to
spdlog by calls to tracef. It's an highly inefficient way of using
LTTng, but it works (and probably lower overhead than writing to a
file).

[1]: https://lttng.org/man/3/lttng-ust/v2.10/
[2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
[3]: https://lttng.org/docs/v2.10/#doc-tracef

Thank you,
Francis

>
>>
>> What are the problems that make your implementation "error-pone"?
>
>
> Use of regex as above.
>
>>
>> Would you mind linking us to said project so we can have a look?
>
>
> It's a project for my employer, and I would need to keep the source closed until I can get an OK. A signal of interest from LTTng would be helpful there. In the meantime, I think it would be fine to share as read-only with the LTTng maintainers if someone wants to send me a Github username.
>
>>
>> I would be interested in seeing at first lttng tracepoint used as Francis
>> demonstrated and see from there were this project can go.
>
> ...
>>
>> > I would be happy to contribute some or all of my implementation if it's
>> > something that the LTTng community would be interested in supporting and
>> > extending.
>>
>> We are clearly open for discussion and helping you improve the project. I am not
>> so sure on supporting and extending it. Others might have a different opinion.
>
>
> Sounds good. Looking forward to connecting via personal email.
>
> Cheers!
> ~br
>
>
> On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> wrote:
>>
>> Hi Brian,
>>
>> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
>> >    4. Boilerplate that does the typical `log(...); auto return_val =
>> >    dlsym(...); log(...); return return_val;` gets generated.
>>
>> As proposed by Francis, this is when you need to "generate" a corresponding
>> tracepoint definition and call the tracepoint() call with the appropriate
>> arguments.
>>
>> As Francis demonstrated we do not see any reason for lttng-ust not to work here
>> given that you compile the libshim object correctly.
>>
>> >    5. `log(...)` is a thin interface to spdlog
>> >    <https://github.com/gabime/spdlog> that handles `__attribute__`-based
>> >    setup and teardown of a logger.
>> >
>> > So at the end of the day, the shim developer provides:
>> >
>> >    - The whitelist of mangled names
>> >    - Implementations of struct "wrappers" that provide custom ostream
>> >    operators
>> >    - A map between type names and wrapper names
>> >
>> > The machinery here seems fairly general-purpose, but I don't presume to be
>> > an expert. My implementation is somewhat error-prone, and my main hope in
>> > reaching out to the mailing list was that LTTng already had some of these
>> > steps better-implemented.
>>
>> AFAIK, lttng does not have an equivalent.
>>
>> > Step #2 is particularly problematic due to
>> > ambiguities in the mangling grammar, and will need support going forward to
>> > generalize well.
>>
>> What is the status of this step in your project?
>>
>> What are the problems that make your implementation "error-pone"?
>>
>> Would you mind linking us to said project so we can have a look?
>>
>> I would be interested in seeing at first lttng tracepoint used as Francis
>> demonstrated and see from there were this project can go.
>>
>> >
>> > I would be happy to contribute some or all of my implementation if it's
>> > something that the LTTng community would be interested in supporting and
>> > extending.
>>
>> We are clearly open for discussion and helping you improve the project. I am not
>> so sure on supporting and extending it. Others might have a different opinion.
>>
>>
>> Cheers
>>
>> --
>> Jonathan Rajotte-Julien
>> EfficiOS



-- 
Francis Deslauriers
Computer Engineer
EfficiOS inc.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

[-- Attachment #1.1: Type: text/plain, Size: 7797 bytes --]

Francis,

These are great suggestions, thanks!

#Third idea:
> Do you know of tracef() [3] ? Using it, you can save any string to a
> UST trace. As a first step, you could directly replace your calls to
> spdlog by calls to tracef. It's an highly inefficient way of using
> LTTng, but it works (and probably lower overhead than writing to a
> file).
>

Replacing spdlog::debug(...) with tracef(...) may be an easy way for me to
get familiar with LTTNg workflows without having to go through a complete
port.

This brings up an interesting question, though, and the answer may motivate
me to close the gap further: What kind of latency reduction can I expect
from moving from spdlog to LLTng? I know this is ill-posed without knowing
more about how many pseudo-tracepoints I'm implementing, the log message
sizes that I'm pushing to disk, etc, etc., but something notional would
help my motivation.


> You mentioned that you wrap structures with ostreams to output them in
> text format. Can you explain this a bit more?
>

I'm just implementing what spdlog suggests:
https://github.com/gabime/spdlog#user-defined-types

Cheers!
~br

On Thu, Jan 24, 2019 at 11:06 AM Francis Deslauriers <
francis.deslauriers@efficios.com> wrote:

> "
>
> Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br@f0cal.com> a écrit :
> >
> > Jonathan,
> >
> > Responses below:
> >
> >> AFAIK, lttng does not have an equivalent.
> >
> >
> > I believe my code could significantly reduce the need for hand-writing
> the tracepoints. But I won't likely take on a port to LTTng immediately, as
> a "vanilla" interposition approach seems to be meeting my requirements.
> >
> >>
> >> > Step #2 is particularly problematic due to
> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> > generalize well.
> >>
> >> What is the status of this step in your project?
> >
> >
> > I demangle the symbol name using c++filt and then use regex to extract
> the list of argument types. Using something like ItaniumPartialDemangler
> would be better.
>
> Hi Brian,
>
> If you have a list of the argument types, you can programmatically
> generate the tracepoint descriptions and callsites accordingly. The
> main blocker I see here is tracing arguments that are pointers to
> classes or containers. We need to be able to map each argument with a
> CTF type [1]. It's easy enough for int, float and char * but it's
> harder for complex structs and data structures.
>
> You mentioned that you wrap structures with ostreams to output them in
> text format. Can you explain this a bit more?
>
> Here are a few ideas:
>
> #First idea
> If you are already defining the printing format and order of each of
> the fields of each structures in your libfoo.so maybe you could do the
> same but in LTTng-UST format. See "my-custom-structure.h" example [2].
>
> #Second idea:
> If you prefer not convert those ostreams wrappers to ctf wrapper, you
> could reuse them to generate CTF_STRINGs.
> 1. Simple data types (int, float, char*) are mapped directly to CTF types,
> 2. Complex data types are wrapped with ostream function,
> 3. Complex data types are saved in the trace as CTF_STRING using the
> ostream.
> All this could be done by the boilerplate scripts I mentioned earlier.
> By using string format for some argument, you don't get the full power
> of LTTng but it will still be faster that saving everything in text.
>
> #Third idea:
> Do you know of tracef() [3] ? Using it, you can save any string to a
> UST trace. As a first step, you could directly replace your calls to
> spdlog by calls to tracef. It's an highly inefficient way of using
> LTTng, but it works (and probably lower overhead than writing to a
> file).
>
> [1]: https://lttng.org/man/3/lttng-ust/v2.10/
> [2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
> [3]: https://lttng.org/docs/v2.10/#doc-tracef
>
> Thank you,
> Francis
>
> >
> >>
> >> What are the problems that make your implementation "error-pone"?
> >
> >
> > Use of regex as above.
> >
> >>
> >> Would you mind linking us to said project so we can have a look?
> >
> >
> > It's a project for my employer, and I would need to keep the source
> closed until I can get an OK. A signal of interest from LTTng would be
> helpful there. In the meantime, I think it would be fine to share as
> read-only with the LTTng maintainers if someone wants to send me a Github
> username.
> >
> >>
> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> demonstrated and see from there were this project can go.
> >
> > ...
> >>
> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> > something that the LTTng community would be interested in supporting
> and
> >> > extending.
> >>
> >> We are clearly open for discussion and helping you improve the project.
> I am not
> >> so sure on supporting and extending it. Others might have a different
> opinion.
> >
> >
> > Sounds good. Looking forward to connecting via personal email.
> >
> > Cheers!
> > ~br
> >
> >
> > On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <
> jonathan.rajotte-julien@efficios.com> wrote:
> >>
> >> Hi Brian,
> >>
> >> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
> >> >    4. Boilerplate that does the typical `log(...); auto return_val =
> >> >    dlsym(...); log(...); return return_val;` gets generated.
> >>
> >> As proposed by Francis, this is when you need to "generate" a
> corresponding
> >> tracepoint definition and call the tracepoint() call with the
> appropriate
> >> arguments.
> >>
> >> As Francis demonstrated we do not see any reason for lttng-ust not to
> work here
> >> given that you compile the libshim object correctly.
> >>
> >> >    5. `log(...)` is a thin interface to spdlog
> >> >    <https://github.com/gabime/spdlog> that handles
> `__attribute__`-based
> >> >    setup and teardown of a logger.
> >> >
> >> > So at the end of the day, the shim developer provides:
> >> >
> >> >    - The whitelist of mangled names
> >> >    - Implementations of struct "wrappers" that provide custom ostream
> >> >    operators
> >> >    - A map between type names and wrapper names
> >> >
> >> > The machinery here seems fairly general-purpose, but I don't presume
> to be
> >> > an expert. My implementation is somewhat error-prone, and my main
> hope in
> >> > reaching out to the mailing list was that LTTng already had some of
> these
> >> > steps better-implemented.
> >>
> >> AFAIK, lttng does not have an equivalent.
> >>
> >> > Step #2 is particularly problematic due to
> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> > generalize well.
> >>
> >> What is the status of this step in your project?
> >>
> >> What are the problems that make your implementation "error-pone"?
> >>
> >> Would you mind linking us to said project so we can have a look?
> >>
> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> demonstrated and see from there were this project can go.
> >>
> >> >
> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> > something that the LTTng community would be interested in supporting
> and
> >> > extending.
> >>
> >> We are clearly open for discussion and helping you improve the project.
> I am not
> >> so sure on supporting and extending it. Others might have a different
> opinion.
> >>
> >>
> >> Cheers
> >>
> >> --
> >> Jonathan Rajotte-Julien
> >> EfficiOS
>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>

[-- Attachment #1.2: Type: text/html, Size: 10431 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Philippe Proulx]

On Fri, Jan 25, 2019 at 4:05 AM Brian Rossa <br@f0cal.com> wrote:
>
> Francis,
>
> These are great suggestions, thanks!
>
>> #Third idea:
>> Do you know of tracef() [3] ? Using it, you can save any string to a
>> UST trace. As a first step, you could directly replace your calls to
>> spdlog by calls to tracef. It's an highly inefficient way of using
>> LTTng, but it works (and probably lower overhead than writing to a
>> file).
>
>
> Replacing spdlog::debug(...) with tracef(...) may be an easy way for me to get familiar with LTTNg workflows without having to go through a complete port.

However please note that spdlog uses {fmt} while tracef() uses a
vsnprintf()-family function, so you would need to adapt the format
strings too.

Phil

>
> This brings up an interesting question, though, and the answer may motivate me to close the gap further: What kind of latency reduction can I expect from moving from spdlog to LLTng? I know this is ill-posed without knowing more about how many pseudo-tracepoints I'm implementing, the log message sizes that I'm pushing to disk, etc, etc., but something notional would help my motivation.
>
>>
>> You mentioned that you wrap structures with ostreams to output them in
>> text format. Can you explain this a bit more?
>
>
> I'm just implementing what spdlog suggests: https://github.com/gabime/spdlog#user-defined-types
>
> Cheers!
> ~br
>
> On Thu, Jan 24, 2019 at 11:06 AM Francis Deslauriers <francis.deslauriers@efficios.com> wrote:
>>
>> "
>>
>> Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br@f0cal.com> a écrit :
>> >
>> > Jonathan,
>> >
>> > Responses below:
>> >
>> >> AFAIK, lttng does not have an equivalent.
>> >
>> >
>> > I believe my code could significantly reduce the need for hand-writing the tracepoints. But I won't likely take on a port to LTTng immediately, as a "vanilla" interposition approach seems to be meeting my requirements.
>> >
>> >>
>> >> > Step #2 is particularly problematic due to
>> >> > ambiguities in the mangling grammar, and will need support going forward to
>> >> > generalize well.
>> >>
>> >> What is the status of this step in your project?
>> >
>> >
>> > I demangle the symbol name using c++filt and then use regex to extract the list of argument types. Using something like ItaniumPartialDemangler would be better.
>>
>> Hi Brian,
>>
>> If you have a list of the argument types, you can programmatically
>> generate the tracepoint descriptions and callsites accordingly. The
>> main blocker I see here is tracing arguments that are pointers to
>> classes or containers. We need to be able to map each argument with a
>> CTF type [1]. It's easy enough for int, float and char * but it's
>> harder for complex structs and data structures.
>>
>> You mentioned that you wrap structures with ostreams to output them in
>> text format. Can you explain this a bit more?
>>
>> Here are a few ideas:
>>
>> #First idea
>> If you are already defining the printing format and order of each of
>> the fields of each structures in your libfoo.so maybe you could do the
>> same but in LTTng-UST format. See "my-custom-structure.h" example [2].
>>
>> #Second idea:
>> If you prefer not convert those ostreams wrappers to ctf wrapper, you
>> could reuse them to generate CTF_STRINGs.
>> 1. Simple data types (int, float, char*) are mapped directly to CTF types,
>> 2. Complex data types are wrapped with ostream function,
>> 3. Complex data types are saved in the trace as CTF_STRING using the ostream.
>> All this could be done by the boilerplate scripts I mentioned earlier.
>> By using string format for some argument, you don't get the full power
>> of LTTng but it will still be faster that saving everything in text.
>>
>> #Third idea:
>> Do you know of tracef() [3] ? Using it, you can save any string to a
>> UST trace. As a first step, you could directly replace your calls to
>> spdlog by calls to tracef. It's an highly inefficient way of using
>> LTTng, but it works (and probably lower overhead than writing to a
>> file).
>>
>> [1]: https://lttng.org/man/3/lttng-ust/v2.10/
>> [2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
>> [3]: https://lttng.org/docs/v2.10/#doc-tracef
>>
>> Thank you,
>> Francis
>>
>> >
>> >>
>> >> What are the problems that make your implementation "error-pone"?
>> >
>> >
>> > Use of regex as above.
>> >
>> >>
>> >> Would you mind linking us to said project so we can have a look?
>> >
>> >
>> > It's a project for my employer, and I would need to keep the source closed until I can get an OK. A signal of interest from LTTng would be helpful there. In the meantime, I think it would be fine to share as read-only with the LTTng maintainers if someone wants to send me a Github username.
>> >
>> >>
>> >> I would be interested in seeing at first lttng tracepoint used as Francis
>> >> demonstrated and see from there were this project can go.
>> >
>> > ...
>> >>
>> >> > I would be happy to contribute some or all of my implementation if it's
>> >> > something that the LTTng community would be interested in supporting and
>> >> > extending.
>> >>
>> >> We are clearly open for discussion and helping you improve the project. I am not
>> >> so sure on supporting and extending it. Others might have a different opinion.
>> >
>> >
>> > Sounds good. Looking forward to connecting via personal email.
>> >
>> > Cheers!
>> > ~br
>> >
>> >
>> > On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> wrote:
>> >>
>> >> Hi Brian,
>> >>
>> >> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
>> >> >    4. Boilerplate that does the typical `log(...); auto return_val =
>> >> >    dlsym(...); log(...); return return_val;` gets generated.
>> >>
>> >> As proposed by Francis, this is when you need to "generate" a corresponding
>> >> tracepoint definition and call the tracepoint() call with the appropriate
>> >> arguments.
>> >>
>> >> As Francis demonstrated we do not see any reason for lttng-ust not to work here
>> >> given that you compile the libshim object correctly.
>> >>
>> >> >    5. `log(...)` is a thin interface to spdlog
>> >> >    <https://github.com/gabime/spdlog> that handles `__attribute__`-based
>> >> >    setup and teardown of a logger.
>> >> >
>> >> > So at the end of the day, the shim developer provides:
>> >> >
>> >> >    - The whitelist of mangled names
>> >> >    - Implementations of struct "wrappers" that provide custom ostream
>> >> >    operators
>> >> >    - A map between type names and wrapper names
>> >> >
>> >> > The machinery here seems fairly general-purpose, but I don't presume to be
>> >> > an expert. My implementation is somewhat error-prone, and my main hope in
>> >> > reaching out to the mailing list was that LTTng already had some of these
>> >> > steps better-implemented.
>> >>
>> >> AFAIK, lttng does not have an equivalent.
>> >>
>> >> > Step #2 is particularly problematic due to
>> >> > ambiguities in the mangling grammar, and will need support going forward to
>> >> > generalize well.
>> >>
>> >> What is the status of this step in your project?
>> >>
>> >> What are the problems that make your implementation "error-pone"?
>> >>
>> >> Would you mind linking us to said project so we can have a look?
>> >>
>> >> I would be interested in seeing at first lttng tracepoint used as Francis
>> >> demonstrated and see from there were this project can go.
>> >>
>> >> >
>> >> > I would be happy to contribute some or all of my implementation if it's
>> >> > something that the LTTng community would be interested in supporting and
>> >> > extending.
>> >>
>> >> We are clearly open for discussion and helping you improve the project. I am not
>> >> so sure on supporting and extending it. Others might have a different opinion.
>> >>
>> >>
>> >> Cheers
>> >>
>> >> --
>> >> Jonathan Rajotte-Julien
>> >> EfficiOS
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

[-- Attachment #1.1: Type: text/plain, Size: 9331 bytes --]

Ok, after further review and some performance analysis on spdlog, I'm
considering skipping the tracef(...) step and jumping to a tracepoint-based
approach. I believe I can use code generation to create both (1) the
tracepoint definitions and (2) function interposition (LD_PRELOAD) shims
calling to #1. This may take some time, but I'll ping this thread if I
encounter any issues. Thanks! ~br

On Fri, Jan 25, 2019 at 9:35 AM Philippe Proulx <eeppeliteloop@gmail.com>
wrote:

> On Fri, Jan 25, 2019 at 4:05 AM Brian Rossa <br@f0cal.com> wrote:
> >
> > Francis,
> >
> > These are great suggestions, thanks!
> >
> >> #Third idea:
> >> Do you know of tracef() [3] ? Using it, you can save any string to a
> >> UST trace. As a first step, you could directly replace your calls to
> >> spdlog by calls to tracef. It's an highly inefficient way of using
> >> LTTng, but it works (and probably lower overhead than writing to a
> >> file).
> >
> >
> > Replacing spdlog::debug(...) with tracef(...) may be an easy way for me
> to get familiar with LTTNg workflows without having to go through a
> complete port.
>
> However please note that spdlog uses {fmt} while tracef() uses a
> vsnprintf()-family function, so you would need to adapt the format
> strings too.
>
> Phil
>
> >
> > This brings up an interesting question, though, and the answer may
> motivate me to close the gap further: What kind of latency reduction can I
> expect from moving from spdlog to LLTng? I know this is ill-posed without
> knowing more about how many pseudo-tracepoints I'm implementing, the log
> message sizes that I'm pushing to disk, etc, etc., but something notional
> would help my motivation.
> >
> >>
> >> You mentioned that you wrap structures with ostreams to output them in
> >> text format. Can you explain this a bit more?
> >
> >
> > I'm just implementing what spdlog suggests:
> https://github.com/gabime/spdlog#user-defined-types
> >
> > Cheers!
> > ~br
> >
> > On Thu, Jan 24, 2019 at 11:06 AM Francis Deslauriers <
> francis.deslauriers@efficios.com> wrote:
> >>
> >> "
> >>
> >> Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br@f0cal.com> a écrit :
> >> >
> >> > Jonathan,
> >> >
> >> > Responses below:
> >> >
> >> >> AFAIK, lttng does not have an equivalent.
> >> >
> >> >
> >> > I believe my code could significantly reduce the need for
> hand-writing the tracepoints. But I won't likely take on a port to LTTng
> immediately, as a "vanilla" interposition approach seems to be meeting my
> requirements.
> >> >
> >> >>
> >> >> > Step #2 is particularly problematic due to
> >> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> >> > generalize well.
> >> >>
> >> >> What is the status of this step in your project?
> >> >
> >> >
> >> > I demangle the symbol name using c++filt and then use regex to
> extract the list of argument types. Using something like
> ItaniumPartialDemangler would be better.
> >>
> >> Hi Brian,
> >>
> >> If you have a list of the argument types, you can programmatically
> >> generate the tracepoint descriptions and callsites accordingly. The
> >> main blocker I see here is tracing arguments that are pointers to
> >> classes or containers. We need to be able to map each argument with a
> >> CTF type [1]. It's easy enough for int, float and char * but it's
> >> harder for complex structs and data structures.
> >>
> >> You mentioned that you wrap structures with ostreams to output them in
> >> text format. Can you explain this a bit more?
> >>
> >> Here are a few ideas:
> >>
> >> #First idea
> >> If you are already defining the printing format and order of each of
> >> the fields of each structures in your libfoo.so maybe you could do the
> >> same but in LTTng-UST format. See "my-custom-structure.h" example [2].
> >>
> >> #Second idea:
> >> If you prefer not convert those ostreams wrappers to ctf wrapper, you
> >> could reuse them to generate CTF_STRINGs.
> >> 1. Simple data types (int, float, char*) are mapped directly to CTF
> types,
> >> 2. Complex data types are wrapped with ostream function,
> >> 3. Complex data types are saved in the trace as CTF_STRING using the
> ostream.
> >> All this could be done by the boilerplate scripts I mentioned earlier.
> >> By using string format for some argument, you don't get the full power
> >> of LTTng but it will still be faster that saving everything in text.
> >>
> >> #Third idea:
> >> Do you know of tracef() [3] ? Using it, you can save any string to a
> >> UST trace. As a first step, you could directly replace your calls to
> >> spdlog by calls to tracef. It's an highly inefficient way of using
> >> LTTng, but it works (and probably lower overhead than writing to a
> >> file).
> >>
> >> [1]: https://lttng.org/man/3/lttng-ust/v2.10/
> >> [2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
> >> [3]: https://lttng.org/docs/v2.10/#doc-tracef
> >>
> >> Thank you,
> >> Francis
> >>
> >> >
> >> >>
> >> >> What are the problems that make your implementation "error-pone"?
> >> >
> >> >
> >> > Use of regex as above.
> >> >
> >> >>
> >> >> Would you mind linking us to said project so we can have a look?
> >> >
> >> >
> >> > It's a project for my employer, and I would need to keep the source
> closed until I can get an OK. A signal of interest from LTTng would be
> helpful there. In the meantime, I think it would be fine to share as
> read-only with the LTTng maintainers if someone wants to send me a Github
> username.
> >> >
> >> >>
> >> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> >> demonstrated and see from there were this project can go.
> >> >
> >> > ...
> >> >>
> >> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> >> > something that the LTTng community would be interested in
> supporting and
> >> >> > extending.
> >> >>
> >> >> We are clearly open for discussion and helping you improve the
> project. I am not
> >> >> so sure on supporting and extending it. Others might have a
> different opinion.
> >> >
> >> >
> >> > Sounds good. Looking forward to connecting via personal email.
> >> >
> >> > Cheers!
> >> > ~br
> >> >
> >> >
> >> > On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <
> jonathan.rajotte-julien@efficios.com> wrote:
> >> >>
> >> >> Hi Brian,
> >> >>
> >> >> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
> >> >> >    4. Boilerplate that does the typical `log(...); auto return_val
> =
> >> >> >    dlsym(...); log(...); return return_val;` gets generated.
> >> >>
> >> >> As proposed by Francis, this is when you need to "generate" a
> corresponding
> >> >> tracepoint definition and call the tracepoint() call with the
> appropriate
> >> >> arguments.
> >> >>
> >> >> As Francis demonstrated we do not see any reason for lttng-ust not
> to work here
> >> >> given that you compile the libshim object correctly.
> >> >>
> >> >> >    5. `log(...)` is a thin interface to spdlog
> >> >> >    <https://github.com/gabime/spdlog> that handles
> `__attribute__`-based
> >> >> >    setup and teardown of a logger.
> >> >> >
> >> >> > So at the end of the day, the shim developer provides:
> >> >> >
> >> >> >    - The whitelist of mangled names
> >> >> >    - Implementations of struct "wrappers" that provide custom
> ostream
> >> >> >    operators
> >> >> >    - A map between type names and wrapper names
> >> >> >
> >> >> > The machinery here seems fairly general-purpose, but I don't
> presume to be
> >> >> > an expert. My implementation is somewhat error-prone, and my main
> hope in
> >> >> > reaching out to the mailing list was that LTTng already had some
> of these
> >> >> > steps better-implemented.
> >> >>
> >> >> AFAIK, lttng does not have an equivalent.
> >> >>
> >> >> > Step #2 is particularly problematic due to
> >> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> >> > generalize well.
> >> >>
> >> >> What is the status of this step in your project?
> >> >>
> >> >> What are the problems that make your implementation "error-pone"?
> >> >>
> >> >> Would you mind linking us to said project so we can have a look?
> >> >>
> >> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> >> demonstrated and see from there were this project can go.
> >> >>
> >> >> >
> >> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> >> > something that the LTTng community would be interested in
> supporting and
> >> >> > extending.
> >> >>
> >> >> We are clearly open for discussion and helping you improve the
> project. I am not
> >> >> so sure on supporting and extending it. Others might have a
> different opinion.
> >> >>
> >> >>
> >> >> Cheers
> >> >>
> >> >> --
> >> >> Jonathan Rajotte-Julien
> >> >> EfficiOS
> >>
> >>
> >>
> >> --
> >> Francis Deslauriers
> >> Computer Engineer
> >> EfficiOS inc.
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

[-- Attachment #1.2: Type: text/html, Size: 12725 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

"Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

[-- Attachment #1.1: Type: text/plain, Size: 1019 bytes --]

Hello,

I have a custom tracing solution that I'm interested in porting to LTTng.
It works as follows:

   1. Input is a whitelist of mangled names taken from a target library,
   libfoo.so.
   2. For each name, generate a logging shim (example
   <https://www.linuxjournal.com/article/7795>) for the corresponding call.
   3. Compile all the logging shims into a shared library, libshim.so.
   4. Trace any application using libfoo by running it as
   LD_PRELOAD=libshim.so ./fooapp

There are two nice things about this approach that I would like to preserve:

   - The shim developer only needs to provide the whitelist and a bit of
   extra annotation. From there, the shim can be be authored using a
   boilerplate generator.
   - The app developer doesn't have to do anything other than pass the
   LD_PRELOAD flag.

The downside is that the only tracepoints are those corresponding to the
whitelist, but I'm fine with that.

Can this kind of "hands-free" developer experience be supported by LTTng?

Thanks!
~br

[-- Attachment #1.2: Type: text/html, Size: 1213 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev