Using lttng-ust's libringbuffer outside lttng-ust

Using lttng-ust's libringbuffer outside lttng-ust

[David Smith]

Over in systemtap land, we're working on implementing a pure userspace
backend, using a project called dyninst (<http://www.dyninst.org/>). As
part of this work, we need to implement a data channel between
systemtap and the target executable(s). Most of the data (print
statement output) flows from probes run in the target back to systemtap.
There is also a need for a control channel the flows from systemtap to
the systemtap runtime present in the target.

I'm looking into the possibility of using libringbuffer from lttng-ust,
which seems like it might fit our needs. Based on my description above
(feel free to ask for more detail), does using libringbuffer make sense?

Assuming it does make sense, I could use some help. Is there any
documentation that I've missed somewhere that describes libringbuffer in
any detail? I've also looked through the test programs in
'lttng-ust/tests' hoping to find a test for just basic libringbuffer
functionality (a standalone producer/consumer), without much luck. (The
code in 'ust-basic-tracing' and 'ust-multi-test' probably comes the
closest.) I started trying to write such a test, but then realized it
made better sense to ask here first.

The next question would be what is the best way of using libringbuffer
outside lttng-ust. One (not great) possibility would be to just copy the
code into systemtap. Of course the problem there is keeping the code up
to date with changes in lttng-ust. The next possibility would be to make
it a separate project (more like userspace-rcu). A bit of work would be
needed here, since while libringbuffer is mostly self-contained, there
is some use of code from 'lttng-ust/lttng-ust-comm' and
'lttng-ust/snprintf' from 'lttng-ust/libringbuffer'.

Thanks for the help.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: [lttng-dev] Using lttng-ust's libringbuffer outside lttng-ust

[Christian Babeux]

Hi David,

> Over in systemtap land, we're working on implementing a pure userspace
> backend, using a project called dyninst (<http://www.dyninst.org/>).

This project looks very interesting!

> Assuming it does make sense, I could use some help. Is there any
> documentation that I've missed somewhere that describes libringbuffer in
> any detail?

I found some old presentation that Mathieu did a Linuxcon in 2010 [1].
The presentation describe the unified kernel ringbuffer on which the
user space one is based. I'm afraid that if you want up-to-date
documentation, the source code will be the best place to look.

> The next question would be what is the best way of using libringbuffer
> outside lttng-ust. One (not great) possibility would be to just copy the
> code into systemtap. Of course the problem there is keeping the code up
> to date with changes in lttng-ust. The next possibility would be to make
> it a separate project (more like userspace-rcu).

Splitting libringbuffer from the UST tree right now would prove to be
quite a challenge. We would need to expose an API to the "clients" of
the ringbuffer. Currently there are two levels of abstraction that one
can use with the ringbuffer: a "high-level" one where you use a
predefined "client" with the associated performance degradation and a
"low-level" one where you know exactly which kind of "client" you are
using with the associated fast-path inlined.

We would need to discuss which level of abstraction we expose to the
user: a generic one or "N apis" for the different low-level "clients".

With that said, I'm curious about which specific features of Dyninst
SystemTAP will be using and also do you think that it would be
possible to integrate Dyninst as an additional instrumentation source
into lttng-ust?

Thanks,

Christian

[1] -  A new unified Lockless Ring Buffer library for efficient kernel tracing:
        http://www.efficios.com/pub/linuxcon2010-tracingsummit/presentation-linuxcon-2010-tracing-mini-summit.pdf

Re: [lttng-dev] Using lttng-ust's libringbuffer outside lttng-ust

[Josh Stone]

On 02/15/2013 12:36 PM, Christian Babeux wrote:
> With that said, I'm curious about which specific features of Dyninst
> SystemTAP will be using and also do you think that it would be
> possible to integrate Dyninst as an additional instrumentation source
> into lttng-ust?

We're using Dyninst mainly like a pure-userspace form of uprobes, so we
can instrument a process without any direct kernel involvement.  In this
mode, stap is perfectly fine for unprivileged users (on their own
processes), and it can be faster too since it's running instrumentation
directly in-process (versus uprobes' breakpoint trap).  Like uprobes,
these instrumentation points can be anywhere in the process.

And sure, I can definitely see lttng using it for similar reasons.
We've already packaged Dyninst in Fedora, if you'd like to try it out.

Josh

Re: [lttng-dev] Using lttng-ust's libringbuffer outside lttng-ust

[David Smith]

On 02/15/2013 02:36 PM, Christian Babeux wrote:
> Hi David,

Thanks for the response.

>> The next question would be what is the best way of using libringbuffer
>> outside lttng-ust. One (not great) possibility would be to just copy the
>> code into systemtap. Of course the problem there is keeping the code up
>> to date with changes in lttng-ust. The next possibility would be to make
>> it a separate project (more like userspace-rcu).
> 
> Splitting libringbuffer from the UST tree right now would prove to be
> quite a challenge. We would need to expose an API to the "clients" of
> the ringbuffer. Currently there are two levels of abstraction that one
> can use with the ringbuffer: a "high-level" one where you use a
> predefined "client" with the associated performance degradation and a
> "low-level" one where you know exactly which kind of "client" you are
> using with the associated fast-path inlined.
> 
> We would need to discuss which level of abstraction we expose to the
> user: a generic one or "N apis" for the different low-level "clients".

In my head, I was planning on using the APIs defined in
libringbuffer/*.h (like lib_ring_buffer_{reserve,commit,write} from
frontend_api.h). But, perhaps we could back up a sec and you could
describe a bit more the low-level vs. high-level levels of abstraction.
Feel free to point me at some code.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Using lttng-ust's libringbuffer outside lttng-ust

[David Smith]

On 02/15/2013 02:36 PM, Christian Babeux wrote:
> Hi David,

Thanks for the response.

>> The next question would be what is the best way of using libringbuffer
>> outside lttng-ust. One (not great) possibility would be to just copy the
>> code into systemtap. Of course the problem there is keeping the code up
>> to date with changes in lttng-ust. The next possibility would be to make
>> it a separate project (more like userspace-rcu).
> 
> Splitting libringbuffer from the UST tree right now would prove to be
> quite a challenge. We would need to expose an API to the "clients" of
> the ringbuffer. Currently there are two levels of abstraction that one
> can use with the ringbuffer: a "high-level" one where you use a
> predefined "client" with the associated performance degradation and a
> "low-level" one where you know exactly which kind of "client" you are
> using with the associated fast-path inlined.
> 
> We would need to discuss which level of abstraction we expose to the
> user: a generic one or "N apis" for the different low-level "clients".

In my head, I was planning on using the APIs defined in
libringbuffer/*.h (like lib_ring_buffer_{reserve,commit,write} from
frontend_api.h). But, perhaps we could back up a sec and you could
describe a bit more the low-level vs. high-level levels of abstraction.
Feel free to point me at some code.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)