Re: LTTng buffer mode

Re: LTTng buffer mode

[Philippe Proulx]

No you cannot.

There's no hidden global buffering scheme option for user space tracing.

From the LTTng documentation:

> In the user space tracing domain, two buffering schemes are available
> when creating a channel:
>
> Per-PID buffering: keep one ring buffer per process.
> Per-UID buffering: keep one ring buffer for all processes of a single user.

and:

> The Linux kernel tracing domain only has one available buffering scheme
> which is to use a single ring buffer for the whole system.

Phil

On Tue, Feb 16, 2016 at 7:55 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:
> Hi LTTng community:
>
>
> I have a question about LTTng buffer mode. According to documentation, LTTng
> supports 2 buffer modes, per-pid, and per-uid. Is there any mode to have a
> global buffer that does not have separate buffers? The reason I asked is
> because, LTTng writes different buffer traces to different directory/file.
> It is much more difficult for us to post-process separate directories. We
> would like to have all traces write into one single buffer and
> directory/file, so it is easier to do post-processing. Thanks.
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

Re: LTTng buffer mode

[Jeffrey Chen]

Thanks.
If we make changes to LTTng to add this feature, would it violate the LTTng design principle in any way? Thanks.


________________________________________
From: Philippe Proulx <eeppeliteloop@gmail.com>
Sent: Tuesday, February 16, 2016 5:02 PM
To: Jeffrey Chen
Cc: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] LTTng buffer mode

No you cannot.

There's no hidden global buffering scheme option for user space tracing.

From the LTTng documentation:

> In the user space tracing domain, two buffering schemes are available
> when creating a channel:
>
> Per-PID buffering: keep one ring buffer per process.
> Per-UID buffering: keep one ring buffer for all processes of a single user.

and:

> The Linux kernel tracing domain only has one available buffering scheme
> which is to use a single ring buffer for the whole system.

Phil

On Tue, Feb 16, 2016 at 7:55 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:
> Hi LTTng community:
>
>
> I have a question about LTTng buffer mode. According to documentation, LTTng
> supports 2 buffer modes, per-pid, and per-uid. Is there any mode to have a
> global buffer that does not have separate buffers? The reason I asked is
> because, LTTng writes different buffer traces to different directory/file.
> It is much more difficult for us to post-process separate directories. We
> would like to have all traces write into one single buffer and
> directory/file, so it is easier to do post-processing. Thanks.
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

Re: LTTng buffer mode

[Mathieu Desnoyers]

Hi,

LTTng implements per-cpu buffers to eliminate false-sharing between
CPUs. Babeltrace and Trace Compass implement iterators that allow
doing the merge of those per-cpu buffers into a single stream on
the fly. Users of Babeltrace and Trace Compass should not have to
worry about doing this merge.

Since LTTng is inherently multi-buffers, we also use this buffer
merging scheme to deal with tracefile rotation, multi-channel, and
combining traces gathered from various processes/user IDs/kernel vs
userspace.

So my question here is: why you say it is much more difficult for
you to deal with those multiple buffers ? Since there are already
libraries in Babeltrace that do that for you, I would expect this
to be a non-issue.

Although it would be technically possible to implement a ring
buffer shared across CPUs, it would be inefficient, and we would
need a good justification for adding this complexity to the
tracers.

Thanks,

Mathieu


----- On Feb 19, 2016, at 4:00 AM, Jeffrey Chen cpthk@hotmail.com wrote:

> Thanks.
> If we make changes to LTTng to add this feature, would it violate the LTTng
> design principle in any way? Thanks.
> 
> 
> ________________________________________
> From: Philippe Proulx <eeppeliteloop@gmail.com>
> Sent: Tuesday, February 16, 2016 5:02 PM
> To: Jeffrey Chen
> Cc: lttng-dev@lists.lttng.org
> Subject: Re: [lttng-dev] LTTng buffer mode
> 
> No you cannot.
> 
> There's no hidden global buffering scheme option for user space tracing.
> 
> From the LTTng documentation:
> 
>> In the user space tracing domain, two buffering schemes are available
>> when creating a channel:
>>
>> Per-PID buffering: keep one ring buffer per process.
>> Per-UID buffering: keep one ring buffer for all processes of a single user.
> 
> and:
> 
>> The Linux kernel tracing domain only has one available buffering scheme
>> which is to use a single ring buffer for the whole system.
> 
> Phil
> 
> On Tue, Feb 16, 2016 at 7:55 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:
>> Hi LTTng community:
>>
>>
>> I have a question about LTTng buffer mode. According to documentation, LTTng
>> supports 2 buffer modes, per-pid, and per-uid. Is there any mode to have a
>> global buffer that does not have separate buffers? The reason I asked is
>> because, LTTng writes different buffer traces to different directory/file.
>> It is much more difficult for us to post-process separate directories. We
>> would like to have all traces write into one single buffer and
>> directory/file, so it is easier to do post-processing. Thanks.
>>
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: LTTng buffer mode

[syed zaidi]

[-- Attachment #1.1: Type: text/plain, Size: 4477 bytes --]

Hi Mathieu,

I have few questions related to the questions above and please forgive me
for my ignorance in advance.
1. When LTTng creates the buffers lets say per uid, the files that show up
in LTTng folder, are those files that LTTng has stopped writing in? What I
mean is, are those files still being written to or they are safe to be
moved.

2. If I want to collect traces lets say every 5 mins for the last 5 mins
while LTTng keeps running. What is the best approach? Do I just run
BabelTrace on the folder and remove all files from the folder to create
space for next files and keep repeating?

3. I understand that I should delete the corresponding .idx file for each
log file that I remove, what about metadata file? Do I remove it or leave
it as is?

4. The trace file produced by Babeltrace, does it guarantee that the file
has traces from all the users and the core till the time stamp of the last
line?'
For example if the last line of the trace file looks like this
23:00:00.912089772,...,...
Does that mean that the trace file has all the traces until that time?

Thanks
Ammar

On Sat, Feb 20, 2016 at 10:17 AM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> Hi,
>
> LTTng implements per-cpu buffers to eliminate false-sharing between
> CPUs. Babeltrace and Trace Compass implement iterators that allow
> doing the merge of those per-cpu buffers into a single stream on
> the fly. Users of Babeltrace and Trace Compass should not have to
> worry about doing this merge.
>
> Since LTTng is inherently multi-buffers, we also use this buffer
> merging scheme to deal with tracefile rotation, multi-channel, and
> combining traces gathered from various processes/user IDs/kernel vs
> userspace.
>
> So my question here is: why you say it is much more difficult for
> you to deal with those multiple buffers ? Since there are already
> libraries in Babeltrace that do that for you, I would expect this
> to be a non-issue.
>
> Although it would be technically possible to implement a ring
> buffer shared across CPUs, it would be inefficient, and we would
> need a good justification for adding this complexity to the
> tracers.
>
> Thanks,
>
> Mathieu
>
>
> ----- On Feb 19, 2016, at 4:00 AM, Jeffrey Chen cpthk@hotmail.com wrote:
>
> > Thanks.
> > If we make changes to LTTng to add this feature, would it violate the
> LTTng
> > design principle in any way? Thanks.
> >
> >
> > ________________________________________
> > From: Philippe Proulx <eeppeliteloop@gmail.com>
> > Sent: Tuesday, February 16, 2016 5:02 PM
> > To: Jeffrey Chen
> > Cc: lttng-dev@lists.lttng.org
> > Subject: Re: [lttng-dev] LTTng buffer mode
> >
> > No you cannot.
> >
> > There's no hidden global buffering scheme option for user space tracing.
> >
> > From the LTTng documentation:
> >
> >> In the user space tracing domain, two buffering schemes are available
> >> when creating a channel:
> >>
> >> Per-PID buffering: keep one ring buffer per process.
> >> Per-UID buffering: keep one ring buffer for all processes of a single
> user.
> >
> > and:
> >
> >> The Linux kernel tracing domain only has one available buffering scheme
> >> which is to use a single ring buffer for the whole system.
> >
> > Phil
> >
> > On Tue, Feb 16, 2016 at 7:55 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:
> >> Hi LTTng community:
> >>
> >>
> >> I have a question about LTTng buffer mode. According to documentation,
> LTTng
> >> supports 2 buffer modes, per-pid, and per-uid. Is there any mode to
> have a
> >> global buffer that does not have separate buffers? The reason I asked is
> >> because, LTTng writes different buffer traces to different
> directory/file.
> >> It is much more difficult for us to post-process separate directories.
> We
> >> would like to have all traces write into one single buffer and
> >> directory/file, so it is easier to do post-processing. Thanks.
> >>
> >>
> >>
> >> _______________________________________________
> >> lttng-dev mailing list
> >> lttng-dev@lists.lttng.org
> >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >>
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

[-- Attachment #1.2: Type: text/html, Size: 6346 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng buffer mode

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 5240 bytes --]

----- On Feb 26, 2016, at 4:12 AM, syed zaidi <zaidi1039@gmail.com> wrote: 

> Hi Mathieu,
> I have few questions related to the questions above and please forgive me for my
> ignorance in advance.
> 1. When LTTng creates the buffers lets say per uid, the files that show up in
> LTTng folder, are those files that LTTng has stopped writing in? What I mean
> is, are those files still being written to or they are safe to be moved.

They are still being written to. 

> 2. If I want to collect traces lets say every 5 mins for the last 5 mins while
> LTTng keeps running. What is the best approach? Do I just run BabelTrace on the
> folder and remove all files from the folder to create space for next files and
> keep repeating?

I would not use Babeltrace on the target, but rather just move the CTF files. 

You could swap between two sessions to do that: lttng supports multiple 
concurrent sessions. Create and start a new session before you stop, 
destroy, and move away the old session, every 5 minutes... 

> 3. I understand that I should delete the corresponding .idx file for each log
> file that I remove, what about metadata file? Do I remove it or leave it as is?

You need to leave it as is. 

> 4. The trace file produced by Babeltrace, does it guarantee that the file has
> traces from all the users and the core till the time stamp of the last line?'
> For example if the last line of the trace file looks like this
> 23:00:00.912089772,...,...
> Does that mean that the trace file has all the traces until that time?

It depends on your tracing mode (discard vs snapshot buffers). In discard 
mode (default if you don't use the --snapshot option on lttng create), 
Babeltrace will print on stderr whenever it encounters discarded events. 

Best regards, 

Mathieu 

> Thanks
> Ammar

> On Sat, Feb 20, 2016 at 10:17 AM, Mathieu Desnoyers <
> mathieu.desnoyers@efficios.com > wrote:

>> Hi,

>> LTTng implements per-cpu buffers to eliminate false-sharing between
>> CPUs. Babeltrace and Trace Compass implement iterators that allow
>> doing the merge of those per-cpu buffers into a single stream on
>> the fly. Users of Babeltrace and Trace Compass should not have to
>> worry about doing this merge.

>> Since LTTng is inherently multi-buffers, we also use this buffer
>> merging scheme to deal with tracefile rotation, multi-channel, and
>> combining traces gathered from various processes/user IDs/kernel vs
>> userspace.

>> So my question here is: why you say it is much more difficult for
>> you to deal with those multiple buffers ? Since there are already
>> libraries in Babeltrace that do that for you, I would expect this
>> to be a non-issue.

>> Although it would be technically possible to implement a ring
>> buffer shared across CPUs, it would be inefficient, and we would
>> need a good justification for adding this complexity to the
>> tracers.

>> Thanks,

>> Mathieu

>> ----- On Feb 19, 2016, at 4:00 AM, Jeffrey Chen cpthk@hotmail.com wrote:

>> > Thanks.
>> > If we make changes to LTTng to add this feature, would it violate the LTTng
>> > design principle in any way? Thanks.


>> > ________________________________________
>> > From: Philippe Proulx < eeppeliteloop@gmail.com >
>> > Sent: Tuesday, February 16, 2016 5:02 PM
>> > To: Jeffrey Chen
>> > Cc: lttng-dev@lists.lttng.org
>> > Subject: Re: [lttng-dev] LTTng buffer mode

>> > No you cannot.

>> > There's no hidden global buffering scheme option for user space tracing.

>> > From the LTTng documentation:

>> >> In the user space tracing domain, two buffering schemes are available
>> >> when creating a channel:

>> >> Per-PID buffering: keep one ring buffer per process.
>> >> Per-UID buffering: keep one ring buffer for all processes of a single user.

>> > and:

>> >> The Linux kernel tracing domain only has one available buffering scheme
>> >> which is to use a single ring buffer for the whole system.

>> > Phil

>> > On Tue, Feb 16, 2016 at 7:55 PM, Jeffrey Chen < cpthk@hotmail.com > wrote:
>> >> Hi LTTng community:


>> >> I have a question about LTTng buffer mode. According to documentation, LTTng
>> >> supports 2 buffer modes, per-pid, and per-uid. Is there any mode to have a
>> >> global buffer that does not have separate buffers? The reason I asked is
>> >> because, LTTng writes different buffer traces to different directory/file.
>> >> It is much more difficult for us to post-process separate directories. We
>> >> would like to have all traces write into one single buffer and
>> >> directory/file, so it is easier to do post-processing. Thanks.



>> >> _______________________________________________
>> >> lttng-dev mailing list
>> >> lttng-dev@lists.lttng.org
>> >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


>> > _______________________________________________
>> > lttng-dev mailing list
>> > lttng-dev@lists.lttng.org
>> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 9254 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

LTTng buffer mode

[Jeffrey Chen]

[-- Attachment #1.1: Type: text/plain, Size: 528 bytes --]

Hi LTTng community:


I have a question about LTTng buffer mode. According to documentation, LTTng supports 2 buffer modes, per-pid, and per-uid. Is there any mode to have a global buffer that does not have separate buffers? The reason I asked is because, LTTng writes different buffer traces to different directory/file. It is much more difficult for us to post-process separate directories. We would like to have all traces write into one single buffer and directory/file, so it is easier to do post-processing. Thanks.


[-- Attachment #1.2: Type: text/html, Size: 953 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev