Re: clarification on limiting the size of the trace file

Re: clarification on limiting the size of the trace file

[Sadon, Mirit]

Hi,

This is what I understood should happen but in the actual run the lttng file fills all the free space in the partition 4GB.

Thanks
Mirit Sadon 

-----Original Message-----
From: Thibault, Daniel [mailto:Daniel.Thibault@drdc-rddc.gc.ca] 
Sent: Monday, May 05, 2014 7:10 PM
To: lttng-dev@lists.lttng.org
Cc: Sadon, Mirit; Eshel, Ayelet
Subject: RE: [lttng-dev] clarification on limiting the size of the trace file

----------------------------------------------------------------------
Date: Mon, 5 May 2014 08:24:39 +0000
From: "Sadon, Mirit" <Mirit.Sadon@emc.com>
Cc: "Eshel, Ayelet" <Ayelet.Eshel@emc.com>

> I am working with the lttng trace to see how it affects are program 
> performance, But if I don't limit the size of the trace file collected it fills all the space in the partition I specify.
> What does it mean if I start the trace and limit the trace file to let say 1GB.
> Would the trace stop after it reaches the limit or it will continue and override the previous collected points?
>
> I tried by using this command when creating the channel but it didn't work.
>
> lttng enable-channel perf --subbuf-size 262144 -C 262144 -W 32 
> --num-subbuf 32 -u
>
> We are working with lttng version 2.04.
>
> Mirit Sadon
> EMC2

   This feature may not have been mature with version 2.0.4; I would recommend upgrading to at least 2.3 if not 2.4.

   Your command sets up a single user-space channel named perf which uses a 32-part buffer of total size 32*262144=8388608 (8 MiB).  The trace is itself limited to 32 chunks of size 262144 each.  By default, user-space channels run in discard mode.  So in this case LTTng should capture about 8 MiB of events to disc, and a further 8 MiB in memory before "giving up" altogether.

   When the trace is started the 8 MiB buffer will start filling up, and the consumer daemon will start dumping the events to the trace file.  Once the trace file saturates at 8 MiB total size, the consumer daemon will stop emptying the buffer.  The buffer will then capture another 8 MiB of events.  Once the buffer is full, LTTng will systematically discard new events as they occur.

   It would be an interesting feature if the snapshot command could be used at this point to dump the 8 MiB of otherwise-lost events to another trace file.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC) Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS) R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada <http://www.valcartier.drdc-rddc.gc.ca/>

Re: clarification on limiting the size of the trace file

[Thibault, Daniel]

> > > We are working with lttng version 2.04.
> > This feature may not have been mature with version 2.0.4; I would recommend upgrading to at least 2.3 if not 2.4.
> Envoyé : 7 mai 2014 01:12
> This is what I understood should happen but in the actual run the lttng file fills all the free space in the partition 4GB.
> Mirit Sadon 

   I just ran a test and in both kernel and user-space tracing modes LTTng 2.3 behaved precisely as expected.  I can only suggest that your problem is probably due to LTTng 2.0.4 not handling the instructions correctly.  Upgrading is the solution.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: clarification on limiting the size of the trace file

[Sadon, Mirit]

Hi,

Thanks, I upgraded and this problem was fixed. 
But I noticed something new.

When I run the "lttng list command" I get the events in different order from two locations I installed the new version.
Did you encounter such an issue before?

Thanks
Mirit 

-----Original Message-----
From: Thibault, Daniel [mailto:Daniel.Thibault@drdc-rddc.gc.ca] 
Sent: Wednesday, May 07, 2014 4:03 PM
To: Sadon, Mirit; lttng-dev@lists.lttng.org
Cc: Eshel, Ayelet; Mirsky, Ilya
Subject: RE: [lttng-dev] clarification on limiting the size of the trace file

> > > We are working with lttng version 2.04.
> > This feature may not have been mature with version 2.0.4; I would recommend upgrading to at least 2.3 if not 2.4.
> Envoyé : 7 mai 2014 01:12
> This is what I understood should happen but in the actual run the lttng file fills all the free space in the partition 4GB.
> Mirit Sadon

   I just ran a test and in both kernel and user-space tracing modes LTTng 2.3 behaved precisely as expected.  I can only suggest that your problem is probably due to LTTng 2.0.4 not handling the instructions correctly.  Upgrading is the solution.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC) Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS) R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada <http://www.valcartier.drdc-rddc.gc.ca/>

Re: clarification on limiting the size of the trace file

[Thibault, Daniel]

-----Message d'origine-----
Envoyé : 8 mai 2014 07:26

> Thanks, I upgraded and this problem was fixed. 
> But I noticed something new.
>
> When I run the "lttng list command" I get the events in different order from two locations I installed the new version.
> Did you encounter such an issue before?
>
> Mirit 

   I'm not sure I understand your question or your concern.  Do you mean that the output of 'lttng list -k' is different each time it is run, or different between otherwise-identical machines?  Or are you talking about 'lttng list -u'?  Or 'lttng list <session_name>'?

   LTTng makes no guarantees that the list of events is sorted in any particular way.  Kernel events normally appear in last-module-loaded-first order, with the events within a module appearing in seemingly arbitrary order (probably last-declared-first within each module).  The ordering of user-space events is truly unpredictable, as it depends on the order in which tracepoint providers register with the session daemon.

   If you need strict ordering of the event listing, you must do it yourself (in a programmatic context) or pipe it through the 'sort' applet (in a shell script context).

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: clarification on limiting the size of the trace file

[Thibault, Daniel]

----------------------------------------------------------------------
Date: Mon, 5 May 2014 16:10:06 +0000

>   Your command sets up a single user-space channel named perf which uses a 32-part buffer of total size 32*262144=8388608 (8 MiB).
>   The trace is itself limited to 32 chunks of size 262144 each.  By default, user-space channels run in discard mode.  So in this case LTTng
> should capture about 8 MiB of events to disc, and a further 8 MiB in memory before "giving up" altogether.
>
>   When the trace is started the 8 MiB buffer will start filling up, and the consumer daemon will start dumping the events to the trace file.
> Once the trace file saturates at 8 MiB total size, the consumer daemon will stop emptying the buffer.  The buffer will then capture
> another 8 MiB of events.  Once the buffer is full, LTTng will systematically discard new events as they occur.
>
>   It would be an interesting feature if the snapshot command could be used at this point to dump the 8 MiB of otherwise-lost events to another trace file.

   My apologies, I misspoke.  The discard/overwrite setting of the channel is independent of the trace file size/count settings, in the sense that it only matters when comparing the relative data rates of the trace providers and consumers.  In the example above, the on-disc trace file does *not* saturate: rather, once all 32 file chunks have been written, the consumer daemon re-uses chunk 0, then chunk 1, etc.  The on-disc trace file in effect becomes a circular buffer itself.

   It could be an interesting feature to have a separate option, say 'tracefile-discard', which would have the trace file, when tracefile-size is specified, prevent re-use of its files (or chunks if also specified).  This could be used to capture a trace of the boot process events, for instance (which you can already do using snapshot+discard mode).  If anyone thinks this may be of use, please submit a feature request on bugs.lttng.org.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: clarification on limiting the size of the trace file

[Thibault, Daniel]

----------------------------------------------------------------------
Date: Mon, 5 May 2014 08:24:39 +0000
From: "Sadon, Mirit" <Mirit.Sadon@emc.com>
Cc: "Eshel, Ayelet" <Ayelet.Eshel@emc.com>

> I am working with the lttng trace to see how it affects are program performance,
> But if I don't limit the size of the trace file collected it fills all the space in the partition I specify.
> What does it mean if I start the trace and limit the trace file to let say 1GB.
> Would the trace stop after it reaches the limit or it will continue and override the previous collected points?
>
> I tried by using this command when creating the channel but it didn't work.
>
> lttng enable-channel perf --subbuf-size 262144 -C 262144 -W 32 --num-subbuf 32 -u
>
> We are working with lttng version 2.04.
>
> Mirit Sadon
> EMC2

   This feature may not have been mature with version 2.0.4; I would recommend upgrading to at least 2.3 if not 2.4.

   Your command sets up a single user-space channel named perf which uses a 32-part buffer of total size 32*262144=8388608 (8 MiB).  The trace is itself limited to 32 chunks of size 262144 each.  By default, user-space channels run in discard mode.  So in this case LTTng should capture about 8 MiB of events to disc, and a further 8 MiB in memory before "giving up" altogether.

   When the trace is started the 8 MiB buffer will start filling up, and the consumer daemon will start dumping the events to the trace file.  Once the trace file saturates at 8 MiB total size, the consumer daemon will stop emptying the buffer.  The buffer will then capture another 8 MiB of events.  Once the buffer is full, LTTng will systematically discard new events as they occur.

   It would be an interesting feature if the snapshot command could be used at this point to dump the 8 MiB of otherwise-lost events to another trace file.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>