Re: User-space tracing by the root lttng-sessiond

Re: User-space tracing by the root lttng-sessiond

[Mathieu Desnoyers]

* Thibault, Daniel (Daniel.Thibault@drdc-rddc.gc.ca) wrote:
>    I ran a little experiment whose results surprised me.  Using a non-tracing group user account, I first made sure I had a root lttng-sessiond running and a user-space lttng-sessiond.  In the first I set up a trace for some kernel events plus all user-space events.  In the second I set up a trace for all user-space events only.  Both traces were started, then the lttng-ust/doc/examples/easy-ust/sample demo was run to generate the user-space events.  Then both traces were stopped and destroyed.
> 
>    I noted there were three lttng-consumerd daemons: one in user-space, the other two owned by root.  Presumably one is servicing the kernel events while the other is servicing the user-space events.

yes

>  (Question: If there were several users active at once, would there be several root user-space lttng-consumerd daemons or still just one?)

no

> 
>    As expected, both traces show the sample user-space events (with no losses, it turns out).  What surprised me, however, is that the pairs of events were not in synch. Since they're all coming out of the same trace provider, I would expect the time-stamp to be issued once (by the tracepoint provider), and then two copies of the event to go into the two trace buffers for eventual storage.  Could it be that the tracepoint provider instead issues two separate events, one for each of its clients?

yes

>  (For the record, the pairs of events were out of synch by at least 4187 ns (670 times out of 100 000), and as much as 3 326 405 ns (once); in 93% of the time root event lags after the user-space event)

can be caused by interruption (for 4k ns) and preemption (for the 3s
hit).

> 
>    This seems problematic to me, since multiple traces capturing the same event won't be able to reconcile their copies with each other.

since timestamps have to match the event physical order in the streams,
we need to assign one timestamp for each buffer. no way around this.

and yes, tracing is not stopping the world, so you can expect to have
slight timing inaccuracy like this. It's a trade-off between perfect
timing accuracy and tracer overhead.

Thanks,

Mathieu


> 
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: User-space tracing by the root lttng-sessiond

[Thibault, Daniel]

> >  (Question: If there were several users active at once, would there be 
> > several root user-space lttng-consumerd daemons or still just one?)
>
> no

   Please clarify: no, there will be several root user-space lttng-consumerd daemons or no, there will be just one root user-space lttng-consumerd daemon?

> >    This seems problematic to me, since multiple traces capturing the same event won't be able to reconcile their copies with each other.
>
> since timestamps have to match the event physical order in the streams, we need to assign one timestamp for each buffer. no way around this.

   Are the time-stamps issued by the tracepoint provider or by the "buffer registrar" (i.e. the time-stamp is looked up when the event is written to the buffer)?

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: User-space tracing by the root lttng-sessiond

[Mathieu Desnoyers]

* Thibault, Daniel (Daniel.Thibault@drdc-rddc.gc.ca) wrote:
> > >  (Question: If there were several users active at once, would there be 
> > > several root user-space lttng-consumerd daemons or still just one?)
> >
> > no
> 
>    Please clarify: no, there will be several root user-space
>    lttng-consumerd daemons or no, there will be just one root
>    user-space lttng-consumerd daemon?

still just one.

> 
> > >    This seems problematic to me, since multiple traces capturing the same event won't be able to reconcile their copies with each other.
> >
> > since timestamps have to match the event physical order in the streams, we need to assign one timestamp for each buffer. no way around this.
> 
>    Are the time-stamps issued by the tracepoint provider or by the
>    "buffer registrar" (i.e. the time-stamp is looked up when the event
>    is written to the buffer)?

the latter.

Thanks,

Mathieu


> 
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

User-space tracing by the root lttng-sessiond

[Thibault, Daniel]

   I ran a little experiment whose results surprised me.  Using a non-tracing group user account, I first made sure I had a root lttng-sessiond running and a user-space lttng-sessiond.  In the first I set up a trace for some kernel events plus all user-space events.  In the second I set up a trace for all user-space events only.  Both traces were started, then the lttng-ust/doc/examples/easy-ust/sample demo was run to generate the user-space events.  Then both traces were stopped and destroyed.

   I noted there were three lttng-consumerd daemons: one in user-space, the other two owned by root.  Presumably one is servicing the kernel events while the other is servicing the user-space events.  (Question: If there were several users active at once, would there be several root user-space lttng-consumerd daemons or still just one?)

   As expected, both traces show the sample user-space events (with no losses, it turns out).  What surprised me, however, is that the pairs of events were not in synch. Since they're all coming out of the same trace provider, I would expect the time-stamp to be issued once (by the tracepoint provider), and then two copies of the event to go into the two trace buffers for eventual storage.  Could it be that the tracepoint provider instead issues two separate events, one for each of its clients?  (For the record, the pairs of events were out of synch by at least 4187 ns (670 times out of 100 000), and as much as 3 326 405 ns (once); in 93% of the time root event lags after the user-space event)

   This seems problematic to me, since multiple traces capturing the same event won't be able to reconcile their copies with each other.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>