Re: LTTng UST vs Syslog, Printf...

Re: LTTng UST vs Syslog, Printf...

[Matthew Khouzam]

I would like to see if the UST logs were ("message %s", string)  of the
printf'ed function.

I remember testing it a while ago and getting about 20x less space for a
trace for 10m events where it was int i=0 ; for(i < 10000000; i++)
trace("val %d", i++);

I formatted recently so I will have to dig a bit but I think the graph
on page 11 discards the fact that since the logs in LTTng are deferred,
they will be logged without stopping the program.

My 0.02c... oh wait, the penny was retired.

Matt



On 13-08-20 02:05 PM, Jim Dumont wrote:
> Hi there,
>
> Has anyone done a recent performance and characteristics comparison between lttngust with syslog and printf?   Things like memory & cpu footprint, tps, i/o, disk space...
>
> I found this Windriver comparison from 2011: https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCoQFjAA&url=https%3A%2F%2Fevents.linuxfoundation.org%2Fslides%2F2011%2Flinuxcon%2Flcna2011_wessel.pdf&ei=Y64TUu6qJqS62AXIqYCYCg&usg=AFQjCNF0Q05MytPYVNWBPnUjB9LEGJfQZA
>
> And if I recall correctly, LTTng UST team also did a printf comparison a while back, but was wondering if someone has done more recent prototyping?   Any comparisons with syslog?
>
> Regards,
>
> /Jim Dumont
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng UST vs Syslog, Printf...

[Mathieu Desnoyers]

* Jim Dumont (jim.dumont@ericsson.com) wrote:
> Hi there,
> 
> Has anyone done a recent performance and characteristics comparison between lttngust with syslog and printf?   Things like memory & cpu footprint, tps, i/o, disk space...

Not that I am aware of.

> 
> I found this Windriver comparison from 2011:
> https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCoQFjAA&url=https%3A%2F%2Fevents.linuxfoundation.org%2Fslides%2F2011%2Flinuxcon%2Flcna2011_wessel.pdf&ei=Y64TUu6qJqS62AXIqYCYCg&usg=AFQjCNF0Q05MytPYVNWBPnUjB9LEGJfQZA

This presentation uses UST 0.x. This is _old_: it predates LTTng 2.0.

> 
> And if I recall correctly, LTTng UST team also did a printf comparison
> a while back, but was wondering if someone has done more recent
> prototyping?  Any comparisons with syslog?

Not at this point. It would be interesting to compare:

- lttng-ust, both in "discard" and "snapshot" modes,
vs
- printf with timestamp,
vs
- syslog

Especially on multi-core systems, with applications generating a _lot_
of log/trace data.

Thanks,

Mathieu

> 
> Regards,
> 
> /Jim Dumont
> 
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: LTTng UST vs Syslog, Printf...

[Mathieu Desnoyers]

* Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> * Jim Dumont (jim.dumont@ericsson.com) wrote:
> > Hi there,
> > 
> > Has anyone done a recent performance and characteristics comparison between lttngust with syslog and printf?   Things like memory & cpu footprint, tps, i/o, disk space...
> 
> Not that I am aware of.
> 
> > 
> > I found this Windriver comparison from 2011:
> > https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCoQFjAA&url=https%3A%2F%2Fevents.linuxfoundation.org%2Fslides%2F2011%2Flinuxcon%2Flcna2011_wessel.pdf&ei=Y64TUu6qJqS62AXIqYCYCg&usg=AFQjCNF0Q05MytPYVNWBPnUjB9LEGJfQZA
> 
> This presentation uses UST 0.x. This is _old_: it predates LTTng 2.0.
> 
> > 
> > And if I recall correctly, LTTng UST team also did a printf comparison
> > a while back, but was wondering if someone has done more recent
> > prototyping?  Any comparisons with syslog?
> 
> Not at this point. It would be interesting to compare:
> 
> - lttng-ust, both in "discard" and "snapshot" modes,
> vs
> - printf with timestamp,
> vs
> - syslog
> 
> Especially on multi-core systems, with applications generating a _lot_
> of log/trace data.

Just to add to the reflexion, there are key differences between
lttng-ust and printf/syslog that makes comparison a bit difficult:

lttng-ust never blocks the application when buffers are full. It either
discards events or overwrites the oldest information (flight recorder
tracing).

printf and syslog will block the application if the disk I/O is not fast
enough.

How would you recommand comparing these ?

Moreover, in order to ensure we can compare those, the printf benchmark
would need to grab a time-stamp, and would need to be performed with one
call per event (no more), since it is only "atomic" from the point of
view of its buffer on a per-call basis. Moreover, printf is not
async-signal-safe (as per signal(7)), so it should be noted that it
cannot be used from a signal handler (whereas lttng-ust can be used from
signal handlers).

Thanks,

Mathieu


> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Regards,
> > 
> > /Jim Dumont
> > 
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: LTTng UST vs Syslog, Printf...

[Jim Dumont]

Hi there,

Totally true that it's a bit apples / oranges as far as comparison goes, but the comparison questions continue to come up...   so some quantitative numbers would be useful.

I would suggest something relatively simple and basic as far as performance & characteristics measurements goes - x apps, y traces / second - measure CPU, memory, i/o, file size across the various mechanisms.

The actual feature delta is very important of course - non blocking, overload protection, run-time filters, flight recorder mode, etc.   That wouldn't require a P&C comparison, some kind of a feature comparison table.

Any other suggestions would be greatly appreciated!

Regards,

/Jim

-----Original Message-----
From: Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com] 
Sent: August-21-13 10:41
To: Jim Dumont
Cc: 'lttng-dev@lists.lttng.org'
Subject: Re: [lttng-dev] LTTng UST vs Syslog, Printf...

* Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> * Jim Dumont (jim.dumont@ericsson.com) wrote:
> > Hi there,
> > 
> > Has anyone done a recent performance and characteristics comparison between lttngust with syslog and printf?   Things like memory & cpu footprint, tps, i/o, disk space...
> 
> Not that I am aware of.
> 
> > 
> > I found this Windriver comparison from 2011:
> > https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1
> > &cad=rja&ved=0CCoQFjAA&url=https%3A%2F%2Fevents.linuxfoundation.org%
> > 2Fslides%2F2011%2Flinuxcon%2Flcna2011_wessel.pdf&ei=Y64TUu6qJqS62AXI
> > qYCYCg&usg=AFQjCNF0Q05MytPYVNWBPnUjB9LEGJfQZA
> 
> This presentation uses UST 0.x. This is _old_: it predates LTTng 2.0.
> 
> > 
> > And if I recall correctly, LTTng UST team also did a printf 
> > comparison a while back, but was wondering if someone has done more 
> > recent prototyping?  Any comparisons with syslog?
> 
> Not at this point. It would be interesting to compare:
> 
> - lttng-ust, both in "discard" and "snapshot" modes, vs
> - printf with timestamp,
> vs
> - syslog
> 
> Especially on multi-core systems, with applications generating a _lot_ 
> of log/trace data.

Just to add to the reflexion, there are key differences between lttng-ust and printf/syslog that makes comparison a bit difficult:

lttng-ust never blocks the application when buffers are full. It either discards events or overwrites the oldest information (flight recorder tracing).

printf and syslog will block the application if the disk I/O is not fast enough.

How would you recommand comparing these ?

Moreover, in order to ensure we can compare those, the printf benchmark would need to grab a time-stamp, and would need to be performed with one call per event (no more), since it is only "atomic" from the point of view of its buffer on a per-call basis. Moreover, printf is not async-signal-safe (as per signal(7)), so it should be noted that it cannot be used from a signal handler (whereas lttng-ust can be used from signal handlers).

Thanks,

Mathieu


> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Regards,
> > 
> > /Jim Dumont
> > 
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: LTTng UST vs Syslog, Printf...

[Michel Dagenais]

 
> I would suggest something relatively simple and basic as far as
> performance & characteristics measurements goes - x apps, y traces /
> second - measure CPU, memory, i/o, file size across the various
> mechanisms.

Yes, something relatively simple should do. For a process writing to syslog, you get a system call, two context switches and a number of other things, as compared to simply writing a few bytes in memory. And you should probably not try to benchmark printk by recording all system calls using it, your kernel will burst.

LTTng UST vs Syslog, Printf...

[Jim Dumont]

Hi there,

Has anyone done a recent performance and characteristics comparison between lttngust with syslog and printf?   Things like memory & cpu footprint, tps, i/o, disk space...

I found this Windriver comparison from 2011: https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCoQFjAA&url=https%3A%2F%2Fevents.linuxfoundation.org%2Fslides%2F2011%2Flinuxcon%2Flcna2011_wessel.pdf&ei=Y64TUu6qJqS62AXIqYCYCg&usg=AFQjCNF0Q05MytPYVNWBPnUjB9LEGJfQZA

And if I recall correctly, LTTng UST team also did a printf comparison a while back, but was wondering if someone has done more recent prototyping?   Any comparisons with syslog?

Regards,

/Jim Dumont