Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

[Douglas Santos]

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing kernel
tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow,
but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Average results with tracing "on":

lttng:  220 ns
ftrace: 260 ns
perf:   740 ns


E5405 system
kernel 2.6.36

-lttng 0.239 + 0.19.2modules + sys_getuid tracepoint + sys_getuid probe
 + remove syscall_trace

-ftrace and perf + sys_getuid tracepoint

Re: Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

[Steven Rostedt]

On Wed, 2010-11-17 at 16:31 -0500, Douglas Santos wrote:
> Hi all,
> 
> This is a response to a benchmark, submitted a few weeks ago, comparing kernel
> tracing options.
> http://lkml.org/lkml/2010/10/28/422
> 
> We followed the methodology described in the link bellow,
> but using the shellscripts posted there to reproduce autotest scripts.
> http://lkml.org/lkml/2010/10/28/261
> 
> We disabled the extra syscall tracing on lttng, for a fair comparison.
> http://lkml.org/lkml/2010/10/28/290
> 
> Average results with tracing "on":
> 
> lttng:  220 ns
> ftrace: 260 ns

Heh, so ftrace got worse with the new kernel?

-- Steve

> perf:   740 ns
> 
> 
> E5405 system
> kernel 2.6.36
> 
> -lttng 0.239 + 0.19.2modules + sys_getuid tracepoint + sys_getuid probe
>  + remove syscall_trace
> 
> -ftrace and perf + sys_getuid tracepoint

Re: Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

[Douglas Santos]

Quoting Steven Rostedt <rostedt@goodmis.org>:

> On Wed, 2010-11-17 at 16:31 -0500, Douglas Santos wrote:
> > Hi all,
> >
> > This is a response to a benchmark, submitted a few weeks ago, comparing
> kernel
> > tracing options.
> > http://lkml.org/lkml/2010/10/28/422
> >
> > We followed the methodology described in the link bellow,
> > but using the shellscripts posted there to reproduce autotest scripts.
> > http://lkml.org/lkml/2010/10/28/261
> >
> > We disabled the extra syscall tracing on lttng, for a fair comparison.
> > http://lkml.org/lkml/2010/10/28/290
> >
> > Average results with tracing "on":
> >
> > lttng:  220 ns
> > ftrace: 260 ns
>
> Heh, so ftrace got worse with the new kernel?

The previous bench was doing tracing "on" minus "off"
average results. They also used autotest scripts, not sure if
it does exactly the same thing.

I'll check if we missed something.

Re: Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

[David Sharp]

On Wed, Nov 17, 2010 at 2:56 PM, Douglas Santos
<douglas.santos@polymtl.ca> wrote:
> Quoting Steven Rostedt <rostedt@goodmis.org>:
>> On Wed, 2010-11-17 at 16:31 -0500, Douglas Santos wrote:
>> > Hi all,
>> >
>> > This is a response to a benchmark, submitted a few weeks ago, comparing
>> kernel
>> > tracing options.
>> > http://lkml.org/lkml/2010/10/28/422
>> >
>> > We followed the methodology described in the link bellow,
>> > but using the shellscripts posted there to reproduce autotest scripts.
>> > http://lkml.org/lkml/2010/10/28/261
>> >
>> > We disabled the extra syscall tracing on lttng, for a fair comparison.
>> > http://lkml.org/lkml/2010/10/28/290
>> >
>> > Average results with tracing "on":
>> >
>> > lttng:  220 ns
>> > ftrace: 260 ns
>>
>> Heh, so ftrace got worse with the new kernel?

Steve, can you explain how you're drawing that conclusion? Did Douglas
run this benchmark before on a previous kernel (I didn't see it if
so)?

- You can't directly compare to my results because of different hardware.
- The methodology for lttng is different (syscall tracing was removed).
- My results were also on 2.6.36

> The previous bench was doing tracing "on" minus "off"
> average results. They also used autotest scripts, not sure if
> it does exactly the same thing.

I think the subtraction is important, or it is at least important to
see what the "off" result is as a baseline of comparison. Otherwise, a
huge portion of the measurement is the cost of making the syscall
itself.

>
> I'll check if we missed something.
>
>
>

Re: Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

[Steven Rostedt]

On Wed, 2010-11-17 at 15:22 -0800, David Sharp wrote:
> On Wed, Nov 17, 2010 at 2:56 PM, Douglas Santos
> <douglas.santos@polymtl.ca> wrote:
> > Quoting Steven Rostedt <rostedt@goodmis.org>:
> >> On Wed, 2010-11-17 at 16:31 -0500, Douglas Santos wrote:
> >> > Hi all,
> >> >
> >> > This is a response to a benchmark, submitted a few weeks ago, comparing
> >> kernel
> >> > tracing options.
> >> > http://lkml.org/lkml/2010/10/28/422
> >> >
> >> > We followed the methodology described in the link bellow,
> >> > but using the shellscripts posted there to reproduce autotest scripts.
> >> > http://lkml.org/lkml/2010/10/28/261
> >> >
> >> > We disabled the extra syscall tracing on lttng, for a fair comparison.
> >> > http://lkml.org/lkml/2010/10/28/290
> >> >
> >> > Average results with tracing "on":
> >> >
> >> > lttng:  220 ns
> >> > ftrace: 260 ns
> >>
> >> Heh, so ftrace got worse with the new kernel?
> 
> Steve, can you explain how you're drawing that conclusion? Did Douglas
> run this benchmark before on a previous kernel (I didn't see it if
> so)?

Oops, no, I was thinking that this was from your tests. I remember
asking you to try the new kernel. I think I got you and Douglas
confused :-)

-- Steve