From: David Sharp <dhsharp@google.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: linux-kernel@vger.kernel.org,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Ingo Molnar <mingo@elte.hu>,
Andrew Morton <akpm@linux-foundation.org>,
Michael Rubin <mrubin@google.com>,
Frederic Weisbecker <fweisbec@gmail.com>
Subject: Re: Benchmarks of kernel tracing options (ftrace and ktrace)
Date: Tue, 16 Nov 2010 15:27:58 -0800 [thread overview]
Message-ID: <AANLkTi=Y2X5EuqiCncPeNefgHs6vvGmnZpsyyddRR2g_@mail.gmail.com> (raw)
In-Reply-To: <1287013830.3673.224.camel@frodo>
On Wed, Oct 13, 2010 at 4:50 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 2010-10-13 at 16:19 -0700, David Sharp wrote:
>> Google uses kernel tracing aggressively in the its data centers. We
>
> Thanks!
>
>> wrote our own kernel tracer, ktrace. However ftrace, perf and LTTng
>> all have a better feature set than ktrace, so we are abandoning that
>> code.
>
> Cool!
>
>>
>> We see several implementations of tracing aimed at the mainline kernel
>> and wanted a fair comparison of each of them to make sure they will
>> not significantly impact performance. A tracing toolkit that is too
>> expensive is not usable in our environment.
>>
>
> [ snip for now (I'm traveling) ]
>
>> This first set of benchmark results compares ftrace to ktrace. The
>> numbers below are the "on" result minus the "off" result for each
>> configuration.
>>
>> ktrace: 200ns (tracepoint: kernel_getuid)
>> ftrace: 224ns (tracepoint: timer:sys_getuid)
>> ftrace: 587ns (tracepoint: syscalls:sys_enter_getuid)
>
>
>> The last result shows that the syscall tracing is about twice as
>> expensive as a normal tracepoint, which is interesting.
>
> Argh, the syscall tracing has a lot of overhead. There is only one
> tracepoint that is hooked into the ptrace code, and will save all
> registers before calling the functions. It enables tracing on all
> syscalls and there's a table that decides whether or not to trace the
> syscall.
>
> So I'm not surprised with the result that the syscall trace point is so
> slow (note, perf uses the same infrastructure).
>
> -- Steve
Steve, Mike told me that you may have some patches to improve overhead
of syscall tracing. I'd be interested in testing them. Can you send
them my way?
Thanks,
d#
next prev parent reply other threads:[~2010-11-16 23:28 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <AANLkTikfYy-kYb1=KbsYwHd0_vcf20d2nPSfFynugz8z@mail.gmail.com>
[not found] ` <AANLkTikKwx6okpX4pxVzTvrVNm=KhUvLQGs0_ziwo6fX@mail.gmail.com>
2010-10-13 23:19 ` Benchmarks of kernel tracing options (ftrace and ktrace) David Sharp
2010-10-13 23:50 ` Steven Rostedt
2010-10-14 0:00 ` Mathieu Desnoyers
2010-10-14 11:27 ` Frederic Weisbecker
2010-10-14 18:04 ` Jason Baron
2010-11-16 23:27 ` David Sharp [this message]
2010-11-16 23:32 ` Steven Rostedt
2010-11-19 1:11 ` Michael Rubin
2010-11-19 1:41 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='AANLkTi=Y2X5EuqiCncPeNefgHs6vvGmnZpsyyddRR2g_@mail.gmail.com' \
--to=dhsharp@google.com \
--cc=akpm@linux-foundation.org \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@elte.hu \
--cc=mrubin@google.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.