From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
To: Ingo Molnar <mingo@kernel.org>
Cc: linux-kernel@vger.kernel.org, Andi Kleen <andi@firstfloor.org>,
Ananth N Mavinakayanahalli <ananth@in.ibm.com>,
Sandeepa Prabhu <sandeepa.prabhu@linaro.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
x86@kernel.org, Steven Rostedt <rostedt@goodmis.org>,
fche@redhat.com, mingo@redhat.com, systemtap@sourceware.org,
"H. Peter Anvin" <hpa@zytor.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits
Date: Thu, 24 Apr 2014 20:38:11 +0900 [thread overview]
Message-ID: <5358F7A3.1000108@hitachi.com> (raw)
In-Reply-To: <20140424090134.GC7768@gmail.com>
(2014/04/24 18:01), Ingo Molnar wrote:
>
> * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>> Introduce kprobe cache to reduce cache misshits for
>> massive multiple kprobes.
>> For stress testing kprobes, we need to activate kprobes
>> as many as possible. This situation causes cache miss
>> hit storm on kprobe hash-list. kprobe hashlist is already
>> enlarged to 4k entries and this is still small for 40k
>> kprobes.
>>
>> For example, when registering 40k probes on the hlist and
>> enabling 20k probes, perf tools shows still a lot of
>> cache-misses are on the get_kprobe.
>> ----
>> Samples: 633 of event 'cache-misses', Event count (approx.): 3414776
>> + 68.13% [k] get_kprobe
>> + 4.38% [k] ftrace_lookup_ip
>> + 2.54% [k] kprobe_ftrace_handler
>> ----
>>
>> Also, I found that the most of the kprobes are not hit.
>> In that case, to reduce cache-misses, we can reduce the
>> random memory access by introducing a per-cpu cache which
>> caches the address of frequently used kprobe data structure
>> and its probe address.
>>
>> With kpcache enabled, the get_kprobe_cached goes down to
>> around 4-5% of cache-misses with 20k probes.
>> ----
>> Samples: 729 of event 'cache-misses', Event count (approx.): 690125
>> + 14.49% [k] ftrace_lookup_ip
>> + 5.61% [k] kprobe_trace_func
>> + 5.17% [k] kprobe_ftrace_handler
>> + 4.62% [k] get_kprobe_cached
>> ----
>>
>> Of course this reduces the enabling time too.
>>
>> Without this fix (just enlarge hash table):
>> (2934 sec, 1 min intervals for each 2000 probes enabled)
>>
>> ----
>> Enabling trace events: start at 1393921862
>> 0 1393921864 a2mp_chan_alloc_skb_cb_38581
>> ...
>> 19999 1393924928 nfs4_open_confirm_done_11785
>> ----
>>
>> With this fix:
>> (2025 sec, 1 min intervals for each 2000 probes enabled)
>
> That's a nice speedup.
Thanks :)
>
> So I don't think this should be a Kconfig entry, just enable it
> unconditionally. That will further simplify the code.
Hmm, it consumes some amount of memory (36KB/core) just for the
case of several thousand of kprobes. On enterprise servers and desktop
it's OK, no problem. But I think, some embedded systems with small
resources will not want that. So, how about enabling Kconfig by default?
Thank you,
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
next prev parent reply other threads:[~2014-04-24 11:38 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-17 8:16 [PATCH -tip v9 00/26] kprobes: introduce NOKPROBE_SYMBOL, bugfixes and scalbility efforts Masami Hiramatsu
2014-04-17 8:16 ` [PATCH -tip v9 01/26] [BUGFIX]kprobes/x86: Fix page-fault handling logic Masami Hiramatsu
2014-04-17 9:58 ` [tip:perf/urgent] kprobes/x86: " tip-bot for Masami Hiramatsu
2014-04-17 8:16 ` [PATCH -tip v9 02/26] kprobes/x86: Allow to handle reentered kprobe on singlestepping Masami Hiramatsu
2014-04-24 10:57 ` [tip:perf/kprobes] kprobes/x86: Allow to handle reentered kprobe on single-stepping tip-bot for Masami Hiramatsu
2014-04-17 8:16 ` [PATCH -tip v9 03/26] kprobes: Prohibit probing on .entry.text code Masami Hiramatsu
2014-04-24 10:57 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 04/26] kprobes: Introduce NOKPROBE_SYMBOL() macro for blacklist Masami Hiramatsu
2014-04-24 10:58 ` [tip:perf/kprobes] kprobes: Introduce NOKPROBE_SYMBOL() macro to maintain kprobes blacklist tip-bot for Masami Hiramatsu
2014-05-01 5:26 ` kprobes broken in linux-next (was Re: [tip:perf/kprobes] kprobes: Introduce NOKPROBE_SYMBOL() macro to maintain kprobes blacklist) Vineet Gupta
2014-05-02 1:13 ` Masami Hiramatsu
2014-05-07 4:56 ` Vineet Gupta
2014-05-07 19:18 ` [tip:perf/kprobes] kprobes: Ensure blacklist data is aligned tip-bot for Vineet Gupta
2014-05-05 20:48 ` [tip:perf/kprobes] kprobes: Introduce NOKPROBE_SYMBOL() macro to maintain kprobes blacklist Tony Luck
2014-05-06 9:25 ` Masami Hiramatsu
2014-05-06 10:03 ` Masami Hiramatsu
2014-05-07 11:19 ` Masami Hiramatsu
2014-05-07 11:55 ` [RFT PATCH -next ] [BUGFIX] kprobes: Fix "Failed to find blacklist" error on ia64 and ppc64 Masami Hiramatsu
2014-05-07 11:59 ` Masami Hiramatsu
2014-05-14 8:19 ` Masami Hiramatsu
2014-05-08 4:47 ` Ananth N Mavinakayanahalli
2014-05-08 5:40 ` Masami Hiramatsu
2014-05-08 6:16 ` Ananth N Mavinakayanahalli
2014-05-09 8:06 ` Masami Hiramatsu
2014-05-26 11:25 ` Suzuki K. Poulose
2014-05-26 11:48 ` Masami Hiramatsu
2014-05-27 6:31 ` [RFT PATCH -next v2] " Masami Hiramatsu
2014-05-29 19:13 ` Suzuki K. Poulose
2014-05-30 2:47 ` Masami Hiramatsu
2014-05-30 3:18 ` [RFT PATCH -next v3] " Masami Hiramatsu
2014-06-06 6:38 ` Masami Hiramatsu
2014-06-17 23:03 ` Tony Luck
2014-06-18 7:56 ` Michael Ellerman
2014-06-18 8:46 ` Masami Hiramatsu
2014-06-19 1:30 ` Michael Ellerman
2014-06-19 4:52 ` Masami Hiramatsu
2014-06-19 6:40 ` Suzuki K. Poulose
2014-06-19 7:26 ` Masami Hiramatsu
2014-06-19 9:45 ` Suzuki K. Poulose
2014-06-19 11:01 ` Masami Hiramatsu
2014-06-19 11:20 ` Masami Hiramatsu
2014-06-20 0:37 ` Michael Ellerman
2014-06-20 2:13 ` Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 05/26] [BUGFIX] kprobes/x86: Prohibit probing on debug_stack_* Masami Hiramatsu
2014-04-24 10:58 ` [tip:perf/kprobes] kprobes, x86: Prohibit probing on debug_stack_*() tip-bot for Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 06/26] [BUGFIX] x86: Prohibit probing on native_set_debugreg/load_idt Masami Hiramatsu
2014-04-24 10:58 ` [tip:perf/kprobes] kprobes, x86: Prohibit probing on native_set_debugreg()/load_idt() tip-bot for Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 07/26] [BUGFIX] x86: Prohibit probing on thunk functions and restore Masami Hiramatsu
2014-04-24 10:58 ` [tip:perf/kprobes] kprobes, " tip-bot for Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 08/26] kprobes/x86: Call exception handlers directly from do_int3/do_debug Masami Hiramatsu
2014-04-24 10:59 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-24 11:26 ` Jiri Kosina
2014-04-17 8:17 ` [PATCH -tip v9 09/26] x86: Call exception_enter after kprobes handled Masami Hiramatsu
2014-04-24 10:59 ` [tip:perf/kprobes] kprobes, " tip-bot for Masami Hiramatsu
2014-06-13 17:14 ` Frederic Weisbecker
2014-06-14 5:44 ` Masami Hiramatsu
2014-06-14 6:47 ` [PATCH -tip ] [Bugfix] x86/kprobes: Fix build errors and blacklist context_track_user Masami Hiramatsu
2014-06-14 8:58 ` [tip:perf/urgent] " tip-bot for Masami Hiramatsu
2014-06-16 15:52 ` [PATCH -tip ] [Bugfix] " Frederic Weisbecker
2014-04-17 8:17 ` [PATCH -tip v9 10/26] kprobes/x86: Allow probe on some kprobe preparation functions Masami Hiramatsu
2014-04-24 10:59 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 11/26] kprobes: Allow probe on some kprobe functions Masami Hiramatsu
2014-04-24 10:59 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 12/26] ftrace/*probes: Allow probing on some functions Masami Hiramatsu
2014-04-24 10:59 ` [tip:perf/kprobes] kprobes, ftrace: " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 13/26] x86: Allow kprobes on text_poke/hw_breakpoint Masami Hiramatsu
2014-04-24 11:00 ` [tip:perf/kprobes] kprobes, x86: Allow kprobes on text_poke/ hw_breakpoint tip-bot for Masami Hiramatsu
2014-04-24 11:26 ` Jiri Kosina
2014-04-17 8:18 ` [PATCH -tip v9 14/26] x86: Use NOKPROBE_SYMBOL() instead of __kprobes annotation Masami Hiramatsu
2014-04-24 11:00 ` [tip:perf/kprobes] kprobes, " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 15/26] kprobes: Use NOKPROBE_SYMBOL macro instead of __kprobes Masami Hiramatsu
2014-04-24 11:00 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 16/26] ftrace/kprobes: Use NOKPROBE_SYMBOL macro in ftrace Masami Hiramatsu
2014-04-24 11:00 ` [tip:perf/kprobes] kprobes, ftrace: " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 17/26] notifier: Use NOKPROBE_SYMBOL macro in notifier Masami Hiramatsu
2014-04-17 14:40 ` Josh Triplett
2014-04-24 11:00 ` [tip:perf/kprobes] kprobes, " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 18/26] sched: Use NOKPROBE_SYMBOL macro in sched Masami Hiramatsu
2014-04-24 11:01 ` [tip:perf/kprobes] kprobes, " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 19/26] kprobes: Show blacklist entries via debugfs Masami Hiramatsu
2014-04-24 11:01 ` [tip:perf/kprobes] " tip-bot for Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 20/26] kprobes: Support blacklist functions in module Masami Hiramatsu
2014-04-24 8:56 ` Ingo Molnar
2014-04-24 11:24 ` Masami Hiramatsu
2014-04-25 8:19 ` Ingo Molnar
2014-04-25 10:12 ` Masami Hiramatsu
2014-04-25 10:55 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 21/26] kprobes: Use NOKPROBE_SYMBOL() in sample modules Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 22/26] kprobes/x86: Use kprobe_blacklist for .kprobes.text and .entry.text Masami Hiramatsu
2014-04-24 8:58 ` Ingo Molnar
2014-04-24 11:22 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 23/26] kprobes/x86: Remove unneeded preempt_disable/enable in interrupt handlers Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 24/26] kprobes: Enlarge hash table to 512 entries Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits Masami Hiramatsu
2014-04-24 9:01 ` Ingo Molnar
2014-04-24 11:38 ` Masami Hiramatsu [this message]
2014-04-25 8:20 ` Ingo Molnar
2014-04-25 9:43 ` Masami Hiramatsu
2014-04-26 7:12 ` Ingo Molnar
2014-04-27 12:49 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 26/26] ftrace: Introduce FTRACE_OPS_FL_SELF_FILTER for ftrace-kprobe Masami Hiramatsu
2014-04-17 8:37 ` [PATCH -tip v9 00/26] kprobes: introduce NOKPROBE_SYMBOL, bugfixes and scalbility efforts Ingo Molnar
2014-04-17 8:53 ` Masami Hiramatsu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5358F7A3.1000108@hitachi.com \
--to=masami.hiramatsu.pt@hitachi.com \
--cc=ananth@in.ibm.com \
--cc=andi@firstfloor.org \
--cc=fche@redhat.com \
--cc=fweisbec@gmail.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
--cc=sandeepa.prabhu@linaro.org \
--cc=systemtap@sourceware.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).