All of lore.kernel.org
 help / color / mirror / Atom feed
From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
To: Ingo Molnar <mingo@kernel.org>
Cc: "Frank Ch. Eigler" <fche@redhat.com>,
	linux-arch@vger.kernel.org,
	Ananth N Mavinakayanahalli <ananth@in.ibm.com>,
	Sandeepa Prabhu <sandeepa.prabhu@linaro.org>,
	x86@kernel.org, lkml <linux-kernel@vger.kernel.org>,
	"Steven Rostedt (Red Hat)" <rostedt@goodmis.org>,
	systemtap@sourceware.org, "David S. Miller" <davem@davemloft.net>
Subject: Re: [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs
Date: Mon, 10 Feb 2014 13:02:41 +0900	[thread overview]
Message-ID: <52F84F61.7060103@hitachi.com> (raw)
In-Reply-To: <20140209143709.GA31608@gmail.com>

(2014/02/09 23:37), Ingo Molnar wrote:
> 
> * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> I guess the second reason is why the stap takes so long time to set 
>> probes. stap tries to register kprobes without disabled flag, that 
>> means we enables thousands of probes (overheads).
>>
>> So the similar thing happens when we enables events as below;
>>
>>   # for i in /sys/kernel/debug/tracing/events/kprobes/* ; do date; echo 1 > $i; done
>>   Wed Jan 29 10:44:50 UTC 2014
>>   ...
>>
>> I tried it and canceled after 4 min passed. It enabled about 17k 
>> events and slowed down my system very much(I almost got hang check 
>> timer).
> 
> Ok, I guess that's the slowdown bug that Frank reported.
> 
>> I think we should have some performance statistics (hit count?) and 
>> if it goes over a threshold, we should stop enabling other events.
> 
> That really feels like a hack. How about fixing the root cause? Does 
> the enabling of all probes have to be so slow?

When I tried to use perf top, most of the time was consumed in
kprobe_ftrace_handler and optimized_callback, both of them
are the handler of kprobes. Since I just tried on a VM guest and it
didn't support NMI nor PMU, thus I have to use a bare metal machine
for deeper investigation (and I'll do).
And yes, it seems that the performance problem comes from probing
and tracing itself.

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com



  parent reply	other threads:[~2014-02-10  4:02 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-12-19  9:03 [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs Masami Hiramatsu
2013-12-19  9:03 ` [PATCH -tip v6 01/22] kprobes: Prohibit probing on .entry.text code Masami Hiramatsu
2013-12-19  9:03 ` [PATCH -tip v6 02/22] kprobes: Introduce NOKPROBE_SYMBOL() macro for blacklist Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 03/22] [BUGFIX] kprobes/x86: Prohibit probing on debug_stack_* Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 04/22] [BUGFIX] x86: Prohibit probing on native_set_debugreg/load_idt Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 05/22] [BUGFIX] x86: Prohibit probing on thunk functions and restore Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 06/22] [BUGFIX] x86: Prohibit probing on memcpy/memset Masami Hiramatsu
2013-12-19  9:37   ` Jovi Zhangwei
2013-12-20  2:37     ` Masami Hiramatsu
2013-12-20  3:07       ` Jovi Zhangwei
2013-12-20  4:42         ` Masami Hiramatsu
2013-12-20  8:31           ` Jovi Zhangwei
2013-12-20  9:21             ` Masami Hiramatsu
2013-12-23  4:51               ` Jovi Zhangwei
2013-12-23 10:59                 ` Masami Hiramatsu
2013-12-24  6:39                   ` Jovi Zhangwei
2013-12-24  8:32                     ` Masami Hiramatsu
2013-12-24  9:53                       ` Jovi Zhangwei
2013-12-24 15:58                         ` Masami Hiramatsu
2013-12-25 14:44                           ` Jovi Zhangwei
2013-12-19  9:04 ` [PATCH -tip v6 07/22] kprobes/x86: Call exception handlers directly from do_int3/do_debug Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 08/22] kprobes/x86: Allow probe on some kprobe preparation functions Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 09/22] kprobes: Allow probe on some kprobe functions Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 10/22] ftrace/kprobes: Allow probing on some preparation functions Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 11/22] x86: Allow kprobes on text_poke/hw_breakpoint Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 12/22] x86: Use NOKPROBE_SYMBOL() instead of __kprobes annotation Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 13/22] kprobes: Use NOKPROBE_SYMBOL macro instead of __kprobes Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 14/22] ftrace/kprobes: Use NOKPROBE_SYMBOL macro in ftrace Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 15/22] notifier: Use NOKPROBE_SYMBOL macro in notifier Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 16/22] sched: Use NOKPROBE_SYMBOL macro in sched Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 17/22] kprobes: Show blacklist entries via debugfs Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 18/22] kprobes: Support blacklist functions in module Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 19/22] kprobes: Use NOKPROBE_SYMBOL() in sample modules Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 20/22] kprobes/x86: Use kprobe_blacklist for .kprobes.text and .entry.text Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 21/22] kprobes/x86: Remove unneeded preempt_disable/enable in interrupt handlers Masami Hiramatsu
2013-12-19  9:04 ` [PATCH -tip v6 22/22] [RFC] kprobes/x86: Add emergency recovery process for bad kprobes Masami Hiramatsu
2013-12-19 20:46 ` [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs Frank Ch. Eigler
2013-12-20  4:21   ` Masami Hiramatsu
2013-12-20  8:20     ` Ingo Molnar
2013-12-20  9:31       ` Masami Hiramatsu
2013-12-20 10:46         ` Ingo Molnar
2013-12-22 21:10           ` Masami Hiramatsu
2013-12-23 13:04             ` Ingo Molnar
2013-12-24  1:19               ` Masami Hiramatsu
2014-01-29 11:22           ` Masami Hiramatsu
2014-02-09 14:37             ` Ingo Molnar
2014-02-09 21:27               ` Frank Ch. Eigler
2014-02-10  4:02               ` Masami Hiramatsu [this message]
2014-02-10 11:31                 ` Masami Hiramatsu
2013-12-20 13:40       ` Frank Ch. Eigler
2013-12-22 21:32         ` Masami Hiramatsu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=52F84F61.7060103@hitachi.com \
    --to=masami.hiramatsu.pt@hitachi.com \
    --cc=ananth@in.ibm.com \
    --cc=davem@davemloft.net \
    --cc=fche@redhat.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=sandeepa.prabhu@linaro.org \
    --cc=systemtap@sourceware.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.