* [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
@ 2013-11-05 21:24 Steven Rostedt
2013-11-06 14:46 ` Steven Rostedt
2013-11-06 19:46 ` Vince Weaver
0 siblings, 2 replies; 7+ messages in thread
From: Steven Rostedt @ 2013-11-05 21:24 UTC (permalink / raw)
To: Linus Torvalds
Cc: LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa, Dave Jones,
Greg Kroah-Hartman, Frederic Weisbecker, Vince Weaver, stable,
Kees Cook
Linus,
Please pull the latest ftrace/urgent tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
ftrace/urgent
Head SHA1: d1356abac98cce8e7765186f8093da1442d74d1d
Steven Rostedt (1):
perf/ftrace: Fix paranoid level for enabling function tracer
----
kernel/trace/trace_event_perf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
---------------------------
commit d1356abac98cce8e7765186f8093da1442d74d1d
Author: Steven Rostedt <rostedt@goodmis.org>
Date: Tue Nov 5 12:51:11 2013 -0500
perf/ftrace: Fix paranoid level for enabling function tracer
The current default perf paranoid level is "1" which has
"perf_paranoid_kernel()" return false, and giving any operations that
use it, access to normal users. Unfortunately, this includes function
tracing and normal users should not be allowed to enable function
tracing by default.
The proper level is defined at "-1" (full perf access), which
"perf_paranoid_tracepoint_raw()" will only give access to. Use that
check instead for enabling function tracing.
Reported-by: Dave Jones <davej@redhat.com>
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: stable@vger.kernel.org # 3.4+
CVE: CVE-2013-2930
Fixes: ced39002f5ea ("ftrace, perf: Add support to use function tracepoint in perf")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
index 80c36bc..78e27e3 100644
--- a/kernel/trace/trace_event_perf.c
+++ b/kernel/trace/trace_event_perf.c
@@ -26,7 +26,7 @@ static int perf_trace_event_perm(struct ftrace_event_call *tp_event,
{
/* The ftrace function trace is allowed only for root. */
if (ftrace_event_is_function(tp_event) &&
- perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
+ perf_paranoid_tracepoint_raw() && !capable(CAP_SYS_ADMIN))
return -EPERM;
/* No tracing, just counting, so no obvious leak */
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
2013-11-05 21:24 [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer Steven Rostedt
@ 2013-11-06 14:46 ` Steven Rostedt
2013-11-06 19:47 ` Steven Rostedt
2013-11-06 19:46 ` Vince Weaver
1 sibling, 1 reply; 7+ messages in thread
From: Steven Rostedt @ 2013-11-06 14:46 UTC (permalink / raw)
To: Linus Torvalds
Cc: LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa, Dave Jones,
Greg Kroah-Hartman, Frederic Weisbecker, Vince Weaver, stable,
Kees Cook
On Tue, 5 Nov 2013 16:24:07 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Linus,
>
> Please pull the latest ftrace/urgent tree, which can be found at:
>
As I usually send you signed tag pull releases, I made one of those too:
Linus,
Please pull the latest ftrace-urgent-3.12 tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
ftrace-urgent-3.12
Tag SHA1: f54bbe2bc77806bf4d544652a2987dd9f784a5ba
Head SHA1: d1356abac98cce8e7765186f8093da1442d74d1d
Steven Rostedt (1):
perf/ftrace: Fix paranoid level for enabling function tracer
----
kernel/trace/trace_event_perf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
---------------------------
commit d1356abac98cce8e7765186f8093da1442d74d1d
Author: Steven Rostedt <rostedt@goodmis.org>
Date: Tue Nov 5 12:51:11 2013 -0500
perf/ftrace: Fix paranoid level for enabling function tracer
The current default perf paranoid level is "1" which has
"perf_paranoid_kernel()" return false, and giving any operations that
use it, access to normal users. Unfortunately, this includes function
tracing and normal users should not be allowed to enable function
tracing by default.
The proper level is defined at "-1" (full perf access), which
"perf_paranoid_tracepoint_raw()" will only give access to. Use that
check instead for enabling function tracing.
Reported-by: Dave Jones <davej@redhat.com>
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: stable@vger.kernel.org # 3.4+
CVE: CVE-2013-2930
Fixes: ced39002f5ea ("ftrace, perf: Add support to use function tracepoint in perf")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
index 80c36bc..78e27e3 100644
--- a/kernel/trace/trace_event_perf.c
+++ b/kernel/trace/trace_event_perf.c
@@ -26,7 +26,7 @@ static int perf_trace_event_perm(struct ftrace_event_call *tp_event,
{
/* The ftrace function trace is allowed only for root. */
if (ftrace_event_is_function(tp_event) &&
- perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
+ perf_paranoid_tracepoint_raw() && !capable(CAP_SYS_ADMIN))
return -EPERM;
/* No tracing, just counting, so no obvious leak */
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
2013-11-05 21:24 [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer Steven Rostedt
2013-11-06 14:46 ` Steven Rostedt
@ 2013-11-06 19:46 ` Vince Weaver
2013-11-06 20:04 ` Vince Weaver
1 sibling, 1 reply; 7+ messages in thread
From: Vince Weaver @ 2013-11-06 19:46 UTC (permalink / raw)
To: Steven Rostedt
Cc: Linus Torvalds, LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
Dave Jones, Greg Kroah-Hartman, Frederic Weisbecker, Kees Cook
I've run this patch on top of 3.12 and it seems pretty solid.
I've run my perf_fuzzer for hours (which wasn't possible w/o
quickly crashing on a stock kernel).
Tested-by: Vince Weaver <vincent.weaver@maine.edu>
On Tue, 5 Nov 2013, Steven Rostedt wrote:
>
> Linus,
>
> Please pull the latest ftrace/urgent tree, which can be found at:
>
>
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
> ftrace/urgent
>
> Head SHA1: d1356abac98cce8e7765186f8093da1442d74d1d
>
>
> Steven Rostedt (1):
> perf/ftrace: Fix paranoid level for enabling function tracer
>
> ----
> kernel/trace/trace_event_perf.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> ---------------------------
> commit d1356abac98cce8e7765186f8093da1442d74d1d
> Author: Steven Rostedt <rostedt@goodmis.org>
> Date: Tue Nov 5 12:51:11 2013 -0500
>
> perf/ftrace: Fix paranoid level for enabling function tracer
>
> The current default perf paranoid level is "1" which has
> "perf_paranoid_kernel()" return false, and giving any operations that
> use it, access to normal users. Unfortunately, this includes function
> tracing and normal users should not be allowed to enable function
> tracing by default.
>
> The proper level is defined at "-1" (full perf access), which
> "perf_paranoid_tracepoint_raw()" will only give access to. Use that
> check instead for enabling function tracing.
>
> Reported-by: Dave Jones <davej@redhat.com>
> Reported-by: Vince Weaver <vincent.weaver@maine.edu>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Jiri Olsa <jolsa@redhat.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: stable@vger.kernel.org # 3.4+
> CVE: CVE-2013-2930
> Fixes: ced39002f5ea ("ftrace, perf: Add support to use function tracepoint in perf")
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
>
> diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
> index 80c36bc..78e27e3 100644
> --- a/kernel/trace/trace_event_perf.c
> +++ b/kernel/trace/trace_event_perf.c
> @@ -26,7 +26,7 @@ static int perf_trace_event_perm(struct ftrace_event_call *tp_event,
> {
> /* The ftrace function trace is allowed only for root. */
> if (ftrace_event_is_function(tp_event) &&
> - perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
> + perf_paranoid_tracepoint_raw() && !capable(CAP_SYS_ADMIN))
> return -EPERM;
>
> /* No tracing, just counting, so no obvious leak */
>
Vince Weaver
vincent.weaver@maine.edu
http://www.eece.maine.edu/~vweaver/
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
2013-11-06 14:46 ` Steven Rostedt
@ 2013-11-06 19:47 ` Steven Rostedt
0 siblings, 0 replies; 7+ messages in thread
From: Steven Rostedt @ 2013-11-06 19:47 UTC (permalink / raw)
To: Steven Rostedt
Cc: Linus Torvalds, LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
Dave Jones, Greg Kroah-Hartman, Frederic Weisbecker,
Vince Weaver, stable, Kees Cook
On Wed, 6 Nov 2013 09:46:49 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
>
> As I usually send you signed tag pull releases, I made one of those too:
>
>
I just included Vince's tested-by tag. Here's a new tag you can pull.
-- Steve
Linus,
Please pull the latest ftrace-urgent-3.12-v2 tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
ftrace-urgent-3.12-v2
Tag SHA1: 84665f63eaea429fc56c9bc8c1b4e2da1ed697d5
Head SHA1: 12ae030d54ef250706da5642fc7697cc60ad0df7
Steven Rostedt (1):
perf/ftrace: Fix paranoid level for enabling function tracer
----
kernel/trace/trace_event_perf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
---------------------------
commit 12ae030d54ef250706da5642fc7697cc60ad0df7
Author: Steven Rostedt <rostedt@goodmis.org>
Date: Tue Nov 5 12:51:11 2013 -0500
perf/ftrace: Fix paranoid level for enabling function tracer
The current default perf paranoid level is "1" which has
"perf_paranoid_kernel()" return false, and giving any operations that
use it, access to normal users. Unfortunately, this includes function
tracing and normal users should not be allowed to enable function
tracing by default.
The proper level is defined at "-1" (full perf access), which
"perf_paranoid_tracepoint_raw()" will only give access to. Use that
check instead for enabling function tracing.
Reported-by: Dave Jones <davej@redhat.com>
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Tested-by: Vince Weaver <vincent.weaver@maine.edu>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: stable@vger.kernel.org # 3.4+
CVE: CVE-2013-2930
Fixes: ced39002f5ea ("ftrace, perf: Add support to use function tracepoint in perf")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
diff --git a/kernel/trace/trace_event_perf.c b/kernel/trace/trace_event_perf.c
index 80c36bc..78e27e3 100644
--- a/kernel/trace/trace_event_perf.c
+++ b/kernel/trace/trace_event_perf.c
@@ -26,7 +26,7 @@ static int perf_trace_event_perm(struct ftrace_event_call *tp_event,
{
/* The ftrace function trace is allowed only for root. */
if (ftrace_event_is_function(tp_event) &&
- perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
+ perf_paranoid_tracepoint_raw() && !capable(CAP_SYS_ADMIN))
return -EPERM;
/* No tracing, just counting, so no obvious leak */
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
2013-11-06 19:46 ` Vince Weaver
@ 2013-11-06 20:04 ` Vince Weaver
2013-11-06 20:04 ` Steven Rostedt
0 siblings, 1 reply; 7+ messages in thread
From: Vince Weaver @ 2013-11-06 20:04 UTC (permalink / raw)
To: Vince Weaver
Cc: Steven Rostedt, Linus Torvalds, LKML, Ingo Molnar,
Peter Zijlstra, Jiri Olsa, Dave Jones, Greg Kroah-Hartman,
Frederic Weisbecker, Kees Cook
On Wed, 6 Nov 2013, Vince Weaver wrote:
> I've run this patch on top of 3.12 and it seems pretty solid.
> I've run my perf_fuzzer for hours (which wasn't possible w/o
> quickly crashing on a stock kernel).
It turns out there are still tracepoint issues though.
I managed to lock my machine hard, even with the patch, but it took a few
hours to trigger:
[11559.476002] WARNING: CPU: 0 PID: 2994 at kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
[11559.476002] Watchdog detected hard LOCKUP on cpu 0
[11559.476002] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative f71882fg evdev coretemp acpi_cpufreq psmouse serio_raw video wmi pcspkr processor thermal_sys button mcs7830 i2c_nforce2 ohci_pci usbnet ohci_hcd sg ehci_pci ehci_hcd sd_mod usbcore usb_common
[11559.476002] CPU: 0 PID: 2994 Comm: perf_fuzzer Tainted: G W 3.12.0 #2
[11559.476002] Hardware name: AOpen DE7000/nMCP7ALPx-DE R1.06 Oct.19.2012, BIOS 080015 10/19/2012
[11559.476002] 00000000000000f5 ffff88011fc07c48 ffffffff8151aad6 ffff88011fc07c58
[11559.476002] ffffffff8103cd68 ffff88011fc07c78 ffffffff810a101c ffff88011b313400
[11559.476002] ffff88011fc07ef8 ffff88011fc07d58 0000000000000000 ffff88011fc07ca8
[11559.476002] Call Trace:
[11559.476002] <NMI> [<ffffffff8151aad6>] ? dump_stack+0x41/0x53
[11559.476002] [<ffffffff8103cd68>] ? warn_slowpath_common+0x81/0x9c
[11559.476002] [<ffffffff810a101c>] ? watchdog_overflow_callback+0x9b/0xa6
[11559.476002] [<ffffffff8103ce25>] ? warn_slowpath_fmt+0x46/0x48
[11559.476002] [<ffffffff810c9e5b>] ? perf_prepare_sample+0x1d0/0x2f4
[11559.476002] [<ffffffff810a101c>] ? watchdog_overflow_callback+0x9b/0xa6
[11559.476002] [<ffffffff810ca4a2>] ? __perf_event_overflow+0x12e/0x1ab
[11559.476002] [<ffffffff810167d3>] ? intel_pmu_lbr_read+0x253/0x349
[11559.476002] [<ffffffff81018c95>] ? intel_pmu_handle_irq+0x2b8/0x34d
[11559.476002] [<ffffffff8101866a>] ? intel_pmu_enable_all+0x4e/0xa7
[11559.476002] [<ffffffff8151edfc>] ? nmi_handle+0xa8/0x13a
[11559.476002] [<ffffffff8151f473>] ? perf_event_nmi_handler+0x24/0x3d
[11559.476002] [<ffffffff8151edb2>] ? nmi_handle+0x5e/0x13a
[11559.476002] [<ffffffff8151ef35>] ? do_nmi+0xa7/0x2bd
[11559.476002] [<ffffffff8151e537>] ? end_repeat_nmi+0x1e/0x2e
[11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
[11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
[11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
[11559.476002] <<EOE>> <IRQ> [<ffffffff8104acaa>] ? __send_signal+0xd8/0x26f
[11559.476002] [<ffffffff8104b074>] ? do_send_sig_info+0x4b/0x75
[11559.476002] [<ffffffff810ca8a5>] ? perf_tp_event+0xe9/0x1f6
[11559.476002] [<ffffffff8111f65e>] ? send_sigio_to_task+0xc6/0xf5
[11559.476002] [<ffffffff8111f737>] ? send_sigio+0xaa/0xdd
[11559.476002] [<ffffffff8111f7d3>] ? kill_fasync+0x69/0x86
[11559.476002] [<ffffffff810cabf9>] ? perf_event_wakeup+0x8c/0xa1
[11559.476002] [<ffffffff810c494d>] ? __irq_work_run+0x5f/0x7b
[11559.476002] [<ffffffff81005f82>] ? smp_trace_irq_work_interrupt+0x4b/0x7e
[11559.476002] [<ffffffff815261ca>] ? trace_irq_work_interrupt+0x6a/0x70
[11559.476002] [<ffffffff810461bf>] ? perf_trace_timer_class+0x39/0x98
[11559.476002] [<ffffffff8151da07>] ? _raw_spin_unlock_irqrestore+0xf/0x11
[11559.476002] [<ffffffff810a3a89>] ? rcu_process_callbacks+0x4a4/0x4b8
[11559.476002] [<ffffffff8105b8f3>] ? enqueue_hrtimer+0x51/0x7b
[11559.476002] [<ffffffff81041027>] ? __do_softirq+0x102/0x208
[11559.476002] [<ffffffff81027608>] ? lapic_next_event+0x1d/0x21
[11559.476002] [<ffffffff8105c187>] ? hrtimer_interrupt+0x119/0x1c6
[11559.476002] [<ffffffff815263dc>] ? call_softirq+0x1c/0x30
[11559.476002] [<ffffffff810041b9>] ? do_softirq+0x38/0x6d
[11559.476002] [<ffffffff81040d1c>] ? irq_exit+0x45/0xa1
[11559.476002] [<ffffffff81526a14>] ? smp_trace_apic_timer_interrupt+0x83/0x91
[11559.476002] [<ffffffff8152597a>] ? trace_apic_timer_interrupt+0x6a/0x70
[11559.476002] <EOI>
[11559.476002] ---[ end trace 1274fa8d9fb13c66 ]---
[11559.476002] perf samples too long (7791497 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
[11559.476002] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 1025.396 msecs
[11570.756001] perf samples too long (7730632 > 20000), lowering kernel.perf_event_max_sample_rate to 6250
[11571.580001] perf samples too long (7670242 > 40000), lowering kernel.perf_event_max_sample_rate to 3250
[11572.428001] perf samples too long (7610324 > 76923), lowering kernel.perf_event_max_sample_rate to 1750
[11573.276001] perf samples too long (7550874 > 142857), lowering kernel.perf_event_max_sample_rate to 1000
[11574.124001] perf samples too long (7491888 > 250000), lowering kernel.perf_event_max_sample_rate to 500
[11574.968001] perf samples too long (7433363 > 500000), lowering kernel.perf_event_max_sample_rate to 250
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer
2013-11-06 20:04 ` Vince Weaver
@ 2013-11-06 20:04 ` Steven Rostedt
2013-11-06 22:00 ` a different perf tracepoint bug Vince Weaver
0 siblings, 1 reply; 7+ messages in thread
From: Steven Rostedt @ 2013-11-06 20:04 UTC (permalink / raw)
To: Vince Weaver
Cc: Linus Torvalds, LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
Dave Jones, Greg Kroah-Hartman, Frederic Weisbecker, Kees Cook
On Wed, 6 Nov 2013 15:04:17 -0500 (EST)
Vince Weaver <vincent.weaver@maine.edu> wrote:
> On Wed, 6 Nov 2013, Vince Weaver wrote:
>
> > I've run this patch on top of 3.12 and it seems pretty solid.
> > I've run my perf_fuzzer for hours (which wasn't possible w/o
> > quickly crashing on a stock kernel).
>
> It turns out there are still tracepoint issues though.
This is a different bug.
>
> I managed to lock my machine hard, even with the patch, but it took a few
> hours to trigger:
>
> [11559.476002] WARNING: CPU: 0 PID: 2994 at kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
> [11559.476002] Watchdog detected hard LOCKUP on cpu 0
> [11559.476002] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative f71882fg evdev coretemp acpi_cpufreq psmouse serio_raw video wmi pcspkr processor thermal_sys button mcs7830 i2c_nforce2 ohci_pci usbnet ohci_hcd sg ehci_pci ehci_hcd sd_mod usbcore usb_common
> [11559.476002] CPU: 0 PID: 2994 Comm: perf_fuzzer Tainted: G W 3.12.0 #2
> [11559.476002] Hardware name: AOpen DE7000/nMCP7ALPx-DE R1.06 Oct.19.2012, BIOS 080015 10/19/2012
> [11559.476002] 00000000000000f5 ffff88011fc07c48 ffffffff8151aad6 ffff88011fc07c58
> [11559.476002] ffffffff8103cd68 ffff88011fc07c78 ffffffff810a101c ffff88011b313400
> [11559.476002] ffff88011fc07ef8 ffff88011fc07d58 0000000000000000 ffff88011fc07ca8
> [11559.476002] Call Trace:
> [11559.476002] <NMI> [<ffffffff8151aad6>] ? dump_stack+0x41/0x53
> [11559.476002] [<ffffffff8103cd68>] ? warn_slowpath_common+0x81/0x9c
> [11559.476002] [<ffffffff810a101c>] ? watchdog_overflow_callback+0x9b/0xa6
> [11559.476002] [<ffffffff8103ce25>] ? warn_slowpath_fmt+0x46/0x48
> [11559.476002] [<ffffffff810c9e5b>] ? perf_prepare_sample+0x1d0/0x2f4
> [11559.476002] [<ffffffff810a101c>] ? watchdog_overflow_callback+0x9b/0xa6
> [11559.476002] [<ffffffff810ca4a2>] ? __perf_event_overflow+0x12e/0x1ab
> [11559.476002] [<ffffffff810167d3>] ? intel_pmu_lbr_read+0x253/0x349
> [11559.476002] [<ffffffff81018c95>] ? intel_pmu_handle_irq+0x2b8/0x34d
> [11559.476002] [<ffffffff8101866a>] ? intel_pmu_enable_all+0x4e/0xa7
> [11559.476002] [<ffffffff8151edfc>] ? nmi_handle+0xa8/0x13a
> [11559.476002] [<ffffffff8151f473>] ? perf_event_nmi_handler+0x24/0x3d
> [11559.476002] [<ffffffff8151edb2>] ? nmi_handle+0x5e/0x13a
> [11559.476002] [<ffffffff8151ef35>] ? do_nmi+0xa7/0x2bd
> [11559.476002] [<ffffffff8151e537>] ? end_repeat_nmi+0x1e/0x2e
> [11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
> [11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
> [11559.476002] [<ffffffff81049982>] ? __sigqueue_alloc+0x9e/0xc4
> [11559.476002] <<EOE>> <IRQ> [<ffffffff8104acaa>] ? __send_signal+0xd8/0x26f
Can you find the exact location of __send_signal()?
-- Steve
> [11559.476002] [<ffffffff8104b074>] ? do_send_sig_info+0x4b/0x75
> [11559.476002] [<ffffffff810ca8a5>] ? perf_tp_event+0xe9/0x1f6
> [11559.476002] [<ffffffff8111f65e>] ? send_sigio_to_task+0xc6/0xf5
> [11559.476002] [<ffffffff8111f737>] ? send_sigio+0xaa/0xdd
> [11559.476002] [<ffffffff8111f7d3>] ? kill_fasync+0x69/0x86
> [11559.476002] [<ffffffff810cabf9>] ? perf_event_wakeup+0x8c/0xa1
> [11559.476002] [<ffffffff810c494d>] ? __irq_work_run+0x5f/0x7b
> [11559.476002] [<ffffffff81005f82>] ? smp_trace_irq_work_interrupt+0x4b/0x7e
> [11559.476002] [<ffffffff815261ca>] ? trace_irq_work_interrupt+0x6a/0x70
> [11559.476002] [<ffffffff810461bf>] ? perf_trace_timer_class+0x39/0x98
> [11559.476002] [<ffffffff8151da07>] ? _raw_spin_unlock_irqrestore+0xf/0x11
> [11559.476002] [<ffffffff810a3a89>] ? rcu_process_callbacks+0x4a4/0x4b8
> [11559.476002] [<ffffffff8105b8f3>] ? enqueue_hrtimer+0x51/0x7b
> [11559.476002] [<ffffffff81041027>] ? __do_softirq+0x102/0x208
> [11559.476002] [<ffffffff81027608>] ? lapic_next_event+0x1d/0x21
> [11559.476002] [<ffffffff8105c187>] ? hrtimer_interrupt+0x119/0x1c6
> [11559.476002] [<ffffffff815263dc>] ? call_softirq+0x1c/0x30
> [11559.476002] [<ffffffff810041b9>] ? do_softirq+0x38/0x6d
> [11559.476002] [<ffffffff81040d1c>] ? irq_exit+0x45/0xa1
> [11559.476002] [<ffffffff81526a14>] ? smp_trace_apic_timer_interrupt+0x83/0x91
> [11559.476002] [<ffffffff8152597a>] ? trace_apic_timer_interrupt+0x6a/0x70
> [11559.476002] <EOI>
> [11559.476002] ---[ end trace 1274fa8d9fb13c66 ]---
> [11559.476002] perf samples too long (7791497 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
> [11559.476002] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 1025.396 msecs
> [11570.756001] perf samples too long (7730632 > 20000), lowering kernel.perf_event_max_sample_rate to 6250
> [11571.580001] perf samples too long (7670242 > 40000), lowering kernel.perf_event_max_sample_rate to 3250
> [11572.428001] perf samples too long (7610324 > 76923), lowering kernel.perf_event_max_sample_rate to 1750
> [11573.276001] perf samples too long (7550874 > 142857), lowering kernel.perf_event_max_sample_rate to 1000
> [11574.124001] perf samples too long (7491888 > 250000), lowering kernel.perf_event_max_sample_rate to 500
> [11574.968001] perf samples too long (7433363 > 500000), lowering kernel.perf_event_max_sample_rate to 250
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: a different perf tracepoint bug
2013-11-06 20:04 ` Steven Rostedt
@ 2013-11-06 22:00 ` Vince Weaver
0 siblings, 0 replies; 7+ messages in thread
From: Vince Weaver @ 2013-11-06 22:00 UTC (permalink / raw)
To: Steven Rostedt
Cc: Linus Torvalds, LKML, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
Dave Jones, Greg Kroah-Hartman, Frederic Weisbecker, Kees Cook
On Wed, 6 Nov 2013, Steven Rostedt wrote:
> > [11559.476002] <<EOE>> <IRQ> [<ffffffff8104acaa>] ? __send_signal+0xd8/0x26f
>
> Can you find the exact location of __send_signal()?
I'm not sure what you mean, but:
addr2line -e ./vmlinux ffffffff8104acaa
linux-kernel/linux-3.12/kernel/signal.c:1095
which is
q = __sigqueue_alloc(sig, t, GFP_ATOMIC | __GFP_NOTRACK_FALSE_POSITIVE,
override_rlimit);
--> if (q) {
list_add_tail(&q->list, &pending->list);
switch ((unsigned long) info) {
case (unsigned long) SEND_SIG_NOINFO:
...
though this might just be a case where tracepoints are happening too
quickly and the kernel gets stuck and can't keep up. Previously I was
using a hacked kernel that made the watchdog continually print rather than
just warn once, the better to see if any progress was being made.
Vince
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-11-06 21:52 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-05 21:24 [GIT PULL][PATCH] perf/ftrace: Fix paranoid level for enabling function tracer Steven Rostedt
2013-11-06 14:46 ` Steven Rostedt
2013-11-06 19:47 ` Steven Rostedt
2013-11-06 19:46 ` Vince Weaver
2013-11-06 20:04 ` Vince Weaver
2013-11-06 20:04 ` Steven Rostedt
2013-11-06 22:00 ` a different perf tracepoint bug Vince Weaver
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).