* suspicious RCU usage (perf)
@ 2013-08-26 14:58 Dave Jones
2013-08-26 16:29 ` Paul E. McKenney
0 siblings, 1 reply; 17+ messages in thread
From: Dave Jones @ 2013-08-26 14:58 UTC (permalink / raw)
To: Linux Kernel; +Cc: Paul McKenney
Another day, another rcu backtrace..
This says rc6, but it's pretty darn close to rc7, I think it was running
a build from Friday.
[260431.854524] ===============================
[260431.855485] [ INFO: suspicious RCU usage. ]
[260431.856437] 3.11.0-rc6+ #12 Not tainted
[260431.857377] -------------------------------
[260431.858318] include/linux/rcupdate.h:779 rcu_read_lock() used illegally while idle!
[260431.859278] other info that might help us debug this:
[260431.862144] RCU used illegally from idle CPU!
rcu_scheduler_active = 1, debug_locks = 0
[260431.864990] RCU used illegally from extended quiescent state!
[260431.865949] 1 lock held by trinity-kid3/15917:
[260431.866901] #0: (rcu_read_lock){.+.+..}, at: [<ffffffff81146d50>] __perf_event_overflow+0x100/0x320
[260431.867896] stack backtrace:
[260431.869815] CPU: 0 PID: 15917 Comm: trinity-kid3 Not tainted 3.11.0-rc6+ #12
[260431.871885] 0000000000000000 ffff8801095d3ad0 ffffffff816f9c6f ffff88023ce86950
[260431.872912] ffff8801095d3b00 ffffffff810be6a7 ffff880079695cd0 0000000000000000
[260431.873910] ffff8801095d3c00 0000000000000000 ffff8801095d3b80 ffffffff81146ef4
[260431.874909] Call Trace:
[260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74
[260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120
[260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320
[260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320
[260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320
[260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0
[260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90
[260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0
[260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0
[260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0
[260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110
[260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110
[260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f
[260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110
[260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50
[260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0
[260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20
[260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0
[260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150
[260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3d
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: suspicious RCU usage (perf) 2013-08-26 14:58 suspicious RCU usage (perf) Dave Jones @ 2013-08-26 16:29 ` Paul E. McKenney 2013-08-26 17:30 ` Steven Rostedt 0 siblings, 1 reply; 17+ messages in thread From: Paul E. McKenney @ 2013-08-26 16:29 UTC (permalink / raw) To: Dave Jones, Linux Kernel; +Cc: rostedt On Mon, Aug 26, 2013 at 10:58:38AM -0400, Dave Jones wrote: > Another day, another rcu backtrace.. > This says rc6, but it's pretty darn close to rc7, I think it was running > a build from Friday. Could you please send your .config? Also, were you running ftrace, perf, RCU event tracing, or what? Looks like you are running ftrace, but I though Steven had set that up so that it could be called from an extended quiescent state. Thanx, Paul > [260431.854524] =============================== > [260431.855485] [ INFO: suspicious RCU usage. ] > [260431.856437] 3.11.0-rc6+ #12 Not tainted > [260431.857377] ------------------------------- > [260431.858318] include/linux/rcupdate.h:779 rcu_read_lock() used illegally while idle! > [260431.859278] other info that might help us debug this: > [260431.862144] RCU used illegally from idle CPU! > rcu_scheduler_active = 1, debug_locks = 0 > [260431.864990] RCU used illegally from extended quiescent state! > [260431.865949] 1 lock held by trinity-kid3/15917: > [260431.866901] #0: (rcu_read_lock){.+.+..}, at: [<ffffffff81146d50>] __perf_event_overflow+0x100/0x320 > [260431.867896] stack backtrace: > [260431.869815] CPU: 0 PID: 15917 Comm: trinity-kid3 Not tainted 3.11.0-rc6+ #12 > [260431.871885] 0000000000000000 ffff8801095d3ad0 ffffffff816f9c6f ffff88023ce86950 > [260431.872912] ffff8801095d3b00 ffffffff810be6a7 ffff880079695cd0 0000000000000000 > [260431.873910] ffff8801095d3c00 0000000000000000 ffff8801095d3b80 ffffffff81146ef4 > [260431.874909] Call Trace: > [260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74 > [260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120 > [260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320 > [260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320 > [260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320 > [260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f > [260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0 > [260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90 > [260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0 > [260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0 > [260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 > [260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0 > [260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110 > [260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 > [260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110 > [260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f > [260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110 > [260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f > [260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50 > [260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 > [260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 > [260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0 > [260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20 > [260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0 > [260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150 > [260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3d > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 16:29 ` Paul E. McKenney @ 2013-08-26 17:30 ` Steven Rostedt 2013-08-26 17:50 ` Dave Jones 0 siblings, 1 reply; 17+ messages in thread From: Steven Rostedt @ 2013-08-26 17:30 UTC (permalink / raw) To: paulmck; +Cc: Dave Jones, Linux Kernel On Mon, 26 Aug 2013 09:29:28 -0700 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > On Mon, Aug 26, 2013 at 10:58:38AM -0400, Dave Jones wrote: > > Another day, another rcu backtrace.. > > This says rc6, but it's pretty darn close to rc7, I think it was running > > a build from Friday. > > Could you please send your .config? Also, were you running ftrace, > perf, RCU event tracing, or what? > > Looks like you are running ftrace, but I though Steven had set that > up so that it could be called from an extended quiescent state. > I know exactly what the issue is. Yes ftrace is safe to call even from these extended quiescent states, the problem is that ftrace is also the infrastructure of other users, where some of those users are not safe. Namely, perf. Right now perf is not safe to trace all functions, as some of those functions have this issue. I was going to add something like: FTRACE_NON_SAFE(rcu_eqs_enter); where it will record locations that are not safe for all users, such that unless a function registers to ftrace with a flag of "FTRACE_FL_NON_SAFE_OK", anything that is on the non safe list (from the macro) will not be traced. Now, how urgent is this fix? perf can only trace functions as root, and there's no reason for perf to be tracing all functions at the moment. But yes, a root user could run that and get this warning. Because I was going to implement this for 3.12 and not for this release. -- Steve ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 17:30 ` Steven Rostedt @ 2013-08-26 17:50 ` Dave Jones 2013-08-26 18:18 ` Steven Rostedt 0 siblings, 1 reply; 17+ messages in thread From: Dave Jones @ 2013-08-26 17:50 UTC (permalink / raw) To: Steven Rostedt; +Cc: paulmck, Linux Kernel On Mon, Aug 26, 2013 at 01:30:41PM -0400, Steven Rostedt wrote: > > On Mon, Aug 26, 2013 at 10:58:38AM -0400, Dave Jones wrote: > > > Another day, another rcu backtrace.. > > > This says rc6, but it's pretty darn close to rc7, I think it was running > > > a build from Friday. > > > > Could you please send your .config? Also, were you running ftrace, > > perf, RCU event tracing, or what? > > > > Looks like you are running ftrace, but I though Steven had set that > > up so that it could be called from an extended quiescent state. > > > > I know exactly what the issue is. Yes ftrace is safe to call even from > these extended quiescent states, the problem is that ftrace is also the > infrastructure of other users, where some of those users are not safe. > Namely, perf. > > Right now perf is not safe to trace all functions, as some of those > functions have this issue. I was going to add something like: > > FTRACE_NON_SAFE(rcu_eqs_enter); > > where it will record locations that are not safe for all users, such > that unless a function registers to ftrace with a flag of > "FTRACE_FL_NON_SAFE_OK", anything that is on the non safe list (from > the macro) will not be traced. > > Now, how urgent is this fix? perf can only trace functions as root, and > there's no reason for perf to be tracing all functions at the moment. > But yes, a root user could run that and get this warning. Because I was > going to implement this for 3.12 and not for this release. This was triggered as a regular user fwiw. I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all. Dave ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 17:50 ` Dave Jones @ 2013-08-26 18:18 ` Steven Rostedt 2013-08-26 18:29 ` Dave Jones 0 siblings, 1 reply; 17+ messages in thread From: Steven Rostedt @ 2013-08-26 18:18 UTC (permalink / raw) To: Dave Jones; +Cc: paulmck, Linux Kernel On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <davej@redhat.com> wrote: > > This was triggered as a regular user fwiw. > I had not been running perf, or any other tracing. It was just left > fuzzing over the weekend with no interaction at all. So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue. > [260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74 > [260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120 > [260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320 > [260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320 > [260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320 > [260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f > [260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0 > [260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90 > [260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0 > [260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0 > [260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 > [260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0 This is the perf function tracing call. > [260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110 > [260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 > [260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110 > [260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f ftrace_call is the mcount trampoline. The only way to get there is via function tracing, and function tracing should only be enabled by root. > [260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110 > [260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f > [260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50 > [260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 > [260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 > [260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 > [260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0 > [260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20 > [260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0 > [260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150 > [260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3d So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box???? -- Steve ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 18:18 ` Steven Rostedt @ 2013-08-26 18:29 ` Dave Jones 2013-08-26 19:03 ` Steven Rostedt 2013-08-26 19:43 ` David Ahern 0 siblings, 2 replies; 17+ messages in thread From: Dave Jones @ 2013-08-26 18:29 UTC (permalink / raw) To: Steven Rostedt; +Cc: paulmck, Linux Kernel On Mon, Aug 26, 2013 at 02:18:14PM -0400, Steven Rostedt wrote: > On Mon, 26 Aug 2013 13:50:12 -0400 > Dave Jones <davej@redhat.com> wrote: > > > > This was triggered as a regular user fwiw. > > I had not been running perf, or any other tracing. It was just left > > fuzzing over the weekend with no interaction at all. > > So you are telling me that ftrace was enabled by a regular user? If so, > that's a huge issue. quite. > So my question to you. If you were not running perf or any other > tracing, and this is all just non-root user. How the hell did perf > function tracing get started on your box???? What mechanisms are available that would trigger it being enabled ? Is there some path through sys_perf_open_event that might be missing a capability check perhaps ? Dave ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 18:29 ` Dave Jones @ 2013-08-26 19:03 ` Steven Rostedt 2013-08-27 12:16 ` Peter Zijlstra 2013-08-26 19:43 ` David Ahern 1 sibling, 1 reply; 17+ messages in thread From: Steven Rostedt @ 2013-08-26 19:03 UTC (permalink / raw) To: Dave Jones; +Cc: paulmck, Linux Kernel, Ingo Molnar, Peter Zijlstra, Jiri Olsa On Mon, 26 Aug 2013 14:29:07 -0400 Dave Jones <davej@redhat.com> wrote: > On Mon, Aug 26, 2013 at 02:18:14PM -0400, Steven Rostedt wrote: > > On Mon, 26 Aug 2013 13:50:12 -0400 > > Dave Jones <davej@redhat.com> wrote: > > > > > > This was triggered as a regular user fwiw. > > > I had not been running perf, or any other tracing. It was just left > > > fuzzing over the weekend with no interaction at all. > > > > So you are telling me that ftrace was enabled by a regular user? If so, > > that's a huge issue. > > quite. > > > So my question to you. If you were not running perf or any other > > tracing, and this is all just non-root user. How the hell did perf > > function tracing get started on your box???? > > What mechanisms are available that would trigger it being enabled ? > > Is there some path through sys_perf_open_event that might be > missing a capability check perhaps ? > That's a question for Ingo, Peter or Jiri. -- Steve ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 19:03 ` Steven Rostedt @ 2013-08-27 12:16 ` Peter Zijlstra 2013-08-30 15:49 ` Frederic Weisbecker 2013-08-30 15:52 ` Frederic Weisbecker 0 siblings, 2 replies; 17+ messages in thread From: Peter Zijlstra @ 2013-08-27 12:16 UTC (permalink / raw) To: Steven Rostedt Cc: Dave Jones, paulmck, Linux Kernel, Ingo Molnar, Jiri Olsa, Frederic Weisbecker On Mon, Aug 26, 2013 at 03:03:04PM -0400, Steven Rostedt wrote: > > Is there some path through sys_perf_open_event that might be > > missing a capability check perhaps ? > > > > That's a question for Ingo, Peter or Jiri. Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code. Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm(). ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-27 12:16 ` Peter Zijlstra @ 2013-08-30 15:49 ` Frederic Weisbecker 2013-08-30 15:52 ` Frederic Weisbecker 1 sibling, 0 replies; 17+ messages in thread From: Frederic Weisbecker @ 2013-08-30 15:49 UTC (permalink / raw) To: Peter Zijlstra Cc: Steven Rostedt, Dave Jones, paulmck, Linux Kernel, Ingo Molnar, Jiri Olsa On Tue, Aug 27, 2013 at 02:16:29PM +0200, Peter Zijlstra wrote: > On Mon, Aug 26, 2013 at 03:03:04PM -0400, Steven Rostedt wrote: > > > Is there some path through sys_perf_open_event that might be > > > missing a capability check perhaps ? > > > > > > > That's a question for Ingo, Peter or Jiri. > > Its not something I've looked at recently, git blames Jiri and fweisbec > for most of that code. > > Permission checks appear to live in > kernel/trace/trace_event_perf.c:perf_trace_event_perm(). Well, it also depends on sysctl kernel.perf_event_paranoid settings. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-27 12:16 ` Peter Zijlstra 2013-08-30 15:49 ` Frederic Weisbecker @ 2013-08-30 15:52 ` Frederic Weisbecker 2013-08-30 15:59 ` Peter Zijlstra 1 sibling, 1 reply; 17+ messages in thread From: Frederic Weisbecker @ 2013-08-30 15:52 UTC (permalink / raw) To: Peter Zijlstra Cc: Steven Rostedt, Dave Jones, paulmck, Linux Kernel, Ingo Molnar, Jiri Olsa On Tue, Aug 27, 2013 at 02:16:29PM +0200, Peter Zijlstra wrote: > On Mon, Aug 26, 2013 at 03:03:04PM -0400, Steven Rostedt wrote: > > > Is there some path through sys_perf_open_event that might be > > > missing a capability check perhaps ? > > > > > > > That's a question for Ingo, Peter or Jiri. > > Its not something I've looked at recently, git blames Jiri and fweisbec > for most of that code. > > Permission checks appear to live in > kernel/trace/trace_event_perf.c:perf_trace_event_perm(). Actually the following condition is weird: /* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) && perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM; We probably intended to do: /* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) || perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM; Can somebody confirm? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-30 15:52 ` Frederic Weisbecker @ 2013-08-30 15:59 ` Peter Zijlstra 2013-08-30 16:06 ` Frederic Weisbecker 0 siblings, 1 reply; 17+ messages in thread From: Peter Zijlstra @ 2013-08-30 15:59 UTC (permalink / raw) To: Frederic Weisbecker Cc: Steven Rostedt, Dave Jones, paulmck, Linux Kernel, Ingo Molnar, Jiri Olsa On Fri, Aug 30, 2013 at 05:52:43PM +0200, Frederic Weisbecker wrote: > On Tue, Aug 27, 2013 at 02:16:29PM +0200, Peter Zijlstra wrote: > > On Mon, Aug 26, 2013 at 03:03:04PM -0400, Steven Rostedt wrote: > > > > Is there some path through sys_perf_open_event that might be > > > > missing a capability check perhaps ? > > > > > > > > > > That's a question for Ingo, Peter or Jiri. > > > > Its not something I've looked at recently, git blames Jiri and fweisbec > > for most of that code. > > > > Permission checks appear to live in > > kernel/trace/trace_event_perf.c:perf_trace_event_perm(). > > Actually the following condition is weird: > > /* The ftrace function trace is allowed only for root. */ > if (ftrace_event_is_function(tp_event) && > perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) > return -EPERM; > That says: If its its a function-event and we're paranoid but we don't have root, bail. > We probably intended to do: > > /* The ftrace function trace is allowed only for root. */ > if (ftrace_event_is_function(tp_event) || > perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) > return -EPERM; > > Can somebody confirm? That would always disallow function-events, no? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-30 15:59 ` Peter Zijlstra @ 2013-08-30 16:06 ` Frederic Weisbecker 0 siblings, 0 replies; 17+ messages in thread From: Frederic Weisbecker @ 2013-08-30 16:06 UTC (permalink / raw) To: Peter Zijlstra Cc: Steven Rostedt, Dave Jones, paulmck, Linux Kernel, Ingo Molnar, Jiri Olsa On Fri, Aug 30, 2013 at 05:59:36PM +0200, Peter Zijlstra wrote: > On Fri, Aug 30, 2013 at 05:52:43PM +0200, Frederic Weisbecker wrote: > > On Tue, Aug 27, 2013 at 02:16:29PM +0200, Peter Zijlstra wrote: > > > On Mon, Aug 26, 2013 at 03:03:04PM -0400, Steven Rostedt wrote: > > > > > Is there some path through sys_perf_open_event that might be > > > > > missing a capability check perhaps ? > > > > > > > > > > > > > That's a question for Ingo, Peter or Jiri. > > > > > > Its not something I've looked at recently, git blames Jiri and fweisbec > > > for most of that code. > > > > > > Permission checks appear to live in > > > kernel/trace/trace_event_perf.c:perf_trace_event_perm(). > > > > Actually the following condition is weird: > > > > /* The ftrace function trace is allowed only for root. */ > > if (ftrace_event_is_function(tp_event) && > > perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) > > return -EPERM; > > > > That says: If its its a function-event and we're paranoid but we don't > have root, bail. Right, I misunderstood and thought we messed up general tracepoint permissions with function events. > > > We probably intended to do: > > > > /* The ftrace function trace is allowed only for root. */ > > if (ftrace_event_is_function(tp_event) || > > perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) > > return -EPERM; > > > > Can somebody confirm? > > That would always disallow function-events, no? Yeah, sorry, returning from holidays require more dense coffee. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 18:29 ` Dave Jones 2013-08-26 19:03 ` Steven Rostedt @ 2013-08-26 19:43 ` David Ahern 2013-08-26 19:49 ` Dave Jones 1 sibling, 1 reply; 17+ messages in thread From: David Ahern @ 2013-08-26 19:43 UTC (permalink / raw) To: Dave Jones, Steven Rostedt, paulmck, Linux Kernel On 8/26/13 12:29 PM, Dave Jones wrote: > On Mon, Aug 26, 2013 at 02:18:14PM -0400, Steven Rostedt wrote: > > On Mon, 26 Aug 2013 13:50:12 -0400 > > Dave Jones <davej@redhat.com> wrote: > > > > > > This was triggered as a regular user fwiw. > > > I had not been running perf, or any other tracing. It was just left > > > fuzzing over the weekend with no interaction at all. > > > > So you are telling me that ftrace was enabled by a regular user? If so, > > that's a huge issue. > > quite. > > > So my question to you. If you were not running perf or any other > > tracing, and this is all just non-root user. How the hell did perf > > function tracing get started on your box???? > > What mechanisms are available that would trigger it being enabled ? > > Is there some path through sys_perf_open_event that might be > missing a capability check perhaps ? Do you have /sys/kernel/debug with access permissions? David ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 19:43 ` David Ahern @ 2013-08-26 19:49 ` Dave Jones 2013-08-27 13:10 ` Steven Rostedt 0 siblings, 1 reply; 17+ messages in thread From: Dave Jones @ 2013-08-26 19:49 UTC (permalink / raw) To: David Ahern; +Cc: Steven Rostedt, paulmck, Linux Kernel On Mon, Aug 26, 2013 at 01:43:15PM -0600, David Ahern wrote: > On 8/26/13 12:29 PM, Dave Jones wrote: > > On Mon, Aug 26, 2013 at 02:18:14PM -0400, Steven Rostedt wrote: > > > On Mon, 26 Aug 2013 13:50:12 -0400 > > > Dave Jones <davej@redhat.com> wrote: > > > > > > > > This was triggered as a regular user fwiw. > > > > I had not been running perf, or any other tracing. It was just left > > > > fuzzing over the weekend with no interaction at all. > > > > > > So you are telling me that ftrace was enabled by a regular user? If so, > > > that's a huge issue. > > > > quite. > > > > > So my question to you. If you were not running perf or any other > > > tracing, and this is all just non-root user. How the hell did perf > > > function tracing get started on your box???? > > > > What mechanisms are available that would trigger it being enabled ? > > > > Is there some path through sys_perf_open_event that might be > > missing a capability check perhaps ? > > Do you have /sys/kernel/debug with access permissions? Ah, yeah, that'll be it. Good catch. Sorry for the false alarm Steve ;) Dave ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-26 19:49 ` Dave Jones @ 2013-08-27 13:10 ` Steven Rostedt 2013-08-27 13:58 ` David Ahern 0 siblings, 1 reply; 17+ messages in thread From: Steven Rostedt @ 2013-08-27 13:10 UTC (permalink / raw) To: Dave Jones; +Cc: David Ahern, paulmck, Linux Kernel On Mon, 26 Aug 2013 15:49:24 -0400 Dave Jones <davej@redhat.com> wrote: > > Do you have /sys/kernel/debug with access permissions? > > Ah, yeah, that'll be it. Good catch. > > Sorry for the false alarm Steve ;) Yeah, but it still does not explain how perf got started. Perf requires a sys_perf_event_open() call to run. -- Steve ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-27 13:10 ` Steven Rostedt @ 2013-08-27 13:58 ` David Ahern 2013-08-27 14:13 ` Dave Jones 0 siblings, 1 reply; 17+ messages in thread From: David Ahern @ 2013-08-27 13:58 UTC (permalink / raw) To: Steven Rostedt; +Cc: Dave Jones, paulmck, Linux Kernel On 8/27/13 7:10 AM, Steven Rostedt wrote: > On Mon, 26 Aug 2013 15:49:24 -0400 > Dave Jones <davej@redhat.com> wrote: > > >> > Do you have /sys/kernel/debug with access permissions? >> >> Ah, yeah, that'll be it. Good catch. >> >> Sorry for the false alarm Steve ;) > > Yeah, but it still does not explain how perf got started. Perf requires > a sys_perf_event_open() call to run. Per Peter's response another option is the paranoia level: # perf event paranoia level: # -1 - not paranoid at all # 0 - disallow raw tracepoint access for unpriv # 1 - disallow cpu events for unpriv # 2 - disallow kernel profiling for unpriv kernel.perf_event_paranoid = -1 I keep my dev box at -1 and set /sys/kernel/debug to 755 to run perf as non-root. David ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: suspicious RCU usage (perf) 2013-08-27 13:58 ` David Ahern @ 2013-08-27 14:13 ` Dave Jones 0 siblings, 0 replies; 17+ messages in thread From: Dave Jones @ 2013-08-27 14:13 UTC (permalink / raw) To: David Ahern; +Cc: Steven Rostedt, paulmck, Linux Kernel On Tue, Aug 27, 2013 at 07:58:12AM -0600, David Ahern wrote: > On 8/27/13 7:10 AM, Steven Rostedt wrote: > > On Mon, 26 Aug 2013 15:49:24 -0400 > > Dave Jones <davej@redhat.com> wrote: > > > > > >> > Do you have /sys/kernel/debug with access permissions? > >> > >> Ah, yeah, that'll be it. Good catch. > >> > >> Sorry for the false alarm Steve ;) > > > > Yeah, but it still does not explain how perf got started. Perf requires > > a sys_perf_event_open() call to run. > > Per Peter's response another option is the paranoia level: > > # perf event paranoia level: > # -1 - not paranoid at all > # 0 - disallow raw tracepoint access for unpriv > # 1 - disallow cpu events for unpriv > # 2 - disallow kernel profiling for unpriv > kernel.perf_event_paranoid = -1 This I have set to 1 on that box. (The default) Dave ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-08-30 16:06 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-08-26 14:58 suspicious RCU usage (perf) Dave Jones 2013-08-26 16:29 ` Paul E. McKenney 2013-08-26 17:30 ` Steven Rostedt 2013-08-26 17:50 ` Dave Jones 2013-08-26 18:18 ` Steven Rostedt 2013-08-26 18:29 ` Dave Jones 2013-08-26 19:03 ` Steven Rostedt 2013-08-27 12:16 ` Peter Zijlstra 2013-08-30 15:49 ` Frederic Weisbecker 2013-08-30 15:52 ` Frederic Weisbecker 2013-08-30 15:59 ` Peter Zijlstra 2013-08-30 16:06 ` Frederic Weisbecker 2013-08-26 19:43 ` David Ahern 2013-08-26 19:49 ` Dave Jones 2013-08-27 13:10 ` Steven Rostedt 2013-08-27 13:58 ` David Ahern 2013-08-27 14:13 ` Dave Jones
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.