* [PATCH] perf probe: Provide more detail with relocation warning @ 2021-05-25 4:37 Ravi Bangoria 2021-05-25 12:48 ` Masami Hiramatsu 2021-05-26 9:01 ` [PATCH] tools/perf: doc: Add permission and sysctl notice Masami Hiramatsu 0 siblings, 2 replies; 16+ messages in thread From: Ravi Bangoria @ 2021-05-25 4:37 UTC (permalink / raw) To: acme; +Cc: ravi.bangoria, jolsa, mhiramat, linux-kernel, aneesh.kumar When run as normal user with default sysctl kernel.kptr_restrict=0 and kernel.perf_event_paranoid=2, perf probe fails with: $ ./perf probe move_page_tables Relocated base symbol is not found! The warning message is not much informative. The reason perf fails is because /proc/kallsyms is restricted by perf_event_paranoid=2 for normal user and thus perf fails to read relocated address of the base symbol. Tweaking kptr_restrict and perf_event_paranoid can change the behavior of perf probe. Also, running as root or privileged user works too. Add these details in the warning message. Plus, kmap->ref_reloc_sym might not be always set even if host_machine is initialized. Above is the example of the same. Remove that comment. Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> --- tools/perf/util/probe-event.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index a78c8d59a555..3a7649835ec9 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -108,7 +108,6 @@ void exit_probe_symbol_maps(void) static struct ref_reloc_sym *kernel_get_ref_reloc_sym(struct map **pmap) { - /* kmap->ref_reloc_sym should be set if host_machine is initialized */ struct kmap *kmap; struct map *map = machine__kernel_map(host_machine); @@ -819,7 +818,10 @@ post_process_kernel_probe_trace_events(struct probe_trace_event *tevs, reloc_sym = kernel_get_ref_reloc_sym(&map); if (!reloc_sym) { - pr_warning("Relocated base symbol is not found!\n"); + pr_warning("Relocated base symbol is not found! " + "Check /proc/sys/kernel/kptr_restrict\n" + "and /proc/sys/kernel/perf_event_paranoid. " + "Or run as privileged perf user.\n\n"); return -EINVAL; } @@ -3025,7 +3027,10 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev, (!pp->retprobe || kretprobe_offset_is_supported())) { reloc_sym = kernel_get_ref_reloc_sym(NULL); if (!reloc_sym) { - pr_warning("Relocated base symbol is not found!\n"); + pr_warning("Relocated base symbol is not found! " + "Check /proc/sys/kernel/kptr_restrict\n" + "and /proc/sys/kernel/perf_event_paranoid. " + "Or run as privileged perf user.\n\n"); ret = -EINVAL; goto out; } -- 2.31.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-25 4:37 [PATCH] perf probe: Provide more detail with relocation warning Ravi Bangoria @ 2021-05-25 12:48 ` Masami Hiramatsu 2021-05-26 4:53 ` Ravi Bangoria 2021-05-26 9:01 ` [PATCH] tools/perf: doc: Add permission and sysctl notice Masami Hiramatsu 1 sibling, 1 reply; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-25 12:48 UTC (permalink / raw) To: Ravi Bangoria; +Cc: acme, jolsa, mhiramat, linux-kernel, aneesh.kumar On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > When run as normal user with default sysctl kernel.kptr_restrict=0 > and kernel.perf_event_paranoid=2, perf probe fails with: > > $ ./perf probe move_page_tables > Relocated base symbol is not found! > > The warning message is not much informative. The reason perf fails > is because /proc/kallsyms is restricted by perf_event_paranoid=2 > for normal user and thus perf fails to read relocated address of > the base symbol. > > Tweaking kptr_restrict and perf_event_paranoid can change the > behavior of perf probe. Also, running as root or privileged user > works too. Add these details in the warning message. > > Plus, kmap->ref_reloc_sym might not be always set even if > host_machine is initialized. Above is the example of the same. > Remove that comment. Yes, those are restricted in some cases. Anyway without priviledged (super) user, perf probe can not set the probe in ftrace. Hmm, I think it should check the effective user-id at first. If it is not super user and the action will access tracefs and kallsyms, it should warn at that point. Thank you, > > Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> > Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> > --- > tools/perf/util/probe-event.c | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) > > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c > index a78c8d59a555..3a7649835ec9 100644 > --- a/tools/perf/util/probe-event.c > +++ b/tools/perf/util/probe-event.c > @@ -108,7 +108,6 @@ void exit_probe_symbol_maps(void) > > static struct ref_reloc_sym *kernel_get_ref_reloc_sym(struct map **pmap) > { > - /* kmap->ref_reloc_sym should be set if host_machine is initialized */ > struct kmap *kmap; > struct map *map = machine__kernel_map(host_machine); > > @@ -819,7 +818,10 @@ post_process_kernel_probe_trace_events(struct probe_trace_event *tevs, > > reloc_sym = kernel_get_ref_reloc_sym(&map); > if (!reloc_sym) { > - pr_warning("Relocated base symbol is not found!\n"); > + pr_warning("Relocated base symbol is not found! " > + "Check /proc/sys/kernel/kptr_restrict\n" > + "and /proc/sys/kernel/perf_event_paranoid. " > + "Or run as privileged perf user.\n\n"); > return -EINVAL; > } > > @@ -3025,7 +3027,10 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev, > (!pp->retprobe || kretprobe_offset_is_supported())) { > reloc_sym = kernel_get_ref_reloc_sym(NULL); > if (!reloc_sym) { > - pr_warning("Relocated base symbol is not found!\n"); > + pr_warning("Relocated base symbol is not found! " > + "Check /proc/sys/kernel/kptr_restrict\n" > + "and /proc/sys/kernel/perf_event_paranoid. " > + "Or run as privileged perf user.\n\n"); > ret = -EINVAL; > goto out; > } > -- > 2.31.1 > -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-25 12:48 ` Masami Hiramatsu @ 2021-05-26 4:53 ` Ravi Bangoria 2021-05-26 6:33 ` Masami Hiramatsu 0 siblings, 1 reply; 16+ messages in thread From: Ravi Bangoria @ 2021-05-26 4:53 UTC (permalink / raw) To: Masami Hiramatsu; +Cc: acme, jolsa, linux-kernel, aneesh.kumar, Ravi Bangoria On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > On Tue, 25 May 2021 10:07:44 +0530 > Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > >> When run as normal user with default sysctl kernel.kptr_restrict=0 >> and kernel.perf_event_paranoid=2, perf probe fails with: >> >> $ ./perf probe move_page_tables >> Relocated base symbol is not found! >> >> The warning message is not much informative. The reason perf fails >> is because /proc/kallsyms is restricted by perf_event_paranoid=2 >> for normal user and thus perf fails to read relocated address of >> the base symbol. >> >> Tweaking kptr_restrict and perf_event_paranoid can change the >> behavior of perf probe. Also, running as root or privileged user >> works too. Add these details in the warning message. >> >> Plus, kmap->ref_reloc_sym might not be always set even if >> host_machine is initialized. Above is the example of the same. >> Remove that comment. > > Yes, those are restricted in some cases. Anyway without priviledged > (super) user, perf probe can not set the probe in ftrace. > > Hmm, I think it should check the effective user-id at first. If it > is not super user and the action will access tracefs and kallsyms, > it should warn at that point. If kptr_restrict=2, perf probe fails with same error even for root user. That's why I thought to just change this warning message. Different combinations of privilege, perf_event_paranoid, kptr_restrict: Normal/Root user | perf_event_paranoid V V kptr_restrict perf probe error ---------------------------------------------------------------- N -1 0 Failed to open kprobe_events: Permission denied N 0 0 Failed to open kprobe_events: Permission denied N 1 0 Failed to open kprobe_events: Permission denied N 2 0 Relocated base symbol is not found! N -1 1 Relocated base symbol is not found! N 0 1 Relocated base symbol is not found! N 1 1 Relocated base symbol is not found! N 2 1 Relocated base symbol is not found! N -1 2 Relocated base symbol is not found! N 0 2 Relocated base symbol is not found! N 1 2 Relocated base symbol is not found! N 2 2 Relocated base symbol is not found! R -1 0 No error. R 0 0 No error. R 1 0 No error. R 2 0 No error. R -1 1 No error. R 0 1 No error. R 1 1 No error. R 2 1 No error. R -1 2 Relocated base symbol is not found! R 0 2 Relocated base symbol is not found! R 1 2 Relocated base symbol is not found! R 2 2 Relocated base symbol is not found! Ravi ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-26 4:53 ` Ravi Bangoria @ 2021-05-26 6:33 ` Masami Hiramatsu 2021-05-26 12:56 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-26 6:33 UTC (permalink / raw) To: Ravi Bangoria; +Cc: acme, jolsa, linux-kernel, aneesh.kumar On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > On Tue, 25 May 2021 10:07:44 +0530 > > Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > >> and kernel.perf_event_paranoid=2, perf probe fails with: > >> > >> $ ./perf probe move_page_tables > >> Relocated base symbol is not found! > >> > >> The warning message is not much informative. The reason perf fails > >> is because /proc/kallsyms is restricted by perf_event_paranoid=2 > >> for normal user and thus perf fails to read relocated address of > >> the base symbol. > >> > >> Tweaking kptr_restrict and perf_event_paranoid can change the > >> behavior of perf probe. Also, running as root or privileged user > >> works too. Add these details in the warning message. > >> > >> Plus, kmap->ref_reloc_sym might not be always set even if > >> host_machine is initialized. Above is the example of the same. > >> Remove that comment. > > > > Yes, those are restricted in some cases. Anyway without priviledged > > (super) user, perf probe can not set the probe in ftrace. > > > > Hmm, I think it should check the effective user-id at first. If it > > is not super user and the action will access tracefs and kallsyms, > > it should warn at that point. > > If kptr_restrict=2, perf probe fails with same error even for root user. > That's why I thought to just change this warning message. Ah, yes. In that case, perf probe must not use the base symbol. (like -D option) OK, then, let's merge this fix. Acked-by: Masami Hiramatsu <mhiramat@kernel.org> Thank you, > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > Normal/Root user > | perf_event_paranoid > V V kptr_restrict perf probe error > ---------------------------------------------------------------- > N -1 0 Failed to open kprobe_events: Permission denied > N 0 0 Failed to open kprobe_events: Permission denied > N 1 0 Failed to open kprobe_events: Permission denied > N 2 0 Relocated base symbol is not found! > > N -1 1 Relocated base symbol is not found! > N 0 1 Relocated base symbol is not found! > N 1 1 Relocated base symbol is not found! > N 2 1 Relocated base symbol is not found! > > N -1 2 Relocated base symbol is not found! > N 0 2 Relocated base symbol is not found! > N 1 2 Relocated base symbol is not found! > N 2 2 Relocated base symbol is not found! > > R -1 0 No error. > R 0 0 No error. > R 1 0 No error. > R 2 0 No error. > > R -1 1 No error. > R 0 1 No error. > R 1 1 No error. > R 2 1 No error. > > R -1 2 Relocated base symbol is not found! > R 0 2 Relocated base symbol is not found! > R 1 2 Relocated base symbol is not found! > R 2 2 Relocated base symbol is not found! > > Ravi -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-26 6:33 ` Masami Hiramatsu @ 2021-05-26 12:56 ` Arnaldo Carvalho de Melo 2021-05-26 14:20 ` Masami Hiramatsu 0 siblings, 1 reply; 16+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-05-26 12:56 UTC (permalink / raw) To: Masami Hiramatsu Cc: Ravi Bangoria, Jiri Olsa, linux-kernel, aneesh.kumar, Peter Zijlstra, Ingo Molnar, Namhyung Kim, Ian Rogers Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu: > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > > >> and kernel.perf_event_paranoid=2, perf probe fails with: > > >> $ ./perf probe move_page_tables > > >> Relocated base symbol is not found! > > >> The warning message is not much informative. The reason perf > > >> fails is because /proc/kallsyms is restricted by > > >> perf_event_paranoid=2 for normal user and thus perf fails to read > > >> relocated address of the base symbol. > > >> Tweaking kptr_restrict and perf_event_paranoid can change the > > >> behavior of perf probe. Also, running as root or privileged user > > >> works too. Add these details in the warning message. > > >> Plus, kmap->ref_reloc_sym might not be always set even if > > >> host_machine is initialized. Above is the example of the same. > > >> Remove that comment. > > > Yes, those are restricted in some cases. Anyway without priviledged > > > (super) user, perf probe can not set the probe in ftrace. > > > Hmm, I think it should check the effective user-id at first. If it > > > is not super user and the action will access tracefs and kallsyms, > > > it should warn at that point. > > If kptr_restrict=2, perf probe fails with same error even for root user. > > That's why I thought to just change this warning message. > Ah, yes. In that case, perf probe must not use the base symbol. > (like -D option) > OK, then, let's merge this fix. > Acked-by: Masami Hiramatsu <mhiramat@kernel.org> Thanks, applied as it improves the current situation. But as a follow up, to further improve this, we can reuse what 'perf trace' has: $ perf trace sleep 1 Error: No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit) Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/' $ sudo mount -o remount,mode=755 /sys/kernel/tracing/ $ perf trace sleep 1 Error: Permission denied. Hint: Check /proc/sys/kernel/perf_event_paranoid setting. Hint: For your workloads it needs to be <= 1 Hint: For system wide tracing it needs to be set to -1. Hint: Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"' Hint: The current value is 2. $ I.e. go the extra step and show what the current value is and what it needs to be to achieve what is being attempted. IOW combine error message with relevant documentation, to save steps. See what 'perf top' does for an unpriv user: $ perf top --stdio Error: Access to performance monitoring and observability operations is limited. Enforced MAC policy settings (SELinux) can limit access to performance monitoring and observability operations. Inspect system audit records for more perf_event access control information and adjusting the policy. Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open access to performance monitoring and observability operations for processes without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. More information can be found at 'Perf events and tool security' document: https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html perf_event_paranoid setting is 2: -1: Allow use of (almost) all events by all users Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK >= 0: Disallow raw and ftrace function tracepoint access >= 1: Disallow CPU event access >= 2: Disallow kernel profiling To make the adjusted perf_event_paranoid setting permanent preserve it in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>) $ - Arnaldo > > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > > > Normal/Root user > > | perf_event_paranoid > > V V kptr_restrict perf probe error > > ---------------------------------------------------------------- > > N -1 0 Failed to open kprobe_events: Permission denied > > N 0 0 Failed to open kprobe_events: Permission denied > > N 1 0 Failed to open kprobe_events: Permission denied > > N 2 0 Relocated base symbol is not found! > > > > N -1 1 Relocated base symbol is not found! > > N 0 1 Relocated base symbol is not found! > > N 1 1 Relocated base symbol is not found! > > N 2 1 Relocated base symbol is not found! > > > > N -1 2 Relocated base symbol is not found! > > N 0 2 Relocated base symbol is not found! > > N 1 2 Relocated base symbol is not found! > > N 2 2 Relocated base symbol is not found! > > > > R -1 0 No error. > > R 0 0 No error. > > R 1 0 No error. > > R 2 0 No error. > > > > R -1 1 No error. > > R 0 1 No error. > > R 1 1 No error. > > R 2 1 No error. > > > > R -1 2 Relocated base symbol is not found! > > R 0 2 Relocated base symbol is not found! > > R 1 2 Relocated base symbol is not found! > > R 2 2 Relocated base symbol is not found! > > > > Ravi > > > -- > Masami Hiramatsu <mhiramat@kernel.org> -- - Arnaldo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-26 12:56 ` Arnaldo Carvalho de Melo @ 2021-05-26 14:20 ` Masami Hiramatsu 2021-06-02 11:52 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-26 14:20 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Ravi Bangoria, Jiri Olsa, linux-kernel, aneesh.kumar, Peter Zijlstra, Ingo Molnar, Namhyung Kim, Ian Rogers On Wed, 26 May 2021 09:56:29 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu: > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > > > >> and kernel.perf_event_paranoid=2, perf probe fails with: > > > > >> $ ./perf probe move_page_tables > > > >> Relocated base symbol is not found! > > > > >> The warning message is not much informative. The reason perf > > > >> fails is because /proc/kallsyms is restricted by > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read > > > >> relocated address of the base symbol. > > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the > > > >> behavior of perf probe. Also, running as root or privileged user > > > >> works too. Add these details in the warning message. > > > > >> Plus, kmap->ref_reloc_sym might not be always set even if > > > >> host_machine is initialized. Above is the example of the same. > > > >> Remove that comment. > > > > > Yes, those are restricted in some cases. Anyway without priviledged > > > > (super) user, perf probe can not set the probe in ftrace. > > > > > Hmm, I think it should check the effective user-id at first. If it > > > > is not super user and the action will access tracefs and kallsyms, > > > > it should warn at that point. > > > > If kptr_restrict=2, perf probe fails with same error even for root user. > > > That's why I thought to just change this warning message. > > > Ah, yes. In that case, perf probe must not use the base symbol. > > (like -D option) > > OK, then, let's merge this fix. > > > Acked-by: Masami Hiramatsu <mhiramat@kernel.org> > > Thanks, applied as it improves the current situation. > > But as a follow up, to further improve this, we can reuse what 'perf trace' has: > > $ perf trace sleep 1 > Error: No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit) > Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/' > $ sudo mount -o remount,mode=755 /sys/kernel/tracing/ > $ perf trace sleep 1 > Error: Permission denied. > Hint: Check /proc/sys/kernel/perf_event_paranoid setting. > Hint: For your workloads it needs to be <= 1 > Hint: For system wide tracing it needs to be set to -1. > Hint: Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"' > Hint: The current value is 2. > $ OK, let me check this. BTW, does perf_event_paranoid affect only perf syscall (and kallsyms), not the tracefs correct? > I.e. go the extra step and show what the current value is and what it > needs to be to achieve what is being attempted. > > IOW combine error message with relevant documentation, to save steps. > > See what 'perf top' does for an unpriv user: > > $ perf top --stdio > Error: > Access to performance monitoring and observability operations is limited. > Enforced MAC policy settings (SELinux) can limit access to performance > monitoring and observability operations. Inspect system audit records for > more perf_event access control information and adjusting the policy. > Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open > access to performance monitoring and observability operations for processes > without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. > More information can be found at 'Perf events and tool security' document: > https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html > perf_event_paranoid setting is 2: > -1: Allow use of (almost) all events by all users > Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK > >= 0: Disallow raw and ftrace function tracepoint access > >= 1: Disallow CPU event access > >= 2: Disallow kernel profiling > To make the adjusted perf_event_paranoid setting permanent preserve it > in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>) Hmm, I would rather like pointing manpages... Would we better to have perf-security.7 manpage? Thank you, > $ > > - Arnaldo > > > > > > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > > > > > Normal/Root user > > > | perf_event_paranoid > > > V V kptr_restrict perf probe error > > > ---------------------------------------------------------------- > > > N -1 0 Failed to open kprobe_events: Permission denied > > > N 0 0 Failed to open kprobe_events: Permission denied > > > N 1 0 Failed to open kprobe_events: Permission denied > > > N 2 0 Relocated base symbol is not found! > > > > > > N -1 1 Relocated base symbol is not found! > > > N 0 1 Relocated base symbol is not found! > > > N 1 1 Relocated base symbol is not found! > > > N 2 1 Relocated base symbol is not found! > > > > > > N -1 2 Relocated base symbol is not found! > > > N 0 2 Relocated base symbol is not found! > > > N 1 2 Relocated base symbol is not found! > > > N 2 2 Relocated base symbol is not found! > > > > > > R -1 0 No error. > > > R 0 0 No error. > > > R 1 0 No error. > > > R 2 0 No error. > > > > > > R -1 1 No error. > > > R 0 1 No error. > > > R 1 1 No error. > > > R 2 1 No error. > > > > > > R -1 2 Relocated base symbol is not found! > > > R 0 2 Relocated base symbol is not found! > > > R 1 2 Relocated base symbol is not found! > > > R 2 2 Relocated base symbol is not found! > > > > > > Ravi > > > > > > -- > > Masami Hiramatsu <mhiramat@kernel.org> > > -- > > - Arnaldo -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-05-26 14:20 ` Masami Hiramatsu @ 2021-06-02 11:52 ` Arnaldo Carvalho de Melo 2021-06-02 12:12 ` Masami Hiramatsu 0 siblings, 1 reply; 16+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-06-02 11:52 UTC (permalink / raw) To: Masami Hiramatsu Cc: Ravi Bangoria, Jiri Olsa, linux-kernel, aneesh.kumar, Peter Zijlstra, Ingo Molnar, Namhyung Kim, Ian Rogers Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu: > On Wed, 26 May 2021 09:56:29 -0300 > Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu: > > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > > > > >> and kernel.perf_event_paranoid=2, perf probe fails with: > > > > > > >> $ ./perf probe move_page_tables > > > > >> Relocated base symbol is not found! > > > > > > >> The warning message is not much informative. The reason perf > > > > >> fails is because /proc/kallsyms is restricted by > > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read > > > > >> relocated address of the base symbol. > > > > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the > > > > >> behavior of perf probe. Also, running as root or privileged user > > > > >> works too. Add these details in the warning message. > > > > > > >> Plus, kmap->ref_reloc_sym might not be always set even if > > > > >> host_machine is initialized. Above is the example of the same. > > > > >> Remove that comment. > > > > > > > Yes, those are restricted in some cases. Anyway without priviledged > > > > > (super) user, perf probe can not set the probe in ftrace. > > > > > > > Hmm, I think it should check the effective user-id at first. If it > > > > > is not super user and the action will access tracefs and kallsyms, > > > > > it should warn at that point. > > > > > > If kptr_restrict=2, perf probe fails with same error even for root user. > > > > That's why I thought to just change this warning message. > > > > > Ah, yes. In that case, perf probe must not use the base symbol. > > > (like -D option) > > > OK, then, let's merge this fix. > > > > > Acked-by: Masami Hiramatsu <mhiramat@kernel.org> > > > > Thanks, applied as it improves the current situation. > > > > But as a follow up, to further improve this, we can reuse what 'perf trace' has: > > > > $ perf trace sleep 1 > > Error: No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit) > > Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/' > > $ sudo mount -o remount,mode=755 /sys/kernel/tracing/ > > $ perf trace sleep 1 > > Error: Permission denied. > > Hint: Check /proc/sys/kernel/perf_event_paranoid setting. > > Hint: For your workloads it needs to be <= 1 > > Hint: For system wide tracing it needs to be set to -1. > > Hint: Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"' > > Hint: The current value is 2. > > $ > > OK, let me check this. > BTW, does perf_event_paranoid affect only perf syscall (and kallsyms), > not the tracefs correct? > > > I.e. go the extra step and show what the current value is and what it > > needs to be to achieve what is being attempted. > > > > IOW combine error message with relevant documentation, to save steps. > > > > See what 'perf top' does for an unpriv user: > > > > $ perf top --stdio > > Error: > > Access to performance monitoring and observability operations is limited. > > Enforced MAC policy settings (SELinux) can limit access to performance > > monitoring and observability operations. Inspect system audit records for > > more perf_event access control information and adjusting the policy. > > Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open > > access to performance monitoring and observability operations for processes > > without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. > > More information can be found at 'Perf events and tool security' document: > > https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html > > perf_event_paranoid setting is 2: > > -1: Allow use of (almost) all events by all users > > Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK > > >= 0: Disallow raw and ftrace function tracepoint access > > >= 1: Disallow CPU event access > > >= 2: Disallow kernel profiling > > To make the adjusted perf_event_paranoid setting permanent preserve it > > in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>) > > Hmm, I would rather like pointing manpages... Man pages are long, if you quote the relevant part of it when the problem takes place, IMHO it helps the user. - Arnaldo > Would we better to have perf-security.7 manpage? > > Thank you, > > > $ > > > > - Arnaldo > > > > > > > > > > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > > > > > > > Normal/Root user > > > > | perf_event_paranoid > > > > V V kptr_restrict perf probe error > > > > ---------------------------------------------------------------- > > > > N -1 0 Failed to open kprobe_events: Permission denied > > > > N 0 0 Failed to open kprobe_events: Permission denied > > > > N 1 0 Failed to open kprobe_events: Permission denied > > > > N 2 0 Relocated base symbol is not found! > > > > > > > > N -1 1 Relocated base symbol is not found! > > > > N 0 1 Relocated base symbol is not found! > > > > N 1 1 Relocated base symbol is not found! > > > > N 2 1 Relocated base symbol is not found! > > > > > > > > N -1 2 Relocated base symbol is not found! > > > > N 0 2 Relocated base symbol is not found! > > > > N 1 2 Relocated base symbol is not found! > > > > N 2 2 Relocated base symbol is not found! > > > > > > > > R -1 0 No error. > > > > R 0 0 No error. > > > > R 1 0 No error. > > > > R 2 0 No error. > > > > > > > > R -1 1 No error. > > > > R 0 1 No error. > > > > R 1 1 No error. > > > > R 2 1 No error. > > > > > > > > R -1 2 Relocated base symbol is not found! > > > > R 0 2 Relocated base symbol is not found! > > > > R 1 2 Relocated base symbol is not found! > > > > R 2 2 Relocated base symbol is not found! > > > > > > > > Ravi > > > > > > > > > -- > > > Masami Hiramatsu <mhiramat@kernel.org> > > > > -- > > > > - Arnaldo > > > -- > Masami Hiramatsu <mhiramat@kernel.org> -- - Arnaldo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-06-02 11:52 ` Arnaldo Carvalho de Melo @ 2021-06-02 12:12 ` Masami Hiramatsu 2021-06-02 13:15 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 16+ messages in thread From: Masami Hiramatsu @ 2021-06-02 12:12 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Ravi Bangoria, Jiri Olsa, linux-kernel, aneesh.kumar, Peter Zijlstra, Ingo Molnar, Namhyung Kim, Ian Rogers On Wed, 2 Jun 2021 08:52:41 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu: > > On Wed, 26 May 2021 09:56:29 -0300 > > Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > > > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu: > > > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > > > > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > > > > > >> and kernel.perf_event_paranoid=2, perf probe fails with: > > > > > > > > >> $ ./perf probe move_page_tables > > > > > >> Relocated base symbol is not found! > > > > > > > > >> The warning message is not much informative. The reason perf > > > > > >> fails is because /proc/kallsyms is restricted by > > > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read > > > > > >> relocated address of the base symbol. > > > > > > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the > > > > > >> behavior of perf probe. Also, running as root or privileged user > > > > > >> works too. Add these details in the warning message. > > > > > > > > >> Plus, kmap->ref_reloc_sym might not be always set even if > > > > > >> host_machine is initialized. Above is the example of the same. > > > > > >> Remove that comment. > > > > > > > > > Yes, those are restricted in some cases. Anyway without priviledged > > > > > > (super) user, perf probe can not set the probe in ftrace. > > > > > > > > > Hmm, I think it should check the effective user-id at first. If it > > > > > > is not super user and the action will access tracefs and kallsyms, > > > > > > it should warn at that point. > > > > > > > > If kptr_restrict=2, perf probe fails with same error even for root user. > > > > > That's why I thought to just change this warning message. > > > > > > > Ah, yes. In that case, perf probe must not use the base symbol. > > > > (like -D option) > > > > OK, then, let's merge this fix. > > > > > > > Acked-by: Masami Hiramatsu <mhiramat@kernel.org> > > > > > > Thanks, applied as it improves the current situation. > > > > > > But as a follow up, to further improve this, we can reuse what 'perf trace' has: > > > > > > $ perf trace sleep 1 > > > Error: No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit) > > > Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/' > > > $ sudo mount -o remount,mode=755 /sys/kernel/tracing/ > > > $ perf trace sleep 1 > > > Error: Permission denied. > > > Hint: Check /proc/sys/kernel/perf_event_paranoid setting. > > > Hint: For your workloads it needs to be <= 1 > > > Hint: For system wide tracing it needs to be set to -1. > > > Hint: Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"' > > > Hint: The current value is 2. > > > $ > > > > OK, let me check this. > > BTW, does perf_event_paranoid affect only perf syscall (and kallsyms), > > not the tracefs correct? > > > > > I.e. go the extra step and show what the current value is and what it > > > needs to be to achieve what is being attempted. > > > > > > IOW combine error message with relevant documentation, to save steps. > > > > > > See what 'perf top' does for an unpriv user: > > > > > > $ perf top --stdio > > > Error: > > > Access to performance monitoring and observability operations is limited. > > > Enforced MAC policy settings (SELinux) can limit access to performance > > > monitoring and observability operations. Inspect system audit records for > > > more perf_event access control information and adjusting the policy. > > > Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open > > > access to performance monitoring and observability operations for processes > > > without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. > > > More information can be found at 'Perf events and tool security' document: > > > https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html > > > perf_event_paranoid setting is 2: > > > -1: Allow use of (almost) all events by all users > > > Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK > > > >= 0: Disallow raw and ftrace function tracepoint access > > > >= 1: Disallow CPU event access > > > >= 2: Disallow kernel profiling > > > To make the adjusted perf_event_paranoid setting permanent preserve it > > > in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>) > > > > Hmm, I would rather like pointing manpages... > > Man pages are long, if you quote the relevant part of it when the > problem takes place, IMHO it helps the user. OK, but please also update man pages (*), which can provide a summarized information for users if they noticed that. And for each place, I will add some messages for letting them know. (*) https://lore.kernel.org/lkml/162204068898.388434.16842705842611255787.stgit@devnote2/ Thank you, > > - Arnaldo > > > Would we better to have perf-security.7 manpage? > > > > Thank you, > > > > > $ > > > > > > - Arnaldo > > > > > > > > > > > > > > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > > > > > > > > > Normal/Root user > > > > > | perf_event_paranoid > > > > > V V kptr_restrict perf probe error > > > > > ---------------------------------------------------------------- > > > > > N -1 0 Failed to open kprobe_events: Permission denied > > > > > N 0 0 Failed to open kprobe_events: Permission denied > > > > > N 1 0 Failed to open kprobe_events: Permission denied > > > > > N 2 0 Relocated base symbol is not found! > > > > > > > > > > N -1 1 Relocated base symbol is not found! > > > > > N 0 1 Relocated base symbol is not found! > > > > > N 1 1 Relocated base symbol is not found! > > > > > N 2 1 Relocated base symbol is not found! > > > > > > > > > > N -1 2 Relocated base symbol is not found! > > > > > N 0 2 Relocated base symbol is not found! > > > > > N 1 2 Relocated base symbol is not found! > > > > > N 2 2 Relocated base symbol is not found! > > > > > > > > > > R -1 0 No error. > > > > > R 0 0 No error. > > > > > R 1 0 No error. > > > > > R 2 0 No error. > > > > > > > > > > R -1 1 No error. > > > > > R 0 1 No error. > > > > > R 1 1 No error. > > > > > R 2 1 No error. > > > > > > > > > > R -1 2 Relocated base symbol is not found! > > > > > R 0 2 Relocated base symbol is not found! > > > > > R 1 2 Relocated base symbol is not found! > > > > > R 2 2 Relocated base symbol is not found! > > > > > > > > > > Ravi > > > > > > > > > > > > -- > > > > Masami Hiramatsu <mhiramat@kernel.org> > > > > > > -- > > > > > > - Arnaldo > > > > > > -- > > Masami Hiramatsu <mhiramat@kernel.org> > > -- > > - Arnaldo -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] perf probe: Provide more detail with relocation warning 2021-06-02 12:12 ` Masami Hiramatsu @ 2021-06-02 13:15 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 16+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-06-02 13:15 UTC (permalink / raw) To: Masami Hiramatsu Cc: Ravi Bangoria, Jiri Olsa, linux-kernel, aneesh.kumar, Peter Zijlstra, Ingo Molnar, Namhyung Kim, Ian Rogers Em Wed, Jun 02, 2021 at 09:12:48PM +0900, Masami Hiramatsu escreveu: > On Wed, 2 Jun 2021 08:52:41 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu: > > > On Wed, 26 May 2021 09:56:29 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > > > I.e. go the extra step and show what the current value is and what it > > > > needs to be to achieve what is being attempted. > > > > IOW combine error message with relevant documentation, to save steps. > > > > See what 'perf top' does for an unpriv user: > > > > $ perf top --stdio > > > > Error: > > > > Access to performance monitoring and observability operations is limited. > > > > Enforced MAC policy settings (SELinux) can limit access to performance > > > > monitoring and observability operations. Inspect system audit records for > > > > more perf_event access control information and adjusting the policy. > > > > Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open > > > > access to performance monitoring and observability operations for processes > > > > without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. > > > > More information can be found at 'Perf events and tool security' document: > > > > https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html > > > > perf_event_paranoid setting is 2: > > > > -1: Allow use of (almost) all events by all users > > > > Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK > > > > >= 0: Disallow raw and ftrace function tracepoint access > > > > >= 1: Disallow CPU event access > > > > >= 2: Disallow kernel profiling > > > > To make the adjusted perf_event_paranoid setting permanent preserve it > > > > in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>) > > > Hmm, I would rather like pointing manpages... > > Man pages are long, if you quote the relevant part of it when the > > problem takes place, IMHO it helps the user. > OK, but please also update man pages (*), which can provide a summarized information > for users if they noticed that. And for each place, I will add some messages > for letting them know. Sure, its not one or the other, its both, i.e. man pages are interesting as the standard reference for a command, while making tools provide information about a specific problem with actionable advice on how to fix the problem at hand is important as well. > (*) https://lore.kernel.org/lkml/162204068898.388434.16842705842611255787.stgit@devnote2/ I saw your patch to the man page and asked Ravi for an Ack/Reviewed-by, Thanks for all your continued work on this! :-) - Arnaldo ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH] tools/perf: doc: Add permission and sysctl notice 2021-05-25 4:37 [PATCH] perf probe: Provide more detail with relocation warning Ravi Bangoria 2021-05-25 12:48 ` Masami Hiramatsu @ 2021-05-26 9:01 ` Masami Hiramatsu 2021-05-26 9:50 ` Ravi Bangoria 2021-05-26 14:51 ` [PATCH v2] " Masami Hiramatsu 1 sibling, 2 replies; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-26 9:01 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, acme Cc: Ravi Bangoria, Jiri Olsa, jolsa, Masami Hiramatsu, linux-kernel, Aneesh Kumar K . V Add a section to notify the permission and sysctl setting for perf probe. And fix some indentations. Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> --- tools/perf/Documentation/perf-probe.txt | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt index ed3ecfa422e1..38f60ac58a8e 100644 --- a/tools/perf/Documentation/perf-probe.txt +++ b/tools/perf/Documentation/perf-probe.txt @@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And LAZY MATCHING ------------- - The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). +The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). e.g. 'a=*' can matches 'a=b', 'a = b', 'a == b' and so on. @@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions FILTER PATTERN -------------- - The filter pattern is a glob matching pattern(s) to filter variables. - In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". +The filter pattern is a glob matching pattern(s) to filter variables. +In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". e.g. With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar". @@ -295,6 +295,17 @@ Add a probe in a source file using special characters by backslash escape ./perf probe -x /opt/test/a.out 'foo\+bar.c:4' +PERMISSIONS AND SYSCTL +---------------------- +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs. + + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option) + + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. + + - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files. + + SEE ALSO -------- linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1] ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] tools/perf: doc: Add permission and sysctl notice 2021-05-26 9:01 ` [PATCH] tools/perf: doc: Add permission and sysctl notice Masami Hiramatsu @ 2021-05-26 9:50 ` Ravi Bangoria 2021-05-26 13:16 ` Masami Hiramatsu 2021-05-26 14:51 ` [PATCH v2] " Masami Hiramatsu 1 sibling, 1 reply; 16+ messages in thread From: Ravi Bangoria @ 2021-05-26 9:50 UTC (permalink / raw) To: Masami Hiramatsu Cc: Jiri Olsa, jolsa, linux-kernel, Aneesh Kumar K . V, Arnaldo Carvalho de Melo, acme, Ravi Bangoria On 5/26/21 2:31 PM, Masami Hiramatsu wrote: > Add a section to notify the permission and sysctl setting > for perf probe. And fix some indentations. Yes, it makes sense to have this detail in man page. Few minor points below ... > > +PERMISSIONS AND SYSCTL > +---------------------- > +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs. > + > + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option) > + > + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. > + > + - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files. 1) Last two points are applicable to kprobes only, not uprobes. Would it make sense to clarify that? 2) For 3rd point, simple perf probe on function entry will work without vmlinux/debuginfo (by using kallsyms). Should we mention that? In any case, Acked-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Ravi ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] tools/perf: doc: Add permission and sysctl notice 2021-05-26 9:50 ` Ravi Bangoria @ 2021-05-26 13:16 ` Masami Hiramatsu 0 siblings, 0 replies; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-26 13:16 UTC (permalink / raw) To: Ravi Bangoria Cc: Jiri Olsa, jolsa, linux-kernel, Aneesh Kumar K . V, Arnaldo Carvalho de Melo, acme On Wed, 26 May 2021 15:20:58 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote: > > > On 5/26/21 2:31 PM, Masami Hiramatsu wrote: > > Add a section to notify the permission and sysctl setting > > for perf probe. And fix some indentations. > > Yes, it makes sense to have this detail in man page. > > Few minor points below ... > > > > > +PERMISSIONS AND SYSCTL > > +---------------------- > > +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs. > > + > > + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option) > > + > > + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. > > + > > + - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files. > > 1) Last two points are applicable to kprobes only, not uprobes. Would > it make sense to clarify that? Ah, right. And the last one may be also related to uprobes. > 2) For 3rd point, simple perf probe on function entry will work without > vmlinux/debuginfo (by using kallsyms). Should we mention that? Hmm, that depends on the options. If we use -k option, it will get the symbols from the given vmlinux. > > In any case, > > Acked-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Thanks! I'll update it for uprobe. > > Ravi -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2] tools/perf: doc: Add permission and sysctl notice 2021-05-26 9:01 ` [PATCH] tools/perf: doc: Add permission and sysctl notice Masami Hiramatsu 2021-05-26 9:50 ` Ravi Bangoria @ 2021-05-26 14:51 ` Masami Hiramatsu 2021-06-02 11:53 ` Arnaldo Carvalho de Melo 1 sibling, 1 reply; 16+ messages in thread From: Masami Hiramatsu @ 2021-05-26 14:51 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, acme Cc: Ravi Bangoria, Jiri Olsa, jolsa, Masami Hiramatsu, linux-kernel, Aneesh Kumar K . V Add a section to notify the permission and sysctl setting for perf probe. And fix some indentations. Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> --- Changes in v2: - Add tracefs remount option for --list command. - Mention uprobe case for kptr_restrict and vmlinux/debuginfo permission. --- tools/perf/Documentation/perf-probe.txt | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt index ed3ecfa422e1..080981d38d7b 100644 --- a/tools/perf/Documentation/perf-probe.txt +++ b/tools/perf/Documentation/perf-probe.txt @@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And LAZY MATCHING ------------- - The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). +The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). e.g. 'a=*' can matches 'a=b', 'a = b', 'a == b' and so on. @@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions FILTER PATTERN -------------- - The filter pattern is a glob matching pattern(s) to filter variables. - In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". +The filter pattern is a glob matching pattern(s) to filter variables. +In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". e.g. With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar". @@ -295,6 +295,19 @@ Add a probe in a source file using special characters by backslash escape ./perf probe -x /opt/test/a.out 'foo\+bar.c:4' +PERMISSIONS AND SYSCTL +---------------------- +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs. + + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option) + + - The system admin can remount the tracefs with 755 (`sudo mount -o remount,mode=755 /sys/kernel/tracing/`) to allow unprivileged user to run the perf probe --list command. + + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. Since the user-space probe doesn't need to access kallsyms, this is only for probing the kernel function (kprobes). + + - Since the perf probe commands read the vmlinux (for kernel) and/or the debuginfo file (including user-space application), you need to ensure that you can read those files. + + SEE ALSO -------- linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1] ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice 2021-05-26 14:51 ` [PATCH v2] " Masami Hiramatsu @ 2021-06-02 11:53 ` Arnaldo Carvalho de Melo 2021-06-02 13:51 ` Ravi Bangoria 0 siblings, 1 reply; 16+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-06-02 11:53 UTC (permalink / raw) To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, Ravi Bangoria, Jiri Olsa, jolsa, linux-kernel, Aneesh Kumar K . V Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu: > Add a section to notify the permission and sysctl setting > for perf probe. And fix some indentations. > > Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Ravi, can I have your Reviewed-by? - Arnaldo > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> > --- > Changes in v2: > - Add tracefs remount option for --list command. > - Mention uprobe case for kptr_restrict and vmlinux/debuginfo permission. > --- > tools/perf/Documentation/perf-probe.txt | 19 ++++++++++++++++--- > 1 file changed, 16 insertions(+), 3 deletions(-) > > diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt > index ed3ecfa422e1..080981d38d7b 100644 > --- a/tools/perf/Documentation/perf-probe.txt > +++ b/tools/perf/Documentation/perf-probe.txt > @@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And > > LAZY MATCHING > ------------- > - The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). > +The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]). > > e.g. > 'a=*' can matches 'a=b', 'a = b', 'a == b' and so on. > @@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions > > FILTER PATTERN > -------------- > - The filter pattern is a glob matching pattern(s) to filter variables. > - In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". > +The filter pattern is a glob matching pattern(s) to filter variables. > +In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")". > > e.g. > With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar". > @@ -295,6 +295,19 @@ Add a probe in a source file using special characters by backslash escape > ./perf probe -x /opt/test/a.out 'foo\+bar.c:4' > > > +PERMISSIONS AND SYSCTL > +---------------------- > +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs. > + > + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option) > + > + - The system admin can remount the tracefs with 755 (`sudo mount -o remount,mode=755 /sys/kernel/tracing/`) to allow unprivileged user to run the perf probe --list command. > + > + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. Since the user-space probe doesn't need to access kallsyms, this is only for probing the kernel function (kprobes). > + > + - Since the perf probe commands read the vmlinux (for kernel) and/or the debuginfo file (including user-space application), you need to ensure that you can read those files. > + > + > SEE ALSO > -------- > linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1] > -- - Arnaldo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice 2021-06-02 11:53 ` Arnaldo Carvalho de Melo @ 2021-06-02 13:51 ` Ravi Bangoria 2021-06-04 13:25 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 16+ messages in thread From: Ravi Bangoria @ 2021-06-02 13:51 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, Jiri Olsa, jolsa, linux-kernel, Aneesh Kumar K . V, Ravi Bangoria On 6/2/21 5:23 PM, Arnaldo Carvalho de Melo wrote: > Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu: >> Add a section to notify the permission and sysctl setting >> for perf probe. And fix some indentations. >> >> Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> > > Ravi, can I have your Reviewed-by? Yes please. Thanks for checking. Reviewed-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice 2021-06-02 13:51 ` Ravi Bangoria @ 2021-06-04 13:25 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 16+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-06-04 13:25 UTC (permalink / raw) To: Ravi Bangoria Cc: Masami Hiramatsu, Arnaldo Carvalho de Melo, Jiri Olsa, jolsa, linux-kernel, Aneesh Kumar K . V Em Wed, Jun 02, 2021 at 07:21:34PM +0530, Ravi Bangoria escreveu: > > > On 6/2/21 5:23 PM, Arnaldo Carvalho de Melo wrote: > > Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu: > > > Add a section to notify the permission and sysctl setting > > > for perf probe. And fix some indentations. > > > > > > Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> > > > > Ravi, can I have your Reviewed-by? > > Yes please. Thanks for checking. > > Reviewed-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com> Thanks, applied. - Arnaldo ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-06-04 13:25 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-25 4:37 [PATCH] perf probe: Provide more detail with relocation warning Ravi Bangoria 2021-05-25 12:48 ` Masami Hiramatsu 2021-05-26 4:53 ` Ravi Bangoria 2021-05-26 6:33 ` Masami Hiramatsu 2021-05-26 12:56 ` Arnaldo Carvalho de Melo 2021-05-26 14:20 ` Masami Hiramatsu 2021-06-02 11:52 ` Arnaldo Carvalho de Melo 2021-06-02 12:12 ` Masami Hiramatsu 2021-06-02 13:15 ` Arnaldo Carvalho de Melo 2021-05-26 9:01 ` [PATCH] tools/perf: doc: Add permission and sysctl notice Masami Hiramatsu 2021-05-26 9:50 ` Ravi Bangoria 2021-05-26 13:16 ` Masami Hiramatsu 2021-05-26 14:51 ` [PATCH v2] " Masami Hiramatsu 2021-06-02 11:53 ` Arnaldo Carvalho de Melo 2021-06-02 13:51 ` Ravi Bangoria 2021-06-04 13:25 ` Arnaldo Carvalho de Melo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.