[PATCH] perf probe: Provide more detail with relocation warning

[PATCH] perf probe: Provide more detail with relocation warning

[Ravi Bangoria]

When run as normal user with default sysctl kernel.kptr_restrict=0
and kernel.perf_event_paranoid=2, perf probe fails with:

  $ ./perf probe move_page_tables
  Relocated base symbol is not found!

The warning message is not much informative. The reason perf fails
is because /proc/kallsyms is restricted by perf_event_paranoid=2
for normal user and thus perf fails to read relocated address of
the base symbol.

Tweaking kptr_restrict and perf_event_paranoid can change the
behavior of perf probe. Also, running as root or privileged user
works too. Add these details in the warning message.

Plus, kmap->ref_reloc_sym might not be always set even if
host_machine is initialized. Above is the example of the same.
Remove that comment.

Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
---
 tools/perf/util/probe-event.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index a78c8d59a555..3a7649835ec9 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -108,7 +108,6 @@ void exit_probe_symbol_maps(void)
 
 static struct ref_reloc_sym *kernel_get_ref_reloc_sym(struct map **pmap)
 {
-	/* kmap->ref_reloc_sym should be set if host_machine is initialized */
 	struct kmap *kmap;
 	struct map *map = machine__kernel_map(host_machine);
 
@@ -819,7 +818,10 @@ post_process_kernel_probe_trace_events(struct probe_trace_event *tevs,
 
 	reloc_sym = kernel_get_ref_reloc_sym(&map);
 	if (!reloc_sym) {
-		pr_warning("Relocated base symbol is not found!\n");
+		pr_warning("Relocated base symbol is not found! "
+			   "Check /proc/sys/kernel/kptr_restrict\n"
+			   "and /proc/sys/kernel/perf_event_paranoid. "
+			   "Or run as privileged perf user.\n\n");
 		return -EINVAL;
 	}
 
@@ -3025,7 +3027,10 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
 			(!pp->retprobe || kretprobe_offset_is_supported())) {
 		reloc_sym = kernel_get_ref_reloc_sym(NULL);
 		if (!reloc_sym) {
-			pr_warning("Relocated base symbol is not found!\n");
+			pr_warning("Relocated base symbol is not found! "
+				   "Check /proc/sys/kernel/kptr_restrict\n"
+				   "and /proc/sys/kernel/perf_event_paranoid. "
+				   "Or run as privileged perf user.\n\n");
 			ret = -EINVAL;
 			goto out;
 		}
-- 
2.31.1

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Masami Hiramatsu]

On Tue, 25 May 2021 10:07:44 +0530
Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:

> When run as normal user with default sysctl kernel.kptr_restrict=0
> and kernel.perf_event_paranoid=2, perf probe fails with:
> 
>   $ ./perf probe move_page_tables
>   Relocated base symbol is not found!
> 
> The warning message is not much informative. The reason perf fails
> is because /proc/kallsyms is restricted by perf_event_paranoid=2
> for normal user and thus perf fails to read relocated address of
> the base symbol.
> 
> Tweaking kptr_restrict and perf_event_paranoid can change the
> behavior of perf probe. Also, running as root or privileged user
> works too. Add these details in the warning message.
> 
> Plus, kmap->ref_reloc_sym might not be always set even if
> host_machine is initialized. Above is the example of the same.
> Remove that comment.

Yes, those are restricted in some cases. Anyway without priviledged
(super) user, perf probe can not set the probe in ftrace.

Hmm, I think it should check the effective user-id at first. If it
is not super user and the action will access tracefs and kallsyms,
it should warn at that point. 

Thank you,

> 
> Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> ---
>  tools/perf/util/probe-event.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index a78c8d59a555..3a7649835ec9 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -108,7 +108,6 @@ void exit_probe_symbol_maps(void)
>  
>  static struct ref_reloc_sym *kernel_get_ref_reloc_sym(struct map **pmap)
>  {
> -	/* kmap->ref_reloc_sym should be set if host_machine is initialized */
>  	struct kmap *kmap;
>  	struct map *map = machine__kernel_map(host_machine);
>  
> @@ -819,7 +818,10 @@ post_process_kernel_probe_trace_events(struct probe_trace_event *tevs,
>  
>  	reloc_sym = kernel_get_ref_reloc_sym(&map);
>  	if (!reloc_sym) {
> -		pr_warning("Relocated base symbol is not found!\n");
> +		pr_warning("Relocated base symbol is not found! "
> +			   "Check /proc/sys/kernel/kptr_restrict\n"
> +			   "and /proc/sys/kernel/perf_event_paranoid. "
> +			   "Or run as privileged perf user.\n\n");
>  		return -EINVAL;
>  	}
>  
> @@ -3025,7 +3027,10 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
>  			(!pp->retprobe || kretprobe_offset_is_supported())) {
>  		reloc_sym = kernel_get_ref_reloc_sym(NULL);
>  		if (!reloc_sym) {
> -			pr_warning("Relocated base symbol is not found!\n");
> +			pr_warning("Relocated base symbol is not found! "
> +				   "Check /proc/sys/kernel/kptr_restrict\n"
> +				   "and /proc/sys/kernel/perf_event_paranoid. "
> +				   "Or run as privileged perf user.\n\n");
>  			ret = -EINVAL;
>  			goto out;
>  		}
> -- 
> 2.31.1
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Ravi Bangoria]

On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> On Tue, 25 May 2021 10:07:44 +0530
> Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> 
>> When run as normal user with default sysctl kernel.kptr_restrict=0
>> and kernel.perf_event_paranoid=2, perf probe fails with:
>>
>>    $ ./perf probe move_page_tables
>>    Relocated base symbol is not found!
>>
>> The warning message is not much informative. The reason perf fails
>> is because /proc/kallsyms is restricted by perf_event_paranoid=2
>> for normal user and thus perf fails to read relocated address of
>> the base symbol.
>>
>> Tweaking kptr_restrict and perf_event_paranoid can change the
>> behavior of perf probe. Also, running as root or privileged user
>> works too. Add these details in the warning message.
>>
>> Plus, kmap->ref_reloc_sym might not be always set even if
>> host_machine is initialized. Above is the example of the same.
>> Remove that comment.
> 
> Yes, those are restricted in some cases. Anyway without priviledged
> (super) user, perf probe can not set the probe in ftrace.
> 
> Hmm, I think it should check the effective user-id at first. If it
> is not super user and the action will access tracefs and kallsyms,
> it should warn at that point.

If kptr_restrict=2, perf probe fails with same error even for root user.
That's why I thought to just change this warning message.

Different combinations of privilege, perf_event_paranoid, kptr_restrict:

   Normal/Root user
    |   perf_event_paranoid
    V    V   kptr_restrict        perf probe error
   ----------------------------------------------------------------
    N   -1    0     Failed to open kprobe_events: Permission denied
    N    0    0     Failed to open kprobe_events: Permission denied
    N    1    0     Failed to open kprobe_events: Permission denied
    N    2    0     Relocated base symbol is not found!
   
    N   -1    1     Relocated base symbol is not found!
    N    0    1     Relocated base symbol is not found!
    N    1    1     Relocated base symbol is not found!
    N    2    1     Relocated base symbol is not found!
   
    N   -1    2     Relocated base symbol is not found!
    N    0    2     Relocated base symbol is not found!
    N    1    2     Relocated base symbol is not found!
    N    2    2     Relocated base symbol is not found!
   
    R   -1    0     No error.
    R    0    0     No error.
    R    1    0     No error.
    R    2    0     No error.
   
    R   -1    1     No error.
    R    0    1     No error.
    R    1    1     No error.
    R    2    1     No error.
   
    R   -1    2     Relocated base symbol is not found!
    R    0    2     Relocated base symbol is not found!
    R    1    2     Relocated base symbol is not found!
    R    2    2     Relocated base symbol is not found!

Ravi

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Masami Hiramatsu]

On Wed, 26 May 2021 10:23:18 +0530
Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:

> 
> 
> On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> > On Tue, 25 May 2021 10:07:44 +0530
> > Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > 
> >> When run as normal user with default sysctl kernel.kptr_restrict=0
> >> and kernel.perf_event_paranoid=2, perf probe fails with:
> >>
> >>    $ ./perf probe move_page_tables
> >>    Relocated base symbol is not found!
> >>
> >> The warning message is not much informative. The reason perf fails
> >> is because /proc/kallsyms is restricted by perf_event_paranoid=2
> >> for normal user and thus perf fails to read relocated address of
> >> the base symbol.
> >>
> >> Tweaking kptr_restrict and perf_event_paranoid can change the
> >> behavior of perf probe. Also, running as root or privileged user
> >> works too. Add these details in the warning message.
> >>
> >> Plus, kmap->ref_reloc_sym might not be always set even if
> >> host_machine is initialized. Above is the example of the same.
> >> Remove that comment.
> > 
> > Yes, those are restricted in some cases. Anyway without priviledged
> > (super) user, perf probe can not set the probe in ftrace.
> > 
> > Hmm, I think it should check the effective user-id at first. If it
> > is not super user and the action will access tracefs and kallsyms,
> > it should warn at that point.
> 
> If kptr_restrict=2, perf probe fails with same error even for root user.
> That's why I thought to just change this warning message.

Ah, yes. In that case, perf probe must not use the base symbol.
(like -D option)
OK, then, let's merge this fix.

Acked-by: Masami Hiramatsu <mhiramat@kernel.org>

Thank you,

> 
> Different combinations of privilege, perf_event_paranoid, kptr_restrict:
> 
>    Normal/Root user
>     |   perf_event_paranoid
>     V    V   kptr_restrict        perf probe error
>    ----------------------------------------------------------------
>     N   -1    0     Failed to open kprobe_events: Permission denied
>     N    0    0     Failed to open kprobe_events: Permission denied
>     N    1    0     Failed to open kprobe_events: Permission denied
>     N    2    0     Relocated base symbol is not found!
>    
>     N   -1    1     Relocated base symbol is not found!
>     N    0    1     Relocated base symbol is not found!
>     N    1    1     Relocated base symbol is not found!
>     N    2    1     Relocated base symbol is not found!
>    
>     N   -1    2     Relocated base symbol is not found!
>     N    0    2     Relocated base symbol is not found!
>     N    1    2     Relocated base symbol is not found!
>     N    2    2     Relocated base symbol is not found!
>    
>     R   -1    0     No error.
>     R    0    0     No error.
>     R    1    0     No error.
>     R    2    0     No error.
>    
>     R   -1    1     No error.
>     R    0    1     No error.
>     R    1    1     No error.
>     R    2    1     No error.
>    
>     R   -1    2     Relocated base symbol is not found!
>     R    0    2     Relocated base symbol is not found!
>     R    1    2     Relocated base symbol is not found!
>     R    2    2     Relocated base symbol is not found!
> 
> Ravi


-- 
Masami Hiramatsu <mhiramat@kernel.org>

[PATCH] tools/perf: doc: Add permission and sysctl notice

[Masami Hiramatsu]

Add a section to notify the permission and sysctl setting
for perf probe. And fix some indentations.

Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 tools/perf/Documentation/perf-probe.txt |   17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt
index ed3ecfa422e1..38f60ac58a8e 100644
--- a/tools/perf/Documentation/perf-probe.txt
+++ b/tools/perf/Documentation/perf-probe.txt
@@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And
 
 LAZY MATCHING
 -------------
- The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
+The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
 
 e.g.
  'a=*' can matches 'a=b', 'a = b', 'a == b' and so on.
@@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions
 
 FILTER PATTERN
 --------------
- The filter pattern is a glob matching pattern(s) to filter variables.
- In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
+The filter pattern is a glob matching pattern(s) to filter variables.
+In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
 
 e.g.
  With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar".
@@ -295,6 +295,17 @@ Add a probe in a source file using special characters by backslash escape
  ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
 
 
+PERMISSIONS AND SYSCTL
+----------------------
+Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
+
+ - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
+
+ - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands.
+
+ - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files.
+
+
 SEE ALSO
 --------
 linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1]

Re: [PATCH] tools/perf: doc: Add permission and sysctl notice

[Ravi Bangoria]

On 5/26/21 2:31 PM, Masami Hiramatsu wrote:
> Add a section to notify the permission and sysctl setting
> for perf probe. And fix some indentations.

Yes, it makes sense to have this detail in man page.

Few minor points below ...

>   
> +PERMISSIONS AND SYSCTL
> +----------------------
> +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
> +
> + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
> +
> + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands.
> +
> + - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files.

1) Last two points are applicable to kprobes only, not uprobes. Would
    it make sense to clarify that?
2) For 3rd point, simple perf probe on function entry will work without
    vmlinux/debuginfo (by using kallsyms). Should we mention that?

In any case,

Acked-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>

Ravi

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Arnaldo Carvalho de Melo]

Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu:
> On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:

> > >> When run as normal user with default sysctl kernel.kptr_restrict=0
> > >> and kernel.perf_event_paranoid=2, perf probe fails with:

> > >>    $ ./perf probe move_page_tables
> > >>    Relocated base symbol is not found!

> > >> The warning message is not much informative. The reason perf
> > >> fails is because /proc/kallsyms is restricted by
> > >> perf_event_paranoid=2 for normal user and thus perf fails to read
> > >> relocated address of the base symbol.

> > >> Tweaking kptr_restrict and perf_event_paranoid can change the
> > >> behavior of perf probe. Also, running as root or privileged user
> > >> works too. Add these details in the warning message.

> > >> Plus, kmap->ref_reloc_sym might not be always set even if
> > >> host_machine is initialized. Above is the example of the same.
> > >> Remove that comment.

> > > Yes, those are restricted in some cases. Anyway without priviledged
> > > (super) user, perf probe can not set the probe in ftrace.

> > > Hmm, I think it should check the effective user-id at first. If it
> > > is not super user and the action will access tracefs and kallsyms,
> > > it should warn at that point.

> > If kptr_restrict=2, perf probe fails with same error even for root user.
> > That's why I thought to just change this warning message.

> Ah, yes. In that case, perf probe must not use the base symbol.
> (like -D option)
> OK, then, let's merge this fix.

> Acked-by: Masami Hiramatsu <mhiramat@kernel.org>

Thanks, applied as it improves the current situation.

But as a follow up, to further improve this, we can reuse what 'perf trace' has:

  $ perf trace sleep 1
  Error:	No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit)
  Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/'
  $ sudo mount -o remount,mode=755 /sys/kernel/tracing/
  $ perf trace sleep 1
  Error:	Permission denied.
  Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
  Hint:	For your workloads it needs to be <= 1
  Hint:	For system wide tracing it needs to be set to -1.
  Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
  Hint:	The current value is 2.
  $ 


I.e. go the extra step and show what the current value is and what it
needs to be to achieve what is being attempted.

IOW combine error message with relevant documentation, to save steps.

See what 'perf top' does for an unpriv user:

  $ perf top --stdio
  Error:
  Access to performance monitoring and observability operations is limited.
  Enforced MAC policy settings (SELinux) can limit access to performance
  monitoring and observability operations. Inspect system audit records for
  more perf_event access control information and adjusting the policy.
  Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
  access to performance monitoring and observability operations for processes
  without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
  More information can be found at 'Perf events and tool security' document:
  https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
  perf_event_paranoid setting is 2:
    -1: Allow use of (almost) all events by all users
        Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
  >= 0: Disallow raw and ftrace function tracepoint access
  >= 1: Disallow CPU event access
  >= 2: Disallow kernel profiling
  To make the adjusted perf_event_paranoid setting permanent preserve it
  in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)
  $

- Arnaldo

> 
> > 
> > Different combinations of privilege, perf_event_paranoid, kptr_restrict:
> > 
> >    Normal/Root user
> >     |   perf_event_paranoid
> >     V    V   kptr_restrict        perf probe error
> >    ----------------------------------------------------------------
> >     N   -1    0     Failed to open kprobe_events: Permission denied
> >     N    0    0     Failed to open kprobe_events: Permission denied
> >     N    1    0     Failed to open kprobe_events: Permission denied
> >     N    2    0     Relocated base symbol is not found!
> >    
> >     N   -1    1     Relocated base symbol is not found!
> >     N    0    1     Relocated base symbol is not found!
> >     N    1    1     Relocated base symbol is not found!
> >     N    2    1     Relocated base symbol is not found!
> >    
> >     N   -1    2     Relocated base symbol is not found!
> >     N    0    2     Relocated base symbol is not found!
> >     N    1    2     Relocated base symbol is not found!
> >     N    2    2     Relocated base symbol is not found!
> >    
> >     R   -1    0     No error.
> >     R    0    0     No error.
> >     R    1    0     No error.
> >     R    2    0     No error.
> >    
> >     R   -1    1     No error.
> >     R    0    1     No error.
> >     R    1    1     No error.
> >     R    2    1     No error.
> >    
> >     R   -1    2     Relocated base symbol is not found!
> >     R    0    2     Relocated base symbol is not found!
> >     R    1    2     Relocated base symbol is not found!
> >     R    2    2     Relocated base symbol is not found!
> > 
> > Ravi
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>

-- 

- Arnaldo

Re: [PATCH] tools/perf: doc: Add permission and sysctl notice

[Masami Hiramatsu]

On Wed, 26 May 2021 15:20:58 +0530
Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:

> 
> 
> On 5/26/21 2:31 PM, Masami Hiramatsu wrote:
> > Add a section to notify the permission and sysctl setting
> > for perf probe. And fix some indentations.
> 
> Yes, it makes sense to have this detail in man page.
> 
> Few minor points below ...
> 
> >   
> > +PERMISSIONS AND SYSCTL
> > +----------------------
> > +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
> > +
> > + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
> > +
> > + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands.
> > +
> > + - Since the perf probe commands read the vmlinux and/or the debuginfo file, you need to ensure that you can read those files.
> 
> 1) Last two points are applicable to kprobes only, not uprobes. Would
>     it make sense to clarify that?

Ah, right. And the last one may be also related to uprobes.

> 2) For 3rd point, simple perf probe on function entry will work without
>     vmlinux/debuginfo (by using kallsyms). Should we mention that?

Hmm, that depends on the options. If we use -k option, it will get the symbols from the given vmlinux.

> 
> In any case,
> 
> Acked-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>

Thanks! I'll update it for uprobe.


> 
> Ravi


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Masami Hiramatsu]

On Wed, 26 May 2021 09:56:29 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu:
> > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> 
> > > >> When run as normal user with default sysctl kernel.kptr_restrict=0
> > > >> and kernel.perf_event_paranoid=2, perf probe fails with:
> 
> > > >>    $ ./perf probe move_page_tables
> > > >>    Relocated base symbol is not found!
> 
> > > >> The warning message is not much informative. The reason perf
> > > >> fails is because /proc/kallsyms is restricted by
> > > >> perf_event_paranoid=2 for normal user and thus perf fails to read
> > > >> relocated address of the base symbol.
> 
> > > >> Tweaking kptr_restrict and perf_event_paranoid can change the
> > > >> behavior of perf probe. Also, running as root or privileged user
> > > >> works too. Add these details in the warning message.
> 
> > > >> Plus, kmap->ref_reloc_sym might not be always set even if
> > > >> host_machine is initialized. Above is the example of the same.
> > > >> Remove that comment.
> 
> > > > Yes, those are restricted in some cases. Anyway without priviledged
> > > > (super) user, perf probe can not set the probe in ftrace.
> 
> > > > Hmm, I think it should check the effective user-id at first. If it
> > > > is not super user and the action will access tracefs and kallsyms,
> > > > it should warn at that point.
> 
> > > If kptr_restrict=2, perf probe fails with same error even for root user.
> > > That's why I thought to just change this warning message.
> 
> > Ah, yes. In that case, perf probe must not use the base symbol.
> > (like -D option)
> > OK, then, let's merge this fix.
> 
> > Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
> 
> Thanks, applied as it improves the current situation.
> 
> But as a follow up, to further improve this, we can reuse what 'perf trace' has:
> 
>   $ perf trace sleep 1
>   Error:	No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit)
>   Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/'
>   $ sudo mount -o remount,mode=755 /sys/kernel/tracing/
>   $ perf trace sleep 1
>   Error:	Permission denied.
>   Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
>   Hint:	For your workloads it needs to be <= 1
>   Hint:	For system wide tracing it needs to be set to -1.
>   Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
>   Hint:	The current value is 2.
>   $ 

OK, let me check this.
BTW, does perf_event_paranoid affect only perf syscall (and kallsyms),
not the tracefs correct?

> I.e. go the extra step and show what the current value is and what it
> needs to be to achieve what is being attempted.
> 
> IOW combine error message with relevant documentation, to save steps.
> 
> See what 'perf top' does for an unpriv user:
> 
>   $ perf top --stdio
>   Error:
>   Access to performance monitoring and observability operations is limited.
>   Enforced MAC policy settings (SELinux) can limit access to performance
>   monitoring and observability operations. Inspect system audit records for
>   more perf_event access control information and adjusting the policy.
>   Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
>   access to performance monitoring and observability operations for processes
>   without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
>   More information can be found at 'Perf events and tool security' document:
>   https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
>   perf_event_paranoid setting is 2:
>     -1: Allow use of (almost) all events by all users
>         Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>   >= 0: Disallow raw and ftrace function tracepoint access
>   >= 1: Disallow CPU event access
>   >= 2: Disallow kernel profiling
>   To make the adjusted perf_event_paranoid setting permanent preserve it
>   in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)

Hmm, I would rather like pointing manpages...

Would we better to have perf-security.7 manpage?

Thank you,

>   $
> 
> - Arnaldo
> 
> > 
> > > 
> > > Different combinations of privilege, perf_event_paranoid, kptr_restrict:
> > > 
> > >    Normal/Root user
> > >     |   perf_event_paranoid
> > >     V    V   kptr_restrict        perf probe error
> > >    ----------------------------------------------------------------
> > >     N   -1    0     Failed to open kprobe_events: Permission denied
> > >     N    0    0     Failed to open kprobe_events: Permission denied
> > >     N    1    0     Failed to open kprobe_events: Permission denied
> > >     N    2    0     Relocated base symbol is not found!
> > >    
> > >     N   -1    1     Relocated base symbol is not found!
> > >     N    0    1     Relocated base symbol is not found!
> > >     N    1    1     Relocated base symbol is not found!
> > >     N    2    1     Relocated base symbol is not found!
> > >    
> > >     N   -1    2     Relocated base symbol is not found!
> > >     N    0    2     Relocated base symbol is not found!
> > >     N    1    2     Relocated base symbol is not found!
> > >     N    2    2     Relocated base symbol is not found!
> > >    
> > >     R   -1    0     No error.
> > >     R    0    0     No error.
> > >     R    1    0     No error.
> > >     R    2    0     No error.
> > >    
> > >     R   -1    1     No error.
> > >     R    0    1     No error.
> > >     R    1    1     No error.
> > >     R    2    1     No error.
> > >    
> > >     R   -1    2     Relocated base symbol is not found!
> > >     R    0    2     Relocated base symbol is not found!
> > >     R    1    2     Relocated base symbol is not found!
> > >     R    2    2     Relocated base symbol is not found!
> > > 
> > > Ravi
> > 
> > 
> > -- 
> > Masami Hiramatsu <mhiramat@kernel.org>
> 
> -- 
> 
> - Arnaldo


-- 
Masami Hiramatsu <mhiramat@kernel.org>

[PATCH v2] tools/perf: doc: Add permission and sysctl notice

[Masami Hiramatsu]

Add a section to notify the permission and sysctl setting
for perf probe. And fix some indentations.

Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 Changes in v2:
  - Add tracefs remount option for --list command.
  - Mention uprobe case for kptr_restrict and vmlinux/debuginfo permission.
---
 tools/perf/Documentation/perf-probe.txt |   19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt
index ed3ecfa422e1..080981d38d7b 100644
--- a/tools/perf/Documentation/perf-probe.txt
+++ b/tools/perf/Documentation/perf-probe.txt
@@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And
 
 LAZY MATCHING
 -------------
- The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
+The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
 
 e.g.
  'a=*' can matches 'a=b', 'a = b', 'a == b' and so on.
@@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions
 
 FILTER PATTERN
 --------------
- The filter pattern is a glob matching pattern(s) to filter variables.
- In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
+The filter pattern is a glob matching pattern(s) to filter variables.
+In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
 
 e.g.
  With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar".
@@ -295,6 +295,19 @@ Add a probe in a source file using special characters by backslash escape
  ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
 
 
+PERMISSIONS AND SYSCTL
+----------------------
+Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
+
+ - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
+
+ - The system admin can remount the tracefs with 755 (`sudo mount -o remount,mode=755 /sys/kernel/tracing/`) to allow unprivileged user to run the perf probe --list command.
+
+ - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. Since the user-space probe doesn't need to access kallsyms, this is only for probing the kernel function (kprobes).
+
+ - Since the perf probe commands read the vmlinux (for kernel) and/or the debuginfo file (including user-space application), you need to ensure that you can read those files.
+
+
 SEE ALSO
 --------
 linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1]

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Arnaldo Carvalho de Melo]

Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu:
> On Wed, 26 May 2021 09:56:29 -0300
> Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu:
> > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > 
> > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0
> > > > >> and kernel.perf_event_paranoid=2, perf probe fails with:
> > 
> > > > >>    $ ./perf probe move_page_tables
> > > > >>    Relocated base symbol is not found!
> > 
> > > > >> The warning message is not much informative. The reason perf
> > > > >> fails is because /proc/kallsyms is restricted by
> > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read
> > > > >> relocated address of the base symbol.
> > 
> > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the
> > > > >> behavior of perf probe. Also, running as root or privileged user
> > > > >> works too. Add these details in the warning message.
> > 
> > > > >> Plus, kmap->ref_reloc_sym might not be always set even if
> > > > >> host_machine is initialized. Above is the example of the same.
> > > > >> Remove that comment.
> > 
> > > > > Yes, those are restricted in some cases. Anyway without priviledged
> > > > > (super) user, perf probe can not set the probe in ftrace.
> > 
> > > > > Hmm, I think it should check the effective user-id at first. If it
> > > > > is not super user and the action will access tracefs and kallsyms,
> > > > > it should warn at that point.
> > 
> > > > If kptr_restrict=2, perf probe fails with same error even for root user.
> > > > That's why I thought to just change this warning message.
> > 
> > > Ah, yes. In that case, perf probe must not use the base symbol.
> > > (like -D option)
> > > OK, then, let's merge this fix.
> > 
> > > Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
> > 
> > Thanks, applied as it improves the current situation.
> > 
> > But as a follow up, to further improve this, we can reuse what 'perf trace' has:
> > 
> >   $ perf trace sleep 1
> >   Error:	No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit)
> >   Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/'
> >   $ sudo mount -o remount,mode=755 /sys/kernel/tracing/
> >   $ perf trace sleep 1
> >   Error:	Permission denied.
> >   Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
> >   Hint:	For your workloads it needs to be <= 1
> >   Hint:	For system wide tracing it needs to be set to -1.
> >   Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
> >   Hint:	The current value is 2.
> >   $ 
> 
> OK, let me check this.
> BTW, does perf_event_paranoid affect only perf syscall (and kallsyms),
> not the tracefs correct?
> 
> > I.e. go the extra step and show what the current value is and what it
> > needs to be to achieve what is being attempted.
> > 
> > IOW combine error message with relevant documentation, to save steps.
> > 
> > See what 'perf top' does for an unpriv user:
> > 
> >   $ perf top --stdio
> >   Error:
> >   Access to performance monitoring and observability operations is limited.
> >   Enforced MAC policy settings (SELinux) can limit access to performance
> >   monitoring and observability operations. Inspect system audit records for
> >   more perf_event access control information and adjusting the policy.
> >   Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
> >   access to performance monitoring and observability operations for processes
> >   without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
> >   More information can be found at 'Perf events and tool security' document:
> >   https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
> >   perf_event_paranoid setting is 2:
> >     -1: Allow use of (almost) all events by all users
> >         Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
> >   >= 0: Disallow raw and ftrace function tracepoint access
> >   >= 1: Disallow CPU event access
> >   >= 2: Disallow kernel profiling
> >   To make the adjusted perf_event_paranoid setting permanent preserve it
> >   in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)
> 
> Hmm, I would rather like pointing manpages...

Man pages are long, if you quote the relevant part of it when the
problem takes place, IMHO it helps the user.

- Arnaldo
 
> Would we better to have perf-security.7 manpage?
> 
> Thank you,
> 
> >   $
> > 
> > - Arnaldo
> > 
> > > 
> > > > 
> > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict:
> > > > 
> > > >    Normal/Root user
> > > >     |   perf_event_paranoid
> > > >     V    V   kptr_restrict        perf probe error
> > > >    ----------------------------------------------------------------
> > > >     N   -1    0     Failed to open kprobe_events: Permission denied
> > > >     N    0    0     Failed to open kprobe_events: Permission denied
> > > >     N    1    0     Failed to open kprobe_events: Permission denied
> > > >     N    2    0     Relocated base symbol is not found!
> > > >    
> > > >     N   -1    1     Relocated base symbol is not found!
> > > >     N    0    1     Relocated base symbol is not found!
> > > >     N    1    1     Relocated base symbol is not found!
> > > >     N    2    1     Relocated base symbol is not found!
> > > >    
> > > >     N   -1    2     Relocated base symbol is not found!
> > > >     N    0    2     Relocated base symbol is not found!
> > > >     N    1    2     Relocated base symbol is not found!
> > > >     N    2    2     Relocated base symbol is not found!
> > > >    
> > > >     R   -1    0     No error.
> > > >     R    0    0     No error.
> > > >     R    1    0     No error.
> > > >     R    2    0     No error.
> > > >    
> > > >     R   -1    1     No error.
> > > >     R    0    1     No error.
> > > >     R    1    1     No error.
> > > >     R    2    1     No error.
> > > >    
> > > >     R   -1    2     Relocated base symbol is not found!
> > > >     R    0    2     Relocated base symbol is not found!
> > > >     R    1    2     Relocated base symbol is not found!
> > > >     R    2    2     Relocated base symbol is not found!
> > > > 
> > > > Ravi
> > > 
> > > 
> > > -- 
> > > Masami Hiramatsu <mhiramat@kernel.org>
> > 
> > -- 
> > 
> > - Arnaldo
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>

-- 

- Arnaldo

Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice

[Arnaldo Carvalho de Melo]

Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu:
> Add a section to notify the permission and sysctl setting
> for perf probe. And fix some indentations.
> 
> Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>

Ravi, can I have your Reviewed-by?

- Arnaldo

> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> ---
>  Changes in v2:
>   - Add tracefs remount option for --list command.
>   - Mention uprobe case for kptr_restrict and vmlinux/debuginfo permission.
> ---
>  tools/perf/Documentation/perf-probe.txt |   19 ++++++++++++++++---
>  1 file changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt
> index ed3ecfa422e1..080981d38d7b 100644
> --- a/tools/perf/Documentation/perf-probe.txt
> +++ b/tools/perf/Documentation/perf-probe.txt
> @@ -226,7 +226,7 @@ So, "source.c:100-120" shows lines between 100th to l20th in source.c file. And
>  
>  LAZY MATCHING
>  -------------
> - The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
> +The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
>  
>  e.g.
>   'a=*' can matches 'a=b', 'a = b', 'a == b' and so on.
> @@ -235,8 +235,8 @@ This provides some sort of flexibility and robustness to probe point definitions
>  
>  FILTER PATTERN
>  --------------
> - The filter pattern is a glob matching pattern(s) to filter variables.
> - In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
> +The filter pattern is a glob matching pattern(s) to filter variables.
> +In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
>  
>  e.g.
>   With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar".
> @@ -295,6 +295,19 @@ Add a probe in a source file using special characters by backslash escape
>   ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
>  
>  
> +PERMISSIONS AND SYSCTL
> +----------------------
> +Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
> +
> + - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
> +
> + - The system admin can remount the tracefs with 755 (`sudo mount -o remount,mode=755 /sys/kernel/tracing/`) to allow unprivileged user to run the perf probe --list command.
> +
> + - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. Since the user-space probe doesn't need to access kallsyms, this is only for probing the kernel function (kprobes).
> +
> + - Since the perf probe commands read the vmlinux (for kernel) and/or the debuginfo file (including user-space application), you need to ensure that you can read those files.
> +
> +
>  SEE ALSO
>  --------
>  linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1]
> 

-- 

- Arnaldo

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Masami Hiramatsu]

On Wed, 2 Jun 2021 08:52:41 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu:
> > On Wed, 26 May 2021 09:56:29 -0300
> > Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 
> > > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu:
> > > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote:
> > > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria <ravi.bangoria@linux.ibm.com> wrote:
> > > 
> > > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0
> > > > > >> and kernel.perf_event_paranoid=2, perf probe fails with:
> > > 
> > > > > >>    $ ./perf probe move_page_tables
> > > > > >>    Relocated base symbol is not found!
> > > 
> > > > > >> The warning message is not much informative. The reason perf
> > > > > >> fails is because /proc/kallsyms is restricted by
> > > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read
> > > > > >> relocated address of the base symbol.
> > > 
> > > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the
> > > > > >> behavior of perf probe. Also, running as root or privileged user
> > > > > >> works too. Add these details in the warning message.
> > > 
> > > > > >> Plus, kmap->ref_reloc_sym might not be always set even if
> > > > > >> host_machine is initialized. Above is the example of the same.
> > > > > >> Remove that comment.
> > > 
> > > > > > Yes, those are restricted in some cases. Anyway without priviledged
> > > > > > (super) user, perf probe can not set the probe in ftrace.
> > > 
> > > > > > Hmm, I think it should check the effective user-id at first. If it
> > > > > > is not super user and the action will access tracefs and kallsyms,
> > > > > > it should warn at that point.
> > > 
> > > > > If kptr_restrict=2, perf probe fails with same error even for root user.
> > > > > That's why I thought to just change this warning message.
> > > 
> > > > Ah, yes. In that case, perf probe must not use the base symbol.
> > > > (like -D option)
> > > > OK, then, let's merge this fix.
> > > 
> > > > Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
> > > 
> > > Thanks, applied as it improves the current situation.
> > > 
> > > But as a follow up, to further improve this, we can reuse what 'perf trace' has:
> > > 
> > >   $ perf trace sleep 1
> > >   Error:	No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit)
> > >   Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/'
> > >   $ sudo mount -o remount,mode=755 /sys/kernel/tracing/
> > >   $ perf trace sleep 1
> > >   Error:	Permission denied.
> > >   Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
> > >   Hint:	For your workloads it needs to be <= 1
> > >   Hint:	For system wide tracing it needs to be set to -1.
> > >   Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
> > >   Hint:	The current value is 2.
> > >   $ 
> > 
> > OK, let me check this.
> > BTW, does perf_event_paranoid affect only perf syscall (and kallsyms),
> > not the tracefs correct?
> > 
> > > I.e. go the extra step and show what the current value is and what it
> > > needs to be to achieve what is being attempted.
> > > 
> > > IOW combine error message with relevant documentation, to save steps.
> > > 
> > > See what 'perf top' does for an unpriv user:
> > > 
> > >   $ perf top --stdio
> > >   Error:
> > >   Access to performance monitoring and observability operations is limited.
> > >   Enforced MAC policy settings (SELinux) can limit access to performance
> > >   monitoring and observability operations. Inspect system audit records for
> > >   more perf_event access control information and adjusting the policy.
> > >   Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
> > >   access to performance monitoring and observability operations for processes
> > >   without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
> > >   More information can be found at 'Perf events and tool security' document:
> > >   https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
> > >   perf_event_paranoid setting is 2:
> > >     -1: Allow use of (almost) all events by all users
> > >         Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
> > >   >= 0: Disallow raw and ftrace function tracepoint access
> > >   >= 1: Disallow CPU event access
> > >   >= 2: Disallow kernel profiling
> > >   To make the adjusted perf_event_paranoid setting permanent preserve it
> > >   in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)
> > 
> > Hmm, I would rather like pointing manpages...
> 
> Man pages are long, if you quote the relevant part of it when the
> problem takes place, IMHO it helps the user.

OK, but please also update man pages (*), which can provide a summarized information
for users if they noticed that. And for each place, I will add some messages
for letting them know.

(*) https://lore.kernel.org/lkml/162204068898.388434.16842705842611255787.stgit@devnote2/

Thank you,

> 
> - Arnaldo
>  
> > Would we better to have perf-security.7 manpage?
> > 
> > Thank you,
> > 
> > >   $
> > > 
> > > - Arnaldo
> > > 
> > > > 
> > > > > 
> > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict:
> > > > > 
> > > > >    Normal/Root user
> > > > >     |   perf_event_paranoid
> > > > >     V    V   kptr_restrict        perf probe error
> > > > >    ----------------------------------------------------------------
> > > > >     N   -1    0     Failed to open kprobe_events: Permission denied
> > > > >     N    0    0     Failed to open kprobe_events: Permission denied
> > > > >     N    1    0     Failed to open kprobe_events: Permission denied
> > > > >     N    2    0     Relocated base symbol is not found!
> > > > >    
> > > > >     N   -1    1     Relocated base symbol is not found!
> > > > >     N    0    1     Relocated base symbol is not found!
> > > > >     N    1    1     Relocated base symbol is not found!
> > > > >     N    2    1     Relocated base symbol is not found!
> > > > >    
> > > > >     N   -1    2     Relocated base symbol is not found!
> > > > >     N    0    2     Relocated base symbol is not found!
> > > > >     N    1    2     Relocated base symbol is not found!
> > > > >     N    2    2     Relocated base symbol is not found!
> > > > >    
> > > > >     R   -1    0     No error.
> > > > >     R    0    0     No error.
> > > > >     R    1    0     No error.
> > > > >     R    2    0     No error.
> > > > >    
> > > > >     R   -1    1     No error.
> > > > >     R    0    1     No error.
> > > > >     R    1    1     No error.
> > > > >     R    2    1     No error.
> > > > >    
> > > > >     R   -1    2     Relocated base symbol is not found!
> > > > >     R    0    2     Relocated base symbol is not found!
> > > > >     R    1    2     Relocated base symbol is not found!
> > > > >     R    2    2     Relocated base symbol is not found!
> > > > > 
> > > > > Ravi
> > > > 
> > > > 
> > > > -- 
> > > > Masami Hiramatsu <mhiramat@kernel.org>
> > > 
> > > -- 
> > > 
> > > - Arnaldo
> > 
> > 
> > -- 
> > Masami Hiramatsu <mhiramat@kernel.org>
> 
> -- 
> 
> - Arnaldo


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] perf probe: Provide more detail with relocation warning

[Arnaldo Carvalho de Melo]

Em Wed, Jun 02, 2021 at 09:12:48PM +0900, Masami Hiramatsu escreveu:
> On Wed, 2 Jun 2021 08:52:41 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu:
> > > On Wed, 26 May 2021 09:56:29 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > > I.e. go the extra step and show what the current value is and what it
> > > > needs to be to achieve what is being attempted.

> > > > IOW combine error message with relevant documentation, to save steps.

> > > > See what 'perf top' does for an unpriv user:

> > > >   $ perf top --stdio
> > > >   Error:
> > > >   Access to performance monitoring and observability operations is limited.
> > > >   Enforced MAC policy settings (SELinux) can limit access to performance
> > > >   monitoring and observability operations. Inspect system audit records for
> > > >   more perf_event access control information and adjusting the policy.
> > > >   Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
> > > >   access to performance monitoring and observability operations for processes
> > > >   without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
> > > >   More information can be found at 'Perf events and tool security' document:
> > > >   https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
> > > >   perf_event_paranoid setting is 2:
> > > >     -1: Allow use of (almost) all events by all users
> > > >         Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
> > > >   >= 0: Disallow raw and ftrace function tracepoint access
> > > >   >= 1: Disallow CPU event access
> > > >   >= 2: Disallow kernel profiling
> > > >   To make the adjusted perf_event_paranoid setting permanent preserve it
> > > >   in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)

> > > Hmm, I would rather like pointing manpages...

> > Man pages are long, if you quote the relevant part of it when the
> > problem takes place, IMHO it helps the user.

> OK, but please also update man pages (*), which can provide a summarized information
> for users if they noticed that. And for each place, I will add some messages
> for letting them know.

Sure, its not one or the other, its both, i.e. man pages are interesting
as the standard reference for a command, while making tools provide
information about a specific problem with actionable advice on how to
fix the problem at hand is important as well.

> (*) https://lore.kernel.org/lkml/162204068898.388434.16842705842611255787.stgit@devnote2/

I saw your patch to the man page and asked Ravi for an Ack/Reviewed-by,

Thanks for all your continued work on this! :-)

- Arnaldo

Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice

[Ravi Bangoria]

On 6/2/21 5:23 PM, Arnaldo Carvalho de Melo wrote:
> Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu:
>> Add a section to notify the permission and sysctl setting
>> for perf probe. And fix some indentations.
>>
>> Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> 
> Ravi, can I have your Reviewed-by?

Yes please. Thanks for checking.

Reviewed-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>

Re: [PATCH v2] tools/perf: doc: Add permission and sysctl notice

[Arnaldo Carvalho de Melo]

Em Wed, Jun 02, 2021 at 07:21:34PM +0530, Ravi Bangoria escreveu:
> 
> 
> On 6/2/21 5:23 PM, Arnaldo Carvalho de Melo wrote:
> > Em Wed, May 26, 2021 at 11:51:29PM +0900, Masami Hiramatsu escreveu:
> > > Add a section to notify the permission and sysctl setting
> > > for perf probe. And fix some indentations.
> > > 
> > > Reported-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> > 
> > Ravi, can I have your Reviewed-by?
> 
> Yes please. Thanks for checking.
> 
> Reviewed-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>

Thanks, applied.

- Arnaldo