[BUG] perf probe can't remove probes

[BUG] perf probe can't remove probes

[Arnaldo Carvalho de Melo]

Masami, have you stumbled on this already?

[root@seventh ~]# perf probe security_locked_down%return 'ret=$retval'
Added new event:
  probe:security_locked_down__return (on security_locked_down%return with ret=$retval)

You can now use it in all perf tools, such as:

	perf record -e probe:security_locked_down__return -aR sleep 1

[root@seventh ~]# perf probe security_locked_down what
Added new event:
  probe:security_locked_down (on security_locked_down with what)

You can now use it in all perf tools, such as:

	perf record -e probe:security_locked_down -aR sleep 1

[root@seventh ~]#


[root@seventh ~]# uname -r
5.10.0-rc3.bpfsign+
[root@seventh ~]# perf probe -l
  probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what)
  probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
[root@seventh ~]# perf probe -D '*:*'
Semantic error :There is non-digit char in line number.

 Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]
    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
                          Show trace event definition of given traceevent for k/uprobe_events.
[root@seventh ~]# perf probe probe:security_locked_down
Semantic error :There is non-digit char in line number.
  Error: Command Parse Error.
[root@seventh ~]# perf probe probe:security_locked_down__return
Semantic error :There is non-digit char in line number.
  Error: Command Parse Error.
[root@seventh ~]# cat /sys/kernel/debug/kprobes/
blacklist  enabled    list
[root@seventh ~]# cat /sys/kernel/debug/kprobes/list
ffffffff8248b350  k  security_locked_down+0x0    [FTRACE]
ffffffff8248b350  r  security_locked_down+0x0    [FTRACE]
[root@seventh ~]#

[root@seventh ~]# cat /etc/fedora-release
Fedora release 33 (Thirty Three)
[root@seventh ~]# gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.1 20201016 (Red Hat 10.2.1-6) (GCC)
[root@seventh ~]# rpm -q elfutils
elfutils-0.182-1.fc33.x86_64
[root@seventh ~]#

- Arnaldo

Re: [BUG] perf probe can't remove probes

[Masami Hiramatsu]

Hi Arnaldo,

On Wed, 25 Nov 2020 14:27:55 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> 
> Masami, have you stumbled on this already?
> 
> [root@seventh ~]# perf probe security_locked_down%return 'ret=$retval'
> Added new event:
>   probe:security_locked_down__return (on security_locked_down%return with ret=$retval)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe:security_locked_down__return -aR sleep 1
> 
> [root@seventh ~]# perf probe security_locked_down what
> Added new event:
>   probe:security_locked_down (on security_locked_down with what)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe:security_locked_down -aR sleep 1
> 
> [root@seventh ~]#
> 
> 
> [root@seventh ~]# uname -r
> 5.10.0-rc3.bpfsign+
> [root@seventh ~]# perf probe -l
>   probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what)
>   probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
> [root@seventh ~]# perf probe -D '*:*'
> Semantic error :There is non-digit char in line number.
> 
>  Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]
>     or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
>     or: perf probe [<options>] --del '[GROUP:]EVENT' ...
>     or: perf probe --list [GROUP:]EVENT ...
>     or: perf probe [<options>] --line 'LINEDESC'
>     or: perf probe [<options>] --vars 'PROBEPOINT'
>     or: perf probe [<options>] --funcs
> 
>     -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
>                           Show trace event definition of given traceevent for k/uprobe_events.

As you can see, "-D" is showing definition. Not delete. (*)
Delete is "-d" or "--del".

(*) this option is for different version of kernel, remote-machine
and boot-time tracing.

> [root@seventh ~]# perf probe probe:security_locked_down
> Semantic error :There is non-digit char in line number.
>   Error: Command Parse Error.
> [root@seventh ~]# perf probe probe:security_locked_down__return
> Semantic error :There is non-digit char in line number.
>   Error: Command Parse Error.

Since you don't pass any option, both are for adding new probe event.

What happen if you run

$ perf probe -d "*:*"

?

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [BUG] perf probe can't remove probes

[Arnaldo Carvalho de Melo]

Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:
> Hi Arnaldo,
> 
> On Wed, 25 Nov 2020 14:27:55 -0300
> Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> > 
> > Masami, have you stumbled on this already?
> > 
> > [root@seventh ~]# perf probe security_locked_down%return 'ret=$retval'
> > Added new event:
> >   probe:security_locked_down__return (on security_locked_down%return with ret=$retval)
> > 
> > You can now use it in all perf tools, such as:
> > 
> > 	perf record -e probe:security_locked_down__return -aR sleep 1
> > 
> > [root@seventh ~]# perf probe security_locked_down what
> > Added new event:
> >   probe:security_locked_down (on security_locked_down with what)
> > 
> > You can now use it in all perf tools, such as:
> > 
> > 	perf record -e probe:security_locked_down -aR sleep 1
> > 
> > [root@seventh ~]#
> > 
> > 
> > [root@seventh ~]# uname -r
> > 5.10.0-rc3.bpfsign+
> > [root@seventh ~]# perf probe -l
> >   probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what)
> >   probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
> > [root@seventh ~]# perf probe -D '*:*'
> > Semantic error :There is non-digit char in line number.
> > 
> >  Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]
> >     or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
> >     or: perf probe [<options>] --del '[GROUP:]EVENT' ...
> >     or: perf probe --list [GROUP:]EVENT ...
> >     or: perf probe [<options>] --line 'LINEDESC'
> >     or: perf probe [<options>] --vars 'PROBEPOINT'
> >     or: perf probe [<options>] --funcs
> > 
> >     -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
> >                           Show trace event definition of given traceevent for k/uprobe_events.
> 
> As you can see, "-D" is showing definition. Not delete. (*)
> Delete is "-d" or "--del".

Yeah, I was in a hurry and looked at just the first line right after the
command, didn't want to forget reporting it so sent the "bug" report,
d0h, sorry about the noise, using -d or --del works.

But having both -d and -D, in retrospect, wasn't such a good idea :-\

- Arnaldo
 
> (*) this option is for different version of kernel, remote-machine
> and boot-time tracing.
> 
> > [root@seventh ~]# perf probe probe:security_locked_down
> > Semantic error :There is non-digit char in line number.
> >   Error: Command Parse Error.
> > [root@seventh ~]# perf probe probe:security_locked_down__return
> > Semantic error :There is non-digit char in line number.
> >   Error: Command Parse Error.
> 
> Since you don't pass any option, both are for adding new probe event.
> 
> What happen if you run
> 
> $ perf probe -d "*:*"
> 
> ?
> 
> Thank you,
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>

-- 

- Arnaldo

Re: [BUG] perf probe can't remove probes

[Masami Hiramatsu]

On Thu, 26 Nov 2020 14:26:03 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:
> > Hi Arnaldo,
> > 
> > On Wed, 25 Nov 2020 14:27:55 -0300
> > Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 
> > > 
> > > Masami, have you stumbled on this already?
> > > 
> > > [root@seventh ~]# perf probe security_locked_down%return 'ret=$retval'
> > > Added new event:
> > >   probe:security_locked_down__return (on security_locked_down%return with ret=$retval)
> > > 
> > > You can now use it in all perf tools, such as:
> > > 
> > > 	perf record -e probe:security_locked_down__return -aR sleep 1
> > > 
> > > [root@seventh ~]# perf probe security_locked_down what
> > > Added new event:
> > >   probe:security_locked_down (on security_locked_down with what)
> > > 
> > > You can now use it in all perf tools, such as:
> > > 
> > > 	perf record -e probe:security_locked_down -aR sleep 1
> > > 
> > > [root@seventh ~]#
> > > 
> > > 
> > > [root@seventh ~]# uname -r
> > > 5.10.0-rc3.bpfsign+
> > > [root@seventh ~]# perf probe -l
> > >   probe:security_locked_down (on security_locked_down@git/bpf/security/security.c with what)
> > >   probe:security_locked_down__return (on security_locked_down%return@git/bpf/security/security.c with ret)
> > > [root@seventh ~]# perf probe -D '*:*'
> > > Semantic error :There is non-digit char in line number.
> > > 
> > >  Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]
> > >     or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
> > >     or: perf probe [<options>] --del '[GROUP:]EVENT' ...
> > >     or: perf probe --list [GROUP:]EVENT ...
> > >     or: perf probe [<options>] --line 'LINEDESC'
> > >     or: perf probe [<options>] --vars 'PROBEPOINT'
> > >     or: perf probe [<options>] --funcs
> > > 
> > >     -D, --definition <[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT [[NAME=]ARG ...]>
> > >                           Show trace event definition of given traceevent for k/uprobe_events.
> > 
> > As you can see, "-D" is showing definition. Not delete. (*)
> > Delete is "-d" or "--del".
> 
> Yeah, I was in a hurry and looked at just the first line right after the
> command, didn't want to forget reporting it so sent the "bug" report,
> d0h, sorry about the noise, using -d or --del works.
> 
> But having both -d and -D, in retrospect, wasn't such a good idea :-\

Sorry for confusing :(

Hmm, would we better to remove -D and keep only --definition?
But it is already there, I think we should keep this option
for backward compatibility.

Thank you,



-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [BUG] perf probe can't remove probes

[Arnaldo Carvalho de Melo]

Em Wed, Dec 16, 2020 at 09:16:40AM +0900, Masami Hiramatsu escreveu:
> On Thu, 26 Nov 2020 14:26:03 -0300
 Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > Em Thu, Nov 26, 2020 at 09:21:25AM +0900, Masami Hiramatsu escreveu:
> > > On Wed, 25 Nov 2020 14:27:55 -0300 Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > As you can see, "-D" is showing definition. Not delete. (*)
> > > Delete is "-d" or "--del".

> > Yeah, I was in a hurry and looked at just the first line right after the
> > command, didn't want to forget reporting it so sent the "bug" report,
> > d0h, sorry about the noise, using -d or --del works.

> > But having both -d and -D, in retrospect, wasn't such a good idea :-\
> 
> Sorry for confusing :(
 
> Hmm, would we better to remove -D and keep only --definition?
> But it is already there, I think we should keep this option
> for backward compatibility.

Nah, since we have it there, lets take this just as a lesson and in the
future use single letter options just for core functionality, leaving
things like --definition to be just a long option.

We can't change it now :-)

- Arnaldo

perf probe can't remove probes

[Arnaldo Carvalho de Melo]

Hi Masami,

	Any idea why listing the source code doesn't work while simply
adding the probe works?

[root@seventh ~]# perf probe verify_pkcs7_signature
Added new event:
  probe:verify_pkcs7_signature (on verify_pkcs7_signature)

You can now use it in all perf tools, such as:

	perf record -e probe:verify_pkcs7_signature -aR sleep 1

[root@seventh ~]#

[root@seventh ~]# readelf -wi /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux | grep DW_AT_producer -m2
    <1c>   DW_AT_producer    : (indirect string, offset: 0x50): GNU AS 2.35
    <2f>   DW_AT_producer    : (indirect string, offset: 0x2daf): GNU C89 10.2.1 20201016 (Red Hat 10.2.1-6) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu90 -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector-strong -fno-strict-overflow -fstack-check=no -fconserve-stack -fcf-protection=none -fno-stack-protector
[root@seventh ~]#

[root@seventh ~]# perf probe -L verify_pkcs7_signature
Specified source line is not found.
  Error: Failed to show lines.
[root@seventh ~]# grep verify_pkcs7_signature /proc/kallsyms
ffffffff9f2617b0 T verify_pkcs7_signature
ffffffffa047ac30 r __ksymtab_verify_pkcs7_signature
ffffffffa0482e38 r __kstrtabns_verify_pkcs7_signature
ffffffffa0483b3b r __kstrtab_verify_pkcs7_signature
[root@seventh ~]# perf probe -L verify_pkcs7_signature
Specified source line is not found.
  Error: Failed to show lines.
[root@seventh ~]# perf probe -v -L verify_pkcs7_signature
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux
fname: /home/acme/git/bpf/include/linux/verification.h, lineno:38
New line range: 38 to 2147483647
path: (null)
Specified source line is not found.
  Error: Failed to show lines. Reason: No such file or directory (Code: -2)
[root@seventh ~]# readelf -wi /lib/modules/5.10.0-rc3.bpfsign+/build/vmlinux | grep -B2 -A8 verify_pkcs7_signature
 <1><1a362a2>: Abbrev Number: 64 (DW_TAG_subprogram)
    <1a362a3>   DW_AT_external    : 1
    <1a362a3>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <1a362a7>   DW_AT_decl_file   : 52
    <1a362a8>   DW_AT_decl_line   : 38
    <1a362a9>   DW_AT_decl_column : 12
    <1a362aa>   DW_AT_prototyped  : 1
    <1a362aa>   DW_AT_type        : <0x1a3333f>
    <1a362ae>   DW_AT_sibling     : <0x1a36313>
 <2><1a362b2>: Abbrev Number: 65 (DW_TAG_formal_parameter)
    <1a362b3>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data
--
 <1><24fa79d>: Abbrev Number: 83 (DW_TAG_subprogram)
    <24fa79e>   DW_AT_external    : 1
    <24fa79e>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <24fa7a2>   DW_AT_decl_file   : 218
    <24fa7a3>   DW_AT_decl_line   : 38
    <24fa7a4>   DW_AT_decl_column : 12
    <24fa7a5>   DW_AT_prototyped  : 1
    <24fa7a5>   DW_AT_type        : <0x24e1e05>
    <24fa7a9>   DW_AT_sibling     : <0x24fa80e>
 <2><24fa7ad>: Abbrev Number: 41 (DW_TAG_formal_parameter)
    <24fa7ae>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data
--
    <26258b8>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
 <1><26258c2>: Abbrev Number: 116 (DW_TAG_variable)
    <26258c3>   DW_AT_name        : (indirect string, offset: 0x10523f): __UNIQUE_ID___addressable_verify_pkcs7_signature164
    <26258c7>   DW_AT_decl_file   : 1
    <26258c8>   DW_AT_decl_line   : 305
    <26258ca>   DW_AT_decl_column : 1
    <26258cb>   DW_AT_type        : <0x2620ca6>
    <26258cf>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
 <1><26258d9>: Abbrev Number: 117 (DW_TAG_subprogram)
    <26258da>   DW_AT_external    : 1
    <26258da>   DW_AT_name        : (indirect string, offset: 0x1050d9): pkcs7_free_message
--
 <1><2625c2e>: Abbrev Number: 120 (DW_TAG_subprogram)
    <2625c2f>   DW_AT_external    : 1
    <2625c2f>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <2625c33>   DW_AT_decl_file   : 1
    <2625c34>   DW_AT_decl_line   : 282
    <2625c36>   DW_AT_decl_column : 5
    <2625c37>   DW_AT_prototyped  : 1
    <2625c37>   DW_AT_type        : <0x26209e9>
    <2625c3b>   DW_AT_low_pc      : 0xffffffff812617b0
    <2625c43>   DW_AT_high_pc     : 0x73
    <2625c4b>   DW_AT_frame_base  : 1 byte block: 9c 	(DW_OP_call_frame_cfa)
--
 <1><495560e>: Abbrev Number: 68 (DW_TAG_subprogram)
    <495560f>   DW_AT_external    : 1
    <495560f>   DW_AT_name        : (indirect string, offset: 0xc2d22): verify_pkcs7_signature
    <4955613>   DW_AT_decl_file   : 61
    <4955614>   DW_AT_decl_line   : 38
    <4955615>   DW_AT_decl_column : 12
    <4955616>   DW_AT_prototyped  : 1
    <4955616>   DW_AT_type        : <0x49519fd>
    <495561a>   DW_AT_sibling     : <0x495567f>
 <2><495561e>: Abbrev Number: 74 (DW_TAG_formal_parameter)
    <495561f>   DW_AT_name        : (indirect string, offset: 0x19ee3d): data

Re: perf probe can't remove probes

[Arnaldo Carvalho de Melo]

Em Tue, Dec 01, 2020 at 10:25:17AM -0300, Arnaldo Carvalho de Melo escreveu:
> Hi Masami,
 
> 	Any idea why listing the source code doesn't work while simply
> adding the probe works?

<SNIP>
 
> [root@seventh ~]# perf probe -L verify_pkcs7_signature
> Specified source line is not found.
>   Error: Failed to show lines.

Yeah, those fixes you sent, when cherry-picked into this bpf-next/master
based branch, do the trick, so nevermind, you fixed this already, these
fixes were already merged by Linus, bpf-next will rebase at some point,
etc. :-)

- Arnaldo

[acme@five bpf]$ git log --oneline -4
0f3147557ea8143d (HEAD -> bpfsign) perf probe: Change function definition check due to broken DWARF
0e104f490d8f86b4 perf probe: Fix to die_entrypc() returns error correctly
9423e1f089643738 libbpf: Check if the kernel supports signatures before associating them
580616996c498b70 libbpf: Attach signature ELF sections to signed ELF program sections
[acme@five bpf]$

[root@seventh ~]# perf probe -L verify_pkcs7_signature
<verify_pkcs7_signature@/home/acme/git/bpf/certs/system_keyring.c:0>
      0  int verify_pkcs7_signature(const void *data, size_t len,
                                   const void *raw_pkcs7, size_t pkcs7_len,
                                   struct key *trusted_keys,
                                   enum key_being_used_for usage,
                                   int (*view_content)(void *ctx,
                                                       const void *data, size_t len,
                                                       size_t asn1hdrlen),
                                   void *ctx)
         {
      9         struct pkcs7_message *pkcs7;
                int ret;

                pkcs7 = pkcs7_parse_message(raw_pkcs7, pkcs7_len);
     13         if (IS_ERR(pkcs7))
     14                 return PTR_ERR(pkcs7);

     16         ret = verify_pkcs7_message_sig(data, len, pkcs7, trusted_keys, usage,
                                               view_content, ctx);

     19         pkcs7_free_message(pkcs7);
     20         pr_devel("<==%s() = %d\n", __func__, ret);
                return ret;
         }
         EXPORT_SYMBOL_GPL(verify_pkcs7_signature);

[root@seventh ~]#