perf probe issues

perf probe issues

[Arnaldo Carvalho de Melo]

Hi Masami,

	While investigating a problem in the annotation code reported by
Linus, with "kaslr" on the kernel command line and this in .config:

[root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config
CONFIG_RANDOMIZE_BASE=y
# CONFIG_DEBUG_INFO is not set
[root@jouet ~]#

[root@jouet ~]# grep -w vfs_write /proc/kallsyms 
ffffffffac2404c0 T vfs_write
[root@jouet ~]# perf probe vfs_write
vfs_write is out of .text, skip it.
  Error: Failed to add events.
[root@jouet ~]# perf probe -v vfs_write
probe-definition(0): vfs_write 
symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
vfs_write is out of .text, skip it.
  Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@jouet ~]# 

Can you try to test with this config to investigate this problem?

We need to improve the error messages, for instance:

"Could not open debuginfo. Try to use symbols."

I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it
doesn't manage to find a vmlinux file, it will try to use kallsyms as a
fallback, is that what is happening here?

- Arnaldo

Re: perf probe issues

[Masami Hiramatsu]

On Tue, 14 Jun 2016 14:05:04 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Hi Masami,
> 
> 	While investigating a problem in the annotation code reported by
> Linus, with "kaslr" on the kernel command line and this in .config:
> 
> [root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config
> CONFIG_RANDOMIZE_BASE=y
> # CONFIG_DEBUG_INFO is not set
> [root@jouet ~]#
> 
> [root@jouet ~]# grep -w vfs_write /proc/kallsyms 
> ffffffffac2404c0 T vfs_write
> [root@jouet ~]# perf probe vfs_write
> vfs_write is out of .text, skip it.
>   Error: Failed to add events.
> [root@jouet ~]# perf probe -v vfs_write
> probe-definition(0): vfs_write 
> symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
> Could not open debuginfo. Try to use symbols.
> vfs_write is out of .text, skip it.
>   Error: Failed to add events. Reason: No such file or directory (Code: -2)
> [root@jouet ~]# 
> 
> Can you try to test with this config to investigate this problem?

Yes, I guess it missed to get start_text because of KASLR...

> 
> We need to improve the error messages, for instance:
> 
> "Could not open debuginfo. Try to use symbols."
> 
> I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it
> doesn't manage to find a vmlinux file, it will try to use kallsyms as a
> fallback, is that what is happening here?

Let me check that the perf symbol/map object uses kallsyms for kernel.
Anyway, if the target is not the kernel, do we change the message
as "trying to use symbols"?


> 
> - Arnaldo


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: perf probe issues

[Arnaldo Carvalho de Melo]

Em Wed, Jun 15, 2016 at 11:02:01AM +0900, Masami Hiramatsu escreveu:
> On Tue, 14 Jun 2016 14:05:04 -0300
> Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 	While investigating a problem in the annotation code reported by
> > Linus, with "kaslr" on the kernel command line and this in .config:

> > [root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config
> > CONFIG_RANDOMIZE_BASE=y
> > # CONFIG_DEBUG_INFO is not set
> > [root@jouet ~]#

> > [root@jouet ~]# grep -w vfs_write /proc/kallsyms 
> > ffffffffac2404c0 T vfs_write
> > [root@jouet ~]# perf probe vfs_write
> > vfs_write is out of .text, skip it.
> >   Error: Failed to add events.
> > [root@jouet ~]# perf probe -v vfs_write
> > probe-definition(0): vfs_write 
> > symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null)
> > 0 arguments
> > Looking at the vmlinux_path (8 entries long)
> > Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
> > Could not open debuginfo. Try to use symbols.
> > vfs_write is out of .text, skip it.
> >   Error: Failed to add events. Reason: No such file or directory (Code: -2)
> > [root@jouet ~]# 

> > Can you try to test with this config to investigate this problem?
 
> Yes, I guess it missed to get start_text because of KASLR...
 
> > We need to improve the error messages, for instance:

> > "Could not open debuginfo. Try to use symbols."

> > I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it
> > doesn't manage to find a vmlinux file, it will try to use kallsyms as a
> > fallback, is that what is happening here?
 
> Let me check that the perf symbol/map object uses kallsyms for kernel.
> Anyway, if the target is not the kernel, do we change the message
> as "trying to use symbols"?

Well, it is _always_ trying to use "symbols" as in "information to map
an address to a symbol", but this information can come from DWARF,
kallsyms, ELF symbol tables, JIT /tmp/ maps, etc :-)

So, if it is going to use kallsyms:

"Trying to use kallsyms"

ELF symtol tables:

"Trying to use the ELF symbol table"

etc.

- Arnaldo

Re: perf probe issues

[Masami Hiramatsu]

On Wed, 15 Jun 2016 14:43:38 -0300
Arnaldo Carvalho de Melo <acme@kernel.org> wrote:

> Em Wed, Jun 15, 2016 at 11:02:01AM +0900, Masami Hiramatsu escreveu:
> > On Tue, 14 Jun 2016 14:05:04 -0300
> > Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > > 	While investigating a problem in the annotation code reported by
> > > Linus, with "kaslr" on the kernel command line and this in .config:
> 
> > > [root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config
> > > CONFIG_RANDOMIZE_BASE=y
> > > # CONFIG_DEBUG_INFO is not set
> > > [root@jouet ~]#

Hmm, could you give me more info? I tried same thing (on Ubuntu 16.04
kernel(4.4 based) which support kaslr by default), but perf probe
succeeded to put an event on vfs_write (I ensured by <debugfs>/kprobes/list).
What would you get if you grep _etext in /proc/kallsyms?
(Ah, perf probe must change error message if it failed to find _etext
 from kallsyms...)


> 
> > > [root@jouet ~]# grep -w vfs_write /proc/kallsyms 
> > > ffffffffac2404c0 T vfs_write
> > > [root@jouet ~]# perf probe vfs_write
> > > vfs_write is out of .text, skip it.
> > >   Error: Failed to add events.
> > > [root@jouet ~]# perf probe -v vfs_write
> > > probe-definition(0): vfs_write 
> > > symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null)
> > > 0 arguments
> > > Looking at the vmlinux_path (8 entries long)
> > > Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
> > > Could not open debuginfo. Try to use symbols.
> > > vfs_write is out of .text, skip it.
> > >   Error: Failed to add events. Reason: No such file or directory (Code: -2)
> > > [root@jouet ~]# 
> 
> > > Can you try to test with this config to investigate this problem?
>  
> > Yes, I guess it missed to get start_text because of KASLR...
>  
> > > We need to improve the error messages, for instance:
> 
> > > "Could not open debuginfo. Try to use symbols."
> 
> > > I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it
> > > doesn't manage to find a vmlinux file, it will try to use kallsyms as a
> > > fallback, is that what is happening here?
>  
> > Let me check that the perf symbol/map object uses kallsyms for kernel.
> > Anyway, if the target is not the kernel, do we change the message
> > as "trying to use symbols"?
> 
> Well, it is _always_ trying to use "symbols" as in "information to map
> an address to a symbol", but this information can come from DWARF,
> kallsyms, ELF symbol tables, JIT /tmp/ maps, etc :-)
> 
> So, if it is going to use kallsyms:
> 
> "Trying to use kallsyms"
> 
> ELF symtol tables:
> 
> "Trying to use the ELF symbol table"

OK, I'll do that.

> 
> etc.
> 
> - Arnaldo


-- 
Masami Hiramatsu <mhiramat@kernel.org>