Re: Fedora 33 and perf probe failures

[Masami Hiramatsu <mhiramat@kernel.org>]

Hi,

On Thu, 19 Nov 2020 09:16:57 +0100
Thomas Richter <tmricht@linux.ibm.com> wrote:

> On 11/19/20 2:07 AM, Masami Hiramatsu wrote:
> > Hi Thomas,
> > 
> > On Wed, 18 Nov 2020 17:37:21 +0100
> > Thomas Richter <tmricht@linux.ibm.com> wrote:
> > 
> >> I debugged the failure of some perf test cases on Fedora 33. The following
> >> test cases fail on s390 and x86_64 using
> > 
> > Thanks for the report!
> > Let me setup Fedora33.
> > 
> >>
> >> [root@f33 yum.repos.d]# uname -a
> >> Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020
> >>                                 x86_64 x86_64 x86_64 GNU/Linux
> >> [root@f33 yum.repos.d]#
> >>
> >> [root@f33 yum.repos.d]# perf test -F 74 77 78
> >> 74: Use vfs_getname probe to get syscall args filenames
> >>                 :Failed to write event: Invalid argument
> >> Error: Failed to add events.
> >> FAILED!
> >> 77: Check open filename arg using perf trace + vfs_getname
> >>                 :Failed to write event: Invalid argument
> >> Error: Failed to add events.
> >> FAILED!
> >> 78: Add vfs_getname probe to get syscall args filenames
> >>                 :Failed to write event: Invalid argument
> >> Error: Failed to add events.
> >> FAILED!
> >> [root@f33 yum.repos.d]#
> > 
> > BTW, do we have vfs_getname in the kernel?
> > 
> 
> No, that is the name of the probe when calling perf probe command,
> as in .../perf/tests/shell/lib/probe_vfs_getname.sh:
> 
> perf probe -q "vfs_getname=getname_flags:...."

Ah, OK.

> The probe is set on kernel function getname_flags()
> in fs/namei.c source file and named vfs_getname inside
> perf/tracing subsystem.

Hmm, this seems debuginfo is not correctly handled.


> >> The root cause is the output of perf probe -L which returns:
> >> [root@f33 yum.repos.d]# perf probe -L getname_flags
> >> Specified source line is not found.
> >> Error: Failed to show lines.
> > 
> > Hmm, could you run it with -vvv option?
> 
> [root@f33 ~]# uname -a
> Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> [root@f33 ~]# cat /etc/fedora-release 
> Fedora release 33 (Thirty Three)
> [root@f33 ~]# perf kallsyms getname_flags
> getname_flags: [kernel] [kernel.kallsyms] 0xffffffffa8325000-0xffffffffa8325060 (0xffffffffa8325000-0xffffffffa8325060)
> [root@f33 ~]# find / -name vmlinux
> /home/richter/linux/arch/x86/boot/compressed/vmlinux
> /home/richter/linux/vmlinux
> /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux
> /sys/kernel/btf/vmlinux
> [root@f33 ~]# nm -g /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux| fgrep getname_flags
> ffffffff81325000 T getname_flags
> [root@f33 ~]# 
> 
> [root@f33 ~]# perf probe -vvv -L getname_flags
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols
> Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
> fname: ./include/linux/fs.h, lineno:2527
> New line range: 2527 to 2147483647
> path: (null)
> Specified source line is not found.

The fname seems wrong, there is a declaration of getname_flags, but not
the definition. Hmm, maybe the dwarf format is different?

>   Error: Failed to show lines. Reason: No such file or directory (Code: -2)
> [root@f33 ~]# 
> 
> > 
> >> [root@f33 yum.repos.d]#
> >>
> >> whereas on my Fedora 32 installation this command shows
> >> [root@t35lp46 perf]# cat /etc/redhat-release
> >> Fedora release 32 (Thirty Two)
> >> [root@t35lp46 perf]#
> >>
> >> [root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10
> >>      68                         return ERR_PTR(-ENOENT);
> >>                         }
> >>                 }
> >>
> >>      72         result->uptr = filename;
> >>      73         result->aname = NULL;
> >>      74         audit_getname(result);
> >>                 return result;
> >> }
> >>
> >> [root@t35lp46 perf]#
> >>
> >> Any ideas how to fix this? Is this failure related to a missing debuginfod
> >> setup, even when I have installed the debuginfo packages for the kernel:
> > 
> > If you can find the vmlinux in debuginfo package, could you try
> > eu-addr2line to find getname_flags?
> >  
> > eu-addr2line -e /somewhere/to/vmlinux getname_flags+0
> 
> [root@f33 ~]# eu-addr2line -e /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux  getname_flags+0
> fs/namei.c:129:1
> [root@f33 ~]# 

OK, so elfutils can decode it.
I'll try to decode dwarf, trace elfutils and find what happens.

Thank you,

> 
> > 
> > Thank you,
> > 
> 
> You are welcome, thanks for looking into this.
> 
> I have added my Fedora32 output for completeness:
> 
> [root@t35lp46 perf]# ./perf probe -vvv -L getname_flags
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/5.10.0-rc4d-perf+/build/vmlinux for symbols
> Open Debuginfo file: /lib/modules/5.10.0-rc4d-perf+/build/vmlinux
> fname: fs/namei.c, lineno:128
> New line range: 128 to 2147483647
> Get 20989 lines from this CU
> path: fs/namei.c
> Search fs/namei.c from debuginfod -> -38
> Failed to find fs/namei.c in debuginfod (8a690c7c154db1283cebb8953128584dd4f94b3d)
> <getname_flags@/root/linux/fs/namei.c:0>
>       0  getname_flags(const char __user *filename, int flags, int *empty)
>          {
>       2         struct filename *result;
>                 char *kname;
>                 int len;
> ......
> 
> root@t35lp46 perf]# eu-addr2line -e ~/linux/vmlinux  getname_flags+0
> fs/namei.c:129:1
> [root@t35lp46 perf]# 
> 
> 
> -- 
> Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
> --
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294


-- 
Masami Hiramatsu <mhiramat@kernel.org>