Re: Fedora 33 and perf probe failures

[Thomas Richter <tmricht@linux.ibm.com>]

On 11/19/20 2:07 AM, Masami Hiramatsu wrote:
> Hi Thomas,
> 
> On Wed, 18 Nov 2020 17:37:21 +0100
> Thomas Richter <tmricht@linux.ibm.com> wrote:
> 
>> I debugged the failure of some perf test cases on Fedora 33. The following
>> test cases fail on s390 and x86_64 using
> 
> Thanks for the report!
> Let me setup Fedora33.
> 
>>
>> [root@f33 yum.repos.d]# uname -a
>> Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020
>>                                 x86_64 x86_64 x86_64 GNU/Linux
>> [root@f33 yum.repos.d]#
>>
>> [root@f33 yum.repos.d]# perf test -F 74 77 78
>> 74: Use vfs_getname probe to get syscall args filenames
>>                 :Failed to write event: Invalid argument
>> Error: Failed to add events.
>> FAILED!
>> 77: Check open filename arg using perf trace + vfs_getname
>>                 :Failed to write event: Invalid argument
>> Error: Failed to add events.
>> FAILED!
>> 78: Add vfs_getname probe to get syscall args filenames
>>                 :Failed to write event: Invalid argument
>> Error: Failed to add events.
>> FAILED!
>> [root@f33 yum.repos.d]#
> 
> BTW, do we have vfs_getname in the kernel?
> 

No, that is the name of the probe when calling perf probe command,
as in .../perf/tests/shell/lib/probe_vfs_getname.sh:

perf probe -q "vfs_getname=getname_flags:...."
The probe is set on kernel function getname_flags()
in fs/namei.c source file and named vfs_getname inside
perf/tracing subsystem.

>>
>> The root cause is the output of perf probe -L which returns:
>> [root@f33 yum.repos.d]# perf probe -L getname_flags
>> Specified source line is not found.
>> Error: Failed to show lines.
> 
> Hmm, could you run it with -vvv option?

[root@f33 ~]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@f33 ~]# cat /etc/fedora-release 
Fedora release 33 (Thirty Three)
[root@f33 ~]# perf kallsyms getname_flags
getname_flags: [kernel] [kernel.kallsyms] 0xffffffffa8325000-0xffffffffa8325060 (0xffffffffa8325000-0xffffffffa8325060)
[root@f33 ~]# find / -name vmlinux
/home/richter/linux/arch/x86/boot/compressed/vmlinux
/home/richter/linux/vmlinux
/usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux
/sys/kernel/btf/vmlinux
[root@f33 ~]# nm -g /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux| fgrep getname_flags
ffffffff81325000 T getname_flags
[root@f33 ~]# 

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
  Error: Failed to show lines. Reason: No such file or directory (Code: -2)
[root@f33 ~]# 

> 
>> [root@f33 yum.repos.d]#
>>
>> whereas on my Fedora 32 installation this command shows
>> [root@t35lp46 perf]# cat /etc/redhat-release
>> Fedora release 32 (Thirty Two)
>> [root@t35lp46 perf]#
>>
>> [root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10
>>      68                         return ERR_PTR(-ENOENT);
>>                         }
>>                 }
>>
>>      72         result->uptr = filename;
>>      73         result->aname = NULL;
>>      74         audit_getname(result);
>>                 return result;
>> }
>>
>> [root@t35lp46 perf]#
>>
>> Any ideas how to fix this? Is this failure related to a missing debuginfod
>> setup, even when I have installed the debuginfo packages for the kernel:
> 
> If you can find the vmlinux in debuginfo package, could you try
> eu-addr2line to find getname_flags?
>  
> eu-addr2line -e /somewhere/to/vmlinux getname_flags+0

[root@f33 ~]# eu-addr2line -e /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux  getname_flags+0
fs/namei.c:129:1
[root@f33 ~]# 

> 
> Thank you,
> 

You are welcome, thanks for looking into this.

I have added my Fedora32 output for completeness:

[root@t35lp46 perf]# ./perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.10.0-rc4d-perf+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.10.0-rc4d-perf+/build/vmlinux
fname: fs/namei.c, lineno:128
New line range: 128 to 2147483647
Get 20989 lines from this CU
path: fs/namei.c
Search fs/namei.c from debuginfod -> -38
Failed to find fs/namei.c in debuginfod (8a690c7c154db1283cebb8953128584dd4f94b3d)
<getname_flags@/root/linux/fs/namei.c:0>
      0  getname_flags(const char __user *filename, int flags, int *empty)
         {
      2         struct filename *result;
                char *kname;
                int len;
......

root@t35lp46 perf]# eu-addr2line -e ~/linux/vmlinux  getname_flags+0
fs/namei.c:129:1
[root@t35lp46 perf]# 


-- 
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294