Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update

[Thomas Richter <tmricht@linux.ibm.com>]

I think I found one issue:

Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
[root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
/usr/lib/debug/lib64/libc-2.33.so.debug
[root@s8360047 ~]#

The file was located in
   /usr/lib/debug/lib64/libc-2.33.so.debug
and hard linked to (or vice versa)
   /usr/lib/debug/usr/lib64/libc-2.33.so.debug

With Fedora 34 the glibc debug file name changed to:
/usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
and
/usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug

This is important. The call chain is

 ...
 try_to_find_probe_trace_events()
 +-> open_debuginfo()
     +-> debuginfo__new(/usr/lib64/libc-2.33.so)
         +-> dso__read_binary_type_filename()

Function dso__read_binary_type_filename() now tries to find the debug file
for /usr/lib64/libc-2.33.so for various distros starting with
FEDORA_DEBUGINFO. Tt builds the file name by prepending
'/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
name change. Now the debug file is not found anymore and various other
distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
file itself is used.

This even worked for Fedora 34 until the glibc 2.33-18 change, because the
/usr/lib/libc-xxx.so file contained the complete symbol table in the
.symtab section. The function debuginfo__find_probe_location() found
the symbol inet_pton. So it did not matter is the debug file was found or not.

With the glib 2.33-18 change most entries in the .symtab section where
moved to the .dynsym section and are not found anymore when the default
/usr/lib/lic-xxx.so is used as fallback. So the issue surfaced.

Second issue:
The symbol 'inet_pton' is defined as WEAK. Even when the correct
debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
the perf probe command fails.
However it works for GLOBAL and LOCAL symbols, so it would be definitly an 
improvement.

I have no clue on dwarf so I put Masami on the addressee list.
This second issue is somehow related to dwarf_getfuncs() and
the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
in the probe_finder.c file. Too far away from my horizon :-)...

On 6/23/21 4:21 PM, Thomas Richter wrote:
> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> [root@m46lp22 perf]# ./perf test 78
> 78: probe libc's inet_pton & backtrace it with ping                 : FAILED!
> [root@m46lp22 perf]#
> 
> It boils down to this command and happens after glibc is update to level 2.33-18.
> 
> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> Probe point 'inet_pton' not found.
>   Error: Failed to add events.
> [root@f34 ~]# rpm -qa | fgrep glibc
> glibc-all-langpacks-2.33-18.fc34.x86_64
> glibc-common-2.33-18.fc34.x86_64
> glibc-langpack-en-2.33-18.fc34.x86_64
> glibc-2.33-18.fc34.x86_64
> glibc-doc-2.33-18.fc34.noarch
> glibc-headers-x86-2.33-18.fc34.noarch
> glibc-devel-2.33-18.fc34.x86_64
> glibc-debugsource-2.33-18.fc34.x86_64
> glibc-debuginfo-2.33-18.fc34.x86_64
> [root@f34 ~]# 
> 
> The symbol inet_pton is now in the .dynsym section of glibc:
> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2419 entries:
>    628: 000000000011ea00   108 FUNC    WEAK   DEFAULT   15 inet_pton@@GLIBC_2.2.5
>   2251: 000000000011e9b0    76 FUNC    GLOBAL DEFAULT   15 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 104 entries:
> [root@f34 ~]#
> 
> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> compared to previous versions.
> 
> Now perf does not find it. In the older version of the library the
> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
> 
> 
> [root@m35lp76 ~]# rpm -qa | fgrep glibc
> glibc-common-2.32-4.fc33.s390x
> glibc-langpack-en-2.32-4.fc33.s390x
> glibc-2.32-4.fc33.s390x
> glibc-headers-s390-2.32-4.fc33.noarch
> glibc-devel-2.32-4.fc33.s390x
> glibc-debuginfo-common-2.32-4.fc33.s390x
> glibc-debuginfo-2.32-4.fc33.s390x
> [root@m35lp76 ~]# 
> 
> 
> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2604 entries:
>    668: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton@@GLIBC_2.2
>   2418: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 28858 entries:
>  20655: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS inet_pton.c
>  20656: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c
>  20657: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end
>  20658: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.hot
>  20659: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.hot
>  20660: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.unlikely
>  20661: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.unlikely
>  20662: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.startup
>  20663: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.startup
>  20664: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.exit
>  20665: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.exit
>  20666: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.start
>  20667: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.end
>  20668: 00000000001440b0   250 FUNC    LOCAL  DEFAULT   13 inet_pton4
>  20669: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.start
>  20670: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.end
>  20671: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.start
>  20672: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.end
>  23591: 00000000001441b0   764 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton_length
>  23871: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __inet_pton
>  24506: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton
>  25831: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI_inet_pton
>  26410: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length
>  27288: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton
> [root@m35lp76 ~]#
> 
> And perf could find the symbol, extract its address and install a probe
> on that address.
> So is this a bug related to the perf tool because it can not handle .dynsym section?
> Or is it releated to glibc's rework?
> 
> Thanks for your help.
> 


-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294