Re: perf test case probe libc fails with latest Fedora34 glibc update

[Thomas Richter <tmricht@linux.ibm.com>]

On 6/25/21 12:34 PM, Jiri Olsa wrote:
> On Wed, Jun 23, 2021 at 04:21:17PM +0200, Thomas Richter wrote:
>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
>> [root@m46lp22 perf]# ./perf test 78
>> 78: probe libc's inet_pton & backtrace it with ping                 : FAILED!
>> [root@m46lp22 perf]#
>>
>> It boils down to this command and happens after glibc is update to level 2.33-18.
>>
>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
>> Probe point 'inet_pton' not found.
>>   Error: Failed to add events.
>> [root@f34 ~]# rpm -qa | fgrep glibc
>> glibc-all-langpacks-2.33-18.fc34.x86_64
>> glibc-common-2.33-18.fc34.x86_64
>> glibc-langpack-en-2.33-18.fc34.x86_64
>> glibc-2.33-18.fc34.x86_64
>> glibc-doc-2.33-18.fc34.noarch
>> glibc-headers-x86-2.33-18.fc34.noarch
>> glibc-devel-2.33-18.fc34.x86_64
>> glibc-debugsource-2.33-18.fc34.x86_64
>> glibc-debuginfo-2.33-18.fc34.x86_64
>> [root@f34 ~]# 
>>
>> The symbol inet_pton is now in the .dynsym section of glibc:
>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
>> Symbol table '.dynsym' contains 2419 entries:
>>    628: 000000000011ea00   108 FUNC    WEAK   DEFAULT   15 inet_pton@@GLIBC_2.2.5
>>   2251: 000000000011e9b0    76 FUNC    GLOBAL DEFAULT   15 __inet_pton_length@@GLIBC_PRIVATE
>> Symbol table '.symtab' contains 104 entries:
>> [root@f34 ~]#
>>
>> The .symtab section does not contain symbol inet_pton. It contains very few symbols
>> compared to previous versions.
>>
>> Now perf does not find it. In the older version of the library the
>> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
>>
>>
>> [root@m35lp76 ~]# rpm -qa | fgrep glibc
>> glibc-common-2.32-4.fc33.s390x
>> glibc-langpack-en-2.32-4.fc33.s390x
>> glibc-2.32-4.fc33.s390x
>> glibc-headers-s390-2.32-4.fc33.noarch
>> glibc-devel-2.32-4.fc33.s390x
>> glibc-debuginfo-common-2.32-4.fc33.s390x
>> glibc-debuginfo-2.32-4.fc33.s390x
>> [root@m35lp76 ~]# 
>>
>>
>> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
>> Symbol table '.dynsym' contains 2604 entries:
>>    668: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton@@GLIBC_2.2
>>   2418: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length@@GLIBC_PRIVATE
>> Symbol table '.symtab' contains 28858 entries:
>>  20655: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS inet_pton.c
>>  20656: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c
>>  20657: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end
>>  20658: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.hot
>>  20659: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.hot
>>  20660: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.unlikely
>>  20661: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.unlikely
>>  20662: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.startup
>>  20663: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.startup
>>  20664: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.exit
>>  20665: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.exit
>>  20666: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.start
>>  20667: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.end
>>  20668: 00000000001440b0   250 FUNC    LOCAL  DEFAULT   13 inet_pton4
>>  20669: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.start
>>  20670: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.end
>>  20671: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.start
>>  20672: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.end
>>  23591: 00000000001441b0   764 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton_length
>>  23871: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __inet_pton
>>  24506: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton
>>  25831: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI_inet_pton
>>  26410: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length
>>  27288: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton
>> [root@m35lp76 ~]#
>>
>> And perf could find the symbol, extract its address and install a probe
>> on that address.
>> So is this a bug related to the perf tool because it can not handle .dynsym section?
>> Or is it releated to glibc's rework?
> 
> hi,
> there's already glibc bug for that:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1975895
> 
> looks like they might restore the old behaviour
> 
> on the other hand, we could consider to read multiple
> symbol tables and merge them together
> 
> I'm not sure there's some hidden issue in such approach,
> otherwise we might be doing that already ;-)
> 
> jirka
> 

Right, Stefan Liebler investigated a similar issue on valgrind (Bug 1965374). He pointed me to the
bug owner...

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294