* perf test case probe libc fails with latest Fedora34 glibc update
@ 2021-06-23 14:21 Thomas Richter
2021-06-25 10:34 ` Jiri Olsa
2021-06-25 10:43 ` Part 2: " Thomas Richter
0 siblings, 2 replies; 11+ messages in thread
From: Thomas Richter @ 2021-06-23 14:21 UTC (permalink / raw)
To: linux-perf-use., Arnaldo Carvalho de Melo; +Cc: Sven Schnelle, Heiko Carstens
I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
[root@m46lp22 perf]# ./perf test 78
78: probe libc's inet_pton & backtrace it with ping : FAILED!
[root@m46lp22 perf]#
It boils down to this command and happens after glibc is update to level 2.33-18.
[root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
Probe point 'inet_pton' not found.
Error: Failed to add events.
[root@f34 ~]# rpm -qa | fgrep glibc
glibc-all-langpacks-2.33-18.fc34.x86_64
glibc-common-2.33-18.fc34.x86_64
glibc-langpack-en-2.33-18.fc34.x86_64
glibc-2.33-18.fc34.x86_64
glibc-doc-2.33-18.fc34.noarch
glibc-headers-x86-2.33-18.fc34.noarch
glibc-devel-2.33-18.fc34.x86_64
glibc-debugsource-2.33-18.fc34.x86_64
glibc-debuginfo-2.33-18.fc34.x86_64
[root@f34 ~]#
The symbol inet_pton is now in the .dynsym section of glibc:
[root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
Symbol table '.dynsym' contains 2419 entries:
628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
Symbol table '.symtab' contains 104 entries:
[root@f34 ~]#
The .symtab section does not contain symbol inet_pton. It contains very few symbols
compared to previous versions.
Now perf does not find it. In the older version of the library the
symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
[root@m35lp76 ~]# rpm -qa | fgrep glibc
glibc-common-2.32-4.fc33.s390x
glibc-langpack-en-2.32-4.fc33.s390x
glibc-2.32-4.fc33.s390x
glibc-headers-s390-2.32-4.fc33.noarch
glibc-devel-2.32-4.fc33.s390x
glibc-debuginfo-common-2.32-4.fc33.s390x
glibc-debuginfo-2.32-4.fc33.s390x
[root@m35lp76 ~]#
readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
Symbol table '.dynsym' contains 2604 entries:
668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE
Symbol table '.symtab' contains 28858 entries:
20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c
20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end
20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot
20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot
20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely
20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely
20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup
20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup
20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit
20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit
20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start
20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end
20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4
20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start
20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end
20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start
20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end
23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length
23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton
24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton
25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton
26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length
27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton
[root@m35lp76 ~]#
And perf could find the symbol, extract its address and install a probe
on that address.
So is this a bug related to the perf tool because it can not handle .dynsym section?
Or is it releated to glibc's rework?
Thanks for your help.
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter
@ 2021-06-25 10:34 ` Jiri Olsa
2021-06-25 10:57 ` Thomas Richter
2021-06-25 10:43 ` Part 2: " Thomas Richter
1 sibling, 1 reply; 11+ messages in thread
From: Jiri Olsa @ 2021-06-25 10:34 UTC (permalink / raw)
To: Thomas Richter
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Namhyung Kim
On Wed, Jun 23, 2021 at 04:21:17PM +0200, Thomas Richter wrote:
> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> [root@m46lp22 perf]# ./perf test 78
> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
> [root@m46lp22 perf]#
>
> It boils down to this command and happens after glibc is update to level 2.33-18.
>
> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> Probe point 'inet_pton' not found.
> Error: Failed to add events.
> [root@f34 ~]# rpm -qa | fgrep glibc
> glibc-all-langpacks-2.33-18.fc34.x86_64
> glibc-common-2.33-18.fc34.x86_64
> glibc-langpack-en-2.33-18.fc34.x86_64
> glibc-2.33-18.fc34.x86_64
> glibc-doc-2.33-18.fc34.noarch
> glibc-headers-x86-2.33-18.fc34.noarch
> glibc-devel-2.33-18.fc34.x86_64
> glibc-debugsource-2.33-18.fc34.x86_64
> glibc-debuginfo-2.33-18.fc34.x86_64
> [root@f34 ~]#
>
> The symbol inet_pton is now in the .dynsym section of glibc:
> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2419 entries:
> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 104 entries:
> [root@f34 ~]#
>
> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> compared to previous versions.
>
> Now perf does not find it. In the older version of the library the
> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
>
>
> [root@m35lp76 ~]# rpm -qa | fgrep glibc
> glibc-common-2.32-4.fc33.s390x
> glibc-langpack-en-2.32-4.fc33.s390x
> glibc-2.32-4.fc33.s390x
> glibc-headers-s390-2.32-4.fc33.noarch
> glibc-devel-2.32-4.fc33.s390x
> glibc-debuginfo-common-2.32-4.fc33.s390x
> glibc-debuginfo-2.32-4.fc33.s390x
> [root@m35lp76 ~]#
>
>
> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2604 entries:
> 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 28858 entries:
> 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c
> 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end
> 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot
> 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot
> 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely
> 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely
> 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup
> 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup
> 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit
> 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit
> 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start
> 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end
> 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start
> 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end
> 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start
> 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end
> 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length
> 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton
> 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton
> 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton
> 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length
> 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton
> [root@m35lp76 ~]#
>
> And perf could find the symbol, extract its address and install a probe
> on that address.
> So is this a bug related to the perf tool because it can not handle .dynsym section?
> Or is it releated to glibc's rework?
hi,
there's already glibc bug for that:
https://bugzilla.redhat.com/show_bug.cgi?id=1975895
looks like they might restore the old behaviour
on the other hand, we could consider to read multiple
symbol tables and merge them together
I'm not sure there's some hidden issue in such approach,
otherwise we might be doing that already ;-)
jirka
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter
2021-06-25 10:34 ` Jiri Olsa
@ 2021-06-25 10:43 ` Thomas Richter
2021-06-29 5:02 ` Masami Hiramatsu
1 sibling, 1 reply; 11+ messages in thread
From: Thomas Richter @ 2021-06-25 10:43 UTC (permalink / raw)
To: linux-perf-use., Arnaldo Carvalho de Melo, Masami Hiramatsu
Cc: Sven Schnelle, Heiko Carstens, Stefan Liebler
I think I found one issue:
Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
[root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
/usr/lib/debug/lib64/libc-2.33.so.debug
[root@s8360047 ~]#
The file was located in
/usr/lib/debug/lib64/libc-2.33.so.debug
and hard linked to (or vice versa)
/usr/lib/debug/usr/lib64/libc-2.33.so.debug
With Fedora 34 the glibc debug file name changed to:
/usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
and
/usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug
This is important. The call chain is
...
try_to_find_probe_trace_events()
+-> open_debuginfo()
+-> debuginfo__new(/usr/lib64/libc-2.33.so)
+-> dso__read_binary_type_filename()
Function dso__read_binary_type_filename() now tries to find the debug file
for /usr/lib64/libc-2.33.so for various distros starting with
FEDORA_DEBUGINFO. Tt builds the file name by prepending
'/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
name change. Now the debug file is not found anymore and various other
distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
file itself is used.
This even worked for Fedora 34 until the glibc 2.33-18 change, because the
/usr/lib/libc-xxx.so file contained the complete symbol table in the
.symtab section. The function debuginfo__find_probe_location() found
the symbol inet_pton. So it did not matter is the debug file was found or not.
With the glib 2.33-18 change most entries in the .symtab section where
moved to the .dynsym section and are not found anymore when the default
/usr/lib/lic-xxx.so is used as fallback. So the issue surfaced.
Second issue:
The symbol 'inet_pton' is defined as WEAK. Even when the correct
debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
the perf probe command fails.
However it works for GLOBAL and LOCAL symbols, so it would be definitly an
improvement.
I have no clue on dwarf so I put Masami on the addressee list.
This second issue is somehow related to dwarf_getfuncs() and
the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
in the probe_finder.c file. Too far away from my horizon :-)...
On 6/23/21 4:21 PM, Thomas Richter wrote:
> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> [root@m46lp22 perf]# ./perf test 78
> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
> [root@m46lp22 perf]#
>
> It boils down to this command and happens after glibc is update to level 2.33-18.
>
> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> Probe point 'inet_pton' not found.
> Error: Failed to add events.
> [root@f34 ~]# rpm -qa | fgrep glibc
> glibc-all-langpacks-2.33-18.fc34.x86_64
> glibc-common-2.33-18.fc34.x86_64
> glibc-langpack-en-2.33-18.fc34.x86_64
> glibc-2.33-18.fc34.x86_64
> glibc-doc-2.33-18.fc34.noarch
> glibc-headers-x86-2.33-18.fc34.noarch
> glibc-devel-2.33-18.fc34.x86_64
> glibc-debugsource-2.33-18.fc34.x86_64
> glibc-debuginfo-2.33-18.fc34.x86_64
> [root@f34 ~]#
>
> The symbol inet_pton is now in the .dynsym section of glibc:
> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2419 entries:
> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 104 entries:
> [root@f34 ~]#
>
> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> compared to previous versions.
>
> Now perf does not find it. In the older version of the library the
> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
>
>
> [root@m35lp76 ~]# rpm -qa | fgrep glibc
> glibc-common-2.32-4.fc33.s390x
> glibc-langpack-en-2.32-4.fc33.s390x
> glibc-2.32-4.fc33.s390x
> glibc-headers-s390-2.32-4.fc33.noarch
> glibc-devel-2.32-4.fc33.s390x
> glibc-debuginfo-common-2.32-4.fc33.s390x
> glibc-debuginfo-2.32-4.fc33.s390x
> [root@m35lp76 ~]#
>
>
> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2604 entries:
> 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 28858 entries:
> 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c
> 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end
> 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot
> 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot
> 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely
> 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely
> 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup
> 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup
> 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit
> 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit
> 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start
> 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end
> 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start
> 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end
> 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start
> 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end
> 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length
> 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton
> 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton
> 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton
> 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length
> 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton
> [root@m35lp76 ~]#
>
> And perf could find the symbol, extract its address and install a probe
> on that address.
> So is this a bug related to the perf tool because it can not handle .dynsym section?
> Or is it releated to glibc's rework?
>
> Thanks for your help.
>
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-25 10:34 ` Jiri Olsa
@ 2021-06-25 10:57 ` Thomas Richter
0 siblings, 0 replies; 11+ messages in thread
From: Thomas Richter @ 2021-06-25 10:57 UTC (permalink / raw)
To: Jiri Olsa
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Namhyung Kim, Stefan Liebler
On 6/25/21 12:34 PM, Jiri Olsa wrote:
> On Wed, Jun 23, 2021 at 04:21:17PM +0200, Thomas Richter wrote:
>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
>> [root@m46lp22 perf]# ./perf test 78
>> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
>> [root@m46lp22 perf]#
>>
>> It boils down to this command and happens after glibc is update to level 2.33-18.
>>
>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
>> Probe point 'inet_pton' not found.
>> Error: Failed to add events.
>> [root@f34 ~]# rpm -qa | fgrep glibc
>> glibc-all-langpacks-2.33-18.fc34.x86_64
>> glibc-common-2.33-18.fc34.x86_64
>> glibc-langpack-en-2.33-18.fc34.x86_64
>> glibc-2.33-18.fc34.x86_64
>> glibc-doc-2.33-18.fc34.noarch
>> glibc-headers-x86-2.33-18.fc34.noarch
>> glibc-devel-2.33-18.fc34.x86_64
>> glibc-debugsource-2.33-18.fc34.x86_64
>> glibc-debuginfo-2.33-18.fc34.x86_64
>> [root@f34 ~]#
>>
>> The symbol inet_pton is now in the .dynsym section of glibc:
>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
>> Symbol table '.dynsym' contains 2419 entries:
>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
>> Symbol table '.symtab' contains 104 entries:
>> [root@f34 ~]#
>>
>> The .symtab section does not contain symbol inet_pton. It contains very few symbols
>> compared to previous versions.
>>
>> Now perf does not find it. In the older version of the library the
>> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
>>
>>
>> [root@m35lp76 ~]# rpm -qa | fgrep glibc
>> glibc-common-2.32-4.fc33.s390x
>> glibc-langpack-en-2.32-4.fc33.s390x
>> glibc-2.32-4.fc33.s390x
>> glibc-headers-s390-2.32-4.fc33.noarch
>> glibc-devel-2.32-4.fc33.s390x
>> glibc-debuginfo-common-2.32-4.fc33.s390x
>> glibc-debuginfo-2.32-4.fc33.s390x
>> [root@m35lp76 ~]#
>>
>>
>> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
>> Symbol table '.dynsym' contains 2604 entries:
>> 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
>> 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE
>> Symbol table '.symtab' contains 28858 entries:
>> 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
>> 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c
>> 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end
>> 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot
>> 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot
>> 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely
>> 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely
>> 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup
>> 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup
>> 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit
>> 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit
>> 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start
>> 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end
>> 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4
>> 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start
>> 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end
>> 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start
>> 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end
>> 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length
>> 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton
>> 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton
>> 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton
>> 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length
>> 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton
>> [root@m35lp76 ~]#
>>
>> And perf could find the symbol, extract its address and install a probe
>> on that address.
>> So is this a bug related to the perf tool because it can not handle .dynsym section?
>> Or is it releated to glibc's rework?
>
> hi,
> there's already glibc bug for that:
> https://bugzilla.redhat.com/show_bug.cgi?id=1975895
>
> looks like they might restore the old behaviour
>
> on the other hand, we could consider to read multiple
> symbol tables and merge them together
>
> I'm not sure there's some hidden issue in such approach,
> otherwise we might be doing that already ;-)
>
> jirka
>
Right, Stefan Liebler investigated a similar issue on valgrind (Bug 1965374). He pointed me to the
bug owner...
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-25 10:43 ` Part 2: " Thomas Richter
@ 2021-06-29 5:02 ` Masami Hiramatsu
2021-06-29 6:42 ` Thomas Richter
2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter
0 siblings, 2 replies; 11+ messages in thread
From: Masami Hiramatsu @ 2021-06-29 5:02 UTC (permalink / raw)
To: Thomas Richter
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
On Fri, 25 Jun 2021 12:43:29 +0200
Thomas Richter <tmricht@linux.ibm.com> wrote:
> I think I found one issue:
>
> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
> /usr/lib/debug/lib64/libc-2.33.so.debug
> [root@s8360047 ~]#
>
> The file was located in
> /usr/lib/debug/lib64/libc-2.33.so.debug
> and hard linked to (or vice versa)
> /usr/lib/debug/usr/lib64/libc-2.33.so.debug
>
> With Fedora 34 the glibc debug file name changed to:
> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
> and
> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug
Oh, the naming scheme has been changed.
>
> This is important. The call chain is
>
> ...
> try_to_find_probe_trace_events()
> +-> open_debuginfo()
> +-> debuginfo__new(/usr/lib64/libc-2.33.so)
> +-> dso__read_binary_type_filename()
>
> Function dso__read_binary_type_filename() now tries to find the debug file
> for /usr/lib64/libc-2.33.so for various distros starting with
> FEDORA_DEBUGINFO. Tt builds the file name by prepending
> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
> name change. Now the debug file is not found anymore and various other
> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
> file itself is used.
Hmm, we might need a generic way to get such debuginfo path among
linux distros... are there any good command?
>
> This even worked for Fedora 34 until the glibc 2.33-18 change, because the
> /usr/lib/libc-xxx.so file contained the complete symbol table in the
> .symtab section. The function debuginfo__find_probe_location() found
> the symbol inet_pton. So it did not matter is the debug file was found or not.
>
> With the glib 2.33-18 change most entries in the .symtab section where
> moved to the .dynsym section and are not found anymore when the default
> /usr/lib/lic-xxx.so is used as fallback. So the issue surfaced.
>
> Second issue:
> The symbol 'inet_pton' is defined as WEAK. Even when the correct
> debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
> the perf probe command fails.
> However it works for GLOBAL and LOCAL symbols, so it would be definitly an
> improvement.
>
> I have no clue on dwarf so I put Masami on the addressee list.
> This second issue is somehow related to dwarf_getfuncs() and
> the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
> in the probe_finder.c file. Too far away from my horizon :-)...
>
> On 6/23/21 4:21 PM, Thomas Richter wrote:
> > I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> > [root@m46lp22 perf]# ./perf test 78
> > 78: probe libc's inet_pton & backtrace it with ping : FAILED!
> > [root@m46lp22 perf]#
> >
> > It boils down to this command and happens after glibc is update to level 2.33-18.
> >
> > [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> > Probe point 'inet_pton' not found.
> > Error: Failed to add events.
Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only
open the library without debuginfo?
> > [root@f34 ~]# rpm -qa | fgrep glibc
> > glibc-all-langpacks-2.33-18.fc34.x86_64
> > glibc-common-2.33-18.fc34.x86_64
> > glibc-langpack-en-2.33-18.fc34.x86_64
> > glibc-2.33-18.fc34.x86_64
> > glibc-doc-2.33-18.fc34.noarch
> > glibc-headers-x86-2.33-18.fc34.noarch
> > glibc-devel-2.33-18.fc34.x86_64
> > glibc-debugsource-2.33-18.fc34.x86_64
> > glibc-debuginfo-2.33-18.fc34.x86_64
> > [root@f34 ~]#
> >
> > The symbol inet_pton is now in the .dynsym section of glibc:
> > [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> > Symbol table '.dynsym' contains 2419 entries:
> > 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
> > 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
> > Symbol table '.symtab' contains 104 entries:
> > [root@f34 ~]#
> >
> > The .symtab section does not contain symbol inet_pton. It contains very few symbols
> > compared to previous versions.
Hmm, if perf probe can not find the debuginfo, it failback to symbol map,
can you add -vvv option to run the perf probe again?
> >
> > Now perf does not find it. In the older version of the library the
> > symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
> >
> >
> > [root@m35lp76 ~]# rpm -qa | fgrep glibc
> > glibc-common-2.32-4.fc33.s390x
> > glibc-langpack-en-2.32-4.fc33.s390x
> > glibc-2.32-4.fc33.s390x
> > glibc-headers-s390-2.32-4.fc33.noarch
> > glibc-devel-2.32-4.fc33.s390x
> > glibc-debuginfo-common-2.32-4.fc33.s390x
> > glibc-debuginfo-2.32-4.fc33.s390x
> > [root@m35lp76 ~]#
> >
> >
> > readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
> > Symbol table '.dynsym' contains 2604 entries:
> > 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> > 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE
> > Symbol table '.symtab' contains 28858 entries:
> > 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> > 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c
> > 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end
> > 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot
> > 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot
> > 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely
> > 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely
> > 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup
> > 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup
> > 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit
> > 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit
> > 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start
> > 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end
> > 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4
> > 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start
> > 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end
> > 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start
> > 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end
> > 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length
> > 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton
> > 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton
> > 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton
> > 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length
> > 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton
> > [root@m35lp76 ~]#
> >
> > And perf could find the symbol, extract its address and install a probe
> > on that address.
> > So is this a bug related to the perf tool because it can not handle .dynsym section?
> > Or is it releated to glibc's rework?
> >
> > Thanks for your help.
> >
Thank you,
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-29 5:02 ` Masami Hiramatsu
@ 2021-06-29 6:42 ` Thomas Richter
2021-06-29 14:06 ` Masami Hiramatsu
2021-06-30 16:02 ` Masami Hiramatsu
2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter
1 sibling, 2 replies; 11+ messages in thread
From: Thomas Richter @ 2021-06-29 6:42 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
On 6/29/21 7:02 AM, Masami Hiramatsu wrote:
> On Fri, 25 Jun 2021 12:43:29 +0200
> Thomas Richter <tmricht@linux.ibm.com> wrote:
>
>> I think I found one issue:
>>
>> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
>> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
>> /usr/lib/debug/lib64/libc-2.33.so.debug
>> [root@s8360047 ~]#
>>
>> The file was located in
>> /usr/lib/debug/lib64/libc-2.33.so.debug
>> and hard linked to (or vice versa)
>> /usr/lib/debug/usr/lib64/libc-2.33.so.debug
>>
>> With Fedora 34 the glibc debug file name changed to:
>> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
>> and
>> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug
>
> Oh, the naming scheme has been changed.
>
>>
>> This is important. The call chain is
>>
>> ...
>> try_to_find_probe_trace_events()
>> +-> open_debuginfo()
>> +-> debuginfo__new(/usr/lib64/libc-2.33.so)
>> +-> dso__read_binary_type_filename()
>>
>> Function dso__read_binary_type_filename() now tries to find the debug file
>> for /usr/lib64/libc-2.33.so for various distros starting with
>> FEDORA_DEBUGINFO. Tt builds the file name by prepending
>> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
>> name change. Now the debug file is not found anymore and various other
>> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
>> file itself is used.
>
> Hmm, we might need a generic way to get such debuginfo path among
> linux distros... are there any good command?
IMHO The only option that makes sense is to use the buildid:
The s390 compiler team recommended this to me.
[root@t35lp46 perf]# file /usr/lib64/libc-2.33.so
/usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped
[root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/
total 7912
-rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug
-rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf
-rw-r--r-- 1 root root 4754 Jun 23 10:23 probes
[root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton'
21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
[root@t35lp46 perf]#
The bad thing is that only the file part xx/yy/debug seems to be agreed upon
between distros. There are multiple/different locations containing .build-id directories.
On the other hand, perf is connected to the kernel, so we could 'convince'
the distro's to use their distro specific locations when they build perf.
[....]
>>
>> Second issue:
>> The symbol 'inet_pton' is defined as WEAK. Even when the correct
>> debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
>> the perf probe command fails.
>> However it works for GLOBAL and LOCAL symbols, so it would be definitly an
>> improvement.
>>
>> I have no clue on dwarf so I put Masami on the addressee list.
>> This second issue is somehow related to dwarf_getfuncs() and
>> the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
>> in the probe_finder.c file. Too far away from my horizon :-)...
>>
>> On 6/23/21 4:21 PM, Thomas Richter wrote:
>>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
>>> [root@m46lp22 perf]# ./perf test 78
>>> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
>>> [root@m46lp22 perf]#
>>>
>>> It boils down to this command and happens after glibc is update to level 2.33-18.
>>>
>>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
>>> Probe point 'inet_pton' not found.
>>> Error: Failed to add events.
>
> Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only
> open the library without debuginfo?
# nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton
00000000001234c0 W inet_pton@@GLIBC_2.2
0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE
# readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton
673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
# nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols
# readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...]
24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton
25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton
26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton
27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length
27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
# readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton
673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
#
>
>>> [root@f34 ~]# rpm -qa | fgrep glibc
>>> glibc-all-langpacks-2.33-18.fc34.x86_64
>>> glibc-common-2.33-18.fc34.x86_64
>>> glibc-langpack-en-2.33-18.fc34.x86_64
>>> glibc-2.33-18.fc34.x86_64
>>> glibc-doc-2.33-18.fc34.noarch
>>> glibc-headers-x86-2.33-18.fc34.noarch
>>> glibc-devel-2.33-18.fc34.x86_64
>>> glibc-debugsource-2.33-18.fc34.x86_64
>>> glibc-debuginfo-2.33-18.fc34.x86_64
>>> [root@f34 ~]#
>>>
>>> The symbol inet_pton is now in the .dynsym section of glibc:
>>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
>>> Symbol table '.dynsym' contains 2419 entries:
>>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
>>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
>>> Symbol table '.symtab' contains 104 entries:
>>> [root@f34 ~]#
>>>
>>> The .symtab section does not contain symbol inet_pton. It contains very few symbols
>>> compared to previous versions.
>
> Hmm, if perf probe can not find the debuginfo, it failback to symbol map,
> can you add -vvv option to run the perf probe again?
>
# ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton
probe-definition(0): inet_pton
symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib64/libc-2.33.so
Try to find probe point from debuginfo.
try_to_find_probe_trace_events ntevs 0
try_to_find_probe_trace_events ret -2
try_to_find_probe_trace_events ntevs2 0
Probe point 'inet_pton' not found.
Error: Failed to add events. Reason: No such file or directory (Code: -2)
#
[....]
>
> Thank you,
>
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update, more info
2021-06-29 5:02 ` Masami Hiramatsu
2021-06-29 6:42 ` Thomas Richter
@ 2021-06-29 7:21 ` Thomas Richter
2021-06-29 14:01 ` Masami Hiramatsu
1 sibling, 1 reply; 11+ messages in thread
From: Thomas Richter @ 2021-06-29 7:21 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
On 6/29/21 7:02 AM, Masami Hiramatsu wrote:
> On Fri, 25 Jun 2021 12:43:29 +0200
> Thomas Richter <tmricht@linux.ibm.com> wrote:
>
Masami,
here is more info on this issue:
# nm /usr/lib64/libc-2.33.so
00000000000a7f30 t __GI_memchr
00000000000a3730 t __GI_memcmp
00000000000a37e0 t __GI_memcpy
00000000000a39a0 t __GI_memmove
00000000000a37d0 t __GI_mempcpy
00000000000a8000 t __GI___rawmemchr
00000000000a6070 t __GI_stpcpy
000000000009d190 t __GI___strcasecmp_l
00000000000a67a0 t __GI_strcat
00000000000a6ec0 t __GI_strchr
00000000000a6b70 t __GI_strcmp
00000000000a5f70 t __GI_strcpy
00000000000a7c70 t __GI_strcspn
00000000000a5ad0 t __GI_strlen
000000000009d200 t __GI___strncasecmp_l
00000000000a6ce0 t __GI_strncmp
00000000000a62a0 t __GI_strncpy
00000000000a5d30 t __GI_strnlen
00000000000a73e0 t __GI_strrchr
00000000000b8e80 t __GI_wmemchr
0000000000111f60 T __memcpy_chk
0000000000111f80 T __memmove_chk
0000000000112040 T __stpcpy_chk
0000000000112100 T __strcpy_chk
#
These are the entries in the .symtab section and can be used by perf:
# ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a __strcpy_chk
# ./perf probe -l
probe_libc:__strcpy_chk (on __strcpy_chk@debug/strcpy_chk.c in /usr/lib64/libc-2.33.so)
[root@t35lp46 perf]#
Hope this helps
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update, more info
2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter
@ 2021-06-29 14:01 ` Masami Hiramatsu
0 siblings, 0 replies; 11+ messages in thread
From: Masami Hiramatsu @ 2021-06-29 14:01 UTC (permalink / raw)
To: Thomas Richter
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
On Tue, 29 Jun 2021 09:21:19 +0200
Thomas Richter <tmricht@linux.ibm.com> wrote:
> On 6/29/21 7:02 AM, Masami Hiramatsu wrote:
> > On Fri, 25 Jun 2021 12:43:29 +0200
> > Thomas Richter <tmricht@linux.ibm.com> wrote:
> >
>
> Masami,
>
> here is more info on this issue:
>
> # nm /usr/lib64/libc-2.33.so
> 00000000000a7f30 t __GI_memchr
> 00000000000a3730 t __GI_memcmp
> 00000000000a37e0 t __GI_memcpy
> 00000000000a39a0 t __GI_memmove
> 00000000000a37d0 t __GI_mempcpy
> 00000000000a8000 t __GI___rawmemchr
> 00000000000a6070 t __GI_stpcpy
> 000000000009d190 t __GI___strcasecmp_l
> 00000000000a67a0 t __GI_strcat
> 00000000000a6ec0 t __GI_strchr
> 00000000000a6b70 t __GI_strcmp
> 00000000000a5f70 t __GI_strcpy
> 00000000000a7c70 t __GI_strcspn
> 00000000000a5ad0 t __GI_strlen
> 000000000009d200 t __GI___strncasecmp_l
> 00000000000a6ce0 t __GI_strncmp
> 00000000000a62a0 t __GI_strncpy
> 00000000000a5d30 t __GI_strnlen
> 00000000000a73e0 t __GI_strrchr
> 00000000000b8e80 t __GI_wmemchr
> 0000000000111f60 T __memcpy_chk
> 0000000000111f80 T __memmove_chk
> 0000000000112040 T __stpcpy_chk
> 0000000000112100 T __strcpy_chk
> #
>
> These are the entries in the .symtab section and can be used by perf:
> # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a __strcpy_chk
Are there any output of the perf probe? It may report that the
event is found by map (symbol) or debuginfo.
> # ./perf probe -l
> probe_libc:__strcpy_chk (on __strcpy_chk@debug/strcpy_chk.c in /usr/lib64/libc-2.33.so)
> [root@t35lp46 perf]#
Thanks! So at least the symbols in .symtab works.
>
> Hope this helps
> --
> Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> --
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-29 6:42 ` Thomas Richter
@ 2021-06-29 14:06 ` Masami Hiramatsu
2021-06-30 16:02 ` Masami Hiramatsu
1 sibling, 0 replies; 11+ messages in thread
From: Masami Hiramatsu @ 2021-06-29 14:06 UTC (permalink / raw)
To: Thomas Richter
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
On Tue, 29 Jun 2021 08:42:15 +0200
Thomas Richter <tmricht@linux.ibm.com> wrote:
> On 6/29/21 7:02 AM, Masami Hiramatsu wrote:
> > On Fri, 25 Jun 2021 12:43:29 +0200
> > Thomas Richter <tmricht@linux.ibm.com> wrote:
> >
> >> I think I found one issue:
> >>
> >> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
> >> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
> >> /usr/lib/debug/lib64/libc-2.33.so.debug
> >> [root@s8360047 ~]#
> >>
> >> The file was located in
> >> /usr/lib/debug/lib64/libc-2.33.so.debug
> >> and hard linked to (or vice versa)
> >> /usr/lib/debug/usr/lib64/libc-2.33.so.debug
> >>
> >> With Fedora 34 the glibc debug file name changed to:
> >> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
> >> and
> >> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug
> >
> > Oh, the naming scheme has been changed.
> >
> >>
> >> This is important. The call chain is
> >>
> >> ...
> >> try_to_find_probe_trace_events()
> >> +-> open_debuginfo()
> >> +-> debuginfo__new(/usr/lib64/libc-2.33.so)
> >> +-> dso__read_binary_type_filename()
> >>
> >> Function dso__read_binary_type_filename() now tries to find the debug file
> >> for /usr/lib64/libc-2.33.so for various distros starting with
> >> FEDORA_DEBUGINFO. Tt builds the file name by prepending
> >> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
> >> name change. Now the debug file is not found anymore and various other
> >> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
> >> file itself is used.
> >
> > Hmm, we might need a generic way to get such debuginfo path among
> > linux distros... are there any good command?
>
> IMHO The only option that makes sense is to use the buildid:
> The s390 compiler team recommended this to me.
>
> [root@t35lp46 perf]# file /usr/lib64/libc-2.33.so
> /usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped
> [root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/
> total 7912
> -rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug
> -rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf
> -rw-r--r-- 1 root root 4754 Jun 23 10:23 probes
> [root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton'
> 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
> 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
> [root@t35lp46 perf]#
This debug file is copied by perf, isn't it?
> The bad thing is that only the file part xx/yy/debug seems to be agreed upon
> between distros. There are multiple/different locations containing .build-id directories.
>
> On the other hand, perf is connected to the kernel, so we could 'convince'
> the distro's to use their distro specific locations when they build perf.
Hmm, or they can add /etc/perf.conf, to provide their customized debuginfo path
with a template pattern.
>
> [....]
>
> >>
> >> Second issue:
> >> The symbol 'inet_pton' is defined as WEAK. Even when the correct
> >> debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
> >> the perf probe command fails.
> >> However it works for GLOBAL and LOCAL symbols, so it would be definitly an
> >> improvement.
> >>
> >> I have no clue on dwarf so I put Masami on the addressee list.
> >> This second issue is somehow related to dwarf_getfuncs() and
> >> the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
> >> in the probe_finder.c file. Too far away from my horizon :-)...
> >>
> >> On 6/23/21 4:21 PM, Thomas Richter wrote:
> >>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> >>> [root@m46lp22 perf]# ./perf test 78
> >>> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
> >>> [root@m46lp22 perf]#
> >>>
> >>> It boils down to this command and happens after glibc is update to level 2.33-18.
> >>>
> >>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> >>> Probe point 'inet_pton' not found.
> >>> Error: Failed to add events.
> >
> > Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only
> > open the library without debuginfo?
>
> # nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton
> 00000000001234c0 W inet_pton@@GLIBC_2.2
> 0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE
> # readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton
> 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> # nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
> nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols
> # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
> 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
> 24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...]
> 24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton
> 25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton
> 26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton
> 27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length
> 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
> # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton
> 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> #
>
> >
> >>> [root@f34 ~]# rpm -qa | fgrep glibc
> >>> glibc-all-langpacks-2.33-18.fc34.x86_64
> >>> glibc-common-2.33-18.fc34.x86_64
> >>> glibc-langpack-en-2.33-18.fc34.x86_64
> >>> glibc-2.33-18.fc34.x86_64
> >>> glibc-doc-2.33-18.fc34.noarch
> >>> glibc-headers-x86-2.33-18.fc34.noarch
> >>> glibc-devel-2.33-18.fc34.x86_64
> >>> glibc-debugsource-2.33-18.fc34.x86_64
> >>> glibc-debuginfo-2.33-18.fc34.x86_64
> >>> [root@f34 ~]#
> >>>
> >>> The symbol inet_pton is now in the .dynsym section of glibc:
> >>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> >>> Symbol table '.dynsym' contains 2419 entries:
> >>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
> >>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
> >>> Symbol table '.symtab' contains 104 entries:
> >>> [root@f34 ~]#
> >>>
> >>> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> >>> compared to previous versions.
> >
> > Hmm, if perf probe can not find the debuginfo, it failback to symbol map,
> > can you add -vvv option to run the perf probe again?
> >
>
> # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton
Ah, OK. This helps me a lot.
> probe-definition(0): inet_pton
> symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null)
> Open Debuginfo file: /usr/lib64/libc-2.33.so
> Try to find probe point from debuginfo.
Hmm, strange. Does the /usr/lib64/libc-2.33.so have .debuginfo??
(or, did you copy it?)
> try_to_find_probe_trace_events ntevs 0
> try_to_find_probe_trace_events ret -2
> try_to_find_probe_trace_events ntevs2 0
> Probe point 'inet_pton' not found.
> Error: Failed to add events. Reason: No such file or directory (Code: -2)
I guess the libc binary may have a partial debuginfo and it has no
full DIE tree. Thus the try_to_find_probe_trace_events() failed.
I'll try to check on the Fedora34.
Thank you,
> #
> [....]
> >
> > Thank you,
> >
>
>
> --
> Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> --
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-29 6:42 ` Thomas Richter
2021-06-29 14:06 ` Masami Hiramatsu
@ 2021-06-30 16:02 ` Masami Hiramatsu
2021-07-02 0:41 ` Masami Hiramatsu
1 sibling, 1 reply; 11+ messages in thread
From: Masami Hiramatsu @ 2021-06-30 16:02 UTC (permalink / raw)
To: Thomas Richter
Cc: linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
Hi Thomas,
OK, I finally found 2 root causes for this issue.
1. debuginfo__new() forgot to call dso__load(). Thus, the dso is
just initialized, but doesn't load the build-id. Then the build-id
based debuginfo file is not searched. It must use map instead of dso.
2. Not sure why, but /usr/lib64/libc-2.33.so is shown as "not stripped"
binary. Then the elfutils is fooled by the information and open it
as a debuginfo file. I think probe-finder.c also need to check the
given file has DIE tree (debuginfo) or not by itself.
Thank you,
On Tue, 29 Jun 2021 08:42:15 +0200
Thomas Richter <tmricht@linux.ibm.com> wrote:
> On 6/29/21 7:02 AM, Masami Hiramatsu wrote:
> > On Fri, 25 Jun 2021 12:43:29 +0200
> > Thomas Richter <tmricht@linux.ibm.com> wrote:
> >
> >> I think I found one issue:
> >>
> >> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
> >> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
> >> /usr/lib/debug/lib64/libc-2.33.so.debug
> >> [root@s8360047 ~]#
> >>
> >> The file was located in
> >> /usr/lib/debug/lib64/libc-2.33.so.debug
> >> and hard linked to (or vice versa)
> >> /usr/lib/debug/usr/lib64/libc-2.33.so.debug
> >>
> >> With Fedora 34 the glibc debug file name changed to:
> >> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
> >> and
> >> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug
> >
> > Oh, the naming scheme has been changed.
> >
> >>
> >> This is important. The call chain is
> >>
> >> ...
> >> try_to_find_probe_trace_events()
> >> +-> open_debuginfo()
> >> +-> debuginfo__new(/usr/lib64/libc-2.33.so)
> >> +-> dso__read_binary_type_filename()
> >>
> >> Function dso__read_binary_type_filename() now tries to find the debug file
> >> for /usr/lib64/libc-2.33.so for various distros starting with
> >> FEDORA_DEBUGINFO. Tt builds the file name by prepending
> >> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
> >> name change. Now the debug file is not found anymore and various other
> >> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
> >> file itself is used.
> >
> > Hmm, we might need a generic way to get such debuginfo path among
> > linux distros... are there any good command?
>
> IMHO The only option that makes sense is to use the buildid:
> The s390 compiler team recommended this to me.
>
> [root@t35lp46 perf]# file /usr/lib64/libc-2.33.so
> /usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped
> [root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/
> total 7912
> -rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug
> -rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf
> -rw-r--r-- 1 root root 4754 Jun 23 10:23 probes
> [root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton'
> 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
> 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
> [root@t35lp46 perf]#
>
> The bad thing is that only the file part xx/yy/debug seems to be agreed upon
> between distros. There are multiple/different locations containing .build-id directories.
>
> On the other hand, perf is connected to the kernel, so we could 'convince'
> the distro's to use their distro specific locations when they build perf.
>
> [....]
>
> >>
> >> Second issue:
> >> The symbol 'inet_pton' is defined as WEAK. Even when the correct
> >> debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
> >> the perf probe command fails.
> >> However it works for GLOBAL and LOCAL symbols, so it would be definitly an
> >> improvement.
> >>
> >> I have no clue on dwarf so I put Masami on the addressee list.
> >> This second issue is somehow related to dwarf_getfuncs() and
> >> the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
> >> in the probe_finder.c file. Too far away from my horizon :-)...
> >>
> >> On 6/23/21 4:21 PM, Thomas Richter wrote:
> >>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> >>> [root@m46lp22 perf]# ./perf test 78
> >>> 78: probe libc's inet_pton & backtrace it with ping : FAILED!
> >>> [root@m46lp22 perf]#
> >>>
> >>> It boils down to this command and happens after glibc is update to level 2.33-18.
> >>>
> >>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> >>> Probe point 'inet_pton' not found.
> >>> Error: Failed to add events.
> >
> > Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only
> > open the library without debuginfo?
>
> # nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton
> 00000000001234c0 W inet_pton@@GLIBC_2.2
> 0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE
> # readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton
> 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> # nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
> nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols
> # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton
> 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c
> 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4
> 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6
> 24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...]
> 24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton
> 25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton
> 26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton
> 27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length
> 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton
> # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton
> 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2
> #
>
> >
> >>> [root@f34 ~]# rpm -qa | fgrep glibc
> >>> glibc-all-langpacks-2.33-18.fc34.x86_64
> >>> glibc-common-2.33-18.fc34.x86_64
> >>> glibc-langpack-en-2.33-18.fc34.x86_64
> >>> glibc-2.33-18.fc34.x86_64
> >>> glibc-doc-2.33-18.fc34.noarch
> >>> glibc-headers-x86-2.33-18.fc34.noarch
> >>> glibc-devel-2.33-18.fc34.x86_64
> >>> glibc-debugsource-2.33-18.fc34.x86_64
> >>> glibc-debuginfo-2.33-18.fc34.x86_64
> >>> [root@f34 ~]#
> >>>
> >>> The symbol inet_pton is now in the .dynsym section of glibc:
> >>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> >>> Symbol table '.dynsym' contains 2419 entries:
> >>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5
> >>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE
> >>> Symbol table '.symtab' contains 104 entries:
> >>> [root@f34 ~]#
> >>>
> >>> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> >>> compared to previous versions.
> >
> > Hmm, if perf probe can not find the debuginfo, it failback to symbol map,
> > can you add -vvv option to run the perf probe again?
> >
>
> # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton
> probe-definition(0): inet_pton
> symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null)
> symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null)
> Open Debuginfo file: /usr/lib64/libc-2.33.so
> Try to find probe point from debuginfo.
> try_to_find_probe_trace_events ntevs 0
> try_to_find_probe_trace_events ret -2
> try_to_find_probe_trace_events ntevs2 0
> Probe point 'inet_pton' not found.
> Error: Failed to add events. Reason: No such file or directory (Code: -2)
> #
> [....]
> >
> > Thank you,
> >
>
>
> --
> Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> --
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
2021-06-30 16:02 ` Masami Hiramatsu
@ 2021-07-02 0:41 ` Masami Hiramatsu
0 siblings, 0 replies; 11+ messages in thread
From: Masami Hiramatsu @ 2021-07-02 0:41 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Thomas Richter, linux-perf-use.,
Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens,
Stefan Liebler
Hi,
On Thu, 1 Jul 2021 01:02:08 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:
> Hi Thomas,
>
> OK, I finally found 2 root causes for this issue.
>
> 1. debuginfo__new() forgot to call dso__load(). Thus, the dso is
> just initialized, but doesn't load the build-id. Then the build-id
> based debuginfo file is not searched. It must use map instead of dso.
I made a fix for this issue using dso__set_build_id() because
using dso__load() which loads symbols just for build-id seems sledgehammer.
So now I can see the perf probe opens build-id based debuginfo correctly.
However, this will not fix your issue because;
> 2. Not sure why, but /usr/lib64/libc-2.33.so is shown as "not stripped"
> binary. Then the elfutils is fooled by the information and open it
> as a debuginfo file. I think probe-finder.c also need to check the
> given file has DIE tree (debuginfo) or not by itself.
this part is not correct. perf-probe is using elfutils, which has
its own debuginfo search algorithm. Thus with or without the above
fix, perf-probe finally opens the correct debuginfo.
(from this reason, I can not show what has been fixed by the command
output in commit message without debug print.)
And with decoding the debuginfo, I finally understand what was wrong.
The problem is that the "inet_pton" is a dynamic symbol, which does *NOT*
exist in the debuginfo. There is "__inet_pton" instead of "inet_pton".
-----
/* Like __inet_pton_length, but use strlen (SRC) as the length of
SRC. */
int
__inet_pton (int af, const char *src, void *dst)
{
return __inet_pton_length (af, src, strlen (src), dst);
}
libc_hidden_def (__inet_pton)
weak_alias (__inet_pton, inet_pton)
libc_hidden_weak (inet_pton)
-----
Thus, if you pass the __inet_pton to perf-probe, it works.
-----
$ ./perf probe -x /lib64/libc.so.6 -L __inet_pton
<__inet_pton@/usr/src/debug/glibc-2.33-18.fc34.aarch64/resolv/inet_pton.c:0>
0 __inet_pton (int af, const char *src, void *dst)
{
return __inet_pton_length (af, src, strlen (src), dst);
}
-----
And -F (--functions) doesn't work yet.
-----
./perf probe -x /usr/lib64/libc-2.33.so -F
@plt
@plt
calloc@plt
free@plt
malloc@plt
memalign@plt
realloc@plt
-----
I think we should remove @plt, and decode ".dynsym" in addition to ".symtab"
by map object for this issue at first. Then we may be able to get the address
of "inet_pton" from map.
Thank you,
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-07-02 0:41 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter
2021-06-25 10:34 ` Jiri Olsa
2021-06-25 10:57 ` Thomas Richter
2021-06-25 10:43 ` Part 2: " Thomas Richter
2021-06-29 5:02 ` Masami Hiramatsu
2021-06-29 6:42 ` Thomas Richter
2021-06-29 14:06 ` Masami Hiramatsu
2021-06-30 16:02 ` Masami Hiramatsu
2021-07-02 0:41 ` Masami Hiramatsu
2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter
2021-06-29 14:01 ` Masami Hiramatsu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).