* perf test case probe libc fails with latest Fedora34 glibc update @ 2021-06-23 14:21 Thomas Richter 2021-06-25 10:34 ` Jiri Olsa 2021-06-25 10:43 ` Part 2: " Thomas Richter 0 siblings, 2 replies; 11+ messages in thread From: Thomas Richter @ 2021-06-23 14:21 UTC (permalink / raw) To: linux-perf-use., Arnaldo Carvalho de Melo; +Cc: Sven Schnelle, Heiko Carstens I just updated Fedora34 to the latest level and discovered that perf test 78 fails: [root@m46lp22 perf]# ./perf test 78 78: probe libc's inet_pton & backtrace it with ping : FAILED! [root@m46lp22 perf]# It boils down to this command and happens after glibc is update to level 2.33-18. [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton Probe point 'inet_pton' not found. Error: Failed to add events. [root@f34 ~]# rpm -qa | fgrep glibc glibc-all-langpacks-2.33-18.fc34.x86_64 glibc-common-2.33-18.fc34.x86_64 glibc-langpack-en-2.33-18.fc34.x86_64 glibc-2.33-18.fc34.x86_64 glibc-doc-2.33-18.fc34.noarch glibc-headers-x86-2.33-18.fc34.noarch glibc-devel-2.33-18.fc34.x86_64 glibc-debugsource-2.33-18.fc34.x86_64 glibc-debuginfo-2.33-18.fc34.x86_64 [root@f34 ~]# The symbol inet_pton is now in the .dynsym section of glibc: [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' Symbol table '.dynsym' contains 2419 entries: 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE Symbol table '.symtab' contains 104 entries: [root@f34 ~]# The .symtab section does not contain symbol inet_pton. It contains very few symbols compared to previous versions. Now perf does not find it. In the older version of the library the symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4: [root@m35lp76 ~]# rpm -qa | fgrep glibc glibc-common-2.32-4.fc33.s390x glibc-langpack-en-2.32-4.fc33.s390x glibc-2.32-4.fc33.s390x glibc-headers-s390-2.32-4.fc33.noarch glibc-devel-2.32-4.fc33.s390x glibc-debuginfo-common-2.32-4.fc33.s390x glibc-debuginfo-2.32-4.fc33.s390x [root@m35lp76 ~]# readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)' Symbol table '.dynsym' contains 2604 entries: 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE Symbol table '.symtab' contains 28858 entries: 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton [root@m35lp76 ~]# And perf could find the symbol, extract its address and install a probe on that address. So is this a bug related to the perf tool because it can not handle .dynsym section? Or is it releated to glibc's rework? Thanks for your help. -- Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter @ 2021-06-25 10:34 ` Jiri Olsa 2021-06-25 10:57 ` Thomas Richter 2021-06-25 10:43 ` Part 2: " Thomas Richter 1 sibling, 1 reply; 11+ messages in thread From: Jiri Olsa @ 2021-06-25 10:34 UTC (permalink / raw) To: Thomas Richter Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Namhyung Kim On Wed, Jun 23, 2021 at 04:21:17PM +0200, Thomas Richter wrote: > I just updated Fedora34 to the latest level and discovered that perf test 78 fails: > [root@m46lp22 perf]# ./perf test 78 > 78: probe libc's inet_pton & backtrace it with ping : FAILED! > [root@m46lp22 perf]# > > It boils down to this command and happens after glibc is update to level 2.33-18. > > [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton > Probe point 'inet_pton' not found. > Error: Failed to add events. > [root@f34 ~]# rpm -qa | fgrep glibc > glibc-all-langpacks-2.33-18.fc34.x86_64 > glibc-common-2.33-18.fc34.x86_64 > glibc-langpack-en-2.33-18.fc34.x86_64 > glibc-2.33-18.fc34.x86_64 > glibc-doc-2.33-18.fc34.noarch > glibc-headers-x86-2.33-18.fc34.noarch > glibc-devel-2.33-18.fc34.x86_64 > glibc-debugsource-2.33-18.fc34.x86_64 > glibc-debuginfo-2.33-18.fc34.x86_64 > [root@f34 ~]# > > The symbol inet_pton is now in the .dynsym section of glibc: > [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' > Symbol table '.dynsym' contains 2419 entries: > 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 > 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE > Symbol table '.symtab' contains 104 entries: > [root@f34 ~]# > > The .symtab section does not contain symbol inet_pton. It contains very few symbols > compared to previous versions. > > Now perf does not find it. In the older version of the library the > symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4: > > > [root@m35lp76 ~]# rpm -qa | fgrep glibc > glibc-common-2.32-4.fc33.s390x > glibc-langpack-en-2.32-4.fc33.s390x > glibc-2.32-4.fc33.s390x > glibc-headers-s390-2.32-4.fc33.noarch > glibc-devel-2.32-4.fc33.s390x > glibc-debuginfo-common-2.32-4.fc33.s390x > glibc-debuginfo-2.32-4.fc33.s390x > [root@m35lp76 ~]# > > > readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)' > Symbol table '.dynsym' contains 2604 entries: > 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE > Symbol table '.symtab' contains 28858 entries: > 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c > 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end > 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot > 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot > 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely > 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely > 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup > 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup > 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit > 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit > 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start > 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end > 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start > 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end > 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start > 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end > 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length > 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton > 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton > 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton > 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length > 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton > [root@m35lp76 ~]# > > And perf could find the symbol, extract its address and install a probe > on that address. > So is this a bug related to the perf tool because it can not handle .dynsym section? > Or is it releated to glibc's rework? hi, there's already glibc bug for that: https://bugzilla.redhat.com/show_bug.cgi?id=1975895 looks like they might restore the old behaviour on the other hand, we could consider to read multiple symbol tables and merge them together I'm not sure there's some hidden issue in such approach, otherwise we might be doing that already ;-) jirka ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-25 10:34 ` Jiri Olsa @ 2021-06-25 10:57 ` Thomas Richter 0 siblings, 0 replies; 11+ messages in thread From: Thomas Richter @ 2021-06-25 10:57 UTC (permalink / raw) To: Jiri Olsa Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Namhyung Kim, Stefan Liebler On 6/25/21 12:34 PM, Jiri Olsa wrote: > On Wed, Jun 23, 2021 at 04:21:17PM +0200, Thomas Richter wrote: >> I just updated Fedora34 to the latest level and discovered that perf test 78 fails: >> [root@m46lp22 perf]# ./perf test 78 >> 78: probe libc's inet_pton & backtrace it with ping : FAILED! >> [root@m46lp22 perf]# >> >> It boils down to this command and happens after glibc is update to level 2.33-18. >> >> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton >> Probe point 'inet_pton' not found. >> Error: Failed to add events. >> [root@f34 ~]# rpm -qa | fgrep glibc >> glibc-all-langpacks-2.33-18.fc34.x86_64 >> glibc-common-2.33-18.fc34.x86_64 >> glibc-langpack-en-2.33-18.fc34.x86_64 >> glibc-2.33-18.fc34.x86_64 >> glibc-doc-2.33-18.fc34.noarch >> glibc-headers-x86-2.33-18.fc34.noarch >> glibc-devel-2.33-18.fc34.x86_64 >> glibc-debugsource-2.33-18.fc34.x86_64 >> glibc-debuginfo-2.33-18.fc34.x86_64 >> [root@f34 ~]# >> >> The symbol inet_pton is now in the .dynsym section of glibc: >> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' >> Symbol table '.dynsym' contains 2419 entries: >> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 >> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE >> Symbol table '.symtab' contains 104 entries: >> [root@f34 ~]# >> >> The .symtab section does not contain symbol inet_pton. It contains very few symbols >> compared to previous versions. >> >> Now perf does not find it. In the older version of the library the >> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4: >> >> >> [root@m35lp76 ~]# rpm -qa | fgrep glibc >> glibc-common-2.32-4.fc33.s390x >> glibc-langpack-en-2.32-4.fc33.s390x >> glibc-2.32-4.fc33.s390x >> glibc-headers-s390-2.32-4.fc33.noarch >> glibc-devel-2.32-4.fc33.s390x >> glibc-debuginfo-common-2.32-4.fc33.s390x >> glibc-debuginfo-2.32-4.fc33.s390x >> [root@m35lp76 ~]# >> >> >> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)' >> Symbol table '.dynsym' contains 2604 entries: >> 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 >> 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE >> Symbol table '.symtab' contains 28858 entries: >> 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c >> 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c >> 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end >> 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot >> 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot >> 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely >> 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely >> 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup >> 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup >> 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit >> 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit >> 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start >> 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end >> 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4 >> 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start >> 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end >> 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start >> 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end >> 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length >> 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton >> 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton >> 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton >> 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length >> 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton >> [root@m35lp76 ~]# >> >> And perf could find the symbol, extract its address and install a probe >> on that address. >> So is this a bug related to the perf tool because it can not handle .dynsym section? >> Or is it releated to glibc's rework? > > hi, > there's already glibc bug for that: > https://bugzilla.redhat.com/show_bug.cgi?id=1975895 > > looks like they might restore the old behaviour > > on the other hand, we could consider to read multiple > symbol tables and merge them together > > I'm not sure there's some hidden issue in such approach, > otherwise we might be doing that already ;-) > > jirka > Right, Stefan Liebler investigated a similar issue on valgrind (Bug 1965374). He pointed me to the bug owner... -- Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter 2021-06-25 10:34 ` Jiri Olsa @ 2021-06-25 10:43 ` Thomas Richter 2021-06-29 5:02 ` Masami Hiramatsu 1 sibling, 1 reply; 11+ messages in thread From: Thomas Richter @ 2021-06-25 10:43 UTC (permalink / raw) To: linux-perf-use., Arnaldo Carvalho de Melo, Masami Hiramatsu Cc: Sven Schnelle, Heiko Carstens, Stefan Liebler I think I found one issue: Fedora 33 named the debug files for glibc 'libc-2.33.so.debug' [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so /usr/lib/debug/lib64/libc-2.33.so.debug [root@s8360047 ~]# The file was located in /usr/lib/debug/lib64/libc-2.33.so.debug and hard linked to (or vice versa) /usr/lib/debug/usr/lib64/libc-2.33.so.debug With Fedora 34 the glibc debug file name changed to: /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug and /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug This is important. The call chain is ... try_to_find_probe_trace_events() +-> open_debuginfo() +-> debuginfo__new(/usr/lib64/libc-2.33.so) +-> dso__read_binary_type_filename() Function dso__read_binary_type_filename() now tries to find the debug file for /usr/lib64/libc-2.33.so for various distros starting with FEDORA_DEBUGINFO. Tt builds the file name by prepending '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34 name change. Now the debug file is not found anymore and various other distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so file itself is used. This even worked for Fedora 34 until the glibc 2.33-18 change, because the /usr/lib/libc-xxx.so file contained the complete symbol table in the .symtab section. The function debuginfo__find_probe_location() found the symbol inet_pton. So it did not matter is the debug file was found or not. With the glib 2.33-18 change most entries in the .symtab section where moved to the .dynsym section and are not found anymore when the default /usr/lib/lic-xxx.so is used as fallback. So the issue surfaced. Second issue: The symbol 'inet_pton' is defined as WEAK. Even when the correct debuginfo file is loaded for libc in function dso__read_binary_type_filename(), the perf probe command fails. However it works for GLOBAL and LOCAL symbols, so it would be definitly an improvement. I have no clue on dwarf so I put Masami on the addressee list. This second issue is somehow related to dwarf_getfuncs() and the probe_point_search_cb() call back function invoked by dwarf_getfuncs() in the probe_finder.c file. Too far away from my horizon :-)... On 6/23/21 4:21 PM, Thomas Richter wrote: > I just updated Fedora34 to the latest level and discovered that perf test 78 fails: > [root@m46lp22 perf]# ./perf test 78 > 78: probe libc's inet_pton & backtrace it with ping : FAILED! > [root@m46lp22 perf]# > > It boils down to this command and happens after glibc is update to level 2.33-18. > > [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton > Probe point 'inet_pton' not found. > Error: Failed to add events. > [root@f34 ~]# rpm -qa | fgrep glibc > glibc-all-langpacks-2.33-18.fc34.x86_64 > glibc-common-2.33-18.fc34.x86_64 > glibc-langpack-en-2.33-18.fc34.x86_64 > glibc-2.33-18.fc34.x86_64 > glibc-doc-2.33-18.fc34.noarch > glibc-headers-x86-2.33-18.fc34.noarch > glibc-devel-2.33-18.fc34.x86_64 > glibc-debugsource-2.33-18.fc34.x86_64 > glibc-debuginfo-2.33-18.fc34.x86_64 > [root@f34 ~]# > > The symbol inet_pton is now in the .dynsym section of glibc: > [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' > Symbol table '.dynsym' contains 2419 entries: > 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 > 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE > Symbol table '.symtab' contains 104 entries: > [root@f34 ~]# > > The .symtab section does not contain symbol inet_pton. It contains very few symbols > compared to previous versions. > > Now perf does not find it. In the older version of the library the > symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4: > > > [root@m35lp76 ~]# rpm -qa | fgrep glibc > glibc-common-2.32-4.fc33.s390x > glibc-langpack-en-2.32-4.fc33.s390x > glibc-2.32-4.fc33.s390x > glibc-headers-s390-2.32-4.fc33.noarch > glibc-devel-2.32-4.fc33.s390x > glibc-debuginfo-common-2.32-4.fc33.s390x > glibc-debuginfo-2.32-4.fc33.s390x > [root@m35lp76 ~]# > > > readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)' > Symbol table '.dynsym' contains 2604 entries: > 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE > Symbol table '.symtab' contains 28858 entries: > 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c > 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end > 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot > 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot > 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely > 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely > 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup > 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup > 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit > 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit > 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start > 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end > 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start > 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end > 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start > 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end > 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length > 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton > 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton > 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton > 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length > 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton > [root@m35lp76 ~]# > > And perf could find the symbol, extract its address and install a probe > on that address. > So is this a bug related to the perf tool because it can not handle .dynsym section? > Or is it releated to glibc's rework? > > Thanks for your help. > -- Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-25 10:43 ` Part 2: " Thomas Richter @ 2021-06-29 5:02 ` Masami Hiramatsu 2021-06-29 6:42 ` Thomas Richter 2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter 0 siblings, 2 replies; 11+ messages in thread From: Masami Hiramatsu @ 2021-06-29 5:02 UTC (permalink / raw) To: Thomas Richter Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler On Fri, 25 Jun 2021 12:43:29 +0200 Thomas Richter <tmricht@linux.ibm.com> wrote: > I think I found one issue: > > Fedora 33 named the debug files for glibc 'libc-2.33.so.debug' > [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so > /usr/lib/debug/lib64/libc-2.33.so.debug > [root@s8360047 ~]# > > The file was located in > /usr/lib/debug/lib64/libc-2.33.so.debug > and hard linked to (or vice versa) > /usr/lib/debug/usr/lib64/libc-2.33.so.debug > > With Fedora 34 the glibc debug file name changed to: > /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug > and > /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug Oh, the naming scheme has been changed. > > This is important. The call chain is > > ... > try_to_find_probe_trace_events() > +-> open_debuginfo() > +-> debuginfo__new(/usr/lib64/libc-2.33.so) > +-> dso__read_binary_type_filename() > > Function dso__read_binary_type_filename() now tries to find the debug file > for /usr/lib64/libc-2.33.so for various distros starting with > FEDORA_DEBUGINFO. Tt builds the file name by prepending > '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34 > name change. Now the debug file is not found anymore and various other > distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so > file itself is used. Hmm, we might need a generic way to get such debuginfo path among linux distros... are there any good command? > > This even worked for Fedora 34 until the glibc 2.33-18 change, because the > /usr/lib/libc-xxx.so file contained the complete symbol table in the > .symtab section. The function debuginfo__find_probe_location() found > the symbol inet_pton. So it did not matter is the debug file was found or not. > > With the glib 2.33-18 change most entries in the .symtab section where > moved to the .dynsym section and are not found anymore when the default > /usr/lib/lic-xxx.so is used as fallback. So the issue surfaced. > > Second issue: > The symbol 'inet_pton' is defined as WEAK. Even when the correct > debuginfo file is loaded for libc in function dso__read_binary_type_filename(), > the perf probe command fails. > However it works for GLOBAL and LOCAL symbols, so it would be definitly an > improvement. > > I have no clue on dwarf so I put Masami on the addressee list. > This second issue is somehow related to dwarf_getfuncs() and > the probe_point_search_cb() call back function invoked by dwarf_getfuncs() > in the probe_finder.c file. Too far away from my horizon :-)... > > On 6/23/21 4:21 PM, Thomas Richter wrote: > > I just updated Fedora34 to the latest level and discovered that perf test 78 fails: > > [root@m46lp22 perf]# ./perf test 78 > > 78: probe libc's inet_pton & backtrace it with ping : FAILED! > > [root@m46lp22 perf]# > > > > It boils down to this command and happens after glibc is update to level 2.33-18. > > > > [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton > > Probe point 'inet_pton' not found. > > Error: Failed to add events. Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only open the library without debuginfo? > > [root@f34 ~]# rpm -qa | fgrep glibc > > glibc-all-langpacks-2.33-18.fc34.x86_64 > > glibc-common-2.33-18.fc34.x86_64 > > glibc-langpack-en-2.33-18.fc34.x86_64 > > glibc-2.33-18.fc34.x86_64 > > glibc-doc-2.33-18.fc34.noarch > > glibc-headers-x86-2.33-18.fc34.noarch > > glibc-devel-2.33-18.fc34.x86_64 > > glibc-debugsource-2.33-18.fc34.x86_64 > > glibc-debuginfo-2.33-18.fc34.x86_64 > > [root@f34 ~]# > > > > The symbol inet_pton is now in the .dynsym section of glibc: > > [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' > > Symbol table '.dynsym' contains 2419 entries: > > 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 > > 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE > > Symbol table '.symtab' contains 104 entries: > > [root@f34 ~]# > > > > The .symtab section does not contain symbol inet_pton. It contains very few symbols > > compared to previous versions. Hmm, if perf probe can not find the debuginfo, it failback to symbol map, can you add -vvv option to run the perf probe again? > > > > Now perf does not find it. In the older version of the library the > > symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4: > > > > > > [root@m35lp76 ~]# rpm -qa | fgrep glibc > > glibc-common-2.32-4.fc33.s390x > > glibc-langpack-en-2.32-4.fc33.s390x > > glibc-2.32-4.fc33.s390x > > glibc-headers-s390-2.32-4.fc33.noarch > > glibc-devel-2.32-4.fc33.s390x > > glibc-debuginfo-common-2.32-4.fc33.s390x > > glibc-debuginfo-2.32-4.fc33.s390x > > [root@m35lp76 ~]# > > > > > > readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)' > > Symbol table '.dynsym' contains 2604 entries: > > 668: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > > 2418: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length@@GLIBC_PRIVATE > > Symbol table '.symtab' contains 28858 entries: > > 20655: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > > 20656: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c > > 20657: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end > > 20658: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.hot > > 20659: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.hot > > 20660: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.unlikely > > 20661: 000000000002b938 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.unlikely > > 20662: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.startup > > 20663: 000000000002ba70 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.startup > > 20664: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c.exit > > 20665: 000000000002b968 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton.c_end.exit > > 20666: 00000000001440b0 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.start > > 20667: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin_inet_pton4.end > > 20668: 00000000001440b0 250 FUNC LOCAL DEFAULT 13 inet_pton4 > > 20669: 00000000001441aa 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.start > > 20670: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton_length.end > > 20671: 00000000001444ac 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.start > > 20672: 00000000001447c4 0 NOTYPE LOCAL HIDDEN 13 .annobin___GI___inet_pton.end > > 23591: 00000000001441b0 764 FUNC LOCAL DEFAULT 13 __GI___inet_pton_length > > 23871: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __inet_pton > > 24506: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI___inet_pton > > 25831: 00000000001444b0 788 FUNC LOCAL DEFAULT 13 __GI_inet_pton > > 26410: 00000000001441b0 764 FUNC GLOBAL DEFAULT 13 __inet_pton_length > > 27288: 00000000001444b0 788 FUNC WEAK DEFAULT 13 inet_pton > > [root@m35lp76 ~]# > > > > And perf could find the symbol, extract its address and install a probe > > on that address. > > So is this a bug related to the perf tool because it can not handle .dynsym section? > > Or is it releated to glibc's rework? > > > > Thanks for your help. > > Thank you, -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-29 5:02 ` Masami Hiramatsu @ 2021-06-29 6:42 ` Thomas Richter 2021-06-29 14:06 ` Masami Hiramatsu 2021-06-30 16:02 ` Masami Hiramatsu 2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter 1 sibling, 2 replies; 11+ messages in thread From: Thomas Richter @ 2021-06-29 6:42 UTC (permalink / raw) To: Masami Hiramatsu Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler On 6/29/21 7:02 AM, Masami Hiramatsu wrote: > On Fri, 25 Jun 2021 12:43:29 +0200 > Thomas Richter <tmricht@linux.ibm.com> wrote: > >> I think I found one issue: >> >> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug' >> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so >> /usr/lib/debug/lib64/libc-2.33.so.debug >> [root@s8360047 ~]# >> >> The file was located in >> /usr/lib/debug/lib64/libc-2.33.so.debug >> and hard linked to (or vice versa) >> /usr/lib/debug/usr/lib64/libc-2.33.so.debug >> >> With Fedora 34 the glibc debug file name changed to: >> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug >> and >> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug > > Oh, the naming scheme has been changed. > >> >> This is important. The call chain is >> >> ... >> try_to_find_probe_trace_events() >> +-> open_debuginfo() >> +-> debuginfo__new(/usr/lib64/libc-2.33.so) >> +-> dso__read_binary_type_filename() >> >> Function dso__read_binary_type_filename() now tries to find the debug file >> for /usr/lib64/libc-2.33.so for various distros starting with >> FEDORA_DEBUGINFO. Tt builds the file name by prepending >> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34 >> name change. Now the debug file is not found anymore and various other >> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so >> file itself is used. > > Hmm, we might need a generic way to get such debuginfo path among > linux distros... are there any good command? IMHO The only option that makes sense is to use the buildid: The s390 compiler team recommended this to me. [root@t35lp46 perf]# file /usr/lib64/libc-2.33.so /usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped [root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/ total 7912 -rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug -rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf -rw-r--r-- 1 root root 4754 Jun 23 10:23 probes [root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton' 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton [root@t35lp46 perf]# The bad thing is that only the file part xx/yy/debug seems to be agreed upon between distros. There are multiple/different locations containing .build-id directories. On the other hand, perf is connected to the kernel, so we could 'convince' the distro's to use their distro specific locations when they build perf. [....] >> >> Second issue: >> The symbol 'inet_pton' is defined as WEAK. Even when the correct >> debuginfo file is loaded for libc in function dso__read_binary_type_filename(), >> the perf probe command fails. >> However it works for GLOBAL and LOCAL symbols, so it would be definitly an >> improvement. >> >> I have no clue on dwarf so I put Masami on the addressee list. >> This second issue is somehow related to dwarf_getfuncs() and >> the probe_point_search_cb() call back function invoked by dwarf_getfuncs() >> in the probe_finder.c file. Too far away from my horizon :-)... >> >> On 6/23/21 4:21 PM, Thomas Richter wrote: >>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails: >>> [root@m46lp22 perf]# ./perf test 78 >>> 78: probe libc's inet_pton & backtrace it with ping : FAILED! >>> [root@m46lp22 perf]# >>> >>> It boils down to this command and happens after glibc is update to level 2.33-18. >>> >>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton >>> Probe point 'inet_pton' not found. >>> Error: Failed to add events. > > Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only > open the library without debuginfo? # nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton 00000000001234c0 W inet_pton@@GLIBC_2.2 0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE # readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 # nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...] 24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton 25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton 26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton 27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 # > >>> [root@f34 ~]# rpm -qa | fgrep glibc >>> glibc-all-langpacks-2.33-18.fc34.x86_64 >>> glibc-common-2.33-18.fc34.x86_64 >>> glibc-langpack-en-2.33-18.fc34.x86_64 >>> glibc-2.33-18.fc34.x86_64 >>> glibc-doc-2.33-18.fc34.noarch >>> glibc-headers-x86-2.33-18.fc34.noarch >>> glibc-devel-2.33-18.fc34.x86_64 >>> glibc-debugsource-2.33-18.fc34.x86_64 >>> glibc-debuginfo-2.33-18.fc34.x86_64 >>> [root@f34 ~]# >>> >>> The symbol inet_pton is now in the .dynsym section of glibc: >>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' >>> Symbol table '.dynsym' contains 2419 entries: >>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 >>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE >>> Symbol table '.symtab' contains 104 entries: >>> [root@f34 ~]# >>> >>> The .symtab section does not contain symbol inet_pton. It contains very few symbols >>> compared to previous versions. > > Hmm, if perf probe can not find the debuginfo, it failback to symbol map, > can you add -vvv option to run the perf probe again? > # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton probe-definition(0): inet_pton symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null) symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null) symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null) symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null) symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null) symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null) Open Debuginfo file: /usr/lib64/libc-2.33.so Try to find probe point from debuginfo. try_to_find_probe_trace_events ntevs 0 try_to_find_probe_trace_events ret -2 try_to_find_probe_trace_events ntevs2 0 Probe point 'inet_pton' not found. Error: Failed to add events. Reason: No such file or directory (Code: -2) # [....] > > Thank you, > -- Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-29 6:42 ` Thomas Richter @ 2021-06-29 14:06 ` Masami Hiramatsu 2021-06-30 16:02 ` Masami Hiramatsu 1 sibling, 0 replies; 11+ messages in thread From: Masami Hiramatsu @ 2021-06-29 14:06 UTC (permalink / raw) To: Thomas Richter Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler On Tue, 29 Jun 2021 08:42:15 +0200 Thomas Richter <tmricht@linux.ibm.com> wrote: > On 6/29/21 7:02 AM, Masami Hiramatsu wrote: > > On Fri, 25 Jun 2021 12:43:29 +0200 > > Thomas Richter <tmricht@linux.ibm.com> wrote: > > > >> I think I found one issue: > >> > >> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug' > >> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so > >> /usr/lib/debug/lib64/libc-2.33.so.debug > >> [root@s8360047 ~]# > >> > >> The file was located in > >> /usr/lib/debug/lib64/libc-2.33.so.debug > >> and hard linked to (or vice versa) > >> /usr/lib/debug/usr/lib64/libc-2.33.so.debug > >> > >> With Fedora 34 the glibc debug file name changed to: > >> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug > >> and > >> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug > > > > Oh, the naming scheme has been changed. > > > >> > >> This is important. The call chain is > >> > >> ... > >> try_to_find_probe_trace_events() > >> +-> open_debuginfo() > >> +-> debuginfo__new(/usr/lib64/libc-2.33.so) > >> +-> dso__read_binary_type_filename() > >> > >> Function dso__read_binary_type_filename() now tries to find the debug file > >> for /usr/lib64/libc-2.33.so for various distros starting with > >> FEDORA_DEBUGINFO. Tt builds the file name by prepending > >> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34 > >> name change. Now the debug file is not found anymore and various other > >> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so > >> file itself is used. > > > > Hmm, we might need a generic way to get such debuginfo path among > > linux distros... are there any good command? > > IMHO The only option that makes sense is to use the buildid: > The s390 compiler team recommended this to me. > > [root@t35lp46 perf]# file /usr/lib64/libc-2.33.so > /usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped > [root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/ > total 7912 > -rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug > -rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf > -rw-r--r-- 1 root root 4754 Jun 23 10:23 probes > [root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton' > 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 > 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton > [root@t35lp46 perf]# This debug file is copied by perf, isn't it? > The bad thing is that only the file part xx/yy/debug seems to be agreed upon > between distros. There are multiple/different locations containing .build-id directories. > > On the other hand, perf is connected to the kernel, so we could 'convince' > the distro's to use their distro specific locations when they build perf. Hmm, or they can add /etc/perf.conf, to provide their customized debuginfo path with a template pattern. > > [....] > > >> > >> Second issue: > >> The symbol 'inet_pton' is defined as WEAK. Even when the correct > >> debuginfo file is loaded for libc in function dso__read_binary_type_filename(), > >> the perf probe command fails. > >> However it works for GLOBAL and LOCAL symbols, so it would be definitly an > >> improvement. > >> > >> I have no clue on dwarf so I put Masami on the addressee list. > >> This second issue is somehow related to dwarf_getfuncs() and > >> the probe_point_search_cb() call back function invoked by dwarf_getfuncs() > >> in the probe_finder.c file. Too far away from my horizon :-)... > >> > >> On 6/23/21 4:21 PM, Thomas Richter wrote: > >>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails: > >>> [root@m46lp22 perf]# ./perf test 78 > >>> 78: probe libc's inet_pton & backtrace it with ping : FAILED! > >>> [root@m46lp22 perf]# > >>> > >>> It boils down to this command and happens after glibc is update to level 2.33-18. > >>> > >>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton > >>> Probe point 'inet_pton' not found. > >>> Error: Failed to add events. > > > > Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only > > open the library without debuginfo? > > # nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton > 00000000001234c0 W inet_pton@@GLIBC_2.2 > 0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE > # readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton > 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > # nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton > nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols > # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton > 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 > 24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...] > 24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton > 25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton > 26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton > 27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length > 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton > # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton > 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > # > > > > >>> [root@f34 ~]# rpm -qa | fgrep glibc > >>> glibc-all-langpacks-2.33-18.fc34.x86_64 > >>> glibc-common-2.33-18.fc34.x86_64 > >>> glibc-langpack-en-2.33-18.fc34.x86_64 > >>> glibc-2.33-18.fc34.x86_64 > >>> glibc-doc-2.33-18.fc34.noarch > >>> glibc-headers-x86-2.33-18.fc34.noarch > >>> glibc-devel-2.33-18.fc34.x86_64 > >>> glibc-debugsource-2.33-18.fc34.x86_64 > >>> glibc-debuginfo-2.33-18.fc34.x86_64 > >>> [root@f34 ~]# > >>> > >>> The symbol inet_pton is now in the .dynsym section of glibc: > >>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' > >>> Symbol table '.dynsym' contains 2419 entries: > >>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 > >>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE > >>> Symbol table '.symtab' contains 104 entries: > >>> [root@f34 ~]# > >>> > >>> The .symtab section does not contain symbol inet_pton. It contains very few symbols > >>> compared to previous versions. > > > > Hmm, if perf probe can not find the debuginfo, it failback to symbol map, > > can you add -vvv option to run the perf probe again? > > > > # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton Ah, OK. This helps me a lot. > probe-definition(0): inet_pton > symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null) > Open Debuginfo file: /usr/lib64/libc-2.33.so > Try to find probe point from debuginfo. Hmm, strange. Does the /usr/lib64/libc-2.33.so have .debuginfo?? (or, did you copy it?) > try_to_find_probe_trace_events ntevs 0 > try_to_find_probe_trace_events ret -2 > try_to_find_probe_trace_events ntevs2 0 > Probe point 'inet_pton' not found. > Error: Failed to add events. Reason: No such file or directory (Code: -2) I guess the libc binary may have a partial debuginfo and it has no full DIE tree. Thus the try_to_find_probe_trace_events() failed. I'll try to check on the Fedora34. Thank you, > # > [....] > > > > Thank you, > > > > > -- > Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany > -- > Vorsitzender des Aufsichtsrats: Gregor Pillen > Geschäftsführung: Dirk Wittkopp > Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-29 6:42 ` Thomas Richter 2021-06-29 14:06 ` Masami Hiramatsu @ 2021-06-30 16:02 ` Masami Hiramatsu 2021-07-02 0:41 ` Masami Hiramatsu 1 sibling, 1 reply; 11+ messages in thread From: Masami Hiramatsu @ 2021-06-30 16:02 UTC (permalink / raw) To: Thomas Richter Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler Hi Thomas, OK, I finally found 2 root causes for this issue. 1. debuginfo__new() forgot to call dso__load(). Thus, the dso is just initialized, but doesn't load the build-id. Then the build-id based debuginfo file is not searched. It must use map instead of dso. 2. Not sure why, but /usr/lib64/libc-2.33.so is shown as "not stripped" binary. Then the elfutils is fooled by the information and open it as a debuginfo file. I think probe-finder.c also need to check the given file has DIE tree (debuginfo) or not by itself. Thank you, On Tue, 29 Jun 2021 08:42:15 +0200 Thomas Richter <tmricht@linux.ibm.com> wrote: > On 6/29/21 7:02 AM, Masami Hiramatsu wrote: > > On Fri, 25 Jun 2021 12:43:29 +0200 > > Thomas Richter <tmricht@linux.ibm.com> wrote: > > > >> I think I found one issue: > >> > >> Fedora 33 named the debug files for glibc 'libc-2.33.so.debug' > >> [root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so > >> /usr/lib/debug/lib64/libc-2.33.so.debug > >> [root@s8360047 ~]# > >> > >> The file was located in > >> /usr/lib/debug/lib64/libc-2.33.so.debug > >> and hard linked to (or vice versa) > >> /usr/lib/debug/usr/lib64/libc-2.33.so.debug > >> > >> With Fedora 34 the glibc debug file name changed to: > >> /usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug > >> and > >> /usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug > > > > Oh, the naming scheme has been changed. > > > >> > >> This is important. The call chain is > >> > >> ... > >> try_to_find_probe_trace_events() > >> +-> open_debuginfo() > >> +-> debuginfo__new(/usr/lib64/libc-2.33.so) > >> +-> dso__read_binary_type_filename() > >> > >> Function dso__read_binary_type_filename() now tries to find the debug file > >> for /usr/lib64/libc-2.33.so for various distros starting with > >> FEDORA_DEBUGINFO. Tt builds the file name by prepending > >> '/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34 > >> name change. Now the debug file is not found anymore and various other > >> distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so > >> file itself is used. > > > > Hmm, we might need a generic way to get such debuginfo path among > > linux distros... are there any good command? > > IMHO The only option that makes sense is to use the buildid: > The s390 compiler team recommended this to me. > > [root@t35lp46 perf]# file /usr/lib64/libc-2.33.so > /usr/lib64/libc-2.33.so: ELF 64-bit MSB shared object, IBM S/390, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=15da6d15f3b4d5042dab1246222479e577fe9190, for GNU/Linux 3.2.0, not stripped > [root@t35lp46 perf]# ll /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/ > total 7912 > -rw-r--r-- 2 root root 6372968 Jun 21 09:01 debug > -rwxr-xr-x 2 root root 1719672 Jun 21 09:01 elf > -rw-r--r-- 1 root root 4754 Jun 23 10:23 probes > [root@t35lp46 perf]# readelf -s /root/.debug/.build-id/15/da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep ' inet_pton' > 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 > 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton > [root@t35lp46 perf]# > > The bad thing is that only the file part xx/yy/debug seems to be agreed upon > between distros. There are multiple/different locations containing .build-id directories. > > On the other hand, perf is connected to the kernel, so we could 'convince' > the distro's to use their distro specific locations when they build perf. > > [....] > > >> > >> Second issue: > >> The symbol 'inet_pton' is defined as WEAK. Even when the correct > >> debuginfo file is loaded for libc in function dso__read_binary_type_filename(), > >> the perf probe command fails. > >> However it works for GLOBAL and LOCAL symbols, so it would be definitly an > >> improvement. > >> > >> I have no clue on dwarf so I put Masami on the addressee list. > >> This second issue is somehow related to dwarf_getfuncs() and > >> the probe_point_search_cb() call back function invoked by dwarf_getfuncs() > >> in the probe_finder.c file. Too far away from my horizon :-)... > >> > >> On 6/23/21 4:21 PM, Thomas Richter wrote: > >>> I just updated Fedora34 to the latest level and discovered that perf test 78 fails: > >>> [root@m46lp22 perf]# ./perf test 78 > >>> 78: probe libc's inet_pton & backtrace it with ping : FAILED! > >>> [root@m46lp22 perf]# > >>> > >>> It boils down to this command and happens after glibc is update to level 2.33-18. > >>> > >>> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton > >>> Probe point 'inet_pton' not found. > >>> Error: Failed to add events. > > > > Hmm, what does "nm" say? Also, Does perf correctly open the debuginfo file or only > > open the library without debuginfo? > > # nm -D /usr/lib64/libc-2.33.so | fgrep inet_pton > 00000000001234c0 W inet_pton@@GLIBC_2.2 > 0000000000123460 T __inet_pton_length@@GLIBC_PRIVATE > # readelf -s /usr/lib64/libc-2.33.so | fgrep inet_pton > 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > # nm -D /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton > nm: /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug: no symbols > # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/debug | fgrep inet_pton > 21092: 0000000000000000 0 FILE LOCAL DEFAULT ABS inet_pton.c > 21105: 0000000000123150 250 FUNC LOCAL DEFAULT 13 inet_pton4 > 21108: 0000000000123250 524 FUNC LOCAL DEFAULT 13 inet_pton6 > 24179: 0000000000123460 90 FUNC LOCAL DEFAULT 13 __GI___inet_pton[...] > 24463: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __inet_pton > 25102: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI___inet_pton > 26439: 00000000001234c0 106 FUNC LOCAL DEFAULT 13 __GI_inet_pton > 27023: 0000000000123460 90 FUNC GLOBAL DEFAULT 13 __inet_pton_length > 27909: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton > # readelf -s /root/.debug/usr/lib64/libc-2.33.so/15da6d15f3b4d5042dab1246222479e577fe9190/elf | fgrep inet_pton > 673: 00000000001234c0 106 FUNC WEAK DEFAULT 13 inet_pton@@GLIBC_2.2 > # > > > > >>> [root@f34 ~]# rpm -qa | fgrep glibc > >>> glibc-all-langpacks-2.33-18.fc34.x86_64 > >>> glibc-common-2.33-18.fc34.x86_64 > >>> glibc-langpack-en-2.33-18.fc34.x86_64 > >>> glibc-2.33-18.fc34.x86_64 > >>> glibc-doc-2.33-18.fc34.noarch > >>> glibc-headers-x86-2.33-18.fc34.noarch > >>> glibc-devel-2.33-18.fc34.x86_64 > >>> glibc-debugsource-2.33-18.fc34.x86_64 > >>> glibc-debuginfo-2.33-18.fc34.x86_64 > >>> [root@f34 ~]# > >>> > >>> The symbol inet_pton is now in the .dynsym section of glibc: > >>> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)' > >>> Symbol table '.dynsym' contains 2419 entries: > >>> 628: 000000000011ea00 108 FUNC WEAK DEFAULT 15 inet_pton@@GLIBC_2.2.5 > >>> 2251: 000000000011e9b0 76 FUNC GLOBAL DEFAULT 15 __inet_pton_length@@GLIBC_PRIVATE > >>> Symbol table '.symtab' contains 104 entries: > >>> [root@f34 ~]# > >>> > >>> The .symtab section does not contain symbol inet_pton. It contains very few symbols > >>> compared to previous versions. > > > > Hmm, if perf probe can not find the debuginfo, it failback to symbol map, > > can you add -vvv option to run the perf probe again? > > > > # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a inet_pton > probe-definition(0): inet_pton > symbol:inet_pton file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > symbol:__v1setjmp file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:longjmp file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:longjmp_target file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:lll_lock_wait_private file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:cond_destroy file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:cond_init file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_arena_max file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_arena_test file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_max_bytes file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_count file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tunable_tcache_unsorted_limit file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_trim_threshold file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_top_pad file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mmap_threshold file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mmap_max file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_perturb file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_mxfast file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_new file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_sbrk_less file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_free file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_tcache_double_free file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_less file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_heap_more file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_sbrk_more file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse_free_list file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_reuse_wait file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_new file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_arena_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_malloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_memalign_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt_free_dyn_thresholds file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_realloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_calloc_retry file:(null) line:0 offset:0 return:0 lazy:(null) > symbol:memory_mallopt file:(null) line:0 offset:0 return:0 lazy:(null) > Open Debuginfo file: /usr/lib64/libc-2.33.so > Try to find probe point from debuginfo. > try_to_find_probe_trace_events ntevs 0 > try_to_find_probe_trace_events ret -2 > try_to_find_probe_trace_events ntevs2 0 > Probe point 'inet_pton' not found. > Error: Failed to add events. Reason: No such file or directory (Code: -2) > # > [....] > > > > Thank you, > > > > > -- > Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany > -- > Vorsitzender des Aufsichtsrats: Gregor Pillen > Geschäftsführung: Dirk Wittkopp > Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update 2021-06-30 16:02 ` Masami Hiramatsu @ 2021-07-02 0:41 ` Masami Hiramatsu 0 siblings, 0 replies; 11+ messages in thread From: Masami Hiramatsu @ 2021-07-02 0:41 UTC (permalink / raw) To: Masami Hiramatsu Cc: Thomas Richter, linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler Hi, On Thu, 1 Jul 2021 01:02:08 +0900 Masami Hiramatsu <mhiramat@kernel.org> wrote: > Hi Thomas, > > OK, I finally found 2 root causes for this issue. > > 1. debuginfo__new() forgot to call dso__load(). Thus, the dso is > just initialized, but doesn't load the build-id. Then the build-id > based debuginfo file is not searched. It must use map instead of dso. I made a fix for this issue using dso__set_build_id() because using dso__load() which loads symbols just for build-id seems sledgehammer. So now I can see the perf probe opens build-id based debuginfo correctly. However, this will not fix your issue because; > 2. Not sure why, but /usr/lib64/libc-2.33.so is shown as "not stripped" > binary. Then the elfutils is fooled by the information and open it > as a debuginfo file. I think probe-finder.c also need to check the > given file has DIE tree (debuginfo) or not by itself. this part is not correct. perf-probe is using elfutils, which has its own debuginfo search algorithm. Thus with or without the above fix, perf-probe finally opens the correct debuginfo. (from this reason, I can not show what has been fixed by the command output in commit message without debug print.) And with decoding the debuginfo, I finally understand what was wrong. The problem is that the "inet_pton" is a dynamic symbol, which does *NOT* exist in the debuginfo. There is "__inet_pton" instead of "inet_pton". ----- /* Like __inet_pton_length, but use strlen (SRC) as the length of SRC. */ int __inet_pton (int af, const char *src, void *dst) { return __inet_pton_length (af, src, strlen (src), dst); } libc_hidden_def (__inet_pton) weak_alias (__inet_pton, inet_pton) libc_hidden_weak (inet_pton) ----- Thus, if you pass the __inet_pton to perf-probe, it works. ----- $ ./perf probe -x /lib64/libc.so.6 -L __inet_pton <__inet_pton@/usr/src/debug/glibc-2.33-18.fc34.aarch64/resolv/inet_pton.c:0> 0 __inet_pton (int af, const char *src, void *dst) { return __inet_pton_length (af, src, strlen (src), dst); } ----- And -F (--functions) doesn't work yet. ----- ./perf probe -x /usr/lib64/libc-2.33.so -F @plt @plt calloc@plt free@plt malloc@plt memalign@plt realloc@plt ----- I think we should remove @plt, and decode ".dynsym" in addition to ".symtab" by map object for this issue at first. Then we may be able to get the address of "inet_pton" from map. Thank you, -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update, more info 2021-06-29 5:02 ` Masami Hiramatsu 2021-06-29 6:42 ` Thomas Richter @ 2021-06-29 7:21 ` Thomas Richter 2021-06-29 14:01 ` Masami Hiramatsu 1 sibling, 1 reply; 11+ messages in thread From: Thomas Richter @ 2021-06-29 7:21 UTC (permalink / raw) To: Masami Hiramatsu Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler On 6/29/21 7:02 AM, Masami Hiramatsu wrote: > On Fri, 25 Jun 2021 12:43:29 +0200 > Thomas Richter <tmricht@linux.ibm.com> wrote: > Masami, here is more info on this issue: # nm /usr/lib64/libc-2.33.so 00000000000a7f30 t __GI_memchr 00000000000a3730 t __GI_memcmp 00000000000a37e0 t __GI_memcpy 00000000000a39a0 t __GI_memmove 00000000000a37d0 t __GI_mempcpy 00000000000a8000 t __GI___rawmemchr 00000000000a6070 t __GI_stpcpy 000000000009d190 t __GI___strcasecmp_l 00000000000a67a0 t __GI_strcat 00000000000a6ec0 t __GI_strchr 00000000000a6b70 t __GI_strcmp 00000000000a5f70 t __GI_strcpy 00000000000a7c70 t __GI_strcspn 00000000000a5ad0 t __GI_strlen 000000000009d200 t __GI___strncasecmp_l 00000000000a6ce0 t __GI_strncmp 00000000000a62a0 t __GI_strncpy 00000000000a5d30 t __GI_strnlen 00000000000a73e0 t __GI_strrchr 00000000000b8e80 t __GI_wmemchr 0000000000111f60 T __memcpy_chk 0000000000111f80 T __memmove_chk 0000000000112040 T __stpcpy_chk 0000000000112100 T __strcpy_chk # These are the entries in the .symtab section and can be used by perf: # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a __strcpy_chk # ./perf probe -l probe_libc:__strcpy_chk (on __strcpy_chk@debug/strcpy_chk.c in /usr/lib64/libc-2.33.so) [root@t35lp46 perf]# Hope this helps -- Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: perf test case probe libc fails with latest Fedora34 glibc update, more info 2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter @ 2021-06-29 14:01 ` Masami Hiramatsu 0 siblings, 0 replies; 11+ messages in thread From: Masami Hiramatsu @ 2021-06-29 14:01 UTC (permalink / raw) To: Thomas Richter Cc: linux-perf-use., Arnaldo Carvalho de Melo, Sven Schnelle, Heiko Carstens, Stefan Liebler On Tue, 29 Jun 2021 09:21:19 +0200 Thomas Richter <tmricht@linux.ibm.com> wrote: > On 6/29/21 7:02 AM, Masami Hiramatsu wrote: > > On Fri, 25 Jun 2021 12:43:29 +0200 > > Thomas Richter <tmricht@linux.ibm.com> wrote: > > > > Masami, > > here is more info on this issue: > > # nm /usr/lib64/libc-2.33.so > 00000000000a7f30 t __GI_memchr > 00000000000a3730 t __GI_memcmp > 00000000000a37e0 t __GI_memcpy > 00000000000a39a0 t __GI_memmove > 00000000000a37d0 t __GI_mempcpy > 00000000000a8000 t __GI___rawmemchr > 00000000000a6070 t __GI_stpcpy > 000000000009d190 t __GI___strcasecmp_l > 00000000000a67a0 t __GI_strcat > 00000000000a6ec0 t __GI_strchr > 00000000000a6b70 t __GI_strcmp > 00000000000a5f70 t __GI_strcpy > 00000000000a7c70 t __GI_strcspn > 00000000000a5ad0 t __GI_strlen > 000000000009d200 t __GI___strncasecmp_l > 00000000000a6ce0 t __GI_strncmp > 00000000000a62a0 t __GI_strncpy > 00000000000a5d30 t __GI_strnlen > 00000000000a73e0 t __GI_strrchr > 00000000000b8e80 t __GI_wmemchr > 0000000000111f60 T __memcpy_chk > 0000000000111f80 T __memmove_chk > 0000000000112040 T __stpcpy_chk > 0000000000112100 T __strcpy_chk > # > > These are the entries in the .symtab section and can be used by perf: > # ./perf probe -vvv -f -x /usr/lib64/libc-2.33.so -a __strcpy_chk Are there any output of the perf probe? It may report that the event is found by map (symbol) or debuginfo. > # ./perf probe -l > probe_libc:__strcpy_chk (on __strcpy_chk@debug/strcpy_chk.c in /usr/lib64/libc-2.33.so) > [root@t35lp46 perf]# Thanks! So at least the symbols in .symtab works. > > Hope this helps > -- > Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany > -- > Vorsitzender des Aufsichtsrats: Gregor Pillen > Geschäftsführung: Dirk Wittkopp > Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-07-02 0:41 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-06-23 14:21 perf test case probe libc fails with latest Fedora34 glibc update Thomas Richter 2021-06-25 10:34 ` Jiri Olsa 2021-06-25 10:57 ` Thomas Richter 2021-06-25 10:43 ` Part 2: " Thomas Richter 2021-06-29 5:02 ` Masami Hiramatsu 2021-06-29 6:42 ` Thomas Richter 2021-06-29 14:06 ` Masami Hiramatsu 2021-06-30 16:02 ` Masami Hiramatsu 2021-07-02 0:41 ` Masami Hiramatsu 2021-06-29 7:21 ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter 2021-06-29 14:01 ` Masami Hiramatsu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.