linux-perf-users.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Thomas Richter <tmricht@linux.ibm.com>
To: "linux-perf-use." <linux-perf-users@vger.kernel.org>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Masami Hiramatsu <mhiramat@kernel.org>
Cc: Sven Schnelle <svens@linux.ibm.com>,
	Heiko Carstens <hca@linux.ibm.com>,
	Stefan Liebler <stli@linux.ibm.com>
Subject: Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update
Date: Fri, 25 Jun 2021 12:43:29 +0200	[thread overview]
Message-ID: <f6752514-eaf9-371e-f81b-0d9e41ebae0c@linux.ibm.com> (raw)
In-Reply-To: <cae863a4-bf83-f87b-6e71-b6a544678cae@linux.ibm.com>

I think I found one issue:

Fedora 33 named the debug files for glibc 'libc-2.33.so.debug'
[root@s8360047 ~]# rpm -ql glibc-debuginfo-2.33-5.fc34.s390x|fgrep libc-2.33.so
/usr/lib/debug/lib64/libc-2.33.so.debug
[root@s8360047 ~]#

The file was located in
   /usr/lib/debug/lib64/libc-2.33.so.debug
and hard linked to (or vice versa)
   /usr/lib/debug/usr/lib64/libc-2.33.so.debug

With Fedora 34 the glibc debug file name changed to:
/usr/lib/debug/lib64/libc-2.33.so-2.33-18.fc34.s390x.debug
and
/usr/lib/debug/usr/lib64/libc-2.33.so-2.33-18.fc34.x86_64.debug

This is important. The call chain is

 ...
 try_to_find_probe_trace_events()
 +-> open_debuginfo()
     +-> debuginfo__new(/usr/lib64/libc-2.33.so)
         +-> dso__read_binary_type_filename()

Function dso__read_binary_type_filename() now tries to find the debug file
for /usr/lib64/libc-2.33.so for various distros starting with
FEDORA_DEBUGINFO. Tt builds the file name by prepending
'/usr/lib/debug' and appending '.debug'. This was ok until the Fedora 34
name change. Now the debug file is not found anymore and various other
distro name schemes are tried. None is found and the /usr/lib64/libc-2.33.so
file itself is used.

This even worked for Fedora 34 until the glibc 2.33-18 change, because the
/usr/lib/libc-xxx.so file contained the complete symbol table in the
.symtab section. The function debuginfo__find_probe_location() found
the symbol inet_pton. So it did not matter is the debug file was found or not.

With the glib 2.33-18 change most entries in the .symtab section where
moved to the .dynsym section and are not found anymore when the default
/usr/lib/lic-xxx.so is used as fallback. So the issue surfaced.

Second issue:
The symbol 'inet_pton' is defined as WEAK. Even when the correct
debuginfo file is loaded for libc in function dso__read_binary_type_filename(),
the perf probe command fails.
However it works for GLOBAL and LOCAL symbols, so it would be definitly an 
improvement.

I have no clue on dwarf so I put Masami on the addressee list.
This second issue is somehow related to dwarf_getfuncs() and
the probe_point_search_cb() call back function invoked by dwarf_getfuncs()
in the probe_finder.c file. Too far away from my horizon :-)...

On 6/23/21 4:21 PM, Thomas Richter wrote:
> I just updated Fedora34 to the latest level and discovered that perf test 78 fails:
> [root@m46lp22 perf]# ./perf test 78
> 78: probe libc's inet_pton & backtrace it with ping                 : FAILED!
> [root@m46lp22 perf]#
> 
> It boils down to this command and happens after glibc is update to level 2.33-18.
> 
> [root@f34 ~]# perf probe -f -x /usr/lib64/libc-2.33.so -a inet_pton
> Probe point 'inet_pton' not found.
>   Error: Failed to add events.
> [root@f34 ~]# rpm -qa | fgrep glibc
> glibc-all-langpacks-2.33-18.fc34.x86_64
> glibc-common-2.33-18.fc34.x86_64
> glibc-langpack-en-2.33-18.fc34.x86_64
> glibc-2.33-18.fc34.x86_64
> glibc-doc-2.33-18.fc34.noarch
> glibc-headers-x86-2.33-18.fc34.noarch
> glibc-devel-2.33-18.fc34.x86_64
> glibc-debugsource-2.33-18.fc34.x86_64
> glibc-debuginfo-2.33-18.fc34.x86_64
> [root@f34 ~]# 
> 
> The symbol inet_pton is now in the .dynsym section of glibc:
> [root@f34 ~]# readelf -sW /usr/lib64/libc-2.33.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2419 entries:
>    628: 000000000011ea00   108 FUNC    WEAK   DEFAULT   15 inet_pton@@GLIBC_2.2.5
>   2251: 000000000011e9b0    76 FUNC    GLOBAL DEFAULT   15 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 104 entries:
> [root@f34 ~]#
> 
> The .symtab section does not contain symbol inet_pton. It contains very few symbols
> compared to previous versions.
> 
> Now perf does not find it. In the older version of the library the
> symbol inet_pton was listed in the .symtab section. Here is the output from version 2.32-4:
> 
> 
> [root@m35lp76 ~]# rpm -qa | fgrep glibc
> glibc-common-2.32-4.fc33.s390x
> glibc-langpack-en-2.32-4.fc33.s390x
> glibc-2.32-4.fc33.s390x
> glibc-headers-s390-2.32-4.fc33.noarch
> glibc-devel-2.32-4.fc33.s390x
> glibc-debuginfo-common-2.32-4.fc33.s390x
> glibc-debuginfo-2.32-4.fc33.s390x
> [root@m35lp76 ~]# 
> 
> 
> readelf -sW /usr/lib64/libc-2.32.so | egrep '(dynsym|symtab|inet_pton)'
> Symbol table '.dynsym' contains 2604 entries:
>    668: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton@@GLIBC_2.2
>   2418: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length@@GLIBC_PRIVATE
> Symbol table '.symtab' contains 28858 entries:
>  20655: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS inet_pton.c
>  20656: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c
>  20657: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end
>  20658: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.hot
>  20659: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.hot
>  20660: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.unlikely
>  20661: 000000000002b938     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.unlikely
>  20662: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.startup
>  20663: 000000000002ba70     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.startup
>  20664: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c.exit
>  20665: 000000000002b968     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton.c_end.exit
>  20666: 00000000001440b0     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.start
>  20667: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin_inet_pton4.end
>  20668: 00000000001440b0   250 FUNC    LOCAL  DEFAULT   13 inet_pton4
>  20669: 00000000001441aa     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.start
>  20670: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton_length.end
>  20671: 00000000001444ac     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.start
>  20672: 00000000001447c4     0 NOTYPE  LOCAL  HIDDEN    13 .annobin___GI___inet_pton.end
>  23591: 00000000001441b0   764 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton_length
>  23871: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __inet_pton
>  24506: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI___inet_pton
>  25831: 00000000001444b0   788 FUNC    LOCAL  DEFAULT   13 __GI_inet_pton
>  26410: 00000000001441b0   764 FUNC    GLOBAL DEFAULT   13 __inet_pton_length
>  27288: 00000000001444b0   788 FUNC    WEAK   DEFAULT   13 inet_pton
> [root@m35lp76 ~]#
> 
> And perf could find the symbol, extract its address and install a probe
> on that address.
> So is this a bug related to the perf tool because it can not handle .dynsym section?
> Or is it releated to glibc's rework?
> 
> Thanks for your help.
> 


-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

  parent reply	other threads:[~2021-06-25 10:43 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-23 14:21 Thomas Richter
2021-06-25 10:34 ` Jiri Olsa
2021-06-25 10:57   ` Thomas Richter
2021-06-25 10:43 ` Thomas Richter [this message]
2021-06-29  5:02   ` Part 2: " Masami Hiramatsu
2021-06-29  6:42     ` Thomas Richter
2021-06-29 14:06       ` Masami Hiramatsu
2021-06-30 16:02       ` Masami Hiramatsu
2021-07-02  0:41         ` Masami Hiramatsu
2021-06-29  7:21     ` perf test case probe libc fails with latest Fedora34 glibc update, more info Thomas Richter
2021-06-29 14:01       ` Masami Hiramatsu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f6752514-eaf9-371e-f81b-0d9e41ebae0c@linux.ibm.com \
    --to=tmricht@linux.ibm.com \
    --cc=acme@kernel.org \
    --cc=hca@linux.ibm.com \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=mhiramat@kernel.org \
    --cc=stli@linux.ibm.com \
    --cc=svens@linux.ibm.com \
    --subject='Re: Part 2: perf test case probe libc fails with latest Fedora34 glibc update' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).