Linux-Trace-Users Archive on lore.kernel.org
 help / color / Atom feed
From: Jiri Olsa <jolsa@redhat.com>
To: ahmadkhorrami <ahmadkhorrami@ut.ac.ir>
Cc: Steven Rostedt <rostedt@goodmis.org>,
	Arnaldo Carvalho de Melo <acme@redhat.com>,
	Linux-trace Users <linux-trace-users@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	linux-trace-users-owner@vger.kernel.org,
	Jin Yao <yao.jin@linux.intel.com>,
	Milian Wolff <milian.wolff@kdab.com>,
	Namhyung Kim <namhyung@kernel.org>,
	Changbin Du <changbin.du@intel.com>,
	Andi Kleen <ak@linux.intel.com>
Subject: Re: Wrong Perf Backtraces
Date: Thu, 26 Mar 2020 10:59:28 +0100
Message-ID: <20200326095928.GF1947699@krava> (raw)
In-Reply-To: <52904543671bcb68ea9109b90c3e4267@ut.ac.ir>

On Thu, Mar 26, 2020 at 03:39:31AM +0430, ahmadkhorrami wrote:
> An here is the second one:
> https://gofile.io/?c=oGxgSM

thanks, so far I don't see that, but I think it's because
the 'inline' code does not resolve the libc dso correctly,
CC-ing few other folks..

so I'm able to get:

	EvJobScheduler 17382 13006.872877:      10000 mem_load_uops_retired.l3_miss:uppp:     7fffd2e06588         5080022 N/A|SNP N/A|TLB N/A|LCK N/A
		    7ffff4b04c74 [unknown] (/lib/x86_64-linux-gnu/libc-2.27.so)
		    7ffff4b072ec malloc+0x27c (/lib/x86_64-linux-gnu/libc-2.27.so)
		    7fffd9872ddd gmalloc+0xd (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
		    7fffd9873391 copyString+0x11 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)

while you see:

	EvJobScheduler 17382 13006.872877:      10000 mem_load_uops_retired.l3_miss:uppp:     7fffd2e06588         5080022 N/A|SNP N/A|TLB N/A|LCK N/A
		    7ffff4b04c74 _int_malloc+0x9a4 (/lib/x86_64-linux-gnu/libc-2.27.so)
		    7ffff4b072ec __GI___libc_malloc+0x27c (inlined)
		    7fffd9872ddd gmalloc+0xd (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
		    7fffd9872ddd gmalloc+0xd (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)

so for some reason I don't resolve 7ffff4b04c74, which might
be the reason I don't see the following address twice as you do:

	    7fffd9872ddd gmalloc+0xd (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
	    7fffd9872ddd gmalloc+0xd (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)

the previous field (7ffff4b072ec) is resolved by the
'inline' code, so I wonder it's related

I needed to make some changes to utils/srcline.c to be able
to properly open dso via buildid cache, I pushed it to:

  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/callchain

here are the steps to get above output:

  download perf.data and perf archive data from:
    https://gofile.io/?c=o95O7N
    http://gofile.io/?c=qk6oXv

  $ unxz ./perf.data.xz
  $ tar xvf perf.data.tar.bz2 -C ~/.debug

compile perf from aobve tree/branch and run:

  $ perf script -i perf.data

I think we might be missing some libraries in 'perf archive'
by not following debug_link section, will check on this

jirka


> Regards
> On 2020-03-26 02:51, ahmadkhorrami wrote:
> 
> > Here is the link for the gmallocn()s:
> > http://gofile.io/?c=qk6oXv
> > I will send the second one as soon as the upload is finished:
> > 
> > Regards.
> > 
> > On 2020-03-26 02:16, Jiri Olsa wrote:
> > 
> > On Thu, Mar 26, 2020 at 02:07:39AM +0430, ahmadkhorrami wrote:
> > 
> > Here is the link for the gmallocn() case:
> > https://gofile.io/?c=o95O7N
> > The output for the second case is big. I have a small one produced
> > several
> > days ago the link of which is as follows:
> > https://gofile.io/?c=OIPCjx
> > looking good, but I still need you t run 'perf archive' on top of your
> > data and send me the perf.data.tar.bz2 it generates, like:
> > 
> > [jolsa@krava perf]$ sudo ./perf record
> > ^C[ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 1.675 MB perf.data (6248 samples) ]
> > 
> > [jolsa@krava perf]$ sudo ./perf archive
> > Now please run:
> > 
> > $ tar xvf perf.data.tar.bz2 -C ~/.debug
> > 
> > wherever you need to run 'perf report' on.
> > 
> > I need that perf.data.tar.bz2 generated from your data
> > 
> > thanks,
> > jirka
> > 
> > Regards.
> > 
> > On 2020-03-26 01:39, Steven Rostedt wrote:
> > 
> > On Wed, 25 Mar 2020 22:02:52 +0100
> > Jiri Olsa <jolsa@redhat.com> wrote:
> > 
> > yea, no luck.. so if you could generate some reasonable small
> > perf.data that
> > shows the issue and send it over together with 'perf archive' data
> > privately
> > to me and to whoever else ask for it, so we don't polute the list..
> > Right. And it may be better if you compress it too.
> > 
> > xz perf.data
> > 
> > and attach the perf.data.xz (and only privately send it to Mr. Olsa).
> > 
> > -- Steve
> > 
> > or if you could put it somewhere on the web/ftp.. that'd be best
> 


  reply index

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <157597d74ff17f781d9de7e7e3defd13@ut.ac.ir>
2020-03-22 20:24 ` ahmadkhorrami
2020-03-23  0:34   ` Steven Rostedt
     [not found]     ` <21b3df4080709f193d62b159887e2a83@ut.ac.ir>
2020-03-23  8:49       ` Jiri Olsa
2020-03-23 10:03         ` ahmadkhorrami
2020-03-25 15:18           ` ahmadkhorrami
2020-03-25 15:46             ` Jiri Olsa
2020-03-25 18:54               ` ahmadkhorrami
2020-03-25 18:58               ` Arnaldo Carvalho de Melo
2020-03-25 19:10                 ` ahmadkhorrami
2020-03-25 19:28                   ` Arnaldo Carvalho de Melo
2020-03-25 20:01                     ` ahmadkhorrami
2020-03-25 20:39                       ` Jiri Olsa
2020-03-25 21:02                         ` Jiri Olsa
2020-03-25 21:09                           ` Steven Rostedt
2020-03-25 21:37                             ` ahmadkhorrami
2020-03-25 21:46                               ` Jiri Olsa
2020-03-25 22:21                                 ` ahmadkhorrami
2020-03-25 23:09                                   ` ahmadkhorrami
2020-03-26  9:59                                     ` Jiri Olsa [this message]
2020-03-26 13:20                                       ` ahmadkhorrami
2020-03-26 15:39                                         ` Jiri Olsa
2020-03-26 18:19                                           ` ahmadkhorrami
2020-03-26 18:21                                             ` ahmadkhorrami
2020-03-27  9:20                                             ` Jiri Olsa
2020-03-27 10:59                                               ` ahmadkhorrami
2020-03-27 11:04                                                 ` ahmadkhorrami
2020-03-27 12:10                                                   ` Milian Wolff
2020-03-27 12:58                                                     ` ahmadkhorrami
2020-03-27 13:25                                                       ` Milian Wolff
2020-03-27 13:33                                                         ` ahmadkhorrami
2020-03-27 18:43                                                   ` ahmadkhorrami
2020-03-27 22:37                                                     ` Jiri Olsa
2020-03-27 23:12                                                       ` ahmadkhorrami
2020-03-28 23:34                                                         ` Jiri Olsa
2020-03-29  0:43                                                           ` ahmadkhorrami
2020-03-29  1:16                                                             ` ahmadkhorrami
2020-03-29 11:19                                                               ` Jiri Olsa
2020-03-29 11:52                                                                 ` ahmadkhorrami
2020-03-29 12:08                                                                   ` Jiri Olsa
2020-03-29 12:39                                                                     ` ahmadkhorrami
2020-03-29 13:50                                                                       ` Milian Wolff
2020-03-29 14:23                                                                         ` ahmadkhorrami
2020-03-29 19:20                                                                         ` Jiri Olsa
2020-03-30  6:09                                                                           ` Milian Wolff
2020-03-30 13:07                                                                             ` Jiri Olsa
2020-03-30 13:49                                                                               ` ahmadkhorrami
2020-03-30 19:05                                                                                 ` ahmadkhorrami
2020-03-30 21:05                                                                                   ` debuginfod-based dwarf downloading, was " Frank Ch. Eigler
2020-03-31  9:26                                                                                     ` Jiri Olsa
2020-03-31 14:00                                                                                       ` Frank Ch. Eigler
2020-03-31  4:43                                                                                   ` ahmadkhorrami
2020-03-31  9:30                                                                                     ` Jiri Olsa
2020-03-31 11:53                                                                                       ` ahmadkhorrami
2020-03-31 12:43                                                                                   ` ahmadkhorrami
2020-03-31 13:20                                                                                     ` Jiri Olsa
2020-03-31 13:39                                                                                       ` ahmadkhorrami
2020-03-31 14:44                                                                                         ` Milian Wolff
2020-03-31 15:02                                                                                           ` ahmadkhorrami
2020-03-31 15:05                                                                                             ` ahmadkhorrami
2020-03-31 15:29                                                                                             ` Milian Wolff
2020-03-31 16:10                                                                                               ` Arnaldo Carvalho de Melo
2020-03-31 19:20                                                                                                 ` ahmadkhorrami
2020-03-31 19:17                                                                                               ` ahmadkhorrami
2020-03-31 20:57                                                                                               ` ahmadkhorrami
2020-04-04  1:01                                                                                                 ` ahmadkhorrami
2020-04-11 16:42                                                                                                   ` ahmadkhorrami
2020-04-11 21:04                                                                                                     ` ahmadkhorrami

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200326095928.GF1947699@krava \
    --to=jolsa@redhat.com \
    --cc=acme@redhat.com \
    --cc=ahmadkhorrami@ut.ac.ir \
    --cc=ak@linux.intel.com \
    --cc=changbin.du@intel.com \
    --cc=linux-trace-users-owner@vger.kernel.org \
    --cc=linux-trace-users@vger.kernel.org \
    --cc=milian.wolff@kdab.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=yao.jin@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Trace-Users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \
		linux-trace-users@vger.kernel.org
	public-inbox-index linux-trace-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git