Linux-Trace-Users Archive on lore.kernel.org
 help / color / Atom feed
From: ahmadkhorrami <ahmadkhorrami@ut.ac.ir>
To: Milian Wolff <milian.wolff@kdab.com>
Cc: Jiri Olsa <jolsa@redhat.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Arnaldo Carvalho de Melo <acme@redhat.com>,
	Linux-trace Users <linux-trace-users@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	linux-trace-users-owner@vger.kernel.org,
	Jin Yao <yao.jin@linux.intel.com>,
	Namhyung Kim <namhyung@kernel.org>,
	Andi Kleen <ak@linux.intel.com>
Subject: Re: Wrong Perf Backtraces
Date: Sat, 11 Apr 2020 21:12:25 +0430
Message-ID: <e624e60c697934beb88cb8744f23c68b@ut.ac.ir> (raw)
In-Reply-To: <9dc590447c61e044934bdad4127297b8@ut.ac.ir>

Hi,
I found the problem by building the libraries from source code. The 
"gmallocn()"s were inlines. So, perhaps it would be better if Perf had 
reported it as "(inlined)" as in the common case.
The second problematic case (repeated 
"gtk_action_muxer_query_action()"s) was purely due to my lack of 
knowledge. Sorry for wasting your time, specially, Mr. Olsa, :D. It was 
inter-library tail-call elimination, which I did not know is possible (I 
thought tail-calls happen only inside each translation unit). In other 
words, the last operation in the callee at 
"gtk_action_muxer_query_action+0x6c" is a jump to the beginning of 
"gtk_action_muxer_query_action()".
And the debug info for the dbg-packages (Ubuntu in my case), seems to be 
OK. I wonder why source lines were reported as 0. Perhaps, the unwinding 
library is bogus.

Finally, I have a new (hopefully, unsilly) question, :D. In the 
following backtrace, __GI___libc_malloc+0x197 is reported as inline. 
While its address (i.e., 0x7ffff4b07207) does not match that of its 
parent function call point (i.e., gmalloc+0x59 which is 0x7fffd9872fb9). 
Is this logical? And, again, there are many such instances in the Perf 
output.

EvJobScheduler 10021  8653.926478:        100 
mem_load_uops_retired.l3_miss:uppp:     7fffd1062a00         5080022 
N/A|SNP N/A|TLB N/A|LCK N/A
    7ffff4b07207 tcache_get+0x197 (inlined)
    7ffff4b07207 __GI___libc_malloc+0x197 (inlined)
    7fffd9872fb9 gmalloc+0x59 (inlined)
    7fffd9872fb9 gmallocn+0x59 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffd9872fb9 gmallocn+0x59 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffd9951e6f _ZN8TextLine8coalesceEP10UnicodeMap+0xff 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffd9952f82 _ZN9TextBlock8coalesceEP10UnicodeMapd+0x752 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffd995bc37 _ZN8TextPage8coalesceEbdb+0x1507 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffd995cb71 _ZN13TextOutputDev7endPageEv+0x31 
(/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
    7fffe803c6d2 _ZL26poppler_page_get_text_pageP12_PopplerPage+0x92 
(/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0)
    7fffe803deb3 poppler_page_get_selection_region+0x63 
(/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0)
    7fffe82ab650 [unknown] 
(/opt/evince-3.28.4/lib/evince/4/backends/libpdfdocument.so)
    7ffff795f165 ev_job_page_data_run+0x2f5 
(/opt/evince-3.28.4/lib/libevview3.so.3.0.0)
    7ffff7961309 ev_job_thread+0xe9 (inlined)
    7ffff7961309 ev_job_thread_proxy+0xe9 
(/opt/evince-3.28.4/lib/libevview3.so.3.0.0)
    7ffff5492194 g_thread_proxy+0x54 
(/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
    7ffff4e686da start_thread+0xda 
(/lib/x86_64-linux-gnu/libpthread-2.27.so)
    7ffff4b9188e __GI___clone+0x3e (inlined)

I am looking forward to any help or suggestions.
Regards.

On 2020-04-04 05:31, ahmadkhorrami wrote:

> Hi,
> I think that I should give up and assume that all illogical repeated 
> addresses are inlines. Thanks everybody, especially, Jirka, Milian, 
> Arnaldo and Steven. Perhaps some day I will be back and try to dig 
> deeper (with your hints and assistance).
> Regards.
> 
> On 2020-04-01 01:27, ahmadkhorrami wrote:
> 
>> Hi,
>> On 2020-03-31 19:59, Milian Wolff wrote:
>> 
>> On Dienstag, 31. März 2020 17:02:37 CEST ahmadkhorrami wrote:
>> 
>> Hi Milian,
>> Thanks for the detailed answer. Well, the bug you mentioned is bad 
>> news.
>> Because I sample using uppp. Perhaps this leads to these weird traces.
>> Please read the full thread from here on:
>> 
>> https://lkml.org/lkml/2018/11/2/86
>> 
>> But as I said - it should be easy to check if this is really the issue 
>> are
>> running into or not: Try to see if you see the problem when you sample 
>> without
>> `ppp`. If not, then you can be pretty sure it's this issue. If you 
>> still see
>> it, then it's something different.
> Well, the problem exists even when ":uppp" is changed to ":u".
> 
> Is this a purely software bug?
> I wouldn't call it that, personally. Rather, it's a limitation in the 
> hardware
> and software. We would need something completely different to "fix" 
> this, i.e.
> something like a deeper LBR. That's btw another alternative you could 
> try:
> `perf record --call-graph lbr` and live with the short call stacks. But 
> at
> least these should be correct (afaik). For me personally they are 
> always far
> too short and thus not practical to use in reality.
> 
> Cheers
   Regards.

  reply index

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <157597d74ff17f781d9de7e7e3defd13@ut.ac.ir>
2020-03-22 20:24 ` ahmadkhorrami
2020-03-23  0:34   ` Steven Rostedt
     [not found]     ` <21b3df4080709f193d62b159887e2a83@ut.ac.ir>
2020-03-23  8:49       ` Jiri Olsa
2020-03-23 10:03         ` ahmadkhorrami
2020-03-25 15:18           ` ahmadkhorrami
2020-03-25 15:46             ` Jiri Olsa
2020-03-25 18:54               ` ahmadkhorrami
2020-03-25 18:58               ` Arnaldo Carvalho de Melo
2020-03-25 19:10                 ` ahmadkhorrami
2020-03-25 19:28                   ` Arnaldo Carvalho de Melo
2020-03-25 20:01                     ` ahmadkhorrami
2020-03-25 20:39                       ` Jiri Olsa
2020-03-25 21:02                         ` Jiri Olsa
2020-03-25 21:09                           ` Steven Rostedt
2020-03-25 21:37                             ` ahmadkhorrami
2020-03-25 21:46                               ` Jiri Olsa
2020-03-25 22:21                                 ` ahmadkhorrami
2020-03-25 23:09                                   ` ahmadkhorrami
2020-03-26  9:59                                     ` Jiri Olsa
2020-03-26 13:20                                       ` ahmadkhorrami
2020-03-26 15:39                                         ` Jiri Olsa
2020-03-26 18:19                                           ` ahmadkhorrami
2020-03-26 18:21                                             ` ahmadkhorrami
2020-03-27  9:20                                             ` Jiri Olsa
2020-03-27 10:59                                               ` ahmadkhorrami
2020-03-27 11:04                                                 ` ahmadkhorrami
2020-03-27 12:10                                                   ` Milian Wolff
2020-03-27 12:58                                                     ` ahmadkhorrami
2020-03-27 13:25                                                       ` Milian Wolff
2020-03-27 13:33                                                         ` ahmadkhorrami
2020-03-27 18:43                                                   ` ahmadkhorrami
2020-03-27 22:37                                                     ` Jiri Olsa
2020-03-27 23:12                                                       ` ahmadkhorrami
2020-03-28 23:34                                                         ` Jiri Olsa
2020-03-29  0:43                                                           ` ahmadkhorrami
2020-03-29  1:16                                                             ` ahmadkhorrami
2020-03-29 11:19                                                               ` Jiri Olsa
2020-03-29 11:52                                                                 ` ahmadkhorrami
2020-03-29 12:08                                                                   ` Jiri Olsa
2020-03-29 12:39                                                                     ` ahmadkhorrami
2020-03-29 13:50                                                                       ` Milian Wolff
2020-03-29 14:23                                                                         ` ahmadkhorrami
2020-03-29 19:20                                                                         ` Jiri Olsa
2020-03-30  6:09                                                                           ` Milian Wolff
2020-03-30 13:07                                                                             ` Jiri Olsa
2020-03-30 13:49                                                                               ` ahmadkhorrami
2020-03-30 19:05                                                                                 ` ahmadkhorrami
2020-03-30 21:05                                                                                   ` debuginfod-based dwarf downloading, was " Frank Ch. Eigler
2020-03-31  9:26                                                                                     ` Jiri Olsa
2020-03-31 14:00                                                                                       ` Frank Ch. Eigler
2020-03-31  4:43                                                                                   ` ahmadkhorrami
2020-03-31  9:30                                                                                     ` Jiri Olsa
2020-03-31 11:53                                                                                       ` ahmadkhorrami
2020-03-31 12:43                                                                                   ` ahmadkhorrami
2020-03-31 13:20                                                                                     ` Jiri Olsa
2020-03-31 13:39                                                                                       ` ahmadkhorrami
2020-03-31 14:44                                                                                         ` Milian Wolff
2020-03-31 15:02                                                                                           ` ahmadkhorrami
2020-03-31 15:05                                                                                             ` ahmadkhorrami
2020-03-31 15:29                                                                                             ` Milian Wolff
2020-03-31 16:10                                                                                               ` Arnaldo Carvalho de Melo
2020-03-31 19:20                                                                                                 ` ahmadkhorrami
2020-03-31 19:17                                                                                               ` ahmadkhorrami
2020-03-31 20:57                                                                                               ` ahmadkhorrami
2020-04-04  1:01                                                                                                 ` ahmadkhorrami
2020-04-11 16:42                                                                                                   ` ahmadkhorrami [this message]
2020-04-11 21:04                                                                                                     ` ahmadkhorrami

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e624e60c697934beb88cb8744f23c68b@ut.ac.ir \
    --to=ahmadkhorrami@ut.ac.ir \
    --cc=acme@redhat.com \
    --cc=ak@linux.intel.com \
    --cc=jolsa@redhat.com \
    --cc=linux-trace-users-owner@vger.kernel.org \
    --cc=linux-trace-users@vger.kernel.org \
    --cc=milian.wolff@kdab.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=yao.jin@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Trace-Users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \
		linux-trace-users@vger.kernel.org
	public-inbox-index linux-trace-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git