Linux-Trace-Users Archive on lore.kernel.org
 help / color / Atom feed
From: ahmadkhorrami <ahmadkhorrami@ut.ac.ir>
To: Milian Wolff <milian.wolff@kdab.com>
Cc: Jiri Olsa <jolsa@redhat.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Arnaldo Carvalho de Melo <acme@redhat.com>,
	Linux-trace Users <linux-trace-users@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	linux-trace-users-owner@vger.kernel.org,
	Jin Yao <yao.jin@linux.intel.com>,
	Namhyung Kim <namhyung@kernel.org>,
	Andi Kleen <ak@linux.intel.com>
Subject: Re: Wrong Perf Backtraces
Date: Sun, 12 Apr 2020 01:34:20 +0430
Message-ID: <f1b212e2b5035bbb2ccfb4e74f6f28cd@ut.ac.ir> (raw)
In-Reply-To: <e624e60c697934beb88cb8744f23c68b@ut.ac.ir>

I traced the code and it jumped into PLT and .... Therefore, it seems 
that it is actually a function call and is incorrectly reported as 
inline. In other words, this is a called function incorrectly reported 
as inline, while the "gmallocn" case was an inline incorrectly omitted. 
Hopefully, I'm right.

Regards.

On 2020-04-11 21:12, ahmadkhorrami wrote:

> Hi,
> I found the problem by building the libraries from source code. The 
> "gmallocn()"s were inlines. So, perhaps it would be better if Perf had 
> reported it as "(inlined)" as in the common case.
> The second problematic case (repeated 
> "gtk_action_muxer_query_action()"s) was purely due to my lack of 
> knowledge. Sorry for wasting your time, specially, Mr. Olsa, :D. It was 
> inter-library tail-call elimination, which I did not know is possible 
> (I thought tail-calls happen only inside each translation unit). In 
> other words, the last operation in the callee at 
> "gtk_action_muxer_query_action+0x6c" is a jump to the beginning of 
> "gtk_action_muxer_query_action()".
> And the debug info for the dbg-packages (Ubuntu in my case), seems to 
> be OK. I wonder why source lines were reported as 0. Perhaps, the 
> unwinding library is bogus.
> 
> Finally, I have a new (hopefully, unsilly) question, :D. In the 
> following backtrace, __GI___libc_malloc+0x197 is reported as inline. 
> While its address (i.e., 0x7ffff4b07207) does not match that of its 
> parent function call point (i.e., gmalloc+0x59 which is 
> 0x7fffd9872fb9). Is this logical? And, again, there are many such 
> instances in the Perf output.
> 
> EvJobScheduler 10021  8653.926478:        100 
> mem_load_uops_retired.l3_miss:uppp:     7fffd1062a00         5080022 
> N/A|SNP N/A|TLB N/A|LCK N/A
> 7ffff4b07207 tcache_get+0x197 (inlined)
> 7ffff4b07207 __GI___libc_malloc+0x197 (inlined)
> 7fffd9872fb9 gmalloc+0x59 (inlined)
> 7fffd9872fb9 gmallocn+0x59 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffd9872fb9 gmallocn+0x59 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffd9951e6f _ZN8TextLine8coalesceEP10UnicodeMap+0xff 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffd9952f82 _ZN9TextBlock8coalesceEP10UnicodeMapd+0x752 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffd995bc37 _ZN8TextPage8coalesceEbdb+0x1507 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffd995cb71 _ZN13TextOutputDev7endPageEv+0x31 
> (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0)
> 7fffe803c6d2 _ZL26poppler_page_get_text_pageP12_PopplerPage+0x92 
> (/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0)
> 7fffe803deb3 poppler_page_get_selection_region+0x63 
> (/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0)
> 7fffe82ab650 [unknown] 
> (/opt/evince-3.28.4/lib/evince/4/backends/libpdfdocument.so)
> 7ffff795f165 ev_job_page_data_run+0x2f5 
> (/opt/evince-3.28.4/lib/libevview3.so.3.0.0)
> 7ffff7961309 ev_job_thread+0xe9 (inlined)
> 7ffff7961309 ev_job_thread_proxy+0xe9 
> (/opt/evince-3.28.4/lib/libevview3.so.3.0.0)
> 7ffff5492194 g_thread_proxy+0x54 
> (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4)
> 7ffff4e686da start_thread+0xda 
> (/lib/x86_64-linux-gnu/libpthread-2.27.so)
> 7ffff4b9188e __GI___clone+0x3e (inlined)
> 
> I am looking forward to any help or suggestions.
> Regards.
> 
> On 2020-04-04 05:31, ahmadkhorrami wrote:
> 
> Hi,
> I think that I should give up and assume that all illogical repeated 
> addresses are inlines. Thanks everybody, especially, Jirka, Milian, 
> Arnaldo and Steven. Perhaps some day I will be back and try to dig 
> deeper (with your hints and assistance).
> Regards.
> 
> On 2020-04-01 01:27, ahmadkhorrami wrote:
> 
> Hi,
> On 2020-03-31 19:59, Milian Wolff wrote:
> 
> On Dienstag, 31. März 2020 17:02:37 CEST ahmadkhorrami wrote:
> 
> Hi Milian,
> Thanks for the detailed answer. Well, the bug you mentioned is bad 
> news.
> Because I sample using uppp. Perhaps this leads to these weird traces.
> Please read the full thread from here on:
> 
> https://lkml.org/lkml/2018/11/2/86
> 
> But as I said - it should be easy to check if this is really the issue 
> are
> running into or not: Try to see if you see the problem when you sample 
> without
> `ppp`. If not, then you can be pretty sure it's this issue. If you 
> still see
> it, then it's something different. Well, the problem exists even when 
> ":uppp" is changed to ":u".
> 
> Is this a purely software bug?
> I wouldn't call it that, personally. Rather, it's a limitation in the 
> hardware
> and software. We would need something completely different to "fix" 
> this, i.e.
> something like a deeper LBR. That's btw another alternative you could 
> try:
> `perf record --call-graph lbr` and live with the short call stacks. But 
> at
> least these should be correct (afaik). For me personally they are 
> always far
> too short and thus not practical to use in reality.
> 
> Cheers
    Regards.

      reply index

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <157597d74ff17f781d9de7e7e3defd13@ut.ac.ir>
2020-03-22 20:24 ` ahmadkhorrami
2020-03-23  0:34   ` Steven Rostedt
     [not found]     ` <21b3df4080709f193d62b159887e2a83@ut.ac.ir>
2020-03-23  8:49       ` Jiri Olsa
2020-03-23 10:03         ` ahmadkhorrami
2020-03-25 15:18           ` ahmadkhorrami
2020-03-25 15:46             ` Jiri Olsa
2020-03-25 18:54               ` ahmadkhorrami
2020-03-25 18:58               ` Arnaldo Carvalho de Melo
2020-03-25 19:10                 ` ahmadkhorrami
2020-03-25 19:28                   ` Arnaldo Carvalho de Melo
2020-03-25 20:01                     ` ahmadkhorrami
2020-03-25 20:39                       ` Jiri Olsa
2020-03-25 21:02                         ` Jiri Olsa
2020-03-25 21:09                           ` Steven Rostedt
2020-03-25 21:37                             ` ahmadkhorrami
2020-03-25 21:46                               ` Jiri Olsa
2020-03-25 22:21                                 ` ahmadkhorrami
2020-03-25 23:09                                   ` ahmadkhorrami
2020-03-26  9:59                                     ` Jiri Olsa
2020-03-26 13:20                                       ` ahmadkhorrami
2020-03-26 15:39                                         ` Jiri Olsa
2020-03-26 18:19                                           ` ahmadkhorrami
2020-03-26 18:21                                             ` ahmadkhorrami
2020-03-27  9:20                                             ` Jiri Olsa
2020-03-27 10:59                                               ` ahmadkhorrami
2020-03-27 11:04                                                 ` ahmadkhorrami
2020-03-27 12:10                                                   ` Milian Wolff
2020-03-27 12:58                                                     ` ahmadkhorrami
2020-03-27 13:25                                                       ` Milian Wolff
2020-03-27 13:33                                                         ` ahmadkhorrami
2020-03-27 18:43                                                   ` ahmadkhorrami
2020-03-27 22:37                                                     ` Jiri Olsa
2020-03-27 23:12                                                       ` ahmadkhorrami
2020-03-28 23:34                                                         ` Jiri Olsa
2020-03-29  0:43                                                           ` ahmadkhorrami
2020-03-29  1:16                                                             ` ahmadkhorrami
2020-03-29 11:19                                                               ` Jiri Olsa
2020-03-29 11:52                                                                 ` ahmadkhorrami
2020-03-29 12:08                                                                   ` Jiri Olsa
2020-03-29 12:39                                                                     ` ahmadkhorrami
2020-03-29 13:50                                                                       ` Milian Wolff
2020-03-29 14:23                                                                         ` ahmadkhorrami
2020-03-29 19:20                                                                         ` Jiri Olsa
2020-03-30  6:09                                                                           ` Milian Wolff
2020-03-30 13:07                                                                             ` Jiri Olsa
2020-03-30 13:49                                                                               ` ahmadkhorrami
2020-03-30 19:05                                                                                 ` ahmadkhorrami
2020-03-30 21:05                                                                                   ` debuginfod-based dwarf downloading, was " Frank Ch. Eigler
2020-03-31  9:26                                                                                     ` Jiri Olsa
2020-03-31 14:00                                                                                       ` Frank Ch. Eigler
2020-03-31  4:43                                                                                   ` ahmadkhorrami
2020-03-31  9:30                                                                                     ` Jiri Olsa
2020-03-31 11:53                                                                                       ` ahmadkhorrami
2020-03-31 12:43                                                                                   ` ahmadkhorrami
2020-03-31 13:20                                                                                     ` Jiri Olsa
2020-03-31 13:39                                                                                       ` ahmadkhorrami
2020-03-31 14:44                                                                                         ` Milian Wolff
2020-03-31 15:02                                                                                           ` ahmadkhorrami
2020-03-31 15:05                                                                                             ` ahmadkhorrami
2020-03-31 15:29                                                                                             ` Milian Wolff
2020-03-31 16:10                                                                                               ` Arnaldo Carvalho de Melo
2020-03-31 19:20                                                                                                 ` ahmadkhorrami
2020-03-31 19:17                                                                                               ` ahmadkhorrami
2020-03-31 20:57                                                                                               ` ahmadkhorrami
2020-04-04  1:01                                                                                                 ` ahmadkhorrami
2020-04-11 16:42                                                                                                   ` ahmadkhorrami
2020-04-11 21:04                                                                                                     ` ahmadkhorrami [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f1b212e2b5035bbb2ccfb4e74f6f28cd@ut.ac.ir \
    --to=ahmadkhorrami@ut.ac.ir \
    --cc=acme@redhat.com \
    --cc=ak@linux.intel.com \
    --cc=jolsa@redhat.com \
    --cc=linux-trace-users-owner@vger.kernel.org \
    --cc=linux-trace-users@vger.kernel.org \
    --cc=milian.wolff@kdab.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=yao.jin@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Trace-Users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \
		linux-trace-users@vger.kernel.org
	public-inbox-index linux-trace-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git