Re: Wrong Perf Backtraces

From: ahmadkhorrami <ahmadkhorrami@ut.ac.ir>
To: Jiri Olsa <jolsa@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>,
	Linux-trace Users <linux-trace-users@vger.kernel.org>,
	Arnaldo Carvalho de Melo <acme@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	linux-trace-users-owner@vger.kernel.org,
	Jin Yao <yao.jin@linux.intel.com>
Subject: Re: Wrong Perf Backtraces
Date: Wed, 25 Mar 2020 19:48:39 +0430	[thread overview]
Message-ID: <8645d3626b4714690925328ab00373d6@ut.ac.ir> (raw)
In-Reply-To: <ee43289d91fcae1c9ef5cc2429dc916b@ut.ac.ir>

Hi,

Could you give me some hints about where the actual problem takes place? 
Is the problem with "Perf" or the hardware part (i.e., "Hardware 
Performance Counters")? Can I revise the problem by simply modifying the 
code? How much work is needed?

Your suggestions will be appreciated, because your experience and 
knowledge in this area is much more.

Regards.

On 2020-03-23 14:33, ahmadkhorrami wrote:

> Hi,
> It seems that my previous e-mail is not sent, properly. So, here is a 
> link to the stackoverflow question:
> https://stackoverflow.com/questions/60766026/wrong-perf-backtraces
> 
> The perf is for Ubuntu 18.04 with the following "uname -a" output:
> Linux Ahmad-Laptop 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 
> 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> I also used a compiled Linux-5.4.7 kernel and its corresponding Perf 
> tool.
> 
> Regards.
> 
> On 2020-03-23 13:19, Jiri Olsa wrote:
> 
> On Mon, Mar 23, 2020 at 07:48:26AM +0430, ahmadkhorrami wrote:
> 
> Here is a link to the detailed question at Stackoverflow:
> 
> https://stackoverflow.com/questions/60766026/wrong-perf-backtraces
> hi,
> what perf version are you running?
> 
> jirka
> 
> I can copy it here, if needed.
> 
> Thanks
> 
> On 2020-03-23 05:04, Steven Rostedt wrote:
> 
> On Mon, 23 Mar 2020 00:54:01 +0430
> ahmadkhorrami <ahmadkhorrami@ut.ac.ir> wrote:
> 
> Hi,
> I used "Perf" to extract call graphs in an evince benchmark. The 
> command
> used is as follows:
> sudo perf record -d --call-graph dwarf -c 10000 e
> mem_load_uops_retired.l3_miss:uppp /opt/evince-3.28.4/bin/evince
> 
> I extracted the backtraces using "perf script" and found out that there
> are many corrupted backtrace instances. Some contained repeated 
> function
> calls, for example two consecutive gmallocn()s exactly at the same
> offsets. There are also some backtraces where the callers and callees 
> do
> not match.
> Could you show some examples of the backtraces you mention?
> 
> Note that that mappings are correct. In other words, each single line 
> of
> the reported backtraces is correct (i.e., addresses match with
> functions). But is seems that there are some function calls in the
> middle, which are missed by "Perf". Strangely, in all runs (and also
> with different sampling frequencies) the problem occurs exactly at the
> same place.
> 
> I am really confused and looking forward to any help. I can also send
> backtraces if needed.
> -- Steve