linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Milian Wolff <milian.wolff@kdab.com>
To: Milian Wolff <mail@milianw.de>
Cc: linux-kernel@vger.kernel.org, Jiri Olsa <jolsa@kernel.org>,
	namhyung@kernel.org, linux-perf-users@vger.kernel.org,
	Arnaldo Carvalho <acme@kernel.org>
Subject: Re: Broken dwarf unwinding - wrong stack pointer register value?
Date: Sun, 21 Oct 2018 22:32:10 +0200	[thread overview]
Message-ID: <1759553.TYpvCCURxk@agathebauer> (raw)
In-Reply-To: <2335309.gnWok9HYb4@agathebauer>

[-- Attachment #1: Type: text/plain, Size: 2408 bytes --]

On Sonntag, 21. Oktober 2018 00:39:51 CEST Milian Wolff wrote:
> Hey all,
> 
> I'm on the quest to figure out why perf regularly fails to unwind (some)
> samples. I am seeing very strange behavior, where an apparently wrong stack
> pointer value is read from the register - see below for more information and
> the end of this (long) mail for my open questions. Any help would be
> greatly appreciated.
> 
> I am currently using this trivial C++ code to reproduce the issue:
> 
> ```
> #include <cmath>
> #include <complex>
> #include <iostream>
> #include <random>
> 
> using namespace std;
> 
> int main()
> {
>     uniform_real_distribution<double> uniform(-1E5, 1E5);
>     default_random_engine engine;
>     double s = 0;
>     for (int i = 0; i < 10000000; ++i) {
>         s += norm(complex<double>(uniform(engine), uniform(engine)));
>     }
>     cout << s << '\n';
>     return 0;
> }
> ```
> 
> I compile it with `g++ -O2 -g` and then record it with `perf record --call-
> graph dwarf`. Using perf script, I then see e.g.:

With my patch to regularly flush the perf script output buffer, we can now 
easily find all broken backtraces and the corresponding debug output via:

    $ perf script --ns -v |& awk -v RS='' '/\[unknown\]/ {print "\n"$0}'

I've pasted the output to the above command from my machine here:
https://paste.kde.org/pmyxwkk1k

This contains 139 samples with broken unwinding, out of 2350 samples in total, 
so about 6% of all samples are broken.

In many cases, the first accessed memory is 0 because a too-low offset into 
the stack is computed from the SP value, similar to the scenario I described 
in my initial mail. In other cases we read garbadge addresses such as 

unwind: access_mem addr 0x7ffc80811cf0 val 408195dfbda90580, offset 24

In all cases except for the the two samples at the very start and end of this 
log, the last offset encountered in access_mem is lower than 72. Remember what 
I wrote in the initial mail - if I manually hack the access_mem function to 
use 72 for one of the broken samples, it made unwinding magically work 
again...

With this addition of data - can anyone sched some light on what's potentially 
going on here? How can we improve this situation?

Thanks
-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]

  reply	other threads:[~2018-10-21 20:32 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-20 22:39 Broken dwarf unwinding - wrong stack pointer register value? Milian Wolff
2018-10-21 20:32 ` Milian Wolff [this message]
2018-10-22 10:35 ` Milian Wolff
2018-10-22 11:17   ` Milian Wolff
2018-10-22 13:58     ` Andi Kleen
2018-10-22 19:26       ` Milian Wolff
2018-10-23  4:03         ` Andi Kleen
2018-10-23 10:34           ` Milian Wolff
2018-10-24 14:48             ` Andi Kleen
2018-10-30 22:34               ` Milian Wolff
2018-11-01 22:08                 ` PEBS level 2/3 breaks dwarf unwinding! [WAS: Re: Broken dwarf unwinding - wrong stack pointer register value?] Milian Wolff
2018-11-02 11:26                   ` Jiri Olsa
2018-11-02 17:56                     ` Milian Wolff
2018-11-05 20:51                       ` Jiri Olsa
2018-11-05 22:54                         ` Milian Wolff
2018-11-06  0:10                           ` Andi Kleen
2018-11-06  8:39                             ` Jiri Olsa
2018-11-06 17:26                               ` Andi Kleen
2018-11-06 20:04                               ` Milian Wolff
2018-11-06 20:24                                 ` Andi Kleen
2018-11-07 22:41                                   ` Milian Wolff
2018-11-08 12:41                                     ` Milian Wolff
2018-11-09  0:55                                       ` Andi Kleen
2018-11-09  0:54                                     ` Andi Kleen
2018-11-10 21:42                             ` Travis Downs
2018-11-11  1:07                               ` Andi Kleen
     [not found]                                 ` <CAOBGo4zirLiKX8VcROAE=kAD0+qkF0E-cBv9DtBiQr=_obDv5w@mail.gmail.com>
2018-11-11  2:54                                   ` Travis Downs
2018-11-12  3:26                                   ` Andi Kleen
2018-11-14 13:20                                     ` Milian Wolff
2018-11-15  2:05                                       ` Travis Downs
2018-11-15  9:10                                         ` Milian Wolff
2018-11-15 19:00                                           ` Andi Kleen
2018-11-15  2:15                                     ` Travis Downs

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1759553.TYpvCCURxk@agathebauer \
    --to=milian.wolff@kdab.com \
    --cc=acme@kernel.org \
    --cc=jolsa@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=mail@milianw.de \
    --cc=namhyung@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).