From: Milian Wolff <milian.wolff@kdab.com>
To: Milian Wolff <mail@milianw.de>
Cc: linux-kernel@vger.kernel.org, Jiri Olsa <jolsa@kernel.org>,
namhyung@kernel.org, linux-perf-users@vger.kernel.org,
Arnaldo Carvalho <acme@kernel.org>
Subject: Re: Broken dwarf unwinding - wrong stack pointer register value?
Date: Sun, 21 Oct 2018 22:32:10 +0200 [thread overview]
Message-ID: <1759553.TYpvCCURxk@agathebauer> (raw)
In-Reply-To: <2335309.gnWok9HYb4@agathebauer>
[-- Attachment #1: Type: text/plain, Size: 2408 bytes --]
On Sonntag, 21. Oktober 2018 00:39:51 CEST Milian Wolff wrote:
> Hey all,
>
> I'm on the quest to figure out why perf regularly fails to unwind (some)
> samples. I am seeing very strange behavior, where an apparently wrong stack
> pointer value is read from the register - see below for more information and
> the end of this (long) mail for my open questions. Any help would be
> greatly appreciated.
>
> I am currently using this trivial C++ code to reproduce the issue:
>
> ```
> #include <cmath>
> #include <complex>
> #include <iostream>
> #include <random>
>
> using namespace std;
>
> int main()
> {
> uniform_real_distribution<double> uniform(-1E5, 1E5);
> default_random_engine engine;
> double s = 0;
> for (int i = 0; i < 10000000; ++i) {
> s += norm(complex<double>(uniform(engine), uniform(engine)));
> }
> cout << s << '\n';
> return 0;
> }
> ```
>
> I compile it with `g++ -O2 -g` and then record it with `perf record --call-
> graph dwarf`. Using perf script, I then see e.g.:
With my patch to regularly flush the perf script output buffer, we can now
easily find all broken backtraces and the corresponding debug output via:
$ perf script --ns -v |& awk -v RS='' '/\[unknown\]/ {print "\n"$0}'
I've pasted the output to the above command from my machine here:
https://paste.kde.org/pmyxwkk1k
This contains 139 samples with broken unwinding, out of 2350 samples in total,
so about 6% of all samples are broken.
In many cases, the first accessed memory is 0 because a too-low offset into
the stack is computed from the SP value, similar to the scenario I described
in my initial mail. In other cases we read garbadge addresses such as
unwind: access_mem addr 0x7ffc80811cf0 val 408195dfbda90580, offset 24
In all cases except for the the two samples at the very start and end of this
log, the last offset encountered in access_mem is lower than 72. Remember what
I wrote in the initial mail - if I manually hack the access_mem function to
use 72 for one of the broken samples, it made unwinding magically work
again...
With this addition of data - can anyone sched some light on what's potentially
going on here? How can we improve this situation?
Thanks
--
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]
next prev parent reply other threads:[~2018-10-21 20:32 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-20 22:39 Broken dwarf unwinding - wrong stack pointer register value? Milian Wolff
2018-10-21 20:32 ` Milian Wolff [this message]
2018-10-22 10:35 ` Milian Wolff
2018-10-22 11:17 ` Milian Wolff
2018-10-22 13:58 ` Andi Kleen
2018-10-22 19:26 ` Milian Wolff
2018-10-23 4:03 ` Andi Kleen
2018-10-23 10:34 ` Milian Wolff
2018-10-24 14:48 ` Andi Kleen
2018-10-30 22:34 ` Milian Wolff
2018-11-01 22:08 ` PEBS level 2/3 breaks dwarf unwinding! [WAS: Re: Broken dwarf unwinding - wrong stack pointer register value?] Milian Wolff
2018-11-02 11:26 ` Jiri Olsa
2018-11-02 17:56 ` Milian Wolff
2018-11-05 20:51 ` Jiri Olsa
2018-11-05 22:54 ` Milian Wolff
2018-11-06 0:10 ` Andi Kleen
2018-11-06 8:39 ` Jiri Olsa
2018-11-06 17:26 ` Andi Kleen
2018-11-06 20:04 ` Milian Wolff
2018-11-06 20:24 ` Andi Kleen
2018-11-07 22:41 ` Milian Wolff
2018-11-08 12:41 ` Milian Wolff
2018-11-09 0:55 ` Andi Kleen
2018-11-09 0:54 ` Andi Kleen
2018-11-10 21:42 ` Travis Downs
2018-11-11 1:07 ` Andi Kleen
[not found] ` <CAOBGo4zirLiKX8VcROAE=kAD0+qkF0E-cBv9DtBiQr=_obDv5w@mail.gmail.com>
2018-11-11 2:54 ` Travis Downs
2018-11-12 3:26 ` Andi Kleen
2018-11-14 13:20 ` Milian Wolff
2018-11-15 2:05 ` Travis Downs
2018-11-15 9:10 ` Milian Wolff
2018-11-15 19:00 ` Andi Kleen
2018-11-15 2:15 ` Travis Downs
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1759553.TYpvCCURxk@agathebauer \
--to=milian.wolff@kdab.com \
--cc=acme@kernel.org \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mail@milianw.de \
--cc=namhyung@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).