From: Milian Wolff <milian.wolff@kdab.com>
To: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>,
linux-kernel@vger.kernel.org, Jiri Olsa <jolsa@kernel.org>,
namhyung@kernel.org, linux-perf-users@vger.kernel.org,
Arnaldo Carvalho <acme@kernel.org>
Subject: Re: PEBS level 2/3 breaks dwarf unwinding! [WAS: Re: Broken dwarf unwinding - wrong stack pointer register value?]
Date: Fri, 02 Nov 2018 18:56:50 +0100 [thread overview]
Message-ID: <3227038.olIWmsCzzY@agathebauer> (raw)
In-Reply-To: <20181102112635.GD5458@krava>
[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]
On Freitag, 2. November 2018 12:26:35 CET Jiri Olsa wrote:
> On Thu, Nov 01, 2018 at 11:08:18PM +0100, Milian Wolff wrote:
> > On Dienstag, 30. Oktober 2018 23:34:35 CET Milian Wolff wrote:
> > > On Mittwoch, 24. Oktober 2018 16:48:18 CET Andi Kleen wrote:
> > > > > Can someone at least confirm whether unwinding from a function
> > > > > prologue
> > > > > via
> > > > > .eh_frame (but without .debug_frame) should actually be possible?
> > > >
> > > > Yes it should be possible. Asynchronous unwind tables should work
> > > > from any instruction.
> >
> > <snip>
> >
> > > We can find `7f91345bdaf8+1 = 7f91345bdaf9" at offset 16 (search for "f9
> > > da
> > > 5b 34 91 7f"). Using that address makes unwinding work for this sample.
> > > What could be the reason for this shift?
> >
> > I believe I have found the culprit: PEBS seems to be at fault here - i.e.
> > the RIP/RSP and the ustack dump of the sample simply don't fit together.
> >
> > Check this out:
> >
> > ```
> > $ for i in $(seq 10); do perf record -q -e "cycles:" --call-graph dwarf
> > ./cpp- inlining > /dev/null; perf script | pcre2grep -c -M
> > "hypot_finite.*\n.*\ [unknown\]"; done
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> >
> > $ for i in $(seq 10); do perf record -q -e "cycles:p" --call-graph dwarf
> > ./
> > cpp-inlining > /dev/null; perf script | pcre2grep -c -M
> > "hypot_finite.*\n.*\ [unknown\]"; done
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> > 0
> >
> > $ for i in $(seq 10); do perf record -q -e "cycles:pp" --call-graph dwarf
> > ./ cpp-inlining > /dev/null; perf script | pcre2grep -c -M
> > "hypot_finite.*\n.*\ [unknown\]"; done
> > 37
> > 39
> > 35
> > 28
> > 40
> > 39
> > 29
> > 37
> > 31
> > 26
> >
> > $ for i in $(seq 10); do perf record -q -e "cycles:ppp" --call-graph dwarf
> > ./ cpp-inlining > /dev/null; perf script | pcre2grep -c -M
> > "hypot_finite.*\n.*\ [unknown\]"; done
> > 79
> > 70
> > 76
> > 77
> > 70
> > 90
> > 64
> > 78
> > 86
> > 74
> > ```
> >
> > Note how precise levels 0 and 1 do not produce any samples where unwinding
> > fails. But precise level 2 produces some, and precise level 3 increases
> > the
> > amount (by ca. ~2x).
> >
> > I can reproduce this pattern on two separate Intel CPUs and kernel
> > versions
> > currently:
> >
> > Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz with 4.18.16-arch1-1-ARCH
> > Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 4.14.78-1-lts
> >
> > Could someone else try this? What about AMD and IBS - is it also affected?
> > What about newer/different Intel CPUs?
>
> I tried on intel and can't actualy see that.. how do the failed samples
> look like? like is there the stack dump attached, what's in the regs?
>
> could you please paste the 'perf report -D' output for some of the
> failed samples?
See here for one case: https://paste.kde.org/prryvdilq
What Intel CPU did you use? What microcode version? Which kernel version?
Generally, isn't what I'm seeing actually a neccessary evil of the ustack
based unwinding in perf? I mean, the general procedure is as follows if I'm
not mistaken:
- PMU triggers interrupt and PEBS stores RIP etc.
- code continous to execute, possibly changing the stack
- PMU interrupt is handled, and a perf sample is generated
- register values are used from "past" status stored in PEBS
- but ustack dump is based on the "current" status
From this latter discrepancy, it must directly follow that *sometimes* the
ustack dump represents a state that cannot be unwound from, no?
Cheers
--
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]
next prev parent reply other threads:[~2018-11-02 17:57 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-20 22:39 Broken dwarf unwinding - wrong stack pointer register value? Milian Wolff
2018-10-21 20:32 ` Milian Wolff
2018-10-22 10:35 ` Milian Wolff
2018-10-22 11:17 ` Milian Wolff
2018-10-22 13:58 ` Andi Kleen
2018-10-22 19:26 ` Milian Wolff
2018-10-23 4:03 ` Andi Kleen
2018-10-23 10:34 ` Milian Wolff
2018-10-24 14:48 ` Andi Kleen
2018-10-30 22:34 ` Milian Wolff
2018-11-01 22:08 ` PEBS level 2/3 breaks dwarf unwinding! [WAS: Re: Broken dwarf unwinding - wrong stack pointer register value?] Milian Wolff
2018-11-02 11:26 ` Jiri Olsa
2018-11-02 17:56 ` Milian Wolff [this message]
2018-11-05 20:51 ` Jiri Olsa
2018-11-05 22:54 ` Milian Wolff
2018-11-06 0:10 ` Andi Kleen
2018-11-06 8:39 ` Jiri Olsa
2018-11-06 17:26 ` Andi Kleen
2018-11-06 20:04 ` Milian Wolff
2018-11-06 20:24 ` Andi Kleen
2018-11-07 22:41 ` Milian Wolff
2018-11-08 12:41 ` Milian Wolff
2018-11-09 0:55 ` Andi Kleen
2018-11-09 0:54 ` Andi Kleen
2018-11-10 21:42 ` Travis Downs
2018-11-11 1:07 ` Andi Kleen
[not found] ` <CAOBGo4zirLiKX8VcROAE=kAD0+qkF0E-cBv9DtBiQr=_obDv5w@mail.gmail.com>
2018-11-11 2:54 ` Travis Downs
2018-11-12 3:26 ` Andi Kleen
2018-11-14 13:20 ` Milian Wolff
2018-11-15 2:05 ` Travis Downs
2018-11-15 9:10 ` Milian Wolff
2018-11-15 19:00 ` Andi Kleen
2018-11-15 2:15 ` Travis Downs
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3227038.olIWmsCzzY@agathebauer \
--to=milian.wolff@kdab.com \
--cc=acme@kernel.org \
--cc=ak@linux.intel.com \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=namhyung@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).