From: ahmadkhorrami <email@example.com> To: Milian Wolff <firstname.lastname@example.org> Cc: Ian Rogers <email@example.com>, Steven Rostedt <firstname.lastname@example.org>, Linux-trace Users <email@example.com>, Arnaldo Carvalho de Melo <firstname.lastname@example.org>, linux-perf-users <email@example.com>, firstname.lastname@example.org Subject: Re: Sp! Re: Perf Script Erroneous User Stack Trace Date: Tue, 16 Jun 2020 21:36:43 +0430 Message-ID: <email@example.com> (raw) In-Reply-To: <5121466.Fqp1Q3JDDc@agathebauer> Hi Milian, Firstly, thanks for the detailed answer! But, it seems that you missed the first thread. There, I said that, I manually used perf_event_open() which dumps raw IPs (which are not unwound). But those raw IPs, themselves were erroneous. Many callchains were single element ones, containing only the innermost RIP, while many others were truncated and not rooted at "_start" or "__GI___clone()". Regards. On 2020-06-16 20:50, Milian Wolff wrote: > On Dienstag, 16. Juni 2020 16:37:08 CEST ahmadkhorrami wrote: > >> Hi, >> The problem doesn't seems to be caused by the assembly code. I checked >> the execution in GDB, and put breakpoint at >> "x264_pixel_avg_w16_avx2+0x4" and ignored with 10, 100, 1000, 10000 >> and >> 100000 occurrences. But, in all cases, GDB, successfully, displayed >> the >> whole backtrace. One of them is as follows: > > <snip> > >> It seems that, there should be something wrong with the kernel-side >> implementation. Could anybody point me to the kernel implementation? I >> think it is dumped, here: >> https://github.com/torvalds/linux/blob/master/kernel/events/core.c#L6786 >> >> But, I do not know where in the kernel, the user call-stack is >> generated. >> Any guesses? > > The kernel does not unwind the user call-stack when you use `perf > record -- > call-graph dwarf`. This is all done in user space at `perf report` > time. The > kernel only copies parts of the stack, in your case 64KB. > > There are tons of ways that can lead to broken unwinding. To figure out > more, > you'll have to dive into `perf report` and try to come with some ideas > yourself: > > a) try to figure out how unwinding should work for that library, does > it have > .eh_frame or does it need debug information for unwinding? If the > latter - run > `strace -e file -f perf script` and check if the separate debug > information > files are found and loaded by perf. > b) run `perf script -v` and inspect the log for your first broken > sample - is > there anything in it that may indicate the reason for the issue? > c) try elfutils instead of libunwind for unwinding, does that make a > difference? > d) dive even deeper into the code to see where and why it fails, > potentially > even within libunwind > > Note that GDB uses a completely different unwinder than perf. Libunwind > is > pretty good, but GDB has even better fallbacks to figure out > backtraces. I > mean it often even works after (partial) stack corruption there ;-) So > just > saying "it works in GDB" doesn't help us too much... > > Good luck!
next prev parent reply index Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-06-14 13:43 ahmadkhorrami 2020-06-15 20:31 ` Steven Rostedt 2020-06-15 20:53 ` Ian Rogers 2020-06-16 10:38 ` Sp! " ahmadkhorrami 2020-06-16 14:37 ` ahmadkhorrami 2020-06-16 16:20 ` Milian Wolff 2020-06-16 17:06 ` ahmadkhorrami [this message] 2020-06-16 17:42 ` ahmadkhorrami 2020-06-16 10:26 ` ahmadkhorrami
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-Trace-Users Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \ email@example.com public-inbox-index linux-trace-users Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users AGPL code for this site: git clone https://public-inbox.org/public-inbox.git