From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 885C9C2BA2B for ; Sat, 11 Apr 2020 16:42:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5C1E720757 for ; Sat, 11 Apr 2020 16:42:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726107AbgDKQmd (ORCPT ); Sat, 11 Apr 2020 12:42:33 -0400 Received: from mail.ut.ac.ir ([80.66.177.10]:58810 "EHLO mail.ut.ac.ir" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726069AbgDKQmc (ORCPT ); Sat, 11 Apr 2020 12:42:32 -0400 Received: from localhost (localhost [127.0.0.1]) by mail.ut.ac.ir (Postfix) with ESMTP id 835741DA0D8; Sat, 11 Apr 2020 21:12:28 +0430 (+0430) Received: from mail.ut.ac.ir ([127.0.0.1]) by localhost (mail.ut.ac.ir [127.0.0.1]) (amavisd-new, port 10024) with LMTP id FcDTx9JJiHz4; Sat, 11 Apr 2020 21:12:26 +0430 (+0430) Received: from mail.ut.ac.ir (mail.ut.ac.ir [194.225.0.10]) by mail.ut.ac.ir (Postfix) with ESMTP id C049C1DAB0E; Sat, 11 Apr 2020 21:12:25 +0430 (+0430) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sat, 11 Apr 2020 21:12:25 +0430 From: ahmadkhorrami To: Milian Wolff Cc: Jiri Olsa , Steven Rostedt , Arnaldo Carvalho de Melo , Linux-trace Users , Peter Zijlstra , linux-trace-users-owner@vger.kernel.org, Jin Yao , Namhyung Kim , Andi Kleen Subject: Re: Wrong Perf Backtraces In-Reply-To: <9dc590447c61e044934bdad4127297b8@ut.ac.ir> References: <821540886fc57d7749edee585a50602f@ut.ac.ir> <8573002.CDJkKcVGEf@agathebauer> <7774721.NyiUUSuA9g@agathebauer> <9dc590447c61e044934bdad4127297b8@ut.ac.ir> Message-ID: X-Sender: ahmadkhorrami@ut.ac.ir User-Agent: Roundcube Webmail/1.3.6 Sender: linux-trace-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-users@vger.kernel.org Hi, I found the problem by building the libraries from source code. The "gmallocn()"s were inlines. So, perhaps it would be better if Perf had reported it as "(inlined)" as in the common case. The second problematic case (repeated "gtk_action_muxer_query_action()"s) was purely due to my lack of knowledge. Sorry for wasting your time, specially, Mr. Olsa, :D. It was inter-library tail-call elimination, which I did not know is possible (I thought tail-calls happen only inside each translation unit). In other words, the last operation in the callee at "gtk_action_muxer_query_action+0x6c" is a jump to the beginning of "gtk_action_muxer_query_action()". And the debug info for the dbg-packages (Ubuntu in my case), seems to be OK. I wonder why source lines were reported as 0. Perhaps, the unwinding library is bogus. Finally, I have a new (hopefully, unsilly) question, :D. In the following backtrace, __GI___libc_malloc+0x197 is reported as inline. While its address (i.e., 0x7ffff4b07207) does not match that of its parent function call point (i.e., gmalloc+0x59 which is 0x7fffd9872fb9). Is this logical? And, again, there are many such instances in the Perf output. EvJobScheduler 10021 8653.926478: 100 mem_load_uops_retired.l3_miss:uppp: 7fffd1062a00 5080022 N/A|SNP N/A|TLB N/A|LCK N/A 7ffff4b07207 tcache_get+0x197 (inlined) 7ffff4b07207 __GI___libc_malloc+0x197 (inlined) 7fffd9872fb9 gmalloc+0x59 (inlined) 7fffd9872fb9 gmallocn+0x59 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffd9872fb9 gmallocn+0x59 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffd9951e6f _ZN8TextLine8coalesceEP10UnicodeMap+0xff (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffd9952f82 _ZN9TextBlock8coalesceEP10UnicodeMapd+0x752 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffd995bc37 _ZN8TextPage8coalesceEbdb+0x1507 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffd995cb71 _ZN13TextOutputDev7endPageEv+0x31 (/usr/lib/x86_64-linux-gnu/libpoppler.so.73.0.0) 7fffe803c6d2 _ZL26poppler_page_get_text_pageP12_PopplerPage+0x92 (/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0) 7fffe803deb3 poppler_page_get_selection_region+0x63 (/usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8.9.0) 7fffe82ab650 [unknown] (/opt/evince-3.28.4/lib/evince/4/backends/libpdfdocument.so) 7ffff795f165 ev_job_page_data_run+0x2f5 (/opt/evince-3.28.4/lib/libevview3.so.3.0.0) 7ffff7961309 ev_job_thread+0xe9 (inlined) 7ffff7961309 ev_job_thread_proxy+0xe9 (/opt/evince-3.28.4/lib/libevview3.so.3.0.0) 7ffff5492194 g_thread_proxy+0x54 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5600.4) 7ffff4e686da start_thread+0xda (/lib/x86_64-linux-gnu/libpthread-2.27.so) 7ffff4b9188e __GI___clone+0x3e (inlined) I am looking forward to any help or suggestions. Regards. On 2020-04-04 05:31, ahmadkhorrami wrote: > Hi, > I think that I should give up and assume that all illogical repeated > addresses are inlines. Thanks everybody, especially, Jirka, Milian, > Arnaldo and Steven. Perhaps some day I will be back and try to dig > deeper (with your hints and assistance). > Regards. > > On 2020-04-01 01:27, ahmadkhorrami wrote: > >> Hi, >> On 2020-03-31 19:59, Milian Wolff wrote: >> >> On Dienstag, 31. März 2020 17:02:37 CEST ahmadkhorrami wrote: >> >> Hi Milian, >> Thanks for the detailed answer. Well, the bug you mentioned is bad >> news. >> Because I sample using uppp. Perhaps this leads to these weird traces. >> Please read the full thread from here on: >> >> https://lkml.org/lkml/2018/11/2/86 >> >> But as I said - it should be easy to check if this is really the issue >> are >> running into or not: Try to see if you see the problem when you sample >> without >> `ppp`. If not, then you can be pretty sure it's this issue. If you >> still see >> it, then it's something different. > Well, the problem exists even when ":uppp" is changed to ":u". > > Is this a purely software bug? > I wouldn't call it that, personally. Rather, it's a limitation in the > hardware > and software. We would need something completely different to "fix" > this, i.e. > something like a deeper LBR. That's btw another alternative you could > try: > `perf record --call-graph lbr` and live with the short call stacks. But > at > least these should be correct (afaik). For me personally they are > always far > too short and thus not practical to use in reality. > > Cheers Regards.