From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,PLING_QUERY, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59DD0C32789 for ; Thu, 8 Nov 2018 12:42:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0225C20685 for ; Thu, 8 Nov 2018 12:42:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kdab.com header.i=@kdab.com header.b="xU9v3aYt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0225C20685 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kdab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726960AbeKHWRR (ORCPT ); Thu, 8 Nov 2018 17:17:17 -0500 Received: from mail.kdab.com ([176.9.126.58]:62470 "EHLO mail.kdab.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726571AbeKHWRR (ORCPT ); Thu, 8 Nov 2018 17:17:17 -0500 Authentication-Results: mail.kdab.com (amavisd-new); dkim=pass (1024-bit key) reason="pass (just generated, assumed good)" header.d=kdab.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kdab.com; h= content-type:content-type:mime-version:references:in-reply-to :organization:message-id:date:date:subject:subject:from:from; s= dkim; t=1541680914; x=1542544915; bh=URKLhhXryJtfzfaRMk3nwYFsslz rb6ATrlX7iqUpdJY=; b=xU9v3aYtnUB4PsmeHmDHxEFROOYqKRW/ZbT4jbdYvEv uarFos3hkNYwS+V115Df3IkrSgQfGJGHkwNa+4Z9eYeOc+Ad7+Klzq1j4bazuXga 8GYhVSNseFSp4u5DPfq/U5l1SX35yiPKVlPCNHy6cKETO/V89fL8RlhnicnBBZkg = X-Virus-Scanned: amavisd-new at kdab.com From: Milian Wolff To: Milian Wolff Cc: Andi Kleen , Jiri Olsa , linux-kernel@vger.kernel.org, Jiri Olsa , namhyung@kernel.org, linux-perf-users@vger.kernel.org, Arnaldo Carvalho Subject: Re: PEBS level 2/3 breaks dwarf unwinding! [WAS: Re: Broken dwarf unwinding - wrong stack pointer register value?] Date: Thu, 08 Nov 2018 13:41:50 +0100 Message-ID: <7105958.qy8IzDpmRc@agathebauer> Organization: KDAB In-Reply-To: <1584560.aXQ729uQp3@agathebauer> References: <2335309.gnWok9HYb4@agathebauer> <20181106202411.GU6218@tassilo.jf.intel.com> <1584560.aXQ729uQp3@agathebauer> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1543118.cLrsxmjIBU"; micalg="sha256"; protocol="application/pkcs7-signature" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --nextPart1543118.cLrsxmjIBU Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" On Mittwoch, 7. November 2018 23:41:31 CET Milian Wolff wrote: > On Dienstag, 6. November 2018 21:24:11 CET Andi Kleen wrote: > > > Where would I look for the source to change here? So far, I only > > > concentrated on the userspace side of perf in tools/perf. > > > > Kind of similar to > > > > a405bad5ad20 perf/x86: Add Haswell specific transaction flag reporting > > fdfbbd07e91f perf: Add generic transaction flags > > > > Report the original (not overwritten) regs->ip and regs->sp > > Thanks a lot Andi! With your help, I have managed to find the exact issue > for my scenario. Turns out, it really is "just" the instruction pointer > that is wrong. I.e. originally we have IP = 0x7feda32ca68c, but with PEBS > we correct that to IP = 7feda32ca688. The SP register value stays the same > according to my printk output. Using the original IP value, we can unwind > correctly since we point to the correct place in the .eh_frame section. The > PEBS IP points to a different position in the .eh_frame section, which is > "too early". > > That brings up some questions: > > - I noticed `perf record --intr-regs`, but the values recorded in the > perf.data file are always the same. I.e. comparing uregs and iregs, I always > see the same values printed by `perf script`. This smells like a bug to me, > but so far I haven't figured out why this happens... The reason seems to be that perf_event_output only takes one set of registers, which then gets handed down into perf_prepare_sample where it gets sampled. Thus if sample type has both PERF_SAMPLE_REGS_USER and PERF_SAMPLE_REGS_INTR set, then by design both will store the same values for user space samples. Can we change this, such that perf_event_output also takes a second set of registers (iregs) that get sampled for PERF_SAMPLE_REGS_INTR? I'm very new to real kernel development, what kind of ABI/API stability guarantees exist for something like "perf_event_output"? > - Independently, when I add a custom printk manually in `arch/x86/events/ > intel/ds.c` at the end of `setup_pebs_sample_data`, then I'm never seeing > any differences between SP in iregs/pebs/regs. Shouldn't it also be > recorded via PEBS? Or is it just chance that I'm never seeing any > difference in setup_pebs_sample_data between iregs->sp and regs->sp? The reason here seems to be that the registers stored in "pebs" are essentially the same as iregs for the setup for `perf record --call-graph dwarf`. The difference is the availability of `pebs->real_ip` which gets used on my system to fixup the IP. SP stays untouched and is thus only truly valid for the untouched IP (which is discarded currently - see above). > - Generally, how do we want to handle this bug? If `--intr-regs` would > actually record a different IP than stored in uregs in the perf.data file, > then we could use that as a fallback for unwinding, when it fails the first > time. Or should we always unwind from that IP? How do we mark the "actual" > frame/IP then, if that differs? > > Thanks -- Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts --nextPart1543118.cLrsxmjIBU Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDEIw ggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqGSIb3DQEBDAUAMIGFMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYD VQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09NT0RPIFJTQSBDZXJ0aWZpY2F0aW9u IEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDkyMzU5NTlaMIGXMQswCQYDVQQGEwJH QjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQK ExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RPIFJTQSBDbGllbnQgQXV0aGVudGlj YXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcxLqf2NHbZhGra0D00SOTq9bus3/mx gUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMqVaVUkbIr5SU0RDX/kSsKwer3H1pT /HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63OTTqRvaq8aWccm+KOMjTcE6s2mj6 RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDaElmOupyTf1Qib+cpukNJnQmwygjD 8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOCATwwggE4MB8GA1UdIwQYMBaAFLuv fgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+lmF86B89K3FIXsSLwDAOBgNVHQ8B Af8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAECjAIMAYGBFUdIAAwTAYDVR0fBEUw QzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2VydGlmaWNhdGlvbkF1 dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUFBzAChi9odHRwOi8vY3J0LmNvbW9k b2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Au Y29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKBKDRPPO5fVs6fl1bsj6JrF/bz9kkI BtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR809AgcYboW+rcTNZ/8u/Hv+GTrNI/ AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px+TbO+88G4zipA2psZaWeEykgzClZ lPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50YJafX8mnahjp3I2Y2mkjhk0xQfhNq bzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99p17bTbY7+1RTWBviN9YJzK8HxzOb XYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmiNOr84JHoy+fNLpfvYc/Q9EtFOI5I SYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2PMnXoc7mbrS22XUSeTwxCTP9bcmU dp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqith7DoJC91WJ8Lce3CVJqb1XWeKIJ8 4F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYRUM1cWcef20Wkyk5S/GFyyPJwG0fR 1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjCCBlQwggU8oAMCAQICEAf6KCF9+1doL2oE OTPysLwwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1h bmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0w OwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWls IENBMB4XDTE3MDUyMzAwMDAwMFoXDTIwMDUyMjIzNTk1OVowggFZMQswCQYDVQQGEwJTRTEPMA0G A1UEERMGNjgzIDMxMRIwEAYDVQQIEwlWYWVybWxhbmQxEDAOBgNVBAcTB0hhZ2ZvcnMxGDAWBgNV BAkTD05vcnJpbmdzIHZhZWcgMjEPMA0GA1UEEhMGQm94IDMwMSYwJAYDVQQKDB1LbGFyw6RsdmRh bGVucyBEYXRha29uc3VsdCBBQjEdMBsGA1UECxMUQSBLREFCIEdyb3VwIENvbXBhbnkxQzBBBgNV BAsMOklzc3VlZCB0aHJvdWdoIEtsYXLDpGx2ZGFsZW5zIERhdGFrb25zdWx0IEFCIEUtUEtJIE1h bmFnZXIxHzAdBgNVBAsTFkNvcnBvcmF0ZSBTZWN1cmUgRW1haWwxFTATBgNVBAMTDE1pbGlhbiBX b2xmZjEkMCIGCSqGSIb3DQEJARYVbWlsaWFuLndvbGZmQGtkYWIuY29tMIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAxrzfNBVvRbiAknuTBXuQnNm9sLIFLo0vbPB6kswk78A3tA++Zn5c lQUHhGlQq1cdYxagnUpqwvG3Sod15mPSOLkAPf/mabLN7p+lFbRaUP+97ZkTZtvb4BCC3osIEFI4 G393OSFWqc2qmIPE/SwSASbAA20Fcaa2M6P1lhOk/ttUh2jIurTPF0wUycIA7lBddrOgaOA8e2m6 iLTNHtlrfRbBaUX91D5ebY+UWmIjXSQ9+CtutMzBkwnF0rZKririvOkklg9VzEGNQVHrQfDF2s/U pOtmtuVSwElauGT/KALyCFuIrYC1pmaKH8S1xODJqiRaf6jH8E+KQzKjyM/ErwIDAQABo4IB1TCC AdEwHwYDVR0jBBgwFoAUgq9sjPjF/pZhfOgfPStxSF7Ei8AwHQYDVR0OBBYEFN+m99RtIuA1bSdw 6b1brOX7X3AJMA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUF BwMEBggrBgEFBQcDAjBGBgNVHSAEPzA9MDsGDCsGAQQBsjEBAgEDBTArMCkGCCsGAQUFBwIBFh1o dHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8EUzBRME+gTaBLhklodHRwOi8vY3Js LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWls Q0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKGSWh0dHA6Ly9jcnQuY29tb2RvY2Eu Y29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcnQwJAYI KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAgBgNVHREEGTAXgRVtaWxpYW4ud29s ZmZAa2RhYi5jb20wDQYJKoZIhvcNAQELBQADggEBABf47LSJADqH+ow9INv3QM1NC/qq2bjxGvsZ 68iD11VEUAFlsYfsVTgQqUirwPVTYenXtwVBELHZyywsui1JxL7HKQetLQegDDP/RyfjReVaWxhy 3OpuItsgLVbru9QVgPifnoBFPtfZcwjeJDmeSbLT8oj4Rd0KYBOIve7WKvsfNPsNwfbLwY2zILkE LjxZcVi2AwZHDyab+dzL/3YcLuJj1lSawBGn7ilpcdZydlv4aye51pD/MemLIYLcylt+ImrmjnTV y+QlAHRF3s5FE8yAr+W1MBD/1bKZCSgFt8VQoAlz3hiQh8QqZp4Zl8WuVL4+mP/mT6VDEWgq/0Bo cukxggJuMIICagIBATCBrDCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hl c3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNV BAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EC EAf6KCF9+1doL2oEOTPysLwwDQYJYIZIAWUDBAIBBQCggZMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgxMTA4MTI0MTUwWjAoBgkqhkiG9w0BCQ8xGzAZMAsGCWCG SAFlAwQBAjAKBggqhkiG9w0DBzAvBgkqhkiG9w0BCQQxIgQgJlP5JaEyDAQfz9ZzoYvAR3qpTJni mhITJxVxQdgzuRIwDQYJKoZIhvcNAQEBBQAEggEAc/agt+U3MTfQCLY1Z4vrIfmIScmTrbvzaGNM +/ryknfSwujWGXQXec+kTL7pSfmp3nGFmD51gHBFnXcOaIFRgYeN3jLRuFQHHOqHn/1I25OQq3Y9 5WA3dlcJaEUf5MoKk5b0COSP/yVZFGdNv2nqiH9bZsSaQ2vSJMBhdmgKumPgSrR48M/c0DiFkZCG kigg1+ZbLCuE/ElfC12FnfnqF3ottSyuSukok8H3uJ/Xb+T4GvpCzzA5tgo6LqWugHdLpCsef2lD NXaZRkOoouKkOUCP0k4CON+a2Eq5Ueof4Y9AnwgWUNkBR8KDdBFTDH2p8C9tEgqdtwUh09fON3QO tgAAAAAAAA== --nextPart1543118.cLrsxmjIBU--