* Cross platform perf reporting
@ 2016-08-12 12:22 Milian Wolff
2016-08-12 14:38 ` David Ahern
2016-08-12 15:10 ` Milian Wolff
0 siblings, 2 replies; 8+ messages in thread
From: Milian Wolff @ 2016-08-12 12:22 UTC (permalink / raw)
To: linux-perf-users
[-- Attachment #1: Type: text/plain, Size: 3147 bytes --]
Hey all,
I'm trying to analyze perf.data files that I recorded on one machine (usually
arm or arm64) on my development machine (x86_64). I have compiled a new
version of perf (and the corresponding dependencies, where needed) to do this.
Now I do:
~~~~~~~~~~~~~~~~~~~~~~~~
aarch64$ perf record -g ...
aarch64$ perf archive
x86_64$ scp aarch64:perf.data* .
x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
x86_64$ perf report |& less
-> lots of errors, none of which are displayed in the TUI, such as:
unwind: target platform=arm64 is not supported
(often repeated)
Failed to open /lib/ld-2.21.so, continuing without symbols
Failed to open /lib/libc-2.21.so, continuing without symbols
Failed to open /lib/libpthread-2.21.so, continuing without symbols
Failed to open /lib/libdl-2.21.so, continuing without symbols
...
~~~~~~~~~~~~~~~~~~~~~~~~
If I try to run `perf report` on the device, I actually see resolved symbol
names and backtraces, e.g.:
~~~~~~~~~~~~~~~~~~~~~~~~
aarch64$ perf report
...
62.93% 0.00% QQmlThread libglib-2.0.so.0.4200.1
[.] g_main_context_iteration
|
---g_main_context_iteration
|
--62.92%--0x55ae0
g_main_context_dispatch
0x322dc4
QCoreApplicationPrivate::sendPostedEvents
|
--62.91%--QCoreApplication::notifyInternal2
QCoreApplication::notify
QCoreApplicationPrivate::notify_helper
0x2decc8
|
--62.89%--0x2de2b4
...
~~~~~~~~~~~~~~~~~~~~~~~~
But if I do the same on my development machine, after extracting the perf
archive, I only see:
~~~~~~~~~~~~~~~~~~~~~~~~
62.93% 0.00% QQmlThread [unknown]
[.] 0x0000007fa58c0eac
|
---0x7fa58c0eac
|
--62.93%--0x7fa58c14a4
|
--62.92%--0x7fa56c43bc
0x7fa5e55dfc
0x7fa56c9be8
0x6e48
0xc7e90
~~~~~~~~~~~~~~~~~~~~~~~~
So my questions:
- How do I get an arm / aarch64 unwinder to work on x86_64? Do I need to
compile libunwind in a special way?
- Is the unwinder what is missing to get the symbols resolved in the report?
Note that e.g. the "libglib" is not reported as "Failed to open", thus I
wonder why I'm not seeing the "g_main_context_iteration" symbol from it in the
report. It seems as if the mapping between address and SO is broken?
Any hints on how to improve that situation would be welcome.
Thanks
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-12 12:22 Cross platform perf reporting Milian Wolff
@ 2016-08-12 14:38 ` David Ahern
2016-08-12 15:10 ` Milian Wolff
1 sibling, 0 replies; 8+ messages in thread
From: David Ahern @ 2016-08-12 14:38 UTC (permalink / raw)
To: Milian Wolff, linux-perf-users
On 8/12/16 6:22 AM, Milian Wolff wrote:
> Hey all,
>
> I'm trying to analyze perf.data files that I recorded on one machine (usually
> arm or arm64) on my development machine (x86_64). I have compiled a new
> version of perf (and the corresponding dependencies, where needed) to do this.
>
> Now I do:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~
> aarch64$ perf record -g ...
> aarch64$ perf archive
>
> x86_64$ scp aarch64:perf.data* .
> x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
> x86_64$ perf report |& less
> -> lots of errors, none of which are displayed in the TUI, such as:
>
> unwind: target platform=arm64 is not supported
> (often repeated)
> Failed to open /lib/ld-2.21.so, continuing without symbols
> Failed to open /lib/libc-2.21.so, continuing without symbols
> Failed to open /lib/libpthread-2.21.so, continuing without symbols
> Failed to open /lib/libdl-2.21.so, continuing without symbols
> ...
> ~~~~~~~~~~~~~~~~~~~~~~~~
>
> If I try to run `perf report` on the device, I actually see resolved symbol
> names and backtraces, e.g.:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~
> aarch64$ perf report
> ...
> 62.93% 0.00% QQmlThread libglib-2.0.so.0.4200.1
> [.] g_main_context_iteration
> |
> ---g_main_context_iteration
> |
> --62.92%--0x55ae0
> g_main_context_dispatch
> 0x322dc4
> QCoreApplicationPrivate::sendPostedEvents
> |
> --62.91%--QCoreApplication::notifyInternal2
> QCoreApplication::notify
> QCoreApplicationPrivate::notify_helper
> 0x2decc8
> |
> --62.89%--0x2de2b4
> ...
> ~~~~~~~~~~~~~~~~~~~~~~~~
>
> But if I do the same on my development machine, after extracting the perf
> archive, I only see:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~
> 62.93% 0.00% QQmlThread [unknown]
> [.] 0x0000007fa58c0eac
> |
> ---0x7fa58c0eac
> |
> --62.93%--0x7fa58c14a4
> |
> --62.92%--0x7fa56c43bc
> 0x7fa5e55dfc
> 0x7fa56c9be8
> 0x6e48
> 0xc7e90
> ~~~~~~~~~~~~~~~~~~~~~~~~
>
> So my questions:
>
> - How do I get an arm / aarch64 unwinder to work on x86_64? Do I need to
> compile libunwind in a special way?
>
> - Is the unwinder what is missing to get the symbols resolved in the report?
> Note that e.g. the "libglib" is not reported as "Failed to open", thus I
> wonder why I'm not seeing the "g_main_context_iteration" symbol from it in the
> report. It seems as if the mapping between address and SO is broken?
>
> Any hints on how to improve that situation would be welcome.
The symfs option was added for this use case. If the dev system does not have the exact same binaries (build id match) perf will not load the symbols. You can create a sysroot that mirrors the runtime environment and then point perf to it using --symfs <path>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-12 12:22 Cross platform perf reporting Milian Wolff
2016-08-12 14:38 ` David Ahern
@ 2016-08-12 15:10 ` Milian Wolff
2016-08-12 15:31 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 8+ messages in thread
From: Milian Wolff @ 2016-08-12 15:10 UTC (permalink / raw)
To: linux-perf-users
[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]
On Friday, August 12, 2016 2:22:29 PM CEST Milian Wolff wrote:
> Hey all,
>
> I'm trying to analyze perf.data files that I recorded on one machine
> (usually arm or arm64) on my development machine (x86_64). I have compiled
> a new version of perf (and the corresponding dependencies, where needed) to
> do this.
>
> Now I do:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~
> aarch64$ perf record -g ...
> aarch64$ perf archive
>
> x86_64$ scp aarch64:perf.data* .
> x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
> x86_64$ perf report |& less
> -> lots of errors, none of which are displayed in the TUI, such as:
>
> unwind: target platform=arm64 is not supported
> (often repeated)
> Failed to open /lib/ld-2.21.so, continuing without symbols
> Failed to open /lib/libc-2.21.so, continuing without symbols
> Failed to open /lib/libpthread-2.21.so, continuing without symbols
> Failed to open /lib/libdl-2.21.so, continuing without symbols
> ...
> ~~~~~~~~~~~~~~~~~~~~~~~~
What I notice, is that `perf buildid-list` does not contain any of these
files. I also don't have the corresponding files in my target's ~/.debug
folder, where everything seems to work as expected. Is that, because there the
symbols are read directly from the file and nothing is cached in ~/.debug?
Can I set an environment variable to let perf know to look in secondary paths
for the libraries? I.e. I hope to make it look into the cross compile sysroot
for these libraries, instead of literally interpreting the path as a valid
path on my development machine.
Thanks
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-12 15:10 ` Milian Wolff
@ 2016-08-12 15:31 ` Arnaldo Carvalho de Melo
2016-08-15 12:20 ` Milian Wolff
0 siblings, 1 reply; 8+ messages in thread
From: Arnaldo Carvalho de Melo @ 2016-08-12 15:31 UTC (permalink / raw)
To: Milian Wolff; +Cc: linux-perf-users, David Ahern, He Kuang
Em Fri, Aug 12, 2016 at 05:10:18PM +0200, Milian Wolff escreveu:
> On Friday, August 12, 2016 2:22:29 PM CEST Milian Wolff wrote:
> > I'm trying to analyze perf.data files that I recorded on one machine
> > (usually arm or arm64) on my development machine (x86_64). I have compiled
> > a new version of perf (and the corresponding dependencies, where needed) to
> > do this.
Adding He Kuang, that did work on this area recently.
> > Now I do:
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~
> > aarch64$ perf record -g ...
> > aarch64$ perf archive
> >
> > x86_64$ scp aarch64:perf.data* .
> > x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
> > x86_64$ perf report |& less
> > -> lots of errors, none of which are displayed in the TUI, such as:
> >
> > unwind: target platform=arm64 is not supported
> > (often repeated)
> > Failed to open /lib/ld-2.21.so, continuing without symbols
> > Failed to open /lib/libc-2.21.so, continuing without symbols
> > Failed to open /lib/libpthread-2.21.so, continuing without symbols
> > Failed to open /lib/libdl-2.21.so, continuing without symbols
> > ...
> > ~~~~~~~~~~~~~~~~~~~~~~~~
>
> What I notice, is that `perf buildid-list` does not contain any of these
That is a limitation that needs fixing, what gets into the build-id are
entries that had hits, DSOs, but those are sample hits, not callchain
hits.
We would have to traverse all callchains and add build-id entries for
the DSOs in it.
I need to do this :-\
But if you have a copy of your target in a file and use --symfs with it it
should work, because then, because it doesn't have the build-id for the DSOs in
the callchains that didn't had hits, it will resort to using just the pathname
when looking for a symtab or the whole ELF for annotation, etc.
> files. I also don't have the corresponding files in my target's ~/.debug
> folder, where everything seems to work as expected. Is that, because there the
> symbols are read directly from the file and nothing is cached in ~/.debug?
That is a possibility, yes. But it should be adding it to ~/.debug, that
is the default, unless you have something disabling that in your
~/.perfconfig or some sort of problem is happening that is preventing it
to do the hardlink/copy.
> Can I set an environment variable to let perf know to look in secondary paths
> for the libraries? I.e. I hope to make it look into the cross compile sysroot
> for these libraries, instead of literally interpreting the path as a valid
> path on my development machine.
That is the --symfs David talked about, no?
[acme@jouet linux]$ perf report -h symfs
Usage: perf report [<options>]
--symfs <directory>
Look for files with symbols relative to this directory
[acme@jouet linux]$
> Thanks
>
> --
> Milian Wolff | milian.wolff@kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-12 15:31 ` Arnaldo Carvalho de Melo
@ 2016-08-15 12:20 ` Milian Wolff
2016-08-15 14:42 ` David Ahern
0 siblings, 1 reply; 8+ messages in thread
From: Milian Wolff @ 2016-08-15 12:20 UTC (permalink / raw)
To: acme; +Cc: linux-perf-users, dsahern, hekuang
[-- Attachment #1: Type: text/plain, Size: 13417 bytes --]
On Friday, August 12, 2016 12:31:25 PM CEST Arnaldo Carvalho de Melo wrote:
> Em Fri, Aug 12, 2016 at 05:10:18PM +0200, Milian Wolff escreveu:
> > On Friday, August 12, 2016 2:22:29 PM CEST Milian Wolff wrote:
> > > I'm trying to analyze perf.data files that I recorded on one machine
> > > (usually arm or arm64) on my development machine (x86_64). I have
> > > compiled
> > > a new version of perf (and the corresponding dependencies, where needed)
> > > to
> > > do this.
>
> Adding He Kuang, that did work on this area recently.
>
> > > Now I do:
> > >
> > > ~~~~~~~~~~~~~~~~~~~~~~~~
> > > aarch64$ perf record -g ...
> > > aarch64$ perf archive
> > >
> > > x86_64$ scp aarch64:perf.data* .
> > > x86_64$ tar xvf perf.data.tar.bz2 -C ~/.debug
> > > x86_64$ perf report |& less
> > > -> lots of errors, none of which are displayed in the TUI, such as:
> > >
> > > unwind: target platform=arm64 is not supported
> > > (often repeated)
> > > Failed to open /lib/ld-2.21.so, continuing without symbols
> > > Failed to open /lib/libc-2.21.so, continuing without symbols
> > > Failed to open /lib/libpthread-2.21.so, continuing without symbols
> > > Failed to open /lib/libdl-2.21.so, continuing without symbols
> > > ...
> > > ~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > What I notice, is that `perf buildid-list` does not contain any of these
>
> That is a limitation that needs fixing, what gets into the build-id are
> entries that had hits, DSOs, but those are sample hits, not callchain
> hits.
>
> We would have to traverse all callchains and add build-id entries for
> the DSOs in it.
>
> I need to do this :-\
>
> But if you have a copy of your target in a file and use --symfs with it it
> should work, because then, because it doesn't have the build-id for the DSOs
> in the callchains that didn't had hits, it will resort to using just the
> pathname when looking for a symtab or the whole ELF for annotation, etc.
>
> > files. I also don't have the corresponding files in my target's ~/.debug
> > folder, where everything seems to work as expected. Is that, because there
> > the symbols are read directly from the file and nothing is cached in
> > ~/.debug?
>
> That is a possibility, yes. But it should be adding it to ~/.debug, that
> is the default, unless you have something disabling that in your
> ~/.perfconfig or some sort of problem is happening that is preventing it
> to do the hardlink/copy.
I now realized that the culprit here is actually a broken platform, i.e. the
libs above simply don't have a build id:
eu-readelf -n libc.so.6
Note section [ 1] '.note.ABI-tag' of 32 bytes at offset 0x270:
Owner Data size Type
GNU 16 VERSION
OS: Linux, ABI: 3.18.0
So no wonder they are not included in perf archive. I think a warning would be
helpful here to tell users that analyzing the data on the remote machine will
require --symfs (which I wasn't aware of, thanks!).
> > Can I set an environment variable to let perf know to look in secondary
> > paths for the libraries? I.e. I hope to make it look into the cross
> > compile sysroot for these libraries, instead of literally interpreting
> > the path as a valid path on my development machine.
>
> That is the --symfs David talked about, no?
>
> [acme@jouet linux]$ perf report -h symfs
>
> Usage: perf report [<options>]
>
> --symfs <directory>
> Look for files with symbols relative to this
> directory
Yes, symfs sounds exactly like what I was looking for. I have now tested this,
and seem to hit some issues with it:
I unpacked a perf archive from a platform which includes a libc with buildit:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ tar -tvf perf.data.tar.bz2 | grep libc-2.19
lrwxrwxrwx root/root 0 2000-01-01 18:36 .build-id/
45/7c5918872338e1b82bf8bf41ae9f55b34604ea -> ../../lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea
drwxr-xr-x root/root 0 2000-01-01 18:36 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/
-rwxr-xr-x root/root 911068 2016-03-01 01:55 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
-rw-r--r-- root/root 0 2000-01-01 18:36 lib/arm-linux-gnueabihf/
libc-2.19.so/457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Analyzing it for some reason isn't using the build id:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ strace -e file -f perf report --stdio -s sym,srcline
|& grep libc-2.19
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) = -1 ENOENT (No
such file or directory)
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316180) = -1 ENOENT (No
such file or directory)
stat("/usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug",
0x7ffde9316200) = -1 ENOENT (No such file or directory)
stat("/usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) =
-1 ENOENT (No such file or directory)
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffde9316200) = -1 ENOENT (No
such file or directory)
stat("/lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffde9316200) = -1
ENOENT (No such file or directory)
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
0.25% 0.00% [.] 0xffffffff4936136c
libc-2.19.so[ffffffff4936136
0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
0.24% 0.00% [.] 0xffffffff49361422
libc-2.19.so[ffffffff4936142
0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
0.21% 0.00% [.] 0xffffffff49361664
libc-2.19.so[ffffffff4936166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Setting symfs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
milian@milian-kdab2:/tmp$ locate libc-2.19.so
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
/home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/lib/arm-
linux-gnueabihf/libc-2.19.so
milian@milian-kdab2:/tmp$ strace -e file -f perf report --symfs=/ssd/milian/
projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/ --stdio -s
sym,srcline |& grep libc-2.19
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 8
stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug", 0x7ffc4da5e740) =
-1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1
ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
...}) = 0
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 10
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
such file or directory)
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
file or directory)
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
0.25% 0.00% [.] 0xffffffff4936136c libc-2.19.so[ffffffff4936136
0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
0.24% 0.00% [.] 0xffffffff49361422 libc-2.19.so[ffffffff4936142
0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
0.21% 0.00% [.] 0xffffffff49361664 libc-2.19.so[ffffffff4936166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note how it does look into symfs sometimes, but not everywhere...
To make things worse, it seems to break the buildid lookup, i.e. when you get
samples in a file that you deployed manually to a target, which is not
included in your sysroot that you pass to symfs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ perf report --stdio -s sym,srcline |& grep particles
34.79% 34.79% [.] main particles.cpp:34
32.62% 32.62% [.] main particles.cpp:33
9.85% 9.85% [.] main particles.cpp:32
$ perf report --symfs=/ssd/milian/projects/pandaboard/debian-8.5-minimal-
armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
/root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
found, continuing without symbols
addr2line_init failed for /root/particles
# above line is repeated a lot
26.99% 26.99% [.] 0x00000000000005a4 particles[5a4]
26.99% 0.00% [.] 0xffffffffffff05a4 particles[ffffffffffff05a4]
21.83% 21.83% [.] 0x00000000000005a6 particles[5a6]
21.83% 0.00% [.] 0xffffffffffff05a6 particles[ffffffffffff05a6]
14.55% 0.00% [.] 0xffffffffffff0596 particles[ffffffffffff0596]
14.54% 14.54% [.] 0x0000000000000596 particles[596]
...
$ strace -e file -f perf report --symfs=/ssd/milian/projects/pandaboard/
debian-8.5-minimal-armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/root/particles", 0x7ffe7535bf40) = -1 EACCES (Permission denied)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", 0x7ffe7535bec0) = -1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/root/particles.debug", 0x7ffe7535bf40) = -1 ENOENT (No such file
or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
usr/lib/debug/root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or directory)
stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
root/.debug/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
directory)
/root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
found, continuing without symbols
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note how the above does not look for the buildid in my ~/.debug anymore...
Cheers
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-15 12:20 ` Milian Wolff
@ 2016-08-15 14:42 ` David Ahern
2016-08-15 15:22 ` Milian Wolff
0 siblings, 1 reply; 8+ messages in thread
From: David Ahern @ 2016-08-15 14:42 UTC (permalink / raw)
To: Milian Wolff, acme; +Cc: linux-perf-users, hekuang
On 8/15/16 6:20 AM, Milian Wolff wrote:
>> That is the --symfs David talked about, no?
>>
>> [acme@jouet linux]$ perf report -h symfs
>>
>> Usage: perf report [<options>]
>>
>> --symfs <directory>
>> Look for files with symbols relative to this
>> directory
>
> Yes, symfs sounds exactly like what I was looking for. I have now tested this,
> and seem to hit some issues with it:
-----8<-----
>
> Setting symfs:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> milian@milian-kdab2:/tmp$ locate libc-2.19.so
> /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so
> /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> 457c5918872338e1b82bf8bf41ae9f55b34604ea
> /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> 457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
> /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> 457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
> /ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/lib/arm-
> linux-gnueabihf/libc-2.19.so
> milian@milian-kdab2:/tmp$ strace -e file -f perf report --symfs=/ssd/milian/
> projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/ --stdio -s
> sym,srcline |& grep libc-2.19
> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 8
> stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
> such file or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
> ...}) = 0
> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
> ...}) = 0
> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug", 0x7ffc4da5e740) =
> -1 ENOENT (No such file or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1
> ENOENT (No such file or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755, st_size=911068,
> ...}) = 0
> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 10
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT (No
> such file or directory)
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No such
> file or directory)
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> 0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
> 0.25% 0.00% [.] 0xffffffff4936136c libc-2.19.so[ffffffff4936136
> 0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
> 0.24% 0.00% [.] 0xffffffff49361422 libc-2.19.so[ffffffff4936142
> 0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
> 0.21% 0.00% [.] 0xffffffff49361664 libc-2.19.so[ffffffff4936166
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Note how it does look into symfs sometimes, but not everywhere...
It should not open files outside of the symfs tree.
If you note perf does find libc in the symfs 3 times but it is not finding the symbol information it wants.
>
> To make things worse, it seems to break the buildid lookup, i.e. when you get
> samples in a file that you deployed manually to a target, which is not
> included in your sysroot that you pass to symfs:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> $ perf report --stdio -s sym,srcline |& grep particles
> 34.79% 34.79% [.] main particles.cpp:34
> 32.62% 32.62% [.] main particles.cpp:33
> 9.85% 9.85% [.] main particles.cpp:32
> $ perf report --symfs=/ssd/milian/projects/pandaboard/debian-8.5-minimal-
> armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
> /root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
> found, continuing without symbols
> addr2line_init failed for /root/particles
> # above line is repeated a lot
> 26.99% 26.99% [.] 0x00000000000005a4 particles[5a4]
> 26.99% 0.00% [.] 0xffffffffffff05a4 particles[ffffffffffff05a4]
> 21.83% 21.83% [.] 0x00000000000005a6 particles[5a6]
> 21.83% 0.00% [.] 0xffffffffffff05a6 particles[ffffffffffff05a6]
> 14.55% 0.00% [.] 0xffffffffffff0596 particles[ffffffffffff0596]
> 14.54% 14.54% [.] 0x0000000000000596 particles[596]
> ...
> $ strace -e file -f perf report --symfs=/ssd/milian/projects/pandaboard/
> debian-8.5-minimal-armhf-2016-06-06/ --stdio -s sym,srcline |& grep particles
>
> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> root/particles", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/root/particles", 0x7ffe7535bf40) = -1 EACCES (Permission denied)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> root/particles", 0x7ffe7535bec0) = -1 ENOENT (No such file or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> usr/lib/debug/root/particles.debug", 0x7ffe7535bf40) = -1 ENOENT (No such file
> or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> usr/lib/debug/root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
> directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or directory)
> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06///
> root/.debug/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
> directory)
> /root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
> found, continuing without symbols
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Note how the above does not look for the buildid in my ~/.debug anymore...
With symfs every file should be below the symfs root directory, including build id files.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-15 14:42 ` David Ahern
@ 2016-08-15 15:22 ` Milian Wolff
2016-08-15 16:08 ` David Ahern
0 siblings, 1 reply; 8+ messages in thread
From: Milian Wolff @ 2016-08-15 15:22 UTC (permalink / raw)
To: dsahern; +Cc: acme, linux-perf-users, hekuang
[-- Attachment #1: Type: text/plain, Size: 7792 bytes --]
On Monday, August 15, 2016 8:42:12 AM CEST David Ahern wrote:
> On 8/15/16 6:20 AM, Milian Wolff wrote:
> >> That is the --symfs David talked about, no?
> >>
> >> [acme@jouet linux]$ perf report -h symfs
> >>
> >> Usage: perf report [<options>]
> >>
> >> --symfs <directory>
> >>
> >> Look for files with symbols relative to this
> >>
> >> directory
> >
> > Yes, symfs sounds exactly like what I was looking for. I have now tested
> > this,
> > and seem to hit some issues with it:
> -----8<-----
>
> > Setting symfs:
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > milian@milian-kdab2:/tmp$ locate libc-2.19.so
> > /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so
> > /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> > 457c5918872338e1b82bf8bf41ae9f55b34604ea
> > /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> > 457c5918872338e1b82bf8bf41ae9f55b34604ea/elf
> > /home/milian/.debug/lib/arm-linux-gnueabihf/libc-2.19.so/
> > 457c5918872338e1b82bf8bf41ae9f55b34604ea/probes
> > /ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/lib/ar
> > m- linux-gnueabihf/libc-2.19.so
> > milian@milian-kdab2:/tmp$ strace -e file -f perf report
> > --symfs=/ssd/milian/
> > projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/ --stdio -s
> > sym,srcline |& grep libc-2.19
> > open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 8
> > stat("/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) = -1 ENOENT
> > (No such file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755,
> > st_size=911068, ...}) = 0
> > open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755,
> > st_size=911068, ...}) = 0
> > open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 9
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so.debug",
> > 0x7ffc4da5e740) = -1 ENOENT (No such file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so", 0x7ffc4da5e740) =
> > -1 ENOENT (No such file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", {st_mode=S_IFREG|0755,
> > st_size=911068, ...}) = 0
> > open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = 10
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > // lib/arm-linux-gnueabihf/.debug/libc-2.19.so", 0x7ffc4da5e740) = -1
> > ENOENT (No such file or directory)
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > open("/lib/arm-linux-gnueabihf/libc-2.19.so", O_RDONLY) = -1 ENOENT (No
> > such file or directory)
> > addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> >
> > 0.25% 0.25% [.] 0x000000000005336c libc-2.19.so[5336c]
> > 0.25% 0.00% [.] 0xffffffff4936136c libc-2.19.so[ffffffff4936136
> > 0.24% 0.24% [.] 0x0000000000053422 libc-2.19.so[53422]
> > 0.24% 0.00% [.] 0xffffffff49361422 libc-2.19.so[ffffffff4936142
> > 0.21% 0.21% [.] 0x0000000000053664 libc-2.19.so[53664]
> > 0.21% 0.00% [.] 0xffffffff49361664 libc-2.19.so[ffffffff4936166
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Note how it does look into symfs sometimes, but not everywhere...
>
> It should not open files outside of the symfs tree.
But apparently that is what happens.
> If you note perf does find libc in the symfs 3 times but it is not finding
> the symbol information it wants.
I only see that it fails to get addr2line debug information from paths
*outside* of the symfs:
addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
> > To make things worse, it seems to break the buildid lookup, i.e. when you
> > get samples in a file that you deployed manually to a target, which is
> > not included in your sysroot that you pass to symfs:
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<snip>
> > $ strace -e file -f perf report --symfs=/ssd/milian/projects/pandaboard/
> > debian-8.5-minimal-armhf-2016-06-06/ --stdio -s sym,srcline |& grep
> > particles
> >
> > open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > root/particles", O_RDONLY) = -1 ENOENT (No such file or directory)
> > stat("/root/particles", 0x7ffe7535bf40) = -1 EACCES (Permission denied)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > root/particles", 0x7ffe7535bec0) = -1 ENOENT (No such file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > usr/lib/debug/root/particles.debug", 0x7ffe7535bf40) = -1 ENOENT (No such
> > file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > usr/lib/debug/root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file
> > or
> > directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or directory)
> > stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
> > //
> > root/.debug/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
> > directory)
> > /root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
> > found, continuing without symbols
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Note how the above does not look for the buildid in my ~/.debug anymore...
>
> With symfs every file should be below the symfs root directory, including
> build id files.
So I should unpack the perf archive on top of my sysroot?
That sounds like a bad idea, considering that I would then get clashes between
e.g. the actual /lib/libc.so.6 in my sysroot and the debug info that is in a
folder called /lib/libc.so.6/ in ~/.debug. And from the strace output above,
it does not look like perf would look into a .debug folder directly in
$symfs/.
It also does not seem to be possible to pass multiple symfs directories, which
could also solve the issue for me. I.e. pass both a path to the sysroot, and
an additional path to wherever I unpacked the perf archive to.
Thanks
--
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cross platform perf reporting
2016-08-15 15:22 ` Milian Wolff
@ 2016-08-15 16:08 ` David Ahern
0 siblings, 0 replies; 8+ messages in thread
From: David Ahern @ 2016-08-15 16:08 UTC (permalink / raw)
To: Milian Wolff; +Cc: acme, linux-perf-users, hekuang, Jiri Olsa, Namhyung Kim
On 8/15/16 9:22 AM, Milian Wolff wrote:
>> It should not open files outside of the symfs tree.
>
> But apparently that is what happens.
sounds like a bug.
>
>> If you note perf does find libc in the symfs 3 times but it is not finding
>> the symbol information it wants.
>
> I only see that it fails to get addr2line debug information from paths
> *outside* of the symfs:
>
> addr2line_init failed for /lib/arm-linux-gnueabihf/libc-2.19.so
that feature needs to handle symfs
>
>>> To make things worse, it seems to break the buildid lookup, i.e. when you
>>> get samples in a file that you deployed manually to a target, which is
>>> not included in your sysroot that you pass to symfs:
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> <snip>
>>> $ strace -e file -f perf report --symfs=/ssd/milian/projects/pandaboard/
>>> debian-8.5-minimal-armhf-2016-06-06/ --stdio -s sym,srcline |& grep
>>> particles
>>>
>>> open("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> root/particles", O_RDONLY) = -1 ENOENT (No such file or directory)
>>> stat("/root/particles", 0x7ffe7535bf40) = -1 EACCES (Permission denied)
>>> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> root/particles", 0x7ffe7535bec0) = -1 ENOENT (No such file or directory)
>>> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> usr/lib/debug/root/particles.debug", 0x7ffe7535bf40) = -1 ENOENT (No such
>>> file or directory)
>>> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> usr/lib/debug/root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file
>>> or
>>> directory)
>>> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> root/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or directory)
>>> stat("/ssd/milian/projects/pandaboard/debian-8.5-minimal-armhf-2016-06-06/
>>> //
>>> root/.debug/particles", 0x7ffe7535bf40) = -1 ENOENT (No such file or
>>> directory)
>>> /root/particles with build id 1c10a17afa25337ab368f8d9e690184e536c6563 not
>>> found, continuing without symbols
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Note how the above does not look for the buildid in my ~/.debug anymore...
>>
>> With symfs every file should be below the symfs root directory, including
>> build id files.
>
> So I should unpack the perf archive on top of my sysroot?
Not if it contains differences.
>
> That sounds like a bad idea, considering that I would then get clashes between
> e.g. the actual /lib/libc.so.6 in my sysroot and the debug info that is in a
> folder called /lib/libc.so.6/ in ~/.debug. And from the strace output above,
> it does not look like perf would look into a .debug folder directly in
> $symfs/.
I thought someone just added support for that.
> It also does not seem to be possible to pass multiple symfs directories, which
> could also solve the issue for me. I.e. pass both a path to the sysroot, and
> an additional path to wherever I unpacked the perf archive to.
When I added the symfs option back in 2010 it was to support many, many versions of embedded images. The images may start with the same sysroot (e.g., system files from the OS vendor) but will differ dramatically with locally built files.
I had a script that took the Wind River sysroot and copied to a tmp directory, added debug files for the system binaries and libraries, and then overlayed the locally built files which contained symbols and kernel files. From there perf would only open files in that directory.
When done, rm -rf tmp_dir and all remnants of that image were gone. I specifically did not want to manage everything in ~/.debug when you consider N variants of images (images you build, co-workers builds, project builds, official builds, etc) and all of those changing on a nearly daily basis.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-08-15 16:08 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-12 12:22 Cross platform perf reporting Milian Wolff
2016-08-12 14:38 ` David Ahern
2016-08-12 15:10 ` Milian Wolff
2016-08-12 15:31 ` Arnaldo Carvalho de Melo
2016-08-15 12:20 ` Milian Wolff
2016-08-15 14:42 ` David Ahern
2016-08-15 15:22 ` Milian Wolff
2016-08-15 16:08 ` David Ahern
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.