Question: perf dso support for /proc/kallsyms

Question: perf dso support for /proc/kallsyms

[leo.yan]

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace
data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file.   So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
run into below flow:

  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

I can observe the function data_read_offset() returns failure, this is caused
by checking the offset sanity "if (offset > dso->data.file_size)"  (I pasted
the whole function code at below in case you want to get more context for it),
but if perf use "/proc/kallsyms" to load kernel symbols, the variable
'dso->data.file_size' will be set to zero thus the sanity checking always
thinks the offset is out of the file size bound.

Now I still don't understand how the dso/map support "/proc/kallsyms" and
have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this?  Or even better if you have fixing
for this, I am glad to test at my side.

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
                                u64 offset, u8 *data, ssize_t size)
{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan

RE: Question: perf dso support for /proc/kallsyms

[Al Grant]

> Now I still don't understand how the dso/map support "/proc/kallsyms" and
> have no idea to fix this issue, though I spent some time to look into it.

The way this is supported is that at record time, pseudo mmap records
are created for the kernel. But depending on permissions these might
not get created. This isn't just an ETM issue, it can happen on Intel too.

What do you see in "perf report -D", do you see a PERF_RECORD_MMAP
record for "[kernel.kallsyms]_text" and possibly some others for loadable
kernel modules in /lib/modules?

Does it all work if you run perf record as sudo? Or if you do

   sudo sysctl kernel.kptr_restrict=0

before you run perf record?

Al
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: Question: perf dso support for /proc/kallsyms

[leo.yan]

Hi Al,

On Fri, Nov 02, 2018 at 12:08:04PM +0000, Al Grant wrote:
> > Now I still don't understand how the dso/map support "/proc/kallsyms" and
> > have no idea to fix this issue, though I spent some time to look into it.
> 
> The way this is supported is that at record time, pseudo mmap records
> are created for the kernel. But depending on permissions these might
> not get created. This isn't just an ETM issue, it can happen on Intel too.

Agree, I also think this is not a specific issue only for Arm platform,
this should be one common issue for how to parse kernel symbols with
kallsyms file.

> What do you see in "perf report -D", do you see a PERF_RECORD_MMAP
> record for "[kernel.kallsyms]_text" and possibly some others for loadable
> kernel modules in /lib/modules?

Yes, I can see PERF_RECORD_MMAP for "[kernel.kallsyms]_text".

0x350 [0x50]: event: 1
.
. ... raw event: size 80 bytes
.  0000:  01 00 00 00 01 00 50 00 ff ff ff ff 00 00 00 00  ......P.........
.  0010:  00 00 08 08 00 00 ff ff ff ff f7 f7 ff ff 00 00  ................
.  0020:  00 00 08 08 00 00 ff ff 5b 6b 65 72 6e 65 6c 2e  ........[kernel.
.  0030:  6b 61 6c 6c 73 79 6d 73 5d 5f 74 65 78 74 00 00  kallsyms]_text..
.  0040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

0x350 [0x50]: PERF_RECORD_MMAP -1/0: [0xffff000008080000(0xfffff7f7ffff) @ 0xffff000008080000]: x [kernel.kallsyms]_text

Below is the infor for checking buildid:

root@debian:~/coresight_test# perf buildid-list
0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-2.27.so
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so
70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-gnu/libc-2.27.so
root@debian:~/coresight_test# ls ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
kallsyms

> Does it all work if you run perf record as sudo? Or if you do
> 
>    sudo sysctl kernel.kptr_restrict=0
> 
> before you run perf record?

Yes, tested this on Juno board with Debian rootFS and logined in
with 'root' user.  I suspected the pointer permission issue so
checked with below command:

root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 
0

Thanks,
Leo Yan

RE: Question: perf dso support for /proc/kallsyms

[Al Grant]

> root@debian:~/coresight_test# perf buildid-list
> 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
> b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
> 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-
> 2.27.so
> 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
> 25829a59e21012cfde7850b30a310cd3a58f531c
> /root/coresight_test/libcstest.so
> 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-
> gnu/libc-2.27.so
> root@debian:~/coresight_test# ls
> ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
> kallsyms

What's in that last file? I've seen it happen that the copy of kallsyms
in ~/.debug has the symbol addresses as zeroes - possibly because it
was created when you didn't have permissions. That's really a bug
in perf, as cacheing a copy of this file with the addresses zeroed out
is kind of pointless. Again, this happens on Intel too.

Then, you can give yourself permissions - but perf's already cached
the file and won't update it!

If you delete it, and then rerun perf record (either as sudo or now
that you've got kptr_restrict=0) you should see it reappear, with
correct kernel addresses.

Perhaps nobody spotted this on Intel because perf report goes directly
to /proc/kallsyms. But it would be an issue if you ran a perf report
on a perf.data from an older kernel and it had to go to ~/.debug.
At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would
mean you couldn't symbolicate any addresses.

Al



>
> > Does it all work if you run perf record as sudo? Or if you do
> >
> >    sudo sysctl kernel.kptr_restrict=0
> >
> > before you run perf record?
>
> Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user.
> I suspected the pointer permission issue so checked with below command:
>
> root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict
> 0
>
> Thanks,
> Leo Yan
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: Question: perf dso support for /proc/kallsyms

[leo.yan]

On Fri, Nov 02, 2018 at 02:12:28PM +0000, Al Grant wrote:
> > root@debian:~/coresight_test# perf buildid-list
> > 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
> > b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
> > 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-
> > 2.27.so
> > 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
> > 25829a59e21012cfde7850b30a310cd3a58f531c
> > /root/coresight_test/libcstest.so
> > 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-
> > gnu/libc-2.27.so
> > root@debian:~/coresight_test# ls
> > ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
> > kallsyms
> 
> What's in that last file?

I can see the correct symbol infos in this file:

root@debian:~/coresight_test# cat ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/kallsyms | head -20
ffff000008080000 t _head
ffff000008080000 T _text
ffff000008080040 t pe_header
ffff000008080044 t coff_header
ffff000008080058 t optional_header
ffff000008080070 t extra_header_fields
ffff0000080800f8 t section_table
ffff000008081000 T do_undefinstr
ffff000008081000 t efi_header_end
ffff000008081000 T _stext
ffff000008081000 T __exception_text_start
ffff000008081268 T do_cp15instr
ffff000008081380 T do_sysinstr
ffff000008081410 T do_debug_exception
ffff000008081550 T do_mem_abort
ffff000008081600 T do_el0_irq_bp_hardening
ffff000008081650 T do_el0_ia_bp_hardening
ffff0000080816f8 T do_sp_pc_abort
ffff0000080817c4 T __exception_text_end
ffff0000080817c8 t bcm2835_handle_irq

> I've seen it happen that the copy of kallsyms
> in ~/.debug has the symbol addresses as zeroes - possibly because it
> was created when you didn't have permissions. That's really a bug
> in perf, as cacheing a copy of this file with the addresses zeroed out
> is kind of pointless. Again, this happens on Intel too.
> 
> Then, you can give yourself permissions - but perf's already cached
> the file and won't update it!
> 
> If you delete it, and then rerun perf record (either as sudo or now
> that you've got kptr_restrict=0) you should see it reappear, with
> correct kernel addresses.
> 
> Perhaps nobody spotted this on Intel because perf report goes directly
> to /proc/kallsyms. But it would be an issue if you ran a perf report
> on a perf.data from an older kernel and it had to go to ~/.debug.
> At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would
> mean you couldn't symbolicate any addresses.

This is another issue related with permission for accessing kernel
pointer values and this will bother much for cross platforms usage
(e.g. 'perf record' on Arm platform and 'perf script' on x86
platform).

At my side, I executed all commands on Arm Juno board and simply to
say the kallsyms support is broken in perf.

Thanks,
Leo Yan

Re: Question: perf dso support for /proc/kallsyms

[Mike Leach]

Hi Leo,

My understanding is that when we decode CoreSight trace - be it on the
system that generated it, or off target device on a separate host
system, we are using the dso files in .debug/ as these represent the
memory layout at the time trace was recorded.

If I look into a recent session copied up to my linux box from DB410,
is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than
the live one generated by /proc/kallsyms. I don't really want to use
the local /proc/kallsyms on my x86 linux box when decoding an ARM
trace captured elsewhere.

So perhaps the problem to be solved is not how to use /proc/kallsyms
if no vmlinux is supplied to the script, but ensure that the
[kernel.kallsyms] is used?

Regards

Mike
On Fri, 2 Nov 2018 at 02:55, <leo.yan@linaro.org> wrote:
>
> Hi all,
>
> Now I found that if use the command 'perf script' for Arm CoreSight trace
> data, it fails to parse kernel symbols if we don't specify kernel vmlinux
> file.   So when we don't specify kernel symbol files then perf tool will
> roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
> run into below flow:
>
>   thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
>   map__load(al.map);
>   dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
>     `-> data_read_offset()
>
> I can observe the function data_read_offset() returns failure, this is caused
> by checking the offset sanity "if (offset > dso->data.file_size)"  (I pasted
> the whole function code at below in case you want to get more context for it),
> but if perf use "/proc/kallsyms" to load kernel symbols, the variable
> 'dso->data.file_size' will be set to zero thus the sanity checking always
> thinks the offset is out of the file size bound.
>
> Now I still don't understand how the dso/map support "/proc/kallsyms" and
> have no idea to fix this issue, though I spent some time to look into it.
>
> Could you give some suggestion for this?  Or even better if you have fixing
> for this, I am glad to test at my side.
>
> static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
>                                 u64 offset, u8 *data, ssize_t size)
> {
>         if (data_file_size(dso, machine))
>                 return -1;
>
>         /* Check the offset sanity. */
>         if (offset > dso->data.file_size)
>                 return -1;
>
>         if (offset + size < offset)
>                 return -1;
>
>         return cached_read(dso, machine, offset, data, size);
> }
>
> Thanks,
> Leo Yan
> _______________________________________________
> CoreSight mailing list
> CoreSight@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/coresight



-- 
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK

Re: Question: perf dso support for /proc/kallsyms

[leo.yan]

Hi Mike,

On Fri, Nov 02, 2018 at 01:08:40PM +0000, Mike Leach wrote:
> Hi Leo,
> 
> My understanding is that when we decode CoreSight trace - be it on the
> system that generated it, or off target device on a separate host
> system, we are using the dso files in .debug/ as these represent the
> memory layout at the time trace was recorded.
> 
> If I look into a recent session copied up to my linux box from DB410,
> is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
> - which seems to me to be the relevant symbol file to use rather than
> the live one generated by /proc/kallsyms. I don't really want to use
> the local /proc/kallsyms on my x86 linux box when decoding an ARM
> trace captured elsewhere.
> 
> So perhaps the problem to be solved is not how to use /proc/kallsyms
> if no vmlinux is supplied to the script, but ensure that the
> [kernel.kallsyms] is used?

Yes, this is good point, the subject should be changed to:
"Question: perf dso support for kallsyms".

I think I made a mistake in my previous email, so clarify it:

~/.debug/\[kernel.kallsyms\]/$BUILDID/kallsyms should has higher
priority than /proc/kallsyms, by default the perf tool should use
kallsyms file under ~/.debug folder.  I also tried to manually specify
the kallsyms file with the command "perf script --kallsyms ./kallsyms",
for both cases perf fails to parse symbols for any kernel address.

Thanks,
Leo Yan

Re: Question: perf dso support for /proc/kallsyms

[leo.yan]

On Fri, Nov 02, 2018 at 10:55:16AM +0800, Leo Yan wrote:
> Hi all,
> 
> Now I found that if use the command 'perf script' for Arm CoreSight trace
> data, it fails to parse kernel symbols if we don't specify kernel vmlinux
> file.   So when we don't specify kernel symbol files then perf tool will
> roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
> run into below flow:
> 
>   thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
>   map__load(al.map);
>   dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
>     `-> data_read_offset()
> 
> I can observe the function data_read_offset() returns failure, this is caused
> by checking the offset sanity "if (offset > dso->data.file_size)"  (I pasted
> the whole function code at below in case you want to get more context for it),
> but if perf use "/proc/kallsyms" to load kernel symbols, the variable
> 'dso->data.file_size' will be set to zero thus the sanity checking always
> thinks the offset is out of the file size bound.
> 
> Now I still don't understand how the dso/map support "/proc/kallsyms" and
> have no idea to fix this issue, though I spent some time to look into it.
> 
> Could you give some suggestion for this?  Or even better if you have fixing
> for this, I am glad to test at my side.

Hi Jiri, Arnaldo,

Could you give some suggestion for this question?  Thanks!

> static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
>                                 u64 offset, u8 *data, ssize_t size)
> {
>         if (data_file_size(dso, machine))
>                 return -1;
> 
>         /* Check the offset sanity. */
>         if (offset > dso->data.file_size)
>                 return -1;
> 
>         if (offset + size < offset)
>                 return -1;
> 
>         return cached_read(dso, machine, offset, data, size);
> }
> 
> Thanks,
> Leo Yan

Re: Question: perf dso support for /proc/kallsyms

[Jiri Olsa]

On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
> Hi all,
> 
> Now I found that if use the command 'perf script' for Arm CoreSight trace
> data, it fails to parse kernel symbols if we don't specify kernel vmlinux
> file.   So when we don't specify kernel symbol files then perf tool will
> roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
> run into below flow:

yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms

> 
>   thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
>   map__load(al.map);
>   dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
>     `-> data_read_offset()

so what is the actual error you see in the perf script?
unresolved samples? could you please describe your config
and workload?

thanks,
jirka

> 
> I can observe the function data_read_offset() returns failure, this is caused
> by checking the offset sanity "if (offset > dso->data.file_size)"  (I pasted
> the whole function code at below in case you want to get more context for it),
> but if perf use "/proc/kallsyms" to load kernel symbols, the variable
> 'dso->data.file_size' will be set to zero thus the sanity checking always
> thinks the offset is out of the file size bound.
> 
> Now I still don't understand how the dso/map support "/proc/kallsyms" and
> have no idea to fix this issue, though I spent some time to look into it.
> 
> Could you give some suggestion for this?  Or even better if you have fixing
> for this, I am glad to test at my side.
> 
> static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
>                                 u64 offset, u8 *data, ssize_t size)
> {
>         if (data_file_size(dso, machine))
>                 return -1;
> 
>         /* Check the offset sanity. */
>         if (offset > dso->data.file_size)
>                 return -1;
> 
>         if (offset + size < offset)
>                 return -1;
> 
>         return cached_read(dso, machine, offset, data, size);
> }
> 
> Thanks,
> Leo Yan

Re: Question: perf dso support for /proc/kallsyms

[leo.yan]

Hi Jiri,

On Wed, Nov 07, 2018 at 03:10:06PM +0100, Jiri Olsa wrote:
> On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
> > Hi all,
> > 
> > Now I found that if use the command 'perf script' for Arm CoreSight trace
> > data, it fails to parse kernel symbols if we don't specify kernel vmlinux
> > file.   So when we don't specify kernel symbol files then perf tool will
> > roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
> > run into below flow:
> 
> yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
> 
> > 
> >   thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
> >   map__load(al.map);
> >   dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
> >     `-> data_read_offset()
> 
> so what is the actual error you see in the perf script?
> unresolved samples? could you please describe your config
> and workload?

So at my side the error is the CoreSight trace decoder fails to
generate samples if the sample has kernel address, thus the decoder
doesn't generate any kernel sample if use kallsyms as dso.

For more detailed info is: the CoreSight decoder needs firstly to get
dso related info by calling dso__data_read_offset() [1], if we use
kallsyms then this function always returns failure then this leads the
docoder to discard all kernel samples.

Regarding to testing case, I can simply run 'uname' program so it can
produce kernel and user space trace data, if I use 'perf script -k
vmlinux' then I can see perf generate samples for kernel related traces,
but if use 'perf script' then only can see samples for user space.

Thanks,
Leo Yan

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/cs-etm.c#n302

> > I can observe the function data_read_offset() returns failure, this is caused
> > by checking the offset sanity "if (offset > dso->data.file_size)"  (I pasted
> > the whole function code at below in case you want to get more context for it),
> > but if perf use "/proc/kallsyms" to load kernel symbols, the variable
> > 'dso->data.file_size' will be set to zero thus the sanity checking always
> > thinks the offset is out of the file size bound.
> > 
> > Now I still don't understand how the dso/map support "/proc/kallsyms" and
> > have no idea to fix this issue, though I spent some time to look into it.
> > 
> > Could you give some suggestion for this?  Or even better if you have fixing
> > for this, I am glad to test at my side.
> > 
> > static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
> >                                 u64 offset, u8 *data, ssize_t size)
> > {
> >         if (data_file_size(dso, machine))
> >                 return -1;
> > 
> >         /* Check the offset sanity. */
> >         if (offset > dso->data.file_size)
> >                 return -1;
> > 
> >         if (offset + size < offset)
> >                 return -1;
> > 
> >         return cached_read(dso, machine, offset, data, size);
> > }
> > 
> > Thanks,
> > Leo Yan

Re: Question: perf dso support for /proc/kallsyms

[Jiri Olsa]

On Thu, Nov 08, 2018 at 04:51:52PM +0800, leo.yan@linaro.org wrote:
> Hi Jiri,
> 
> On Wed, Nov 07, 2018 at 03:10:06PM +0100, Jiri Olsa wrote:
> > On Fri, Nov 02, 2018 at 10:55:16AM +0800, leo.yan@linaro.org wrote:
> > > Hi all,
> > > 
> > > Now I found that if use the command 'perf script' for Arm CoreSight trace
> > > data, it fails to parse kernel symbols if we don't specify kernel vmlinux
> > > file.   So when we don't specify kernel symbol files then perf tool will
> > > roll back to use /proc/kallsyms for kernel symbols parsing, as result it will
> > > run into below flow:
> > 
> > yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
> > 
> > > 
> > >   thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
> > >   map__load(al.map);
> > >   dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
> > >     `-> data_read_offset()
> > 
> > so what is the actual error you see in the perf script?
> > unresolved samples? could you please describe your config
> > and workload?
> 
> So at my side the error is the CoreSight trace decoder fails to
> generate samples if the sample has kernel address, thus the decoder
> doesn't generate any kernel sample if use kallsyms as dso.
> 
> For more detailed info is: the CoreSight decoder needs firstly to get
> dso related info by calling dso__data_read_offset() [1], if we use
> kallsyms then this function always returns failure then this leads the
> docoder to discard all kernel samples.

I haven't checked on this code for some time but AFAICS
before you call dso__data_read_offset you need to check
  dso->data.status != DSO_DATA_STATUS_ERROR

map__load call ahead should take care on setting this and
find the source of the data

check the intel_pt code (intel_pt_walk_next_insn)
or for example grab_bb in perf script

I guess you need to have either vmlinux or kcore in place
to get some data out of it

jirka