Re: Per-CPU variables in modules and pahole

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Thu, Dec 10, 2020 at 3:49 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Dec 10, 2020 at 3:42 PM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Thu, Dec 10, 2020 at 09:02:05AM -0800, Andrii Nakryiko wrote:
> >
> > SNIP
> >
> > >
> > > yes, ELF symbol's value is 4, but when iterating DWARF variables
> > > (0x10e70 + 4) is returned. It does look like a special handling of
> > > modules. I missed that libdw does some special things for specifically
> > > modules. Further debugging yesterday showed that 0x10e70 roughly
> > > corresponds to the offset of .data..per_cpu if you count all the
> > > allocatable data sections that come before it. So I think you are
> > > right. We should probably centralize the logic of kernel module
> > > detection so that we can handle these module vs non-module differences
> > > properly.
> > >
> > > >
> > > > not sure this is related but looks like similar issue I had to
> > > > solve for modules functions, as described in the changelog:
> > > > (not merged yet)
> > > >
> > > >     btf_encoder: Detect kernel module ftrace addresses
> > > >
> > > >     ...
> > > >     There's one tricky point with kernel modules wrt Elf object,
> > > >     which we get from dwfl_module_getelf function. This function
> > > >     performs all possible relocations, including __mcount_loc
> > > >     section.
> > > >
> > > >     So addrs array contains relocated values, which we need take
> > > >     into account when we compare them to functions values which
> > > >     are relative to their sections.
> > > >     ...
> > > >
> > > > The 0x10e74 value could be relocated 4.. but it's me guessing,
> > > > because not sure where you see that address exactly
> > >
> > >
> > > It comes up in cu__encode_btf(), var->ip.addr is not 4, as we expect it to be.
> >
> > I'm taking section sh_addr for each function and relocate
> > the addr value for kernel modules, check setup_functions
> > function
> >
> > I don't see this being somehow centralized, looks simple
> > enough to me for each case
>
> I meant centralized detection of whether we are working with the
> module or vmlinux or something else. setup_functions() currently has
> very specific heuristic for that. So I'd like to extract that or come
> up with some other way that won't be so function specific
> (__start_mcount_loc symbol vs __mcount_loc section).
>

This seems to be unnecessary, actually. We already record
btfe->percpu_base_addr, which for vmlinux is always zero, while for
module non-zero. So just subtracting this base addr before looking up
ELF symbol solves the problem for me and still works for vmlinux. So
I'm going with that for now.

> >
> > jirka
> >