Re: [PATCH v4] btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.

[Hao Luo <haoluo-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>]

Hi Andrii,

Thanks for taking a look! I can post a v5 based on your thorough and
detailed analysis this week. Some of my other work on sched consumed
most of my bandwidth last week and I very appreciate your help! I
apologize if it is felt I dropped this patch and will follow up within
this week.

Hao

On Mon, Jun 29, 2020 at 1:26 PM Andrii Nakryiko
<andrii.nakryiko-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> On Fri, Jun 19, 2020 at 1:30 PM Hao Luo <haoluo-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> >
> > Hi, Andrii,
> >
> > I agree that we'd better put a hold on this patch until we find out
> > the reason for the 'unconventional' symbols. I'll _try_ to figure it
> > out, but not able to fully commit my time on this patch. I thought I'd
> > better publish this patch as the DATASEC and VARs are generated
> > correctly in format, so that anyone can use it to generate the vmlinux
> > and continue the development on libbpf based on your ksym work (i.e.
> > typed ksyms).
> >
>
> Hey Hao,
>
> It's a pity that you are willing to drop this while being half-step
> away from doing this properly. See my other reply, there is something
> fishy with variable__name() and variable__type_size() and how it
> calculates/caches values. ELF itself has all the data (based ELF
> symbol data) and it seems to be correct. And I was correct to be
> suspicious about just filtering out such variables, because in my case
> you'd filter out a good chunk of variables for no good reason: 74
> variables, which is a 26% of all per-CPU variables in my kernel.
>
> It's up to you, of course, but it would be nice to fix it up (just
> switching to elf_sym__size() and elf_sym__name() would be fine, you'd
> just need to make sure to add elf_sym_name() result into a string
> buffer; unless Arnaldo has some better alternatives) and publish
> complete v5, that would get merged into pahole. After that, building
> on my .ksym work in libbpf should get you to what you need from BPF
> pretty quickly (plus some kernel-side logic to recognize these per-CPU
> variables).
>
> > Hao
> >
> > On Fri, Jun 19, 2020 at 12:58 PM Andrii Nakryiko
> > <andrii.nakryiko-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > On Thu, Jun 18, 2020 at 12:49 AM Hao Luo <haoluo-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> > > >
> > > > On SMP systems, the global percpu variables are placed in a special
> > > > '.data..percpu' section, which is stored in a segment whose initial
> > > > address is set to 0, the addresses of per-CPU variables are relative
> > > > positive addresses [1].
> > > >
> > > > This patch extracts these variables from vmlinux and places them with
> > > > their type information in BTF. More specifically, when BTF is encoded,
> > > > we find the index of the '.data..percpu' section and then traverse
> > > > the symbol table to find those global objects which are in this section.
> > > > For each of these objects, we push a BTF_KIND_VAR into the types buffer,
> > > > and a BTF_VAR_SECINFO into another buffer, percpu_secinfo. When all the
> > > > CUs have finished processing, we push a BTF_KIND_DATASEC into the
> > > > btfe->types buffer, followed by the percpu_secinfo's content.
> > > >
> > > > In a v5.7-rc7 linux kernel, I was able to extract 291 such variables.
> > > > The build time overhead is small and the space overhead is also small.
> > > >
> > > > Testing:
> > > >
> > > > - vmlinux size has increased by ~12kb.
> > > >   Before:
> > > >    $ readelf -SW vmlinux | grep BTF
> > > >    [25] .BTF              PROGBITS        ffffffff821a905c 13a905c 2d2bf8 00
> > > >   After:
> > > >    $ pahole -J vmlinux
> > > >    $ readelf -SW vmlinux  | grep BTF
> > > >    [25] .BTF              PROGBITS        ffffffff821a905c 13a905c 2d5bca 00
> > > >
> > > > - Common global percpu VARs and DATASEC are found in BTF section.
> > > >   $ bpftool btf dump file vmlinux | grep runqueues
> > > >   [14098] VAR 'runqueues' type_id=13725, linkage=global-alloc
> > > >
> > > >   $ bpftool btf dump file vmlinux | grep 'cpu_stopper'
> > > >   [17592] STRUCT 'cpu_stopper' size=72 vlen=5
> > > >   [17612] VAR 'cpu_stopper' type_id=17592, linkage=static
> > > >
> > > >   $ bpftool btf dump file vmlinux | grep ' DATASEC '
> > > >   [63652] DATASEC '.data..percpu' size=0 vlen=294
> > >
> > > probably forgot to update the example, I'd imagine size wouldn't be 0 anymore?
> > >
>
> [...]