Re: [RFT] Testing 1.22

From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Andrii Nakryiko <andrii@kernel.org>, Jiri Olsa <jolsa@kernel.org>,
	dwarves@vger.kernel.org, bpf <bpf@vger.kernel.org>,
	Kernel Team <kernel-team@fb.com>
Subject: Re: [RFT] Testing 1.22
Date: Fri, 4 Jun 2021 19:55:17 -0700	[thread overview]
Message-ID: <CAEf4BzbQ9w2smTMK5uwGGjyZ_mjDy-TGxd6m8tiDd3T_nJ7khQ@mail.gmail.com> (raw)
In-Reply-To: <YLjtwB+nGYvcCfgC@kernel.org>

On Thu, Jun 3, 2021 at 7:57 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> Em Sat, May 29, 2021 at 05:40:17PM -0700, Andrii Nakryiko escreveu:
> > On Fri, May 28, 2021 at 12:45 PM Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
> > > commit b579a18a1ea0ee84b90b5302f597dda2edf2f61b
> > > Author: Arnaldo Carvalho de Melo <acme@redhat.com>
> > > Date:   Fri May 28 16:41:30 2021 -0300
> > >
> > >     pahole: Allow encoding BTF into a detached file
> > >
> > >     Previously the newly encoded BTF info was stored into a ELF section in
> > >     the file where the DWARF info was obtained, but it is useful to just
> > >     dump it into a separate file, do it.
> > >
> > >     Requested-by: Andrii Nakryiko <andrii@kernel.org>
> > >     Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> > >
> >
> > Looks good, see few minor comments below. At some point it probably
> > would make sense to formalize "btf_encoder" as a struct with its own
> > state instead of passing in multiple variables. It would probably also
>
> Take a look at the tmp.master branch at:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=tmp.master

Oh wow, that's a lot of commits! :) Great that you decided to do this
refactoring, thanks!

>
> that btf_elf class isn't used anymore by btf_loader, that uses only
> libbpf's APIs, and now we have a btf_encoder class with all the globals,
> etc, more baby steps are needed to finally ditch btf_elf altogether and
> move on to the parallelization.

So do you plan to try to parallelize as a next step? I'm pretty
confident about BTF encoding part: dump each CU into its own BTF, use
btf__add_type() to merge multiple BTFs together. Just need to re-map
IDs (libbpf internally has API to visit each field that contains
type_id, it's well-defined enough to expose that as a public API, if
necessary). Then final btf_dedup().

But the DWARF loading and parsing part is almost a black box to me, so
I'm not sure how much work it would involve.

>
> I'm doing 'pahole -J vmlinux && btfdiff' after each cset and doing it
> very piecemeal as I'm doing will help bisecting any subtle bug this may
> introduce.
>
> > allow to parallelize BTF generation, where each CU would proceed in
> > parallel generating local BTF, and then the final pass would merge and
> > dedup BTFs. Currently reading and processing DWARF is the slowest part
> > of the DWARF-to-BTF conversion, parallelization and maybe some other
> > optimization seems like the only way to speed the process up.
> >
> > Acked-by: Andrii Nakryiko <andrii@kernel.org>
>
> Thanks!
>
> - Arnaldo