Re: [RFT] Testing 1.22

[Arnaldo Carvalho de Melo <acme@kernel.org>]

Em Sat, May 29, 2021 at 05:40:17PM -0700, Andrii Nakryiko escreveu:
> On Fri, May 28, 2021 at 12:45 PM Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
> >
> > Em Thu, May 27, 2021 at 01:41:13PM -0700, Andrii Nakryiko escreveu:
> > > On Thu, May 27, 2021 at 12:57 PM Arnaldo <arnaldo.melo@gmail.com> wrote:
> > > > On May 27, 2021 4:14:17 PM GMT-03:00, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> > > > >If we make 1.22 mandatory there will be no good reason to make 1.23
> > > > >mandatory again. So I will have absolutely no inclination to work on
> > > > >this, for example. So we are just wasting a chance to clean up the
> > > > >Kbuild story w.r.t. pahole. And we are talking about just a few days
> > > > >at most, while we do have a reasonable work around on the kernel side.
> >
> > > > So there were patches for stop using objcopy, which we thought could
> > > > uncover some can of worms, were there patches for the detached BTF
> > > > file?
> >
> > > No, there weren't, if I remember correctly. What's the concern,
> > > though? That detached BTF file isn't even an ELF, so it's
> > > btf__get_raw_data() and write it to the file. Done.
> >
> > See patch below, lightly tested, now working on making pahole accept raw
> > BTF files out of /sys/kernel/btf/
> >
> > Please test, and if works as expected, try to bolt this into the kbuild
> > process, as intended.
> 
> So while looking through this I found --skip_encoding_btf_vars and I
> just sent a fix to disable per-CPU var BTF generation for versions
> 1.18 through 1.21. I think that's a better solution than all the

That is already in akpm's tree, cool.

I also forgot that I asked Han to have a way to disable this new feature
as it had gone thru several back and forths so could still have some
problem.

> previously proposed ones. But it also means we have no good reason to
> force 1.22+ as minimal version.

> But in either case, this is good feature and will definitely be useful
> going forward. See my comments below.

Sure
 
> > commit b579a18a1ea0ee84b90b5302f597dda2edf2f61b
> > Author: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Date:   Fri May 28 16:41:30 2021 -0300
> >
> >     pahole: Allow encoding BTF into a detached file
> >
> >     Previously the newly encoded BTF info was stored into a ELF section in
> >     the file where the DWARF info was obtained, but it is useful to just
> >     dump it into a separate file, do it.
> >
> >     Requested-by: Andrii Nakryiko <andrii@kernel.org>
> >     Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> >
> 
> Looks good, see few minor comments below. At some point it probably
> would make sense to formalize "btf_encoder" as a struct with its own
> state instead of passing in multiple variables. It would probably also

Yeah, this all was made in haste, to have features out of the door ASAP,
etc. I hate global variables and this code is full of it.

> allow to parallelize BTF generation, where each CU would proceed in
> parallel generating local BTF, and then the final pass would merge and
> dedup BTFs. Currently reading and processing DWARF is the slowest part

yeah, would be wonderful to have someone working on this.

> of the DWARF-to-BTF conversion, parallelization and maybe some other
> optimization seems like the only way to speed the process up.

agreed
 
> Acked-by: Andrii Nakryiko <andrii@kernel.org>
> 
> > diff --git a/btf_encoder.c b/btf_encoder.c
> > index 033c927b537dad1e..bc3ac72968cea826 100644
> > --- a/btf_encoder.c
> > +++ b/btf_encoder.c
> > @@ -21,6 +21,14 @@
> >  #include <stdlib.h> /* for qsort() and bsearch() */
> >  #include <inttypes.h>
> >
> > +#include <sys/types.h>
> > +#include <sys/stat.h>
> > +#include <fcntl.h>
> > +
> > +#include <unistd.h>
> > +
> > +#include <errno.h>
> > +
> >  /*
> >   * This corresponds to the same macro defined in
> >   * include/linux/kallsyms.h
> > @@ -267,14 +275,62 @@ static struct btf_elf *btfe;
> >  static uint32_t array_index_id;
> >  static bool has_index_type;
> >
> > -int btf_encoder__encode()
> > +static int btf_encoder__dump(struct btf *btf, const char *filename)
> > +{
> > +       uint32_t raw_btf_size;
> > +       const void *raw_btf_data;
> > +       int fd, err;
> > +
> > +       /* Empty file, nothing to do, so... done! */
> > +       if (btf__get_nr_types(btf) == 0)
> > +               return 0;
> > +
> > +       if (btf__dedup(btf, NULL, NULL)) {
> > +               fprintf(stderr, "%s: btf__dedup failed!\n", __func__);
> > +               return -1;
> > +       }
> > +
> > +       raw_btf_data = btf__get_raw_data(btf, &raw_btf_size);
> > +       if (raw_btf_data == NULL) {
> > +                fprintf(stderr, "%s: btf__get_raw_data failed!\n", __func__);
> 
> indentation seems off here and in few places below

Yeah, I fixed those now
 
> > +               return -1;
> > +       }
> > +
> > +       fd = open(filename, O_WRONLY | O_CREAT);
> > +       if (fd < 0) {
> > +                fprintf(stderr, "%s: Couldn't open %s for writing the raw BTF info: %s\n", __func__, filename, strerror(errno));
> > +               return -1;
> > +       }
> > +       err = write(fd, raw_btf_data, raw_btf_size);
> > +       if (err < 0)
> > +                fprintf(stderr, "%s: Couldn't open %s for writing the raw BTF info: %s\n", __func__, filename, strerror(errno));
> 
> nit: copy-pasted error message is wrong

fixed
 
> > +
> > +       close(fd);
> > +
> > +       if (err != raw_btf_size) {
> > +                fprintf(stderr, "%s: Could only write %d bytes to %s of raw BTF info out of %d, aborting\n", __func__, err, filename, raw_btf_size);
> > +               unlink(filename);
> > +               err = -1;
> > +       } else {
> > +               /* go from bytes written == raw_btf_size to an indication that all went fine */
> > +               err = 0;
> > +       }
> > +
> > +       return err;
> > +}
> > +
> 
> [...]

-- 

- Arnaldo