Re: pahole -J usage in kernel scripts/link-vmlinux.sh

[Bill Wendling <morbo@google.com>]

On Fri, Mar 19, 2021 at 3:44 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
> >
> > Hi BTF folks,
> >
> > I have discovered some problems with pahole -J.
> > Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
> >
> > ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> > pahole -J .tmp_vmlinux.btf
> > llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
> >
> > pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> > Why doesn't pahole provide a command generating an object file with just the .BTF section?
> >
>
> We just recently discussed adding this. So the reason is that
> historically pahole never had this feature and no one bothered to add
> it.
>
> >
> >
> > When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> > I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
>
> I don't think anything changed in 1.20 about how .BTF is added. pahole
> still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
> just double-checked that after running pahole -j .tmp_vmlinux.btf has
> non-allocatable .BTF section:
>
> $ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
>   [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
> 000510 00   A  0   0  1
>   [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
> 4c8056 00      0   0  1
>
>
> > This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
> >
> > In particular, there are two bugs:
> >
> > * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> >    sh_offset != sh_addr (mod max-page-size)
> >
> >   Section Headers:
> >     [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
> >     [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
> > -  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
> > +  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096
> >
> > * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> >    Because of the first bug, pahole -J rewritten object file generally has small offsets.
> >    If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> >    a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
> >
> >    error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
> >
> >    This is very rare, though.
> >
> >
> > So I suggest:
> >
> > * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
>
> so nothing changed (at least as of 1.20) about how pahole adds .BTF,
> so I'd like to understand why our observations differ
>
Here's one things we're seeing (note, we're using a kernel based on
4.15). Before we run pahole, we have this. Notice the offset of the
'.text' section:

  [ 0]                   NULL            0000000000000000 000000
000000 00      0   0  0
  [ 1] .text             PROGBITS        ffffffff81000000 200000
e0169a 00  AX  0   0 4096
  [ 2] .notes            NOTE            ffffffff81e0169c 100169c
000024 00   A  0   0  4
...
  [24] .BTF              PROGBITS        ffffffff825caf40 17caf40
000000 00  WA  0   0  1
  [25] .BTF_ids          PROGBITS        ffffffff825caf40 17caf40
0004c0 00   A  0   0  1

After running pahole, we get this:

  [ 0]                   NULL            0000000000000000 000000
000000 00      0   0  0
  [ 1] .text             PROGBITS        ffffffff81000000 001000
e0169a 00  AX  0   0 4096
  [ 2] .notes            NOTE            ffffffff81e0169c e0269c
000024 00   A  0   0  4
...
  [24] .BTF              PROGBITS        ffffffff825caf40 11ecf40
3c1ebe 00  WA  0   0  1
  [25] .BTF_ids          PROGBITS        ffffffff825caf40 15aedfe
0004c0 00   A  0   0  1

The offset of '.text' changed. This is because `elf_update` decides
what the offsets should be. Did the linker scripts change between 4.15
and top-of-tree to mark .BTF as non-allocatable?

-bw

> >    Place the new non-SHF_ALLOC .BTF section at the end.
> >    Then not rewriting PT_LOAD will be justified.
> > * Add a new option incorporating the current pahole -J + llvm-objcopy
>
> pahole is already using llvm-objcopy. Do you mean to add a mode where
> pahole will dump a separate ELF object file with just .BTF? Or
> something else?