* pahole -J usage in kernel scripts/link-vmlinux.sh
@ 2021-03-17 23:26 Fangrui Song
2021-03-19 22:44 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Fangrui Song @ 2021-03-17 23:26 UTC (permalink / raw)
To: dwarves; +Cc: Bill Wendling
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
Why doesn't pahole provide a command generating an object file with just the .BTF section?
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
sh_offset != sh_addr (mod max-page-size)
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
+ [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
Place the new non-SHF_ALLOC .BTF section at the end.
Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pahole -J usage in kernel scripts/link-vmlinux.sh
2021-03-17 23:26 pahole -J usage in kernel scripts/link-vmlinux.sh Fangrui Song
@ 2021-03-19 22:44 ` Andrii Nakryiko
2021-03-19 23:30 ` Bill Wendling
0 siblings, 1 reply; 6+ messages in thread
From: Andrii Nakryiko @ 2021-03-19 22:44 UTC (permalink / raw)
To: Fangrui Song; +Cc: dwarves, Bill Wendling
On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
>
> Hi BTF folks,
>
> I have discovered some problems with pahole -J.
> Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
>
> ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> pahole -J .tmp_vmlinux.btf
> llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
>
> pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> Why doesn't pahole provide a command generating an object file with just the .BTF section?
>
We just recently discussed adding this. So the reason is that
historically pahole never had this feature and no one bothered to add
it.
>
>
> When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
I don't think anything changed in 1.20 about how .BTF is added. pahole
still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
just double-checked that after running pahole -j .tmp_vmlinux.btf has
non-allocatable .BTF section:
$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
[13] .BTF_ids PROGBITS ffffffff822b6804 14b6804
000510 00 A 0 0 1
[43] .BTF PROGBITS 0000000000000000 18bc0e8c
4c8056 00 0 0 1
> This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
>
> In particular, there are two bugs:
>
> * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> sh_offset != sh_addr (mod max-page-size)
>
> Section Headers:
> [Nr] Name Type Address Off Size ES Flg Lk Inf Al
> [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
> - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
> + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
>
> * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> Because of the first bug, pahole -J rewritten object file generally has small offsets.
> If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
>
> error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
>
> This is very rare, though.
>
>
> So I suggest:
>
> * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
so nothing changed (at least as of 1.20) about how pahole adds .BTF,
so I'd like to understand why our observations differ
> Place the new non-SHF_ALLOC .BTF section at the end.
> Then not rewriting PT_LOAD will be justified.
> * Add a new option incorporating the current pahole -J + llvm-objcopy
pahole is already using llvm-objcopy. Do you mean to add a mode where
pahole will dump a separate ELF object file with just .BTF? Or
something else?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pahole -J usage in kernel scripts/link-vmlinux.sh
2021-03-19 22:44 ` Andrii Nakryiko
@ 2021-03-19 23:30 ` Bill Wendling
2021-03-19 23:38 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Bill Wendling @ 2021-03-19 23:30 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: Fangrui Song, dwarves
On Fri, Mar 19, 2021 at 3:44 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
> >
> > Hi BTF folks,
> >
> > I have discovered some problems with pahole -J.
> > Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
> >
> > ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> > pahole -J .tmp_vmlinux.btf
> > llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
> >
> > pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> > Why doesn't pahole provide a command generating an object file with just the .BTF section?
> >
>
> We just recently discussed adding this. So the reason is that
> historically pahole never had this feature and no one bothered to add
> it.
>
> >
> >
> > When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> > I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
>
> I don't think anything changed in 1.20 about how .BTF is added. pahole
> still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
> just double-checked that after running pahole -j .tmp_vmlinux.btf has
> non-allocatable .BTF section:
>
> $ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
> [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804
> 000510 00 A 0 0 1
> [43] .BTF PROGBITS 0000000000000000 18bc0e8c
> 4c8056 00 0 0 1
>
>
> > This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
> >
> > In particular, there are two bugs:
> >
> > * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> > sh_offset != sh_addr (mod max-page-size)
> >
> > Section Headers:
> > [Nr] Name Type Address Off Size ES Flg Lk Inf Al
> > [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
> > - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
> > + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
> >
> > * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> > Because of the first bug, pahole -J rewritten object file generally has small offsets.
> > If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> > a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
> >
> > error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
> >
> > This is very rare, though.
> >
> >
> > So I suggest:
> >
> > * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
>
> so nothing changed (at least as of 1.20) about how pahole adds .BTF,
> so I'd like to understand why our observations differ
>
Here's one things we're seeing (note, we're using a kernel based on
4.15). Before we run pahole, we have this. Notice the offset of the
'.text' section:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1
After running pahole, we get this:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1
The offset of '.text' changed. This is because `elf_update` decides
what the offsets should be. Did the linker scripts change between 4.15
and top-of-tree to mark .BTF as non-allocatable?
-bw
> > Place the new non-SHF_ALLOC .BTF section at the end.
> > Then not rewriting PT_LOAD will be justified.
> > * Add a new option incorporating the current pahole -J + llvm-objcopy
>
> pahole is already using llvm-objcopy. Do you mean to add a mode where
> pahole will dump a separate ELF object file with just .BTF? Or
> something else?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pahole -J usage in kernel scripts/link-vmlinux.sh
2021-03-19 23:30 ` Bill Wendling
@ 2021-03-19 23:38 ` Andrii Nakryiko
2021-03-19 23:47 ` Bill Wendling
0 siblings, 1 reply; 6+ messages in thread
From: Andrii Nakryiko @ 2021-03-19 23:38 UTC (permalink / raw)
To: Bill Wendling; +Cc: Fangrui Song, dwarves
On Fri, Mar 19, 2021 at 4:30 PM Bill Wendling <morbo@google.com> wrote:
>
> On Fri, Mar 19, 2021 at 3:44 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
> > >
> > > Hi BTF folks,
> > >
> > > I have discovered some problems with pahole -J.
> > > Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
> > >
> > > ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> > > pahole -J .tmp_vmlinux.btf
> > > llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
> > >
> > > pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> > > Why doesn't pahole provide a command generating an object file with just the .BTF section?
> > >
> >
> > We just recently discussed adding this. So the reason is that
> > historically pahole never had this feature and no one bothered to add
> > it.
> >
> > >
> > >
> > > When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> > > I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
> >
> > I don't think anything changed in 1.20 about how .BTF is added. pahole
> > still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
> > just double-checked that after running pahole -j .tmp_vmlinux.btf has
> > non-allocatable .BTF section:
> >
> > $ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
> > [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804
> > 000510 00 A 0 0 1
> > [43] .BTF PROGBITS 0000000000000000 18bc0e8c
> > 4c8056 00 0 0 1
> >
> >
> > > This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
> > >
> > > In particular, there are two bugs:
> > >
> > > * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> > > sh_offset != sh_addr (mod max-page-size)
> > >
> > > Section Headers:
> > > [Nr] Name Type Address Off Size ES Flg Lk Inf Al
> > > [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
> > > - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
> > > + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
> > >
> > > * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> > > Because of the first bug, pahole -J rewritten object file generally has small offsets.
> > > If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> > > a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
> > >
> > > error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
> > >
> > > This is very rare, though.
> > >
> > >
> > > So I suggest:
> > >
> > > * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
> >
> > so nothing changed (at least as of 1.20) about how pahole adds .BTF,
> > so I'd like to understand why our observations differ
> >
> Here's one things we're seeing (note, we're using a kernel based on
> 4.15). Before we run pahole, we have this. Notice the offset of the
> '.text' section:
>
> [ 0] NULL 0000000000000000 000000
> 000000 00 0 0 0
> [ 1] .text PROGBITS ffffffff81000000 200000
> e0169a 00 AX 0 0 4096
> [ 2] .notes NOTE ffffffff81e0169c 100169c
> 000024 00 A 0 0 4
> ...
> [24] .BTF PROGBITS ffffffff825caf40 17caf40
> 000000 00 WA 0 0 1
> [25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
> 0004c0 00 A 0 0 1
>
> After running pahole, we get this:
>
> [ 0] NULL 0000000000000000 000000
> 000000 00 0 0 0
> [ 1] .text PROGBITS ffffffff81000000 001000
> e0169a 00 AX 0 0 4096
> [ 2] .notes NOTE ffffffff81e0169c e0269c
> 000024 00 A 0 0 4
> ...
> [24] .BTF PROGBITS ffffffff825caf40 11ecf40
> 3c1ebe 00 WA 0 0 1
> [25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
> 0004c0 00 A 0 0 1
>
> The offset of '.text' changed. This is because `elf_update` decides
> what the offsets should be. Did the linker scripts change between 4.15
> and top-of-tree to mark .BTF as non-allocatable?
maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not
used anymore. We only dump .BTF section into a separate ELF, which is
linked into the final vmlinux image as the next step. So pahole
doesn't rewrite the final binary.
>
> -bw
>
> > > Place the new non-SHF_ALLOC .BTF section at the end.
> > > Then not rewriting PT_LOAD will be justified.
> > > * Add a new option incorporating the current pahole -J + llvm-objcopy
> >
> > pahole is already using llvm-objcopy. Do you mean to add a mode where
> > pahole will dump a separate ELF object file with just .BTF? Or
> > something else?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pahole -J usage in kernel scripts/link-vmlinux.sh
2021-03-19 23:38 ` Andrii Nakryiko
@ 2021-03-19 23:47 ` Bill Wendling
2021-03-20 17:21 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Bill Wendling @ 2021-03-19 23:47 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: Fangrui Song, dwarves
On Fri, Mar 19, 2021 at 4:38 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Fri, Mar 19, 2021 at 4:30 PM Bill Wendling <morbo@google.com> wrote:
> >
> > On Fri, Mar 19, 2021 at 3:44 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
> > > >
> > > > Hi BTF folks,
> > > >
> > > > I have discovered some problems with pahole -J.
> > > > Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
> > > >
> > > > ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> > > > pahole -J .tmp_vmlinux.btf
> > > > llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
> > > >
> > > > pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> > > > Why doesn't pahole provide a command generating an object file with just the .BTF section?
> > > >
> > >
> > > We just recently discussed adding this. So the reason is that
> > > historically pahole never had this feature and no one bothered to add
> > > it.
> > >
> > > >
> > > >
> > > > When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> > > > I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
> > >
> > > I don't think anything changed in 1.20 about how .BTF is added. pahole
> > > still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
> > > just double-checked that after running pahole -j .tmp_vmlinux.btf has
> > > non-allocatable .BTF section:
> > >
> > > $ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
> > > [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804
> > > 000510 00 A 0 0 1
> > > [43] .BTF PROGBITS 0000000000000000 18bc0e8c
> > > 4c8056 00 0 0 1
> > >
> > >
> > > > This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
> > > >
> > > > In particular, there are two bugs:
> > > >
> > > > * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> > > > sh_offset != sh_addr (mod max-page-size)
> > > >
> > > > Section Headers:
> > > > [Nr] Name Type Address Off Size ES Flg Lk Inf Al
> > > > [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
> > > > - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
> > > > + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
> > > >
> > > > * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> > > > Because of the first bug, pahole -J rewritten object file generally has small offsets.
> > > > If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> > > > a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
> > > >
> > > > error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
> > > >
> > > > This is very rare, though.
> > > >
> > > >
> > > > So I suggest:
> > > >
> > > > * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
> > >
> > > so nothing changed (at least as of 1.20) about how pahole adds .BTF,
> > > so I'd like to understand why our observations differ
> > >
> > Here's one things we're seeing (note, we're using a kernel based on
> > 4.15). Before we run pahole, we have this. Notice the offset of the
> > '.text' section:
> >
> > [ 0] NULL 0000000000000000 000000
> > 000000 00 0 0 0
> > [ 1] .text PROGBITS ffffffff81000000 200000
> > e0169a 00 AX 0 0 4096
> > [ 2] .notes NOTE ffffffff81e0169c 100169c
> > 000024 00 A 0 0 4
> > ...
> > [24] .BTF PROGBITS ffffffff825caf40 17caf40
> > 000000 00 WA 0 0 1
> > [25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
> > 0004c0 00 A 0 0 1
> >
> > After running pahole, we get this:
> >
> > [ 0] NULL 0000000000000000 000000
> > 000000 00 0 0 0
> > [ 1] .text PROGBITS ffffffff81000000 001000
> > e0169a 00 AX 0 0 4096
> > [ 2] .notes NOTE ffffffff81e0169c e0269c
> > 000024 00 A 0 0 4
> > ...
> > [24] .BTF PROGBITS ffffffff825caf40 11ecf40
> > 3c1ebe 00 WA 0 0 1
> > [25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
> > 0004c0 00 A 0 0 1
> >
> > The offset of '.text' changed. This is because `elf_update` decides
> > what the offsets should be. Did the linker scripts change between 4.15
> > and top-of-tree to mark .BTF as non-allocatable?
>
> maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not
> used anymore. We only dump .BTF section into a separate ELF, which is
> linked into the final vmlinux image as the next step. So pahole
> doesn't rewrite the final binary.
>
Right. The problem is with the follow-up command:
${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
--strip-all ${1} ${2} 2>/dev/null
The file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation),
so this command fails to write a valid ELF file.
-bw
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pahole -J usage in kernel scripts/link-vmlinux.sh
2021-03-19 23:47 ` Bill Wendling
@ 2021-03-20 17:21 ` Andrii Nakryiko
0 siblings, 0 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2021-03-20 17:21 UTC (permalink / raw)
To: Bill Wendling; +Cc: Fangrui Song, dwarves
On Fri, Mar 19, 2021 at 4:47 PM Bill Wendling <morbo@google.com> wrote:
>
> On Fri, Mar 19, 2021 at 4:38 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Fri, Mar 19, 2021 at 4:30 PM Bill Wendling <morbo@google.com> wrote:
> > >
> > > On Fri, Mar 19, 2021 at 3:44 PM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Wed, Mar 17, 2021 at 4:28 PM Fangrui Song <maskray@google.com> wrote:
> > > > >
> > > > > Hi BTF folks,
> > > > >
> > > > > I have discovered some problems with pahole -J.
> > > > > Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
> > > > >
> > > > > ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group
> > > > > pahole -J .tmp_vmlinux.btf
> > > > > llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
> > > > >
> > > > > pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section.
> > > > > Why doesn't pahole provide a command generating an object file with just the .BTF section?
> > > > >
> > > >
> > > > We just recently discussed adding this. So the reason is that
> > > > historically pahole never had this feature and no one bothered to add
> > > > it.
> > > >
> > > > >
> > > > >
> > > > > When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 ,
> > > > > I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.
> > > >
> > > > I don't think anything changed in 1.20 about how .BTF is added. pahole
> > > > still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I
> > > > just double-checked that after running pahole -j .tmp_vmlinux.btf has
> > > > non-allocatable .BTF section:
> > > >
> > > > $ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
> > > > [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804
> > > > 000510 00 A 0 0 1
> > > > [43] .BTF PROGBITS 0000000000000000 18bc0e8c
> > > > 4c8056 00 0 0 1
> > > >
> > > >
> > > > > This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
> > > > >
> > > > > In particular, there are two bugs:
> > > > >
> > > > > * pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its
> > > > > sh_offset != sh_addr (mod max-page-size)
> > > > >
> > > > > Section Headers:
> > > > > [Nr] Name Type Address Off Size ES Flg Lk Inf Al
> > > > > [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
> > > > > - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096
> > > > > + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
> > > > >
> > > > > * pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
> > > > > Because of the first bug, pahole -J rewritten object file generally has small offsets.
> > > > > If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within
> > > > > a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
> > > > >
> > > > > error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
> > > > >
> > > > > This is very rare, though.
> > > > >
> > > > >
> > > > > So I suggest:
> > > > >
> > > > > * pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.
> > > >
> > > > so nothing changed (at least as of 1.20) about how pahole adds .BTF,
> > > > so I'd like to understand why our observations differ
> > > >
> > > Here's one things we're seeing (note, we're using a kernel based on
> > > 4.15). Before we run pahole, we have this. Notice the offset of the
> > > '.text' section:
> > >
> > > [ 0] NULL 0000000000000000 000000
> > > 000000 00 0 0 0
> > > [ 1] .text PROGBITS ffffffff81000000 200000
> > > e0169a 00 AX 0 0 4096
> > > [ 2] .notes NOTE ffffffff81e0169c 100169c
> > > 000024 00 A 0 0 4
> > > ...
> > > [24] .BTF PROGBITS ffffffff825caf40 17caf40
> > > 000000 00 WA 0 0 1
> > > [25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
> > > 0004c0 00 A 0 0 1
> > >
> > > After running pahole, we get this:
> > >
> > > [ 0] NULL 0000000000000000 000000
> > > 000000 00 0 0 0
> > > [ 1] .text PROGBITS ffffffff81000000 001000
> > > e0169a 00 AX 0 0 4096
> > > [ 2] .notes NOTE ffffffff81e0169c e0269c
> > > 000024 00 A 0 0 4
> > > ...
> > > [24] .BTF PROGBITS ffffffff825caf40 11ecf40
> > > 3c1ebe 00 WA 0 0 1
> > > [25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
> > > 0004c0 00 A 0 0 1
> > >
> > > The offset of '.text' changed. This is because `elf_update` decides
> > > what the offsets should be. Did the linker scripts change between 4.15
> > > and top-of-tree to mark .BTF as non-allocatable?
> >
> > maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not
> > used anymore. We only dump .BTF section into a separate ELF, which is
> > linked into the final vmlinux image as the next step. So pahole
> > doesn't rewrite the final binary.
> >
> Right. The problem is with the follow-up command:
>
> ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
> --strip-all ${1} ${2} 2>/dev/null
>
> The file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation),
> so this command fails to write a valid ELF file.
Oh, I see. Yeah, I didn't get that from the original email. So seems
like llvm-objcopy that pahole invokes internally will corrupt that
.tmp_vmlinux.btf file to the point of another objcopy not being able
to handle it. Interesting.
But regardless, as I said, we already discussed this feature and think
that it's a good addition. I'd like to be able to both get object file
with just .BTF (for kernel linking process) and pure binary contents
of .BTF section (for other, CO-RE-related reasons, especially for
legacy systems). I think Arnaldo is onboard as well. So please feel
free to send patches.
>
> -bw
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-03-20 17:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-17 23:26 pahole -J usage in kernel scripts/link-vmlinux.sh Fangrui Song
2021-03-19 22:44 ` Andrii Nakryiko
2021-03-19 23:30 ` Bill Wendling
2021-03-19 23:38 ` Andrii Nakryiko
2021-03-19 23:47 ` Bill Wendling
2021-03-20 17:21 ` Andrii Nakryiko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).