[PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sean Christopherson]

Merge module sections only when using Clang LTO.  With gcc-10, merging
sections does not appear to update the symbol tables for the module,
e.g. 'readelf -s' shows the value that a symbol would have had, if
sections were not merged.

The stale symbol table breaks gdb's function disassambler, and presumably
other things, e.g.

  gdb -batch -ex "file arch/x86/kvm/kvm.ko" -ex "disassemble kvm_init"

reads the wrong bytes and dumps garbage.

Fixes: dd2776222abb ("kbuild: lto: merge module sections")
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---

This is obviously the quick and dirty approach to fixing the problem,
presumably there is a way to actually update the symbols, but that is far,
far outside my area of expertise.

IIUC how the disassemblers work, the section headers are correctly updated,
e.g. objdump displays the correct info, and even gdb's disassembler shows
the "correct" offset, it's just the symbol tables that are out of whack.

An earlier version of the LTO series did have exactly this #ifdef, but
it was dropped when no one objected to Kees' suggestion to unconditionally
merge sections.  https://lkml.kernel.org/r/202010141548.47CB1BC@keescook

 scripts/module.lds.S | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index 168cd27e6122..3580c6d02957 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -25,6 +25,7 @@ SECTIONS {
 	 * -ffunction-sections, which increases the size of the final module.
 	 * Merge the split sections in the final binary.
 	 */
+#ifdef CONFIG_LTO_CLANG
 	.bss : {
 		*(.bss .bss.[0-9a-zA-Z_]*)
 		*(.bss..L*)
@@ -41,6 +42,7 @@ SECTIONS {
 	}
 
 	.text : { *(.text .text.[0-9a-zA-Z_]*) }
+#endif
 }
 
 /* bring in arch-specific sections */
-- 
2.31.0.rc2.261.g7f71774620-goog

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sami Tolvanen]

On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Merge module sections only when using Clang LTO.  With gcc-10, merging
> sections does not appear to update the symbol tables for the module,
> e.g. 'readelf -s' shows the value that a symbol would have had, if
> sections were not merged.

I'm fine with limiting this to LTO only, but it would be helpful to
understand which sections are actually getting merged here. Are you
compiling the kernel with -ffunction-sections and/or -fdata-sections?
Does this issue only happen with gcc 10?

Sami

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sean Christopherson]

On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > sections does not appear to update the symbol tables for the module,
> > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > sections were not merged.
> 
> I'm fine with limiting this to LTO only, but it would be helpful to
> understand which sections are actually getting merged here.

It doesn't appear to matter which sections get merged, the tables only show the
correct data if there is no merging whatsoever, e.g. allowing merging for any
one of the four types (.bss, .data, .rodata and .text) results in breakage.
AFAICT, merging any sections causes the layout to change and throw off the
symbol tables. 

> Are you compiling the kernel with -ffunction-sections and/or -fdata-sections?

I tried both.  Default off, and forcing those flags by hacking the Makefile had
no effect.

> Does this issue only happen with gcc 10?

gcc-7 shows the same behavior, I haven't checked anything older or anything in
between.

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sami Tolvanen]

On Tue, Mar 23, 2021 at 9:36 AM Sean Christopherson <seanjc@google.com> wrote:
>
> On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > > sections does not appear to update the symbol tables for the module,
> > > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > > sections were not merged.
> >
> > I'm fine with limiting this to LTO only, but it would be helpful to
> > understand which sections are actually getting merged here.
>
> It doesn't appear to matter which sections get merged, the tables only show the
> correct data if there is no merging whatsoever, e.g. allowing merging for any
> one of the four types (.bss, .data, .rodata and .text) results in breakage.
> AFAICT, merging any sections causes the layout to change and throw off the
> symbol tables.

Thanks for the clarification. I can reproduce this issue with gcc +
bfd if any of the sections are merged, but gcc + lld produces valid
symbol tables.

Perhaps someone more familiar with bfd can comment on whether this is
a bug or a feature, and if there's a flag we can pass to bfd that
would fix the issue. In the meanwhile, this patch looks like a
reasonable workaround to me.

Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Sami Tolvanen <samitolvanen@google.com>

Sami

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sean Christopherson]

On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> On Tue, Mar 23, 2021 at 9:36 AM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> > > >
> > > > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > > > sections does not appear to update the symbol tables for the module,
> > > > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > > > sections were not merged.
> > >
> > > I'm fine with limiting this to LTO only, but it would be helpful to
> > > understand which sections are actually getting merged here.
> >
> > It doesn't appear to matter which sections get merged, the tables only show the
> > correct data if there is no merging whatsoever, e.g. allowing merging for any
> > one of the four types (.bss, .data, .rodata and .text) results in breakage.
> > AFAICT, merging any sections causes the layout to change and throw off the
> > symbol tables.
> 
> Thanks for the clarification. I can reproduce this issue with gcc +
> bfd if any of the sections are merged, but gcc + lld produces valid
> symbol tables.

FWIW, clang + bfd also produces mangled tables, so it does appear to be bfd
specific.

> Perhaps someone more familiar with bfd can comment on whether this is
> a bug or a feature, and if there's a flag we can pass to bfd that
> would fix the issue. In the meanwhile, this patch looks like a
> reasonable workaround to me.
> 
> Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
> Tested-by: Sami Tolvanen <samitolvanen@google.com>
> 
> Sami

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Kees Cook]

On Wed, Mar 24, 2021 at 10:45:36PM +0000, Sean Christopherson wrote:
> On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > On Tue, Mar 23, 2021 at 9:36 AM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > > On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> > > > >
> > > > > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > > > > sections does not appear to update the symbol tables for the module,
> > > > > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > > > > sections were not merged.
> > > >
> > > > I'm fine with limiting this to LTO only, but it would be helpful to
> > > > understand which sections are actually getting merged here.
> > >
> > > It doesn't appear to matter which sections get merged, the tables only show the
> > > correct data if there is no merging whatsoever, e.g. allowing merging for any
> > > one of the four types (.bss, .data, .rodata and .text) results in breakage.
> > > AFAICT, merging any sections causes the layout to change and throw off the
> > > symbol tables.
> > 
> > Thanks for the clarification. I can reproduce this issue with gcc +
> > bfd if any of the sections are merged, but gcc + lld produces valid
> > symbol tables.
> 
> FWIW, clang + bfd also produces mangled tables, so it does appear to be bfd
> specific.

Are you able to open a bug against bfd for this?

> > Perhaps someone more familiar with bfd can comment on whether this is
> > a bug or a feature, and if there's a flag we can pass to bfd that
> > would fix the issue. In the meanwhile, this patch looks like a
> > reasonable workaround to me.
> > 
> > Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
> > Tested-by: Sami Tolvanen <samitolvanen@google.com>

Thanks, I'll get this sent to Linus.

-- 
Kees Cook

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Kees Cook]

On Mon, 22 Mar 2021 16:44:38 -0700, Sean Christopherson wrote:
> Merge module sections only when using Clang LTO.  With gcc-10, merging
> sections does not appear to update the symbol tables for the module,
> e.g. 'readelf -s' shows the value that a symbol would have had, if
> sections were not merged.
> 
> The stale symbol table breaks gdb's function disassambler, and presumably
> other things, e.g.
> 
> [...]

Applied to for-linus/lto, thanks!

[1/1] kbuild: lto: Merge module sections if and only if CONFIG_LTO_CLANG is enabled
      https://git.kernel.org/kees/c/8b382ebc86a8

-- 
Kees Cook

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Sean Christopherson]

On Wed, Mar 31, 2021, Kees Cook wrote:
> On Wed, Mar 24, 2021 at 10:45:36PM +0000, Sean Christopherson wrote:
> > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > On Tue, Mar 23, 2021 at 9:36 AM Sean Christopherson <seanjc@google.com> wrote:
> > > >
> > > > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > > > On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> > > > > >
> > > > > > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > > > > > sections does not appear to update the symbol tables for the module,
> > > > > > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > > > > > sections were not merged.
> > > > >
> > > > > I'm fine with limiting this to LTO only, but it would be helpful to
> > > > > understand which sections are actually getting merged here.
> > > >
> > > > It doesn't appear to matter which sections get merged, the tables only show the
> > > > correct data if there is no merging whatsoever, e.g. allowing merging for any
> > > > one of the four types (.bss, .data, .rodata and .text) results in breakage.
> > > > AFAICT, merging any sections causes the layout to change and throw off the
> > > > symbol tables.
> > > 
> > > Thanks for the clarification. I can reproduce this issue with gcc +
> > > bfd if any of the sections are merged, but gcc + lld produces valid
> > > symbol tables.
> > 
> > FWIW, clang + bfd also produces mangled tables, so it does appear to be bfd
> > specific.
> 
> Are you able to open a bug against bfd for this?

Yes?  I'll ping you when I run into trouble ;-)

Re: [PATCH] kbuild: Merge module sections if and only if CONFIG_LTO_CLANG is enabled

[Kees Cook]

On Wed, Mar 31, 2021 at 08:07:18PM +0000, Sean Christopherson wrote:
> On Wed, Mar 31, 2021, Kees Cook wrote:
> > On Wed, Mar 24, 2021 at 10:45:36PM +0000, Sean Christopherson wrote:
> > > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > > On Tue, Mar 23, 2021 at 9:36 AM Sean Christopherson <seanjc@google.com> wrote:
> > > > >
> > > > > On Tue, Mar 23, 2021, Sami Tolvanen wrote:
> > > > > > On Mon, Mar 22, 2021 at 4:44 PM Sean Christopherson <seanjc@google.com> wrote:
> > > > > > >
> > > > > > > Merge module sections only when using Clang LTO.  With gcc-10, merging
> > > > > > > sections does not appear to update the symbol tables for the module,
> > > > > > > e.g. 'readelf -s' shows the value that a symbol would have had, if
> > > > > > > sections were not merged.
> > > > > >
> > > > > > I'm fine with limiting this to LTO only, but it would be helpful to
> > > > > > understand which sections are actually getting merged here.
> > > > >
> > > > > It doesn't appear to matter which sections get merged, the tables only show the
> > > > > correct data if there is no merging whatsoever, e.g. allowing merging for any
> > > > > one of the four types (.bss, .data, .rodata and .text) results in breakage.
> > > > > AFAICT, merging any sections causes the layout to change and throw off the
> > > > > symbol tables.
> > > > 
> > > > Thanks for the clarification. I can reproduce this issue with gcc +
> > > > bfd if any of the sections are merged, but gcc + lld produces valid
> > > > symbol tables.
> > > 
> > > FWIW, clang + bfd also produces mangled tables, so it does appear to be bfd
> > > specific.
> > 
> > Are you able to open a bug against bfd for this?
> 
> Yes?  I'll ping you when I run into trouble ;-)

Hm, I can't tell if any of these are duplicates:

https://sourceware.org/bugzilla/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=SUSPENDED&bug_status=WAITING&bug_status=REOPENED&component=ld&list_id=59962&product=binutils&query_format=advanced&short_desc=symbol&short_desc_type=allwordssubstr

-- 
Kees Cook