eh_frame confusion

eh_frame confusion

[Rasmus Villemoes]

I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
to gcc-8 all object files now end up having .eh_frame section. For
vmlinux, that's not a problem, because they all get discarded in
arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
modules, which doesn't seem to be useful - given that everything worked
just fine with gcc-7, and I don't see anything in the module loader that
handles .eh_frame.

The reason I care is that my target has a rather tight rootfs budget,
and the .eh_frame section seem to occupy 10-30% of the file size
(obviously very depending on the particular module).

Comparing the .foo.o.cmd files, I don't see change in options that might
explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
is apparently no longer supported in gcc-8). Both before and after, there's

-fno-dwarf2-cfi-asm

about which gcc's documentation says

'-fno-dwarf2-cfi-asm'
     Emit DWARF unwind info as compiler generated '.eh_frame' section
     instead of using GAS '.cfi_*' directives.

Looking into where that comes from got me even more confused, because
both arm and unicore32 say

# Never generate .eh_frame
KBUILD_CFLAGS           += $(call cc-option,-fno-dwarf2-cfi-asm)

while the ppc32 case at hand says

# FIXME: the module load should be taught about the additional relocs
# generated by this.
# revert to pre-gcc-4.4 behaviour of .eh_frame

but prior to gcc-8, .eh_frame didn't seem to get generated anyway.

Can .eh_frame sections be discarded for modules (on ppc32 at least), or
is there some magic that makes them necessary when building with gcc-8?

Rasmus

Re: eh_frame confusion

[Segher Boessenkool]

On Mon, Mar 02, 2020 at 11:56:05AM +0100, Rasmus Villemoes wrote:
> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
> to gcc-8 all object files now end up having .eh_frame section.

Since GCC 8, we enable -fasynchronous-unwind-tables by default for
PowerPC.  See https://gcc.gnu.org/r259298 .

> For
> vmlinux, that's not a problem, because they all get discarded in
> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
> modules, which doesn't seem to be useful - given that everything worked
> just fine with gcc-7, and I don't see anything in the module loader that
> handles .eh_frame.

It is useful for debugging.  Not many people debug the kernel like this,
of course.


Segher

Re: eh_frame confusion

[Naveen N. Rao]

Segher Boessenkool wrote:
> On Mon, Mar 02, 2020 at 11:56:05AM +0100, Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section.
> 
> Since GCC 8, we enable -fasynchronous-unwind-tables by default for
> PowerPC.  See https://gcc.gnu.org/r259298 .
> 
>> For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
> 
> It is useful for debugging.  Not many people debug the kernel like this,
> of course.

I'm trying to understand if we need that. Other architectures seems to 
pass -fasynchronous-unwind-tables only for the vdso, but disable it for 
the kernel build. I suppose we can do the same.

If using -fno-asynchronous-unwind-tables, would crash/perf have 
problems?

- Naveen

eh_frame confusion

[Naveen N. Rao]

Rasmus Villemoes wrote:
> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
> to gcc-8 all object files now end up having .eh_frame section. For
> vmlinux, that's not a problem, because they all get discarded in
> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
> modules, which doesn't seem to be useful - given that everything worked
> just fine with gcc-7, and I don't see anything in the module loader that
> handles .eh_frame.
> 
> The reason I care is that my target has a rather tight rootfs budget,
> and the .eh_frame section seem to occupy 10-30% of the file size
> (obviously very depending on the particular module).
> 
> Comparing the .foo.o.cmd files, I don't see change in options that might
> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
> is apparently no longer supported in gcc-8). Both before and after, there's
> 
> -fno-dwarf2-cfi-asm
> 
> about which gcc's documentation says
> 
> '-fno-dwarf2-cfi-asm'
>      Emit DWARF unwind info as compiler generated '.eh_frame' section
>      instead of using GAS '.cfi_*' directives.
> 
> Looking into where that comes from got me even more confused, because
> both arm and unicore32 say
> 
> # Never generate .eh_frame
> KBUILD_CFLAGS           += $(call cc-option,-fno-dwarf2-cfi-asm)
> 
> while the ppc32 case at hand says
> 
> # FIXME: the module load should be taught about the additional relocs
> # generated by this.
> # revert to pre-gcc-4.4 behaviour of .eh_frame

Michael opened a task to look into this recently and I had spent some 
time last week on this. The original commit/discussion adding 
-fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being 
handled by our module loader:
http://lkml.kernel.org/r/20090224065112.GA6690@bombadil.infradead.org

However, that is now handled thanks to commit 9f751b82b491d:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d

I did a test build and a simple module loaded fine, so I think 
-fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen 
some breakages with it. Michael?

> 
> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
> 
> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
> is there some magic that makes them necessary when building with gcc-8?

As Segher points out, it looks like we need to add 
-fno-asynchronous-unwind-tables. Most other architectures seem to use 
that too.


- Naveen

eh_frame confusion

[Naveen N. Rao]

Naveen N. Rao wrote:
> Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section. For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
>> 
>> The reason I care is that my target has a rather tight rootfs budget,
>> and the .eh_frame section seem to occupy 10-30% of the file size
>> (obviously very depending on the particular module).
>> 
>> Comparing the .foo.o.cmd files, I don't see change in options that might
>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>> is apparently no longer supported in gcc-8). Both before and after, there's
>> 
>> -fno-dwarf2-cfi-asm
>> 
>> about which gcc's documentation says
>> 
>> '-fno-dwarf2-cfi-asm'
>>      Emit DWARF unwind info as compiler generated '.eh_frame' section
>>      instead of using GAS '.cfi_*' directives.
>> 
>> Looking into where that comes from got me even more confused, because
>> both arm and unicore32 say
>> 
>> # Never generate .eh_frame
>> KBUILD_CFLAGS           += $(call cc-option,-fno-dwarf2-cfi-asm)
>> 
>> while the ppc32 case at hand says
>> 
>> # FIXME: the module load should be taught about the additional relocs
>> # generated by this.
>> # revert to pre-gcc-4.4 behaviour of .eh_frame
> 
> Michael opened a task to look into this recently and I had spent some 
> time last week on this. The original commit/discussion adding 
> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being 
> handled by our module loader:
> http://lkml.kernel.org/r/20090224065112.GA6690@bombadil.infradead.org
> 
> However, that is now handled thanks to commit 9f751b82b491d:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
> 
> I did a test build and a simple module loaded fine, so I think 
> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen 
> some breakages with it. Michael?
> 
>> 
>> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
>> 
>> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
>> is there some magic that makes them necessary when building with gcc-8?
> 
> As Segher points out, it looks like we need to add 
> -fno-asynchronous-unwind-tables. Most other architectures seem to use 
> that too.

Can you check if the below patch works? I am yet to test this in more 
detail, but would be good to know the implications for ppc32.

- Naveen


---
diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index f35730548e42..5b5bf98b8217 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -239,10 +239,7 @@ KBUILD_CFLAGS += $(call cc-option,-mno-vsx)
 KBUILD_CFLAGS += $(call cc-option,-mno-spe)
 KBUILD_CFLAGS += $(call cc-option,-mspe=no)
 
-# FIXME: the module load should be taught about the additional relocs
-# generated by this.
-# revert to pre-gcc-4.4 behaviour of .eh_frame
-KBUILD_CFLAGS  += $(call cc-option,-fno-dwarf2-cfi-asm)
+KBUILD_CFLAGS  += $(call cc-option,-fno-asynchronous-unwind-tables)
 
 # Never use string load/store instructions as they are
 # often slow when they are implemented at all
diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
index e147bbdc12cd..d43b0b18137c 100644
--- a/arch/powerpc/kernel/vdso32/Makefile
+++ b/arch/powerpc/kernel/vdso32/Makefile
@@ -25,6 +25,7 @@ KCOV_INSTRUMENT := n
 UBSAN_SANITIZE := n
 
 ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
+       -fasynchronous-unwind-tables \
        -Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
 asflags-y := -D__VDSO32__ -s
 
diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index 32ebb3522ea1..b2cbb5c49bad 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -13,6 +13,7 @@ KCOV_INSTRUMENT := n
 UBSAN_SANITIZE := n
 
 ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
+       -fasynchronous-unwind-tables \
        -Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=both
 asflags-y := -D__VDSO64__ -s
 
 

Re: eh_frame confusion

[Rasmus Villemoes]

On 02/03/2020 18.32, Naveen N. Rao wrote:
> Naveen N. Rao wrote:
>> Michael opened a task to look into this recently and I had spent some
>> time last week on this. The original commit/discussion adding
>> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
>> handled by our module loader:
>> http://lkml.kernel.org/r/20090224065112.GA6690@bombadil.infradead.org
>>
>> However, that is now handled thanks to commit 9f751b82b491d:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
>>
>>
>> I did a test build and a simple module loaded fine, so I think
>> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
>> some breakages with it. Michael?
>>
>>>
>>> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
>>>
>>> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
>>> is there some magic that makes them necessary when building with gcc-8?
>>
>> As Segher points out, it looks like we need to add
>> -fno-asynchronous-unwind-tables. Most other architectures seem to use
>> that too.

Yes. Thanks, Segher, that explains that part.

> Can you check if the below patch works? I am yet to test this in more
> detail, but would be good to know the implications for ppc32.

I'll see if that produces a bootable kernel, but I think I'd prefer a
more piecemeal approach.

One patch to add -fno-asynchronous-unwind-tables (given that other
arches do it unconditionally I don't think cc-option is needed), with a
commit log saying something like "no-op for gcc < 8, prevents .eh_frame
sections that are discarded anyway for vmlinux and waste disk space for
modules". Then another patch can get rid of -fno-dwarf2-cfi-asm if
that's no longer required.

Rasmus

eh_frame confusion

[Naveen N. Rao]

Naveen N. Rao wrote:
> Naveen N. Rao wrote:
>> Rasmus Villemoes wrote:
<snip>
> Can you check if the below patch works? I am yet to test this in more 
> detail, but would be good to know the implications for ppc32.
> 
> - Naveen
> 
> 
> ---
> diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
> index f35730548e42..5b5bf98b8217 100644
> --- a/arch/powerpc/Makefile
> +++ b/arch/powerpc/Makefile
> @@ -239,10 +239,7 @@ KBUILD_CFLAGS += $(call cc-option,-mno-vsx)
>  KBUILD_CFLAGS += $(call cc-option,-mno-spe)
>  KBUILD_CFLAGS += $(call cc-option,-mspe=no)
>  
> -# FIXME: the module load should be taught about the additional relocs
> -# generated by this.
> -# revert to pre-gcc-4.4 behaviour of .eh_frame
> -KBUILD_CFLAGS  += $(call cc-option,-fno-dwarf2-cfi-asm)
> +KBUILD_CFLAGS  += $(call cc-option,-fno-asynchronous-unwind-tables)

In terms of the CFI information, the primary difference I see with 
-fno-dwarf2-cfi-asm is that when dumping the debug frames, CIE indicates 
version 3, while otherwise (i.e., without -fno-dwarf2-cfi-asm and 
with/without -fasynchronous-unwind-tables), it is version 1, regardless 
of -gdwarf-2/-gdwarf-4. There are few more minor changes, but none of 
these looked significant to me.

>  
>  # Never use string load/store instructions as they are
>  # often slow when they are implemented at all
> diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
> index e147bbdc12cd..d43b0b18137c 100644
> --- a/arch/powerpc/kernel/vdso32/Makefile
> +++ b/arch/powerpc/kernel/vdso32/Makefile
> @@ -25,6 +25,7 @@ KCOV_INSTRUMENT := n
>  UBSAN_SANITIZE := n
>  
>  ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> +       -fasynchronous-unwind-tables \
>         -Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
>  asflags-y := -D__VDSO32__ -s
>  
> diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
> index 32ebb3522ea1..b2cbb5c49bad 100644
> --- a/arch/powerpc/kernel/vdso64/Makefile
> +++ b/arch/powerpc/kernel/vdso64/Makefile
> @@ -13,6 +13,7 @@ KCOV_INSTRUMENT := n
>  UBSAN_SANITIZE := n
>  
>  ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> +       -fasynchronous-unwind-tables \
>         -Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=both
>  asflags-y := -D__VDSO64__ -s

The above vdso hunks can be dropped since all our VDSO are assembly, so 
the above have no impact.


- Naveen

Re: eh_frame confusion

[Michael Ellerman]

"Naveen N. Rao" <naveen.n.rao@linux.ibm.com> writes:
> Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section. For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
>> 
>> The reason I care is that my target has a rather tight rootfs budget,
>> and the .eh_frame section seem to occupy 10-30% of the file size
>> (obviously very depending on the particular module).
>> 
>> Comparing the .foo.o.cmd files, I don't see change in options that might
>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>> is apparently no longer supported in gcc-8). Both before and after, there's
>> 
>> -fno-dwarf2-cfi-asm
>> 
>> about which gcc's documentation says
>> 
>> '-fno-dwarf2-cfi-asm'
>>      Emit DWARF unwind info as compiler generated '.eh_frame' section
>>      instead of using GAS '.cfi_*' directives.
>> 
>> Looking into where that comes from got me even more confused, because
>> both arm and unicore32 say
>> 
>> # Never generate .eh_frame
>> KBUILD_CFLAGS           += $(call cc-option,-fno-dwarf2-cfi-asm)
>> 
>> while the ppc32 case at hand says
>> 
>> # FIXME: the module load should be taught about the additional relocs
>> # generated by this.
>> # revert to pre-gcc-4.4 behaviour of .eh_frame
>
> Michael opened a task to look into this recently and I had spent some 
> time last week on this. The original commit/discussion adding 
> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being 
> handled by our module loader:
> http://lkml.kernel.org/r/20090224065112.GA6690@bombadil.infradead.org

I opened that issue purely based on noticing the wart in the Makefile,
not because I'd actually tested it.

> However, that is now handled thanks to commit 9f751b82b491d:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d

Haha, written by me, what an idiot.

So the Makefile hack can presumably be dropped, because the module
loader can handle the relocations.

And then maybe we also want to turn off the unwind tables, but that
would be a separate patch.

> I did a test build and a simple module loaded fine, so I think 
> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen 
> some breakages with it. Michael?

No, as I said above it was just reading the Makefile.

cheers

Re: eh_frame confusion

[Naveen N. Rao]

Michael Ellerman wrote:
> "Naveen N. Rao" <naveen.n.rao@linux.ibm.com> writes:
>> Rasmus Villemoes wrote:
>>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>>> to gcc-8 all object files now end up having .eh_frame section. For
>>> vmlinux, that's not a problem, because they all get discarded in
>>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>>> modules, which doesn't seem to be useful - given that everything worked
>>> just fine with gcc-7, and I don't see anything in the module loader that
>>> handles .eh_frame.
>>> 
>>> The reason I care is that my target has a rather tight rootfs budget,
>>> and the .eh_frame section seem to occupy 10-30% of the file size
>>> (obviously very depending on the particular module).
>>> 
>>> Comparing the .foo.o.cmd files, I don't see change in options that might
>>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>>> is apparently no longer supported in gcc-8). Both before and after, there's
>>> 
>>> -fno-dwarf2-cfi-asm
>>> 
>>> about which gcc's documentation says
>>> 
>>> '-fno-dwarf2-cfi-asm'
>>>      Emit DWARF unwind info as compiler generated '.eh_frame' section
>>>      instead of using GAS '.cfi_*' directives.
>>> 
>>> Looking into where that comes from got me even more confused, because
>>> both arm and unicore32 say
>>> 
>>> # Never generate .eh_frame
>>> KBUILD_CFLAGS           += $(call cc-option,-fno-dwarf2-cfi-asm)
>>> 
>>> while the ppc32 case at hand says
>>> 
>>> # FIXME: the module load should be taught about the additional relocs
>>> # generated by this.
>>> # revert to pre-gcc-4.4 behaviour of .eh_frame
>>
>> Michael opened a task to look into this recently and I had spent some 
>> time last week on this. The original commit/discussion adding 
>> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being 
>> handled by our module loader:
>> http://lkml.kernel.org/r/20090224065112.GA6690@bombadil.infradead.org
> 
> I opened that issue purely based on noticing the wart in the Makefile,
> not because I'd actually tested it.
> 
>> However, that is now handled thanks to commit 9f751b82b491d:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
> 
> Haha, written by me, what an idiot.
> 
> So the Makefile hack can presumably be dropped, because the module
> loader can handle the relocations.
> 
> And then maybe we also want to turn off the unwind tables, but that
> would be a separate patch.
> 
>> I did a test build and a simple module loaded fine, so I think 
>> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen 
>> some breakages with it. Michael?
> 
> No, as I said above it was just reading the Makefile.

Ok, thanks for clarifying. To test, I did 'allmodconfig' builds across 
three environments:
- gcc (Ubuntu 9.2.1-9ubuntu2) 9.2.1 20191008 -- ppc64le
- gcc (SUSE Linux) 7.5.0 -- ppc64le
- gcc (GCC) 8.2.1 20181215 (Red Hat 8.2.1-6) -- ppc64 (BE)

Then, used the below command to list all relocations in the modules:
$ find . -name '*.ko' | xargs -n 1 readelf -Wr  | grep -v "Relocation " | grep -v "Offset " | cut -d' ' -f4 | sort | uniq

R_PPC64_ADDR32
R_PPC64_ADDR64
R_PPC64_ENTRY
R_PPC64_REL24
R_PPC64_REL32
R_PPC64_REL64
R_PPC64_TOC
R_PPC64_TOC16_HA
R_PPC64_TOC16_LO
R_PPC64_TOC16_LO_DS

All three environments show up similar set of relocations, all of which 
we handle in the module loader today.

If Rasmus/Christophe can confirm that this is true for ppc32 as well, 
then we should be fine.

- Naveen