[RFC PATCH 0/2] ARM large kernels

[RFC PATCH 0/2] ARM large kernels

[Ard Biesheuvel]

Hello all,

After Russell has pulled most of the large kernel fixes that are
specific to the ARM tree, these two patches are the remaining ones
that are required to successfully build 'allyesconfig' on ARM (*)

The first one updates scripts/kallsyms.c to ignore _veneer symbols.
These are emitted by the linker, and due to the fact that the kallsyms
region is empty in the first pass, and populated in the second, the
number of emitted veneers changes, which changes the contents of
the symbol table etc etc

The second patch keeps the contents of drivers/built-in.o into as
separate inputs into the final link stage. This ensures that the
linker can emit the veneers against out-of-range functions in a
location that is not out of range itself.

(*) It produces a ~75 MB zImage; whether it runs is another question

Ard Biesheuvel (2):
  Kbuild: kallsyms: ignore veneers emitted by the ARM linker
  Kbuild: avoid partial linking of drivers/built-in.o

 Makefile           | 12 +++++++++++-
 scripts/kallsyms.c | 17 ++++++++++++++---
 2 files changed, 25 insertions(+), 4 deletions(-)

-- 
1.8.3.2

[RFC PATCH 1/2] Kbuild: kallsyms: ignore veneers emitted by the ARM linker

[Ard Biesheuvel]

When linking large kernels on ARM, the linker will insert veneers
(i.e., PLT like stubs) when function symbols are out of reach for
the ordinary relative branch/branch-and-link instructions.

However, due to the fact that the kallsyms region sits in .rodata,
which is between .text and .init.text, additional veneers may be
emitted in the second pass due to the fact that the size of the
kallsyms region itself has pushed the .init.text section further
down, resulting in additional veneers to be emitted.

So ignore the veneers when generating the symbol table. Veneers
have no corresponding source code, and they will not turn up in
backtraces anyway.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 scripts/kallsyms.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
index c6d33bd15b04..6668c87d599f 100644
--- a/scripts/kallsyms.c
+++ b/scripts/kallsyms.c
@@ -212,6 +212,12 @@ static int symbol_valid(struct sym_entry *s)
 		"_SDA_BASE_",		/* ppc */
 		"_SDA2_BASE_",		/* ppc */
 		NULL };
+
+	static char *special_suffixes[] = {
+		"_compiled.",		/* gcc < 3.0: "gcc[0-9]_compiled." */
+		"_veneer",		/* arm */
+		NULL };
+
 	int i;
 	int offset = 1;
 
@@ -244,13 +250,18 @@ static int symbol_valid(struct sym_entry *s)
 	}
 
 	/* Exclude symbols which vary between passes. */
-	if (strstr((char *)s->sym + offset, "_compiled."))
-		return 0;
-
 	for (i = 0; special_symbols[i]; i++)
 		if( strcmp((char *)s->sym + offset, special_symbols[i]) == 0 )
 			return 0;
 
+	for (i = 0; special_suffixes[i]; i++) {
+		char *sym_name = (char *)s->sym + offset;
+		int l = strlen(sym_name) - strlen(special_suffixes[i]);
+
+		if (l >= 0 && strcmp(sym_name + l, special_suffixes[i]) == 0)
+			return 0;
+	}
+
 	return 1;
 }
 
-- 
1.8.3.2

[RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Ard Biesheuvel]

The recursive partial linking of vmlinux can result in a
drivers/built-in.o that is so huge that it interferes with
the ability of the linker to emit veneers in the final link
stage if the symbols are out of reach. This is caused by the
fact that those veneers, which should be emitted close enough
to the original call site, can only be emitted after the .text
section of drivers/built-in.o, whose size pushes those veneers
out of range.

So instead, avoid building drivers/built-in.o, and instead, add
the constituent parts to the command line of the final link.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 Makefile | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index e734965b1604..1eb6c246a586 100644
--- a/Makefile
+++ b/Makefile
@@ -558,7 +558,7 @@ scripts: scripts_basic include/config/auto.conf include/config/tristate.conf \
 
 # Objects we will link into vmlinux / subdirs we need to visit
 init-y		:= init/
-drivers-y	:= drivers/ sound/ firmware/
+drivers-y	:= sound/ firmware/
 net-y		:= net/
 libs-y		:= lib/
 core-y		:= usr/
@@ -569,6 +569,16 @@ ifeq ($(dot-config),1)
 -include include/config/auto.conf
 
 ifeq ($(KBUILD_EXTMOD),)
+
+# drivers/built-in.o can become huge, which interferes with the linker's
+# ability to emit stubs for branch targets that are out of reach for the
+# ordinary relative branch instructions
+include $(srctree)/drivers/Makefile
+drivers-y += $(addprefix drivers/,$(sort $(obj-y)))
+drivers-m += $(addprefix drivers/,$(sort $(obj-m)))
+obj-y :=
+obj-m :=
+
 # Read in dependencies to all Kconfig* files, make sure to run
 # oldconfig if changes are detected.
 -include include/config/auto.conf.cmd
-- 
1.8.3.2

Re: [RFC PATCH 1/2] Kbuild: kallsyms: ignore veneers emitted by the ARM linker

[Michal Marek]

On 2015-03-30 13:49, Ard Biesheuvel wrote:
> When linking large kernels on ARM, the linker will insert veneers
> (i.e., PLT like stubs) when function symbols are out of reach for
> the ordinary relative branch/branch-and-link instructions.
> 
> However, due to the fact that the kallsyms region sits in .rodata,
> which is between .text and .init.text, additional veneers may be
> emitted in the second pass due to the fact that the size of the
> kallsyms region itself has pushed the .init.text section further
> down, resulting in additional veneers to be emitted.
> 
> So ignore the veneers when generating the symbol table. Veneers
> have no corresponding source code, and they will not turn up in
> backtraces anyway.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
>  scripts/kallsyms.c | 17 ++++++++++++++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
> index c6d33bd15b04..6668c87d599f 100644
> --- a/scripts/kallsyms.c
> +++ b/scripts/kallsyms.c
> @@ -212,6 +212,12 @@ static int symbol_valid(struct sym_entry *s)
>  		"_SDA_BASE_",		/* ppc */
>  		"_SDA2_BASE_",		/* ppc */
>  		NULL };
> +
> +	static char *special_suffixes[] = {
> +		"_compiled.",		/* gcc < 3.0: "gcc[0-9]_compiled." */

Your newly added comment suggests that this can go away, since we have
been requiring at least gcc 3.2 since some time. But it can be removed
in a follow-up cleanup.


> +		"_veneer",		/* arm */
> +		NULL };
> +
>  	int i;
>  	int offset = 1;
>  
> @@ -244,13 +250,18 @@ static int symbol_valid(struct sym_entry *s)
>  	}
>  
>  	/* Exclude symbols which vary between passes. */
> -	if (strstr((char *)s->sym + offset, "_compiled."))
> -		return 0;
> -
>  	for (i = 0; special_symbols[i]; i++)
>  		if( strcmp((char *)s->sym + offset, special_symbols[i]) == 0 )
>  			return 0;
>  
> +	for (i = 0; special_suffixes[i]; i++) {
> +		char *sym_name = (char *)s->sym + offset;

You should declare and set this useful helper a few lines earlier and
use it in the other statements that test the symbol name.

Michal

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Michal Marek]

On 2015-03-30 13:49, Ard Biesheuvel wrote:
> The recursive partial linking of vmlinux can result in a
> drivers/built-in.o that is so huge that it interferes with
> the ability of the linker to emit veneers in the final link
> stage if the symbols are out of reach. This is caused by the
> fact that those veneers, which should be emitted close enough
> to the original call site, can only be emitted after the .text
> section of drivers/built-in.o, whose size pushes those veneers
> out of range.

Is this a limitation of a particular ARM ABI or a limitation of a state
of the art ARM linker or something else? If such a hack is necessary, it
needs to be accompanied with an explanation as to in which environments
it is needed, whether it can be removed at some point in future, what is
the exact error it causes, etc. Also, are you able to gauge the
limitation? Will it at some point affect fs/built-in.o or
drivers/net/built-in.o?


> So instead, avoid building drivers/built-in.o, and instead, add
> the constituent parts to the command line of the final link.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
>  Makefile | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index e734965b1604..1eb6c246a586 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -558,7 +558,7 @@ scripts: scripts_basic include/config/auto.conf include/config/tristate.conf \
>  
>  # Objects we will link into vmlinux / subdirs we need to visit
>  init-y		:= init/
> -drivers-y	:= drivers/ sound/ firmware/
> +drivers-y	:= sound/ firmware/
>  net-y		:= net/
>  libs-y		:= lib/
>  core-y		:= usr/
> @@ -569,6 +569,16 @@ ifeq ($(dot-config),1)
>  -include include/config/auto.conf
>  
>  ifeq ($(KBUILD_EXTMOD),)
> +
> +# drivers/built-in.o can become huge, which interferes with the linker's
> +# ability to emit stubs for branch targets that are out of reach for the
> +# ordinary relative branch instructions
> +include $(srctree)/drivers/Makefile
> +drivers-y += $(addprefix drivers/,$(sort $(obj-y)))
> +drivers-m += $(addprefix drivers/,$(sort $(obj-m)))

I think this will break should we ever put a .c file in drivers/ directly.

Michal

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Ard Biesheuvel]

On 30 March 2015 at 14:38, Michal Marek <mmarek@suse.cz> wrote:
> On 2015-03-30 13:49, Ard Biesheuvel wrote:
>> The recursive partial linking of vmlinux can result in a
>> drivers/built-in.o that is so huge that it interferes with
>> the ability of the linker to emit veneers in the final link
>> stage if the symbols are out of reach. This is caused by the
>> fact that those veneers, which should be emitted close enough
>> to the original call site, can only be emitted after the .text
>> section of drivers/built-in.o, whose size pushes those veneers
>> out of range.
>
> Is this a limitation of a particular ARM ABI or a limitation of a state
> of the art ARM linker or something else? If such a hack is necessary, it
> needs to be accompanied with an explanation as to in which environments
> it is needed, whether it can be removed at some point in future, what is
> the exact error it causes, etc. Also, are you able to gauge the
> limitation? Will it at some point affect fs/built-in.o or
> drivers/net/built-in.o?
>

The limitation results from the fact that ARM branch instructions are
limited to 24-bits of relative offset, and depending on the
instruction set (ARM or Thumb) this translates to +/- 32 MB or +/- 16
MB respectively.

Note that using -ffunction-sections works around the problem as well,
but it typically uses more space. There are other archs that add this,
to work around a similar issue.

The errors it causes are build time linker errors, i.e., after
crossing a certain size threshold, you just cannot build vmlinux
anymore.

It is not something that we would able to remove in the future, as it
is simply a result of the ISA capabilities and the build strategy that
relies heavily on partial linking. Other linkers would not be able to
do a better job, as the drivers/built-in.o(.text) section just exceeds
the maximum size that allows a branch instruction to jump out of it.

I am not sure whether it would affect other subdirs, but it is not
entirely unlikely.

A more structural approach would be to limit the contents of the
built-in.o files to the mod_init/_exit/_etc input sections, and let
those pull in everything else through a libbuilt-in.a at the same
level that contains the .text/.data etc, each of which would be
combined with the constituent ones as we do for built-in.o now. I
haven't played around with this extensively, though.

>
>> So instead, avoid building drivers/built-in.o, and instead, add
>> the constituent parts to the command line of the final link.
>>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> ---
>>  Makefile | 12 +++++++++++-
>>  1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/Makefile b/Makefile
>> index e734965b1604..1eb6c246a586 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -558,7 +558,7 @@ scripts: scripts_basic include/config/auto.conf include/config/tristate.conf \
>>
>>  # Objects we will link into vmlinux / subdirs we need to visit
>>  init-y               := init/
>> -drivers-y    := drivers/ sound/ firmware/
>> +drivers-y    := sound/ firmware/
>>  net-y                := net/
>>  libs-y               := lib/
>>  core-y               := usr/
>> @@ -569,6 +569,16 @@ ifeq ($(dot-config),1)
>>  -include include/config/auto.conf
>>
>>  ifeq ($(KBUILD_EXTMOD),)
>> +
>> +# drivers/built-in.o can become huge, which interferes with the linker's
>> +# ability to emit stubs for branch targets that are out of reach for the
>> +# ordinary relative branch instructions
>> +include $(srctree)/drivers/Makefile
>> +drivers-y += $(addprefix drivers/,$(sort $(obj-y)))
>> +drivers-m += $(addprefix drivers/,$(sort $(obj-m)))
>
> I think this will break should we ever put a .c file in drivers/ directly.
>

Yes, it will

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Russell King - ARM Linux]

On Mon, Mar 30, 2015 at 02:38:35PM +0200, Michal Marek wrote:
> Is this a limitation of a particular ARM ABI or a limitation of a state
> of the art ARM linker or something else?

It's a limitation of the ARM ISA.

Normal PC-relative branches, which are emitted by the C compiler, can
branch +/- 32MB for ARM, or +/- 16MB of Thumb.  Beyond that, the address
offset is not representable in the instruction.

To get around that "modern" linkers are able to insert stubs as
necessary inbetween the object .text sections to extend the range of
these branches (by emitting a chunk of code possibly with some data to
extend the range of the branch.)

Obviously, gcc can't know before hand that the sum total of all the
small object files is going to cause problems, so the compiler can't
do this on an "as necessary basis".

For most practical kernels, this is not a problem; they normally fit
within the PC-relative range.  However, the exception is allyesconfig.

The question is: how far do we go with allyesconfig... do we want it
to work, or is reaching the final link sufficient?  If we do tweak
stuff to allow the link to work, are we going to try running it?

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Ard Biesheuvel]

On 30 March 2015 at 15:26, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Mar 30, 2015 at 02:38:35PM +0200, Michal Marek wrote:
>> Is this a limitation of a particular ARM ABI or a limitation of a state
>> of the art ARM linker or something else?
>
> It's a limitation of the ARM ISA.
>
> Normal PC-relative branches, which are emitted by the C compiler, can
> branch +/- 32MB for ARM, or +/- 16MB of Thumb.  Beyond that, the address
> offset is not representable in the instruction.
>
> To get around that "modern" linkers are able to insert stubs as
> necessary inbetween the object .text sections to extend the range of
> these branches (by emitting a chunk of code possibly with some data to
> extend the range of the branch.)
>
> Obviously, gcc can't know before hand that the sum total of all the
> small object files is going to cause problems, so the compiler can't
> do this on an "as necessary basis".
>
> For most practical kernels, this is not a problem; they normally fit
> within the PC-relative range.  However, the exception is allyesconfig.
>
> The question is: how far do we go with allyesconfig... do we want it
> to work, or is reaching the final link sufficient?  If we do tweak
> stuff to allow the link to work, are we going to try running it?
>

That is an excellent question, hence the RFC in the subject line.

Note that the other patch, the one against kallsyms, addresses the
issue where the distance between the beginning of .text and the end of
.init.text  exceeds this limit, which is not as unlikely as the issue
that this patch addresses, where just drivers/built-in.o in isolation
already exceeds this limit.

So I am quite happy to drop this, especially as we can add
-ffunction-sections as well. (Haven't tested this myself though: after
building allyesconfig on my poor laptop a couple of times, I am
sending it on a well deserved vacation for the rest of the week)

Thanks,
Ard.

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Michal Marek]

On 2015-03-30 15:31, Ard Biesheuvel wrote:
> On 30 March 2015 at 15:26, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
>> On Mon, Mar 30, 2015 at 02:38:35PM +0200, Michal Marek wrote:
>>> Is this a limitation of a particular ARM ABI or a limitation of a state
>>> of the art ARM linker or something else?
>>
>> It's a limitation of the ARM ISA.
>>
>> Normal PC-relative branches, which are emitted by the C compiler, can
>> branch +/- 32MB for ARM, or +/- 16MB of Thumb.  Beyond that, the address
>> offset is not representable in the instruction.

Thank you both for the explanation!


>> The question is: how far do we go with allyesconfig... do we want it
>> to work, or is reaching the final link sufficient?

It certainly is more useful as a test tool if the baseline is a
successful compile and link. Because you can have genuine link errors
due to missing symbols.


>> If we do tweak
>> stuff to allow the link to work, are we going to try running it?

Good question. I myself always treated all{yes,mod}config as a build
test only and never dared to run it. Allyesconfig produces a giant
kernel image and allmodconfig builds binfmt_script as a module. And if
people used all*config for boot tests, they would probably be sending
patches to tweak the Kconfigs for that purpose. And this is not the case
as far as I can tell.


> That is an excellent question, hence the RFC in the subject line.
> 
> Note that the other patch, the one against kallsyms, addresses the
> issue where the distance between the beginning of .text and the end of
> .init.text  exceeds this limit, which is not as unlikely as the issue
> that this patch addresses, where just drivers/built-in.o in isolation
> already exceeds this limit.
> 
> So I am quite happy to drop this, especially as we can add
> -ffunction-sections as well.

What you could do is to add a Kconfig option to arch/arm/Kconfig adding
-ffunction-sections to the compiler flags. Then allyesconfig would
select it and work around the problem in a somewhat elegant way.

Michal

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Ard Biesheuvel]

On 30 March 2015 at 16:13, Michal Marek <mmarek@suse.cz> wrote:
> On 2015-03-30 15:31, Ard Biesheuvel wrote:
>> On 30 March 2015 at 15:26, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>>> On Mon, Mar 30, 2015 at 02:38:35PM +0200, Michal Marek wrote:
>>>> Is this a limitation of a particular ARM ABI or a limitation of a state
>>>> of the art ARM linker or something else?
>>>
>>> It's a limitation of the ARM ISA.
>>>
>>> Normal PC-relative branches, which are emitted by the C compiler, can
>>> branch +/- 32MB for ARM, or +/- 16MB of Thumb.  Beyond that, the address
>>> offset is not representable in the instruction.
>
> Thank you both for the explanation!
>
>
>>> The question is: how far do we go with allyesconfig... do we want it
>>> to work, or is reaching the final link sufficient?
>
> It certainly is more useful as a test tool if the baseline is a
> successful compile and link. Because you can have genuine link errors
> due to missing symbols.
>

Agreed

>
>>> If we do tweak
>>> stuff to allow the link to work, are we going to try running it?
>
> Good question. I myself always treated all{yes,mod}config as a build
> test only and never dared to run it. Allyesconfig produces a giant
> kernel image and allmodconfig builds binfmt_script as a module. And if
> people used all*config for boot tests, they would probably be sending
> patches to tweak the Kconfigs for that purpose. And this is not the case
> as far as I can tell.
>

Russell should confirm this, but I think running such a large kernel
is non-trivial on ARM, since the decompressor should make room for the
decompressed image by moving itself upward in memory, and it may
overwrite the device tree binary in the process.

>
>> That is an excellent question, hence the RFC in the subject line.
>>
>> Note that the other patch, the one against kallsyms, addresses the
>> issue where the distance between the beginning of .text and the end of
>> .init.text  exceeds this limit, which is not as unlikely as the issue
>> that this patch addresses, where just drivers/built-in.o in isolation
>> already exceeds this limit.
>>
>> So I am quite happy to drop this, especially as we can add
>> -ffunction-sections as well.
>
> What you could do is to add a Kconfig option to arch/arm/Kconfig adding
> -ffunction-sections to the compiler flags. Then allyesconfig would
> select it and work around the problem in a somewhat elegant way.
>

Excellent idea! Arnd hasn't chimed in yet, but he is the one doing
lots and lots of randconfig builds and other test builds, so I will
wait for him to confirm that this is a useful thing to have.

Thanks,
Ard.

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Nicolas Pitre]

On Mon, 30 Mar 2015, Ard Biesheuvel wrote:

> On 30 March 2015 at 16:13, Michal Marek <mmarek@suse.cz> wrote:
> > On 2015-03-30 15:31, Ard Biesheuvel wrote:
> >> On 30 March 2015 at 15:26, Russell King - ARM Linux
> >> <linux@arm.linux.org.uk> wrote:
> >>> On Mon, Mar 30, 2015 at 02:38:35PM +0200, Michal Marek wrote:
> >>>> Is this a limitation of a particular ARM ABI or a limitation of a state
> >>>> of the art ARM linker or something else?
> >>>
> >>> It's a limitation of the ARM ISA.
> >>>
> >>> Normal PC-relative branches, which are emitted by the C compiler, can
> >>> branch +/- 32MB for ARM, or +/- 16MB of Thumb.  Beyond that, the address
> >>> offset is not representable in the instruction.
> >
> > Thank you both for the explanation!
> >
> >
> >>> The question is: how far do we go with allyesconfig... do we want it
> >>> to work, or is reaching the final link sufficient?
> >
> > It certainly is more useful as a test tool if the baseline is a
> > successful compile and link. Because you can have genuine link errors
> > due to missing symbols.
> >
> 
> Agreed
> 
> >
> >>> If we do tweak
> >>> stuff to allow the link to work, are we going to try running it?
> >
> > Good question. I myself always treated all{yes,mod}config as a build
> > test only and never dared to run it. Allyesconfig produces a giant
> > kernel image and allmodconfig builds binfmt_script as a module. And if
> > people used all*config for boot tests, they would probably be sending
> > patches to tweak the Kconfigs for that purpose. And this is not the case
> > as far as I can tell.
> >
> 
> Russell should confirm this, but I think running such a large kernel
> is non-trivial on ARM, since the decompressor should make room for the
> decompressed image by moving itself upward in memory, and it may
> overwrite the device tree binary in the process.

Loading the device tree at a different address should be easy.  You just 
need to load it at the top of RAM. You may also start by attempting to 
boot a plain Image (uncompressed) kernel binary.

> >> That is an excellent question, hence the RFC in the subject line.
> >>
> >> Note that the other patch, the one against kallsyms, addresses the
> >> issue where the distance between the beginning of .text and the end of
> >> .init.text  exceeds this limit, which is not as unlikely as the issue
> >> that this patch addresses, where just drivers/built-in.o in isolation
> >> already exceeds this limit.
> >>
> >> So I am quite happy to drop this, especially as we can add
> >> -ffunction-sections as well.
> >
> > What you could do is to add a Kconfig option to arch/arm/Kconfig adding
> > -ffunction-sections to the compiler flags. Then allyesconfig would
> > select it and work around the problem in a somewhat elegant way.
> >
> 
> Excellent idea! Arnd hasn't chimed in yet, but he is the one doing
> lots and lots of randconfig builds and other test builds, so I will
> wait for him to confirm that this is a useful thing to have.

I'm using -ffunction-sections as well for the kernel size reduction work 
I'm currently doing.  The linker script has to be adapted so .text.* is 
specified along .text otherwise those functions end up appended at the 
end of the binary.


Nicolas

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Dave Martin]

On Tue, Mar 31, 2015 at 11:22:29AM -0400, Nicolas Pitre wrote:
> On Mon, 30 Mar 2015, Ard Biesheuvel wrote:
> 
> > On 30 March 2015 at 16:13, Michal Marek <mmarek@suse.cz> wrote:

[...]

> > > What you could do is to add a Kconfig option to arch/arm/Kconfig adding
> > > -ffunction-sections to the compiler flags. Then allyesconfig would
> > > select it and work around the problem in a somewhat elegant way.
> > >
> > 
> > Excellent idea! Arnd hasn't chimed in yet, but he is the one doing
> > lots and lots of randconfig builds and other test builds, so I will
> > wait for him to confirm that this is a useful thing to have.
> 
> I'm using -ffunction-sections as well for the kernel size reduction work 
> I'm currently doing.  The linker script has to be adapted so .text.* is 
> specified along .text otherwise those functions end up appended at the 
> end of the binary.

Interesting ... do you also mean using --gc-sections at link time?
We'd need to avoid pruning needed code that has no explicit caller,
and functions that are part of the kernel/module ABI but not used
within vmlinux.

The GCC docs suggest that -ffunction-sections may impact performance
and/or increase code size, but I don't know by how much.  Maybe it
interferes with inling.

Cheers
---Dave

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Russell King - ARM Linux]

On Tue, Mar 31, 2015 at 05:27:22PM +0100, Dave Martin wrote:
> On Tue, Mar 31, 2015 at 11:22:29AM -0400, Nicolas Pitre wrote:
> > I'm using -ffunction-sections as well for the kernel size reduction work 
> > I'm currently doing.  The linker script has to be adapted so .text.* is 
> > specified along .text otherwise those functions end up appended at the 
> > end of the binary.
> 
> Interesting ... do you also mean using --gc-sections at link time?
> We'd need to avoid pruning needed code that has no explicit caller,
> and functions that are part of the kernel/module ABI but not used
> within vmlinux.
> 
> The GCC docs suggest that -ffunction-sections may impact performance
> and/or increase code size, but I don't know by how much.  Maybe it
> interferes with inling.

I've mentioned these options over the course of the last couple of
months.  David Woodhouse replied in February, so it's worth reading
up on that work.

David Woodhouse wrote:
| In many kernel configurations there are actually quite a lot of
| functions that are never called, and I was quite surprised the first
| time I played with this stuff.
| 
| There are a few ways of dealing with it. One is to use
| -ffunction-section -fdata-sections --gc-sections as you noted. I once
| also played with using GCC's --combine during the brief period that it
| was supported and not *entirely* broken, with similar effects:
| https://lwn.net/Articles/197097/
| 
| These days, the better answer is probably LTO. We could potentially
| still look at --gc-sections, but I suspect we're better off using LTO
| and just filing toolchain bugs until everything that --gc-sections
| *would* have dropped is also dropped from the LTO build :)
| 
| Unless --gc-sections actually speeds up the build in a significant way;
| a full LTO link of the kernel takes insane amounts of memory IIRC.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Nicolas Pitre]

On Tue, 31 Mar 2015, Dave Martin wrote:

> On Tue, Mar 31, 2015 at 11:22:29AM -0400, Nicolas Pitre wrote:
> > On Mon, 30 Mar 2015, Ard Biesheuvel wrote:
> > 
> > > On 30 March 2015 at 16:13, Michal Marek <mmarek@suse.cz> wrote:
> 
> [...]
> 
> > > > What you could do is to add a Kconfig option to arch/arm/Kconfig adding
> > > > -ffunction-sections to the compiler flags. Then allyesconfig would
> > > > select it and work around the problem in a somewhat elegant way.
> > > >
> > > 
> > > Excellent idea! Arnd hasn't chimed in yet, but he is the one doing
> > > lots and lots of randconfig builds and other test builds, so I will
> > > wait for him to confirm that this is a useful thing to have.
> > 
> > I'm using -ffunction-sections as well for the kernel size reduction work 
> > I'm currently doing.  The linker script has to be adapted so .text.* is 
> > specified along .text otherwise those functions end up appended at the 
> > end of the binary.
> 
> Interesting ... do you also mean using --gc-sections at link time?

Yes.

> We'd need to avoid pruning needed code that has no explicit caller,
> and functions that are part of the kernel/module ABI but not used
> within vmlinux.

Those are usually located in special sections, like the initcall table, 
the CPU entry table, etc. The linker allows for those sections to be 
marked with KEEP() not to prune them.

> The GCC docs suggest that -ffunction-sections may impact performance
> and/or increase code size, but I don't know by how much.  Maybe it
> interferes with inling.

It doesn't interfere with inlining. However it impose a section 
alignment on every function. That still can be overriden though. Also, I 
suppose that gcc may not assume that calls to a global function that 
happens to be located in the same C file will be close by anymore, 
however I don't see this having any impact on ARM code generation.


Nicolas

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Nicolas Pitre]

On Tue, 31 Mar 2015, Russell King - ARM Linux wrote:

> On Tue, Mar 31, 2015 at 05:27:22PM +0100, Dave Martin wrote:
> > On Tue, Mar 31, 2015 at 11:22:29AM -0400, Nicolas Pitre wrote:
> > > I'm using -ffunction-sections as well for the kernel size reduction work 
> > > I'm currently doing.  The linker script has to be adapted so .text.* is 
> > > specified along .text otherwise those functions end up appended at the 
> > > end of the binary.
> > 
> > Interesting ... do you also mean using --gc-sections at link time?
> > We'd need to avoid pruning needed code that has no explicit caller,
> > and functions that are part of the kernel/module ABI but not used
> > within vmlinux.
> > 
> > The GCC docs suggest that -ffunction-sections may impact performance
> > and/or increase code size, but I don't know by how much.  Maybe it
> > interferes with inling.
> 
> I've mentioned these options over the course of the last couple of
> months.  David Woodhouse replied in February, so it's worth reading
> up on that work.

Yup, I did.

> David Woodhouse wrote:
> | In many kernel configurations there are actually quite a lot of
> | functions that are never called, and I was quite surprised the first
> | time I played with this stuff.
> | 
> | There are a few ways of dealing with it. One is to use
> | -ffunction-section -fdata-sections --gc-sections as you noted. I once
> | also played with using GCC's --combine during the brief period that it
> | was supported and not *entirely* broken, with similar effects:
> | https://lwn.net/Articles/197097/
> | 
> | These days, the better answer is probably LTO. We could potentially
> | still look at --gc-sections, but I suspect we're better off using LTO
> | and just filing toolchain bugs until everything that --gc-sections
> | *would* have dropped is also dropped from the LTO build :)
> | 
> | Unless --gc-sections actually speeds up the build in a significant way;
> | a full LTO link of the kernel takes insane amounts of memory IIRC.

LTO indirectly achieve the same effect (and more), however it is 
frigging heavy and unwieldy for kernel builds.  And it has many subtle 
issues of its own that --gc-sections doesn't have.  I'm therefore 
looking at both in parallel.


Nicolas

Re: [RFC PATCH 2/2] Kbuild: avoid partial linking of drivers/built-in.o

[Dave Martin]

On Tue, Mar 31, 2015 at 02:42:24PM -0400, Nicolas Pitre wrote:
> On Tue, 31 Mar 2015, Dave Martin wrote:

[...]

> > We'd need to avoid pruning needed code that has no explicit caller,
> > and functions that are part of the kernel/module ABI but not used
> > within vmlinux.
> 
> Those are usually located in special sections, like the initcall table, 
> the CPU entry table, etc. The linker allows for those sections to be 
> marked with KEEP() not to prune them.

True.  See what breaks, I guess.

> > The GCC docs suggest that -ffunction-sections may impact performance
> > and/or increase code size, but I don't know by how much.  Maybe it
> > interferes with inling.
> 
> It doesn't interfere with inlining. However it impose a section 
> alignment on every function. That still can be overriden though. Also, I 
> suppose that gcc may not assume that calls to a global function that 
> happens to be located in the same C file will be close by anymore, 
> however I don't see this having any impact on ARM code generation.

More veneers may be introduced during linking, and it's likely that
cache/TLB locality would be affected, but I've no idea whether the
impact would be significant.

Cheers
---Dave