[GIT] kbuild/lto changes for 3.15-rc1

[GIT] kbuild/lto changes for 3.15-rc1

[Michal Marek]

Hi Linus,

besides the kbuild branch, here is the LTO build support by Andi. It is
a separate branch, because it depends on other patches by Andi which
were merged through other trees. The link-time-optimization build is an
experimental feature, so there one kconfig option to enable it and
another kconfig option to disable it (behind a door with a sign "Beware
of the Leopard"...), so that it is not enabled by
allmodconfig/allyesconfig.

Thanks,
Michal

The following changes since commit 176ab02d4916f09d5d8cb63372d142df4378cdea:

  Merge branch 'x86-asmlinkage-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip (2014-03-31 14:13:25 -0700)

are available in the git repository at:


  git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild.git lto

for you to fetch changes up to 19a3cc83353e3bb4bc28769f8606139a3d350d2d:

  Kbuild, lto: Add Link Time Optimization support v3 (2014-04-07 21:51:13 +0200)

----------------------------------------------------------------
Andi Kleen (2):
      Kbuild, lto: Set TMPDIR for LTO v4
      Kbuild, lto: Add Link Time Optimization support v3

 Documentation/lto-build  | 173 +++++++++++++++++++++++++++++++++++++++++++++++
 Makefile                 |  20 +++++-
 arch/x86/Kconfig         |   2 +-
 init/Kconfig             |  73 ++++++++++++++++++++
 kernel/gcov/Kconfig      |   2 +-
 lib/Kconfig.debug        |   2 +-
 scripts/Makefile.lto     |  84 +++++++++++++++++++++++
 scripts/Makefile.modpost |   7 +-
 scripts/link-vmlinux.sh  |   2 +-
 9 files changed, 357 insertions(+), 8 deletions(-)
 create mode 100644 Documentation/lto-build
 create mode 100644 scripts/Makefile.lto

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

On Mon, Apr 07, 2014 at 10:19:19PM +0200, Michal Marek wrote:
> Hi Linus,
> 
> besides the kbuild branch, here is the LTO build support by Andi. It is
> a separate branch, because it depends on other patches by Andi which
> were merged through other trees. The link-time-optimization build is an
> experimental feature, so there one kconfig option to enable it and
> another kconfig option to disable it (behind a door with a sign "Beware
> of the Leopard"...), so that it is not enabled by
> allmodconfig/allyesconfig.

Thanks Michal,

I should add this version builds quite a bit slower than other LTO trees,
because the single pass kallsyms didn't quite make it, so the Link
Time Optimization is re-done 2-3 times if KALLSYMS is enabled. That 
will be hopefully fixed next cycle.

LTO will always be somewhat slower than a normal build, but right now 
it's a lot slower because of this, depending on the size of the vmlinux.

-Andi

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Linus Torvalds]

On Mon, Apr 7, 2014 at 1:19 PM, Michal Marek <mmarek@suse.cz> wrote:
>
> besides the kbuild branch, here is the LTO build support by Andi. It is
> a separate branch, because it depends on other patches by Andi which
> were merged through other trees. The link-time-optimization build is an
> experimental feature, so there one kconfig option to enable it and
> another kconfig option to disable it (behind a door with a sign "Beware
> of the Leopard"...), so that it is not enabled by
> allmodconfig/allyesconfig.

So right now, I see several reasons not to merge it ("It's so
experimental that we don't even want to encourage people to test it"
to "it's not fully fleshed out yet and makes compile times _much_
longer").

And yet nobody has actually talked about why I *should* merge it.

Which - I think understandably - makes me less than enthusiastic.

So I think I'll let this wait a bit longer, _unless_ people start
talking about the upsides. How much smaller is the end result? How
much faster is it? How much more beautiful is it? Does it make new
cool things possible? Are those cool things really close on the
horizon and really want this merged even though it's not really quite
ready yet?

So please: convince me. Andi?

              Linus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[josh]

On Tue, Apr 08, 2014 at 08:26:09AM -0700, Linus Torvalds wrote:
> On Mon, Apr 7, 2014 at 1:19 PM, Michal Marek <mmarek@suse.cz> wrote:
> > besides the kbuild branch, here is the LTO build support by Andi. It is
> > a separate branch, because it depends on other patches by Andi which
> > were merged through other trees. The link-time-optimization build is an
> > experimental feature, so there one kconfig option to enable it and
> > another kconfig option to disable it (behind a door with a sign "Beware
> > of the Leopard"...), so that it is not enabled by
> > allmodconfig/allyesconfig.
> 
> So right now, I see several reasons not to merge it ("It's so
> experimental that we don't even want to encourage people to test it"
> to "it's not fully fleshed out yet and makes compile times _much_
> longer").
> 
> And yet nobody has actually talked about why I *should* merge it.
> 
> Which - I think understandably - makes me less than enthusiastic.
> 
> So I think I'll let this wait a bit longer, _unless_ people start
> talking about the upsides. How much smaller is the end result? How
> much faster is it? How much more beautiful is it? Does it make new
> cool things possible? Are those cool things really close on the
> horizon and really want this merged even though it's not really quite
> ready yet?
> 
> So please: convince me.

I'd really like to see this feature go in to enable ongoing efforts to
make the kernel much smaller, so I'll give it a shot:

In addition to making the kernel smaller and such (I'll leave the
specific stats there to Andi), here's the key awesomeness of LTO that
you, personally, should find useful and compelling: LTO will eliminate
the need to add many lower-level Kconfig symbols to compile out bits of
the kernel.  Normally, compiling out a piece of kernel infrastructure
requires adding a kernel config symbol for that infrastructure, and
making all the other kernel bits that use that infrastructure depend on
it (which is an extra step beyond just "include the header and use it").
LTO will allow the compiler to automatically drop bits of infrastructure
(such as bits of lib/, kernel/, etc) that aren't used.

And with some additional GCC smarts (which exist today for C++ classes,
but would need adding for C structure-of-function-pointers objects), GCC
with LTO could even optimize away functions that are only used in the
function pointer of a data structure for which there are no callers.
So, it'd be possible to compile out high-level user-facing items like
syscalls, and have all the infrastructure go away all the way down.

So, with LTO merged, the next time you see a patch to add yet another
Kconfig symbol to compile out some low-level kernel infrastructure when
not needed, you can say "no, I don't want to add more configuration
complexity; compile out the high-level bits and use LTO to make the
compiler drop the infrastructure".  And we can potentially start ripping
out *existing* Kconfig infrastructure symbols like that, too.

- Josh Triplett

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Linus Torvalds]

On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
>
> In addition to making the kernel smaller and such (I'll leave the
> specific stats there to Andi), here's the key awesomeness of LTO that
> you, personally, should find useful and compelling: LTO will eliminate
> the need to add many lower-level Kconfig symbols to compile out bits of
> the kernel.

Actually that, to me, is a negative right now.

Since there's no way we'll make LTO the default in the foreseeable
future, people starting to use it like that is just a bad bad thing.

So really, the main advantage of LTO would be any actual optimizations
it can do. And call me anal, but I want *numbers* for that before I
merge it. Not handwaving. I'm not actually aware of how well - if at
all - code generation actually improves.

                Linus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> >
> > In addition to making the kernel smaller and such (I'll leave the
> > specific stats there to Andi), here's the key awesomeness of LTO that
> > you, personally, should find useful and compelling: LTO will eliminate
> > the need to add many lower-level Kconfig symbols to compile out bits of
> > the kernel.
> 
> Actually that, to me, is a negative right now.
> 
> Since there's no way we'll make LTO the default in the foreseeable
> future, people starting to use it like that is just a bad bad thing.
> 
> So really, the main advantage of LTO would be any actual optimizations
> it can do. And call me anal, but I want *numbers* for that before I
> merge it. Not handwaving. I'm not actually aware of how well - if at
> all - code generation actually improves.

Well it looks very different if you look at the generated code. gcc becomes
a lot more aggressive.

But as I said there's currently no significant performance improvement known,
so if your only goal is better performance this patch (as currently) 
known is not a big winner.  My suspicion is that's mostly because
the standard benchmarks we run are not too compiler sensitive.

However the users seem to care about the other benefits, like code size.

And there may well be loads that are compiler sensitive.
As Honza posted, for non kernel workloads LTO is known to have large benefits.

Besides at this point it's pretty much just some additions to the Makefiles.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ingo Molnar]

* Andi Kleen <ak@linux.intel.com> wrote:

> On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > >
> > > In addition to making the kernel smaller and such (I'll leave the
> > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > you, personally, should find useful and compelling: LTO will eliminate
> > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > the kernel.
> > 
> > Actually that, to me, is a negative right now.
> > 
> > Since there's no way we'll make LTO the default in the foreseeable
> > future, people starting to use it like that is just a bad bad thing.
> > 
> > So really, the main advantage of LTO would be any actual 
> > optimizations it can do. And call me anal, but I want *numbers* 
> > for that before I merge it. Not handwaving. I'm not actually aware 
> > of how well - if at all - code generation actually improves.
> 
> Well it looks very different if you look at the generated code. gcc 
> becomes a lot more aggressive.
> 
> But as I said there's currently no significant performance 
> improvement known, so if your only goal is better performance this 
> patch (as currently) known is not a big winner.  My suspicion is 
> that's mostly because the standard benchmarks we run are not too 
> compiler sensitive.
> 
> However the users seem to care about the other benefits, like code 
> size.
> 
> And there may well be loads that are compiler sensitive. As Honza 
> posted, for non kernel workloads LTO is known to have large 
> benefits.
> 
> Besides at this point it's pretty much just some additions to the 
> Makefiles.

So the reason I've been mostly ignoring the LTO patches myself (I only 
took LTO related changes that had other justifications such as 
cleanups) is that I've actually implemented full LTO in a userspace 
project myself, and my experience was:

  1) There was very little if any measurable LTO runtime speedup, 
     despite agressive GCC options and despite user-space generally 
     offering more optimizations opportunities than kernel space.

  2) LTO with current build tools meant a 1.5x-3x build speed
     slowdown (on a very fast box with tons of CPUs and RAM),
     which made LTO essentially a non-starter for development
     work. (And that was with the Gold linker.)

and looking at your characterisation of LTO you only conceded
#1 much after you started pushing LTO and you are clearly trying
to avoid talking about #2 while it's very much relevant...

I'm willing to be convinced by actual numbers, and LTO tooling might 
eventually improve, etc., but right now LTO is much ado about very 
little, being pushed in a somewhat dishonest way.

Thanks,

	Ingo

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Markus Trippelsdorf]

On 2014.04.09 at 08:01 +0200, Ingo Molnar wrote:
> 
> * Andi Kleen <ak@linux.intel.com> wrote:
> 
> > On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > > >
> > > > In addition to making the kernel smaller and such (I'll leave the
> > > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > > you, personally, should find useful and compelling: LTO will eliminate
> > > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > > the kernel.
> > > 
> > > Actually that, to me, is a negative right now.
> > > 
> > > Since there's no way we'll make LTO the default in the foreseeable
> > > future, people starting to use it like that is just a bad bad thing.
> > > 
> > > So really, the main advantage of LTO would be any actual 
> > > optimizations it can do. And call me anal, but I want *numbers* 
> > > for that before I merge it. Not handwaving. I'm not actually aware 
> > > of how well - if at all - code generation actually improves.
> > 
> > Well it looks very different if you look at the generated code. gcc 
> > becomes a lot more aggressive.
> > 
> > But as I said there's currently no significant performance 
> > improvement known, so if your only goal is better performance this 
> > patch (as currently) known is not a big winner.  My suspicion is 
> > that's mostly because the standard benchmarks we run are not too 
> > compiler sensitive.
> > 
> > However the users seem to care about the other benefits, like code 
> > size.
> > 
> > And there may well be loads that are compiler sensitive. As Honza 
> > posted, for non kernel workloads LTO is known to have large 
> > benefits.
> > 
> > Besides at this point it's pretty much just some additions to the 
> > Makefiles.
> 
> So the reason I've been mostly ignoring the LTO patches myself (I only 
> took LTO related changes that had other justifications such as 
> cleanups) is that I've actually implemented full LTO in a userspace 
> project myself, and my experience was:
> 
>   1) There was very little if any measurable LTO runtime speedup, 
>      despite agressive GCC options and despite user-space generally 
>      offering more optimizations opportunities than kernel space.
> 
>   2) LTO with current build tools meant a 1.5x-3x build speed
>      slowdown (on a very fast box with tons of CPUs and RAM),
>      which made LTO essentially a non-starter for development
>      work. (And that was with the Gold linker.)
> 
> and looking at your characterisation of LTO you only conceded
> #1 much after you started pushing LTO and you are clearly trying
> to avoid talking about #2 while it's very much relevant...
> 
> I'm willing to be convinced by actual numbers, and LTO tooling might 
> eventually improve, etc., but right now LTO is much ado about very 
> little, being pushed in a somewhat dishonest way.

I did some measurements on Andi's lto-3.14 branch:

options    size     build time
------------------------------
-O2        4408880  1:56.98
-flto -O2  4213072  2:36.22
-Os        3833248  1:45.13         
-flto -Os  3651504  2:34.51

This was measured on my AMD 4 core machine with a monolithic .config
where "CONFIG_MODULES is not set". The compiler is gcc trunk (4.9).
So on x86_86 you get 5% size reduction for 25-30% build time slowdown.

The huge RAM requirements of LTO are solved with gcc-4.9.

As for tooling support, the next version of binutils will support
automatic loading of the liblto_plugin and that will allow one to get
rid of the gcc wrappers (gcc-ar, etc.). 

It is unfortunate that a special version of binutils is still required
to build the kernel, but that will change as soon as all "ld -r"
invocations are eliminated (I think Sam Ravnborg has a patch for this).

-- 
Markus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ingo Molnar]

* Markus Trippelsdorf <markus@trippelsdorf.de> wrote:

> On 2014.04.09 at 08:01 +0200, Ingo Molnar wrote:
> > 
> > * Andi Kleen <ak@linux.intel.com> wrote:
> > 
> > > On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > > > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > > > >
> > > > > In addition to making the kernel smaller and such (I'll leave the
> > > > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > > > you, personally, should find useful and compelling: LTO will eliminate
> > > > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > > > the kernel.
> > > > 
> > > > Actually that, to me, is a negative right now.
> > > > 
> > > > Since there's no way we'll make LTO the default in the foreseeable
> > > > future, people starting to use it like that is just a bad bad thing.
> > > > 
> > > > So really, the main advantage of LTO would be any actual 
> > > > optimizations it can do. And call me anal, but I want *numbers* 
> > > > for that before I merge it. Not handwaving. I'm not actually aware 
> > > > of how well - if at all - code generation actually improves.
> > > 
> > > Well it looks very different if you look at the generated code. gcc 
> > > becomes a lot more aggressive.
> > > 
> > > But as I said there's currently no significant performance 
> > > improvement known, so if your only goal is better performance this 
> > > patch (as currently) known is not a big winner.  My suspicion is 
> > > that's mostly because the standard benchmarks we run are not too 
> > > compiler sensitive.
> > > 
> > > However the users seem to care about the other benefits, like code 
> > > size.
> > > 
> > > And there may well be loads that are compiler sensitive. As Honza 
> > > posted, for non kernel workloads LTO is known to have large 
> > > benefits.
> > > 
> > > Besides at this point it's pretty much just some additions to the 
> > > Makefiles.
> > 
> > So the reason I've been mostly ignoring the LTO patches myself (I only 
> > took LTO related changes that had other justifications such as 
> > cleanups) is that I've actually implemented full LTO in a userspace 
> > project myself, and my experience was:
> > 
> >   1) There was very little if any measurable LTO runtime speedup, 
> >      despite agressive GCC options and despite user-space generally 
> >      offering more optimizations opportunities than kernel space.
> > 
> >   2) LTO with current build tools meant a 1.5x-3x build speed
> >      slowdown (on a very fast box with tons of CPUs and RAM),
> >      which made LTO essentially a non-starter for development
> >      work. (And that was with the Gold linker.)
> > 
> > and looking at your characterisation of LTO you only conceded
> > #1 much after you started pushing LTO and you are clearly trying
> > to avoid talking about #2 while it's very much relevant...
> > 
> > I'm willing to be convinced by actual numbers, and LTO tooling might 
> > eventually improve, etc., but right now LTO is much ado about very 
> > little, being pushed in a somewhat dishonest way.
> 
> I did some measurements on Andi's lto-3.14 branch:
> 
> options    size     build time
> ------------------------------
> -O2        4408880  1:56.98
> -flto -O2  4213072  2:36.22
> -Os        3833248  1:45.13         
> -flto -Os  3651504  2:34.51
> 
> This was measured on my AMD 4 core machine with a monolithic .config
> where "CONFIG_MODULES is not set". The compiler is gcc trunk (4.9).
> So on x86_86 you get 5% size reduction for 25-30% build time slowdown.

Note that the build slowdowns you measured are more like 30-45%:

   156.22/116.98 == 33.5% slowdown
   154.51/105.13 == 46.9% slowdown

not 25-30%.

But yeah, that sounds about right and is obviously relevant data, and 
goes beyond 'a bit slower' .

Also note that the 5% size reduction due to LTO consists of two big 
parts:

  - removal of unused facilities in that .config
  - true optimizations

it would be important to know the proportion of true optimizations, 
because those are that matter most. Unused facilities will take up a 
bit of RAM, and perhaps fragment the CPU cache a tiny bit, but aren't 
nearly as relevant as true optimizations.

So the '[LTO is] not a big winner' characterisation given by Andi is a 
euphemism AFAICS, a more accurate description would be something like: 
'LTO does not help kernel performance measurably and it slows down 
kernel development'.

Thanks,

	Ingo

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Markus Trippelsdorf]

On 2014.04.14 at 12:32 +0200, Ingo Molnar wrote:
> 
> * Markus Trippelsdorf <markus@trippelsdorf.de> wrote:
> 
> > On 2014.04.09 at 08:01 +0200, Ingo Molnar wrote:
> > > 
> > > * Andi Kleen <ak@linux.intel.com> wrote:
> > > 
> > > > On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > > > > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > > > > >
> > > > > > In addition to making the kernel smaller and such (I'll leave the
> > > > > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > > > > you, personally, should find useful and compelling: LTO will eliminate
> > > > > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > > > > the kernel.
> > > > > 
> > > > > Actually that, to me, is a negative right now.
> > > > > 
> > > > > Since there's no way we'll make LTO the default in the foreseeable
> > > > > future, people starting to use it like that is just a bad bad thing.
> > > > > 
> > > > > So really, the main advantage of LTO would be any actual 
> > > > > optimizations it can do. And call me anal, but I want *numbers* 
> > > > > for that before I merge it. Not handwaving. I'm not actually aware 
> > > > > of how well - if at all - code generation actually improves.
> > > > 
> > > > Well it looks very different if you look at the generated code. gcc 
> > > > becomes a lot more aggressive.
> > > > 
> > > > But as I said there's currently no significant performance 
> > > > improvement known, so if your only goal is better performance this 
> > > > patch (as currently) known is not a big winner.  My suspicion is 
> > > > that's mostly because the standard benchmarks we run are not too 
> > > > compiler sensitive.
> > > > 
> > > > However the users seem to care about the other benefits, like code 
> > > > size.
> > > > 
> > > > And there may well be loads that are compiler sensitive. As Honza 
> > > > posted, for non kernel workloads LTO is known to have large 
> > > > benefits.
> > > > 
> > > > Besides at this point it's pretty much just some additions to the 
> > > > Makefiles.
> > > 
> > > So the reason I've been mostly ignoring the LTO patches myself (I only 
> > > took LTO related changes that had other justifications such as 
> > > cleanups) is that I've actually implemented full LTO in a userspace 
> > > project myself, and my experience was:
> > > 
> > >   1) There was very little if any measurable LTO runtime speedup, 
> > >      despite agressive GCC options and despite user-space generally 
> > >      offering more optimizations opportunities than kernel space.
> > > 
> > >   2) LTO with current build tools meant a 1.5x-3x build speed
> > >      slowdown (on a very fast box with tons of CPUs and RAM),
> > >      which made LTO essentially a non-starter for development
> > >      work. (And that was with the Gold linker.)
> > > 
> > > and looking at your characterisation of LTO you only conceded
> > > #1 much after you started pushing LTO and you are clearly trying
> > > to avoid talking about #2 while it's very much relevant...
> > > 
> > > I'm willing to be convinced by actual numbers, and LTO tooling might 
> > > eventually improve, etc., but right now LTO is much ado about very 
> > > little, being pushed in a somewhat dishonest way.
> > 
> > I did some measurements on Andi's lto-3.14 branch:
> > 
> > options    size     build time
> > ------------------------------
> > -O2        4408880  1:56.98
> > -flto -O2  4213072  2:36.22
> > -Os        3833248  1:45.13         
> > -flto -Os  3651504  2:34.51
> > 
> > This was measured on my AMD 4 core machine with a monolithic .config
> > where "CONFIG_MODULES is not set". The compiler is gcc trunk (4.9).
> > So on x86_86 you get 5% size reduction for 25-30% build time slowdown.
> 
> Note that the build slowdowns you measured are more like 30-45%:
> 
>    156.22/116.98 == 33.5% slowdown
>    154.51/105.13 == 46.9% slowdown
> 
> not 25-30%.

It's a matter of definition. I've computed the slowdown by taking the
difference: (a - b) * 100 / a

But you're right that -flto is not an option that one would use for
daily development.

-- 
Markus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ingo Molnar]

* Markus Trippelsdorf <markus@trippelsdorf.de> wrote:

> On 2014.04.14 at 12:32 +0200, Ingo Molnar wrote:
> > 
> > * Markus Trippelsdorf <markus@trippelsdorf.de> wrote:
> > 
> > > On 2014.04.09 at 08:01 +0200, Ingo Molnar wrote:
> > > > 
> > > > * Andi Kleen <ak@linux.intel.com> wrote:
> > > > 
> > > > > On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > > > > > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > > > > > >
> > > > > > > In addition to making the kernel smaller and such (I'll leave the
> > > > > > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > > > > > you, personally, should find useful and compelling: LTO will eliminate
> > > > > > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > > > > > the kernel.
> > > > > > 
> > > > > > Actually that, to me, is a negative right now.
> > > > > > 
> > > > > > Since there's no way we'll make LTO the default in the foreseeable
> > > > > > future, people starting to use it like that is just a bad bad thing.
> > > > > > 
> > > > > > So really, the main advantage of LTO would be any actual 
> > > > > > optimizations it can do. And call me anal, but I want *numbers* 
> > > > > > for that before I merge it. Not handwaving. I'm not actually aware 
> > > > > > of how well - if at all - code generation actually improves.
> > > > > 
> > > > > Well it looks very different if you look at the generated code. gcc 
> > > > > becomes a lot more aggressive.
> > > > > 
> > > > > But as I said there's currently no significant performance 
> > > > > improvement known, so if your only goal is better performance this 
> > > > > patch (as currently) known is not a big winner.  My suspicion is 
> > > > > that's mostly because the standard benchmarks we run are not too 
> > > > > compiler sensitive.
> > > > > 
> > > > > However the users seem to care about the other benefits, like code 
> > > > > size.
> > > > > 
> > > > > And there may well be loads that are compiler sensitive. As Honza 
> > > > > posted, for non kernel workloads LTO is known to have large 
> > > > > benefits.
> > > > > 
> > > > > Besides at this point it's pretty much just some additions to the 
> > > > > Makefiles.
> > > > 
> > > > So the reason I've been mostly ignoring the LTO patches myself (I only 
> > > > took LTO related changes that had other justifications such as 
> > > > cleanups) is that I've actually implemented full LTO in a userspace 
> > > > project myself, and my experience was:
> > > > 
> > > >   1) There was very little if any measurable LTO runtime speedup, 
> > > >      despite agressive GCC options and despite user-space generally 
> > > >      offering more optimizations opportunities than kernel space.
> > > > 
> > > >   2) LTO with current build tools meant a 1.5x-3x build speed
> > > >      slowdown (on a very fast box with tons of CPUs and RAM),
> > > >      which made LTO essentially a non-starter for development
> > > >      work. (And that was with the Gold linker.)
> > > > 
> > > > and looking at your characterisation of LTO you only conceded
> > > > #1 much after you started pushing LTO and you are clearly trying
> > > > to avoid talking about #2 while it's very much relevant...
> > > > 
> > > > I'm willing to be convinced by actual numbers, and LTO tooling might 
> > > > eventually improve, etc., but right now LTO is much ado about very 
> > > > little, being pushed in a somewhat dishonest way.
> > > 
> > > I did some measurements on Andi's lto-3.14 branch:
> > > 
> > > options    size     build time
> > > ------------------------------
> > > -O2        4408880  1:56.98
> > > -flto -O2  4213072  2:36.22
> > > -Os        3833248  1:45.13         
> > > -flto -Os  3651504  2:34.51
> > > 
> > > This was measured on my AMD 4 core machine with a monolithic .config
> > > where "CONFIG_MODULES is not set". The compiler is gcc trunk (4.9).
> > > So on x86_86 you get 5% size reduction for 25-30% build time slowdown.
> > 
> > Note that the build slowdowns you measured are more like 30-45%:
> > 
> >    156.22/116.98 == 33.5% slowdown
> >    154.51/105.13 == 46.9% slowdown
> > 
> > not 25-30%.
> 
> It's a matter of definition. I've computed the slowdown by taking the
> difference: (a - b) * 100 / a

That definition is IMHO deficient to the level of being broken - for 
example "it got slower by 300%" would be impossible to achieve, as 
slowdowns get larger it just converges to 100% from below, which is 
not very intuitive.

So the typical definition for "X got slower by Y%" is:

    Z = X*1.Y

For small deltas it does not matter, for larger ones it does. Anyway, 
we seem to agree :)

Thanks,

	Ingo

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Josh Triplett]

On Mon, Apr 14, 2014 at 12:32:05PM +0200, Ingo Molnar wrote:
> 
> * Markus Trippelsdorf <markus@trippelsdorf.de> wrote:
> 
> > On 2014.04.09 at 08:01 +0200, Ingo Molnar wrote:
> > > 
> > > * Andi Kleen <ak@linux.intel.com> wrote:
> > > 
> > > > On Tue, Apr 08, 2014 at 03:44:25PM -0700, Linus Torvalds wrote:
> > > > > On Tue, Apr 8, 2014 at 1:49 PM,  <josh@joshtriplett.org> wrote:
> > > > > >
> > > > > > In addition to making the kernel smaller and such (I'll leave the
> > > > > > specific stats there to Andi), here's the key awesomeness of LTO that
> > > > > > you, personally, should find useful and compelling: LTO will eliminate
> > > > > > the need to add many lower-level Kconfig symbols to compile out bits of
> > > > > > the kernel.
> > > > > 
> > > > > Actually that, to me, is a negative right now.
> > > > > 
> > > > > Since there's no way we'll make LTO the default in the foreseeable
> > > > > future, people starting to use it like that is just a bad bad thing.
> > > > > 
> > > > > So really, the main advantage of LTO would be any actual 
> > > > > optimizations it can do. And call me anal, but I want *numbers* 
> > > > > for that before I merge it. Not handwaving. I'm not actually aware 
> > > > > of how well - if at all - code generation actually improves.
> > > > 
> > > > Well it looks very different if you look at the generated code. gcc 
> > > > becomes a lot more aggressive.
> > > > 
> > > > But as I said there's currently no significant performance 
> > > > improvement known, so if your only goal is better performance this 
> > > > patch (as currently) known is not a big winner.  My suspicion is 
> > > > that's mostly because the standard benchmarks we run are not too 
> > > > compiler sensitive.
> > > > 
> > > > However the users seem to care about the other benefits, like code 
> > > > size.
> > > > 
> > > > And there may well be loads that are compiler sensitive. As Honza 
> > > > posted, for non kernel workloads LTO is known to have large 
> > > > benefits.
> > > > 
> > > > Besides at this point it's pretty much just some additions to the 
> > > > Makefiles.
> > > 
> > > So the reason I've been mostly ignoring the LTO patches myself (I only 
> > > took LTO related changes that had other justifications such as 
> > > cleanups) is that I've actually implemented full LTO in a userspace 
> > > project myself, and my experience was:
> > > 
> > >   1) There was very little if any measurable LTO runtime speedup, 
> > >      despite agressive GCC options and despite user-space generally 
> > >      offering more optimizations opportunities than kernel space.
> > > 
> > >   2) LTO with current build tools meant a 1.5x-3x build speed
> > >      slowdown (on a very fast box with tons of CPUs and RAM),
> > >      which made LTO essentially a non-starter for development
> > >      work. (And that was with the Gold linker.)
> > > 
> > > and looking at your characterisation of LTO you only conceded
> > > #1 much after you started pushing LTO and you are clearly trying
> > > to avoid talking about #2 while it's very much relevant...
> > > 
> > > I'm willing to be convinced by actual numbers, and LTO tooling might 
> > > eventually improve, etc., but right now LTO is much ado about very 
> > > little, being pushed in a somewhat dishonest way.
> > 
> > I did some measurements on Andi's lto-3.14 branch:
> > 
> > options    size     build time
> > ------------------------------
> > -O2        4408880  1:56.98
> > -flto -O2  4213072  2:36.22
> > -Os        3833248  1:45.13         
> > -flto -Os  3651504  2:34.51
> > 
> > This was measured on my AMD 4 core machine with a monolithic .config
> > where "CONFIG_MODULES is not set". The compiler is gcc trunk (4.9).
> > So on x86_86 you get 5% size reduction for 25-30% build time slowdown.
> 
> Note that the build slowdowns you measured are more like 30-45%:
> 
>    156.22/116.98 == 33.5% slowdown
>    154.51/105.13 == 46.9% slowdown
> 
> not 25-30%.
> 
> But yeah, that sounds about right and is obviously relevant data, and 
> goes beyond 'a bit slower' .
> 
> Also note that the 5% size reduction due to LTO consists of two big 
> parts:
> 
>   - removal of unused facilities in that .config
>   - true optimizations
> 
> it would be important to know the proportion of true optimizations, 
> because those are that matter most. Unused facilities will take up a 
> bit of RAM, and perhaps fragment the CPU cache a tiny bit, but aren't 
> nearly as relevant as true optimizations.

RAM usage and vmlinux size are worthwhile optimization goals as well,
and LTO makes a difference there, both directly (the 5% already
observed) and through additional changes (working to help LTO throw away
more unused code, such as helpers and lib/ bits).  And doing a better
job of compiling out unused bits means less proliferation of Kconfig
options; with LTO, we don't need a Kconfig option (and associated
dependencies added elsewhere) for every random helper or library
function.

> and it slows down 
> kernel development'.

No, it doesn't slow down development builds; it makes kernel builds
slower if and only if LTO is turned on, which most kernel developers
won't need to do.  On the other hand, distro and embedded kernels can do
so for final builds, and developers pushing to minimize the kernel can
turn it on for their own work as needed.

- Josh Triplett

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

> > and it slows down 
> > kernel development'.
> 
> No, it doesn't slow down development builds; it makes kernel builds
> slower if and only if LTO is turned on, which most kernel developers
> won't need to do.  On the other hand, distro and embedded kernels can do
> so for final builds, and developers pushing to minimize the kernel can
> turn it on for their own work as needed.

Yes and the patch even takes special care that you will not enable
it by mistake.

-Andi

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ingo Molnar]

* Josh Triplett <josh@joshtriplett.org> wrote:

> > and it slows down kernel development'.
>
> No, it doesn't slow down development builds; it makes kernel builds 
> slower if and only if LTO is turned on, which most kernel developers 
> won't need to do.
>
> On the other hand, distro and embedded kernels can do so for final 
> builds, and developers pushing to minimize the kernel can turn it on 
> for their own work as needed.

1)

If LTO will be as fragile for the kernel as it is for user space, then 
low level developers will be advised to enable it during testing.

2)

Developers and testers tend to clone distro configs to get to a 
working .config: via 'make localconfig' and similar methods. This 
means that options like this tend to spread into development 
environments.

I saw that with DEBUG_INFO frequently: I detest the option with a 
passion because it's such a drag on build time (but not as slow as 
LTO), still time and time again it shows up in a .config of mine.

So the "it does not slow down development" argument is simply not 
true. It clearly has such a cost.

LTO might still be a net win in the end, if the upsides are strong 
enough, but I am wary of arguments that try to ignore or underestimate 
the weak points.

Thanks,

	Ingo

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ralf Baechle]

On Mon, Apr 14, 2014 at 06:00:04PM -0700, Josh Triplett wrote:

> > and it slows down 
> > kernel development'.
> 
> No, it doesn't slow down development builds; it makes kernel builds
> slower if and only if LTO is turned on, which most kernel developers
> won't need to do.  On the other hand, distro and embedded kernels can do
> so for final builds, and developers pushing to minimize the kernel can

LTO inherently is going to slow down development because it does inflate
the testing matrix - a developer really should test an LTO build.

That said, the increased checking of the source code for validity across
compilation units done by the LTO final link is a benefit by itself.  With
my MIPS maintainer head on I can say it's required fixes / cleanups of
several thousand lines which have already been merged several kernel
versions ago because they all were beneficial even without LTO.  A mere
three small commits preparing arch/mips for LTO support and that don't
make sense without LTO are remaining.  So it's not a support/testing
nightmare.

And while the code size reduction is less for MIPS than what others have
reported for their platforms (I'm still investigating) is still is enough
that embedded developers would commit murder for.

  Ralf

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Sam Ravnborg]

> 
> And while the code size reduction is less for MIPS than what others have
> reported for their platforms (I'm still investigating) is still is enough
> that embedded developers would commit murder for.

I have experimented a little with a patch that links all of vmlinux in one step.
I compared the text size of vmlinux without and with -ffunction-sections.

With a defconfig build on x86 (32 bit) I got following results:

                        size     difference
singlelink          10266506
function-sections    9487369         779137  7,5%

So this is a reduction of ~800 kb by enabling -ffunction-sections which
allows the linker to throw away unused sections.

I have not boot tested the kernel so chances are that too much was thrown out by the linker.
But this is an option that has much smaller cost to use than lto.
And seems to benefit nicely in size.

I have not tried this wihtout my singlelink patch - but I assume similar results.
[My dev box is an Intel atom - so things are slow to build]

	Sam

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Markus Trippelsdorf]

On 2014.04.15 at 13:19 +0200, Sam Ravnborg wrote:
> > 
> > And while the code size reduction is less for MIPS than what others have
> > reported for their platforms (I'm still investigating) is still is enough
> > that embedded developers would commit murder for.
> 
> I have experimented a little with a patch that links all of vmlinux in one step.
> I compared the text size of vmlinux without and with -ffunction-sections.
> 
> With a defconfig build on x86 (32 bit) I got following results:
> 
>                         size     difference
> singlelink          10266506
> function-sections    9487369         779137  7,5%
> 
> So this is a reduction of ~800 kb by enabling -ffunction-sections which
> allows the linker to throw away unused sections.
> 
> I have not boot tested the kernel so chances are that too much was thrown out by the linker.
> But this is an option that has much smaller cost to use than lto.
> And seems to benefit nicely in size.
> 
> I have not tried this wihtout my singlelink patch - but I assume similar results.

No, it wouldn't work, because you cannot mix -r and --gc-sections (or
gold's --icf (identical code folding)).

-- 
Markus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Sam Ravnborg]

On Tue, Apr 15, 2014 at 01:36:02PM +0200, Markus Trippelsdorf wrote:
> On 2014.04.15 at 13:19 +0200, Sam Ravnborg wrote:
> > > 
> > > And while the code size reduction is less for MIPS than what others have
> > > reported for their platforms (I'm still investigating) is still is enough
> > > that embedded developers would commit murder for.
> > 
> > I have experimented a little with a patch that links all of vmlinux in one step.
> > I compared the text size of vmlinux without and with -ffunction-sections.
> > 
> > With a defconfig build on x86 (32 bit) I got following results:
> > 
> >                         size     difference
> > singlelink          10266506
> > function-sections    9487369         779137  7,5%
> > 
> > So this is a reduction of ~800 kb by enabling -ffunction-sections which
> > allows the linker to throw away unused sections.
> > 
> > I have not boot tested the kernel so chances are that too much was thrown out by the linker.
> > But this is an option that has much smaller cost to use than lto.
> > And seems to benefit nicely in size.
> > 
> > I have not tried this wihtout my singlelink patch - but I assume similar results.
> 
> No, it wouldn't work, because you cannot mix -r and --gc-sections (or
> gold's --icf (identical code folding)).
With make allnoconfig I see a 5% decrease in text size by applying -ffunction-sections.
This is with latest mainline and no other than the following applied:

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 602f57e..51bac0a 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -171,6 +171,8 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
 KBUILD_CFLAGS += $(call cc-option,-mno-sse -mno-mmx -mno-sse2 -mno-3dnow,)
 KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
 
+KBUILD_CFLAGS += -ffunction-sections
+
 KBUILD_CFLAGS += $(mflags-y)
 KBUILD_AFLAGS += $(mflags-y)
 

What can then explain this size difference?

	Sam

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Markus Trippelsdorf]

On 2014.04.15 at 20:19 +0200, Sam Ravnborg wrote:
> On Tue, Apr 15, 2014 at 01:36:02PM +0200, Markus Trippelsdorf wrote:
> > On 2014.04.15 at 13:19 +0200, Sam Ravnborg wrote:
> > > > 
> > > > And while the code size reduction is less for MIPS than what others have
> > > > reported for their platforms (I'm still investigating) is still is enough
> > > > that embedded developers would commit murder for.
> > > 
> > > I have experimented a little with a patch that links all of vmlinux in one step.
> > > I compared the text size of vmlinux without and with -ffunction-sections.
> > > 
> > > With a defconfig build on x86 (32 bit) I got following results:
> > > 
> > >                         size     difference
> > > singlelink          10266506
> > > function-sections    9487369         779137  7,5%
> > > 
> > > So this is a reduction of ~800 kb by enabling -ffunction-sections which
> > > allows the linker to throw away unused sections.
> > > 
> > > I have not boot tested the kernel so chances are that too much was thrown out by the linker.
> > > But this is an option that has much smaller cost to use than lto.
> > > And seems to benefit nicely in size.
> > > 
> > > I have not tried this wihtout my singlelink patch - but I assume similar results.
> > 
> > No, it wouldn't work, because you cannot mix -r and --gc-sections (or
> > gold's --icf (identical code folding)).

BTW using --gc-sections during vmlinux link time will not work, because
it will "garbage collect" the whole kernel away.

> With make allnoconfig I see a 5% decrease in text size by applying -ffunction-sections.
> This is with latest mainline and no other than the following applied:
> 
> diff --git a/arch/x86/Makefile b/arch/x86/Makefile
> index 602f57e..51bac0a 100644
> --- a/arch/x86/Makefile
> +++ b/arch/x86/Makefile
> @@ -171,6 +171,8 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
>  KBUILD_CFLAGS += $(call cc-option,-mno-sse -mno-mmx -mno-sse2 -mno-3dnow,)
>  KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
>  
> +KBUILD_CFLAGS += -ffunction-sections
> +
>  KBUILD_CFLAGS += $(mflags-y)
>  KBUILD_AFLAGS += $(mflags-y)
>  
> 
> What can then explain this size difference?

That is a good question. Because one would expect that adding
-ffunction-sections should increase the size (and it does with my
config).

-- 
Markus

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Ingo Molnar]

* Markus Trippelsdorf <markus@trippelsdorf.de> wrote:

> On 2014.04.15 at 20:19 +0200, Sam Ravnborg wrote:
> > On Tue, Apr 15, 2014 at 01:36:02PM +0200, Markus Trippelsdorf wrote:
> > > On 2014.04.15 at 13:19 +0200, Sam Ravnborg wrote:
> > > > > 
> > > > > And while the code size reduction is less for MIPS than what others have
> > > > > reported for their platforms (I'm still investigating) is still is enough
> > > > > that embedded developers would commit murder for.
> > > > 
> > > > I have experimented a little with a patch that links all of vmlinux in one step.
> > > > I compared the text size of vmlinux without and with -ffunction-sections.
> > > > 
> > > > With a defconfig build on x86 (32 bit) I got following results:
> > > > 
> > > >                         size     difference
> > > > singlelink          10266506
> > > > function-sections    9487369         779137  7,5%
> > > > 
> > > > So this is a reduction of ~800 kb by enabling -ffunction-sections which
> > > > allows the linker to throw away unused sections.
> > > > 
> > > > I have not boot tested the kernel so chances are that too much was thrown out by the linker.
> > > > But this is an option that has much smaller cost to use than lto.
> > > > And seems to benefit nicely in size.
> > > > 
> > > > I have not tried this wihtout my singlelink patch - but I assume similar results.
> > > 
> > > No, it wouldn't work, because you cannot mix -r and --gc-sections (or
> > > gold's --icf (identical code folding)).
> 
> BTW using --gc-sections during vmlinux link time will not work, because
> it will "garbage collect" the whole kernel away.
> 
> > With make allnoconfig I see a 5% decrease in text size by applying -ffunction-sections.
> > This is with latest mainline and no other than the following applied:
> > 
> > diff --git a/arch/x86/Makefile b/arch/x86/Makefile
> > index 602f57e..51bac0a 100644
> > --- a/arch/x86/Makefile
> > +++ b/arch/x86/Makefile
> > @@ -171,6 +171,8 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
> >  KBUILD_CFLAGS += $(call cc-option,-mno-sse -mno-mmx -mno-sse2 -mno-3dnow,)
> >  KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
> >  
> > +KBUILD_CFLAGS += -ffunction-sections
> > +
> >  KBUILD_CFLAGS += $(mflags-y)
> >  KBUILD_AFLAGS += $(mflags-y)
> >  
> > 
> > What can then explain this size difference?

I haven't looked at this for some time, but won't the linker eliminate 
unused input sections by default, unless told not to do that? (You 
have to add --no-gc-sections explicitly to turn this off.)

That would mean that with per function sections, unused global 
functions are eliminated. 'Unused' here means that the section only 
contains symbols that are not referenced anywhere else. With 
-ffunction-sections we'll have a lot of small per function sections, 
with a single symbol in them (the function) most of the time, IIRC.

> That is a good question. Because one would expect that adding 
> -ffunction-sections should increase the size (and it does with my 
> config).

--gc-sections appears to be platform dependent, so I suspect it 
depends on the platform.

But if -ffunction-sections can be made to work and has fewer downsides 
than the upsides then that approach should be tried first, and then 
LTO should be compared to that baseline.

Thanks,

	Ingo

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

>   1) There was very little if any measurable LTO runtime speedup, 
>      despite agressive GCC options and despite user-space generally 
>      offering more optimizations opportunities than kernel space.

See Honza's email. There are lots of benefits in various
large projects.

Also BTW compiler technology is not static. It's often
hard to interpolate from older releases, and to see
project benefits for different projects.

>   2) LTO with current build tools meant a 1.5x-3x build speed
>      slowdown (on a very fast box with tons of CPUs and RAM),

I see about 40-60% build time penalty with gcc 4.9 / single link.

>      which made LTO essentially a non-starter for development
>      work. (And that was with the Gold linker.)

I see LTO as a "release build", as opposed to a development build.
I don't think it's a big problem in such a model. If you don't
like that model, just don't enable it.

> I'm willing to be convinced by actual numbers, and LTO tooling might 
> eventually improve, etc., but right now LTO is much ado about very 
> little, being pushed in a somewhat dishonest way.

The concrete tangible advantages at this point are code size
on smaller configs.  There are a variety of users who use it for that.

I think it's worth merging for that alone.

-Andi

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

Hi Linus,

> So right now, I see several reasons not to merge it ("It's so
> experimental that we don't even want to encourage people to test it"

I don't want them to enable it during allyesconfig because they
might need more than 4GB of RAM to build it (especially with gcc 
4.8, 4.9 is better). But allyesconfig is a special case. More standard
kernels with smaller vmlinux don't have this problem, but build
somewhat slower.

> to "it's not fully fleshed out yet and makes compile times _much_
> longer").

It's functionally stable, I have a number of users who
don't report any problems.

> 
> And yet nobody has actually talked about why I *should* merge it.
> 
> Which - I think understandably - makes me less than enthusiastic.
> 
> So I think I'll let this wait a bit longer, _unless_ people start
> talking about the upsides. How much smaller is the end result? How
> much faster is it? How much more beautiful is it? Does it make new

The smaller part is mainly visible with small kernels, because
it's very good at throwing out unused code there.  All the
stuff in kernel etc. that is not used.

For example Tim Bird saw ~11% binary reduction on ARM with his 
configs [1]. We also see some reduction in small configs.

Some of the static measures like nice, for example
a LTO kernel has ~4% less calls.

We did some performance tests, but at least in the standard
macro benchmarks we do there wasn't a clear performance
win.  LKP had a small win, but nothing dramatic.
But I would like others to test it on their workloads.

In principle LTO can do cool optimizations, like propagating
constants into functions (e.g. generate specialized versions
of some code). I experimented a bit with this, however
it currently seems to bloat the code quite a bit.

There are some other possible future optimizations
that can be enabled by a global optimizer.

Honza may have more reasons for LTO.

Other benefits are global warnings and some additional
type checking. The LTO log files are really useful
to do global call graph analysis and similar.

-Andi

[1] http://elinux.org/images/9/9e/Bird-Kernel-Size-Optimization-LCJ-2013.pdf

-- 
ak@linux.intel.com -- Speaking for myself only

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Jan Hubicka]

> Hi Linus,
> 
> > So right now, I see several reasons not to merge it ("It's so
> > experimental that we don't even want to encourage people to test it"
> 
> I don't want them to enable it during allyesconfig because they
> might need more than 4GB of RAM to build it (especially with gcc 
> 4.8, 4.9 is better). But allyesconfig is a special case. More standard
> kernels with smaller vmlinux don't have this problem, but build
> somewhat slower.
> 
> > to "it's not fully fleshed out yet and makes compile times _much_
> > longer").
> 
> It's functionally stable, I have a number of users who
> don't report any problems.
> 
> > 
> > And yet nobody has actually talked about why I *should* merge it.
> > 
> > Which - I think understandably - makes me less than enthusiastic.
> > 
> > So I think I'll let this wait a bit longer, _unless_ people start
> > talking about the upsides. How much smaller is the end result? How
> > much faster is it? How much more beautiful is it? Does it make new
> 
> The smaller part is mainly visible with small kernels, because
> it's very good at throwing out unused code there.  All the
> stuff in kernel etc. that is not used.
> 
> For example Tim Bird saw ~11% binary reduction on ARM with his 
> configs [1]. We also see some reduction in small configs.
> 
> Some of the static measures like nice, for example
> a LTO kernel has ~4% less calls.
> 
> We did some performance tests, but at least in the standard
> macro benchmarks we do there wasn't a clear performance
> win.  LKP had a small win, but nothing dramatic.
> But I would like others to test it on their workloads.
> 
> In principle LTO can do cool optimizations, like propagating
> constants into functions (e.g. generate specialized versions
> of some code). I experimented a bit with this, however
> it currently seems to bloat the code quite a bit.
> 
> There are some other possible future optimizations
> that can be enabled by a global optimizer.
> 
> Honza may have more reasons for LTO.

My basic understanding of LTO benefits is about the following.

 1) Today LTO will quite reliably reduce code size. 
    Andi mentione 11% of kernel. It is not that unusual to get over 30% code
    size reduction.

    Generally it is a lot easier to fine tune hot spots than to throw away all
    unnecesary code in all possible configurations your project might have 
    (+ optimize by hand code layout).  So even well hand optimized programs
    benefits from LTO in code size.

 2) If build machinery is well structured, code size reduction also translates
    to compile time improvements (GCC spends a lot of time in codegen).
    This holds only for full rebuild (not compile/edit cycle) and only for
    projects that do not rebuild one binary many times (like both kernel and
    GCC does) or do not link many times large LTO library (like Firefox).

    For SPEC2k6, the LTO build time is faster, for Firefox it is about the same
    For GCC the bootstrap time is slower since our build system needs reorg (we
    use old libtool that requires fat LTO files and we rebuild every binary
    twice just to get a checksum, we also link whole backend as static library
    to every frontend binary) 

    I hope that not in too distant future we will be able to build majority of
    distro with LTO and get LTO build times better than non-LTO.
    I also have some longer term plans for compile-edit development model that
    won't trigger reoptimization of whole binary, but that is bit more of
    research stage right now.

 3) On really large projects, LTO may need a lot of memory.
    This is basically problem of kernel/firefox and chromium. Notihng else
    on my installation has similarly large binary.  We are improving this from
    release to release and I believe 4.9 is doing pretty well so I can build
    those things on my 8GB laptop w/o swap storms.

 4) LTO brings noticeable performance wins on average, but it is largely benchmark
    dependent; some see huge improvemnts, others no improvements at all.

    Basic observation is there is not much that LTO can do that can not be done
    by hand.  Careful developer can just identify the important spot and
    restructure the soruces.
    The runtime benefits are more visible on bigger, bloated and less
    optimized projects than on hand tuned video encoder implementation.
    I believe Kernel largely falls into hand tuned category despite its size.

    I am in progress of trying to benchmark GCC 4.9 LTO for Firefox/Libreoffice
    and Chromium and will publish once it is done.
    Just as very quick data point (not too serious) I just run Dromaeo
    benchmarks on firefox comparing default and LTO build, the overall
    difference is 7% http://dromaeo.com/?id=219677,219678,219672,219676
    (first two tests are default build, second two are LTO).
    This is a lot more than I expected given that dromaeo tests largery JIT
    generated code and I am sure it is a common benchmark well hand optimized.

    Vladimir has some SPEC2k scores http://vmakarov.fedorapeople.org/spec/
    and in a brief summary at http://vmakarov.fedorapeople.org/spec/peak.html

    I would be curious about the results on Kernel.

 5) LTO combine very well with profile feedback directed optimization (FDO). 
    One of problem of compiler (and compiler developer) is what to do with
    all the extra freedom one suddenly gets.  FDO really helps.

    GCC 4.9 will optimize code layout with FDO (something Martin
    Liska implemented based on Taras Glek's analysis) and do a lot better
    inlining than without.
    I hope the static code (without FDO) layout will also get better in relese
    after 4.9.  This makes programs to start measurably faster and touch less
    pages. http://arxiv.org/pdf/1403.6997.pdf

 6) LTO will pay back more in long term.

    It is not only because LTO implementation in GCC (and LLVM) has more room
    for improvement than the per-file optimizers.

    Main things is that despite the aim to be transparent to user,
    LTO is invasive change.  Existing programs was developed and tuned for
    per-file optimization model and many of them contains a lot of hacks to
    work around its limitation (such as a lot of inline code in headers, etc.)
    With LTO becoming mainstream, developers will have time to work on different
    hacks ;)

    I also believe that in addition to optimization, we will see more static
    analysis built around in future.

In 2010 I started working on project making large apps working with GCC LTO
(http://arxiv.org/pdf/1010.2196.pdf). It is a long run to a moving target.
I am very happy Andi did the hard work on getting Kernel work and would like
to see the patches upstream. There are measurable improvements now and more
users we get, the faster will LTO improve.

Honza
> 
> Other benefits are global warnings and some additional
> type checking. The LTO log files are really useful
> to do global call graph analysis and similar.
> 
> -Andi
> 
> [1] http://elinux.org/images/9/9e/Bird-Kernel-Size-Optimization-LCJ-2013.pdf
> 
> -- 
> ak@linux.intel.com -- Speaking for myself only

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Andi Kleen]

Thanks Honza. Just one comment:

>     The runtime benefits are more visible on bigger, bloated and less
>     optimized projects than on hand tuned video encoder implementation.
>     I believe Kernel largely falls into hand tuned category despite its size.

In my experience there's a lot of badly tuned code in the kernel these days,
especially when you go outside the core code (i.e. into drivers/*)

Or code that used to be tuned, but isn't aftermore after several years
of feature additions and bug fixes. The kernel code quality is also quite varying.

So anything the compiler can do helps.

>     I would be curious about the results on Kernel.

We saw some upsides in performance with some standard tests, but nothing 
too significant.

-Andi

Re: [GIT] kbuild/lto changes for 3.15-rc1

[Tim Bird]

FWIW, I'd really like to see this go in as an experimental feature.

Andi has already quoted my size results, which I thought were pretty
good, as well as given a pointer to my size optimization presentation.

Some of what follows is in the presentation, but here is a summary:
There are other automated reductions that I experimented with, that
are made more effective (or are only possible) with LTO.  Examples of
these include:
 - elimination of unused kernel command line handlers,
 - automated elimination of unused syscalls (through whole-system analysis), and
 - an experimental system I developed for doing structure reduction.

LTO shows promise for allowing more automation in configuration
handling (that is, requiring less CONFIG options).

People should definitely be warned off using this in any production
setting, but I think it's valuable for developers experimenting with
tiny-size systems to have this easily available in mainline.

On Tue, Apr 8, 2014 at 3:49 PM, Andi Kleen <ak@linux.intel.com> wrote:
> Hi Linus,
>
>> So right now, I see several reasons not to merge it ("It's so
>> experimental that we don't even want to encourage people to test it"
>
> I don't want them to enable it during allyesconfig because they
> might need more than 4GB of RAM to build it (especially with gcc
> 4.8, 4.9 is better). But allyesconfig is a special case. More standard
> kernels with smaller vmlinux don't have this problem, but build
> somewhat slower.
>
>> to "it's not fully fleshed out yet and makes compile times _much_
>> longer").
>
> It's functionally stable, I have a number of users who
> don't report any problems.
>
>>
>> And yet nobody has actually talked about why I *should* merge it.
>>
>> Which - I think understandably - makes me less than enthusiastic.
>>
>> So I think I'll let this wait a bit longer, _unless_ people start
>> talking about the upsides. How much smaller is the end result? How
>> much faster is it? How much more beautiful is it? Does it make new
>
> The smaller part is mainly visible with small kernels, because
> it's very good at throwing out unused code there.  All the
> stuff in kernel etc. that is not used.
>
> For example Tim Bird saw ~11% binary reduction on ARM with his
> configs [1]. We also see some reduction in small configs.
>
> Some of the static measures like nice, for example
> a LTO kernel has ~4% less calls.
>
> We did some performance tests, but at least in the standard
> macro benchmarks we do there wasn't a clear performance
> win.  LKP had a small win, but nothing dramatic.
> But I would like others to test it on their workloads.
>
> In principle LTO can do cool optimizations, like propagating
> constants into functions (e.g. generate specialized versions
> of some code). I experimented a bit with this, however
> it currently seems to bloat the code quite a bit.
>
> There are some other possible future optimizations
> that can be enabled by a global optimizer.
>
> Honza may have more reasons for LTO.
>
> Other benefits are global warnings and some additional
> type checking. The LTO log files are really useful
> to do global call graph analysis and similar.
>
> -Andi
>
> [1] http://elinux.org/images/9/9e/Bird-Kernel-Size-Optimization-LCJ-2013.pdf
>
> --
> ak@linux.intel.com -- Speaking for myself only
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
 -- Tim Bird
Senior Software Engineer, Sony Mobile
Architecture Group Chair, CE Workgroup, Linux Foundation