flto automake

flto automake

[James Feist]

If you aren't planning on enabling flto in a repo you can ignore this email.


I've created a new class flto-automake 
https://github.com/openbmc/meta-phosphor/blob/master/classes/flto-automake.bbclass 
if you are enabling -flto in an autotools project you might be 
interested in. When using static libraries you may run into undefined 
references when building with Yocto. Yocto by default uses:

export AR = "${HOST_PREFIX}ar"
export RANLIB = "${HOST_PREFIX}ranlib"

for AR and RANLIB. This new class replaces these with the gcc equivalent 
wrappers that can work with the flto object files. This doesn't seem to 
be an issue for the CI build, it only shows up for Yocto builds. 
Reference: 
http://stackoverflow.com/questions/25878407/how-can-i-use-lto-with-static-libraries#comment67660913_25878408

For Cmake I suggest looking at the way bmcweb handles it in CMakesLists:
https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt#L92

Thanks,

James

Re: flto automake

[Brad Bishop]

> On Dec 18, 2018, at 1:08 PM, James Feist <james.feist@linux.intel.com> wrote:
> 
> If you aren't planning on enabling flto in a repo you can ignore this email.
> 
> 
> I've created a new class flto-automake https://github.com/openbmc/meta-phosphor/blob/master/classes/flto-automake.bbclass if you are enabling -flto in an autotools project you might be interested in. When using static libraries you may run into undefined references when building with Yocto. Yocto by default uses:
> 
> export AR = "${HOST_PREFIX}ar"
> export RANLIB = "${HOST_PREFIX}ranlib"
> 
> for AR and RANLIB. This new class replaces these with the gcc equivalent wrappers that can work with the flto object files. This doesn't seem to be an issue for the CI build, it only shows up for Yocto builds. Reference: http://stackoverflow.com/questions/25878407/how-can-i-use-lto-with-static-libraries#comment67660913_25878408
> 
> For Cmake I suggest looking at the way bmcweb handles it in CMakesLists:
> https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt#L92
> 
> Thanks,
> 
> James

Thanks James

I wonder if this is something they would be interested in upstream.  I’ve copied
the oe-core mailing list for possible comment.

-brad

Re: [OE-core] flto automake

[Burton, Ross]

On Wed, 19 Dec 2018 at 00:42, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:

> I wonder if this is something they would be interested in upstream.  I’ve copied
> the oe-core mailing list for possible comment.

Thanks for forwarding this to us.  The class is very simple and just does:

PACKAGECONFIG_CONFARGS += " AR=${TARGET_PREFIX}gcc-ar \
RANLIB=${TARGET_PREFIX}gcc-ranlib"

That's quite an abuse of PACKAGECONFIG_CONFARGS, I'd recommend just
setting AR and RANLIB directly in the class, they'll override the
defaults in bitbake.conf.

That however leads to the question of should we just change the
defaults? Does anything break if we use gcc-ar instead of ar?

Ross

Re: flto automake

[Burton, Ross]

On Wed, 19 Dec 2018 at 00:42, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:

> I wonder if this is something they would be interested in upstream.  I’ve copied
> the oe-core mailing list for possible comment.

Thanks for forwarding this to us.  The class is very simple and just does:

PACKAGECONFIG_CONFARGS += " AR=${TARGET_PREFIX}gcc-ar \
RANLIB=${TARGET_PREFIX}gcc-ranlib"

That's quite an abuse of PACKAGECONFIG_CONFARGS, I'd recommend just
setting AR and RANLIB directly in the class, they'll override the
defaults in bitbake.conf.

That however leads to the question of should we just change the
defaults? Does anything break if we use gcc-ar instead of ar?

Ross

Re: [OE-core] flto automake

[Lei YU]

On Wed, Dec 19, 2018 at 9:07 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Wed, 19 Dec 2018 at 00:42, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> > I wonder if this is something they would be interested in upstream.  I’ve copied
> > the oe-core mailing list for possible comment.
>
> Thanks for forwarding this to us.  The class is very simple and just does:
>
> PACKAGECONFIG_CONFARGS += " AR=${TARGET_PREFIX}gcc-ar \
> RANLIB=${TARGET_PREFIX}gcc-ranlib"

Oh I like it in a common class! We have "ugly" AC_SUBST for AR and RANLIB to
gcc one in https://github.com/openbmc/phosphor-time-manager/blob/master/configure.ac#L10-L11
Let's see how upstream think about this.

Re: [OE-core] flto automake

[Burton, Ross]

On Wed, 19 Dec 2018 at 01:07, Burton, Ross <ross.burton@intel.com> wrote:

> That however leads to the question of should we just change the
> defaults? Does anything break if we use gcc-ar instead of ar?

Answer: yes.

| x86_64-poky-linux-gcc-ar  rc libgcov.a $objects
| x86_64-poky-linux-gcc-ar: Cannot find plugin 'liblto_plugin.so'
| Makefile:932: recipe for target 'libgcov.a' failed
ERROR: Task (/home/ross/Yocto/poky/meta/recipes-devtools/gcc/libgcc-initial_8.2.bb:do_compile)
failed with exit code '1'

I don't know enough about gcc...

Ross

Re: flto automake

[Burton, Ross]

On Wed, 19 Dec 2018 at 01:07, Burton, Ross <ross.burton@intel.com> wrote:

> That however leads to the question of should we just change the
> defaults? Does anything break if we use gcc-ar instead of ar?

Answer: yes.

| x86_64-poky-linux-gcc-ar  rc libgcov.a $objects
| x86_64-poky-linux-gcc-ar: Cannot find plugin 'liblto_plugin.so'
| Makefile:932: recipe for target 'libgcov.a' failed
ERROR: Task (/home/ross/Yocto/poky/meta/recipes-devtools/gcc/libgcc-initial_8.2.bb:do_compile)
failed with exit code '1'

I don't know enough about gcc...

Ross

Re: [OE-core] flto automake

[Richard Purdie]

On Wed, 2018-12-19 at 10:00 +0000, Burton, Ross wrote:
> On Wed, 19 Dec 2018 at 01:07, Burton, Ross <ross.burton@intel.com>
> wrote:
> 
> > That however leads to the question of should we just change the
> > defaults? Does anything break if we use gcc-ar instead of ar?
> 
> Answer: yes.
> 
> > x86_64-poky-linux-gcc-ar  rc libgcov.a $objects
> > x86_64-poky-linux-gcc-ar: Cannot find plugin 'liblto_plugin.so'
> > Makefile:932: recipe for target 'libgcov.a' failed
> ERROR: Task (/home/ross/Yocto/poky/meta/recipes-devtools/gcc/libgcc-
> initial_8.2.bb:do_compile)
> failed with exit code '1'
> 
> I don't know enough about gcc...

The -initial compiler doesn't have lto compiled into it so you probably
need to use binutils ar for that part of the bootstrap.

Cheers,

Richard

Re: flto automake

[Richard Purdie]

On Wed, 2018-12-19 at 10:00 +0000, Burton, Ross wrote:
> On Wed, 19 Dec 2018 at 01:07, Burton, Ross <ross.burton@intel.com>
> wrote:
> 
> > That however leads to the question of should we just change the
> > defaults? Does anything break if we use gcc-ar instead of ar?
> 
> Answer: yes.
> 
> > x86_64-poky-linux-gcc-ar  rc libgcov.a $objects
> > x86_64-poky-linux-gcc-ar: Cannot find plugin 'liblto_plugin.so'
> > Makefile:932: recipe for target 'libgcov.a' failed
> ERROR: Task (/home/ross/Yocto/poky/meta/recipes-devtools/gcc/libgcc-
> initial_8.2.bb:do_compile)
> failed with exit code '1'
> 
> I don't know enough about gcc...

The -initial compiler doesn't have lto compiled into it so you probably
need to use binutils ar for that part of the bootstrap.

Cheers,

Richard

Re: [OE-core] flto automake

[Khem Raj]

On Tue, Dec 18, 2018 at 4:42 PM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
>
>
> > On Dec 18, 2018, at 1:08 PM, James Feist <james.feist@linux.intel.com> wrote:
> >
> > If you aren't planning on enabling flto in a repo you can ignore this email.
> >
> >
> > I've created a new class flto-automake https://github.com/openbmc/meta-phosphor/blob/master/classes/flto-automake.bbclass if you are enabling -flto in an autotools project you might be interested in. When using static libraries you may run into undefined references when building with Yocto. Yocto by default uses:


The changes are right but I think flto should also be a DISTRO_FEATURE
and this change should become effective only when flto is in
DISTRO_FEATURES

> >
> > export AR = "${HOST_PREFIX}ar"
> > export RANLIB = "${HOST_PREFIX}ranlib"
> >
> > for AR and RANLIB. This new class replaces these with the gcc equivalent wrappers that can work with the flto object files. This doesn't seem to be an issue for the CI build, it only shows up for Yocto builds. Reference: http://stackoverflow.com/questions/25878407/how-can-i-use-lto-with-static-libraries#comment67660913_25878408
> >
> > For Cmake I suggest looking at the way bmcweb handles it in CMakesLists:
> > https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt#L92

hardcoding is not right. We have cross versions of these tools
installed during OE build and we also have options to use non-gcc
compilers, e.g. llvm provides its own verisons of llvm-ar and
llvm-ranlib and llvm-nm which should be used when using clang
or other llvm based static compilers.
This is right fix for cross compiling with CMake
https://github.com/YoeDistro/meta-ti/blob/yoe/mut/recipes-bsp/rwmem/rwmem/0001-Use-CMake-provided-vars-for-ar-and-ranlib.patch

> >
> > Thanks,
> >
> > James
>
> Thanks James
>
> I wonder if this is something they would be interested in upstream.  I’ve copied
> the oe-core mailing list for possible comment.
>
> -brad
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: flto automake

[Khem Raj]

On Tue, Dec 18, 2018 at 4:42 PM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
>
>
> > On Dec 18, 2018, at 1:08 PM, James Feist <james.feist@linux.intel.com> wrote:
> >
> > If you aren't planning on enabling flto in a repo you can ignore this email.
> >
> >
> > I've created a new class flto-automake https://github.com/openbmc/meta-phosphor/blob/master/classes/flto-automake.bbclass if you are enabling -flto in an autotools project you might be interested in. When using static libraries you may run into undefined references when building with Yocto. Yocto by default uses:


The changes are right but I think flto should also be a DISTRO_FEATURE
and this change should become effective only when flto is in
DISTRO_FEATURES

> >
> > export AR = "${HOST_PREFIX}ar"
> > export RANLIB = "${HOST_PREFIX}ranlib"
> >
> > for AR and RANLIB. This new class replaces these with the gcc equivalent wrappers that can work with the flto object files. This doesn't seem to be an issue for the CI build, it only shows up for Yocto builds. Reference: http://stackoverflow.com/questions/25878407/how-can-i-use-lto-with-static-libraries#comment67660913_25878408
> >
> > For Cmake I suggest looking at the way bmcweb handles it in CMakesLists:
> > https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt#L92

hardcoding is not right. We have cross versions of these tools
installed during OE build and we also have options to use non-gcc
compilers, e.g. llvm provides its own verisons of llvm-ar and
llvm-ranlib and llvm-nm which should be used when using clang
or other llvm based static compilers.
This is right fix for cross compiling with CMake
https://github.com/YoeDistro/meta-ti/blob/yoe/mut/recipes-bsp/rwmem/rwmem/0001-Use-CMake-provided-vars-for-ar-and-ranlib.patch

> >
> > Thanks,
> >
> > James
>
> Thanks James
>
> I wonder if this is something they would be interested in upstream.  I’ve copied
> the oe-core mailing list for possible comment.
>
> -brad
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] flto automake

[Khem Raj]

On Tue, Dec 18, 2018 at 5:07 PM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Wed, 19 Dec 2018 at 00:42, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> > I wonder if this is something they would be interested in upstream.  I’ve copied
> > the oe-core mailing list for possible comment.
>
> Thanks for forwarding this to us.  The class is very simple and just does:
>
> PACKAGECONFIG_CONFARGS += " AR=${TARGET_PREFIX}gcc-ar \
> RANLIB=${TARGET_PREFIX}gcc-ranlib"
>
> That's quite an abuse of PACKAGECONFIG_CONFARGS, I'd recommend just
> setting AR and RANLIB directly in the class, they'll override the
> defaults in bitbake.conf.
>
> That however leads to the question of should we just change the
> defaults? Does anything break if we use gcc-ar instead of ar?

Dont think that is a good idea in general since gcc-ar is a wrapper
around normal ar
passing --plugin=/path/to/lto/plugin/so and that may not be right
thing to do always.

>
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: flto automake

[Khem Raj]

On Tue, Dec 18, 2018 at 5:07 PM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Wed, 19 Dec 2018 at 00:42, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> > I wonder if this is something they would be interested in upstream.  I’ve copied
> > the oe-core mailing list for possible comment.
>
> Thanks for forwarding this to us.  The class is very simple and just does:
>
> PACKAGECONFIG_CONFARGS += " AR=${TARGET_PREFIX}gcc-ar \
> RANLIB=${TARGET_PREFIX}gcc-ranlib"
>
> That's quite an abuse of PACKAGECONFIG_CONFARGS, I'd recommend just
> setting AR and RANLIB directly in the class, they'll override the
> defaults in bitbake.conf.
>
> That however leads to the question of should we just change the
> defaults? Does anything break if we use gcc-ar instead of ar?

Dont think that is a good idea in general since gcc-ar is a wrapper
around normal ar
passing --plugin=/path/to/lto/plugin/so and that may not be right
thing to do always.

>
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 01:20, Khem Raj <raj.khem@gmail.com> wrote:
> > That however leads to the question of should we just change the
> > defaults? Does anything break if we use gcc-ar instead of ar?
>
> Dont think that is a good idea in general since gcc-ar is a wrapper
> around normal ar
> passing --plugin=/path/to/lto/plugin/so and that may not be right
> thing to do always.

Well, I built core-image-sato core-image-weston with AR=gcc-ar
RANLIB=gcc-ranlib globally and it worked fine.

As we build the cross gcc we know what is in it.  Other toolchains be
they clang, ICC, or external GCCs, can override AR just like they do
the other variables.

Ross

Re: flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 01:20, Khem Raj <raj.khem@gmail.com> wrote:
> > That however leads to the question of should we just change the
> > defaults? Does anything break if we use gcc-ar instead of ar?
>
> Dont think that is a good idea in general since gcc-ar is a wrapper
> around normal ar
> passing --plugin=/path/to/lto/plugin/so and that may not be right
> thing to do always.

Well, I built core-image-sato core-image-weston with AR=gcc-ar
RANLIB=gcc-ranlib globally and it worked fine.

As we build the cross gcc we know what is in it.  Other toolchains be
they clang, ICC, or external GCCs, can override AR just like they do
the other variables.

Ross

Re: [OE-core] flto automake

[Khem Raj]

On Thu, Dec 20, 2018 at 4:37 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Thu, 20 Dec 2018 at 01:20, Khem Raj <raj.khem@gmail.com> wrote:
> > > That however leads to the question of should we just change the
> > > defaults? Does anything break if we use gcc-ar instead of ar?
> >
> > Dont think that is a good idea in general since gcc-ar is a wrapper
> > around normal ar
> > passing --plugin=/path/to/lto/plugin/so and that may not be right
> > thing to do always.
>
> Well, I built core-image-sato core-image-weston with AR=gcc-ar
> RANLIB=gcc-ranlib globally and it worked fine.

Please build a lot more. These images cover only handful of recipes
there are problems like

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576

which we might have to contend.

>
> As we build the cross gcc we know what is in it.  Other toolchains be
> they clang, ICC, or external GCCs, can override AR just like they do
> the other variables.
>

Yes they can, but what doe this buy us ? except we make Core toolchain
gcc dependent one step more.

> Ross

Re: flto automake

[Khem Raj]

On Thu, Dec 20, 2018 at 4:37 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Thu, 20 Dec 2018 at 01:20, Khem Raj <raj.khem@gmail.com> wrote:
> > > That however leads to the question of should we just change the
> > > defaults? Does anything break if we use gcc-ar instead of ar?
> >
> > Dont think that is a good idea in general since gcc-ar is a wrapper
> > around normal ar
> > passing --plugin=/path/to/lto/plugin/so and that may not be right
> > thing to do always.
>
> Well, I built core-image-sato core-image-weston with AR=gcc-ar
> RANLIB=gcc-ranlib globally and it worked fine.

Please build a lot more. These images cover only handful of recipes
there are problems like

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576

which we might have to contend.

>
> As we build the cross gcc we know what is in it.  Other toolchains be
> they clang, ICC, or external GCCs, can override AR just like they do
> the other variables.
>

Yes they can, but what doe this buy us ? except we make Core toolchain
gcc dependent one step more.

> Ross

Re: [OE-core] flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > As we build the cross gcc we know what is in it.  Other toolchains be
> > they clang, ICC, or external GCCs, can override AR just like they do
> > the other variables.
>
> Yes they can, but what doe this buy us ? except we make Core toolchain
> gcc dependent one step more.

The core toolchain is currently GCC dependent already by definition.
I'm actually about to start working on abstracting the core so it
doesn't make any assumptions, like how there's no hard assumption of
glibc in the core.

Ross

Re: flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > As we build the cross gcc we know what is in it.  Other toolchains be
> > they clang, ICC, or external GCCs, can override AR just like they do
> > the other variables.
>
> Yes they can, but what doe this buy us ? except we make Core toolchain
> gcc dependent one step more.

The core toolchain is currently GCC dependent already by definition.
I'm actually about to start working on abstracting the core so it
doesn't make any assumptions, like how there's no hard assumption of
glibc in the core.

Ross

Re: [OE-core] flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > Well, I built core-image-sato core-image-weston with AR=gcc-ar
> > RANLIB=gcc-ranlib globally and it worked fine.
>
> Please build a lot more. These images cover only handful of recipes
> there are problems like
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576
>
> which we might have to contend.

That bug from two years ago suggests that the gcc-ar wrapper isn't
required if GCC has been configured properly.  I guess we need a test
case.

Ross

Re: flto automake

[Burton, Ross]

On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > Well, I built core-image-sato core-image-weston with AR=gcc-ar
> > RANLIB=gcc-ranlib globally and it worked fine.
>
> Please build a lot more. These images cover only handful of recipes
> there are problems like
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576
>
> which we might have to contend.

That bug from two years ago suggests that the gcc-ar wrapper isn't
required if GCC has been configured properly.  I guess we need a test
case.

Ross

Re: [OE-core] flto automake

[Khem Raj]

On Thu, Dec 20, 2018 at 7:48 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > > Well, I built core-image-sato core-image-weston with AR=gcc-ar
> > > RANLIB=gcc-ranlib globally and it worked fine.
> >
> > Please build a lot more. These images cover only handful of recipes
> > there are problems like
> >
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576
> >
> > which we might have to contend.
>
> That bug from two years ago suggests that the gcc-ar wrapper isn't
> required if GCC has been configured properly.  I guess we need a test
> case.

cmake's approach of defined COMPILER_AR and COMPILER_RANLIB etc seems
better to me. where it can be uses selectively only when LTO is in
play.

>
> Ross

Re: flto automake

[Khem Raj]

On Thu, Dec 20, 2018 at 7:48 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Thu, 20 Dec 2018 at 15:37, Khem Raj <raj.khem@gmail.com> wrote:
> > > Well, I built core-image-sato core-image-weston with AR=gcc-ar
> > > RANLIB=gcc-ranlib globally and it worked fine.
> >
> > Please build a lot more. These images cover only handful of recipes
> > there are problems like
> >
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77576
> >
> > which we might have to contend.
>
> That bug from two years ago suggests that the gcc-ar wrapper isn't
> required if GCC has been configured properly.  I guess we need a test
> case.

cmake's approach of defined COMPILER_AR and COMPILER_RANLIB etc seems
better to me. where it can be uses selectively only when LTO is in
play.

>
> Ross