[Buildroot] [PATCH v2 1/4] package/Makefile.in: use gcc wrappers for binutils tools

[Buildroot] [PATCH v2 1/4] package/Makefile.in: use gcc wrappers for binutils tools

[Norbert Lange]

This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
normal binutils tools. The difference is that with the
wrappers, gcc plugins will be automatically picked up.

gcc 4.7 introduced these wrappers, to detect the prefix and
keep gcc specifics out of Makefile.in, a new variable
BR2_TOOLCHAIN_BUTILS_PREFIX will be used to carry the
prefix on supported versions.

Note that binutils added some automatic loading with the
'bfd-plugins' directory (somewhere around 2.28), but
the first implementation had issues, and generally depends on
correctly setup symlinks (often broken, may point to some
other gcc's library). The wrappers always work painless.

The original motivation (now ~2 years in use) was to add
"-flto -ffat-lto-objects" to both BR2_TARGET_OPTIMIZATION and
BR2_TARGET_LDFLAGS, and have target binaries lto optimized.

Not all packages will compile with this option, further work
could white/blacklist packages (adding -fno-lto to the
options).

Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
v1->v2:
*   support older gcc versions missing those wrappers
---
 package/Makefile.in | 6 +++---
 toolchain/Config.in | 7 +++++++
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 51f5cbce4f..665edec539 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -198,15 +198,15 @@ TARGET_CROSS = $(HOST_DIR)/bin/$(TOOLCHAIN_EXTERNAL_PREFIX)-
 endif
 
 # Define TARGET_xx variables for all common binutils/gcc
-TARGET_AR       = $(TARGET_CROSS)ar
+TARGET_AR       = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))ar
 TARGET_AS       = $(TARGET_CROSS)as
 TARGET_CC       = $(TARGET_CROSS)gcc
 TARGET_CPP      = $(TARGET_CROSS)cpp
 TARGET_CXX      = $(TARGET_CROSS)g++
 TARGET_FC       = $(TARGET_CROSS)gfortran
 TARGET_LD       = $(TARGET_CROSS)ld
-TARGET_NM       = $(TARGET_CROSS)nm
-TARGET_RANLIB   = $(TARGET_CROSS)ranlib
+TARGET_NM       = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))nm
+TARGET_RANLIB   = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))ranlib
 TARGET_READELF  = $(TARGET_CROSS)readelf
 TARGET_OBJCOPY  = $(TARGET_CROSS)objcopy
 TARGET_OBJDUMP  = $(TARGET_CROSS)objdump
diff --git a/toolchain/Config.in b/toolchain/Config.in
index db2ab0f059..7a2becc09a 100644
--- a/toolchain/Config.in
+++ b/toolchain/Config.in
@@ -732,4 +732,11 @@ config BR2_TOOLCHAIN_HAS_LIBQUADMATH
 	bool
 	default y if BR2_i386 || BR2_x86_64
 
+# gcc ships with wrappers that will automatically pass arguments
+# to the binutils tools.
+# So far, those are paths to necessary linker plugins
+config BR2_TOOLCHAIN_BUTILS_PREFIX
+	string
+	default "gcc-" if BR2_TOOLCHAIN_GCC_AT_LEAST_4_7
+
 endmenu
-- 
2.28.0

[Buildroot] [PATCH v2 2/4] package/gcc: use binutils wrappers for target libs

[Norbert Lange]

This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
normal binutils tools. The difference is that with the
wrappers, gcc plugins will be automatically picked up,
which might be necessary with some flags.

Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
v1->v2:
*   Only set those wrappers if gc 4.7 or newer
---
 package/gcc/gcc.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
index 114c2887a0..a24ffe3ee2 100644
--- a/package/gcc/gcc.mk
+++ b/package/gcc/gcc.mk
@@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error
 endif
 
 # Propagate options used for target software building to GCC target libs
+ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
+HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib
+endif
 HOST_GCC_COMMON_CONF_ENV += CFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CFLAGS)"
 HOST_GCC_COMMON_CONF_ENV += CXXFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CXXFLAGS)"
 
-- 
2.28.0

[Buildroot] [PATCH v2 3/4] package/busybox: explicitly state binutil paths

[Norbert Lange]

Pass paths to ar, nm, ranlib binaries. This allows linker-plugins
to be used.

Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
 package/busybox/busybox.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 01f7331221..221b8adf59 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -94,6 +94,9 @@ BUSYBOX_MAKE_ENV += \
 endif
 
 BUSYBOX_MAKE_OPTS = \
+	AR="$(TARGET_AR)" \
+	NM="$(TARGET_NM)" \
+	RANLIB="$(TARGET_RANLIB)" \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
 	PREFIX="$(TARGET_DIR)" \
-- 
2.28.0

[Buildroot] [PATCH v2 4/4] package/glibc: force -fno-lto

[Norbert Lange]

glibc requires compilation barriers between files, and will
fail with LTO enabled.
So force LTO off by appending -fno-lto to the flags, but only
if the GCC version is recent enough that an LTO build would
be possible.

Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
 package/glibc/glibc.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 4721177d83..6d07495edc 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -67,6 +67,11 @@ ifeq ($(BR2_ENABLE_DEBUG),y)
 GLIBC_EXTRA_CFLAGS += -g
 endif
 
+# glibc explicitly requires compile barriers between files
+ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
+GLIBC_EXTRA_CFLAGS += -fno-lto
+endif
+
 # The stubs.h header is not installed by install-headers, but is
 # needed for the gcc build. An empty stubs.h will work, as explained
 # in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
-- 
2.28.0

Re: [Buildroot] [PATCH v2 1/4] package/Makefile.in: use gcc wrappers for binutils tools

[Yann E. MORIN]

Norbert, All,

On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
> normal binutils tools. The difference is that with the
> wrappers, gcc plugins will be automatically picked up.
> 
> gcc 4.7 introduced these wrappers, to detect the prefix and
> keep gcc specifics out of Makefile.in, a new variable
> BR2_TOOLCHAIN_BUTILS_PREFIX will be used to carry the
> prefix on supported versions.
> 
> Note that binutils added some automatic loading with the
> 'bfd-plugins' directory (somewhere around 2.28), but
> the first implementation had issues, and generally depends on
> correctly setup symlinks (often broken, may point to some
> other gcc's library). The wrappers always work painless.
> 
> The original motivation (now ~2 years in use) was to add
> "-flto -ffat-lto-objects" to both BR2_TARGET_OPTIMIZATION and
> BR2_TARGET_LDFLAGS, and have target binaries lto optimized.
> 
> Not all packages will compile with this option, further work
> could white/blacklist packages (adding -fno-lto to the
> options).
> 
> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
> v1->v2:
> *   support older gcc versions missing those wrappers
> ---
>  package/Makefile.in | 6 +++---
>  toolchain/Config.in | 7 +++++++
>  2 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 51f5cbce4f..665edec539 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -198,15 +198,15 @@ TARGET_CROSS = $(HOST_DIR)/bin/$(TOOLCHAIN_EXTERNAL_PREFIX)-
>  endif
>  
>  # Define TARGET_xx variables for all common binutils/gcc
> -TARGET_AR       = $(TARGET_CROSS)ar
> +TARGET_AR       = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))ar
>  TARGET_AS       = $(TARGET_CROSS)as
>  TARGET_CC       = $(TARGET_CROSS)gcc
>  TARGET_CPP      = $(TARGET_CROSS)cpp
>  TARGET_CXX      = $(TARGET_CROSS)g++
>  TARGET_FC       = $(TARGET_CROSS)gfortran
>  TARGET_LD       = $(TARGET_CROSS)ld
> -TARGET_NM       = $(TARGET_CROSS)nm
> -TARGET_RANLIB   = $(TARGET_CROSS)ranlib
> +TARGET_NM       = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))nm
> +TARGET_RANLIB   = $(TARGET_CROSS)$(call qstrip,$(BR2_TOOLCHAIN_BUTILS_PREFIX))ranlib
>  TARGET_READELF  = $(TARGET_CROSS)readelf
>  TARGET_OBJCOPY  = $(TARGET_CROSS)objcopy
>  TARGET_OBJDUMP  = $(TARGET_CROSS)objdump
> diff --git a/toolchain/Config.in b/toolchain/Config.in
> index db2ab0f059..7a2becc09a 100644
> --- a/toolchain/Config.in
> +++ b/toolchain/Config.in
> @@ -732,4 +732,11 @@ config BR2_TOOLCHAIN_HAS_LIBQUADMATH
>  	bool
>  	default y if BR2_i386 || BR2_x86_64
>  
> +# gcc ships with wrappers that will automatically pass arguments
> +# to the binutils tools.
> +# So far, those are paths to necessary linker plugins
> +config BR2_TOOLCHAIN_BUTILS_PREFIX
> +	string
> +	default "gcc-" if BR2_TOOLCHAIN_GCC_AT_LEAST_4_7

There was no need for a Kconfig option, so I moved that to the Makefile
side.

Applied to master, thanks.

Regards,
Yann E. MORIN.

>  endmenu
> -- 
> 2.28.0

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 3/4] package/busybox: explicitly state binutil paths

[Yann E. MORIN]

Norbet, All,

On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> Pass paths to ar, nm, ranlib binaries. This allows linker-plugins
> to be used.

I have expanded/reworded the commit log to explain why this is needed
specifically for busybox.

Applied to master, thanks.

Regards,
Yann E. MORIN.

> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
>  package/busybox/busybox.mk | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
> index 01f7331221..221b8adf59 100644
> --- a/package/busybox/busybox.mk
> +++ b/package/busybox/busybox.mk
> @@ -94,6 +94,9 @@ BUSYBOX_MAKE_ENV += \
>  endif
>  
>  BUSYBOX_MAKE_OPTS = \
> +	AR="$(TARGET_AR)" \
> +	NM="$(TARGET_NM)" \
> +	RANLIB="$(TARGET_RANLIB)" \
>  	CC="$(TARGET_CC)" \
>  	ARCH=$(KERNEL_ARCH) \
>  	PREFIX="$(TARGET_DIR)" \
> -- 
> 2.28.0

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/4] package/gcc: use binutils wrappers for target libs

[Yann E. MORIN]

Nrobert, All,

On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
> normal binutils tools. The difference is that with the
> wrappers, gcc plugins will be automatically picked up,
> which might be necessary with some flags.
> 
> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
> v1->v2:
> *   Only set those wrappers if gc 4.7 or newer
> ---
>  package/gcc/gcc.mk | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
> index 114c2887a0..a24ffe3ee2 100644
> --- a/package/gcc/gcc.mk
> +++ b/package/gcc/gcc.mk
> @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error
>  endif
>  
>  # Propagate options used for target software building to GCC target libs
> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
> +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib

So we were a bit skeptical about this one.

First, we believe that only the final gcc should be using those. Indeed,
we do not need much of the initial gcc, except libgcc.a, which anyway
gets overwitten by the final gcc, so we do not care that the libgcc.a is
built with LTO (and thus the gcc wrappers).

Second, gcc initial is only used to build glibc, and as your patch 4
states, glibc must not be built with LTO, we we also do not care that
the initial gcc uses its wrappers to build libgcc.

Third, shouldn't we be using the tupple-prefixed wrappers? Or does it
just happens that gcc will internally add its build directory to PATH so
that it can find the un-prefixed just-built ones?

For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH,
that are just simple wrapper scripts that trace their being called, and
then call to the actual gcca-r et al.:

    #!/bin/sh
    me="${0}"
    my_name="${0##*/}"
    found=false
    for i in $(which -a "${my_name}"); do
        if [ "${i}" -ef "${me}" ]; then
            found=true
        elif ${found}; then
            me="${i}"
        fi
    done
    if ! ${found}; then
        exit 42
    fi
    printf "YEM: wrapped %s\n" "${me}"
    exec "${me}" "${@}"

Turns out that the 'YEM: wrapped' string occurs nowhere in the build
directory (so it does not end up in config.log or whatever), and it
does not even appear on the stdout/stderr either.

So I am wondering how we can exercise these... Any hint?

Regards,
Yann E. MORIN.

> +endif
>  HOST_GCC_COMMON_CONF_ENV += CFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CFLAGS)"
>  HOST_GCC_COMMON_CONF_ENV += CXXFLAGS_FOR_TARGET="$(GCC_COMMON_TARGET_CXXFLAGS)"
>  
> -- 
> 2.28.0

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/4] package/gcc: use binutils wrappers for target libs

[Norbert Lange]

Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN
<yann.morin.1998@free.fr>:
>
> Nrobert, All,
>
> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> > This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
> > normal binutils tools. The difference is that with the
> > wrappers, gcc plugins will be automatically picked up,
> > which might be necessary with some flags.
> >
> > Signed-off-by: Norbert Lange <nolange79@gmail.com>
> > ---
> > v1->v2:
> > *   Only set those wrappers if gc 4.7 or newer
> > ---
> >  package/gcc/gcc.mk | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
> > index 114c2887a0..a24ffe3ee2 100644
> > --- a/package/gcc/gcc.mk
> > +++ b/package/gcc/gcc.mk
> > @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error
> >  endif
> >
> >  # Propagate options used for target software building to GCC target libs
> > +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
> > +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib
>
> So we were a bit skeptical about this one.
>
> First, we believe that only the final gcc should be using those. Indeed,
> we do not need much of the initial gcc, except libgcc.a, which anyway
> gets overwitten by the final gcc, so we do not care that the libgcc.a is
> built with LTO (and thus the gcc wrappers).
>
> Second, gcc initial is only used to build glibc, and as your patch 4
> states, glibc must not be built with LTO, we we also do not care that
> the initial gcc uses its wrappers to build libgcc.
>
> Third, shouldn't we be using the tupple-prefixed wrappers? Or does it
> just happens that gcc will internally add its build directory to PATH so
> that it can find the un-prefixed just-built ones?
>
> For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH,
> that are just simple wrapper scripts that trace their being called, and
> then call to the actual gcca-r et al.:
>
>     #!/bin/sh
>     me="${0}"
>     my_name="${0##*/}"
>     found=false
>     for i in $(which -a "${my_name}"); do
>         if [ "${i}" -ef "${me}" ]; then
>             found=true
>         elif ${found}; then
>             me="${i}"
>         fi
>     done
>     if ! ${found}; then
>         exit 42
>     fi
>     printf "YEM: wrapped %s\n" "${me}"
>     exec "${me}" "${@}"
>
> Turns out that the 'YEM: wrapped' string occurs nowhere in the build
> directory (so it does not end up in config.log or whatever), and it
> does not even appear on the stdout/stderr either.
>
> So I am wondering how we can exercise these... Any hint?

Can't tell anymore, I went to crosstool-ng for the compiler some time ago,
so unlike my other patches I dont use them anymore.

From my memory, it affected some of the additional libs gcc builds,
libstdc++, some sanitizers maybe.
there was not absolute paths or tuples as the various stages should
use the wrappers from the previous ones. I believe the final compiler
ignores PATH and uses the stage2-compiler with abs path to build the
included target libraries.

I am just sure it was very deliberately done *this* way.

Norbert
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 4/4] package/glibc: force -fno-lto

[Yann E. MORIN]

Norbert, All,

On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> glibc requires compilation barriers between files, and will
> fail with LTO enabled.
> So force LTO off by appending -fno-lto to the flags, but only
> if the GCC version is recent enough that an LTO build would
> be possible.
> 
> Signed-off-by: Norbert Lange <nolange79@gmail.com>

Applied to master, thanks.

Are the other C libraries (uClibc, musl) also affected by such an issue?

Regards,
Yann E. MORIN.

> ---
>  package/glibc/glibc.mk | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
> index 4721177d83..6d07495edc 100644
> --- a/package/glibc/glibc.mk
> +++ b/package/glibc/glibc.mk
> @@ -67,6 +67,11 @@ ifeq ($(BR2_ENABLE_DEBUG),y)
>  GLIBC_EXTRA_CFLAGS += -g
>  endif
>  
> +# glibc explicitly requires compile barriers between files
> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
> +GLIBC_EXTRA_CFLAGS += -fno-lto
> +endif
> +
>  # The stubs.h header is not installed by install-headers, but is
>  # needed for the gcc build. An empty stubs.h will work, as explained
>  # in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
> -- 
> 2.28.0

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 4/4] package/glibc: force -fno-lto

[Norbert Lange]

[-- Attachment #1.1: Type: text/plain, Size: 890 bytes --]

Yann E. MORIN <yann.morin.1998@free.fr> schrieb am So., 9. Jan. 2022, 23:31:

> Norbert, All,
>
> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
> > glibc requires compilation barriers between files, and will
> > fail with LTO enabled.
> > So force LTO off by appending -fno-lto to the flags, but only
> > if the GCC version is recent enough that an LTO build would
> > be possible.
> >
> > Signed-off-by: Norbert Lange <nolange79@gmail.com>
>
> Applied to master, thanks.
>
> Are the other C libraries (uClibc, musl) also affected by such an issue?
>

Likely, and unless they are tested with LTO I would not trust them to not
have weird runtime issues.

Ideally there would be a whitelist for packages doing fine with LTO, for
now I can only vouch for the couple I am using LTOed since years.

Since a few distros now use LTO by default, hopefully things will get
better.

Norbert

[-- Attachment #1.2: Type: text/html, Size: 1543 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 4/4] package/glibc: force -fno-lto

[Arnout Vandecappelle]

On 10/01/2022 00:09, Norbert Lange wrote:
> 
> 
> Yann E. MORIN <yann.morin.1998@free.fr <mailto:yann.morin.1998@free.fr>> schrieb 
> am So., 9. Jan. 2022, 23:31:
> 
>     Norbert, All,
> 
>     On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
>      > glibc requires compilation barriers between files, and will
>      > fail with LTO enabled.
>      > So force LTO off by appending -fno-lto to the flags, but only
>      > if the GCC version is recent enough that an LTO build would
>      > be possible.
>      >
>      > Signed-off-by: Norbert Lange <nolange79@gmail.com
>     <mailto:nolange79@gmail.com>>
> 
>     Applied to master, thanks.
> 
>     Are the other C libraries (uClibc, musl) also affected by such an issue?
> 
> 
> Likely, and unless they are tested with LTO I would not trust them to not have 
> weird runtime issues.
> 
> Ideally there would be a whitelist for packages doing fine with LTO, for now I 
> can only vouch for the couple I am using LTOed since years.

  Up to now, the idea was that packages that support LTO would check for it 
themselves in their configure step and enable it if available.

  It would be a good idea though to add a global option to enable LTO, similar 
to SSP etc. The first step for that would be to add a hidden symbol that 
indicates if the toolchain has LTO, and that gets selected by the toolchains 
that have LTO. Then we have to look at each external toolchain to verify if LTO 
is available, and add a checker function to validate that it is correct. 
Finally, we can randomly turn on the LTO option in the genrandconfig script, and 
fix the fallout on abo.

  Regards,
  Arnout

> 
> Since a few distros now use LTO by default, hopefully things will get better.
> 
> Norbert
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/4] package/gcc: use binutils wrappers for target libs

[Arnout Vandecappelle]

On 09/01/2022 22:08, Norbert Lange wrote:
> Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN
> <yann.morin.1998@free.fr>:
>>
>> Nrobert, All,
>>
>> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
>>> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
>>> normal binutils tools. The difference is that with the
>>> wrappers, gcc plugins will be automatically picked up,
>>> which might be necessary with some flags.
>>>
>>> Signed-off-by: Norbert Lange <nolange79@gmail.com>
>>> ---
>>> v1->v2:
>>> *   Only set those wrappers if gc 4.7 or newer
>>> ---
>>>   package/gcc/gcc.mk | 3 +++
>>>   1 file changed, 3 insertions(+)
>>>
>>> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
>>> index 114c2887a0..a24ffe3ee2 100644
>>> --- a/package/gcc/gcc.mk
>>> +++ b/package/gcc/gcc.mk
>>> @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error
>>>   endif
>>>
>>>   # Propagate options used for target software building to GCC target libs
>>> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)
>>> +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm RANLIB_FOR_TARGET=gcc-ranlib
>>
>> So we were a bit skeptical about this one.
>>
>> First, we believe that only the final gcc should be using those. Indeed,
>> we do not need much of the initial gcc, except libgcc.a, which anyway
>> gets overwitten by the final gcc, so we do not care that the libgcc.a is
>> built with LTO (and thus the gcc wrappers). >>
>> Second, gcc initial is only used to build glibc, and as your patch 4
>> states, glibc must not be built with LTO, we we also do not care that
>> the initial gcc uses its wrappers to build libgcc.

  Although I may have been the one to make this obvservation to begin with, I'm 
starting to come back on that. Even though indeed it is probably not very useful 
to apply the wrappers to gcc-initial, it doesn't hurt either. And by putting it 
in _COMMON_CONF_ENV, we make sure that (almost) all of the options are together 
in gcc.mk. Note that --enable-lto is also in COMMON_OPTS.

>>
>> Third, shouldn't we be using the tupple-prefixed wrappers? Or does it
>> just happens that gcc will internally add its build directory to PATH so
>> that it can find the un-prefixed just-built ones?
>>
>> For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH,
>> that are just simple wrapper scripts that trace their being called, and
>> then call to the actual gcca-r et al.:
>>
>>      #!/bin/sh
>>      me="${0}"
>>      my_name="${0##*/}"
>>      found=false
>>      for i in $(which -a "${my_name}"); do
>>          if [ "${i}" -ef "${me}" ]; then
>>              found=true
>>          elif ${found}; then
>>              me="${i}"
>>          fi
>>      done
>>      if ! ${found}; then
>>          exit 42
>>      fi
>>      printf "YEM: wrapped %s\n" "${me}"
>>      exec "${me}" "${@}"
>>
>> Turns out that the 'YEM: wrapped' string occurs nowhere in the build
>> directory (so it does not end up in config.log or whatever), and it
>> does not even appear on the stdout/stderr either.
>>
>> So I am wondering how we can exercise these... Any hint?
> 
> Can't tell anymore, I went to crosstool-ng for the compiler some time ago,
> so unlike my other patches I dont use them anymore. >
>  From my memory, it affected some of the additional libs gcc builds,
> libstdc++, some sanitizers maybe.
> there was not absolute paths or tuples as the various stages should
> use the wrappers from the previous ones. I believe the final compiler
> ignores PATH and uses the stage2-compiler with abs path to build the
> included target libraries.

  That it doesn't look in PATH indeed sounds feasible.

  However, what is very surprising is that GCC doesn't automatically use the 
plugins when --enable-lto is passed. Before blindly adding some random options, 
we want to really understand why it is needed, in case maybe things are already 
broken in some other way, and to be able in the future to make changes and 
understand if they're OK or not.

  That said, I have checked and indeed, without these options it seems to use 
plain ar instead of gcc-ar.

  So I looked a bit deeper, and found this really old bug about enabling lto for 
libgcc [1]. It refers to a bunch of other issues found while doing this.

  For me, however, the conclusion is that the patch should be applied, but with 
better explanation. That way, you can set BR2_TARGET_OPTIMIZATION to -flto and 
get an LTO'ed libgcc/libstdc++/etc. If LTO is not used, it doesn't harm to use 
the wrapper.

  There's just one little concern though: I think the wrappers aren't even 
generated unless LTO is enabled. But that point is moot - since GCC N (where N > 
4.9 and < 9), LTO is enabled by default, and we don't pass --disable-lto if LTO 
is not enabled. In other words, nowadays the BR2_GCC_ENABLE_LTO symbol is useless.


  Regards,
  Arnout


[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59893



> 
> I am just sure it was very deliberately done *this* way.
> 
> Norbert
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/4] package/gcc: use binutils wrappers for target libs

[Arnout Vandecappelle]

On 10/01/2022 11:43, Arnout Vandecappelle wrote:
>
>
> On 09/01/2022 22:08, Norbert Lange wrote:
>> Am So., 9. Jan. 2022 um 16:16 Uhr schrieb Yann E. MORIN
>> <yann.morin.1998@free.fr>:
>>>
>>> Nrobert, All,
>>>
>>> On 2020-10-18 00:17 +0200, Norbert Lange spake thusly:
>>>> This will use gcc-ar, gcc-nm and gcc-ranlib instead of the
>>>> normal binutils tools. The difference is that with the
>>>> wrappers, gcc plugins will be automatically picked up,
>>>> which might be necessary with some flags.
>>>>
>>>> Signed-off-by: Norbert Lange <nolange79@gmail.com>
>>>> ---
>>>> v1->v2:
>>>> *   Only set those wrappers if gc 4.7 or newer
>>>> ---
>>>>   package/gcc/gcc.mk | 3 +++
>>>>   1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
>>>> index 114c2887a0..a24ffe3ee2 100644
>>>> --- a/package/gcc/gcc.mk
>>>> +++ b/package/gcc/gcc.mk
>>>> @@ -99,6 +99,9 @@ GCC_COMMON_TARGET_CFLAGS += -Wno-error
>>>>   endif
>>>>
>>>>   # Propagate options used for target software building to GCC target libs
>>>> +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_4_7),y)

  This is actually not needed - the lowest GCC version we have for the internal 
toolchain is 8.


>>>> +HOST_GCC_COMMON_CONF_ENV += AR_FOR_TARGET=gcc-ar NM_FOR_TARGET=gcc-nm 
>>>> RANLIB_FOR_TARGET=gcc-ranlib
>>>
>>> So we were a bit skeptical about this one.
>>>
>>> First, we believe that only the final gcc should be using those. Indeed,
>>> we do not need much of the initial gcc, except libgcc.a, which anyway
>>> gets overwitten by the final gcc, so we do not care that the libgcc.a is
>>> built with LTO (and thus the gcc wrappers). >>
>>> Second, gcc initial is only used to build glibc, and as your patch 4
>>> states, glibc must not be built with LTO, we we also do not care that
>>> the initial gcc uses its wrappers to build libgcc.
>
>  Although I may have been the one to make this obvservation to begin with, I'm 
> starting to come back on that. Even though indeed it is probably not very 
> useful to apply the wrappers to gcc-initial, it doesn't hurt either. And by 
> putting it in _COMMON_CONF_ENV, we make sure that (almost) all of the options 
> are together in gcc.mk. Note that --enable-lto is also in COMMON_OPTS.
>
>>>
>>> Third, shouldn't we be using the tupple-prefixed wrappers? Or does it
>>> just happens that gcc will internally add its build directory to PATH so
>>> that it can find the un-prefixed just-built ones?
>>>
>>> For that last point, I added gcc-ar, gcc-nm, and gcc-ranlib in my PATH,
>>> that are just simple wrapper scripts that trace their being called, and
>>> then call to the actual gcca-r et al.:
>>>
>>>      #!/bin/sh
>>>      me="${0}"
>>>      my_name="${0##*/}"
>>>      found=false
>>>      for i in $(which -a "${my_name}"); do
>>>          if [ "${i}" -ef "${me}" ]; then
>>>              found=true
>>>          elif ${found}; then
>>>              me="${i}"
>>>          fi
>>>      done
>>>      if ! ${found}; then
>>>          exit 42
>>>      fi
>>>      printf "YEM: wrapped %s\n" "${me}"
>>>      exec "${me}" "${@}"
>>>
>>> Turns out that the 'YEM: wrapped' string occurs nowhere in the build
>>> directory (so it does not end up in config.log or whatever), and it
>>> does not even appear on the stdout/stderr either.
>>>
>>> So I am wondering how we can exercise these... Any hint?
>>
>> Can't tell anymore, I went to crosstool-ng for the compiler some time ago,
>> so unlike my other patches I dont use them anymore. >
>>  From my memory, it affected some of the additional libs gcc builds,
>> libstdc++, some sanitizers maybe.
>> there was not absolute paths or tuples as the various stages should
>> use the wrappers from the previous ones. I believe the final compiler
>> ignores PATH and uses the stage2-compiler with abs path to build the
>> included target libraries.
>
>  That it doesn't look in PATH indeed sounds feasible.
>
>  However, what is very surprising is that GCC doesn't automatically use the 
> plugins when --enable-lto is passed. Before blindly adding some random 
> options, we want to really understand why it is needed, in case maybe things 
> are already broken in some other way, and to be able in the future to make 
> changes and understand if they're OK or not.
>
>  That said, I have checked and indeed, without these options it seems to use 
> plain ar instead of gcc-ar.
>
>  So I looked a bit deeper, and found this really old bug about enabling lto 
> for libgcc [1]. It refers to a bunch of other issues found while doing this.
>
>  For me, however, the conclusion is that the patch should be applied, but with 
> better explanation. That way, you can set BR2_TARGET_OPTIMIZATION to -flto and 
> get an LTO'ed libgcc/libstdc++/etc. If LTO is not used, it doesn't harm to use 
> the wrapper.

  I summarized this discussion and applied to master, thanks.


  Regards,
  Arnout


>
>  There's just one little concern though: I think the wrappers aren't even 
> generated unless LTO is enabled. But that point is moot - since GCC N (where N 
> > 4.9 and < 9), LTO is enabled by default, and we don't pass --disable-lto if 
> LTO is not enabled. In other words, nowadays the BR2_GCC_ENABLE_LTO symbol is 
> useless.
>
>
>  Regards,
>  Arnout
>
>
> [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59893
>
>
>
>>
>> I am just sure it was very deliberately done *this* way.
>>
>> Norbert
>> _______________________________________________
>> buildroot mailing list
>> buildroot@buildroot.org
>> https://lists.buildroot.org/mailman/listinfo/buildroot
>>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot