* [PATCH] gcc: enable zstd compression of LTO bytecode
@ 2022-02-03 17:26 Enrico Scholz
2022-02-03 19:23 ` [OE-core] " Khem Raj
2022-02-03 21:59 ` Richard Purdie
0 siblings, 2 replies; 9+ messages in thread
From: Enrico Scholz @ 2022-02-03 17:26 UTC (permalink / raw)
To: openembedded-core; +Cc: Enrico Scholz
gcc-10+ supports zstd compression of LTO bytecode. Install the
corresponding package to enable this feature in a deterministic way.
NOTE: previously built LTO object files (without this compression)
must be regenerated; gcc will fail else with
| lto1: internal compiler error: original not compressed with zstd
in this case.
NOTE: it seems to be possible that zstd support is enabled non
deterministically (perhaps by host system pollution).
I had the problem that the SDK gcc was built with zstd while the
cross gcc was built without it. Libraries (built by cross gcc and
uncompressed hence) could not be used with the SDK gcc.
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
index f07a5be507b3..62db035760aa 100644
--- a/meta/recipes-devtools/gcc/gcc-11.2.inc
+++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
@@ -11,7 +11,7 @@ BINV = "11.2.0"
FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
-NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
+NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
--
2.34.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 17:26 [PATCH] gcc: enable zstd compression of LTO bytecode Enrico Scholz
@ 2022-02-03 19:23 ` Khem Raj
2022-02-03 21:40 ` Enrico Scholz
2022-02-03 21:59 ` Richard Purdie
1 sibling, 1 reply; 9+ messages in thread
From: Khem Raj @ 2022-02-03 19:23 UTC (permalink / raw)
To: Enrico Scholz; +Cc: Patches and discussions about the oe-core layer
On Thu, Feb 3, 2022 at 9:26 AM Enrico Scholz via
lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
>
> gcc-10+ supports zstd compression of LTO bytecode. Install the
> corresponding package to enable this feature in a deterministic way.
>
> NOTE: previously built LTO object files (without this compression)
> must be regenerated; gcc will fail else with
>
> | lto1: internal compiler error: original not compressed with zstd
>
> in this case.
>
> NOTE: it seems to be possible that zstd support is enabled non
> deterministically (perhaps by host system pollution).
>
> I had the problem that the SDK gcc was built with zstd while the
> cross gcc was built without it. Libraries (built by cross gcc and
> uncompressed hence) could not be used with the SDK gcc.
>
this means regenerating entire shared state isnt it ?
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
> meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
> index f07a5be507b3..62db035760aa 100644
> --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> @@ -11,7 +11,7 @@ BINV = "11.2.0"
> FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
>
> DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>
> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161292): https://lists.openembedded.org/g/openembedded-core/message/161292
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 19:23 ` [OE-core] " Khem Raj
@ 2022-02-03 21:40 ` Enrico Scholz
2022-02-03 21:48 ` Khem Raj
0 siblings, 1 reply; 9+ messages in thread
From: Enrico Scholz @ 2022-02-03 21:40 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
Khem Raj <raj.khem@gmail.com> writes:
>> gcc-10+ supports zstd compression of LTO bytecode. Install the
>> corresponding package to enable this feature in a deterministic way.
>>
>> NOTE: previously built LTO object files (without this compression)
>> must be regenerated; gcc will fail else with
>>
>> | lto1: internal compiler error: original not compressed with zstd
>>
>> in this case.
>>
>> NOTE: it seems to be possible that zstd support is enabled non
>> deterministically (perhaps by host system pollution).
>>
>> I had the problem that the SDK gcc was built with zstd while the
>> cross gcc was built without it. Libraries (built by cross gcc and
>> uncompressed hence) could not be used with the SDK gcc.
>
> this means regenerating entire shared state isnt it ?
yes (at least when lto.inc is used (which is *not* by default)). When a
bad sstate really matters for 'master', perhaps there can be wait for a
change in glibc or so before applying this patch.
Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make
sense? When this flag is missing, '--without-zstd' must be added to
EXTRA_OECONF. Also some early [vardeps] must be set (where?) to avoid
the sstate problem.
Globally enabling 'zstd' appears easier because drawbacks (costs for
compression + decompression) should be negligible for zstd.
Enrico
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 21:40 ` Enrico Scholz
@ 2022-02-03 21:48 ` Khem Raj
0 siblings, 0 replies; 9+ messages in thread
From: Khem Raj @ 2022-02-03 21:48 UTC (permalink / raw)
To: Enrico Scholz; +Cc: Patches and discussions about the oe-core layer
On Thu, Feb 3, 2022 at 1:40 PM Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
>
> Khem Raj <raj.khem@gmail.com> writes:
>
> >> gcc-10+ supports zstd compression of LTO bytecode. Install the
> >> corresponding package to enable this feature in a deterministic way.
> >>
> >> NOTE: previously built LTO object files (without this compression)
> >> must be regenerated; gcc will fail else with
> >>
> >> | lto1: internal compiler error: original not compressed with zstd
> >>
> >> in this case.
> >>
> >> NOTE: it seems to be possible that zstd support is enabled non
> >> deterministically (perhaps by host system pollution).
> >>
> >> I had the problem that the SDK gcc was built with zstd while the
> >> cross gcc was built without it. Libraries (built by cross gcc and
> >> uncompressed hence) could not be used with the SDK gcc.
> >
> > this means regenerating entire shared state isnt it ?
>
> yes (at least when lto.inc is used (which is *not* by default)). When a
> bad sstate really matters for 'master', perhaps there can be wait for a
> change in glibc or so before applying this patch.
>
> Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make
> sense? When this flag is missing, '--without-zstd' must be added to
> EXTRA_OECONF. Also some early [vardeps] must be set (where?) to avoid
> the sstate problem.
>
> Globally enabling 'zstd' appears easier because drawbacks (costs for
> compression + decompression) should be negligible for zstd.
I think switching to zstd is fine.
>
>
>
> Enrico
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 17:26 [PATCH] gcc: enable zstd compression of LTO bytecode Enrico Scholz
2022-02-03 19:23 ` [OE-core] " Khem Raj
@ 2022-02-03 21:59 ` Richard Purdie
2022-02-03 22:11 ` Khem Raj
2022-02-04 19:00 ` Enrico Scholz
1 sibling, 2 replies; 9+ messages in thread
From: Richard Purdie @ 2022-02-03 21:59 UTC (permalink / raw)
To: enrico.scholz, openembedded-core
On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org
wrote:
> gcc-10+ supports zstd compression of LTO bytecode. Install the
> corresponding package to enable this feature in a deterministic way.
>
> NOTE: previously built LTO object files (without this compression)
> must be regenerated; gcc will fail else with
>
> | lto1: internal compiler error: original not compressed with zstd
>
> in this case.
>
> NOTE: it seems to be possible that zstd support is enabled non
> deterministically (perhaps by host system pollution).
>
> I had the problem that the SDK gcc was built with zstd while the
> cross gcc was built without it. Libraries (built by cross gcc and
> uncompressed hence) could not be used with the SDK gcc.
>
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
> meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
> index f07a5be507b3..62db035760aa 100644
> --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> @@ -11,7 +11,7 @@ BINV = "11.2.0"
> FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
>
> DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>
> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>
Should we be passing some flag to gcc explicitly to configure this
deterministicly?
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 21:59 ` Richard Purdie
@ 2022-02-03 22:11 ` Khem Raj
2022-02-03 22:21 ` Richard Purdie
2022-02-04 19:00 ` Enrico Scholz
1 sibling, 1 reply; 9+ messages in thread
From: Khem Raj @ 2022-02-03 22:11 UTC (permalink / raw)
To: Richard Purdie; +Cc: enrico.scholz, openembedded-core
[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]
On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via
> lists.openembedded.org
> wrote:
> > gcc-10+ supports zstd compression of LTO bytecode. Install the
> > corresponding package to enable this feature in a deterministic way.
> >
> > NOTE: previously built LTO object files (without this compression)
> > must be regenerated; gcc will fail else with
> >
> > | lto1: internal compiler error: original not compressed with zstd
> >
> > in this case.
> >
> > NOTE: it seems to be possible that zstd support is enabled non
> > deterministically (perhaps by host system pollution).
> >
> > I had the problem that the SDK gcc was built with zstd while the
> > cross gcc was built without it. Libraries (built by cross gcc and
> > uncompressed hence) could not be used with the SDK gcc.
> >
> > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> > ---
> > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc
> b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > index f07a5be507b3..62db035760aa 100644
> > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > @@ -11,7 +11,7 @@ BINV = "11.2.0"
> > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
> >
> > DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native
> flex-native"
> > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native
> flex-native zstd-native"
> >
> > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> >
>
> Should we be passing some flag to gcc explicitly to configure this
> deterministicly?
Perhaps use —with-zstd configure switch maybe via package config but it’s
better to enable it always
>
>
> Cheers,
>
> Richard
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161323):
> https://lists.openembedded.org/g/openembedded-core/message/161323
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
[-- Attachment #2: Type: text/html, Size: 3795 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 22:11 ` Khem Raj
@ 2022-02-03 22:21 ` Richard Purdie
0 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2022-02-03 22:21 UTC (permalink / raw)
To: Khem Raj; +Cc: enrico.scholz, openembedded-core
On Thu, 2022-02-03 at 14:11 -0800, Khem Raj wrote:
>
>
> On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org
> > wrote:
> > > gcc-10+ supports zstd compression of LTO bytecode. Install the
> > > corresponding package to enable this feature in a deterministic way.
> > >
> > > NOTE: previously built LTO object files (without this compression)
> > > must be regenerated; gcc will fail else with
> > >
> > > | lto1: internal compiler error: original not compressed with zstd
> > >
> > > in this case.
> > >
> > > NOTE: it seems to be possible that zstd support is enabled non
> > > deterministically (perhaps by host system pollution).
> > >
> > > I had the problem that the SDK gcc was built with zstd while the
> > > cross gcc was built without it. Libraries (built by cross gcc and
> > > uncompressed hence) could not be used with the SDK gcc.
> > >
> > > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> > > ---
> > > meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-
> > devtools/gcc/gcc-11.2.inc
> > > index f07a5be507b3..62db035760aa 100644
> > > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> > > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > > @@ -11,7 +11,7 @@ BINV = "11.2.0"
> > > FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
> > >
> > > DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> > > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-
> > > native"
> > > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-
> > > native
> > zstd-native"
> > >
> > > LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> > >
> >
> > Should we be passing some flag to gcc explicitly to configure this
> > deterministicly?
> >
>
>
> Perhaps use —with-zstd configure switch maybe via package config but it’s
> better to enable it always
>
Agreed, I'm fine with enabling it, I'd just like to be explicit about what we're
doing.
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-03 21:59 ` Richard Purdie
2022-02-03 22:11 ` Khem Raj
@ 2022-02-04 19:00 ` Enrico Scholz
2022-02-04 21:00 ` Khem Raj
1 sibling, 1 reply; 9+ messages in thread
From: Enrico Scholz @ 2022-02-04 19:00 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-core
Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> | lto1: internal compiler error: original not compressed with zstd
>>
>> DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
>> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
>> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>>
>> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>>
>
> Should we be passing some flag to gcc explicitly to configure this
> deterministicly?
Both '--with-zstd' and '--without-zstd' make it deterministic (require
zstd.h and abort when non-existing resp. ignore it when existing).
Perhaps '--without-zstd' should be added in the hardknott and honister
branches to prevent pollution by host environment.
Enrico
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode
2022-02-04 19:00 ` Enrico Scholz
@ 2022-02-04 21:00 ` Khem Raj
0 siblings, 0 replies; 9+ messages in thread
From: Khem Raj @ 2022-02-04 21:00 UTC (permalink / raw)
To: Enrico Scholz
Cc: Richard Purdie, Patches and discussions about the oe-core layer
On Fri, Feb 4, 2022 at 11:00 AM Enrico Scholz via
lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
>
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>
> >> | lto1: internal compiler error: original not compressed with zstd
> >>
> >> DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> >> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> >> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
> >>
> >> LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> >>
> >
> > Should we be passing some flag to gcc explicitly to configure this
> > deterministicly?
>
> Both '--with-zstd' and '--without-zstd' make it deterministic (require
> zstd.h and abort when non-existing resp. ignore it when existing).
>
> Perhaps '--without-zstd' should be added in the hardknott and honister
> branches to prevent pollution by host environment.
>
yes makes sense.
>
>
> Enrico
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161379): https://lists.openembedded.org/g/openembedded-core/message/161379
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-02-04 21:01 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-03 17:26 [PATCH] gcc: enable zstd compression of LTO bytecode Enrico Scholz
2022-02-03 19:23 ` [OE-core] " Khem Raj
2022-02-03 21:40 ` Enrico Scholz
2022-02-03 21:48 ` Khem Raj
2022-02-03 21:59 ` Richard Purdie
2022-02-03 22:11 ` Khem Raj
2022-02-03 22:21 ` Richard Purdie
2022-02-04 19:00 ` Enrico Scholz
2022-02-04 21:00 ` Khem Raj
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.