* [RFC PATCH v2 0/2] Improve native/cross reproducibility
@ 2021-11-30 22:37 Jacob Kroon
2021-11-30 22:37 ` [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries Jacob Kroon
2021-11-30 22:37 ` [RFC PATCH v2 2/2] Improve native reproducibility in recipes Jacob Kroon
0 siblings, 2 replies; 9+ messages in thread
From: Jacob Kroon @ 2021-11-30 22:37 UTC (permalink / raw)
To: openembedded-core
This patch series is not intended for merge. I only send it out to
highlight where the problems are and to get some discussion going on
how/if we want to improve the sitation.
This is a patch series that tries to improve the reproducibility of the
native/cross binaries when building in different directories. This has
been tested on a Fedora 35 system which uses gcc 11.2.1 at the time of
writing.
The RUNTIME hack is questionable, maybe there is a better way to enforce
a fixed RUNTIME entry size during linking.
If in the end we do come up with a solution, then it should be tested on
the autobuilders, since otherwise this is going to degrade overtime. It
would then be important that the build paths are of significantly different
lengths. TMPDIR=/tmp/sysrootA/ and TMPDIR=/tmp/sysrootB/ will *not* uncover all
rpath problems.
Another thing that would be useful is if buildhistory could start
monitoring the depsig.do_populate_sysroot files, since that would highlight
changes in the actual file contents, which is currently not covered by
buildhistory.
The end result of this patch series is that I can build python3-native
in two different build paths, and the resuling sysroot-components/x86_64/
directories are identical, except for the 'fixmepath.cmd' files, which
are not included in the hash equiv. comparisons. Even so, there remains a lot of
native builds that are going to need to be fixed in similar ways as the
ones in this patch series.
Changes in v2:
* Fixed building icedtea7-native/openjdk-8-native by prepending a '/'
to the rpath padding
* Squashed all the recipe-fixes together in one commit for now
Jacob
Jacob Kroon (2):
bitbake.conf: Pad rpath and remove build ID in native binaries
Improve native reproducibility in recipes
meta/classes/chrpath.bbclass | 3 +
meta/conf/bitbake.conf | 5 +-
...sysroot-and-debug-prefix-map-from-co.patch | 78 -------------------
.../openssl/openssl/strip-buildinfo.patch | 13 ++++
.../openssl/openssl_3.0.0.bb | 10 +--
meta/recipes-core/ncurses/ncurses.inc | 4 +
.../util-linux/util-linux_2.37.2.bb | 2 +-
.../libtool/libtool-native_2.4.6.bb | 1 +
...ism.patch => perl-cross-determinism.patch} | 0
.../perl-cross/perlcross_1.3.6.bb | 4 +-
meta/recipes-devtools/perl/perl_5.34.0.bb | 5 ++
.../pkgconfig/pkgconfig_git.bb | 1 +
.../python/python3/determinism.patch | 15 ++++
.../recipes-devtools/python/python3_3.10.0.bb | 8 ++
14 files changed, 62 insertions(+), 87 deletions(-)
delete mode 100644 meta/recipes-connectivity/openssl/openssl/0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch
create mode 100644 meta/recipes-connectivity/openssl/openssl/strip-buildinfo.patch
rename meta/recipes-devtools/perl-cross/files/{determinism.patch => perl-cross-determinism.patch} (100%)
create mode 100644 meta/recipes-devtools/python/python3/determinism.patch
^ permalink raw reply [flat|nested] 9+ messages in thread* [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-11-30 22:37 [RFC PATCH v2 0/2] Improve native/cross reproducibility Jacob Kroon @ 2021-11-30 22:37 ` Jacob Kroon 2021-12-01 23:11 ` [OE-core] " Richard Purdie 2021-11-30 22:37 ` [RFC PATCH v2 2/2] Improve native reproducibility in recipes Jacob Kroon 1 sibling, 1 reply; 9+ messages in thread From: Jacob Kroon @ 2021-11-30 22:37 UTC (permalink / raw) To: openembedded-core Try to make sure that the RUNTIME dynamic entry size is the same for all binaries produced with the native compiler. This is necessary in order to produce identical binaries when using differently sized buildpaths. I've tried using only patchelf, and keeping the linker flags as they are, but I am unable to produce identical binaries. Has anyone else managed to do this with patchelf ? If not, maybe we can write a new tool that can handle it ? The build-id also needs to be removed since it is calculated based on the data present at link time. This includes STAGING_LIBDIR_NATIVE and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily preserved since some recipes will execute the binaries during do_install() (for example python3-native). Later on these are removed in chrpath.bbclass. This hack is the first step for producing identical native binaries when using different build paths. 'zstd-native' is a working example. Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> --- meta/classes/chrpath.bbclass | 3 +++ meta/conf/bitbake.conf | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/meta/classes/chrpath.bbclass b/meta/classes/chrpath.bbclass index 26b984c4db..56a68768e0 100644 --- a/meta/classes/chrpath.bbclass +++ b/meta/classes/chrpath.bbclass @@ -24,6 +24,9 @@ def process_file_linux(cmd, fpath, rootdir, baseprefix, tmpdir, d, break_hardlin new_rpaths = [] modified = False for rpath in rpaths: + if rpath.startswith('/rpath-padding-'): + modified = True + continue # If rpath is already dynamic copy it to new_rpath and continue if rpath.find("$ORIGIN") != -1: new_rpaths.append(rpath) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index fba99e8f0c..61124e10b2 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -580,6 +580,7 @@ BUILDSDK_CXXFLAGS = "${BUILDSDK_CFLAGS}" export CXXFLAGS = "${TARGET_CXXFLAGS}" TARGET_CXXFLAGS = "${TARGET_CFLAGS}" +RPATH_PADDING ?= "/rpath-padding-${@'x' * (512 - len(d.expand('${STAGING_LIBDIR_NATIVE}:${STAGING_BASE_LIBDIR_NATIVE}:/rpath-padding-')))}" export BUILD_LDFLAGS = "-L${STAGING_LIBDIR_NATIVE} \ -L${STAGING_BASE_LIBDIR_NATIVE} \ -Wl,--enable-new-dtags \ @@ -587,7 +588,9 @@ export BUILD_LDFLAGS = "-L${STAGING_LIBDIR_NATIVE} \ -Wl,-rpath-link,${STAGING_BASE_LIBDIR_NATIVE} \ -Wl,-rpath,${STAGING_LIBDIR_NATIVE} \ -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} \ - -Wl,-O1" + -Wl,-rpath,${RPATH_PADDING} \ + -Wl,-O1 \ + -Wl,--build-id=none" BUILDSDK_LDFLAGS = "-Wl,-O1" ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-11-30 22:37 ` [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries Jacob Kroon @ 2021-12-01 23:11 ` Richard Purdie 2021-12-02 10:19 ` Jacob Kroon 0 siblings, 1 reply; 9+ messages in thread From: Richard Purdie @ 2021-12-01 23:11 UTC (permalink / raw) To: Jacob Kroon, openembedded-core On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: > Try to make sure that the RUNTIME dynamic entry size is the same for all > binaries produced with the native compiler. This is necessary in order to > produce identical binaries when using differently sized buildpaths. I've > tried using only patchelf, and keeping the linker flags as they are, but > I am unable to produce identical binaries. Has anyone else managed to do > this with patchelf ? If not, maybe we can write a new tool that can handle it ? > > The build-id also needs to be removed since it is calculated based on > the data present at link time. This includes STAGING_LIBDIR_NATIVE > and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily > preserved since some recipes will execute the binaries during do_install() > (for example python3-native). Later on these are removed in chrpath.bbclass. > > This hack is the first step for producing identical native binaries when using > different build paths. 'zstd-native' is a working example. > > Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> > --- > meta/classes/chrpath.bbclass | 3 +++ > meta/conf/bitbake.conf | 5 ++++- > 2 files changed, 7 insertions(+), 1 deletion(-) I'm a little torn on this. Our other option would be to hardcoded a specific dummy path and then edit it later to the correct value. That may be neater than adding the padding. It will change the end binaries but hopefully only after they're installed so should give the same net end result more neatly? If we separate out the build-id patch we could hopefully get that piece merged as that shouldn't be controversial? Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-12-01 23:11 ` [OE-core] " Richard Purdie @ 2021-12-02 10:19 ` Jacob Kroon 2021-12-02 10:51 ` Richard Purdie 0 siblings, 1 reply; 9+ messages in thread From: Jacob Kroon @ 2021-12-02 10:19 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 12/2/21 00:11, Richard Purdie wrote: > On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: >> Try to make sure that the RUNTIME dynamic entry size is the same for all >> binaries produced with the native compiler. This is necessary in order to >> produce identical binaries when using differently sized buildpaths. I've >> tried using only patchelf, and keeping the linker flags as they are, but >> I am unable to produce identical binaries. Has anyone else managed to do >> this with patchelf ? If not, maybe we can write a new tool that can handle it ? >> >> The build-id also needs to be removed since it is calculated based on >> the data present at link time. This includes STAGING_LIBDIR_NATIVE >> and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily >> preserved since some recipes will execute the binaries during do_install() >> (for example python3-native). Later on these are removed in chrpath.bbclass. >> >> This hack is the first step for producing identical native binaries when using >> different build paths. 'zstd-native' is a working example. >> >> Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> >> --- >> meta/classes/chrpath.bbclass | 3 +++ >> meta/conf/bitbake.conf | 5 ++++- >> 2 files changed, 7 insertions(+), 1 deletion(-) > > I'm a little torn on this. Our other option would be to hardcoded a specific > dummy path and then edit it later to the correct value. That may be neater than > adding the padding. It will change the end binaries but hopefully only after > they're installed so should give the same net end result more neatly? > Hmm not sure I follow. This patch adds a new dummy rpath entry, "/rpath-padding-xxx...", then we remove it in chrpath. I don't know what other value we would like to put there. If I understand you correctly, we could perhaps pad one of the ones we already pass -Wl,-rpath,${STAGING_LIBDIR_NATIVE} -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} with spaces, like: -Wl,-rpath,${STAGING_LIBDIR_NATIVE} -Wl,-rpath,"${STAGING_BASE_LIBDIR_NATIVE}${RPATH_PADDING}" If that works that would be less intrusive I think. > If we separate out the build-id patch we could hopefully get that piece merged > as that shouldn't be controversial? > Yes, I can split it out into a separate patch. But now that I've looked at this for a while, I've asked myself what good does all this do ? The only optimization I can think of is that if we rebuild a native recipes, and the sysroot component turns out the same, then we don't need to create a new sstate cache entry. So we save disk space, but disk space is cheap. We still need to build it. What I would like is to have a common sstate dir for multiple build directories. So if I build libtool-native in one build path, then at my other build path it would just pick it up from sstate cache when I build there. In the end, is that something that would be possible ? Jacob ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-12-02 10:19 ` Jacob Kroon @ 2021-12-02 10:51 ` Richard Purdie 2021-12-02 11:03 ` Jacob Kroon 0 siblings, 1 reply; 9+ messages in thread From: Richard Purdie @ 2021-12-02 10:51 UTC (permalink / raw) To: Jacob Kroon, openembedded-core On Thu, 2021-12-02 at 11:19 +0100, Jacob Kroon wrote: > On 12/2/21 00:11, Richard Purdie wrote: > > On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: > > > Try to make sure that the RUNTIME dynamic entry size is the same for all > > > binaries produced with the native compiler. This is necessary in order to > > > produce identical binaries when using differently sized buildpaths. I've > > > tried using only patchelf, and keeping the linker flags as they are, but > > > I am unable to produce identical binaries. Has anyone else managed to do > > > this with patchelf ? If not, maybe we can write a new tool that can handle it ? > > > > > > The build-id also needs to be removed since it is calculated based on > > > the data present at link time. This includes STAGING_LIBDIR_NATIVE > > > and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily > > > preserved since some recipes will execute the binaries during do_install() > > > (for example python3-native). Later on these are removed in chrpath.bbclass. > > > > > > This hack is the first step for producing identical native binaries when using > > > different build paths. 'zstd-native' is a working example. > > > > > > Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> > > > --- > > > meta/classes/chrpath.bbclass | 3 +++ > > > meta/conf/bitbake.conf | 5 ++++- > > > 2 files changed, 7 insertions(+), 1 deletion(-) > > > > I'm a little torn on this. Our other option would be to hardcoded a specific > > dummy path and then edit it later to the correct value. That may be neater than > > adding the padding. It will change the end binaries but hopefully only after > > they're installed so should give the same net end result more neatly? > > > > Hmm not sure I follow. This patch adds a new dummy rpath entry, > "/rpath-padding-xxx...", then we remove it in chrpath. I don't know what > other value we would like to put there. If I understand you correctly, > we could perhaps pad one of the ones we already pass > > -Wl,-rpath,${STAGING_LIBDIR_NATIVE} > -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} > > with spaces, like: > > -Wl,-rpath,${STAGING_LIBDIR_NATIVE} > -Wl,-rpath,"${STAGING_BASE_LIBDIR_NATIVE}${RPATH_PADDING}" I'm wondering if: -Wl,-rpath,/not/exist/our-native-libdir-marker -Wl,-rpath,/not/exist/our-native-base-libdir-marker would work. > If that works that would be less intrusive I think. > > > If we separate out the build-id patch we could hopefully get that piece merged > > as that shouldn't be controversial? > > > > Yes, I can split it out into a separate patch. > > But now that I've looked at this for a while, I've asked myself what > good does all this do ? The only optimization I can think of is that if > we rebuild a native recipes, and the sysroot component turns out the > same, then we don't need to create a new sstate cache entry. So we save > disk space, but disk space is cheap. We still need to build it. What I > would like is to have a common sstate dir for multiple build > directories. So if I build libtool-native in one build path, then at my > other build path it would just pick it up from sstate cache when I build > there. In the end, is that something that would be possible ? We originally started here with gcc-cross so lets consider that and multiple build directories where a patch changes gcc-cross in a way that is irrelavent to the output. The "win" is that regardless of whether I build in location A or B, I get the same gcc-cross binary. Hash-equiv will then not rebuild the target binaries. Yes, I pay the price of a gcc-cross rebuild but hashequiv saves the targets rebuilding. Currently it would only happen if you always build gcc-cross in a specific build path. Like everything, it is a question of looking at the changes and deciding whether they are worth any maintenance burden/code complication or additional overhead they generate. I don't know the answer here yet but I do appreciate the research in helping get us data to make decisions on! Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-12-02 10:51 ` Richard Purdie @ 2021-12-02 11:03 ` Jacob Kroon 2021-12-02 11:09 ` Richard Purdie 0 siblings, 1 reply; 9+ messages in thread From: Jacob Kroon @ 2021-12-02 11:03 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 12/2/21 11:51, Richard Purdie wrote: > On Thu, 2021-12-02 at 11:19 +0100, Jacob Kroon wrote: >> On 12/2/21 00:11, Richard Purdie wrote: >>> On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: >>>> Try to make sure that the RUNTIME dynamic entry size is the same for all >>>> binaries produced with the native compiler. This is necessary in order to >>>> produce identical binaries when using differently sized buildpaths. I've >>>> tried using only patchelf, and keeping the linker flags as they are, but >>>> I am unable to produce identical binaries. Has anyone else managed to do >>>> this with patchelf ? If not, maybe we can write a new tool that can handle it ? >>>> >>>> The build-id also needs to be removed since it is calculated based on >>>> the data present at link time. This includes STAGING_LIBDIR_NATIVE >>>> and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily >>>> preserved since some recipes will execute the binaries during do_install() >>>> (for example python3-native). Later on these are removed in chrpath.bbclass. >>>> >>>> This hack is the first step for producing identical native binaries when using >>>> different build paths. 'zstd-native' is a working example. >>>> >>>> Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> >>>> --- >>>> meta/classes/chrpath.bbclass | 3 +++ >>>> meta/conf/bitbake.conf | 5 ++++- >>>> 2 files changed, 7 insertions(+), 1 deletion(-) >>> >>> I'm a little torn on this. Our other option would be to hardcoded a specific >>> dummy path and then edit it later to the correct value. That may be neater than >>> adding the padding. It will change the end binaries but hopefully only after >>> they're installed so should give the same net end result more neatly? >>> >> >> Hmm not sure I follow. This patch adds a new dummy rpath entry, >> "/rpath-padding-xxx...", then we remove it in chrpath. I don't know what >> other value we would like to put there. If I understand you correctly, >> we could perhaps pad one of the ones we already pass >> >> -Wl,-rpath,${STAGING_LIBDIR_NATIVE} >> -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} >> >> with spaces, like: >> >> -Wl,-rpath,${STAGING_LIBDIR_NATIVE} >> -Wl,-rpath,"${STAGING_BASE_LIBDIR_NATIVE}${RPATH_PADDING}" > > > I'm wondering if: > > -Wl,-rpath,/not/exist/our-native-libdir-marker > -Wl,-rpath,/not/exist/our-native-base-libdir-marker > > would work. > Right, I'll give it a try. >> If that works that would be less intrusive I think. >> >>> If we separate out the build-id patch we could hopefully get that piece merged >>> as that shouldn't be controversial? >>> >> >> Yes, I can split it out into a separate patch. >> >> But now that I've looked at this for a while, I've asked myself what >> good does all this do ? The only optimization I can think of is that if >> we rebuild a native recipes, and the sysroot component turns out the >> same, then we don't need to create a new sstate cache entry. So we save >> disk space, but disk space is cheap. We still need to build it. What I >> would like is to have a common sstate dir for multiple build >> directories. So if I build libtool-native in one build path, then at my >> other build path it would just pick it up from sstate cache when I build >> there. In the end, is that something that would be possible ? > > We originally started here with gcc-cross so lets consider that and multiple > build directories where a patch changes gcc-cross in a way that is irrelavent to > the output. > > The "win" is that regardless of whether I build in location A or B, I get the > same gcc-cross binary. Hash-equiv will then not rebuild the target binaries. > Yes, I pay the price of a gcc-cross rebuild but hashequiv saves the targets > rebuilding. > > Currently it would only happen if you always build gcc-cross in a specific build > path. > I know the build path will change if I upgrade to a new version of gcc, but then the output is most definitely gonna change as well. > Like everything, it is a question of looking at the changes and deciding whether > they are worth any maintenance burden/code complication or additional overhead > they generate. I don't know the answer here yet but I do appreciate the research > in helping get us data to make decisions on! > I was thinking if it was possible to add a "build-path-does-not-matter" .bbclass that would make the signatures independent of build path and then scan the output to make sure it didn't contain any references to the build path. Then those recipes who didn't depend on build path could inherit from that class, and then maybe their sstate could be reused from multiple build directories ? Not sure reliable it would be though.. Jacob ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-12-02 11:03 ` Jacob Kroon @ 2021-12-02 11:09 ` Richard Purdie 2021-12-02 14:49 ` Jacob Kroon 0 siblings, 1 reply; 9+ messages in thread From: Richard Purdie @ 2021-12-02 11:09 UTC (permalink / raw) To: Jacob Kroon, openembedded-core On Thu, 2021-12-02 at 12:03 +0100, Jacob Kroon wrote: > On 12/2/21 11:51, Richard Purdie wrote: > > On Thu, 2021-12-02 at 11:19 +0100, Jacob Kroon wrote: > > > On 12/2/21 00:11, Richard Purdie wrote: > > > > On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: > > > > > Try to make sure that the RUNTIME dynamic entry size is the same for all > > > > > binaries produced with the native compiler. This is necessary in order to > > > > > produce identical binaries when using differently sized buildpaths. I've > > > > > tried using only patchelf, and keeping the linker flags as they are, but > > > > > I am unable to produce identical binaries. Has anyone else managed to do > > > > > this with patchelf ? If not, maybe we can write a new tool that can handle it ? > > > > > > > > > > The build-id also needs to be removed since it is calculated based on > > > > > the data present at link time. This includes STAGING_LIBDIR_NATIVE > > > > > and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily > > > > > preserved since some recipes will execute the binaries during do_install() > > > > > (for example python3-native). Later on these are removed in chrpath.bbclass. > > > > > > > > > > This hack is the first step for producing identical native binaries when using > > > > > different build paths. 'zstd-native' is a working example. > > > > > > > > > > Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> > > > > > --- > > > > > meta/classes/chrpath.bbclass | 3 +++ > > > > > meta/conf/bitbake.conf | 5 ++++- > > > > > 2 files changed, 7 insertions(+), 1 deletion(-) > > > > > > > > I'm a little torn on this. Our other option would be to hardcoded a specific > > > > dummy path and then edit it later to the correct value. That may be neater than > > > > adding the padding. It will change the end binaries but hopefully only after > > > > they're installed so should give the same net end result more neatly? > > > > > > > > > > Hmm not sure I follow. This patch adds a new dummy rpath entry, > > > "/rpath-padding-xxx...", then we remove it in chrpath. I don't know what > > > other value we would like to put there. If I understand you correctly, > > > we could perhaps pad one of the ones we already pass > > > > > > -Wl,-rpath,${STAGING_LIBDIR_NATIVE} > > > -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} > > > > > > with spaces, like: > > > > > > -Wl,-rpath,${STAGING_LIBDIR_NATIVE} > > > -Wl,-rpath,"${STAGING_BASE_LIBDIR_NATIVE}${RPATH_PADDING}" > > > > > > I'm wondering if: > > > > -Wl,-rpath,/not/exist/our-native-libdir-marker > > -Wl,-rpath,/not/exist/our-native-base-libdir-marker > > > > would work. > > > > Right, I'll give it a try. > > > > If that works that would be less intrusive I think. > > > > > > > If we separate out the build-id patch we could hopefully get that piece merged > > > > as that shouldn't be controversial? > > > > > > > > > > Yes, I can split it out into a separate patch. > > > > > > But now that I've looked at this for a while, I've asked myself what > > > good does all this do ? The only optimization I can think of is that if > > > we rebuild a native recipes, and the sysroot component turns out the > > > same, then we don't need to create a new sstate cache entry. So we save > > > disk space, but disk space is cheap. We still need to build it. What I > > > would like is to have a common sstate dir for multiple build > > > directories. So if I build libtool-native in one build path, then at my > > > other build path it would just pick it up from sstate cache when I build > > > there. In the end, is that something that would be possible ? > > > > We originally started here with gcc-cross so lets consider that and multiple > > build directories where a patch changes gcc-cross in a way that is irrelavent to > > the output. > > > > The "win" is that regardless of whether I build in location A or B, I get the > > same gcc-cross binary. Hash-equiv will then not rebuild the target binaries. > > Yes, I pay the price of a gcc-cross rebuild but hashequiv saves the targets > > rebuilding. > > > > Currently it would only happen if you always build gcc-cross in a specific build > > path. > > > > I know the build path will change if I upgrade to a new version of gcc, > but then the output is most definitely gonna change as well. > > > Like everything, it is a question of looking at the changes and deciding whether > > they are worth any maintenance burden/code complication or additional overhead > > they generate. I don't know the answer here yet but I do appreciate the research > > in helping get us data to make decisions on! > > > > I was thinking if it was possible to add a "build-path-does-not-matter" > .bbclass that would make the signatures independent of build path and > then scan the output to make sure it didn't contain any references to > the build path. Then those recipes who didn't depend on build path could > inherit from that class, and then maybe their sstate could be reused > from multiple build directories ? Not sure reliable it would be though.. Another crazy thought is our sstate really is already path independent, regardless of the binary content. You could therefore make the hash function replace the path with a fixed string. The downside is that doesn't work well on binaries due to offsets, alignment and so on. As I read the above I was reminded that insane.bbclass does sanity check the output for build paths and does have a configurable control mechanism. It doesn't do that for the populate_sysroot output though since it is for do_package. Lots to think about here but you're right that adding some kind of scanner to mark up recipes over time would help us preserve this. Cheers, Richard ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [OE-core] [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries 2021-12-02 11:09 ` Richard Purdie @ 2021-12-02 14:49 ` Jacob Kroon 0 siblings, 0 replies; 9+ messages in thread From: Jacob Kroon @ 2021-12-02 14:49 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 12/2/21 12:09, Richard Purdie wrote: > On Thu, 2021-12-02 at 12:03 +0100, Jacob Kroon wrote: >> On 12/2/21 11:51, Richard Purdie wrote: >>> On Thu, 2021-12-02 at 11:19 +0100, Jacob Kroon wrote: >>>> On 12/2/21 00:11, Richard Purdie wrote: >>>>> On Tue, 2021-11-30 at 23:37 +0100, Jacob Kroon wrote: >>>>>> Try to make sure that the RUNTIME dynamic entry size is the same for all >>>>>> binaries produced with the native compiler. This is necessary in order to >>>>>> produce identical binaries when using differently sized buildpaths. I've >>>>>> tried using only patchelf, and keeping the linker flags as they are, but >>>>>> I am unable to produce identical binaries. Has anyone else managed to do >>>>>> this with patchelf ? If not, maybe we can write a new tool that can handle it ? >>>>>> >>>>>> The build-id also needs to be removed since it is calculated based on >>>>>> the data present at link time. This includes STAGING_LIBDIR_NATIVE >>>>>> and STAGING_BASE_LIBDIR_NATIVE. Both will differ and they need to be temporarily >>>>>> preserved since some recipes will execute the binaries during do_install() >>>>>> (for example python3-native). Later on these are removed in chrpath.bbclass. >>>>>> >>>>>> This hack is the first step for producing identical native binaries when using >>>>>> different build paths. 'zstd-native' is a working example. >>>>>> >>>>>> Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> >>>>>> --- >>>>>> meta/classes/chrpath.bbclass | 3 +++ >>>>>> meta/conf/bitbake.conf | 5 ++++- >>>>>> 2 files changed, 7 insertions(+), 1 deletion(-) >>>>> >>>>> I'm a little torn on this. Our other option would be to hardcoded a specific >>>>> dummy path and then edit it later to the correct value. That may be neater than >>>>> adding the padding. It will change the end binaries but hopefully only after >>>>> they're installed so should give the same net end result more neatly? >>>>> >>>> >>>> Hmm not sure I follow. This patch adds a new dummy rpath entry, >>>> "/rpath-padding-xxx...", then we remove it in chrpath. I don't know what >>>> other value we would like to put there. If I understand you correctly, >>>> we could perhaps pad one of the ones we already pass >>>> >>>> -Wl,-rpath,${STAGING_LIBDIR_NATIVE} >>>> -Wl,-rpath,${STAGING_BASE_LIBDIR_NATIVE} >>>> >>>> with spaces, like: >>>> >>>> -Wl,-rpath,${STAGING_LIBDIR_NATIVE} >>>> -Wl,-rpath,"${STAGING_BASE_LIBDIR_NATIVE}${RPATH_PADDING}" >>> >>> >>> I'm wondering if: >>> >>> -Wl,-rpath,/not/exist/our-native-libdir-marker >>> -Wl,-rpath,/not/exist/our-native-base-libdir-marker >>> >>> would work. >>> >> >> Right, I'll give it a try. >> Unfortunatley this breaks building python3-native. Although it compiles, during the build the python build scripts tries to import the created modules, and if this fails (which it does) it renames the modules: > *** WARNING: renaming "_curses" since importing it failed: libncurses.so.5: cannot open shared object file: No such file or directory > *** WARNING: renaming "_curses_panel" since importing it failed: libpanel.so.5: cannot open shared object file: No such file or directory > *** WARNING: renaming "_ssl" since importing it failed: libssl.so.3: cannot open shared object file: No such file or directory > *** WARNING: renaming "_hashlib" since importing it failed: libssl.so.3: cannot open shared object file: No such file or directory > *** WARNING: renaming "nis" since importing it failed: libnsl.so.3: cannot open shared object file: No such file or directory > *** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directory I suppose it tries to import using the built python which has those phony rpaths, and can't find the per-recipe-sysroot lbncurses.so.5/libpanel.so.5/etc and fails. The new modules will be called: > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu_failed.so > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/nis.cpython-310-x86_64-linux-gnu_failed.so > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/_hashlib.cpython-310-x86_64-linux-gnu_failed.so > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu_failed.so > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/_curses_panel.cpython-310-x86_64-linux-gnu_failed.so > sysroots-components/x86_64/python3-native/usr/lib/python3.10/lib-dynload/_curses.cpython-310-x86_64-linux-gnu_failed.so which means any subsequent recipe that uses python3-native will fail to import any of those modules. I suspect it might not just be python that wants to run the produced binaries during the build itself. >>>> If that works that would be less intrusive I think. >>>> >>>>> If we separate out the build-id patch we could hopefully get that piece merged >>>>> as that shouldn't be controversial? >>>>> >>>> >>>> Yes, I can split it out into a separate patch. >>>> >>>> But now that I've looked at this for a while, I've asked myself what >>>> good does all this do ? The only optimization I can think of is that if >>>> we rebuild a native recipes, and the sysroot component turns out the >>>> same, then we don't need to create a new sstate cache entry. So we save >>>> disk space, but disk space is cheap. We still need to build it. What I >>>> would like is to have a common sstate dir for multiple build >>>> directories. So if I build libtool-native in one build path, then at my >>>> other build path it would just pick it up from sstate cache when I build >>>> there. In the end, is that something that would be possible ? >>> >>> We originally started here with gcc-cross so lets consider that and multiple >>> build directories where a patch changes gcc-cross in a way that is irrelavent to >>> the output. >>> >>> The "win" is that regardless of whether I build in location A or B, I get the >>> same gcc-cross binary. Hash-equiv will then not rebuild the target binaries. >>> Yes, I pay the price of a gcc-cross rebuild but hashequiv saves the targets >>> rebuilding. >>> >>> Currently it would only happen if you always build gcc-cross in a specific build >>> path. >>> >> >> I know the build path will change if I upgrade to a new version of gcc, >> but then the output is most definitely gonna change as well. >> >>> Like everything, it is a question of looking at the changes and deciding whether >>> they are worth any maintenance burden/code complication or additional overhead >>> they generate. I don't know the answer here yet but I do appreciate the research >>> in helping get us data to make decisions on! >>> >> >> I was thinking if it was possible to add a "build-path-does-not-matter" >> .bbclass that would make the signatures independent of build path and >> then scan the output to make sure it didn't contain any references to >> the build path. Then those recipes who didn't depend on build path could >> inherit from that class, and then maybe their sstate could be reused >> from multiple build directories ? Not sure reliable it would be though.. > > Another crazy thought is our sstate really is already path independent, > regardless of the binary content. You could therefore make the hash function > replace the path with a fixed string. The downside is that doesn't work well on > binaries due to offsets, alignment and so on. > > As I read the above I was reminded that insane.bbclass does sanity check the > output for build paths and does have a configurable control mechanism. It > doesn't do that for the populate_sysroot output though since it is for > do_package. > > Lots to think about here but you're right that adding some kind of scanner to > mark up recipes over time would help us preserve this. Jacob ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v2 2/2] Improve native reproducibility in recipes 2021-11-30 22:37 [RFC PATCH v2 0/2] Improve native/cross reproducibility Jacob Kroon 2021-11-30 22:37 ` [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries Jacob Kroon @ 2021-11-30 22:37 ` Jacob Kroon 1 sibling, 0 replies; 9+ messages in thread From: Jacob Kroon @ 2021-11-30 22:37 UTC (permalink / raw) To: openembedded-core Avoid encoding build-specific paths in the resulting binaries. Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> --- ...sysroot-and-debug-prefix-map-from-co.patch | 78 ------------------- .../openssl/openssl/strip-buildinfo.patch | 13 ++++ .../openssl/openssl_3.0.0.bb | 10 +-- meta/recipes-core/ncurses/ncurses.inc | 4 + .../util-linux/util-linux_2.37.2.bb | 2 +- .../libtool/libtool-native_2.4.6.bb | 1 + ...ism.patch => perl-cross-determinism.patch} | 0 .../perl-cross/perlcross_1.3.6.bb | 4 +- meta/recipes-devtools/perl/perl_5.34.0.bb | 5 ++ .../pkgconfig/pkgconfig_git.bb | 1 + .../python/python3/determinism.patch | 15 ++++ .../recipes-devtools/python/python3_3.10.0.bb | 8 ++ 12 files changed, 55 insertions(+), 86 deletions(-) delete mode 100644 meta/recipes-connectivity/openssl/openssl/0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch create mode 100644 meta/recipes-connectivity/openssl/openssl/strip-buildinfo.patch rename meta/recipes-devtools/perl-cross/files/{determinism.patch => perl-cross-determinism.patch} (100%) create mode 100644 meta/recipes-devtools/python/python3/determinism.patch diff --git a/meta/recipes-connectivity/openssl/openssl/0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch b/meta/recipes-connectivity/openssl/openssl/0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch deleted file mode 100644 index 60890c666d..0000000000 --- a/meta/recipes-connectivity/openssl/openssl/0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch +++ /dev/null @@ -1,78 +0,0 @@ -From 5985253f2c9025d7c127443a3a9938946f80c2a1 Mon Sep 17 00:00:00 2001 -From: =?UTF-8?q?Martin=20Hundeb=C3=B8ll?= <martin@geanix.com> -Date: Tue, 6 Nov 2018 14:50:47 +0100 -Subject: [PATCH] buildinfo: strip sysroot and debug-prefix-map from compiler - info -MIME-Version: 1.0 -Content-Type: text/plain; charset=UTF-8 -Content-Transfer-Encoding: 8bit - -The openssl build system generates buildinf.h containing the full -compiler command line used to compile objects. This breaks -reproducibility, as the compile command is baked into libcrypto, where -it is used when running `openssl version -f`. - -Add stripped build variables for the compiler and cflags lines, and use -those when generating buildinfo.h. - -This is based on a similar patch for older openssl versions: -https://patchwork.openembedded.org/patch/147229/ - -Upstream-Status: Inappropriate [OE specific] -Signed-off-by: Martin Hundebøll <martin@geanix.com> - -Update to fix buildpaths qa issue for '-fmacro-prefix-map'. - -Signed-off-by: Kai Kang <kai.kang@windriver.com> - -Update to fix buildpaths qa issue for '-ffile-prefix-map'. - -Signed-off-by: Khem Raj <raj.khem@gmail.com> - ---- - Configurations/unix-Makefile.tmpl | 12 +++++++++++- - crypto/build.info | 2 +- - 2 files changed, 12 insertions(+), 2 deletions(-) - -diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl -index f88a70f..528cdef 100644 ---- a/Configurations/unix-Makefile.tmpl -+++ b/Configurations/unix-Makefile.tmpl -@@ -471,13 +471,23 @@ BIN_LDFLAGS={- join(' ', $target{bin_lflags} || (), - '$(CNF_LDFLAGS)', '$(LDFLAGS)') -} - BIN_EX_LIBS=$(CNF_EX_LIBS) $(EX_LIBS) - --# CPPFLAGS_Q is used for one thing only: to build up buildinf.h -+# *_Q variables are used for one thing only: to build up buildinf.h - CPPFLAGS_Q={- $cppflags1 =~ s|([\\"])|\\$1|g; - $cppflags2 =~ s|([\\"])|\\$1|g; - $lib_cppflags =~ s|([\\"])|\\$1|g; - join(' ', $lib_cppflags || (), $cppflags2 || (), - $cppflags1 || ()) -} - -+CFLAGS_Q={- for (@{$config{CFLAGS}}) { -+ s|-fdebug-prefix-map=[^ ]+|-fdebug-prefix-map=|g; -+ s|-fmacro-prefix-map=[^ ]+|-fmacro-prefix-map=|g; -+ s|-ffile-prefix-map=[^ ]+|-ffile-prefix-map=|g; -+ } -+ join(' ', @{$config{CFLAGS}}) -} -+ -+CC_Q={- $config{CC} =~ s|--sysroot=[^ ]+|--sysroot=recipe-sysroot|g; -+ join(' ', $config{CC}) -} -+ - PERLASM_SCHEME= {- $target{perlasm_scheme} -} - - # For x86 assembler: Set PROCESSOR to 386 if you want to support -diff --git a/crypto/build.info b/crypto/build.info -index efca6cc..eda433e 100644 ---- a/crypto/build.info -+++ b/crypto/build.info -@@ -109,7 +109,7 @@ DEFINE[../libcrypto]=$UPLINKDEF - - DEPEND[info.o]=buildinf.h - DEPEND[cversion.o]=buildinf.h --GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(LIB_CFLAGS) $(CPPFLAGS_Q)" "$(PLATFORM)" -+GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC_Q) $(CFLAGS_Q) $(CPPFLAGS_Q)" "$(PLATFORM)" - - GENERATE[uplink-x86.s]=../ms/uplink-x86.pl - GENERATE[uplink-x86_64.s]=../ms/uplink-x86_64.pl diff --git a/meta/recipes-connectivity/openssl/openssl/strip-buildinfo.patch b/meta/recipes-connectivity/openssl/openssl/strip-buildinfo.patch new file mode 100644 index 0000000000..0a4a60273d --- /dev/null +++ b/meta/recipes-connectivity/openssl/openssl/strip-buildinfo.patch @@ -0,0 +1,13 @@ +Index: openssl-3.0.0/crypto/build.info +=================================================================== +--- openssl-3.0.0.orig/crypto/build.info ++++ openssl-3.0.0/crypto/build.info +@@ -109,7 +109,7 @@ DEFINE[../libcrypto]=$UPLINKDEF + + DEPEND[info.o]=buildinf.h + DEPEND[cversion.o]=buildinf.h +-GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(LIB_CFLAGS) $(CPPFLAGS_Q)" "$(PLATFORM)" ++GENERATE[buildinf.h]=../util/mkbuildinf.pl "empty" + + GENERATE[uplink-x86.s]=../ms/uplink-x86.pl + GENERATE[uplink-x86_64.s]=../ms/uplink-x86_64.pl diff --git a/meta/recipes-connectivity/openssl/openssl_3.0.0.bb b/meta/recipes-connectivity/openssl/openssl_3.0.0.bb index 8852a51ca8..ccfd16b79b 100644 --- a/meta/recipes-connectivity/openssl/openssl_3.0.0.bb +++ b/meta/recipes-connectivity/openssl/openssl_3.0.0.bb @@ -9,10 +9,10 @@ LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=c75985e733726beaba57bc5253e96d04" SRC_URI = "http://www.openssl.org/source/openssl-${PV}.tar.gz \ file://run-ptest \ - file://0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch \ file://afalg.patch \ file://0001-Configure-do-not-tweak-mips-cflags.patch \ file://armv8-32bit.patch \ + file://strip-buildinfo.patch \ " SRC_URI:append:class-nativesdk = " \ @@ -46,10 +46,6 @@ EXTRA_OECONF:append:libc-musl:powerpc64 = " no-asm" EXTRA_OECONF:class-native = "--with-rand-seed=os,devrandom" EXTRA_OECONF:class-nativesdk = "--with-rand-seed=os,devrandom" -# Relying on hardcoded built-in paths causes openssl-native to not be relocateable from sstate. -CFLAGS:append:class-native = " -DOPENSSLDIR=/not/builtin -DENGINESDIR=/not/builtin" -CFLAGS:append:class-nativesdk = " -DOPENSSLDIR=/not/builtin -DENGINESDIR=/not/builtin" - # This allows disabling deprecated or undesirable crypto algorithms. # The default is to trust upstream choices. DEPRECATED_CRYPTO_FLAGS ?= "" @@ -131,6 +127,10 @@ do_configure () { perl ${B}/configdata.pm --dump } +do_compile:class-native () { + oe_runmake OPENSSLDIR=/non/existent ENGINESDIR=/non/existent MODULESDIR=/non/existent +} + do_install () { oe_runmake DESTDIR="${D}" MANDIR="${mandir}" MANSUFFIX=ssl install diff --git a/meta/recipes-core/ncurses/ncurses.inc b/meta/recipes-core/ncurses/ncurses.inc index a0ecd8a80b..3c15498dd4 100644 --- a/meta/recipes-core/ncurses/ncurses.inc +++ b/meta/recipes-core/ncurses/ncurses.inc @@ -91,10 +91,14 @@ ncurses_configure() { --with-manpage-format=normal \ --without-manpage-renames \ --disable-stripping \ + ${EXTRA_CLASS_FLAGS} \ "$@" || return 1 cd .. } +EXTRA_CLASS_FLAGS = "" +EXTRA_CLASS_FLAGS:class-native = "--datadir=/non/existent --with-terminfo-dirs=/non/existent" + # Override the function from the autotools class; ncurses requires a # patched autoconf213 to generate the configure script. This autoconf # is not available so that the shipped script will be used. diff --git a/meta/recipes-core/util-linux/util-linux_2.37.2.bb b/meta/recipes-core/util-linux/util-linux_2.37.2.bb index d609c30067..09f83eb4dd 100644 --- a/meta/recipes-core/util-linux/util-linux_2.37.2.bb +++ b/meta/recipes-core/util-linux/util-linux_2.37.2.bb @@ -83,7 +83,7 @@ EXTRA_OECONF = "\ " EXTRA_OECONF:append:class-target = " --enable-setpriv" -EXTRA_OECONF:append:class-native = " --without-cap-ng --disable-setpriv" +EXTRA_OECONF:append:class-native = " --without-cap-ng --disable-setpriv --runstatedir=/non/existent SYSCONFSTATICDIR=/non/existent" EXTRA_OECONF:append:class-nativesdk = " --without-cap-ng --disable-setpriv" EXTRA_OECONF:append = " --disable-hwclock-gplv3" diff --git a/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb b/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb index 3b20ce3e69..ea19b86d4a 100644 --- a/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb +++ b/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb @@ -7,6 +7,7 @@ SRC_URI += "file://prefix.patch" inherit native EXTRA_OECONF = " --with-libtool-sysroot=${STAGING_DIR_NATIVE}" +CACHED_CONFIGUREVARS += "lt_cv_sys_dlsearch_path=/non/existent" do_configure:prepend () { # Remove any existing libtool m4 since old stale versions would break diff --git a/meta/recipes-devtools/perl-cross/files/determinism.patch b/meta/recipes-devtools/perl-cross/files/perl-cross-determinism.patch similarity index 100% rename from meta/recipes-devtools/perl-cross/files/determinism.patch rename to meta/recipes-devtools/perl-cross/files/perl-cross-determinism.patch diff --git a/meta/recipes-devtools/perl-cross/perlcross_1.3.6.bb b/meta/recipes-devtools/perl-cross/perlcross_1.3.6.bb index 2759ef8a53..dab7f4558f 100644 --- a/meta/recipes-devtools/perl-cross/perlcross_1.3.6.bb +++ b/meta/recipes-devtools/perl-cross/perlcross_1.3.6.bb @@ -15,7 +15,7 @@ SRC_URI = "https://github.com/arsv/perl-cross/releases/download/${PV}/perl-cross file://0001-configure_tool.sh-do-not-quote-the-argument-to-comma.patch \ file://0001-perl-cross-add-LDFLAGS-when-linking-libperl.patch \ file://0001-configure_path.sh-do-not-hardcode-prefix-lib-as-libr.patch \ - file://determinism.patch \ + file://perl-cross-determinism.patch \ file://0001-cnf-configure_func_sel.sh-disable-thread_safe_nl_lan.patch \ file://0001-Makefile-check-the-file-if-patched-or-not.patch \ " @@ -33,7 +33,7 @@ do_compile () { do_install:class-native() { mkdir -p ${D}/${datadir}/perl-cross/ - cp -rf ${S}/* ${D}/${datadir}/perl-cross/ + cp -rfL ${S}/* ${D}/${datadir}/perl-cross/ } BBCLASSEXTEND = "native" diff --git a/meta/recipes-devtools/perl/perl_5.34.0.bb b/meta/recipes-devtools/perl/perl_5.34.0.bb index 16d45ccff3..0b74d5f072 100644 --- a/meta/recipes-devtools/perl/perl_5.34.0.bb +++ b/meta/recipes-devtools/perl/perl_5.34.0.bb @@ -97,6 +97,9 @@ do_configure:class-native() { -Dvendorprefix=${prefix} \ -Ui_xlocale \ ${PACKAGECONFIG_CONFARGS} + + # See the comment above + sed -i -e "s,${STAGING_DIR_NATIVE},/non/existent,g" config.h } do_configure:append() { @@ -395,3 +398,5 @@ SSTATE_HASHEQUIV_FILEMAP = " \ populate_sysroot:*/lib*/perl5/config.sh:${TMPDIR} \ populate_sysroot:*/lib*/perl5/config.sh:${COREBASE} \ " + +EXTRA_STAGING_FIXMES:append:class-native = " RPATH_PADDING" diff --git a/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb b/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb index c220bafd90..a7b2cae624 100644 --- a/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb +++ b/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb @@ -28,6 +28,7 @@ inherit autotools # so just continue that behaviour. # EXTRA_OECONF += "--disable-indirect-deps" +EXTRA_OECONF:append:class-native = " --libdir=/non/existent --with-pc-path=/non/existent" PACKAGECONFIG ??= "glib" PACKAGECONFIG:class-native = "" diff --git a/meta/recipes-devtools/python/python3/determinism.patch b/meta/recipes-devtools/python/python3/determinism.patch new file mode 100644 index 0000000000..eca7755d4e --- /dev/null +++ b/meta/recipes-devtools/python/python3/determinism.patch @@ -0,0 +1,15 @@ +Index: Python-3.10.0/Makefile.pre.in +=================================================================== +--- Python-3.10.0.orig/Makefile.pre.in ++++ Python-3.10.0/Makefile.pre.in +@@ -791,8 +791,8 @@ Modules/getbuildinfo.o: $(PARSER_OBJS) \ + + Modules/getpath.o: $(srcdir)/Modules/getpath.c Makefile + $(CC) -c $(PY_CORE_CFLAGS) -DPYTHONPATH='"$(PYTHONPATH)"' \ +- -DPREFIX='"$(prefix)"' \ +- -DEXEC_PREFIX='"$(exec_prefix)"' \ ++ -DPREFIX='"/non/existent"' \ ++ -DEXEC_PREFIX='"/non/existent"' \ + -DVERSION='"$(VERSION)"' \ + -DVPATH='"$(VPATH)"' \ + -o $@ $(srcdir)/Modules/getpath.c diff --git a/meta/recipes-devtools/python/python3_3.10.0.bb b/meta/recipes-devtools/python/python3_3.10.0.bb index e3300b6495..ba2e9f7dcb 100644 --- a/meta/recipes-devtools/python/python3_3.10.0.bb +++ b/meta/recipes-devtools/python/python3_3.10.0.bb @@ -40,6 +40,7 @@ SRC_URI:append:class-native = " \ file://0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch \ file://12-distutils-prefix-is-inside-staging-area.patch \ file://0001-Don-t-search-system-for-headers-libraries.patch \ + file://determinism.patch \ " SRC_URI[sha256sum] = "5a99f8e7a6a11a7b98b4e75e0d1303d3832cada5534068f69c7b6222a7b1b002" @@ -79,6 +80,8 @@ DEPENDS:append:class-nativesdk = " python3-native" # force to use the mutex+cond implementation (https://bugs.python.org/issue41710) CFLAGS += "-DHAVE_BROKEN_POSIX_SEMAPHORES" +CFLAGS:append:class-native = " -ffile-prefix-map=${WORKDIR}=/usr/src" + EXTRA_OECONF = " --without-ensurepip --enable-shared --with-platlibdir=${baselib}" EXTRA_OECONF:append:class-native = " --bindir=${bindir}/${PN}" @@ -94,6 +97,7 @@ CACHED_CONFIGUREVARS = " \ ac_cv_file__dev_ptc=no \ ac_cv_working_tzset=yes \ " +CACHED_CONFIGUREVARS:append:class-native = " ac_cv_prog_cc_g=no" # PGO currently causes builds to not be reproducible so disable by default, see YOCTO #13407 PACKAGECONFIG:class-target ??= "readline gdbm ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)}" @@ -180,6 +184,8 @@ do_install:append() { # More info: http://benno.id.au/blog/2013/01/15/python-determinism rm ${D}${libdir}/python${PYTHON_MAJMIN}/test/__pycache__/test_range.cpython* rm ${D}${libdir}/python${PYTHON_MAJMIN}/test/__pycache__/test_xml_etree.cpython* + + find ${D}${libdir}/python${PYTHON_MAJMIN} -name __pycache__ | xargs -n1 rm -r } do_install:append:class-nativesdk () { @@ -398,3 +404,5 @@ SYSROOT_PREPROCESS_FUNCS += " py3_sysroot_cleanup" py3_sysroot_cleanup () { rm -rf ${SYSROOT_DESTDIR}${libdir}/python${PYTHON_MAJMIN}/test } + +EXTRA_STAGING_FIXMES:append:class-native = " RPATH_PADDING WORKDIR" ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-12-02 14:49 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-30 22:37 [RFC PATCH v2 0/2] Improve native/cross reproducibility Jacob Kroon 2021-11-30 22:37 ` [RFC PATCH v2 1/2] bitbake.conf: Pad rpath and remove build ID in native binaries Jacob Kroon 2021-12-01 23:11 ` [OE-core] " Richard Purdie 2021-12-02 10:19 ` Jacob Kroon 2021-12-02 10:51 ` Richard Purdie 2021-12-02 11:03 ` Jacob Kroon 2021-12-02 11:09 ` Richard Purdie 2021-12-02 14:49 ` Jacob Kroon 2021-11-30 22:37 ` [RFC PATCH v2 2/2] Improve native reproducibility in recipes Jacob Kroon
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.