[PATCH 0/4] improve reproducibility of several packages

[PATCH 0/4] improve reproducibility of several packages

[Juro Bystricky]

These are independent patches with the same theme: leaking build host
information in various distributed files.


Juro Bystricky (4):
  gnutls: improve reproducibility
  grub-efi_2.02.bb: improve reproducibility
  x11perf: improve reproducibility
  sudo: improve reproducibility

 meta/recipes-bsp/grub/grub-efi_2.02.bb          | 10 ++++++++++
 meta/recipes-extended/sudo/sudo.inc             |  6 ++++++
 meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb |  4 ++++
 meta/recipes-support/gnutls/gnutls.inc          |  5 +++++
 4 files changed, 25 insertions(+)

-- 
2.7.4

[PATCH 1/4] gnutls: improve reproducibility

[Juro Bystricky]

Sanitize generated config.h containing build host reference such as:

 #define POSIX_SHELL "<builddir>/tmp/hosttools/bash"

The path for POSIX_SHELL is not only wrong for the cross-compiled target,
it also prevents reproducible build.

Prior the compilation we edit the value to:

 #define POSIX_SHELL "bash"

With this change we can build reproducible libgnutls-bin, libgnutls-dev
and libgnutls-dbg packages.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-support/gnutls/gnutls.inc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-support/gnutls/gnutls.inc b/meta/recipes-support/gnutls/gnutls.inc
index 29b5dd6..5b45b27 100644
--- a/meta/recipes-support/gnutls/gnutls.inc
+++ b/meta/recipes-support/gnutls/gnutls.inc
@@ -53,6 +53,11 @@ do_configure_prepend() {
 	done
 }
 
+do_configure_append_class-target () {
+	# Remove build host references from config.h
+	sed -i -e 's:${HOSTTOOLS_DIR}/::g' ${B}/config.h
+}
+
 PACKAGES =+ "${PN}-openssl ${PN}-xx"
 
 FILES_${PN}-dev += "${bindir}/gnutls-cli-debug"
-- 
2.7.4

[PATCH 2/4] grub-efi_2.02.bb: improve reproducibility

[Juro Bystricky]

Remove several build host references from modinfo.sh files.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-bsp/grub/grub-efi_2.02.bb | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/recipes-bsp/grub/grub-efi_2.02.bb b/meta/recipes-bsp/grub/grub-efi_2.02.bb
index b949beb..44e32a8 100644
--- a/meta/recipes-bsp/grub/grub-efi_2.02.bb
+++ b/meta/recipes-bsp/grub/grub-efi_2.02.bb
@@ -41,6 +41,16 @@ do_install_class-native() {
 	install -m 755 grub-mkimage ${D}${bindir}
 }
 
+do_install_append_class-target() {
+    # Remove build host references...
+    find "${D}" -name modinfo.sh -type f -exec \
+        sed -i \
+        -e 's,--sysroot=${STAGING_DIR_TARGET},,g' \
+        -e 's|${DEBUG_PREFIX_MAP}||g' \
+        -e 's:${RECIPE_SYSROOT_NATIVE}::g' \
+        {} +
+}
+
 GRUB_BUILDIN ?= "boot linux ext2 fat serial part_msdos part_gpt normal efi_gop iso9660 search"
 
 do_deploy() {
-- 
2.7.4

[PATCH 3/4] x11perf: improve reproducibility

[Juro Bystricky]

Remove build host refeences.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb b/meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb
index 4e93558..a06aa26 100644
--- a/meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb
+++ b/meta/recipes-graphics/xorg-app/x11perf_1.6.0.bb
@@ -13,6 +13,10 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=428ca4d67a41fcd4fc3283dce9bbda7e \
 
 PE = "1"
 
+do_install_append_class-target () {
+    sed -i -e 's:${HOSTTOOLS_DIR}/::g' ${D}${bindir}/x11perfcomp
+}
+
 FILES_${PN} += "${libdir}/X11/x11perfcomp/*"
 
 SRC_URI[md5sum] = "f0b24e4d8beb622a419e8431e1c03cd7"
-- 
2.7.4

[PATCH 4/4] sudo: improve reproducibility

[Juro Bystricky]

Delete various build host references from the internally
generated file sudo_usage.h. The references get compiled into
executables, which leads to non-reproducible builds.
The removed references (configure options) were only used as part
of the sudo "usage", and even then only when ran as root.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-extended/sudo/sudo.inc | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-extended/sudo/sudo.inc b/meta/recipes-extended/sudo/sudo.inc
index 8c16d83..80ec0ae 100644
--- a/meta/recipes-extended/sudo/sudo.inc
+++ b/meta/recipes-extended/sudo/sudo.inc
@@ -27,6 +27,12 @@ EXTRA_OECONF_append_libc-musl = " --disable-hardening "
 
 # mksigname/mksiglist are used on build host to generate source files
 do_compile_prepend () {
+	# Remove build host references from sudo_usage.h
+	sed -i  \
+	    -e 's,--with-libtool-sysroot=${STAGING_DIR_TARGET},,g' \
+	    -e 's,--build=${BUILD_SYS},,g' \
+	    -e 's,--host=${HOST_SYS},,g' \
+	    ${B}/src/sudo_usage.h
 	oe_runmake SSP_CFLAGS="" SSP_LDFLAGS="" CC="$BUILD_CC" CFLAGS="$BUILD_CFLAGS" CPPFLAGS="$BUILD_CPPFLAGS -I${S}/include -I${S} -I${B}"  -C lib/util mksigname mksiglist
 }
 
-- 
2.7.4

Re: [PATCH 1/4] gnutls: improve reproducibility

[Andre McCurdy]

On Thu, Nov 16, 2017 at 11:48 AM, Juro Bystricky
<juro.bystricky@intel.com> wrote:
> Sanitize generated config.h containing build host reference such as:
>
>  #define POSIX_SHELL "<builddir>/tmp/hosttools/bash"
>
> The path for POSIX_SHELL is not only wrong for the cross-compiled target,
> it also prevents reproducible build.
>
> Prior the compilation we edit the value to:
>
>  #define POSIX_SHELL "bash"
>
> With this change we can build reproducible libgnutls-bin, libgnutls-dev
> and libgnutls-dbg packages.

Is there a configure option we could use (perhaps via
CACHED_CONFIGUREVARS) to achieve the same end result without needing
to run sed on config.h?

> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
>  meta/recipes-support/gnutls/gnutls.inc | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/meta/recipes-support/gnutls/gnutls.inc b/meta/recipes-support/gnutls/gnutls.inc
> index 29b5dd6..5b45b27 100644
> --- a/meta/recipes-support/gnutls/gnutls.inc
> +++ b/meta/recipes-support/gnutls/gnutls.inc
> @@ -53,6 +53,11 @@ do_configure_prepend() {
>         done
>  }
>
> +do_configure_append_class-target () {
> +       # Remove build host references from config.h
> +       sed -i -e 's:${HOSTTOOLS_DIR}/::g' ${B}/config.h
> +}
> +
>  PACKAGES =+ "${PN}-openssl ${PN}-xx"
>
>  FILES_${PN}-dev += "${bindir}/gnutls-cli-debug"
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 1/4] gnutls: improve reproducibility

[Bystricky, Juro]

> Is there a configure option we could use (perhaps via
> CACHED_CONFIGUREVARS) to achieve the same end result without needing
> to run sed on config.h?

Sorry Andre, I am not quite sure I understand the question. Do you suggest to have a "config.h"
cached somewhere instead of generating the file at build time?

Re: [PATCH 1/4] gnutls: improve reproducibility

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

On Fri, Nov 17, 2017 at 5:23 PM, Bystricky, Juro <juro.bystricky@intel.com>
wrote:

>
> > Is there a configure option we could use (perhaps via
> > CACHED_CONFIGUREVARS) to achieve the same end result without needing
> > to run sed on config.h?
>
> Sorry Andre, I am not quite sure I understand the question. Do you suggest
> to have a "config.h"
> cached somewhere instead of generating the file at build time?


config.h is generally generated from config.h.in by the configure script
(and the config.h.in is generated in some cases by autoheader), and the
values of those defines are generally based on autoconf test results, which
are often cached. By supplying different cached results to those tests, you
can change the emitted config.h without having to explicitly and manually
do so.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1506 bytes --]

Re: [PATCH 1/4] gnutls: improve reproducibility

[Andre McCurdy]

On Fri, Nov 17, 2017 at 5:08 PM, Christopher Larson <kergoth@gmail.com> wrote:
> On Fri, Nov 17, 2017 at 5:23 PM, Bystricky, Juro <juro.bystricky@intel.com>
> wrote:
>> > Is there a configure option we could use (perhaps via
>> > CACHED_CONFIGUREVARS) to achieve the same end result without needing
>> > to run sed on config.h?
>>
>> Sorry Andre, I am not quite sure I understand the question. Do you suggest
>> to have a "config.h"
>> cached somewhere instead of generating the file at build time?
>
> config.h is generally generated from config.h.in by the configure script
> (and the config.h.in is generated in some cases by autoheader), and the
> values of those defines are generally based on autoconf test results, which
> are often cached. By supplying different cached results to those tests, you
> can change the emitted config.h without having to explicitly and manually do
> so.

Yes, exactly that.

Looking at the gnutls configure script, it seems that POSIX_SHELL is
detected using a custom test (rather than using a standard autotools
macro) and doesn't respect any over-rides, so running sed on config.h
may indeed be the easiest approach, but you should confirm that first.

Note also that if POSIX_SHELL is used at all at run-time then setting
it to "bash" is probably not the right approach as gnutls has no
run-time dependency on bash (and so there's no guarantee that bash
will be available in the target rootfs). If POSIX_SHELL is only used
at build time and never gets embedded in the build output then perhaps
completely removing POSIX_SHELL from any headers etc installed by
gnutls may be a better fix than just removing the path?