[PATCH 0/1] python-xcbgen: improve reproducibility

[PATCH 0/1] python-xcbgen: improve reproducibility

[Juro Bystricky]

This patch addresses the last remaining non-reproducible package for
core-image-minimal (with current "sumo").

A short explanation:
When building core-image-minimal in two different folders with current "sumo"
(http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo ) :

$ source oe-build-env build_1
local.conf:
PACKAGE_CLASSES="package_deb package_rpm package_ipk"
MACHINE="qemux86-64"
$ bitbake core-image-minimal

and

$ source oe-build-env build_2

local.conf:
PACKAGE_CLASSES="package_deb package_rpm package_ipk"
MACHINE="qemux86-64"

$ bitbake core-image-minimal

After binary comparing all packages in build_1/tmp/deploy/[ipk,rpm.deb] to
build_2/tmp/deploy/[ipk,rpm,deb], we get these results:

           IPK   DEB  RPM
Same:         0    0    0
Different: 3952 3952 3952
Total:     3952 3952 3952

Repeating the same with (same host):

$ source oe-build-env build_repro_1
local.conf:
PACKAGE_CLASSES="package_deb package_rpm package_ipk"
MACHINE="qemux86-64"
INHERIT+="reproducible_build"
$ bitbake core-image-minimal

and

$ source oe-build-env build_repro_2

local.conf:
PACKAGE_CLASSES="package_deb package_rpm package_ipk"
MACHINE="qemux86-64"
INHERIT+="reproducible_build"

$ bitbake core-image-minimal

After binary comparing all packages in build_repro_1/tmp/deploy/[ipk,rpm.deb] to
build_repro_2/tmp/deploy/[ipk,rpm,deb], we get these results:

           IPK   DEB  RPM
Same:      3951 3951 3951
Different:    1    1    1
Total:     3952 3952 3952

The remaining package that does not build reproducibly is python-xcbgen.
This is because the package contains .pyc files which contain build-time timestamps.
The python files are compiled with host Python3, which may or may not support
SOURCE_DATE_EPOCH. However, both oe-core python-native and python3-native do support
SOURCE_DATE_EPOCH, so to build the package reproducibly all we need to do is
compile the python files with python3-native. Ideally, we would use python3-native
unconditionally, but building python3-native itself can be computationally quite expensive.
So, as a compromise, we use python3-native only when BUILD_REPRODUCIBLE_BINARIES = '1'
host Python3 otherwise.


Juro Bystricky (1):
  python-xcbgen_1.12: improve reproducibility

 meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb | 5 +++++
 1 file changed, 5 insertions(+)

-- 
2.7.4

[PATCH 1/1] python-xcbgen_1.12: improve reproducibility

[Juro Bystricky]

In order to remove timestamps from all .pyc files we need to recompile
them with python3-native, as we cannot rely on the host python being
able to do that. Both python-native and python3-native derive the timestamp
from SOURCE_DATE_EPOCH if present.
However, building python3-native can be computationally expensive, so
we resort to python3-native only when building reproducible packages:
(BUILD_REPRODUCIBLE_BINARIES = '1'), otherwise we use the host python3.

[YOCTO #12543]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb b/meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb
index 25a8373..02a42af 100644
--- a/meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb
+++ b/meta/recipes-graphics/xorg-proto/xcb-proto_1.12.bb
@@ -20,6 +20,11 @@ SRC_URI[sha256sum] = "5922aba4c664ab7899a29d92ea91a87aa4c1fc7eb5ee550325c3216c48
 
 inherit autotools pkgconfig
 
+# We prefer using host Python, but for reproducible builds we need to resort to
+# python3-native in order to have timestamps in compiled modules based on SOURCE_DATE_EPOCH.
+# We cannot assume host Python supports SOURCE_DATE_EPOCH.
+inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'python3native', '')}
+
 # Force the use of Python 3 and a specific library path so we don't need to
 # depend on python3-native
 CACHED_CONFIGUREVARS += "PYTHON=python3 am_cv_python_pythondir=${libdir}/xcb-proto"
-- 
2.7.4

Re: [PATCH 1/1] python-xcbgen_1.12: improve reproducibility

[Burton, Ross]

On 6 April 2018 at 20:57, Juro Bystricky <juro.bystricky@intel.com> wrote:
> In order to remove timestamps from all .pyc files we need to recompile
> them with python3-native, as we cannot rely on the host python being
> able to do that. Both python-native and python3-native derive the timestamp
> from SOURCE_DATE_EPOCH if present.
> However, building python3-native can be computationally expensive, so
> we resort to python3-native only when building reproducible packages:
> (BUILD_REPRODUCIBLE_BINARIES = '1'), otherwise we use the host python3.

The logical extension of this is that every recipe that ships .py
needs to depend on python3-native so that it can recompile.

However, there's another solution...

.pyc files are basically a four byte magic number, a four byte
timestamp, and then the byte code.  It won't be that many lines of
Python in a do_package[postfunc] to search for __pycache__/*.pyc files
in the packages and replace the timestamp with SOURCE_DATE_EPOCH if
reproducible builds are enabled.

Ross

Re: [PATCH 1/1] python-xcbgen_1.12: improve reproducibility

[Bystricky, Juro]

This would solve the .pyc timestamps problem. However, from reproducibility point of view, it is a bit more desirable to use python-native, as - in principle - different host Python versions can produce different compiled code as well. 
So it is not the timestamps only. 

________________________________________
From: Burton, Ross [ross.burton@intel.com]
Sent: Tuesday, April 10, 2018 5:02 PM
To: Bystricky, Juro
Cc: OE-core; Purdie, Richard; Juro Bystricky; Martin Jansa
Subject: Re: [PATCH 1/1] python-xcbgen_1.12: improve reproducibility

On 6 April 2018 at 20:57, Juro Bystricky <juro.bystricky@intel.com> wrote:
> In order to remove timestamps from all .pyc files we need to recompile
> them with python3-native, as we cannot rely on the host python being
> able to do that. Both python-native and python3-native derive the timestamp
> from SOURCE_DATE_EPOCH if present.
> However, building python3-native can be computationally expensive, so
> we resort to python3-native only when building reproducible packages:
> (BUILD_REPRODUCIBLE_BINARIES = '1'), otherwise we use the host python3.

The logical extension of this is that every recipe that ships .py
needs to depend on python3-native so that it can recompile.

However, there's another solution...

.pyc files are basically a four byte magic number, a four byte
timestamp, and then the byte code.  It won't be that many lines of
Python in a do_package[postfunc] to search for __pycache__/*.pyc files
in the packages and replace the timestamp with SOURCE_DATE_EPOCH if
reproducible builds are enabled.

Ross