[PATCH v2 0/6] Reproducible binaries

[PATCH v2 0/6] Reproducible binaries

[Juro Bystricky]

This patch set (V2) contains several patches aimed to achieve reproducible binaries.
Building reproducible binaries may remove certain intentional
randomness intended for increased security. Hence, it is reasonable
to expect there will be cases where this is not desirable.
The user can select his/her preferences via the variable
BUILD_REPRODUCIBLE_BINARIES. The variable defaults to "0" (do not
build reproducible binaries) in order to minimize any potential
regressions.

For debian packages we get a lot of binary identical packages simply by
exporting SOURCE_DATE_EPOCH. This is done automatically when
BUILD_REPRODUCIBLE_BINARIES="1".

For rootfs we get much fewer differences by modified prelinking and by
ensuring various timestamps are reproducible.

For example, building core-image-minimal with this patchset,
using the following settings in the local.conf:

    BUILD_REPRODUCIBLE_BINARIES="1"
    LDCONFIGDEPEND=""
    IMAGE_CMD_TAR="tar -v --sort=name"

    #Optional user specified timestams:
    REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK="1483228800"
    REPRODUCIBLE_TIMESTAMP_ROOTFS="1483228800"

we can build binary identical core-image-minimal-rootfs.tar.bz2 images.
(Tested on the same machine, two different build folders, images built at different
times)
Eventually, it will be possible to build identical identical core-image-minimal-rootfs.ext4
as well. (Note in this test case the rootfs is built without pre-built ldconfig aux-cache).
This patchset does not address the reproducibility of the linux kernel nor
the reproducibility of linux kernel modules.



Juro Bystricky (6):
  bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES
  base.bbclass: initial support for binary reproducibility
  image-prelink.bbclass: support binary reproducibility
  rootfs-postcommands.bbclass: support binary reproducibility
  busybox.inc: improve reproducibility
  image.bbclass: support binary reproducibility

 meta/classes/base.bbclass                | 82 ++++++++++++++++++++++++++++++++
 meta/classes/image-prelink.bbclass       | 12 ++++-
 meta/classes/image.bbclass               | 12 +++++
 meta/classes/rootfs-postcommands.bbclass | 24 ++++++++--
 meta/conf/bitbake.conf                   | 11 +++++
 meta/recipes-core/busybox/busybox.inc    |  3 ++
 6 files changed, 140 insertions(+), 4 deletions(-)

-- 
2.7.4

[PATCH v2 1/6] bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES

[Juro Bystricky]

Building reproducible binaries may remove certain intentional
randomness intended for increased security. Hence, it is reasonable
to expect there will be cases where this is not desirable.
The user can select his/her preferences via the variable
BUILD_REPRODUCIBLE_BINARIES. The variable defaults to "0" (do not
build reproducible binaries) in order to minimize any potential
regressions. (Once the reproducible binaries code is mature enough,
it can be set to "1".)
If the variable BUILD_REPRODUCIBLE_BINARIES is set to "1",
timestamp values taken from additional variables will be optionally used
when building binary reproducible images:

    REPRODUCIBLE_TIMESTAMP_ROOTFS
        If the value is specified, all files mtime will be set to this value.
        In addition, /etc/timestamp and /etc/version will both contain the value.
        If no value is specified, timestamp will be derived from the top git commit.

    REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK
        Value passed via environment variable PRELINK_TIMESTAMP to the prelink program.
        If the value is specified, the value will be used.
        If no value is specified, timestamp will be derived from the top git commit.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/conf/bitbake.conf | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 227babd..6ce1a1a 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -859,3 +859,14 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
 
 MLPREFIX ??= ""
 MULTILIB_VARIANTS ??= ""
+
+BUILD_REPRODUCIBLE_BINARIES ??= "0"
+BUILD_REPRODUCIBLE_BINARIES[export] = "1"
+
+# Unix timestamp
+REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
+REPRODUCIBLE_TIMESTAMP_ROOTFS[export] = "1"
+
+# Unix timestamp
+REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK ??= ""
+REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK[export] = "1"
-- 
2.7.4

Re: [PATCH v2 1/6] bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES

[Richard Purdie]

On Mon, 2017-05-01 at 13:58 -0700, Juro Bystricky wrote:
> Building reproducible binaries may remove certain intentional
> randomness intended for increased security. Hence, it is reasonable
> to expect there will be cases where this is not desirable.
> The user can select his/her preferences via the variable
> BUILD_REPRODUCIBLE_BINARIES. The variable defaults to "0" (do not
> build reproducible binaries) in order to minimize any potential
> regressions. (Once the reproducible binaries code is mature enough,
> it can be set to "1".)
> If the variable BUILD_REPRODUCIBLE_BINARIES is set to "1",
> timestamp values taken from additional variables will be optionally
> used
> when building binary reproducible images:
> 
>     REPRODUCIBLE_TIMESTAMP_ROOTFS
>         If the value is specified, all files mtime will be set to
> this value.
>         In addition, /etc/timestamp and /etc/version will both
> contain the value.
>         If no value is specified, timestamp will be derived from the
> top git commit.
> 
>     REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK
>         Value passed via environment variable PRELINK_TIMESTAMP to
> the prelink program.
>         If the value is specified, the value will be used.
>         If no value is specified, timestamp will be derived from the
> top git commit.
> 
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
>  meta/conf/bitbake.conf | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 227babd..6ce1a1a 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -859,3 +859,14 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends
> \
>  
>  MLPREFIX ??= ""
>  MULTILIB_VARIANTS ??= ""
> +
> +BUILD_REPRODUCIBLE_BINARIES ??= "0"
> +BUILD_REPRODUCIBLE_BINARIES[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
> +REPRODUCIBLE_TIMESTAMP_ROOTFS[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK ??= ""
> +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK[export] = "1"

Please don't add new global exports in bitbake.conf. Changing the value
of this will cause everything to rebuild (e.g. recompile) since the
exported environment goes to all tasks. We really don't want to do that
if it only affects the image generation.

I'll give this a bit more thought/review but wanted to comment on this
whilst I see it/remember.

Cheers,

Richard

Re: [PATCH v2 1/6] bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES

[Bystricky, Juro]

I see your point. The original idea was to keep all related variables in one place. There is 
one variable ( BUILD_REPRODUCIBLE_BINARIES ) that I think should be global,
as it should be visible by all tasks (well, a lot of tasks). The rest can be moved to more appropriate places.


________________________________________
From: Richard Purdie [richard.purdie@linuxfoundation.org]
Sent: Monday, May 01, 2017 4:13 PM
To: Bystricky, Juro; openembedded-core@lists.openembedded.org
Cc: joshua.g.lock@linux.intel.com; Burton, Ross; martin.jansa@gmail.com; raj.khem@gmail.com; jurobystricky@hotmail.com
Subject: Re: [PATCH v2 1/6] bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES

On Mon, 2017-05-01 at 13:58 -0700, Juro Bystricky wrote:
> Building reproducible binaries may remove certain intentional
> randomness intended for increased security. Hence, it is reasonable
> to expect there will be cases where this is not desirable.
> The user can select his/her preferences via the variable
> BUILD_REPRODUCIBLE_BINARIES. The variable defaults to "0" (do not
> build reproducible binaries) in order to minimize any potential
> regressions. (Once the reproducible binaries code is mature enough,
> it can be set to "1".)
> If the variable BUILD_REPRODUCIBLE_BINARIES is set to "1",
> timestamp values taken from additional variables will be optionally
> used
> when building binary reproducible images:
>
>     REPRODUCIBLE_TIMESTAMP_ROOTFS
>         If the value is specified, all files mtime will be set to
> this value.
>         In addition, /etc/timestamp and /etc/version will both
> contain the value.
>         If no value is specified, timestamp will be derived from the
> top git commit.
>
>     REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK
>         Value passed via environment variable PRELINK_TIMESTAMP to
> the prelink program.
>         If the value is specified, the value will be used.
>         If no value is specified, timestamp will be derived from the
> top git commit.
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
>  meta/conf/bitbake.conf | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 227babd..6ce1a1a 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -859,3 +859,14 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends
> \
>
>  MLPREFIX ??= ""
>  MULTILIB_VARIANTS ??= ""
> +
> +BUILD_REPRODUCIBLE_BINARIES ??= "0"
> +BUILD_REPRODUCIBLE_BINARIES[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
> +REPRODUCIBLE_TIMESTAMP_ROOTFS[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK ??= ""
> +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK[export] = "1"

Please don't add new global exports in bitbake.conf. Changing the value
of this will cause everything to rebuild (e.g. recompile) since the
exported environment goes to all tasks. We really don't want to do that
if it only affects the image generation.

I'll give this a bit more thought/review but wanted to comment on this
whilst I see it/remember.

Cheers,

Richard

Re: [PATCH v2 1/6] bitbake.conf: new variable BUILD_REPRODUCIBLE_BINARIES

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3528 bytes --]

I think you can define them in bitbake.conf, but then export them only
where needed.

On Tue, May 2, 2017 at 2:35 AM, Bystricky, Juro <juro.bystricky@intel.com>
wrote:

> I see your point. The original idea was to keep all related variables in
> one place. There is
> one variable ( BUILD_REPRODUCIBLE_BINARIES ) that I think should be global,
> as it should be visible by all tasks (well, a lot of tasks). The rest can
> be moved to more appropriate places.
>
>
> ________________________________________
> From: Richard Purdie [richard.purdie@linuxfoundation.org]
> Sent: Monday, May 01, 2017 4:13 PM
> To: Bystricky, Juro; openembedded-core@lists.openembedded.org
> Cc: joshua.g.lock@linux.intel.com; Burton, Ross; martin.jansa@gmail.com;
> raj.khem@gmail.com; jurobystricky@hotmail.com
> Subject: Re: [PATCH v2 1/6] bitbake.conf: new variable
> BUILD_REPRODUCIBLE_BINARIES
>
> On Mon, 2017-05-01 at 13:58 -0700, Juro Bystricky wrote:
> > Building reproducible binaries may remove certain intentional
> > randomness intended for increased security. Hence, it is reasonable
> > to expect there will be cases where this is not desirable.
> > The user can select his/her preferences via the variable
> > BUILD_REPRODUCIBLE_BINARIES. The variable defaults to "0" (do not
> > build reproducible binaries) in order to minimize any potential
> > regressions. (Once the reproducible binaries code is mature enough,
> > it can be set to "1".)
> > If the variable BUILD_REPRODUCIBLE_BINARIES is set to "1",
> > timestamp values taken from additional variables will be optionally
> > used
> > when building binary reproducible images:
> >
> >     REPRODUCIBLE_TIMESTAMP_ROOTFS
> >         If the value is specified, all files mtime will be set to
> > this value.
> >         In addition, /etc/timestamp and /etc/version will both
> > contain the value.
> >         If no value is specified, timestamp will be derived from the
> > top git commit.
> >
> >     REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK
> >         Value passed via environment variable PRELINK_TIMESTAMP to
> > the prelink program.
> >         If the value is specified, the value will be used.
> >         If no value is specified, timestamp will be derived from the
> > top git commit.
> >
> > Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> > ---
> >  meta/conf/bitbake.conf | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 227babd..6ce1a1a 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -859,3 +859,14 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends
> > \
> >
> >  MLPREFIX ??= ""
> >  MULTILIB_VARIANTS ??= ""
> > +
> > +BUILD_REPRODUCIBLE_BINARIES ??= "0"
> > +BUILD_REPRODUCIBLE_BINARIES[export] = "1"
> > +
> > +# Unix timestamp
> > +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
> > +REPRODUCIBLE_TIMESTAMP_ROOTFS[export] = "1"
> > +
> > +# Unix timestamp
> > +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK ??= ""
> > +REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK[export] = "1"
>
> Please don't add new global exports in bitbake.conf. Changing the value
> of this will cause everything to rebuild (e.g. recompile) since the
> exported environment goes to all tasks. We really don't want to do that
> if it only affects the image generation.
>
> I'll give this a bit more thought/review but wanted to comment on this
> whilst I see it/remember.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 4738 bytes --]

[PATCH v2 2/6] base.bbclass: initial support for binary reproducibility

[Juro Bystricky]

Conditionally set some environment variables in order to achieve
improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
set to "1", we set the following environment variables:

export PYTHONHASHSEED=0
export PERL_HASH_SEED=0
export TZ="UTC"

We also export and set SOURCE_DATE_EPOCH. The value for this variable
is obtained after source code for a recipe has been unpacked, but before it is
patched. If the code comes from a GIT repo, we get the timestamp from the top
commit. (This usually corresponds to the mktime of "changelog".)
Otherwise we go through all files and get the timestamp from the youngest
one. We create a timestamp for each recipe. The timestamp is stored in the file
'src_date_epoch.txt'. Later on, each task reads this file and sets SOURCE_DATE_EPOCH
based on the value found in the file.

[YOCTO#11178]
[YOCTO#11179]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/classes/base.bbclass | 82 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index e29821f..f2b2d97 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -10,6 +10,52 @@ inherit utility-tasks
 inherit metadata_scm
 inherit logging
 
+def get_git_src_date_epoch(d, path):
+    import subprocess
+    saved_cwd = os.getcwd()
+    os.chdir(path)
+    src_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct']))
+    os.chdir(saved_cwd)
+    return src_date_epoch
+
+def create_src_date_epoch_stamp(d):
+    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+        path = d.getVar('S')
+        src_date_epoch = 0
+        filename_dbg = None
+
+        if path.endswith('/git'):
+            src_date_epoch = get_git_src_date_epoch(d, path)
+        else:
+            exclude = set(["temp", "licenses", "patches", "recipe-sysroot-native", "recipe-sysroot" ])
+            for root, dirs, files in os.walk(path, topdown=True):
+                dirs[:] = [d for d in dirs if d not in exclude]
+                if root.endswith('/git'):
+                    src_date_epoch = get_git_src_date_epoch(d, root)
+                    break
+
+                for fname in files:
+                    filename = os.path.join(root, fname)
+                    try:
+                        mtime = int(os.path.getmtime(filename))
+                    except:
+                        mtime = 0
+                    if mtime > src_date_epoch:
+                        src_date_epoch = mtime
+                        filename_dbg = filename
+
+        # Most likely an empty folder
+        if src_date_epoch == 0:
+            bb.warn("Unable to determine src_date_epoch! path:%s" % path)
+
+        f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
+        f.write(str(src_date_epoch))
+        f.close()
+
+        if filename_dbg != None:
+            bb.debug(1," src_date_epoch %d derived from: %s" % (src_date_epoch, filename_dbg))
+
+
 OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
 OE_IMPORTS[type] = "list"
 
@@ -173,6 +219,7 @@ python base_do_unpack() {
     try:
         fetcher = bb.fetch2.Fetch(src_uri, d)
         fetcher.unpack(d.getVar('WORKDIR'))
+        create_src_date_epoch_stamp(d)
     except bb.fetch2.BBFetchException as e:
         bb.fatal(str(e))
 }
@@ -383,9 +430,43 @@ def set_packagetriplet(d):
 
     settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)
 
+
+export PYTHONHASHSEED
+export PERL_HASH_SEED
+export SOURCE_DATE_EPOCH
+
+BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED "
+
 python () {
     import string, re
 
+    # Create reproducible_environment
+
+    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+        import subprocess
+        d.setVar('PYTHONHASHSEED', '0')
+        d.setVar('PERL_HASH_SEED', '0')
+        d.setVar('TZ', 'UTC')
+
+        path = d.getVar('S')
+        epochfile = os.path.join(path,'src_date_epoch.txt')
+        if os.path.isfile(epochfile):
+            f = open(epochfile, 'r')
+            src_date_epoch = f.read()
+            f.close()
+            bb.debug(1, "src_date_epoch stamp found ---> stamp %s" % src_date_epoch)
+            d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
+        else:
+            bb.debug(1, "src_date_epoch stamp not found.")
+            d.setVar('SOURCE_DATE_EPOCH', '0')
+    else:
+        if 'PYTHONHASHSEED' in os.environ:
+            del os.environ['PYTHONHASHSEED']
+        if 'PERL_HASH_SEED' in os.environ:
+            del os.environ['PERL_HASH_SEED']
+        if 'SOURCE_DATE_EPOCH' in os.environ:
+            del os.environ['SOURCE_DATE_EPOCH']
+
     # Handle PACKAGECONFIG
     #
     # These take the form:
@@ -678,6 +759,7 @@ python () {
             bb.warn("Recipe %s is marked as only being architecture specific but seems to have machine specific packages?! The recipe may as well mark itself as machine specific directly." % d.getVar("PN"))
 }
 
+
 addtask cleansstate after do_clean
 python do_cleansstate() {
         sstate_clean_cachefiles(d)
-- 
2.7.4

Re: [PATCH v2 2/6] base.bbclass: initial support for binary reproducibility

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 9470 bytes --]

For some recipes (llvm, chromium, chromium-wayland) I've noticed this
function to fail with:

ERROR: llvm3.3-3.3-r0 do_unpack: Error executing a python function in
exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:base_do_unpack(d)
     0003:
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass',
lineno: 215, function: base_do_unpack
     0211:
     0212:    try:
     0213:        fetcher = bb.fetch2.Fetch(src_uri, d)
     0214:        fetcher.unpack(d.getVar('WORKDIR'))
 *** 0215:        create_src_date_epoch_stamp(d)
     0216:    except bb.fetch2.BBFetchException as e:
     0217:        bb.fatal(str(e))
     0218:}
     0219:
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass',
lineno: 34, function: create_src_date_epoch_stamp
     0030:            exclude = set(["temp", "licenses", "patches",
"recipe-sysroot-native", "recipe-sysroot" ])
     0031:            for root, dirs, files in os.walk(path, topdown=True):
     0032:                dirs[:] = [d for d in dirs if d not in exclude]
     0033:                if root.endswith('/git'):
 *** 0034:                    src_date_epoch = get_git_src_date_epoch(d, root)
     0035:                    break
     0036:
     0037:                for fname in files:
     0038:                    filename = os.path.join(root, fname)
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass',
lineno: 17, function: get_git_src_date_epoch
     0013:def get_git_src_date_epoch(d, path):
     0014:    import subprocess
     0015:    saved_cwd = os.getcwd()
     0016:    os.chdir(path)
 *** 0017:    src_date_epoch =
int(subprocess.check_output(['git','log','-1','--pretty=%ct']))
     0018:    os.chdir(saved_cwd)
     0019:    return src_date_epoch
     0020:
     0021:def create_src_date_epoch_stamp(d):
File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output
     0622:        # empty string. That is maintained here for
backwards compatibility.
     0623:        kwargs['input'] = '' if
kwargs.get('universal_newlines', False) else b''
     0624:
     0625:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 *** 0626:               **kwargs).stdout
     0627:
     0628:
     0629:class CompletedProcess(object):
     0630:    """A process that has finished running.
File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run
     0704:            raise
     0705:        retcode = process.poll()
     0706:        if check and retcode:
     0707:            raise CalledProcessError(retcode, process.args,
 *** 0708:                                     output=stdout, stderr=stderr)
     0709:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0710:
     0711:
     0712:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command '['git', 'log',
'-1', '--pretty=%ct']' returned non-zero exit status 128

ERROR: llvm3.3-3.3-r0 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in:
/home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe-linux/llvm3.3/3.3-r0/temp/log.do_unpack.25005
NOTE: recipe llvm3.3-3.3-r0: task do_unpack: Failed
ERROR: Task (/home/jenkins/oe/world/shr-core/meta-openembedded/meta-oe/recipes-core/llvm/llvm3.3_3.3.bb:do_unpack)
failed with exit code '1'


Maybe I don't have the latest version of this patch (I'm not using
your poky-contrib branch yet), but it should fail with nicer message
when git log fails for whatever reason.


On Mon, May 1, 2017 at 10:59 PM, Juro Bystricky <juro.bystricky@intel.com>
wrote:

> Conditionally set some environment variables in order to achieve
> improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
> set to "1", we set the following environment variables:
>
> export PYTHONHASHSEED=0
> export PERL_HASH_SEED=0
> export TZ="UTC"
>
> We also export and set SOURCE_DATE_EPOCH. The value for this variable
> is obtained after source code for a recipe has been unpacked, but before
> it is
> patched. If the code comes from a GIT repo, we get the timestamp from the
> top
> commit. (This usually corresponds to the mktime of "changelog".)
> Otherwise we go through all files and get the timestamp from the youngest
> one. We create a timestamp for each recipe. The timestamp is stored in the
> file
> 'src_date_epoch.txt'. Later on, each task reads this file and sets
> SOURCE_DATE_EPOCH
> based on the value found in the file.
>
> [YOCTO#11178]
> [YOCTO#11179]
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
>  meta/classes/base.bbclass | 82 ++++++++++++++++++++++++++++++
> +++++++++++++++++
>  1 file changed, 82 insertions(+)
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index e29821f..f2b2d97 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -10,6 +10,52 @@ inherit utility-tasks
>  inherit metadata_scm
>  inherit logging
>
> +def get_git_src_date_epoch(d, path):
> +    import subprocess
> +    saved_cwd = os.getcwd()
> +    os.chdir(path)
> +    src_date_epoch = int(subprocess.check_output(['
> git','log','-1','--pretty=%ct']))
> +    os.chdir(saved_cwd)
> +    return src_date_epoch
> +
> +def create_src_date_epoch_stamp(d):
> +    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> +        path = d.getVar('S')
> +        src_date_epoch = 0
> +        filename_dbg = None
> +
> +        if path.endswith('/git'):
> +            src_date_epoch = get_git_src_date_epoch(d, path)
> +        else:
> +            exclude = set(["temp", "licenses", "patches",
> "recipe-sysroot-native", "recipe-sysroot" ])
> +            for root, dirs, files in os.walk(path, topdown=True):
> +                dirs[:] = [d for d in dirs if d not in exclude]
> +                if root.endswith('/git'):
> +                    src_date_epoch = get_git_src_date_epoch(d, root)
> +                    break
> +
> +                for fname in files:
> +                    filename = os.path.join(root, fname)
> +                    try:
> +                        mtime = int(os.path.getmtime(filename))
> +                    except:
> +                        mtime = 0
> +                    if mtime > src_date_epoch:
> +                        src_date_epoch = mtime
> +                        filename_dbg = filename
> +
> +        # Most likely an empty folder
> +        if src_date_epoch == 0:
> +            bb.warn("Unable to determine src_date_epoch! path:%s" % path)
> +
> +        f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
> +        f.write(str(src_date_epoch))
> +        f.close()
> +
> +        if filename_dbg != None:
> +            bb.debug(1," src_date_epoch %d derived from: %s" %
> (src_date_epoch, filename_dbg))
> +
> +
>  OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
>  OE_IMPORTS[type] = "list"
>
> @@ -173,6 +219,7 @@ python base_do_unpack() {
>      try:
>          fetcher = bb.fetch2.Fetch(src_uri, d)
>          fetcher.unpack(d.getVar('WORKDIR'))
> +        create_src_date_epoch_stamp(d)
>      except bb.fetch2.BBFetchException as e:
>          bb.fatal(str(e))
>  }
> @@ -383,9 +430,43 @@ def set_packagetriplet(d):
>
>      settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)
>
> +
> +export PYTHONHASHSEED
> +export PERL_HASH_SEED
> +export SOURCE_DATE_EPOCH
> +
> +BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED
> "
> +
>  python () {
>      import string, re
>
> +    # Create reproducible_environment
> +
> +    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> +        import subprocess
> +        d.setVar('PYTHONHASHSEED', '0')
> +        d.setVar('PERL_HASH_SEED', '0')
> +        d.setVar('TZ', 'UTC')
> +
> +        path = d.getVar('S')
> +        epochfile = os.path.join(path,'src_date_epoch.txt')
> +        if os.path.isfile(epochfile):
> +            f = open(epochfile, 'r')
> +            src_date_epoch = f.read()
> +            f.close()
> +            bb.debug(1, "src_date_epoch stamp found ---> stamp %s" %
> src_date_epoch)
> +            d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
> +        else:
> +            bb.debug(1, "src_date_epoch stamp not found.")
> +            d.setVar('SOURCE_DATE_EPOCH', '0')
> +    else:
> +        if 'PYTHONHASHSEED' in os.environ:
> +            del os.environ['PYTHONHASHSEED']
> +        if 'PERL_HASH_SEED' in os.environ:
> +            del os.environ['PERL_HASH_SEED']
> +        if 'SOURCE_DATE_EPOCH' in os.environ:
> +            del os.environ['SOURCE_DATE_EPOCH']
> +
>      # Handle PACKAGECONFIG
>      #
>      # These take the form:
> @@ -678,6 +759,7 @@ python () {
>              bb.warn("Recipe %s is marked as only being architecture
> specific but seems to have machine specific packages?! The recipe may as
> well mark itself as machine specific directly." % d.getVar("PN"))
>  }
>
> +
>  addtask cleansstate after do_clean
>  python do_cleansstate() {
>          sstate_clean_cachefiles(d)
> --
> 2.7.4
>
>

[-- Attachment #2: Type: text/html, Size: 11745 bytes --]

Re: [PATCH v2 2/6] base.bbclass: initial support for binary reproducibility

[Bystricky, Juro]

[-- Attachment #1: Type: text/plain, Size: 10024 bytes --]

Thanks, it seems there is not-too-great error checking implemented. There is an assumption if there is
a git folder there is also a git repo (and "git" binary in PATH). Seems the assumption is wrong.
I'll add some error checking/improved error message. The problem with git repos is they modify the
file mod timestamps upon checkout, so it is not possible to find the one that is truly the youngest.
So the code tries to get the timestamp of the top commit instead.

________________________________
From: Martin Jansa [martin.jansa@gmail.com]
Sent: Wednesday, June 14, 2017 1:30 PM
To: Bystricky, Juro
Cc: Patches and discussions about the oe-core layer; Richard Purdie; Joshua G Lock; Burton, Ross; Khem Raj; Juro Bystricky
Subject: Re: [PATCH v2 2/6] base.bbclass: initial support for binary reproducibility

For some recipes (llvm, chromium, chromium-wayland) I've noticed this function to fail with:


ERROR: llvm3.3-3.3-r0 do_unpack: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:base_do_unpack(d)
     0003:
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass', lineno: 215, function: base_do_unpack
     0211:
     0212:    try:
     0213:        fetcher = bb.fetch2.Fetch(src_uri, d)
     0214:        fetcher.unpack(d.getVar('WORKDIR'))
 *** 0215:        create_src_date_epoch_stamp(d)
     0216:    except bb.fetch2.BBFetchException as e:
     0217:        bb.fatal(str(e))
     0218:}
     0219:
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass', lineno: 34, function: create_src_date_epoch_stamp
     0030:            exclude = set(["temp", "licenses", "patches", "recipe-sysroot-native", "recipe-sysroot" ])
     0031:            for root, dirs, files in os.walk(path, topdown=True):
     0032:                dirs[:] = [d for d in dirs if d not in exclude]
     0033:                if root.endswith('/git'):
 *** 0034:                    src_date_epoch = get_git_src_date_epoch(d, root)
     0035:                    break
     0036:
     0037:                for fname in files:
     0038:                    filename = os.path.join(root, fname)
File: '/home/jenkins/oe/world/shr-core/openembedded-core/meta/classes/base.bbclass', lineno: 17, function: get_git_src_date_epoch
     0013:def get_git_src_date_epoch(d, path):
     0014:    import subprocess
     0015:    saved_cwd = os.getcwd()
     0016:    os.chdir(path)
 *** 0017:    src_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct']))
     0018:    os.chdir(saved_cwd)
     0019:    return src_date_epoch
     0020:
     0021:def create_src_date_epoch_stamp(d):
File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output
     0622:        # empty string. That is maintained here for backwards compatibility.
     0623:        kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
     0624:
     0625:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 *** 0626:               **kwargs).stdout
     0627:
     0628:
     0629:class CompletedProcess(object):
     0630:    """A process that has finished running.
File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run
     0704:            raise
     0705:        retcode = process.poll()
     0706:        if check and retcode:
     0707:            raise CalledProcessError(retcode, process.args,
 *** 0708:                                     output=stdout, stderr=stderr)
     0709:    return CompletedProcess(process.args, retcode, stdout, stderr)
     0710:
     0711:
     0712:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command '['git', 'log', '-1', '--pretty=%ct']' returned non-zero exit status 128

ERROR: llvm3.3-3.3-r0 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe-linux/llvm3.3/3.3-r0/temp/log.do_unpack.25005
NOTE: recipe llvm3.3-3.3-r0: task do_unpack: Failed
ERROR: Task (/home/jenkins/oe/world/shr-core/meta-openembedded/meta-oe/recipes-core/llvm/llvm3.3_3.3.bb:do_unpack) failed with exit code '1'


Maybe I don't have the latest version of this patch (I'm not using your poky-contrib branch yet), but it should fail with nicer message when git log fails for whatever reason.

On Mon, May 1, 2017 at 10:59 PM, Juro Bystricky <juro.bystricky@intel.com<mailto:juro.bystricky@intel.com>> wrote:
Conditionally set some environment variables in order to achieve
improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
set to "1", we set the following environment variables:

export PYTHONHASHSEED=0
export PERL_HASH_SEED=0
export TZ="UTC"

We also export and set SOURCE_DATE_EPOCH. The value for this variable
is obtained after source code for a recipe has been unpacked, but before it is
patched. If the code comes from a GIT repo, we get the timestamp from the top
commit. (This usually corresponds to the mktime of "changelog".)
Otherwise we go through all files and get the timestamp from the youngest
one. We create a timestamp for each recipe. The timestamp is stored in the file
'src_date_epoch.txt'. Later on, each task reads this file and sets SOURCE_DATE_EPOCH
based on the value found in the file.

[YOCTO#11178]
[YOCTO#11179]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com<mailto:juro.bystricky@intel.com>>
---
 meta/classes/base.bbclass | 82 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index e29821f..f2b2d97 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -10,6 +10,52 @@ inherit utility-tasks
 inherit metadata_scm
 inherit logging

+def get_git_src_date_epoch(d, path):
+    import subprocess
+    saved_cwd = os.getcwd()
+    os.chdir(path)
+    src_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct']))
+    os.chdir(saved_cwd)
+    return src_date_epoch
+
+def create_src_date_epoch_stamp(d):
+    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+        path = d.getVar('S')
+        src_date_epoch = 0
+        filename_dbg = None
+
+        if path.endswith('/git'):
+            src_date_epoch = get_git_src_date_epoch(d, path)
+        else:
+            exclude = set(["temp", "licenses", "patches", "recipe-sysroot-native", "recipe-sysroot" ])
+            for root, dirs, files in os.walk(path, topdown=True):
+                dirs[:] = [d for d in dirs if d not in exclude]
+                if root.endswith('/git'):
+                    src_date_epoch = get_git_src_date_epoch(d, root)
+                    break
+
+                for fname in files:
+                    filename = os.path.join(root, fname)
+                    try:
+                        mtime = int(os.path.getmtime(filename))
+                    except:
+                        mtime = 0
+                    if mtime > src_date_epoch:
+                        src_date_epoch = mtime
+                        filename_dbg = filename
+
+        # Most likely an empty folder
+        if src_date_epoch == 0:
+            bb.warn("Unable to determine src_date_epoch! path:%s" % path)
+
+        f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
+        f.write(str(src_date_epoch))
+        f.close()
+
+        if filename_dbg != None:
+            bb.debug(1," src_date_epoch %d derived from: %s" % (src_date_epoch, filename_dbg))
+
+
 OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
 OE_IMPORTS[type] = "list"

@@ -173,6 +219,7 @@ python base_do_unpack() {
     try:
         fetcher = bb.fetch2.Fetch(src_uri, d)
         fetcher.unpack(d.getVar('WORKDIR'))
+        create_src_date_epoch_stamp(d)
     except bb.fetch2.BBFetchException as e:
         bb.fatal(str(e))
 }
@@ -383,9 +430,43 @@ def set_packagetriplet(d):

     settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)

+
+export PYTHONHASHSEED
+export PERL_HASH_SEED
+export SOURCE_DATE_EPOCH
+
+BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED "
+
 python () {
     import string, re

+    # Create reproducible_environment
+
+    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+        import subprocess
+        d.setVar('PYTHONHASHSEED', '0')
+        d.setVar('PERL_HASH_SEED', '0')
+        d.setVar('TZ', 'UTC')
+
+        path = d.getVar('S')
+        epochfile = os.path.join(path,'src_date_epoch.txt')
+        if os.path.isfile(epochfile):
+            f = open(epochfile, 'r')
+            src_date_epoch = f.read()
+            f.close()
+            bb.debug(1, "src_date_epoch stamp found ---> stamp %s" % src_date_epoch)
+            d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
+        else:
+            bb.debug(1, "src_date_epoch stamp not found.")
+            d.setVar('SOURCE_DATE_EPOCH', '0')
+    else:
+        if 'PYTHONHASHSEED' in os.environ:
+            del os.environ['PYTHONHASHSEED']
+        if 'PERL_HASH_SEED' in os.environ:
+            del os.environ['PERL_HASH_SEED']
+        if 'SOURCE_DATE_EPOCH' in os.environ:
+            del os.environ['SOURCE_DATE_EPOCH']
+
     # Handle PACKAGECONFIG
     #
     # These take the form:
@@ -678,6 +759,7 @@ python () {
             bb.warn("Recipe %s is marked as only being architecture specific but seems to have machine specific packages?! The recipe may as well mark itself as machine specific directly." % d.getVar("PN"))
 }

+
 addtask cleansstate after do_clean
 python do_cleansstate() {
         sstate_clean_cachefiles(d)
--
2.7.4



[-- Attachment #2: Type: text/html, Size: 14876 bytes --]

[PATCH v2 3/6] image-prelink.bbclass: support binary reproducibility

[Juro Bystricky]

Conditionally support binary reproducibility in built images.
If BUILD_REPRODUCIBLE_BINARIES = 1 then:

1. Do not randomize library addresses
2. Set/export PRELINK_TIMESTAMP to a reproducible value.
   If REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK is specified, then the value will
   be used. Otherwise the timestamp will be derived from the top git commit.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/classes/image-prelink.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image-prelink.bbclass b/meta/classes/image-prelink.bbclass
index 4157df0..046660e 100644
--- a/meta/classes/image-prelink.bbclass
+++ b/meta/classes/image-prelink.bbclass
@@ -36,7 +36,17 @@ prelink_image () {
 	dynamic_loader=$(linuxloader)
 
 	# prelink!
-	${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+	if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+		bbnote " prelink: BUILD_REPRODUCIBLE_BINARIES..."
+		if [ "$REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK" = "" ]; then
+			export PRELINK_TIMESTAMP=`git log -1 --pretty=%ct `
+		else
+			export PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK
+		fi
+		${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+	else
+		${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+	fi
 
 	# Remove the prelink.conf if we had to add it.
 	if [ "$dummy_prelink_conf" = "true" ]; then
-- 
2.7.4

[PATCH v2 4/6] rootfs-postcommands.bbclass: support binary reproducibility

[Juro Bystricky]

Conditionally support binary reproducibility of rootfs images.
If BUILD_REPRODUCIBLE_BINARIES = 1 then:

1. set /etc/timestamp to a reproducible value
2. set /etc/version to a reproducible value

The reproducible value is taken from the variable REPRODUCIBLE_TIMESTAMP_ROOTFS.
If the variable is not specified, the timestamp value is derived from
the top git commit.

[YOCTO#11176]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/classes/rootfs-postcommands.bbclass | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index 498174a..072b43d 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -48,6 +48,7 @@ ROOTFS_POSTPROCESS_COMMAND_append_qemuall = "${SSH_DISABLE_DNS_LOOKUP}"
 SORT_PASSWD_POSTPROCESS_COMMAND ??= " sort_passwd; "
 python () {
     d.appendVar('ROOTFS_POSTPROCESS_COMMAND', '${SORT_PASSWD_POSTPROCESS_COMMAND}')
+    d.appendVar('ROOTFS_POSTPROCESS_COMMAND', 'rootfs_reproducible;')
 }
 
 systemd_create_users () {
@@ -243,10 +244,12 @@ python write_image_manifest () {
         os.symlink(os.path.basename(manifest_name), manifest_link)
 }
 
-# Can be use to create /etc/timestamp during image construction to give a reasonably
+# Can be used to create /etc/timestamp during image construction to give a reasonably
 # sane default time setting
 rootfs_update_timestamp () {
-	date -u +%4Y%2m%2d%2H%2M%2S >${IMAGE_ROOTFS}/etc/timestamp
+	if [ "$BUILD_REPRODUCIBLE_BINARIES" = "0" ]; then
+		date -u +%4Y%2m%2d%2H%2M%2S >${IMAGE_ROOTFS}/etc/timestamp
+	fi
 }
 
 # Prevent X from being started
@@ -286,7 +289,6 @@ rootfs_sysroot_relativelinks () {
 	sysroot-relativelinks.py ${SDK_OUTPUT}/${SDKTARGETSYSROOT}
 }
 
-
 # Generated test data json file
 python write_image_test_data() {
     from oe.data import export2json
@@ -302,3 +304,19 @@ python write_image_test_data() {
        os.remove(testdata_link)
     os.symlink(os.path.basename(testdata), testdata_link)
 }
+
+# Perform any additional adjustments needed to make rootf binary reproducible
+rootfs_reproducible () {
+	if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+		if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
+			REPRODUCIBLE_TIMESTAMP_ROOTFS=`git log -1 --pretty=%ct`
+		fi
+
+		# Convert UTC into %4Y%2m%2d%2H%2M%2S
+		sformatted=`date -u -d @$REPRODUCIBLE_TIMESTAMP_ROOTFS +%4Y%2m%2d%2H%2M%2S`
+		echo $sformatted > ${IMAGE_ROOTFS}/etc/version
+		bbnote "rootfs_reproducible: set /etc/version to $sformatted"
+		echo $sformatted > ${IMAGE_ROOTFS}/etc/timestamp
+		bbnote "rootfs_reproducible: set /etc/timestamp to $sformatted"
+	fi
+}
-- 
2.7.4

[PATCH v2 5/6] busybox.inc: improve reproducibility

[Juro Bystricky]

For reproducible builds do not generate build timestamp as part of
the version string.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/recipes-core/busybox/busybox.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index 375632d..3e991f9 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -138,6 +138,9 @@ do_configure () {
 
 do_compile() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
+	if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+		export KCONFIG_NOTIMESTAMP=1
+	fi
 	if [ "${BUSYBOX_SPLIT_SUID}" = "1" -a x`grep "CONFIG_FEATURE_INDIVIDUAL=y" .config` = x ]; then
 	# split the .config into two parts, and make two busybox binaries
 		if [ -e .config.orig ]; then
-- 
2.7.4

Re: [PATCH v2 5/6] busybox.inc: improve reproducibility

[Andre McCurdy]

On Mon, May 1, 2017 at 1:59 PM, Juro Bystricky <juro.bystricky@intel.com> wrote:
> For reproducible builds do not generate build timestamp as part of
> the version string.

Maybe just do that by default?

> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
>  meta/recipes-core/busybox/busybox.inc | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
> index 375632d..3e991f9 100644
> --- a/meta/recipes-core/busybox/busybox.inc
> +++ b/meta/recipes-core/busybox/busybox.inc
> @@ -138,6 +138,9 @@ do_configure () {
>
>  do_compile() {
>         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> +       if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
> +               export KCONFIG_NOTIMESTAMP=1
> +       fi
>         if [ "${BUSYBOX_SPLIT_SUID}" = "1" -a x`grep "CONFIG_FEATURE_INDIVIDUAL=y" .config` = x ]; then
>         # split the .config into two parts, and make two busybox binaries
>                 if [ -e .config.orig ]; then
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[PATCH v2 6/6] image.bbclass: support binary reproducibility

[Juro Bystricky]

Added a new task "reproducible_final_image_task".
If binary reproducibility is desired ($BUILD_REPRODUCIBLE_BINARIES" = "1"),
then recursivley modify mtimes of all files to a reproducible vale.
The value is obtained via REPRODUCIBLE_TIMESTAMP_ROOTFS.
This task is executed as the very last step in image creation, once all
the files in the image have been finalized.

[YOCTO#11176]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/classes/image.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 405fd73..c311de5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -617,3 +617,15 @@ do_bundle_initramfs () {
 	:
 }
 addtask bundle_initramfs after do_image_complete
+
+reproducible_final_image_task () {
+    if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+        if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
+            REPRODUCIBLE_TIMESTAMP_ROOTFS=`git log -1 --pretty=%ct`
+        fi
+        # Set mtime of all files to a reproducible value
+        bbnote "reproducible_final_image_task: mtime set to $REPRODUCIBLE_TIMESTAMP_ROOTFS"
+        find  ${IMAGE_ROOTFS} -exec touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \;
+    fi
+}
+IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; "
-- 
2.7.4