[Buildroot] [PATCH 00/15] Reproducible builds

[Buildroot] [PATCH 00/15] Reproducible builds

[Jérôme Pouiller]

This series try to continue work initiated by Gilles Chanteperdrix:
  http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
  http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
  - I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
  - I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
  - They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

This version focuses on timestamps. It provide good enough results as soon as
OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files.
Only patch called "remove full path from .pyc" try to solve this issue. Another
big step could be done by removing rpaths from ELF generated with libtool.

Other thing known to break reproducibility:
  - use of lzop (it unconditionally include timestamps in result)
  - since gcc versions supporting SOURCE_DATE_EPOCH are not widely available,
    external toolchains probably won't work.
  - /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION,
    ccache is incompatible with reproducible
  - debug symbols are not reproducible

Since this feature is experimental I did not (yet) reported these
incompatibilities in menuconfig.

Gilles Chanteperdrix (3):
  reproducibility: generate SOURCE_DATE_EPOCH
  reproducibility/linux: override build timestamp
  reproducibility/busybox: disable build timestamps

J?r?me Pouiller (12):
  gcc6: honor SOURCE_DATE_EPOCH
  gcc5: honor SOURCE_DATE_EPOCH
  reproducible: add '-n' to gzip invocations
  fs/tar: make results reproducible
  reproducibility/linux: inhibit build-id
  reproducible: lock modification times in $TARGET_DIR
  fakedate: new package
  reproducible: enable fakedate
  python2: generate reproducible .pyc
  python3: generate reproducible .pyc
  python2: remove full path from .pyc
  python3: remove full path from .pyc

 Makefile                                           |   5 +
 fs/common.mk                                       |   3 +
 fs/tar/tar.mk                                      |   2 +-
 linux/linux.mk                                     |  15 ++
 package/busybox/busybox.mk                         |   6 +
 package/fakedate/fakedate                          |  28 +++
 package/fakedate/fakedate.mk                       |  14 ++
 .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch    | 213 +++++++++++++++++++
 .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch    | 234 +++++++++++++++++++++
 package/python/python.mk                           |  36 ++--
 package/python3/python3.mk                         |  35 +--
 support/dependencies/check-host-tar.sh             |   5 +-
 toolchain/toolchain/toolchain.mk                   |   4 +
 13 files changed, 570 insertions(+), 30 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk
 create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
 create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch

-- 
1.9.1

[Buildroot] [PATCH 01/15] gcc6: honor SOURCE_DATE_EPOCH

[Jérôme Pouiller]

Support $SOURCE_DATE_EPOCH is necessary to create reproducible build.
Unfortunately, it is not yet widely available. This patch add support for
SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 6.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch    | 234 +++++++++++++++++++++
 1 file changed, 234 insertions(+)
 create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch

diff --git a/package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch b/package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
new file mode 100644
index 0000000..3ed2e25
--- /dev/null
+++ b/package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
@@ -0,0 +1,234 @@
+From e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934 Mon Sep 17 00:00:00 2001
+From: doko <doko@138bc75d-0d04-0410-961f-82ee72b054a4>
+Date: Thu, 28 Apr 2016 09:12:05 +0000
+Subject: [PATCH] gcc/c-family/ChangeLog:
+
+Upstream status:
+  Accepted.
+  See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* c-common.c (get_source_date_epoch): New function, gets the environment
+	variable SOURCE_DATE_EPOCH and parses it as long long with error
+	handling.
+	* c-common.h (get_source_date_epoch): Prototype.
+	* c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch.
+
+gcc/ChangeLog:
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable.
+
+libcpp/ChangeLog:
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* include/cpplib.h (cpp_init_source_date_epoch): Prototype.
+	* init.c (cpp_init_source_date_epoch): New function.
+	* internal.h: Added source_date_epoch variable to struct
+	cpp_reader to store a reproducible date.
+	* macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from
+	pfile->source_date_epoch instead of localtime if source_date_epoch is
+	set, to be used for __DATE__ and __TIME__ macros to help reproducible
+	builds.
+
+Signed-off-by: Eduard Sanou <dhole@openmailbox.org>
+Signed-off-by: Matthias Klose <doko@debian.org>
+
+git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk at 235550 138bc75d-0d04-0410-961f-82ee72b054a4
+---
+ gcc/c-family/c-common.c |   33 +++++++++++++++++++++++++++++++++
+ gcc/c-family/c-common.h |    5 +++++
+ gcc/c-family/c-lex.c    |    3 +++
+ gcc/doc/cppenv.texi     |   17 +++++++++++++++++
+ libcpp/include/cpplib.h |    3 +++
+ libcpp/init.c           |    9 ++++++++-
+ libcpp/internal.h       |    4 ++++
+ libcpp/macro.c          |   21 ++++++++++++++-------
+
+diff --git a/gcc/c-family/c-common.c b/gcc/c-family/c-common.c
+index 1f0d76a..c086dee 100644
+--- a/gcc/c-family/c-common.c
++++ b/gcc/c-family/c-common.c
+@@ -12784,4 +12784,37 @@ valid_array_size_p (location_t loc, tree type, tree name)
+   return true;
+ }
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++time_t
++get_source_date_epoch ()
++{
++  char *source_date_epoch;
++  long long epoch;
++  char *endptr;
++
++  source_date_epoch = getenv ("SOURCE_DATE_EPOCH");
++  if (!source_date_epoch)
++    return (time_t) -1;
++
++  errno = 0;
++  epoch = strtoll (source_date_epoch, &endptr, 10);
++  if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
++      || (errno != 0 && epoch == 0))
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "strtoll: %s\n", xstrerror(errno));
++  if (endptr == source_date_epoch)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "no digits were found: %s\n", endptr);
++  if (*endptr != '\0')
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "trailing garbage: %s\n", endptr);
++  if (epoch < 0)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "value must be nonnegative: %lld \n", epoch);
++
++  return (time_t) epoch;
++}
++
+ #include "gt-c-family-c-common.h"
+diff --git a/gcc/c-family/c-common.h b/gcc/c-family/c-common.h
+index 1309549..3a7805f 100644
+--- a/gcc/c-family/c-common.h
++++ b/gcc/c-family/c-common.h
+@@ -1471,4 +1471,9 @@ extern bool valid_array_size_p (location_t, tree, tree);
+ extern bool cilk_ignorable_spawn_rhs_op (tree);
+ extern bool cilk_recognize_spawn (tree, tree *);
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++extern time_t get_source_date_epoch (void);
++
+ #endif /* ! GCC_C_COMMON_H */
+diff --git a/gcc/c-family/c-lex.c b/gcc/c-family/c-lex.c
+index 6b020a4..ff7eb25 100644
+--- a/gcc/c-family/c-lex.c
++++ b/gcc/c-family/c-lex.c
+@@ -388,6 +388,9 @@ c_lex_with_flags (tree *value, location_t *loc, unsigned char *cpp_flags,
+   enum cpp_ttype type;
+   unsigned char add_flags = 0;
+   enum overflow_type overflow = OT_NONE;
++  time_t source_date_epoch = get_source_date_epoch ();
++
++  cpp_init_source_date_epoch (parse_in, source_date_epoch);
+ 
+   timevar_push (TV_CPP);
+  retry:
+diff --git a/gcc/doc/cppenv.texi b/gcc/doc/cppenv.texi
+index 22c8cb3..e958e93 100644
+--- a/gcc/doc/cppenv.texi
++++ b/gcc/doc/cppenv.texi
+@@ -79,4 +79,21 @@ main input file is omitted.
+ @ifclear cppmanual
+ @xref{Preprocessor Options}.
+ @end ifclear
++
++ at item SOURCE_DATE_EPOCH
++
++If this variable is set, its value specifies a UNIX timestamp to be
++used in replacement of the current date and time in the @code{__DATE__}
++and @code{__TIME__} macros, so that the embedded timestamps become
++reproducible.
++
++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp,
++defined as the number of seconds (excluding leap seconds) since
++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of
++@samp{@command{date +%s}}.
++
++The value should be a known timestamp such as the last modification
++time of the source or package and it should be set by the build
++process.
++
+ @end vtable
+diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
+index 35b0375..4998b3a 100644
+--- a/libcpp/include/cpplib.h
++++ b/libcpp/include/cpplib.h
+@@ -784,6 +784,9 @@ extern void cpp_init_special_builtins (cpp_reader *);
+ /* Set up built-ins like __FILE__.  */
+ extern void cpp_init_builtins (cpp_reader *, int);
+ 
++/* Initialize the source_date_epoch value.  */
++extern void cpp_init_source_date_epoch (cpp_reader *, time_t);
++
+ /* This is called after options have been parsed, and partially
+    processed.  */
+ extern void cpp_post_options (cpp_reader *);
+diff --git a/libcpp/init.c b/libcpp/init.c
+index 4343075..f5ff85b 100644
+--- a/libcpp/init.c
++++ b/libcpp/init.c
+@@ -533,8 +533,15 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
+     _cpp_define_builtin (pfile, "__OBJC__ 1");
+ }
+ 
++/* Initialize the source_date_epoch value.  */
++void
++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch)
++{
++  pfile->source_date_epoch = source_date_epoch; 
++}
++
+ /* Sanity-checks are dependent on command-line options, so it is
+-   called as a subroutine of cpp_read_main_file ().  */
++   called as a subroutine of cpp_read_main_file.  */
+ #if CHECKING_P
+ static void sanity_checks (cpp_reader *);
+ static void sanity_checks (cpp_reader *pfile)
+diff --git a/libcpp/internal.h b/libcpp/internal.h
+index 9ce8707..e3eb26b 100644
+--- a/libcpp/internal.h
++++ b/libcpp/internal.h
+@@ -502,6 +502,10 @@ struct cpp_reader
+   const unsigned char *date;
+   const unsigned char *time;
+ 
++  /* Externally set timestamp to replace current date and time useful for
++     reproducibility.  */
++  time_t source_date_epoch;
++
+   /* EOF token, and a token forcing paste avoidance.  */
+   cpp_token avoid_paste;
+   cpp_token eof;
+diff --git a/libcpp/macro.c b/libcpp/macro.c
+index c251553..c2a8376 100644
+--- a/libcpp/macro.c
++++ b/libcpp/macro.c
+@@ -357,13 +357,20 @@ _cpp_builtin_macro_text (cpp_reader *pfile, cpp_hashnode *node,
+ 	  time_t tt;
+ 	  struct tm *tb = NULL;
+ 
+-	  /* (time_t) -1 is a legitimate value for "number of seconds
+-	     since the Epoch", so we have to do a little dance to
+-	     distinguish that from a genuine error.  */
+-	  errno = 0;
+-	  tt = time(NULL);
+-	  if (tt != (time_t)-1 || errno == 0)
+-	    tb = localtime (&tt);
++	  /* Set a reproducible timestamp for __DATE__ and __TIME__ macro
++	     usage if SOURCE_DATE_EPOCH is defined.  */
++	  if (pfile->source_date_epoch != (time_t) -1)
++	     tb = gmtime (&pfile->source_date_epoch);
++	  else
++	    {
++	      /* (time_t) -1 is a legitimate value for "number of seconds
++		 since the Epoch", so we have to do a little dance to
++		 distinguish that from a genuine error.  */
++	      errno = 0;
++	      tt = time (NULL);
++	      if (tt != (time_t)-1 || errno == 0)
++		tb = localtime (&tt);
++	    }
+ 
+ 	  if (tb)
+ 	    {
+-- 
+1.7.1
+
-- 
1.9.1

[Buildroot] [PATCH 02/15] gcc5: honor SOURCE_DATE_EPOCH

[Jérôme Pouiller]

Support $SOURCE_DATE_EPOCH is necessary to create reproducible build.
Unfortunately, it is not yet widely available. This patch add support for
SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 5.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch    | 213 +++++++++++++++++++++
 1 file changed, 213 insertions(+)
 create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch

diff --git a/package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch b/package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
new file mode 100644
index 0000000..0b644a7
--- /dev/null
+++ b/package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
@@ -0,0 +1,213 @@
+From: J?r?me Pouiller <jezz@sysmic.org>
+Date: Fri, 04 Nov 2016 11:45:07 +0100
+Subject: [PATCH] Honor SOURCE_DATE_EPOCH
+
+Upstream status:
+  Accepted.
+  See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934
+
+Current patch is backported from equivalent written for gcc 6.1
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* c-common.c (get_source_date_epoch): New function, gets the environment
+	variable SOURCE_DATE_EPOCH and parses it as long long with error
+	handling.
+	* c-common.h (get_source_date_epoch): Prototype.
+	* c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch.
+
+gcc/ChangeLog:
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable.
+
+libcpp/ChangeLog:
+
+2016-04-28  Eduard Sanou  <dhole@openmailbox.org>
+	    Matthias Klose  <doko@debian.org>
+
+	* include/cpplib.h (cpp_init_source_date_epoch): Prototype.
+	* init.c (cpp_init_source_date_epoch): New function.
+	* internal.h: Added source_date_epoch variable to struct
+	cpp_reader to store a reproducible date.
+	* macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from
+	pfile->source_date_epoch instead of localtime if source_date_epoch is
+	set, to be used for __DATE__ and __TIME__ macros to help reproducible
+	builds.
+
+Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
+
+---
+ gcc/c-family/c-common.c |   33 +++++++++++++++++++++++++++++++++
+ gcc/c-family/c-common.h |    6 ++++++
+ gcc/c-family/c-lex.c    |    3 +++
+ gcc/doc/cppenv.texi     |   17 +++++++++++++++++
+ libcpp/include/cpplib.h |    3 +++
+ libcpp/init.c           |    7 +++++++
+ libcpp/internal.h       |    4 ++++
+ libcpp/macro.c          |   21 ++++++++++++++-------
+ 8 files changed, 87 insertions(+), 7 deletions(-)
+
+--- a/gcc/c-family/c-common.c
++++ b/gcc/c-family/c-common.c
+@@ -12318,4 +12318,37 @@ pointer_to_zero_sized_aggr_p (tree t)
+   return (TYPE_SIZE (t) && integer_zerop (TYPE_SIZE (t)));
+ }
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++time_t
++get_source_date_epoch ()
++{
++  char *source_date_epoch;
++  long long epoch;
++  char *endptr;
++
++  source_date_epoch = getenv ("SOURCE_DATE_EPOCH");
++  if (!source_date_epoch)
++    return (time_t) -1;
++
++  errno = 0;
++  epoch = strtoll (source_date_epoch, &endptr, 10);
++  if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
++      || (errno != 0 && epoch == 0))
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "strtoll: %s\n", xstrerror(errno));
++  if (endptr == source_date_epoch)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "no digits were found: %s\n", endptr);
++  if (*endptr != '\0')
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "trailing garbage: %s\n", endptr);
++  if (epoch < 0)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "value must be nonnegative: %lld \n", epoch);
++
++  return (time_t) epoch;
++}
++
+ #include "gt-c-family-c-common.h"
+--- a/gcc/c-family/c-common.h
++++ b/gcc/c-family/c-common.h
+@@ -1437,4 +1437,10 @@ extern bool contains_cilk_spawn_stmt (tr
+ extern tree cilk_for_number_of_iterations (tree);
+ extern bool check_no_cilk (tree, const char *, const char *,
+ 		           location_t loc = UNKNOWN_LOCATION);
++
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++extern time_t get_source_date_epoch (void);
++
+ #endif /* ! GCC_C_COMMON_H */
+--- a/gcc/c-family/c-lex.c
++++ b/gcc/c-family/c-lex.c
+@@ -402,6 +402,9 @@ c_lex_with_flags (tree *value, location_
+   enum cpp_ttype type;
+   unsigned char add_flags = 0;
+   enum overflow_type overflow = OT_NONE;
++  time_t source_date_epoch = get_source_date_epoch ();
++
++  cpp_init_source_date_epoch (parse_in, source_date_epoch);
+ 
+   timevar_push (TV_CPP);
+  retry:
+--- a/gcc/doc/cppenv.texi
++++ b/gcc/doc/cppenv.texi
+@@ -79,4 +79,21 @@ main input file is omitted.
+ @ifclear cppmanual
+ @xref{Preprocessor Options}.
+ @end ifclear
++
++ at item SOURCE_DATE_EPOCH
++
++If this variable is set, its value specifies a UNIX timestamp to be
++used in replacement of the current date and time in the @code{__DATE__}
++and @code{__TIME__} macros, so that the embedded timestamps become
++reproducible.
++
++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp,
++defined as the number of seconds (excluding leap seconds) since
++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of
++@samp{@command{date +%s}}.
++
++The value should be a known timestamp such as the last modification
++time of the source or package and it should be set by the build
++process.
++
+ @end vtable
+--- a/libcpp/include/cpplib.h
++++ b/libcpp/include/cpplib.h
+@@ -775,6 +775,9 @@ extern void cpp_init_special_builtins (c
+ /* Set up built-ins like __FILE__.  */
+ extern void cpp_init_builtins (cpp_reader *, int);
+ 
++/* Initialize the source_date_epoch value.  */
++extern void cpp_init_source_date_epoch (cpp_reader *, time_t);
++
+ /* This is called after options have been parsed, and partially
+    processed.  */
+ extern void cpp_post_options (cpp_reader *);
+--- a/libcpp/init.c
++++ b/libcpp/init.c
+@@ -530,6 +530,13 @@ cpp_init_builtins (cpp_reader *pfile, in
+     _cpp_define_builtin (pfile, "__OBJC__ 1");
+ }
+ 
++/* Initialize the source_date_epoch value.  */
++void
++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch)
++{
++  pfile->source_date_epoch = source_date_epoch; 
++}
++
+ /* Sanity-checks are dependent on command-line options, so it is
+    called as a subroutine of cpp_read_main_file ().  */
+ #if ENABLE_CHECKING
+--- a/libcpp/internal.h
++++ b/libcpp/internal.h
+@@ -502,6 +502,10 @@ struct cpp_reader
+   const unsigned char *date;
+   const unsigned char *time;
+ 
++  /* Externally set timestamp to replace current date and time useful for
++     reproducibility.  */
++  time_t source_date_epoch;
++
+   /* EOF token, and a token forcing paste avoidance.  */
+   cpp_token avoid_paste;
+   cpp_token eof;
+--- a/libcpp/macro.c
++++ b/libcpp/macro.c
+@@ -350,13 +350,20 @@ _cpp_builtin_macro_text (cpp_reader *pfi
+ 	  time_t tt;
+ 	  struct tm *tb = NULL;
+ 
+-	  /* (time_t) -1 is a legitimate value for "number of seconds
+-	     since the Epoch", so we have to do a little dance to
+-	     distinguish that from a genuine error.  */
+-	  errno = 0;
+-	  tt = time(NULL);
+-	  if (tt != (time_t)-1 || errno == 0)
+-	    tb = localtime (&tt);
++	  /* Set a reproducible timestamp for __DATE__ and __TIME__ macro
++	     usage if SOURCE_DATE_EPOCH is defined.  */
++	  if (pfile->source_date_epoch != (time_t) -1)
++	     tb = gmtime (&pfile->source_date_epoch);
++	  else
++	    {
++	      /* (time_t) -1 is a legitimate value for "number of seconds
++		 since the Epoch", so we have to do a little dance to
++		 distinguish that from a genuine error.  */
++	      errno = 0;
++	      tt = time (NULL);
++	      if (tt != (time_t)-1 || errno == 0)
++		tb = localtime (&tt);
++	    }
+ 
+ 	  if (tb)
+ 	    {
-- 
1.9.1

[Buildroot] [PATCH 03/15] reproducibility: generate SOURCE_DATE_EPOCH

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
environment variable which contains either the date of Buildroot last
commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been change, even though some of them
may not have changed in fact.

The meaning of SOURCE_DATE_EPOCH is specified by the following
specification:
  https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index eff814b..17e49e4 100644
--- a/Makefile
+++ b/Makefile
@@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
+SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
1.9.1

[Buildroot] [PATCH 04/15] reproducible: add '-n' to gzip invocations

[Jérôme Pouiller]

Default invocation to gzip include timestamp in output file. This feature is
incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.

The environment variable GZIP can hold a set of default options for gzip. So
instead to find all gzip invocation in build process, we just export 'GZIP=-n'.

Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop
include timestamp and does not provide any way to disable it.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 17e49e4..c21ba4a 100644
--- a/Makefile
+++ b/Makefile
@@ -249,6 +249,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export GZIP=-n
 export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
-- 
1.9.1

[Buildroot] [PATCH 05/15] fs/tar: make results reproducible

[Jérôme Pouiller]

In order to make tar images reproducible, we use --sort flag. However,
this flags is available only from tar 1.28. So we also bump necessary
host-tar version.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 fs/tar/tar.mk                          | 2 +-
 support/dependencies/check-host-tar.sh | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
index 11c69c5..5a1b263 100644
--- a/fs/tar/tar.mk
+++ b/fs/tar/tar.mk
@@ -7,7 +7,7 @@
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 define ROOTFS_TAR_CMD
-	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
+	tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) .
 endef
 
 $(eval $(call ROOTFS_TARGET,tar))
diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
index 932d3c4..cef6d82 100755
--- a/support/dependencies/check-host-tar.sh
+++ b/support/dependencies/check-host-tar.sh
@@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then
   minor=0
 fi
 
-# Minimal version = 1.17 (previous versions do not correctly unpack archives
-# containing hard-links if the --strip-components option is used).
+# Minimal version = 1.28 (previous versions do not does not support --sort=name)
 major_min=1
-minor_min=17
+minor_min=28
 if [ $major -gt $major_min ]; then
 	echo $tar
 else
-- 
1.9.1

[Buildroot] [PATCH 06/15] reproducibility/linux: override build timestamp

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Linux kernel include a few information about build environment in its binary.
This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build
information when BR2_REPRODUCIBLE is enabled.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 988427c..7e826cc 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
-- 
1.9.1

[Buildroot] [PATCH 07/15] reproducibility/linux: inhibit build-id

[Jérôme Pouiller]

By default, Linux kernel enable 'build-id'. 'build-id' tends to add random
bytes in section .notes of kernel image[1]:

  $ readelf -Wn .../vmlinux
  Displaying notes found at file offset 0x00008000 with length 0x00000024:
    Owner                 Data size       Description
    GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2

Therefore, we patch kernel Makefile to disable 'build-id'.

[1] https://kernelnewbies.org/BuildId

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 linux/linux.mk | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 7e826cc..a63d1f3 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define LINUX_REMOVE_BUILD_ID
+	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
+endif
+
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
 LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
-- 
1.9.1

[Buildroot] [PATCH 08/15] reproducibility/busybox: disable build timestamps

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Busybox include a few information about build environment in its binary. This
feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build
information when BR2_REPRODUCIBLE is enabled.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index fc23a90..f4a241d 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
-- 
1.9.1

[Buildroot] [PATCH 09/15] reproducible: lock modification times in $TARGET_DIR

[Jérôme Pouiller]

Make sure all files in $TARGET_DIR has a defined modification time before to
generate filesystems.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 fs/common.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 2dbef4d..981dcb1 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -95,6 +95,9 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+ifeq ($$(BR2_REPRODUCIBLE),y)
+	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
+endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
-- 
1.9.1

[Buildroot] [PATCH 10/15] fakedate: new package

[Jérôme Pouiller]

`date' is widely used by packages to include build information in their
binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

Instead to find all `date' invocation in build process, we add small tool
allowing to alway return same date.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/fakedate/fakedate    | 28 ++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 14 ++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
new file mode 100755
index 0000000..2eded22
--- /dev/null
+++ b/package/fakedate/fakedate
@@ -0,0 +1,28 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# Licence: GPL
+# Created: 2016-11-04 16:31:18+01:00
+# Main authors:
+#     - J?r?me Pouiller <jezz@sysmic.org>
+#
+
+PATH=/bin:/usr/bin
+LOG=/dev/null
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    INHIBIT=0
+    for i in "$@"; do
+        case $i in
+        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
+            INHIBIT=1
+            ;;
+        esac
+    done
+    if [ $INHIBIT -eq 0 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec date "$@"
diff --git a/package/fakedate/fakedate.mk b/package/fakedate/fakedate.mk
new file mode 100644
index 0000000..e81ce5d
--- /dev/null
+++ b/package/fakedate/fakedate.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))
-- 
1.9.1

[Buildroot] [PATCH 11/15] reproducible: enable fakedate

[Jérôme Pouiller]

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 toolchain/toolchain/toolchain.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index d317e91..b88dd94 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -10,6 +10,10 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TOOLCHAIN_DEPENDENCIES += host-fakedate
+endif
+
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
 # Apply a hack that Rick Felker suggested[1] to avoid conflicts between libc
-- 
1.9.1

[Buildroot] [PATCH 12/15] python2: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time,
we remove .pyc before to compile. However, I wonder if it wouldn't be simpler
to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python/python.mk | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index cc65376..b0ff1fd 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -226,6 +226,26 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+# However, do not remove .pyc if source .py is not present.
+ifneq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_REPRODUCIBLE)),)
+define PYTHON_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
@@ -245,16 +265,6 @@ endef
 PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
-define PYTHON_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .pyo files
 define PYTHON_REMOVE_PYO_FILES
 	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyo' -print0 | \
-- 
1.9.1

[Buildroot] [PATCH 13/15] python3: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time,
we remove .pyc before to compile. However, I wonder if it wouldn't be simpler
to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python3/python3.mk | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index b3f31c0..158c29c 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -219,6 +219,25 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
+define PYTHON3_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
@@ -238,16 +257,6 @@ endef
 PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
-define PYTHON3_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .opt-1.pyc and
 # .opt-2.pyc files, since they can't work without their non-optimized
 # variant.
-- 
1.9.1

[Buildroot] [PATCH 14/15] python2: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch change the way
`pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python/python.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index b0ff1fd..35f971e 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -248,9 +248,9 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH 15/15] python3: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch change the way
`pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python3/python3.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 158c29c..3b2dd31 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -240,9 +240,9 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH 00/15] Reproducible builds

[Thomas Petazzoni]

Hello,

On Thu, 17 Nov 2016 11:00:24 +0100, J?r?me Pouiller wrote:
> This series try to continue work initiated by Gilles Chanteperdrix:
>   http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
>   http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

Thanks for taking over this effort. This is definitely interesting.

> Other thing known to break reproducibility:
>   - use of lzop (it unconditionally include timestamps in result)

I guess we can patch lzop to avoid this issue, right?

>   - since gcc versions supporting SOURCE_DATE_EPOCH are not widely available,
>     external toolchains probably won't work.

Instead of patching gcc, can we solve the problem in the toolchain
wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__
by passing -D__DATE__=... -D__TIME__=... to gcc ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/15] Reproducible builds

[Jérôme Pouiller]

On 2016-11-17 12:13, Thomas Petazzoni wrote:
> On Thu, 17 Nov 2016 11:00:24 +0100, J?r?me Pouiller wrote:
[...]
>> Other thing known to break reproducibility:
>>   - use of lzop (it unconditionally include timestamps in result)
> 
> I guess we can patch lzop to avoid this issue, right?

I guess also.


>>   - since gcc versions supporting SOURCE_DATE_EPOCH are not widely 
>> available,
>>     external toolchains probably won't work.
> 
> Instead of patching gcc, can we solve the problem in the toolchain
> wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__
> by passing -D__DATE__=... -D__TIME__=... to gcc ?

Yes, I can switch to this solution.

-- 
J?r?me Pouiller

[Buildroot] [PATCH 00/15] Reproducible builds

[Thomas Petazzoni]

Hello,

On Thu, 17 Nov 2016 14:17:41 +0100, J?r?me Pouiller wrote:

> >>   - since gcc versions supporting SOURCE_DATE_EPOCH are not widely 
> >> available,
> >>     external toolchains probably won't work.  
> > 
> > Instead of patching gcc, can we solve the problem in the toolchain
> > wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__
> > by passing -D__DATE__=... -D__TIME__=... to gcc ?  
> 
> Yes, I can switch to this solution.

I think it's a reasonable use of the wrapper, and makes the thing work
fine for external toolchain as well, which is nice.

Another (unrelated) question: how can we test this stuff? You're
providing the initial steps for it, but at some point, we will want to
validate which packages behave properly and which packages don't behave
properly. Have you already thought of automated testing we can do (in
the autobuilders perhaps?) to exercise this "reproducible builds"
functionality? Should we from time to time do a given builds twice,
each time in a different environment (different date, different time,
different user, different path, different timezone, different locale,
etc.) ?

To what extent do we try to be reproducible? Do we try to have
something that produces the same result when executed multiple times,
but on the same machine, in the same environment (i.e only date/time
changes) ? Or do we try to go the point where a given Buildroot .config
gives a byte identical output regardless of the host machine on which
you do the build? It would be great to know what our definition of
"reproducible is", and document it in the help text of the
BR2_REPRODUCIBLE option.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/15] Reproducible builds

[Jérôme Pouiller]

On 2016-11-17 15:08, Thomas Petazzoni wrote:
[...]
> Another (unrelated) question: how can we test this stuff? You're
> providing the initial steps for it, but at some point, we will want to
> validate which packages behave properly and which packages don't behave
> properly. Have you already thought of automated testing we can do (in
> the autobuilders perhaps?) to exercise this "reproducible builds"
> functionality? Should we from time to time do a given builds twice,
> each time in a different environment (different date, different time,
> different user, different path, different timezone, different locale,
> etc.) ?

In a first time, compiling each configuration twice, diffing 
$(TARGET_DIR)
and find guilty packages using packages-file-list.txt should be enough 
to find
a many errors. We may also grep $TARGET_DIR for most common error 
patterns
(hostname, user, ...)

In a second time, I think reproducible build is an excellent tool to 
validate
top level parallelization.

Finally, for more accurate checks, we need to compile same configuration 
on
different host. I didn't check autobuilder sources, but it seems it 
would
involve a new autobuilder design.


> To what extent do we try to be reproducible? Do we try to have
> something that produces the same result when executed multiple times,
> but on the same machine, in the same environment (i.e only date/time
> changes) ? Or do we try to go the point where a given Buildroot .config
> gives a byte identical output regardless of the host machine on which
> you do the build? It would be great to know what our definition of
> "reproducible is", and document it in the help text of the
> BR2_REPRODUCIBLE option.

Ideally, "reproducible" mean that a binary result only depends on 
Buildroot
version and user configuration (sure, if a custom script have a 
non-reproducible
behavior, result won't be reproducible).
Currently, there are few known (and doubtless many unknown) bugs:
    - build paths have to be the sames
    - lzop shouldn't be used

BR,

-- 
J?r?me Pouiller

[Buildroot] [PATCH 00/15] Reproducible builds

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 09:49:51 +0100, J?r?me Pouiller wrote:

> In a first time, compiling each configuration twice, diffing 
> $(TARGET_DIR)
> and find guilty packages using packages-file-list.txt should be enough 
> to find
> a many errors. We may also grep $TARGET_DIR for most common error 
> patterns
> (hostname, user, ...)

OK. So for now we concentrate on timestamp-related variations.

> In a second time, I think reproducible build is an excellent tool to 
> validate top level parallelization.

Yes, right.

> Finally, for more accurate checks, we need to compile same configuration 
> on different host. I didn't check autobuilder sources, but it seems it 
> would involve a new autobuilder design.

This is clearly not possible with the current autobuilder design,
indeed. It will require a fairly major rework. However, we are already
thinking of changing the autobuilder architecture to have the
configuration generated not by the build slaves themselves, but instead
by the server. This could be a first step towards having the server
generate a given configuration, and give it to two separate
autobuilder slaves. But it's a lot of work to do.

> Ideally, "reproducible" mean that a binary result only depends on 
> Buildroot version and user configuration (sure, if a custom script have a 
> non-reproducible behavior, result won't be reproducible).

Right. I believe we should document in the BR2_REPRODUCIBLE help text
what is our current definition of reproducible. I.e that for the moment
we only make builds reproducibles if they are built on the same
machine, in the same path.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com