[Buildroot] [PATCH v2 00/15] Reproducible builds

[Buildroot] [PATCH v2 00/15] Reproducible builds

[Jérôme Pouiller]

This series try to continue work initiated by Gilles Chanteperdrix:
  http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
  http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
  - I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
  - I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
  - They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

This version focuses on timestamps. It provide good enough results as soon as
OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files.
Only patch called "remove full path from .pyc" try to solve this issue. Another
big step could be done by removing rpaths from ELF generated with libtool.

Other thing known to break reproducibility:
  - use of lzop (it unconditionally include timestamps in result)
  - /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION,
    ccache is incompatible with reproducible
  - debug symbols are not reproducible

Since this feature is experimental I did not (yet) reported these
incompatibilities in menuconfig.

v2:
  - overload __TIME__ and __DATE__ instead of patching gcc
  - improve BR2_REPRODUCIBLE help text

Gilles Chanteperdrix (3):
  reproducibility: generate SOURCE_DATE_EPOCH
  reproducibility/linux: override build timestamp
  reproducibility/busybox: disable build timestamps

J?r?me Pouiller (12):
  reproducible: fix DATE/TIME macros in toolchain-wrapper
  reproducible: add '-n' to gzip invocations
  fs/tar: make results reproducible
  reproducibility/linux: inhibit build-id
  reproducible: lock modification times in $TARGET_DIR
  fakedate: new package
  reproducible: enable fakedate
  python2: generate reproducible .pyc
  python3: generate reproducible .pyc
  python2: remove full path from .pyc
  python3: remove full path from .pyc
  reproducible: improve help text

 Config.in                              |  5 +++++
 Makefile                               |  5 +++++
 fs/common.mk                           |  3 +++
 fs/tar/tar.mk                          |  2 +-
 linux/linux.mk                         | 15 ++++++++++++++
 package/busybox/busybox.mk             |  6 ++++++
 package/fakedate/fakedate              | 28 ++++++++++++++++++++++++++
 package/fakedate/fakedate.mk           | 14 +++++++++++++
 package/python/python.mk               | 36 ++++++++++++++++++++++------------
 package/python3/python3.mk             | 35 +++++++++++++++++++++------------
 support/dependencies/check-host-tar.sh |  5 ++---
 toolchain/toolchain-wrapper.c          |  9 +++++++++
 toolchain/toolchain-wrapper.mk         |  5 +++++
 toolchain/toolchain/toolchain.mk       |  4 ++++
 14 files changed, 142 insertions(+), 30 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

-- 
1.9.1

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
environment variable which contains either the date of Buildroot last
commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been change, even though some of them
may not have changed in fact.

The meaning of SOURCE_DATE_EPOCH is specified by the following
specification:
  https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index eff814b..17e49e4 100644
--- a/Makefile
+++ b/Makefile
@@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
+SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
1.9.1

[Buildroot] [PATCH v2 02/15] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

The use __DATE__ and __TIME__ are one of most common sources of
non-reproducible binaries. We take advantage of toolchain-wrapper to fix
these macros.

Notice this workaround will be useless this next gcc versions since it will
include suport for SOURCE_DATE_EPOCH variable.

Note: quoting of $(TOOLCHAIN_WRAPPER_ARGS) is a mess, but I did not find easier
way to do (without involving `$(shell ...)')

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 toolchain/toolchain-wrapper.c  | 9 +++++++++
 toolchain/toolchain-wrapper.mk | 5 +++++
 2 files changed, 14 insertions(+)

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index 925d013..28709ec 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -75,6 +75,15 @@ static char *predef_args[] = {
 #if defined(BR_MIPS_TARGET_BIG_ENDIAN) || defined(BR_ARC_TARGET_BIG_ENDIAN)
 	"-EB",
 #endif
+#if defined(BR_REPRODUCIBLE_TIME) || defined(BR_REPRODUCIBLE_DATE)
+	"-Wno-builtin-macro-redefined",
+#endif
+#ifdef BR_REPRODUCIBLE_TIME
+	"-D__TIME__=" BR_REPRODUCIBLE_TIME,
+#endif
+#ifdef BR_REPRODUCIBLE_DATE
+	"-D__DATE__=" BR_REPRODUCIBLE_DATE,
+#endif
 #ifdef BR_ADDITIONAL_CFLAGS
 	BR_ADDITIONAL_CFLAGS
 #endif
diff --git a/toolchain/toolchain-wrapper.mk b/toolchain/toolchain-wrapper.mk
index e7aa5fb..7067134 100644
--- a/toolchain/toolchain-wrapper.mk
+++ b/toolchain/toolchain-wrapper.mk
@@ -30,6 +30,11 @@ ifeq ($(BR2_CCACHE_USE_BASEDIR),y)
 TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_BASEDIR='"$(BASE_DIR)"'
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_TIME="\"\\\"`date -d @$(SOURCE_DATE_EPOCH) "+%T"`\\\"\""
+TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_DATE="\"\\\"`date -d @$(SOURCE_DATE_EPOCH) "+%b %e %Y"`\\\"\""
+endif
+
 define TOOLCHAIN_WRAPPER_BUILD
 	$(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \
 		-s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \
-- 
1.9.1

[Buildroot] [PATCH v2 03/15] reproducible: add '-n' to gzip invocations

[Jérôme Pouiller]

Default invocation to gzip include timestamp in output file. This feature is
incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.

The environment variable GZIP can hold a set of default options for gzip. So
instead to find all gzip invocation in build process, we just export 'GZIP=-n'.

Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop
include timestamp and does not provide any way to disable it.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 17e49e4..c21ba4a 100644
--- a/Makefile
+++ b/Makefile
@@ -249,6 +249,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export GZIP=-n
 export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
-- 
1.9.1

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Jérôme Pouiller]

In order to make tar images reproducible, we use --sort flag. However,
this flags is available only from tar 1.28. So we also bump necessary
host-tar version.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 fs/tar/tar.mk                          | 2 +-
 support/dependencies/check-host-tar.sh | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
index 11c69c5..5a1b263 100644
--- a/fs/tar/tar.mk
+++ b/fs/tar/tar.mk
@@ -7,7 +7,7 @@
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 define ROOTFS_TAR_CMD
-	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
+	tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) .
 endef
 
 $(eval $(call ROOTFS_TARGET,tar))
diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
index 932d3c4..cef6d82 100755
--- a/support/dependencies/check-host-tar.sh
+++ b/support/dependencies/check-host-tar.sh
@@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then
   minor=0
 fi
 
-# Minimal version = 1.17 (previous versions do not correctly unpack archives
-# containing hard-links if the --strip-components option is used).
+# Minimal version = 1.28 (previous versions do not does not support --sort=name)
 major_min=1
-minor_min=17
+minor_min=28
 if [ $major -gt $major_min ]; then
 	echo $tar
 else
-- 
1.9.1

[Buildroot] [PATCH v2 05/15] reproducibility/linux: override build timestamp

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Linux kernel include a few information about build environment in its binary.
This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build
information when BR2_REPRODUCIBLE is enabled.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 988427c..7e826cc 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
-- 
1.9.1

[Buildroot] [PATCH v2 06/15] reproducibility/linux: inhibit build-id

[Jérôme Pouiller]

By default, Linux kernel enable 'build-id'. 'build-id' tends to add random
bytes in section .notes of kernel image[1]:

  $ readelf -Wn .../vmlinux
  Displaying notes found at file offset 0x00008000 with length 0x00000024:
    Owner                 Data size       Description
    GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2

Therefore, we patch kernel Makefile to disable 'build-id'.

[1] https://kernelnewbies.org/BuildId

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 linux/linux.mk | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 7e826cc..a63d1f3 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define LINUX_REMOVE_BUILD_ID
+	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
+endif
+
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
 LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
-- 
1.9.1

[Buildroot] [PATCH v2 07/15] reproducibility/busybox: disable build timestamps

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Busybox include a few information about build environment in its binary. This
feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build
information when BR2_REPRODUCIBLE is enabled.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index fc23a90..f4a241d 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
-- 
1.9.1

[Buildroot] [PATCH v2 08/15] reproducible: lock modification times in $TARGET_DIR

[Jérôme Pouiller]

Make sure all files in $TARGET_DIR has a defined modification time before to
generate filesystems.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 fs/common.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 2dbef4d..981dcb1 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -95,6 +95,9 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+ifeq ($$(BR2_REPRODUCIBLE),y)
+	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
+endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
-- 
1.9.1

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Jérôme Pouiller]

`date' is widely used by packages to include build information in their
binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

Instead to find all `date' invocation in build process, we add small tool
allowing to alway return same date.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/fakedate/fakedate    | 28 ++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 14 ++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
new file mode 100755
index 0000000..2eded22
--- /dev/null
+++ b/package/fakedate/fakedate
@@ -0,0 +1,28 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# Licence: GPL
+# Created: 2016-11-04 16:31:18+01:00
+# Main authors:
+#     - J?r?me Pouiller <jezz@sysmic.org>
+#
+
+PATH=/bin:/usr/bin
+LOG=/dev/null
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    INHIBIT=0
+    for i in "$@"; do
+        case $i in
+        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
+            INHIBIT=1
+            ;;
+        esac
+    done
+    if [ $INHIBIT -eq 0 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec date "$@"
diff --git a/package/fakedate/fakedate.mk b/package/fakedate/fakedate.mk
new file mode 100644
index 0000000..e81ce5d
--- /dev/null
+++ b/package/fakedate/fakedate.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))
-- 
1.9.1

[Buildroot] [PATCH v2 10/15] reproducible: enable fakedate

[Jérôme Pouiller]

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 toolchain/toolchain/toolchain.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index d317e91..b88dd94 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -10,6 +10,10 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TOOLCHAIN_DEPENDENCIES += host-fakedate
+endif
+
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
 # Apply a hack that Rick Felker suggested[1] to avoid conflicts between libc
-- 
1.9.1

[Buildroot] [PATCH v2 11/15] python2: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time,
we remove .pyc before to compile. However, I wonder if it wouldn't be simpler
to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python/python.mk | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index cc65376..b0ff1fd 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -226,6 +226,26 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+# However, do not remove .pyc if source .py is not present.
+ifneq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_REPRODUCIBLE)),)
+define PYTHON_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
@@ -245,16 +265,6 @@ endef
 PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
-define PYTHON_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .pyo files
 define PYTHON_REMOVE_PYO_FILES
 	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyo' -print0 | \
-- 
1.9.1

[Buildroot] [PATCH v2 12/15] python3: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time,
we remove .pyc before to compile. However, I wonder if it wouldn't be simpler
to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python3/python3.mk | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index b3f31c0..158c29c 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -219,6 +219,25 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
+define PYTHON3_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
@@ -238,16 +257,6 @@ endef
 PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
-define PYTHON3_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .opt-1.pyc and
 # .opt-2.pyc files, since they can't work without their non-optimized
 # variant.
-- 
1.9.1

[Buildroot] [PATCH v2 13/15] python2: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch change the way
`pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python/python.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index b0ff1fd..35f971e 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -248,9 +248,9 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v2 14/15] python3: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch change the way
`pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/python3/python3.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 158c29c..3b2dd31 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -240,9 +240,9 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v2 15/15] reproducible: improve help text

[Jérôme Pouiller]

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Config.in | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Config.in b/Config.in
index 5cf0c4d..2a81202 100644
--- a/Config.in
+++ b/Config.in
@@ -707,6 +707,11 @@ config BR2_REPRODUCIBLE
 	  this allows to generate exactly identical binaries from one
 	  build to the other, including on different machines.
 
+	  Some restrictions are known on current implementation:
+	    - Build paths ($OUTDIR and $TOPDIR) have to be the sames for all
+	      builds
+	    - Use of lzop is incompatible
+
 	  This is labeled as an experimental feature, as not all
 	  packages behave properly to ensure reproducibility.
 
-- 
1.9.1

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:10 +0100, J?r?me Pouiller wrote:

> diff --git a/Makefile b/Makefile
> index eff814b..17e49e4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ=UTC
>  export LANG=C
>  export LC_ALL=C
> +export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)

Why do you do this instead of just doing

export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))

> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)

Why don't we change the way BR2_VERSION is defined in order to avoid having to do this?

BR2_VERSION_DATE = 2016.11
BR2_VERSION = $(BR2_VERSION)-rc1

for example.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 02/15] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:11 +0100, J?r?me Pouiller wrote:

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_TIME="\"\\\"`date -d @$(SOURCE_DATE_EPOCH) "+%T"`\\\"\""
> +TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_DATE="\"\\\"`date -d @$(SOURCE_DATE_EPOCH) "+%b %e %Y"`\\\"\""
> +endif

Do we need to encode the dates in the wrapper, or should we have the
wrapper read at runtime the SOURCE_DATE_EPOCH environment variable?

I believe the latter is simpler, and matches better what future gcc
versions will do.

The only drawback is that people using the toolchain (through the
wrapper) outside of Buildroot will not benefit from the "fixed" date,
unless they define SOURCE_DATE_EPOCH in their environment. But I
believe this is OK, because this is anyway the behavior that they would
get by using the newest gcc versions that have this feature.

It would make all the quoting mess a bit simpler :)

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 03/15] reproducible: add '-n' to gzip invocations

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:12 +0100, J?r?me Pouiller wrote:
> Default invocation to gzip include timestamp in output file. This feature is
> incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
> 
> The environment variable GZIP can hold a set of default options for gzip. So
> instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
> 
> Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop
> include timestamp and does not provide any way to disable it.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

If nobody complains about this patch, I'm going to apply it to the next
branch soon.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:13 +0100, J?r?me Pouiller wrote:

> -# Minimal version = 1.17 (previous versions do not correctly unpack archives
> -# containing hard-links if the --strip-components option is used).
> +# Minimal version = 1.28 (previous versions do not does not support --sort=name)
>  major_min=1
> -minor_min=17
> +minor_min=28

Unfortunately, this version dependency is not really acceptable I'm
afraid. Ubuntu 14.04, which is really not that old, only has tar 1.27.

And the old Debian 6.0 that I have for the autobuilders to test "old
systems" uses tar 1.23.

Can we find a better way?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 05/15] reproducibility/linux: override build timestamp

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:14 +0100, J?r?me Pouiller wrote:

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +LINUX_MAKE_ENV += \
> +	KBUILD_BUILD_VERSION=1 \
> +	KBUILD_BUILD_USER=buildroot \
> +	KBUILD_BUILD_HOST=buildroot \
> +	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"

With your "fakedate" script, do we really need this last line?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 07/15] reproducibility/busybox: disable build timestamps

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:16 +0100, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> Busybox include a few information about build environment in its binary. This
> feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build
> information when BR2_REPRODUCIBLE is enabled.
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

I will apply this one to the next branch soon, unless someone complains.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 08/15] reproducible: lock modification times in $TARGET_DIR

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:17 +0100, J?r?me Pouiller wrote:
> Make sure all files in $TARGET_DIR has a defined modification time before to

has -> have

> generate filesystems.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Same: I'll apply soon if nobody complains.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:18 +0100, J?r?me Pouiller wrote:
> `date' is widely used by packages to include build information in their
> binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.
> 
> Instead to find all `date' invocation in build process, we add small tool
> allowing to alway return same date.

Instead of having to identify all `date' invocations in the different
packages, this commit adds a small tool that allows to always return
the same date.

> +PATH=/bin:/usr/bin

It is not really nice to override the PATH. I guess you want to remove
$(HOST_DIR)/usr/bin from the PATH to not call yourself recursively, but
I think we should do better than assuming /bin:/usr/bin is OK.

> +LOG=/dev/null

This variable is used by?

> +if [ -n "$SOURCE_DATE_EPOCH" ]; then
> +    INHIBIT=0
> +    for i in "$@"; do
> +        case $i in
> +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
> +            INHIBIT=1
> +            ;;
> +        esac
> +    done
> +    if [ $INHIBIT -eq 0 ]; then
> +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
> +        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
> +        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
> +    fi
> +fi
> +
> +exec date "$@"

Could you explain a bit the logic here?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 10/15] reproducible: enable fakedate

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 10:10:19 +0100, J?r?me Pouiller wrote:

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +TOOLCHAIN_DEPENDENCIES += host-fakedate
> +endif

Unfortunately, this means that fakedate is only installed at the end of
the toolchain build process. So the entire toolchain is built without
fakedate installed.

Is it because we assume that the toolchain build process is safe (i.e
it doesn't call "date") ?

Perhaps it should be added in DEPENDENCIES_HOST_PREREQ instead ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Jérôme Pouiller]

On 2016-11-18 12:40, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 10:10:13 +0100, J?r?me Pouiller wrote:
> 
>> -# Minimal version = 1.17 (previous versions do not correctly unpack 
>> archives
>> -# containing hard-links if the --strip-components option is used).
>> +# Minimal version = 1.28 (previous versions do not does not support 
>> --sort=name)
>>  major_min=1
>> -minor_min=17
>> +minor_min=28
> 
> Unfortunately, this version dependency is not really acceptable I'm
> afraid. Ubuntu 14.04, which is really not that old, only has tar 1.27.
> 
> And the old Debian 6.0 that I have for the autobuilders to test "old
> systems" uses tar 1.23.
> 
> Can we find a better way?

My workstation only have tar 1.27 installed. When tar version is wrong, 
compilation
does not fail. Instead, host-tar is compiled before any other packages 
(see
support/dependencies/check-host-tar.mk)

--
J?r?me Pouiller

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 14:02:58 +0100, J?r?me Pouiller wrote:

> My workstation only have tar 1.27 installed. When tar version is wrong, 
> compilation does not fail. Instead, host-tar is compiled before any
> other packages (see support/dependencies/check-host-tar.mk)

Yes, I know, but it's a pain to force everyone using an old distro to
build host-tar. Maybe we should make this conditional on
BR2_REPRODUCIBLE?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Jérôme Pouiller]

On 2016-11-18 14:29, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 14:02:58 +0100, J?r?me Pouiller wrote:
> 
>> My workstation only have tar 1.27 installed. When tar version is 
>> wrong,
>> compilation does not fail. Instead, host-tar is compiled before any
>> other packages (see support/dependencies/check-host-tar.mk)
> 
> Yes, I know, but it's a pain to force everyone using an old distro to
> build host-tar. Maybe we should make this conditional on
> BR2_REPRODUCIBLE?

Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months 
(maybe
simultaneous with next Buildroot version).

So, IMHO it not justified to add a dirty condition in check-host-tar.sh.

--
J?r?me Pouiller

[Buildroot] [PATCH v2 02/15] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

On 2016-11-18 12:37, Thomas Petazzoni wrote:
> On Fri, 18 Nov 2016 10:10:11 +0100, J?r?me Pouiller wrote:
> 
>> +ifeq ($(BR2_REPRODUCIBLE),y)
>> +TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_TIME="\"\\\"`date -d 
>> @$(SOURCE_DATE_EPOCH) "+%T"`\\\"\""
>> +TOOLCHAIN_WRAPPER_ARGS += -DBR_REPRODUCIBLE_DATE="\"\\\"`date -d 
>> @$(SOURCE_DATE_EPOCH) "+%b %e %Y"`\\\"\""
>> +endif
> 
> Do we need to encode the dates in the wrapper, or should we have the
> wrapper read at runtime the SOURCE_DATE_EPOCH environment variable?
> 
> I believe the latter is simpler, and matches better what future gcc
> versions will do.
> 
> The only drawback is that people using the toolchain (through the
> wrapper) outside of Buildroot will not benefit from the "fixed" date,
> unless they define SOURCE_DATE_EPOCH in their environment. But I
> believe this is OK, because this is anyway the behavior that they would
> get by using the newest gcc versions that have this feature.

Agreed.

--
J?r?me Pouiller

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Jérôme Pouiller]

On 2016-11-18 12:33, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 10:10:10 +0100, J?r?me Pouiller wrote:
> 
>> diff --git a/Makefile b/Makefile
>> index eff814b..17e49e4 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>>  export TZ=UTC
>>  export LANG=C
>>  export LC_ALL=C
>> +export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
> 
> Why do you do this instead of just doing
> 
> export SOURCE_DATE_EPOCH = $(if $(wildcard
> $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))

I wanted keep all exported variables together, but ok.


>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 
>> --format=%at)
>> +SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 
>> 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
> 
> Why don't we change the way BR2_VERSION is defined in order to avoid
> having to do this?
> 
> BR2_VERSION_DATE = 2016.11
> BR2_VERSION = $(BR2_VERSION)-rc1
> 
> for example.

OK

-- 
J?r?me Pouiller

[Buildroot] [PATCH v2 10/15] reproducible: enable fakedate

[Jérôme Pouiller]

On 2016-11-18 12:49, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 10:10:19 +0100, J?r?me Pouiller wrote:
> 
>> +ifeq ($(BR2_REPRODUCIBLE),y)
>> +TOOLCHAIN_DEPENDENCIES += host-fakedate
>> +endif
> 
> Unfortunately, this means that fakedate is only installed at the end of
> the toolchain build process. So the entire toolchain is built without
> fakedate installed.
> 
> Is it because we assume that the toolchain build process is safe (i.e
> it doesn't call "date") ?

During my tests, fakedate was build before host-gcc-initial. Indeed, I
was lucky (in add, I think toolchain build process is safe).


> Perhaps it should be added in DEPENDENCIES_HOST_PREREQ instead ?

Right.

-- 
J?r?me Pouiller

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Thomas Petazzoni]

Hello,

On Fri, 18 Nov 2016 14:44:50 +0100, J?r?me Pouiller wrote:

> > Yes, I know, but it's a pain to force everyone using an old distro to
> > build host-tar. Maybe we should make this conditional on
> > BR2_REPRODUCIBLE?  
> 
> Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months 
> (maybe simultaneous with next Buildroot version).
> 
> So, IMHO it not justified to add a dirty condition in check-host-tar.sh.

I think you don't really realize how big companies work. They will be
using such version in 3 or 5 years maybe.

We have people still using RHEL5, even though it's almost 10 years old,
and its support going to stop next year. Supporting old distributions
is important for Buildroot, so you can't just sweep away the problem by
pretending that it doesn't exist with modern distributions.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Arnout Vandecappelle]

On 18-11-16 22:28, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 14:44:50 +0100, J?r?me Pouiller wrote:
> 
>>> Yes, I know, but it's a pain to force everyone using an old distro to
>>> build host-tar. Maybe we should make this conditional on
>>> BR2_REPRODUCIBLE?  
>>
>> Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months 
>> (maybe simultaneous with next Buildroot version).
>>
>> So, IMHO it not justified to add a dirty condition in check-host-tar.sh.
> 
> I think you don't really realize how big companies work. They will be
> using such version in 3 or 5 years maybe.
> 
> We have people still using RHEL5, even though it's almost 10 years old,
> and its support going to stop next year. Supporting old distributions
> is important for Buildroot, so you can't just sweep away the problem by
> pretending that it doesn't exist with modern distributions.

 I'm absolutely with Thomas on this one.

 Also, it doesn't have to be very dirty. In check-host-tar.sh the major_min and
minor_min can be passed as arguments, and in check-host-tar.mk we can do

ifeq ($(BR2_REPRODUCIBLE),y)
TAR_VERSION_MIN = 1 28
else
TAR_VERSION_MIN = 1 17
endif
ifeq (,$(call suitable-host-package,tar,$(TAR) $(TAR_VERSION_MIN))
...


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Arnout Vandecappelle]

 In addition to Thomas's comments:

On 18-11-16 10:10, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
> environment variable which contains either the date of Buildroot last
> commit if running from a git repository, or the latest release date.
> 
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
                                    changed


 Regards,
 Arnout

> may not have changed in fact.
> 
> The meaning of SOURCE_DATE_EPOCH is specified by the following
> specification:
>   https://reproducible-builds.org/specs/source-date-epoch/
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index eff814b..17e49e4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ=UTC
>  export LANG=C
>  export LC_ALL=C
> +export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
> +SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
>  endif
>  
>  # To put more focus on warnings, be less verbose as default
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Arnout Vandecappelle]

On 18-11-16 12:33, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 10:10:10 +0100, J?r?me Pouiller wrote:
> 
>> diff --git a/Makefile b/Makefile
>> index eff814b..17e49e4 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>>  export TZ=UTC
>>  export LANG=C
>>  export LC_ALL=C
>> +export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
> 
> Why do you do this instead of just doing
> 
> export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> 
>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>> +SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
> 
> Why don't we change the way BR2_VERSION is defined in order to avoid having to do this?
> 
> BR2_VERSION_DATE = 2016.11
> BR2_VERSION = $(BR2_VERSION)-rc1

 Or maybe even easier:

BR2_VERSION = 2016.11-rc1
BR2_VERSION_EPOCH = 1479545409

(where the EPOCH is the actual time the release is cut, instead of some fixed
time in the month).

 This is particularly useful for 2016.11.1, which otherwise would have the same
EPOCH as 2016.11.


 Regards,
 Arnout

> 
> for example.
> 
> Thanks,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 03/15] reproducible: add '-n' to gzip invocations

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> Default invocation to gzip include timestamp in output file. This feature is
> incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
> 
> The environment variable GZIP can hold a set of default options for gzip. So
> instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
> 
> Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop

 Unless I'm very mistaken, they are NOT impacted by this problem.

> include timestamp and does not provide any way to disable it.
 ^ does include a timestamp and does not...

> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  Makefile | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/Makefile b/Makefile
> index 17e49e4..c21ba4a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -249,6 +249,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ=UTC
>  export LANG=C
>  export LC_ALL=C
> +export GZIP=-n
>  export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)

 In this whole hunk, there are spaces missing:

export LC_ALL = C
export GZIP = -n
...

You only need to fix the ones you add of course.

 All of the above can be fixed up while applying, so:

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

>  SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>  SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> In order to make tar images reproducible, we use --sort flag. However,
> this flags is available only from tar 1.28. So we also bump necessary
> host-tar version.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  fs/tar/tar.mk                          | 2 +-
>  support/dependencies/check-host-tar.sh | 5 ++---
>  2 files changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
> index 11c69c5..5a1b263 100644
> --- a/fs/tar/tar.mk
> +++ b/fs/tar/tar.mk
> @@ -7,7 +7,7 @@
>  TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
>  
>  define ROOTFS_TAR_CMD
> -	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
> +	tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) .

 Instead of requiring a specific version of tar, I would prefer to use the same
solution we already have in the git wrapper, and which is also what is proposed
by reproducible-builds.org:

find src -print0 | LC_ALL=C sort -z |
    tar --null -T - --no-recursion -cf $@ --numeric-owner -C $(TARGET_DIR)

 The find | sort part we probably want to factor into the rootfs infra, because
the same thing will have to be done for all other rootfs types as well. But that
refactoring can be done later (when the other rootfs types are tackled).

 Regards,
 Arnout

>  endef
>  
>  $(eval $(call ROOTFS_TARGET,tar))
> diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
> index 932d3c4..cef6d82 100755
> --- a/support/dependencies/check-host-tar.sh
> +++ b/support/dependencies/check-host-tar.sh
> @@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then
>    minor=0
>  fi
>  
> -# Minimal version = 1.17 (previous versions do not correctly unpack archives
> -# containing hard-links if the --strip-components option is used).
> +# Minimal version = 1.28 (previous versions do not does not support --sort=name)
>  major_min=1
> -minor_min=17
> +minor_min=28
>  if [ $major -gt $major_min ]; then
>  	echo $tar
>  else
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 06/15] reproducibility/linux: inhibit build-id

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> By default, Linux kernel enable 'build-id'. 'build-id' tends to add random
> bytes in section .notes of kernel image[1]:

 Err, no, these are not random bytes, these are a sha1 of the content of the
file. If the build ID changes, it means the content has changed.

 One common way that the build ID can differ while the output files don't differ
is because of the source path that is recorded in the debug sections (which are
stripped in the end). But I think that reproducible builds when the source path
differs are very far away at this point...

> 
>   $ readelf -Wn .../vmlinux
>   Displaying notes found at file offset 0x00008000 with length 0x00000024:
>     Owner                 Data size       Description
>     GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
>       Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2
> 
> Therefore, we patch kernel Makefile to disable 'build-id'.
> 
> [1] https://kernelnewbies.org/BuildId
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  linux/linux.mk | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 7e826cc..a63d1f3 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
>  endef
>  LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
>  
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define LINUX_REMOVE_BUILD_ID
> +	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile

 build-id is also used in the VDSO, and there it is really mandatory to have it.

 Did you encounter a concrete problem with the build ID? And it didn't occur for
the VDSO, only for the vmlinux image? Could you trace it back to the individual
object file that has a different build ID?

 Regards,
 Arnout

> +endef
> +LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
> +endif
> +
>  ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
>  LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
>  else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 07/15] reproducibility/busybox: disable build timestamps

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> Busybox include a few information about build environment in its binary. This
> feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build
> information when BR2_REPRODUCIBLE is enabled.
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  package/busybox/busybox.mk | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
> index fc23a90..f4a241d 100644
> --- a/package/busybox/busybox.mk
> +++ b/package/busybox/busybox.mk
> @@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
>  	$(TARGET_MAKE_ENV) \
>  	CFLAGS="$(BUSYBOX_CFLAGS)" \
>  	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
> +
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +BUSYBOX_MAKE_ENV += \
> +	KCONFIG_NOTIMESTAMP=1
> +endif
> +
>  BUSYBOX_MAKE_OPTS = \
>  	CC="$(TARGET_CC)" \
>  	ARCH=$(KERNEL_ARCH) \
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 07/15] reproducibility/busybox: disable build timestamps

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> Busybox include a few information about build environment in its binary. This
> feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build
> information when BR2_REPRODUCIBLE is enabled.

 Commit message is not very English :-)

Busybox includes some information about the build environment in its binary. For
BR2_REPRODUCIBLE, remove that information.


 Regards,
 Arnout

> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/busybox/busybox.mk | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
> index fc23a90..f4a241d 100644
> --- a/package/busybox/busybox.mk
> +++ b/package/busybox/busybox.mk
> @@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
>  	$(TARGET_MAKE_ENV) \
>  	CFLAGS="$(BUSYBOX_CFLAGS)" \
>  	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
> +
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +BUSYBOX_MAKE_ENV += \
> +	KCONFIG_NOTIMESTAMP=1
> +endif
> +
>  BUSYBOX_MAKE_OPTS = \
>  	CC="$(TARGET_CC)" \
>  	ARCH=$(KERNEL_ARCH) \
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 08/15] reproducible: lock modification times in $TARGET_DIR

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> Make sure all files in $TARGET_DIR has a defined modification time before to
> generate filesystems.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  fs/common.mk | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/fs/common.mk b/fs/common.mk
> index 2dbef4d..981dcb1 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -95,6 +95,9 @@ endif
>  	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
>  		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
>  		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
> +ifeq ($$(BR2_REPRODUCIBLE),y)
> +	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)

 I'm not sure if this is really a good idea in general. I'm thinking for
instance of the GStreamer plugin registry, which is rebuilt based on timestamp.
Now, that particular one is not an issue because we disable it, but I expect
there will be others like that. And unfortunately that's a runtime issue.

 Still, it's impossible to predict this kind of issue, so we have to solve those
problems when they come. Therefore,

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> +endif
>  	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
>  	chmod a+x $$(FAKEROOT_SCRIPT)
>  	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Thomas Petazzoni]

Hello,

On Sat, 19 Nov 2016 09:51:27 +0100, Arnout Vandecappelle wrote:

>  Or maybe even easier:
> 
> BR2_VERSION = 2016.11-rc1
> BR2_VERSION_EPOCH = 1479545409

Yes, this seems like a good idea. It's a bit annoying that we have to
update this, but it's the cleanest solution.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v2 01/15] reproducibility: generate SOURCE_DATE_EPOCH

[Arnout Vandecappelle]

On 19-11-16 10:51, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 19 Nov 2016 09:51:27 +0100, Arnout Vandecappelle wrote:
> 
>>  Or maybe even easier:
>>
>> BR2_VERSION = 2016.11-rc1
>> BR2_VERSION_EPOCH = 1479545409
> 
> Yes, this seems like a good idea. It's a bit annoying that we have to
> update this, but it's the cleanest solution.

 An additional advantage is that you can leave it empty when no release has been
made yet, and error out in that case if no git is available.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> `date' is widely used by packages to include build information in their
> binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.
> 
> Instead to find all `date' invocation in build process, we add small tool
> allowing to alway return same date.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/fakedate/fakedate    | 28 ++++++++++++++++++++++++++++
>  package/fakedate/fakedate.mk | 14 ++++++++++++++
>  2 files changed, 42 insertions(+)
>  create mode 100755 package/fakedate/fakedate
>  create mode 100644 package/fakedate/fakedate.mk
> 
> diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
> new file mode 100755
> index 0000000..2eded22
> --- /dev/null
> +++ b/package/fakedate/fakedate
> @@ -0,0 +1,28 @@
> +#!/bin/sh
> +# vim: set sw=4 expandtab:
> +#
> +# Licence: GPL

 Please use a proper copyright blurb. Yes, it's long, but it's also more
accurate. You seem to be saying here that it is GPLv1 only, which is most likely
not what you want.

> +# Created: 2016-11-04 16:31:18+01:00
> +# Main authors:
> +#     - J?r?me Pouiller <jezz@sysmic.org>
> +#
> +
> +PATH=/bin:/usr/bin
> +LOG=/dev/null
> +if [ -n "$SOURCE_DATE_EPOCH" ]; then
> +    INHIBIT=0

 INHIBIT is a bit vague. How about DATE_IS_FORCED?

> +    for i in "$@"; do
> +        case $i in
> +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)

 We use [^-] everywhere else. Note that this pattern will also match something
like -rfrood, i.e. --reference=frood. Fixing that becomes tricky without regexp.

 Anyway, the -d option doesn't really need to be checked. 'date -d foo -d bar'
will ignore the first -d, so things work OK. It's just that you get the spurious
warning. So we could limit to checking -f, and limit to -f|--file=*). In that
case, if someone passes something like -uf we'll get an error and the build will
most likely terminate, so that particular error can be fixed.

> +            INHIBIT=1

 You could add a break here.

> +            ;;
> +        esac
> +    done
> +    if [ $INHIBIT -eq 0 ]; then
> +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2

 Is it really needed to print this warning?

> +        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
> +        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
> +    fi
> +fi
> +
> +exec date "$@"
> diff --git a/package/fakedate/fakedate.mk b/package/fakedate/fakedate.mk
> new file mode 100644
> index 0000000..e81ce5d
> --- /dev/null
> +++ b/package/fakedate/fakedate.mk
> @@ -0,0 +1,14 @@
> +################################################################################
> +#
> +# fakedate
> +#
> +################################################################################
> +
> +# source included in buildroot
> +HOST_FAKEDATE_LICENSE = GPLv2+

 This is inconsistent with the script itself, which specifies GPLv1 only :-P

 Regards,
 Arnout

> +
> +define HOST_FAKEDATE_INSTALL_CMDS
> +	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/date
> +endef
> +
> +$(eval $(host-generic-package))
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 10/15] reproducible: enable fakedate

[Arnout Vandecappelle]

On 18-11-16 14:53, J?r?me Pouiller wrote:
> On 2016-11-18 12:49, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Fri, 18 Nov 2016 10:10:19 +0100, J?r?me Pouiller wrote:
>>
>>> +ifeq ($(BR2_REPRODUCIBLE),y)
>>> +TOOLCHAIN_DEPENDENCIES += host-fakedate
>>> +endif
>>
>> Unfortunately, this means that fakedate is only installed at the end of
>> the toolchain build process. So the entire toolchain is built without
>> fakedate installed.
>>
>> Is it because we assume that the toolchain build process is safe (i.e
>> it doesn't call "date") ?
> 
> During my tests, fakedate was build before host-gcc-initial. Indeed, I
> was lucky (in add, I think toolchain build process is safe).
> 
> 
>> Perhaps it should be added in DEPENDENCIES_HOST_PREREQ instead ?
> 
> Right.

 And that can be done in the top-level BR2_REPRODUCIBLE condition.

 Regards,
 Arnout

> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 11/15] python2: generate reproducible .pyc

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before to
> compile .pyc files.

 I wonder if it is worth doing this globally.

 Normally, the .py file will come out of a tarball, so it's modification time is
fixed.

 So this is only about generated .py files. How often does that happen? Isn't it
better to deal with that per package?


 Regardless, this patch really does three things so should probably be three
patches:

1. Move the PYTHON_REMOVE_PYC_FILES hunk around. I really prefer this to be a
separate patch where you promise that nothing else change, because review is a
lot more difficult when things move around.

2. Do not remove .pyc if source .py is not present. This is in fact independent
of reproducible builds.

3. Do the fix time.

> 
> In order to guarantee .pyc are regenerated regardless their modification time,
> we remove .pyc before to compile. However, I wonder if it wouldn't be simpler
> to always call compile_all with 'force' flag.

 That would indeed be simpler. But actually, doesn't compile_all recompile
whenever the timestamp has changed? So it will really recompile everything after
you have done the FIX_TIME, because the timestamp has changed... So removing
(and therefore moving the hunk) is not needed.


 Regards,
 Arnout

> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/python/python.mk | 30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)
> 
> diff --git a/package/python/python.mk b/package/python/python.mk
> index cc65376..b0ff1fd 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -226,6 +226,26 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
>  
> +# Normally, *.pyc files should not have been compiled, but just in
> +# case, we make sure we remove all of them.
> +# However, do not remove .pyc if source .py is not present.
> +ifneq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_REPRODUCIBLE)),)
> +define PYTHON_REMOVE_PYC_FILES
> +	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +		sed -z -e s/py$$/pyc/ | \
> +		xargs -0 --no-run-if-empty rm -f
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
> +endif
> +
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON_FIX_TIME
> +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
> +endif
> +
>  define PYTHON_CREATE_PYC_FILES
>  	PYTHONPATH="$(PYTHON_PATH)" \
>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> @@ -245,16 +265,6 @@ endef
>  PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
>  endif
>  
> -# Normally, *.pyc files should not have been compiled, but just in
> -# case, we make sure we remove all of them.
> -ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
> -define PYTHON_REMOVE_PYC_FILES
> -	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | \
> -		xargs -0 --no-run-if-empty rm -f
> -endef
> -PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
> -endif
> -
>  # In all cases, we don't want to keep the optimized .pyo files
>  define PYTHON_REMOVE_PYO_FILES
>  	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyo' -print0 | \
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 11/15] python2: generate reproducible .pyc

[Arnout Vandecappelle]

On 19-11-16 11:41, Arnout Vandecappelle wrote:
> 
> On 18-11-16 10:10, J?r?me Pouiller wrote:
>> > .pyc files contain modification time of .py source. In order to make
>> > build reproducible, we fix modification time of all .py before to
>> > compile .pyc files.
>  I wonder if it is worth doing this globally.
> 
>  Normally, the .py file will come out of a tarball, so it's modification time is
> fixed.
> 
>  So this is only about generated .py files. How often does that happen? Isn't it
> better to deal with that per package?

 Forget what I said, python's own install doesn't seem to retain timestamps, so
there are a lot of py files with build-dependent timestamps. So this patch is
certainly needed.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 13/15] python2: remove full path from .pyc

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> .pyc files include path to source .py file. This patch change the way
                                                               s
> `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.
                                                  ^the
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  package/python/python.mk | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/package/python/python.mk b/package/python/python.mk
> index b0ff1fd..35f971e 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -248,9 +248,9 @@ endif
>  
>  define PYTHON_CREATE_PYC_FILES
>  	PYTHONPATH="$(PYTHON_PATH)" \
> -	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> -		support/scripts/pycompile.py \
> -		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
> +	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> +		$(TOPDIR)/support/scripts/pycompile.py \
> +		usr/lib/python$(PYTHON_VERSION_MAJOR)
>  endef
>  
>  ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 14/15] python3: remove full path from .pyc

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> .pyc files include path to source .py file. This patch change the way
> `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.
> 

 Same as for python.

> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  package/python3/python3.mk | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/package/python3/python3.mk b/package/python3/python3.mk
> index 158c29c..3b2dd31 100644
> --- a/package/python3/python3.mk
> +++ b/package/python3/python3.mk
> @@ -240,9 +240,9 @@ endif
>  
>  define PYTHON3_CREATE_PYC_FILES
>  	PYTHONPATH="$(PYTHON3_PATH)" \
> -	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
> -		support/scripts/pycompile.py \
> -		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
> +	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
> +		$(TOPDIR)/support/scripts/pycompile.py \
> +		usr/lib/python$(PYTHON3_VERSION_MAJOR)
>  endef
>  
>  ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 15/15] reproducible: improve help text

[Arnout Vandecappelle]

On 18-11-16 10:10, J?r?me Pouiller wrote:
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  Config.in | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/Config.in b/Config.in
> index 5cf0c4d..2a81202 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -707,6 +707,11 @@ config BR2_REPRODUCIBLE
>  	  this allows to generate exactly identical binaries from one
>  	  build to the other, including on different machines.
>  
> +	  Some restrictions are known on current implementation:
> +	    - Build paths ($OUTDIR and $TOPDIR) have to be the sames for all
> +	      builds
> +	    - Use of lzop is incompatible

 I don't think it makes sense to document the lzop part. For example, I doubt
any of the filesystems except for tar and maybe squashfs are reproducible at the
moment (they probably put things in filesystem order). The lzop thing is really
a small aspect compared to that.

 So I think we should just document the restriction that it has to be the same
output directory:

	  The current implementation is restricted to builds with the same
	  output directory. Many (absolute) paths are recorded in intermediary
	  files, and it is very likely that some of these paths leak into the
	  target rootfs. If you build with the same O=... path, however, the
	  result is identical.

(I'm not 100% satisfied with this explanation but it's a start.)

 Regards,
 Arnout

> +
>  	  This is labeled as an experimental feature, as not all
>  	  packages behave properly to ensure reproducibility.
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Jérôme Pouiller]

On Saturday 19 November 2016 11:21:39 Arnout Vandecappelle wrote:
> On 18-11-16 10:10, J?r?me Pouiller wrote:
[...]
> > +    for i in "$@"; do
> > +        case $i in
> > +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
> 
>  We use [^-] everywhere else.

It seems this syntax is a bashism. From glob(7): "POSIX has declared
the effect of a wildcard pattern "[^...]" to be undefined" (and I
confirm it does not work with dash)

>  Note that this pattern will also match something
> like -rfrood, i.e. --reference=frood. Fixing that becomes tricky without regexp.

hmmm... yes, it matches -rfrood (and it is what we want), but it does not
match --reference=frood, isn't?

 
>  Anyway, the -d option doesn't really need to be checked. 'date -d foo -d bar'
> will ignore the first -d, so things work OK. It's just that you get the spurious
> warning. So we could limit to checking -f, and limit to -f|--file=*). In that
> case, if someone passes something like -uf we'll get an error and the build will
> most likely terminate, so that particular error can be fixed.

You are right. However, since it may produce unexpected situation, I
prefer to identify precisely the cases where fakedate is used. 

[...]
> > +            ;;
> > +        esac
> > +    done
> > +    if [ $INHIBIT -eq 0 ]; then
> > +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
> 
>  Is it really needed to print this warning?

From user point of view, result of `date' when fakedate is installed
is unexpected. I prefer to warn.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Jérôme Pouiller]

Hello,

On Friday 18 November 2016 12:48:06 Thomas Petazzoni wrote:
> On Fri, 18 Nov 2016 10:10:18 +0100, J?r?me Pouiller wrote:
[...]
> > +PATH=/bin:/usr/bin
> 
> It is not really nice to override the PATH. I guess you want to remove
> $(HOST_DIR)/usr/bin from the PATH to not call yourself recursively, but
> I think we should do better than assuming /bin:/usr/bin is OK.

My initial idea was something based on:
    sed "s/@@DATE_CMD@@/`which date`/"

However, I worried about people who add $HOST_DIR in their $PATH and
call host-fakedate-reinstall.

However, stripping $(dirname $0) from PATH during runtime seems a good
idea. I will try that.

> > +LOG=/dev/null
> 
> This variable is used by?

In my initial version, I logged calls to fakedate in $BUILD_DIR.
I removed the log file, but it may be convenient to easily restore it
if necessary.

> > +if [ -n "$SOURCE_DATE_EPOCH" ]; then
> > +    INHIBIT=0
> > +    for i in "$@"; do
> > +        case $i in
> > +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
> > +            INHIBIT=1
> > +            ;;
> > +        esac
> > +    done
> > +    if [ $INHIBIT -eq 0 ]; then
> > +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
> > +        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
> > +        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
> > +    fi
> > +fi
> > +
> > +exec date "$@"
> 
> Could you explain a bit the logic here?

If this script is called with '--date', '--file' or any aliases of
these option, just call `date' as usual. Else, this script force
returned date.


BR,

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Arnout Vandecappelle]

On 19-11-16 14:06, J?r?me Pouiller wrote:
> On Saturday 19 November 2016 11:21:39 Arnout Vandecappelle wrote:
>> On 18-11-16 10:10, J?r?me Pouiller wrote:
> [...]
>>> +    for i in "$@"; do
>>> +        case $i in
>>> +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
>>
>>  We use [^-] everywhere else.
> 
> It seems this syntax is a bashism. From glob(7): "POSIX has declared
> the effect of a wildcard pattern "[^...]" to be undefined" (and I
> confirm it does not work with dash)
> 
>>  Note that this pattern will also match something
>> like -rfrood, i.e. --reference=frood. Fixing that becomes tricky without regexp.
> 
> hmmm... yes, it matches -rfrood (and it is what we want), but it does not
> match --reference=frood, isn't?

 -rfrood and --reference=frood are the same thing, so no, we don't want it to
match -rfrood.


>>  Anyway, the -d option doesn't really need to be checked. 'date -d foo -d bar'
>> will ignore the first -d, so things work OK. It's just that you get the spurious
>> warning. So we could limit to checking -f, and limit to -f|--file=*). In that
>> case, if someone passes something like -uf we'll get an error and the build will
>> most likely terminate, so that particular error can be fixed.
> 
> You are right. However, since it may produce unexpected situation, I
> prefer to identify precisely the cases where fakedate is used. 

 I would also prefer that, but I don't think it's possible without relying on
regex. This could work:

if echo "$i" | grep -qE '^-([urI]*d|-date|[urI]*f|-file)'; then

I notice now that you forgot a pattern for 'date --date yesterday' - that one is
handled as well by the regex above.

> 
> [...]
>>> +            ;;
>>> +        esac
>>> +    done
>>> +    if [ $INHIBIT -eq 0 ]; then
>>> +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
>>
>>  Is it really needed to print this warning?
> 
> From user point of view, result of `date' when fakedate is installed
> is unexpected. I prefer to warn.

 I'm just worried that it might confuse some script that captures stderr and
tries to do something with it.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Jérôme Pouiller]

On Saturday 19 November 2016 14:26:40 Arnout Vandecappelle wrote:
> 
> On 19-11-16 14:06, J?r?me Pouiller wrote:
> > On Saturday 19 November 2016 11:21:39 Arnout Vandecappelle wrote:
> >> On 18-11-16 10:10, J?r?me Pouiller wrote:
[...]
> >>  Note that this pattern will also match something
> >> like -rfrood, i.e. --reference=frood. Fixing that becomes tricky without regexp.
> > 
> > hmmm... yes, it matches -rfrood (and it is what we want), but it does not
> > match --reference=frood, isn't?
> 
>  -rfrood and --reference=frood are the same thing, so no, we don't want it to
> match -rfrood.

Ok, I get the point.


> >>  Anyway, the -d option doesn't really need to be checked. 'date -d foo -d bar'
> >> will ignore the first -d, so things work OK. It's just that you get the spurious
> >> warning. So we could limit to checking -f, and limit to -f|--file=*). In that
> >> case, if someone passes something like -uf we'll get an error and the build will
> >> most likely terminate, so that particular error can be fixed.
> > 
> > You are right. However, since it may produce unexpected situation, I
> > prefer to identify precisely the cases where fakedate is used. 
> 
>  I would also prefer that, but I don't think it's possible without relying on
> regex. This could work:
> 
> if echo "$i" | grep -qE '^-([urI]*d|-date|[urI]*f|-file)'; then

It begins to be complex, but I do not see better ways.


> I notice now that you forgot a pattern for 'date --date yesterday' - that one is
> handled as well by the regex above.
> 
> > 
> > [...]
> >>> +            ;;
> >>> +        esac
> >>> +    done
> >>> +    if [ $INHIBIT -eq 0 ]; then
> >>> +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
> >>
> >>  Is it really needed to print this warning?
> > 
> > From user point of view, result of `date' when fakedate is installed
> > is unexpected. I prefer to warn.
> 
>  I'm just worried that it might confuse some script that captures stderr and
> tries to do something with it.

I suggest to keep it until we find a script that captures stderr.

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 03/15] reproducible: add '-n' to gzip invocations

[Jérôme Pouiller]

Hello,

On Saturday 19 November 2016 10:02:46 Arnout Vandecappelle wrote:
> 
> On 18-11-16 10:10, J?r?me Pouiller wrote:
> > Default invocation to gzip include timestamp in output file. This feature is
> > incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
> > 
> > The environment variable GZIP can hold a set of default options for gzip. So
> > instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
> > 
> > Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop
> 
>  Unless I'm very mistaken, they are NOT impacted by this problem.

Sure!

BR,

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 05/15] reproducibility/linux: override build timestamp

[Jérôme Pouiller]

On Friday 18 November 2016 12:40:35 Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Nov 2016 10:10:14 +0100, J?r?me Pouiller wrote:
> 
> > +ifeq ($(BR2_REPRODUCIBLE),y)
> > +LINUX_MAKE_ENV += \
> > +	KBUILD_BUILD_VERSION=1 \
> > +	KBUILD_BUILD_USER=buildroot \
> > +	KBUILD_BUILD_HOST=buildroot \
> > +	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
> 
> With your "fakedate" script, do we really need this last line?

I prefer to use native solution when upstream provide one. I hope one
day all packages will be fixed by upstream and we could get ride of
fakedate.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Jérôme Pouiller]

On Friday 18 November 2016 22:28:23 Thomas Petazzoni wrote:
> On Fri, 18 Nov 2016 14:44:50 +0100, J?r?me Pouiller wrote:
> 
> > > Yes, I know, but it's a pain to force everyone using an old distro to
> > > build host-tar. Maybe we should make this conditional on
> > > BR2_REPRODUCIBLE?  
> > 
> > Last Ubuntu LTS is 16.04 and Debian 9 will be stable in a few months 
> > (maybe simultaneous with next Buildroot version).
> > 
> > So, IMHO it not justified to add a dirty condition in check-host-tar.sh.
> 
> I think you don't really realize how big companies work. They will be
> using such version in 3 or 5 years maybe.
> 
> We have people still using RHEL5, even though it's almost 10 years old,
> and its support going to stop next year. Supporting old distributions
> is important for Buildroot, so you can't just sweep away the problem by
> pretending that it doesn't exist with modern distributions.

This patch did not break support for old distributions, but OK.

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 04/15] fs/tar: make results reproducible

[Jérôme Pouiller]

On Saturday 19 November 2016 10:12:24 Arnout Vandecappelle wrote:
> On 18-11-16 10:10, J?r?me Pouiller wrote:
> > In order to make tar images reproducible, we use --sort flag. However,
> > this flags is available only from tar 1.28. So we also bump necessary
> > host-tar version.
> > 
> > This work was sponsored by `BA Robotic Systems'.
> > 
> > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> > ---
> >  fs/tar/tar.mk                          | 2 +-
> >  support/dependencies/check-host-tar.sh | 5 ++---
> >  2 files changed, 3 insertions(+), 4 deletions(-)
> > 
> > diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
> > index 11c69c5..5a1b263 100644
> > --- a/fs/tar/tar.mk
> > +++ b/fs/tar/tar.mk
> > @@ -7,7 +7,7 @@
> >  TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
> >  
> >  define ROOTFS_TAR_CMD
> > -	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
> > +	tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) .
> 
>  Instead of requiring a specific version of tar, I would prefer to use the same
> solution we already have in the git wrapper, and which is also what is proposed
> by reproducible-builds.org:
> 
> find src -print0 | LC_ALL=C sort -z |
>     tar --null -T - --no-recursion -cf $@ --numeric-owner -C $(TARGET_DIR)
> 
>  The find | sort part we probably want to factor into the rootfs infra, because
> the same thing will have to be done for all other rootfs types as well. But that
> refactoring can be done later (when the other rootfs types are tackled).

It makes sense. I will do that.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 06/15] reproducibility/linux: inhibit build-id

[Jérôme Pouiller]

On Saturday 19 November 2016 10:31:51 Arnout Vandecappelle wrote:
> 
> On 18-11-16 10:10, J?r?me Pouiller wrote:
> > By default, Linux kernel enable 'build-id'. 'build-id' tends to add random
> > bytes in section .notes of kernel image[1]:
> 
>  Err, no, these are not random bytes, these are a sha1 of the content of the
> file. If the build ID changes, it means the content has changed.
> 
>  One common way that the build ID can differ while the output files don't differ
> is because of the source path that is recorded in the debug sections (which are
> stripped in the end). But I think that reproducible builds when the source path
> differs are very far away at this point...

Oh, you are right! I didn't understood why my build-id changes while all
sections was identical. In fact, in some circumstances, symbol order in
debug sections is not defined. I don't know yet exactly why nor how to
fix it.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v2 05/15] reproducibility/linux: override build timestamp

[Arnout Vandecappelle]

On 19-11-16 14:53, J?r?me Pouiller wrote:
> On Friday 18 November 2016 12:40:35 Thomas Petazzoni wrote:
>> Hello,
>>
>> On Fri, 18 Nov 2016 10:10:14 +0100, J?r?me Pouiller wrote:
>>
>>> +ifeq ($(BR2_REPRODUCIBLE),y)
>>> +LINUX_MAKE_ENV += \
>>> +	KBUILD_BUILD_VERSION=1 \
>>> +	KBUILD_BUILD_USER=buildroot \
>>> +	KBUILD_BUILD_HOST=buildroot \
>>> +	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
>>
>> With your "fakedate" script, do we really need this last line?
> 
> I prefer to use native solution when upstream provide one. I hope one
> day all packages will be fixed by upstream and we could get ride of
> fakedate.

 +1 to that. Could you explain this in the commit message?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2 09/15] fakedate: new package

[Jérôme Pouiller]

Hello Arnoult,

On Saturday 19 November 2016 14:26:40 Arnout Vandecappelle wrote:
> 
> On 19-11-16 14:06, J?r?me Pouiller wrote:
> > On Saturday 19 November 2016 11:21:39 Arnout Vandecappelle wrote:
> >> On 18-11-16 10:10, J?r?me Pouiller wrote:
> > [...]
> >>> +    for i in "$@"; do
> >>> +        case $i in
> >>> +        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
> >>
> >>  We use [^-] everywhere else.
> > 
> > It seems this syntax is a bashism. From glob(7): "POSIX has declared
> > the effect of a wildcard pattern "[^...]" to be undefined" (and I
> > confirm it does not work with dash)
> > 
> >>  Note that this pattern will also match something
> >> like -rfrood, i.e. --reference=frood. Fixing that becomes tricky without regexp.
> > 
> > hmmm... yes, it matches -rfrood (and it is what we want), but it does not
> > match --reference=frood, isn't?
> 
>  -rfrood and --reference=frood are the same thing, so no, we don't want it to
> match -rfrood.
> 
> 
> >>  Anyway, the -d option doesn't really need to be checked. 'date -d foo -d bar'
> >> will ignore the first -d, so things work OK. It's just that you get the spurious
> >> warning. So we could limit to checking -f, and limit to -f|--file=*). In that
> >> case, if someone passes something like -uf we'll get an error and the build will
> >> most likely terminate, so that particular error can be fixed.
> > 
> > You are right. However, since it may produce unexpected situation, I
> > prefer to identify precisely the cases where fakedate is used. 
> 
>  I would also prefer that, but I don't think it's possible without relying on
> regex. This could work:
> 
> if echo "$i" | grep -qE '^-([urI]*d|-date|[urI]*f|-file)'; then

From manual page, only option -u and -R do not take arguments. In add, we
also have to inhibit fakedate is --reference (or -r) is detected. So, I
think that the expression should be:

   '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr