[Buildroot] Reproducible builds

[Buildroot] Reproducible builds

[Gilles Chanteperdrix]

Hi,

some time ago, I worked on getting reproducible builds for
buildroot. I take "reproducible builds" in a broader sense than what
Debian does for instance: in the case of buildroot, we would like
the built binaries to be as independent of the build environment as
possible, so that in addition to getting identical binaries when
building twice on the same machine, we also get identical binaries
when building on a different distribution. So far, I got
reproducible builds for a (relatively reduced) subset of buildroot
packages, on a Debian jessie and Slackware 14.1, with as many
differences in the distributions configuration as I could find
(timezone, locale, date, time, etc... even the awk flavor used). The
only thing I have to get identical is the build path, this is what
Debian does, a patch has been proposed a long time ago to allow gcc
to avoid the dependence between the __FILE__ macro and the build
path, but it does not seem to have been merged:
https://mail-index.netbsd.org/tech-toolchain/2009/02/17/msg000577.html

I expected to be able to clean up the patches before submission (as
many patches do the same thing, like overriding the __DATE__ and
__TIME__ macros in order to avoid depending on the build time, so
this could probably be made generic by buildroot core), but I am not
going to have time soon to do that, so here come the patches as they
are, in order to spark discussion and gather remarks, before I have
time to do more. The patches are based on the 2015.11.1 release.

Regards.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Gilles Chanteperdrix]

Making the builds reproducible involves turning off build timestamps,
which some users may find annoying. So make the reproducible builds
optional.
---
 Config.in | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Config.in b/Config.in
index d795361..3cdd659 100644
--- a/Config.in
+++ b/Config.in
@@ -623,6 +623,9 @@ config BR2_COMPILER_PARANOID_UNSAFE_PATH
 
 endmenu
 
+config BR2_REPRODUCIBLE
+	bool "Try and remove all sources of non-reproducibility from the build"
+
 endmenu
 
 source "toolchain/Config.in"
-- 
2.7.4

[Buildroot] [PATCH 02/34] reproducibility: override locale and timezone

[Gilles Chanteperdrix]

---
 Makefile | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Makefile b/Makefile
index ec7dccb..86b2ed2 100644
--- a/Makefile
+++ b/Makefile
@@ -218,6 +218,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
 -include $(BR2_CONFIG)
 endif
 
+export BR2_REPRODUCIBLE
+ifeq ($(BR2_REPRODUCIBLE),y)
+export TZ=UTC
+export LANG=C
+export LC_ALL=C
+endif
+
 # To put more focus on warnings, be less verbose as default
 # Use 'make V=1' to see the full commands
 ifeq ("$(origin V)", "command line")
-- 
2.7.4

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Gilles Chanteperdrix]

When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
environment variable which contains either the date of buildroot last
commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been change, even though some of them
may not have changed in fact.
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Makefile b/Makefile
index 86b2ed2..c1d2961 100644
--- a/Makefile
+++ b/Makefile
@@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
+	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
+	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
2.7.4

[Buildroot] [PATCH 04/34] reproducibility: make rootfs.tar reproducible

[Gilles Chanteperdrix]

By generating a tarball with a deterministic file order and date.
---
 fs/tar/tar.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
index 28219cf..06a4d7c 100644
--- a/fs/tar/tar.mk
+++ b/fs/tar/tar.mk
@@ -6,8 +6,19 @@
 
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
+ifneq ($(BR2_REPRODUCIBLE),y)
 define ROOTFS_TAR_CMD
 	tar -c$(TAR_OPTS)f $@ --numeric-owner -C $(TARGET_DIR) .
 endef
+else
+define ROOTFS_TAR_CMD
+	cd $(TARGET_DIR) && { \
+		find . -\( -! -type d -o -empty -\) -print0 | \
+		sort -z | \
+		tar --null -T - -c$(TAR_OPTS)f $@ --mtime=@$(SOURCE_DATE_EPOCH) --numeric-owner; \
+	}
+endef
+endif
+
 
 $(eval $(call ROOTFS_TARGET,tar))
-- 
2.7.4

[Buildroot] [PATCH 05/34] reproducibility/linux: override build timestamp

[Gilles Chanteperdrix]

with SOURCE_DATE_EPOCH
---
 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 9507837..d882702 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -74,6 +74,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
-- 
2.7.4

[Buildroot] [PATCH 06/34] reproducibility/busybox: disable build timestamps

[Gilles Chanteperdrix]

---
 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 7f6dbd6..bee7489 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -32,6 +32,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
-- 
2.7.4

[Buildroot] [PATCH 07/34] reproducibility/libgcrypt: override timestamps

[Gilles Chanteperdrix]

with SOURCE_DATE_EPOCH
---
 package/libgcrypt/0003-honor-source-date-epoch.patch | 15 +++++++++++++++
 package/libgcrypt/libgcrypt.mk                       |  1 +
 2 files changed, 16 insertions(+)
 create mode 100644 package/libgcrypt/0003-honor-source-date-epoch.patch

diff --git a/package/libgcrypt/0003-honor-source-date-epoch.patch b/package/libgcrypt/0003-honor-source-date-epoch.patch
new file mode 100644
index 0000000..0f41c1b
--- /dev/null
+++ b/package/libgcrypt/0003-honor-source-date-epoch.patch
@@ -0,0 +1,15 @@
+--- libgcrypt-1.6.4/configure.ac.orig	2015-09-08 08:27:26.000000000 +0200
++++ libgcrypt-1.6.4/configure.ac	2016-02-23 21:31:19.290079195 +0100
+@@ -1993,7 +1993,11 @@ changequote([,])dnl
+ BUILD_FILEVERSION="${BUILD_FILEVERSION}mym4_revision_dec"
+ AC_SUBST(BUILD_FILEVERSION)
+ 
+-BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
++if test x"${SOURCE_DATE_EPOCH}" = x; then
++   	BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
++else
++	BUILD_TIMESTAMP=`date -u -d @${SOURCE_DATE_EPOCH} +%Y-%m-%dT%H:%M+0000 2>/dev/null || date -d @${SOURCE_DATE_EPOCH}`
++fi
+ AC_SUBST(BUILD_TIMESTAMP)
+ AC_DEFINE_UNQUOTED(BUILD_TIMESTAMP, "$BUILD_TIMESTAMP",
+                    [The time this package was configured for a build])
diff --git a/package/libgcrypt/libgcrypt.mk b/package/libgcrypt/libgcrypt.mk
index 53d1d6c..768e42f 100644
--- a/package/libgcrypt/libgcrypt.mk
+++ b/package/libgcrypt/libgcrypt.mk
@@ -12,6 +12,7 @@ LIBGCRYPT_SITE = ftp://ftp.gnupg.org/gcrypt/libgcrypt
 LIBGCRYPT_INSTALL_STAGING = YES
 LIBGCRYPT_DEPENDENCIES = libgpg-error
 LIBGCRYPT_CONFIG_SCRIPTS = libgcrypt-config
+LIBGCRYPT_AUTORECONF = YES
 
 LIBGCRYPT_CONF_ENV = \
 	ac_cv_sys_symbol_underscore=no
-- 
2.7.4

[Buildroot] [PATCH 08/34] reproducibility/libgpg-error: override timestamps

[Gilles Chanteperdrix]

with SOURCE_DATE_EPOCH
---
 package/libgpg-error/0002-honor-source-date-epoch.patch | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 package/libgpg-error/0002-honor-source-date-epoch.patch

diff --git a/package/libgpg-error/0002-honor-source-date-epoch.patch b/package/libgpg-error/0002-honor-source-date-epoch.patch
new file mode 100644
index 0000000..2df7cca
--- /dev/null
+++ b/package/libgpg-error/0002-honor-source-date-epoch.patch
@@ -0,0 +1,15 @@
+--- libgpg-error-1.12/configure.ac~	2016-02-23 20:30:21.000000000 +0100
++++ libgpg-error-1.12/configure.ac	2016-02-23 21:16:20.284133994 +0100
+@@ -198,7 +198,11 @@ changequote([,])dnl
+ BUILD_FILEVERSION="${BUILD_FILEVERSION}0,mym4_revision_dec"
+ AC_SUBST(BUILD_FILEVERSION)
+ 
+-BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
++if test x"${SOURCE_DATE_EPOCH}" = x; then
++   	BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
++else
++	BUILD_TIMESTAMP=`date -u -d @${SOURCE_DATE_EPOCH} +%Y-%m-%dT%H:%M+0000 2>/dev/null || date -d @${SOURCE_DATE_EPOCH}`
++fi
+ AC_SUBST(BUILD_TIMESTAMP)
+ AC_DEFINE_UNQUOTED(BUILD_TIMESTAMP, "$BUILD_TIMESTAMP",
+                    [The time this package was configured for a build])
-- 
2.7.4

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gilles Chanteperdrix]

glib2 installs compiled python bytecode in /usr/share/glib-2.0/codegen,
likely for the gdbus-codegen program.

This bytecode is not compiled with the buildroot host python, which
causes a first source of non reproducibility, fix that by using
buildroot host python.

But this is not sufficient, compiling the python bytecode with the
same interpreter in different environments yields different binaries,
so, since buildroot users are unlikely to need the qdbus-codegen
program on target, add an option to remove it. The option is disabled
by default.
---
 package/libglib2/Config.in   | 10 ++++++++++
 package/libglib2/libglib2.mk | 18 ++++++++++++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/package/libglib2/Config.in b/package/libglib2/Config.in
index 7cbfea5..8a39805 100644
--- a/package/libglib2/Config.in
+++ b/package/libglib2/Config.in
@@ -13,6 +13,16 @@ config BR2_PACKAGE_LIBGLIB2
 
 	  http://www.gtk.org/
 
+if BR2_PACKAGE_LIBGLIB2
+
+config BR2_PACKAGE_LIBGLIB2_CODEGEN
+	bool "install Glib D-Bus code and documentation generator"
+	default y
+	help
+	  Install Glib D-Bus code and documentation generator.
+
+endif
+
 comment "libglib2 needs a toolchain w/ wchar, threads"
 	depends on BR2_USE_MMU
 	depends on !BR2_USE_WCHAR || !BR2_TOOLCHAIN_HAS_THREADS
diff --git a/package/libglib2/libglib2.mk b/package/libglib2/libglib2.mk
index 8cf055d..7feba90 100644
--- a/package/libglib2/libglib2.mk
+++ b/package/libglib2/libglib2.mk
@@ -99,9 +99,9 @@ HOST_LIBGLIB2_CONF_OPTS = \
 	--disable-systemtap \
 	--disable-xattr
 
-LIBGLIB2_DEPENDENCIES = host-pkgconf host-libglib2 libffi zlib $(if $(BR2_NEEDS_GETTEXT),gettext) host-gettext
+LIBGLIB2_DEPENDENCIES = host-pkgconf host-libglib2 libffi zlib $(if $(BR2_NEEDS_GETTEXT),gettext) host-gettext host-python
 
-HOST_LIBGLIB2_DEPENDENCIES = host-pkgconf host-libffi host-zlib host-gettext
+HOST_LIBGLIB2_DEPENDENCIES = host-pkgconf host-libffi host-zlib host-gettext host-python
 
 ifneq ($(BR2_ENABLE_LOCALE),y)
 LIBGLIB2_DEPENDENCIES += libiconv
@@ -137,6 +137,20 @@ ifneq ($(BR2_PACKAGE_GDB),y)
 LIBGLIB2_POST_INSTALL_TARGET_HOOKS += LIBGLIB2_REMOVE_GDB_FILES
 endif
 
+ifeq ($(BR2_PACKAGE_LIBGLIB2_CODEGEN),)
+define LIBGLIB2_REMOVE_CODEGEN
+	rm $(TARGET_DIR)/usr/bin/gdbus-codegen
+	rm -rf $(TARGET_DIR)/usr/share/glib-2.0/codegen
+	rmdir --ignore-fail-on-non-empty $(TARGET_DIR)/usr/share/glib-2.0
+endef
+
+LIBGLIB2_POST_INSTALL_TARGET_HOOKS += LIBGLIB2_REMOVE_CODEGEN
+endif
+
+LIBGLIB2_CONF_OPTS += --with-python=$(HOST_DIR)/usr/bin/python
+
+HOST_LIBGLIB2_CONF_OPTS += --with-python=$(HOST_DIR)/usr/bin/python
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
-- 
2.7.4

[Buildroot] [PATCH 10/34] reproducibility/sysstat: override gcc macros

[Gilles Chanteperdrix]

Override the __DATE__ and __TIME__ macros with values obtained from
the package SOURCE_DATE_EPOCH.
---
 package/sysstat/sysstat.mk | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/package/sysstat/sysstat.mk b/package/sysstat/sysstat.mk
index 7568a4b..5cc6cd6 100644
--- a/package/sysstat/sysstat.mk
+++ b/package/sysstat/sysstat.mk
@@ -17,6 +17,25 @@ SYSSTAT_DEPENDENCIES += gettext
 SYSSTAT_MAKE_OPTS += LFLAGS="$(TARGET_LDFLAGS) -lintl"
 endif
 
+SYSSTAT_CONF_OPTS += \
+	sa_lib_dir=/usr/lib/sa
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+SYSSTAT_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
+SYSSTAT_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
+
+SYSSTAT_CONF_OPTS += \
+	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(SYSSTAT_DIR)/.br_date_override.h"
+
+define SYSSTAT_OVERRIDE_DATE_HOOK
+	echo "#define buildroot_stringify(x) #x" > $(SYSSTAT_DIR)/.br_date_override.h
+	echo "#define __DATE__ buildroot_stringify($(SYSSTAT_SOURCE_DATE))" >> $(SYSSTAT_DIR)/.br_date_override.h
+	echo "#define __TIME__ buildroot_stringify($(SYSSTAT_SOURCE_TIME))" >> $(SYSSTAT_DIR)/.br_date_override.h
+endef
+
+SYSSTAT_POST_PATCH_HOOKS += SYSSTAT_OVERRIDE_DATE_HOOK
+endif
+
 # The isag tool is a post processing script that depends on tcl/tk
 # among other things. So we don't install it.
 SYSSTAT_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) INSTALL_ISAG=n install
-- 
2.7.4

[Buildroot] [PATCH 11/34] reproducibility/vim: make vim reproducible

[Gilles Chanteperdrix]

There are three reasons why vim is not reproducible:
- it embeds the build timestamp, fix that by overriding them with
  SOURCE_DATE_EPOCH;
- it embeds the build host and user, fix that by using its configure
  script --with-compiledby option
- it generates an awk script which depends on the version of awk present
  on the build machine, fix that by forcing the interpreter to be
  /usr/bin/awk, it is what is available on buildroot rootfs anyway.
---
 package/vim/vim.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/package/vim/vim.mk b/package/vim/vim.mk
index a6b75d0..475fd3c 100644
--- a/package/vim/vim.mk
+++ b/package/vim/vim.mk
@@ -34,6 +34,29 @@ else
 VIM_CONF_OPTS += --disable-selinux
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+VIM_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
+VIM_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
+
+VIM_CONF_OPTS += \
+	--with-compiledby=buildroot \
+	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(VIM_DIR)/.br_date_override.h"
+
+define VIM_OVERRIDE_DATE_HOOK
+	echo "#define buildroot_stringify(x) #x" > $(VIM_DIR)/.br_date_override.h
+	echo "#define __DATE__ buildroot_stringify($(VIM_SOURCE_DATE))" >> $(VIM_DIR)/.br_date_override.h
+	echo "#define __TIME__ buildroot_stringify($(VIM_SOURCE_TIME))" >> $(VIM_DIR)/.br_date_override.h
+endef
+
+VIM_POST_PATCH_HOOKS += VIM_OVERRIDE_DATE_HOOK
+
+define VIM_FIX_AWK_REFERENCE
+	sed -i 's,^#!.*awk,#!/usr/bin/awk,' $(TARGET_DIR)/usr/share/vim/vim74/tools/mve.awk
+endef
+
+VIM_POST_INSTALL_TARGET_HOOKS += VIM_FIX_AWK_REFERENCE
+endif
+
 define VIM_INSTALL_TARGET_CMDS
 	cd $(@D)/src; \
 		$(MAKE) DESTDIR=$(TARGET_DIR) installvimbin; \
-- 
2.7.4

[Buildroot] [PATCH 12/34] reproducibility/wireshark: remove documentation

[Gilles Chanteperdrix]

HTML documentation contains a link to a mail address specific to
the distribution of the build machine. Remove html documentation
from the rootfs as it is unlikely to be needed. At this chance
also remove the txt documentation.
---
 package/wireshark/wireshark.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/wireshark/wireshark.mk b/package/wireshark/wireshark.mk
index ffadf80..43adb1e 100644
--- a/package/wireshark/wireshark.mk
+++ b/package/wireshark/wireshark.mk
@@ -28,4 +28,10 @@ WIRESHARK_CONF_OPTS = \
 	--with-libsmi=no \
 	--includedir=$(STAGING_DIR)/usr/include
 
+define WIRESHARK_REMOVE_DOCS
+	find $(TARGET_DIR)/usr/share/wireshark -\( -name '*.txt' -o -name '*.html' -\) -delete
+endef
+
+WIRESHARK_POST_INSTALL_TARGET_HOOKS += WIRESHARK_REMOVE_DOCS
+
 $(eval $(autotools-package))
-- 
2.7.4

[Buildroot] [PATCH 13/34] reproducibility/libtool: fix reproducibility issues

[Gilles Chanteperdrix]

- remove libtool and libtoolize; they are full of references to the
  build environment tools, and unlikely to be needed on target;
- do not use /etc/ld.so.conf contents in libtool search path: they are
  not relevant on target, and cause a useless dependency of libltdl
  on the build environment.
---
 package/libtool/0001-fix-cross-compilation.patch | 11 +++++++++++
 package/libtool/libtool.mk                       |  8 ++++++--
 2 files changed, 17 insertions(+), 2 deletions(-)
 create mode 100644 package/libtool/0001-fix-cross-compilation.patch

diff --git a/package/libtool/0001-fix-cross-compilation.patch b/package/libtool/0001-fix-cross-compilation.patch
new file mode 100644
index 0000000..c203371
--- /dev/null
+++ b/package/libtool/0001-fix-cross-compilation.patch
@@ -0,0 +1,11 @@
+--- libtool-2.4.6/m4/libtool.m4~	2015-01-20 17:15:19.000000000 +0100
++++ libtool-2.4.6/m4/libtool.m4	2016-02-27 21:02:21.513684396 +0100
+@@ -2873,7 +2873,7 @@ linux* | k*bsd*-gnu | kopensolaris*-gnu
+   # 'ldconfig -N -X -v | grep ^/' on 64bit Fedora does not report /usr/lib64,
+   # even though it is searched at run-time.  Try to do the best guess by
+   # appending ld.so.conf contents (and includes) to the search path.
+-  if test -f /etc/ld.so.conf; then
++  if test "$cross_compiling" != "yes" -a -f /etc/ld.so.conf; then
+     lt_ld_extra=`awk '/^include / { system(sprintf("cd /etc; cat %s 2>/dev/null", \[$]2)); skip = 1; } { if (!skip) print \[$]0; skip = 0; }' < /etc/ld.so.conf | $SED -e 's/#.*//;/^[	 ]*hwcap[	 ]/d;s/[:,	]/ /g;s/=[^=]*$//;s/=[^= ]* / /g;s/"//g;/^$/d' | tr '\n' ' '`
+     sys_lib_dlsearch_path_spec="/lib /usr/lib $lt_ld_extra"
+   fi
diff --git a/package/libtool/libtool.mk b/package/libtool/libtool.mk
index 8688f8b..c177626 100644
--- a/package/libtool/libtool.mk
+++ b/package/libtool/libtool.mk
@@ -8,7 +8,7 @@ LIBTOOL_VERSION = 2.4.6
 LIBTOOL_SOURCE = libtool-$(LIBTOOL_VERSION).tar.xz
 LIBTOOL_SITE = $(BR2_GNU_MIRROR)/libtool
 LIBTOOL_INSTALL_STAGING = YES
-LIBTOOL_CONF_ENV = HELP2MAN=true
+LIBTOOL_CONF_OPTS = HELP2MAN=true
 LIBTOOL_DEPENDENCIES = host-m4
 LIBTOOL_LICENSE = GPLv2+
 LIBTOOL_LICENSE_FILES = COPYING
@@ -33,9 +33,13 @@ define LIBTOOL_AVOID_AUTORECONF_HOOK
 	find $(@D) -name configure -exec touch '{}' \;
 	find $(@D) -name Makefile.in -exec touch '{}' \;
 endef
-LIBTOOL_PRE_CONFIGURE_HOOKS += LIBTOOL_AVOID_AUTORECONF_HOOK
 HOST_LIBTOOL_PRE_CONFIGURE_HOOKS += LIBTOOL_AVOID_AUTORECONF_HOOK
 
+define LIBTOOL_REMOVE_LIBTOOL
+	rm $(TARGET_DIR)/usr/bin/libtool $(TARGET_DIR)/usr/bin/libtoolize
+endef
+LIBTOOL_POST_INSTALL_TARGET_HOOKS += LIBTOOL_REMOVE_LIBTOOL
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
-- 
2.7.4

[Buildroot] [PATCH 14/34] reproducibility/python: also remove pyo files

[Gilles Chanteperdrix]

when BR2_PACKAGE_PYTHON_PY_ONLY is selected. Make that option the
default when BR2_REPRODUCIBLE is selected in order to avoid problems
with timestamps embedded in compiled python files.
---
 package/python/Config.in | 3 ++-
 package/python/python.mk | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/package/python/Config.in b/package/python/Config.in
index fa83e77..2c62559 100644
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -19,7 +19,8 @@ if BR2_PACKAGE_PYTHON
 
 choice
 	prompt "python module format to install"
-	default BR2_PACKAGE_PYTHON_PYC_ONLY
+	default BR2_PACKAGE_PYTHON_PYC_ONLY if !BR2_REPRODUCIBLE
+	default BR2_PACKAGE_PYTHON_PY_ONLY if BR2_REPRODUCIBLE
 	help
 	  Select Python module format to install on target (py, pyc or both)
 
diff --git a/package/python/python.mk b/package/python/python.mk
index 80ffbde..5c5ebb8 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -225,7 +225,7 @@ endif
 
 ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
 define PYTHON_FINALIZE_TARGET
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py[co]' -print0 | xargs -0 rm -f
 endef
 endif
 
-- 
2.7.4

[Buildroot] [PATCH 15/34] reproducibility/gzip: avoid gzip timestamp in rootfs.tar.gz

[Gilles Chanteperdrix]

---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/common.mk b/fs/common.mk
index 528e194..ea248bb 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -48,7 +48,7 @@ ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
 ROOTFS_$(2)_COMPRESS_EXT = .gz
-ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
+ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c $(if $(BR2_REPRODUCIBLE),-n)
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
 ROOTFS_$(2)_COMPRESS_EXT = .bz2
-- 
2.7.4

[Buildroot] [PATCH 16/34] reproducibility/lzo: forbid lzo with reproducible builds

[Gilles Chanteperdrix]

The lzo archive contains a timestamp.
---
 fs/tar/Config.in | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/fs/tar/Config.in b/fs/tar/Config.in
index 63663ec..7e4b603 100644
--- a/fs/tar/Config.in
+++ b/fs/tar/Config.in
@@ -33,6 +33,7 @@ config BR2_TARGET_ROOTFS_TAR_LZMA
 
 config BR2_TARGET_ROOTFS_TAR_LZO
 	bool "lzo"
+	depends on !BR2_REPRODUCIBLE
 	help
 	  Do compress the tarball with lzop.
 
@@ -43,6 +44,12 @@ config BR2_TARGET_ROOTFS_TAR_XZ
 
 endchoice
 
+if BR2_REPRODUCIBLE
+
+comment "lzo compression is only available with non-reproducible builds"
+
+endif
+
 config BR2_TARGET_ROOTFS_TAR_OPTIONS
 	string "other random options to pass to tar"
 	depends on BR2_TARGET_ROOTFS_TAR
-- 
2.7.4

[Buildroot] [PATCH 17/34] reproducibility/libnspr: override build timestamps

[Gilles Chanteperdrix]

with SOURCE_DATE_EPOCH
---
 package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch | 71 ++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch

diff --git a/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
new file mode 100644
index 0000000..1a0fb7c
--- /dev/null
+++ b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
@@ -0,0 +1,71 @@
+--- libnspr-4.10.10/nspr/lib/libc/src/Makefile.in~	2015-10-16 15:22:19.000000000 +0200
++++ libnspr-4.10.10/nspr/lib/libc/src/Makefile.in	2016-02-27 16:11:13.745483215 +0100
+@@ -103,7 +103,7 @@ ECHO = echo
+ TINC = $(OBJDIR)/_pl_bld.h
+ PROD = $(notdir $(SHARED_LIBRARY))
+ NOW = $(MOD_DEPTH)/config/$(OBJDIR)/now
+-SH_DATE = $(shell date "+%Y-%m-%d %T")
++SH_DATE = $(shell date `test -n "$$SOURCE_DATE_EPOCH" && echo -d @$$SOURCE_DATE_EPOCH` "+%Y-%m-%d %T")
+ SH_NOW = $(shell $(NOW))
+ 
+ ifeq ($(NS_USE_GCC)_$(OS_ARCH),_WINNT)
+--- libnspr-4.10.10/nspr/lib/ds/Makefile.in~	2015-10-16 15:22:19.000000000 +0200
++++ libnspr-4.10.10/nspr/lib/ds/Makefile.in	2016-02-27 16:11:28.115738828 +0100
+@@ -101,7 +101,7 @@ ECHO = echo
+ TINC = $(OBJDIR)/_pl_bld.h
+ PROD = $(notdir $(SHARED_LIBRARY))
+ NOW = $(MOD_DEPTH)/config/$(OBJDIR)/now
+-SH_DATE = $(shell date "+%Y-%m-%d %T")
++SH_DATE = $(shell date `test -n "$$SOURCE_DATE_EPOCH" && echo -d @$$SOURCE_DATE_EPOCH` "+%Y-%m-%d %T")
+ SH_NOW = $(shell $(NOW))
+ 
+ ifeq ($(NS_USE_GCC)_$(OS_ARCH),_WINNT)
+--- libnspr-4.10.10/nspr/lib/prstreams/Makefile.in~	2015-10-16 15:22:19.000000000 +0200
++++ libnspr-4.10.10/nspr/lib/prstreams/Makefile.in	2016-02-27 16:11:39.799279987 +0100
+@@ -105,7 +105,7 @@ ECHO = echo
+ TINC = $(OBJDIR)/_pl_bld.h
+ PROD = $(notdir $(SHARED_LIBRARY))
+ NOW = $(MOD_DEPTH)/config/$(OBJDIR)/now
+-SH_DATE = $(shell date "+%Y-%m-%d %T")
++SH_DATE = $(shell date `test -n "$$SOURCE_DATE_EPOCH" && echo -d @$$SOURCE_DATE_EPOCH` "+%Y-%m-%d %T")
+ SH_NOW = $(shell $(NOW))
+ 
+ ifeq ($(OS_ARCH), WINNT)
+--- libnspr-4.10.10/nspr/pr/src/Makefile.in~	2015-10-16 15:22:19.000000000 +0200
++++ libnspr-4.10.10/nspr/pr/src/Makefile.in	2016-02-27 16:18:55.560364508 +0100
+@@ -315,7 +315,7 @@ PROD = $(notdir $(SHARED_LIBRARY))
+ endif
+ 
+ NOW = $(MOD_DEPTH)/config/$(OBJDIR)/now
+-SH_DATE = $(shell date "+%Y-%m-%d %T")
++SH_DATE = $(shell date `test -n "$$SOURCE_DATE_EPOCH" && echo -d @$$SOURCE_DATE_EPOCH` "+%Y-%m-%d %T")
+ SH_NOW = $(shell $(NOW))
+ 
+ ifeq ($(NS_USE_GCC)_$(OS_ARCH),_WINNT)
+--- libnspr-4.10.10/nspr/config/now.c~	2015-10-16 15:22:19.000000000 +0200
++++ libnspr-4.10.10/nspr/config/now.c	2016-02-27 16:29:53.862069883 +0100
+@@ -4,6 +4,7 @@
+  * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
+ 
+ #include <stdio.h>
++#include <stdlib.h>
+ #include <time.h>
+ 
+ int main(int argc, char **argv)
+@@ -27,10 +28,15 @@ int main(int argc, char **argv)
+     now = (1000000i64) * sec;
+     fprintf(stdout, "%I64d", now);
+ #else
++    const char *source_date_epoch = getenv("SOURCE_DATE_EPOCH");
+     long long now;
+     time_t sec;
+ 
+-    sec = time(NULL);
++    if (!source_date_epoch)
++      sec = time(NULL);
++    else
++      sec = (time_t)atoll(source_date_epoch);
++
+     now = (1000000LL) * sec;
+     fprintf(stdout, "%lld", now);
+ #endif
-- 
2.7.4

[Buildroot] [PATCH 18/34] readline: fix installed libraries permission

[Gilles Chanteperdrix]

In order to avoid strip failing to strip the library.
---
 package/readline/0009-avoid-strip-perm-issue.patch | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100644 package/readline/0009-avoid-strip-perm-issue.patch

diff --git a/package/readline/0009-avoid-strip-perm-issue.patch b/package/readline/0009-avoid-strip-perm-issue.patch
new file mode 100644
index 0000000..8415ca2
--- /dev/null
+++ b/package/readline/0009-avoid-strip-perm-issue.patch
@@ -0,0 +1,11 @@
+--- readline-6.3/support/shlib-install~	2012-07-10 21:58:08.000000000 +0200
++++ readline-6.3/support/shlib-install	2016-02-27 01:19:09.878310797 +0100
+@@ -73,7 +73,7 @@ fi
+ case "$host_os" in
+ hpux*|darwin*|macosx*|linux*|solaris2*)
+ 	if [ -z "$uninstall" ]; then
+-		chmod 555 ${INSTALLDIR}/${LIBNAME}
++		chmod 755 ${INSTALLDIR}/${LIBNAME}
+ 	fi ;;
+ cygwin*|mingw*)
+ 	IMPLIBNAME=`echo ${LIBNAME} \
-- 
2.7.4

[Buildroot] [PATCH 19/34] reproducibility/python: make python reproducible

[Gilles Chanteperdrix]

python build is not reproducible for two reasons:
- the libpython library embeds the build timestamps, fix this by
  overriding the __DATE__ and __TIME__ macro using SOURCE_DATE_EPOCH;
- _sysconfigdata.py and config/Makefile on target includes references
  through INSTALL and MKDIR_P variables to host tools, fix this by
  overriding these variables values with the corresponding paths on
  buildroot rootfs.
---
 package/python/python.mk | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index 5c5ebb8..dc064df 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -143,7 +143,9 @@ PYTHON_CONF_OPTS += \
 	--disable-tk		\
 	--disable-nis		\
 	--disable-dbm		\
-	--disable-pyo-build
+	--disable-pyo-build	\
+	ac_cv_path_install=/usr/bin/install \
+	ac_cv_path_mkdir=/bin/mkdir
 
 # This is needed to make sure the Python build process doesn't try to
 # regenerate those files with the pgen program. Otherwise, it builds
@@ -214,6 +216,22 @@ endif
 # Provided to other packages
 PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/:$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+PYTHON_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
+PYTHON_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
+
+PYTHON_CONF_OPTS += \
+	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(PYTHON_DIR)/.br_date_override.h"
+
+define PYTHON_OVERRIDE_DATE_HOOK
+	echo "#define buildroot_stringify(x) #x" > $(PYTHON_DIR)/.br_date_override.h
+	echo "#define __DATE__ buildroot_stringify($(PYTHON_SOURCE_DATE))" >> $(PYTHON_DIR)/.br_date_override.h
+	echo "#define __TIME__ buildroot_stringify($(PYTHON_SOURCE_TIME))" >> $(PYTHON_DIR)/.br_date_override.h
+endef
+
+PYTHON_POST_PATCH_HOOKS += PYTHON_OVERRIDE_DATE_HOOK
+endif
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
-- 
2.7.4

[Buildroot] [PATCH 20/34] reproducibility/iproute2: do not rely on host directories

[Gilles Chanteperdrix]

In order to find whether iptables/xtables are installed on target
---
 .../0003-avoid-checking-host-directories.patch        | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 package/iproute2/0003-avoid-checking-host-directories.patch

diff --git a/package/iproute2/0003-avoid-checking-host-directories.patch b/package/iproute2/0003-avoid-checking-host-directories.patch
new file mode 100644
index 0000000..ee82e8e
--- /dev/null
+++ b/package/iproute2/0003-avoid-checking-host-directories.patch
@@ -0,0 +1,19 @@
+--- iproute2-4.3.0/configure~	2016-02-27 20:31:57.000000000 +0100
++++ iproute2-4.3.0/configure	2016-02-27 20:32:05.301478415 +0100
+@@ -184,16 +184,6 @@ check_ipt_lib_dir()
+ 		return
+ 	fi
+ 
+-	for dir in /lib /usr/lib /usr/local/lib
+-	do
+-		for file in $dir/{xtables,iptables}/lib*t_*so ; do
+-			if [ -f $file ]; then
+-				echo ${file%/*}
+-				echo "IPT_LIB_DIR:=${file%/*}" >> Config
+-				return
+-			fi
+-		done
+-	done
+ 	echo "not found!"
+ }
+ 
-- 
2.7.4

[Buildroot] [PATCH 21/34] hddtemp: add

[Gilles Chanteperdrix]

---
 package/Config.in            |  1 +
 package/hddtemp/Config.in    |  8 ++++++++
 package/hddtemp/hddtemp.hash |  2 ++
 package/hddtemp/hddtemp.mk   | 19 +++++++++++++++++++
 4 files changed, 30 insertions(+)
 create mode 100644 package/hddtemp/Config.in
 create mode 100644 package/hddtemp/hddtemp.hash
 create mode 100644 package/hddtemp/hddtemp.mk

diff --git a/package/Config.in b/package/Config.in
index bdc3063..76665e6 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1458,6 +1458,7 @@ comment "Utilities"
 	source "package/file/Config.in"
 	source "package/gnupg/Config.in"
 	source "package/gnupg2/Config.in"
+	source "package/hddtemp/Config.in"
 	source "package/inotify-tools/Config.in"
 	source "package/lockfile-progs/Config.in"
 	source "package/logrotate/Config.in"
diff --git a/package/hddtemp/Config.in b/package/hddtemp/Config.in
new file mode 100644
index 0000000..0d28250
--- /dev/null
+++ b/package/hddtemp/Config.in
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_HDDTEMP
+	bool "hddtemp"
+	help
+	  hddtemp is a small utility that gives you the temperature of your
+	  hard drive by reading S.M.A.R.T. informations, for drives that
+	  support this feature. 
+
+	  http://www.guzu.net/linux/hddtemp.php
diff --git a/package/hddtemp/hddtemp.hash b/package/hddtemp/hddtemp.hash
new file mode 100644
index 0000000..c308989
--- /dev/null
+++ b/package/hddtemp/hddtemp.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	618541584054093d53be8a2d9e81c97174f30f00af91cb8700a97e442d79ef5b  hddtemp-0.3-beta15.tar.bz2
diff --git a/package/hddtemp/hddtemp.mk b/package/hddtemp/hddtemp.mk
new file mode 100644
index 0000000..a51548d
--- /dev/null
+++ b/package/hddtemp/hddtemp.mk
@@ -0,0 +1,19 @@
+################################################################################
+#
+# hddtemp
+#
+################################################################################
+
+HDDTEMP_VERSION = 0.3-beta15
+HDDTEMP_SOURCE = hddtemp-$(HDDTEMP_VERSION).tar.bz2
+HDDTEMP_SITE = http://download.savannah.gnu.org/releases/hddtemp
+HDDTEMP_LICENSE = GPLv2
+HDDTEMP_LICENSE_FILES = GPL-2
+
+define HDDTEMP_REMOVE_MAN
+	rm -rf $(TARGET_DIR)/usr/man
+endef
+
+HDDTEMP_POST_INSTALL_TARGET_HOOKS += HDDTEMP_REMOVE_MAN
+
+$(eval $(autotools-package))
-- 
2.7.4

[Buildroot] [PATCH 22/34] High availability: add packages

[Gilles Chanteperdrix]

cluster-glue
corosync
heartbeat
libqb
pacemaker
---
 package/Config.in                                  |   8 ++
 package/cluster-glue/0001-fix-cross-compile.patch  |  26 +++++
 package/cluster-glue/Config.in                     |   9 ++
 package/cluster-glue/cluster-glue.hash             |   2 +
 package/cluster-glue/cluster-glue.mk               |  23 +++++
 package/corosync/Config.in                         |   8 ++
 package/corosync/corosync.hash                     |   2 +
 package/corosync/corosync.mk                       |  17 ++++
 package/heartbeat/0001-fix-cross-compilation.patch | 107 +++++++++++++++++++++
 .../heartbeat/0002-avoid-strip-perm-issue.patch    |  10 ++
 package/heartbeat/Config.in                        |  10 ++
 package/heartbeat/heartbeat.hash                   |   2 +
 package/heartbeat/heartbeat.mk                     |  37 +++++++
 package/libqb/Config.in                            |  11 +++
 package/libqb/libqb.hash                           |   2 +
 package/libqb/libqb.mk                             |  20 ++++
 package/pacemaker/Config.in                        |   9 ++
 package/pacemaker/pacemaker.hash                   |   2 +
 package/pacemaker/pacemaker.mk                     |  32 ++++++
 19 files changed, 337 insertions(+)
 create mode 100644 package/cluster-glue/0001-fix-cross-compile.patch
 create mode 100644 package/cluster-glue/Config.in
 create mode 100644 package/cluster-glue/cluster-glue.hash
 create mode 100644 package/cluster-glue/cluster-glue.mk
 create mode 100644 package/corosync/Config.in
 create mode 100644 package/corosync/corosync.hash
 create mode 100644 package/corosync/corosync.mk
 create mode 100644 package/heartbeat/0001-fix-cross-compilation.patch
 create mode 100644 package/heartbeat/0002-avoid-strip-perm-issue.patch
 create mode 100644 package/heartbeat/Config.in
 create mode 100644 package/heartbeat/heartbeat.hash
 create mode 100644 package/heartbeat/heartbeat.mk
 create mode 100644 package/libqb/Config.in
 create mode 100644 package/libqb/libqb.hash
 create mode 100644 package/libqb/libqb.mk
 create mode 100644 package/pacemaker/Config.in
 create mode 100644 package/pacemaker/pacemaker.hash
 create mode 100644 package/pacemaker/pacemaker.mk

diff --git a/package/Config.in b/package/Config.in
index 76665e6..e09655d 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -902,6 +902,14 @@ menu "Hardware handling"
 	source "package/urg/Config.in"
 endmenu
 
+menu "High availability"
+	source "package/cluster-glue/Config.in"
+	source "package/corosync/Config.in"
+	source "package/heartbeat/Config.in"
+	source "package/libqb/Config.in"
+	source "package/pacemaker/Config.in"
+endmenu
+
 menu "Javascript"
 	source "package/angularjs/Config.in"
 	source "package/bootstrap/Config.in"
diff --git a/package/cluster-glue/0001-fix-cross-compile.patch b/package/cluster-glue/0001-fix-cross-compile.patch
new file mode 100644
index 0000000..2f8bd47
--- /dev/null
+++ b/package/cluster-glue/0001-fix-cross-compile.patch
@@ -0,0 +1,26 @@
+--- cluster-glue-0a7add1d9996/configure.ac~	2014-10-16 19:24:17.000000000 +0200
++++ cluster-glue-0a7add1d9996/configure.ac	2016-02-26 15:39:02.732961215 +0100
+@@ -231,23 +231,6 @@ esac
+ AC_SUBST(INITDIR)
+ 
+ AC_MSG_NOTICE(Sanitizing libdir: ${libdir})
+-case $libdir in
+-  dnl For consistency with Heartbeat, map NONE->$prefix
+-  *prefix*|NONE)
+-    AC_MSG_CHECKING(which lib directory to use)
+-    for aDir in lib64 lib
+-    do
+-      trydir="${exec_prefix}/${aDir}"
+-      if
+-        test -d ${trydir}
+-      then
+-        libdir=${trydir}
+-        break
+-      fi
+-    done
+-    AC_MSG_RESULT($libdir);
+-    ;;
+-esac
+ 
+ DLOPEN_FORCE_FLAGS=""
+ AC_SUBST(DLOPEN_FORCE_FLAGS)
diff --git a/package/cluster-glue/Config.in b/package/cluster-glue/Config.in
new file mode 100644
index 0000000..8b39152
--- /dev/null
+++ b/package/cluster-glue/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_CLUSTER_GLUE
+	bool "cluster-glue"
+	select BR2_PACKAGE_UTIL_LINUX
+	select BR2_PACKAGE_UTIL_LINUX_LIBUUID
+	help
+	  Cluster Glue is a set of libraries, tools and utilities suitable for
+	  the Heartbeat/Pacemaker cluster stack.
+
+	  http://www.linux-ha.org/wiki/Cluster_Glue
diff --git a/package/cluster-glue/cluster-glue.hash b/package/cluster-glue/cluster-glue.hash
new file mode 100644
index 0000000..abd1a74
--- /dev/null
+++ b/package/cluster-glue/cluster-glue.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	c47bfffbfe11401379d5248fc7518a3de9532c02090d614f314037fc3a7c4ceb  0a7add1d9996.tar.bz2
diff --git a/package/cluster-glue/cluster-glue.mk b/package/cluster-glue/cluster-glue.mk
new file mode 100644
index 0000000..abe3697
--- /dev/null
+++ b/package/cluster-glue/cluster-glue.mk
@@ -0,0 +1,23 @@
+################################################################################
+#
+# cluster-glue
+#
+################################################################################
+
+CLUSTER_GLUE_VERSION = 0a7add1d9996
+CLUSTER_GLUE_SOURCE = $(CLUSTER_GLUE_VERSION).tar.bz2
+CLUSTER_GLUE_SITE =  http://hg.linux-ha.org/glue/archive
+CLUSTER_GLUE_LICENSE = LGPLv2.1+ (library), GPLv2+ (programs)
+CLUSTER_GLUE_LICENSE_FILES = COPYING COPYING.LIB
+CLUSTER_GLUE_DEPENDENCIES += host-pkgconf host-python libtool util-linux bzip2 libglib2 libxml2
+
+CLUSTER_GLUE_INSTALL_STAGING = YES
+CLUSTER_GLUE_AUTORECONF = YES
+
+CLUSTER_GLUE_CONF_ENV += PYTHON=$(HOST_DIR)/usr/bin/python
+
+CLUSTER_GLUE_CONF_OPTS += \
+	--with-initdir=/etc/init.d \
+	ac_cv_path_XML2CONFIG="$(STAGING_DIR)/usr/bin/xml2-config"
+
+$(eval $(autotools-package))
diff --git a/package/corosync/Config.in b/package/corosync/Config.in
new file mode 100644
index 0000000..3855136
--- /dev/null
+++ b/package/corosync/Config.in
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_COROSYNC
+	bool "corosync"
+	help
+	  The Corosync Cluster Engine is a Group Communication System with
+	  additional features for implementing high availability within
+	  applications. 
+
+	  https://corosync.github.io/corosync/
diff --git a/package/corosync/corosync.hash b/package/corosync/corosync.hash
new file mode 100644
index 0000000..dbfddd0
--- /dev/null
+++ b/package/corosync/corosync.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	64ee50f783a4bfa8a63fd576090cf7dd1810f3c60e01b4886723b87c5ff2b013  corosync-1.4.8.tar.gz
diff --git a/package/corosync/corosync.mk b/package/corosync/corosync.mk
new file mode 100644
index 0000000..7d36c75
--- /dev/null
+++ b/package/corosync/corosync.mk
@@ -0,0 +1,17 @@
+################################################################################
+#
+# corosync
+#
+################################################################################
+
+COROSYNC_VERSION = 1.4.8
+COROSYNC_SOURCE = corosync-$(COROSYNC_VERSION).tar.gz
+COROSYNC_SITE = http://build.clusterlabs.org/corosync/releases
+COROSYNC_LICENSE = BSD-3c except for crypto.[ch], public domain for crypto.[ch]
+COROSYNC_LICENSE_FILES = LICENSE
+COROSYNC_INSTALL_STAGING = YES
+COROSYNC_DEPENDENCIES += host-pkgconf libnss
+
+COROSYNC_CONF_OPTS += ac_cv_func_fnmatch_works=yes
+
+$(eval $(autotools-package))
diff --git a/package/heartbeat/0001-fix-cross-compilation.patch b/package/heartbeat/0001-fix-cross-compilation.patch
new file mode 100644
index 0000000..4a653f6
--- /dev/null
+++ b/package/heartbeat/0001-fix-cross-compilation.patch
@@ -0,0 +1,107 @@
+--- Heartbeat-3-0-958e11be8686/configure.ac~	2016-02-25 19:13:54.488125586 +0100
++++ Heartbeat-3-0-958e11be8686/configure.ac	2016-02-26 12:28:12.366938641 +0100
+@@ -274,55 +274,7 @@ dnl Test to see whether library director
+ dnl ************************************************************************
+ 
+ AC_MSG_CHECKING(for proper library directory suffix)
+-tmpdir=$srcdir
+-tmpCfile=${tmpdir}/$$test.c
+-tmpOutfile=${tmpdir}/$$test
+-echo 'int main(int argc, char** argv) { return(1);}' >$tmpCfile
+-if
+-  ${CC} ${CFLAGS} ${tmpCfile} -o ${tmpOutfile}
+-then
+-  LIBC=`${LDD} ${tmpOutfile} | grep libc | sed -e 's%.*=> *%%' -e 's% .*$%%'`
+-  LibCdir=`dirname $LIBC`
+-  dirlist=`echo $LibCdir | tr '/' ' '`
+-  LibDirSuffix=unknown
+-  for dir in $dirlist
+-  do
+-    case $dir in
+-      *lib*)	LibDirSuffix=$dir; break;;
+-      *);;
+-    esac
+-  done
+-  case $LibDirSuffix in
+-    unknown)	LibDirSuffix=`basename $LibCdir`;;
+-  esac
+-  OutFileType=`file $tmpOutfile`
+-  rm -f $tmpCfile $tmpOutfile
+-else
+-  AC_MSG_ERROR([Cannot Compile trivial C program])
+-fi
+-#
+-#	The code above doesn't work right everywhere
+-#		(like Fedora and OpenBSD) 
+-#
+-case ${LibDirSuffix} in
+-	*lib*)	: Cool;;
+-	*)	: Sigh...
+-		case $OutFileType in
+-		  *64-bit*)	
+-		  	case $host_os in
+-			  openbsd*)	LibDirSuffix=lib;;
+-		  	  *)		LibDirSuffix=lib64;;
+-			esac;;
+-		  *32-bit*)	LibDirSuffix=lib;;
+-		  *)		LibDirSuffix=lib;;
+-		esac;;
+-esac
+-#
+-#	This may not yet be quite right for PPC where the default
+-#	is to produce 32-bit binaries, even though the OS is 64-bit
+-#	or for that matter for system Z, But, it's a lot better than
+-#	it used to be.
+-#
++LibDirSuffix=lib
+ AC_MSG_RESULT($LibDirSuffix)
+ 
+ case $libdir in
+@@ -386,7 +338,6 @@ else
+    AC_MSG_FAILURE(Core development headers were not found)
+ fi
+ 
+-CPPFLAGS="$CPPFLAGS -I${prefix}/include/heartbeat"
+ AC_CHECK_HEADERS(glue_config.h)
+ if test "$ac_cv_header_glue_config_h" != "yes";  then
+    AC_MSG_FAILURE(Core development headers were not found)
+@@ -2022,24 +1973,11 @@ AC_CHECK_HEADERS(bzlib.h, , [bz2_install
+ AC_CHECK_LIB(bz2, BZ2_bzBuffToBuffCompress , , [bz2_installed="no"])
+ AM_CONDITIONAL(BUILD_BZ2_COMPRESS_MODULE, test "x${bz2_installed}" = "xyes")
+ 
+-dnl check if there are getpid() inconsistency
+-dnl	Note: reduce LIBS; in particular, ltdl can cause confusion.
+-dnl	Possibly better:  move 'LIBS="$LIBS -lltdl"' from above to beyond here.
+-dnl
+-AC_MSG_CHECKING(for getpid() consistency in multi-process/threads program)
+-ac_save_LIBS=$LIBS
+-LIBS="-lpthread"
+-AC_TRY_RUN(`cat $srcdir/config/pidtest.c`, 
+-AC_MSG_RESULT(ok), 
+-[AC_MSG_RESULT(fail); AC_DEFINE(GETPID_INCONSISTENT, 1 , [pid inconsistent])],)
+-LIBS=$ac_save_LIBS
+-
+ dnl check byte order
+ AC_MSG_CHECKING(for byteorder)
+-AC_TRY_RUN(`cat $srcdir/config/byteorder_test.c`,
+-[AC_MSG_RESULT(little-endian); AC_DEFINE(CONFIG_LITTLE_ENDIAN, 1, [little-endian])],
+-[AC_MSG_RESULT(big-endian); AC_DEFINE(CONFIG_BIG_ENDIAN, 1, [big-endian])],)
+-
++AC_C_BIGENDIAN(
++	[AC_MSG_RESULT(big-endian); AC_DEFINE(CONFIG_BIG_ENDIAN, 1, [big-endian])],
++	[AC_MSG_RESULT(little-endian); AC_DEFINE(CONFIG_LITTLE_ENDIAN, 1, [little-endian])])
+ 
+ dnl
+ dnl Lex and yacc can't be trusted to produce code that won't produce
+--- heartbeat-958e11be8686/membership/ccm/Makefile.am~	2015-02-09 17:04:41.000000000 +0100
++++ heartbeat-958e11be8686/membership/ccm/Makefile.am	2016-02-26 13:06:41.678114297 +0100
+@@ -21,8 +21,7 @@ MAINTAINERCLEANFILES    = Makefile.in
+ 
+ AM_CPPFLAGS		= -I$(top_builddir)/include -I$(top_srcdir)/include \
+ 			-I$(top_builddir)/linux-ha -I$(top_srcdir)/linux-ha \
+-			-I$(top_builddir)/libltdl -I$(top_srcdir)/libltdl \
+-                        -I$(includedir)/clplumbing
++			-I$(top_builddir)/libltdl -I$(top_srcdir)/libltdl
+ 
+ halibexecdir		= $(libexecdir)/@HB_PKG@
+ ccmlibdir		= $(halibexecdir)
diff --git a/package/heartbeat/0002-avoid-strip-perm-issue.patch b/package/heartbeat/0002-avoid-strip-perm-issue.patch
new file mode 100644
index 0000000..f4e805a
--- /dev/null
+++ b/package/heartbeat/0002-avoid-strip-perm-issue.patch
@@ -0,0 +1,10 @@
+--- Heartbeat-3-0-958e11be8686/tools/Makefile.am~	2015-02-09 17:04:41.000000000 +0100
++++ Heartbeat-3-0-958e11be8686/tools/Makefile.am	2016-02-27 01:13:40.102433992 +0100
+@@ -49,6 +49,6 @@ cl_respawn_LDADD	= -lplumb	\
+ 
+ install-data-hook:    # install-exec-hook doesn't work (!)
+ 	-chgrp $(apigid) $(DESTDIR)/$(habindir)/cl_status
+-	-chmod g+s,a-w $(DESTDIR)/$(habindir)/cl_status
++	-chmod g+s,g-w,o-w $(DESTDIR)/$(habindir)/cl_status
+ 
+ .PHONY: install-exec-hook
diff --git a/package/heartbeat/Config.in b/package/heartbeat/Config.in
new file mode 100644
index 0000000..0f8ee80
--- /dev/null
+++ b/package/heartbeat/Config.in
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_HEARTBEAT
+	bool "heartbeat"
+	select BR2_PACKAGE_UTIL_LINUX
+	select BR2_PACKAGE_UTIL_LINUX_LIBUUID
+	select BR2_PACKAGE_IPROUTE2
+	help
+	  Heartbeat is a daemon that provides cluster infrastructure
+	  (communication and membership) services to its clients.
+
+	  http://www.linux-ha.org/wiki/Heartbeat
diff --git a/package/heartbeat/heartbeat.hash b/package/heartbeat/heartbeat.hash
new file mode 100644
index 0000000..ff85472
--- /dev/null
+++ b/package/heartbeat/heartbeat.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	851d2add2c129fef9fede764fec80229e1f6e7295e0e979950d10258648b462c  958e11be8686.tar.bz2
diff --git a/package/heartbeat/heartbeat.mk b/package/heartbeat/heartbeat.mk
new file mode 100644
index 0000000..eedf40f
--- /dev/null
+++ b/package/heartbeat/heartbeat.mk
@@ -0,0 +1,37 @@
+################################################################################
+#
+# heartbeat
+#
+################################################################################
+HEARTBEAT_VERSION = 958e11be8686
+HEARTBEAT_SOURCE = $(HEARTBEAT_VERSION).tar.bz2
+HEARTBEAT_SITE =  http://hg.linux-ha.org/heartbeat-STABLE_3_0/archive
+HEARTBEAT_LICENSE = LGPLv2.1+ (library), GPLv2+ (programs)
+HEARTBEAT_LICENSE_FILES = doc/COPYING doc/COPYING.LGPL
+HEARTBEAT_DEPENDENCIES += host-pkgconf host-python libtool util-linux bzip2 cluster-glue
+HEARTBEAT_INSTALL_STAGING = YES
+HEARTBEAT_AUTORECONF = YES
+
+HEARTBEAT_CONF_ENV += PYTHON=$(HOST_DIR)/usr/bin/python
+
+HEARTBEAT_CONF_OPTS += \
+	CPPFLAGS="$(TARGET_CPPFLAGS) -I$(STAGING_DIR)/usr/include/heartbeat -I$(STAGING_DIR)/usr/include/clplumbing" \
+	--with-initdir=/etc/init.d \
+	--with-initdefdir=/etc/sysconfig \
+	ac_cv_path_IP2UTIL=/sbin/ip
+
+ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
+define HEARTBEAT_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/share/heartbeat -name '*.py' -print0 | xargs -0 rm -f
+endef
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
+define HEARTBEAT_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/share/heartbeat -name '*.py[co]' -print0 | xargs -0 rm -f
+endef
+endif
+
+TARGET_FINALIZE_HOOKS += HEARTBEAT_FINALIZE_TARGET
+
+$(eval $(autotools-package))
diff --git a/package/libqb/Config.in b/package/libqb/Config.in
new file mode 100644
index 0000000..bafe166
--- /dev/null
+++ b/package/libqb/Config.in
@@ -0,0 +1,11 @@
+config BR2_PACKAGE_LIBQB
+	bool "libqb"
+	help
+	  libqb is a library with the primary purpose of providing high
+	  performance client server reusable features. It provides high
+	  performance logging, tracing, ipc, and poll.
+
+	  The initial features of libqb comes from the features of corosync that
+	  were thought to useful to other projects.
+
+	  https://github.com/ClusterLabs/libqb/wiki
diff --git a/package/libqb/libqb.hash b/package/libqb/libqb.hash
new file mode 100644
index 0000000..34afcb3
--- /dev/null
+++ b/package/libqb/libqb.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	11d69568a3b99c841535fdeacdad92f0d95af73a73beb77a3912dd29dc434744 libqb-0.17.2.tar.gz
diff --git a/package/libqb/libqb.mk b/package/libqb/libqb.mk
new file mode 100644
index 0000000..a300626
--- /dev/null
+++ b/package/libqb/libqb.mk
@@ -0,0 +1,20 @@
+################################################################################
+#
+# libqb
+#
+################################################################################
+
+LIBQB_VERSION = 0.17.2
+LIBQB_SITE = $(call github,ClusterLabs,libqb,v$(LIBQB_VERSION))
+LIBQB_SOURCE = libqb-$(LIBQB_VERSION).tar.gz
+LIBQB_LICENSE = LGPLv2.1+
+LIBQB_LICENSE_FILES = COPYING
+LIBQB_INSTALL_STAGING = YES
+LIBQB_AUTORECONF = YES
+
+define LIBQB_CREATE_M4_DIR
+	mkdir -p $(@D)/m4
+endef
+LIBQB_POST_PATCH_HOOKS += LIBQB_CREATE_M4_DIR
+
+$(eval $(autotools-package))
diff --git a/package/pacemaker/Config.in b/package/pacemaker/Config.in
new file mode 100644
index 0000000..3719bb5
--- /dev/null
+++ b/package/pacemaker/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PACEMAKER
+	bool "pacemaker"
+	select BR2_PACKAGE_UTIL_LINUX
+	select BR2_PACKAGE_UTIL_LINUX_LIBUUID
+	help
+	  High-Availability cluster resource manager for Heartbeat as
+	  well as Corosync. 
+
+	  http://clusterlabs.org/
diff --git a/package/pacemaker/pacemaker.hash b/package/pacemaker/pacemaker.hash
new file mode 100644
index 0000000..f06a18c
--- /dev/null
+++ b/package/pacemaker/pacemaker.hash
@@ -0,0 +1,2 @@
+# Locally calculated after checking pgp signature
+sha256	2f73043a5de5e66a47462d37cd68c4c6861f71a03c0ea02eb4ae2b1a550cd7b0  pacemaker-Pacemaker-1.1.14.tar.gz
diff --git a/package/pacemaker/pacemaker.mk b/package/pacemaker/pacemaker.mk
new file mode 100644
index 0000000..d987277
--- /dev/null
+++ b/package/pacemaker/pacemaker.mk
@@ -0,0 +1,32 @@
+################################################################################
+#
+# pacemaker
+#
+################################################################################
+
+PACEMAKER_VERSION = Pacemaker-1.1.14
+PACEMAKER_SITE = $(call github,ClusterLabs,pacemaker,$(PACEMAKER_VERSION))
+PACEMAKER_LICENSE = LGPLv2.1 (library), GPLv2 (programs)
+PACEMAKER_LICENSE_FILES = COPYING COPYING.LIB
+PACEMAKER_DEPENDENCIES += host-pkgconf host-python libtool util-linux bzip2 libxml2 libxslt corosync heartbeat libqb
+PACEMAKER_INSTALL_STAGING = YES
+PACEMAKER_AUTORECONF = YES
+
+PACEMAKER_CONF_ENV += PYTHON=$(HOST_DIR)/usr/bin/python
+
+# This program configure script expects the proper -I flags in CFLAGS and not
+# CPPFLAGS
+PACEMAKER_CONF_OPTS += \
+	CFLAGS="$(TARGET_CFLAGS) -I$(STAGING_DIR)/usr/include -I$(STAGING_DIR)/usr/include/heartbeat" \
+	ac_cv_path_HTML2TXT="" \
+	ac_cv_path_HELP2MAN="" \
+	ac_cv_path_POD2MAN="" \
+	ac_cv_path_ASCIIDOC="" \
+	ac_cv_path_PUBLICAN="" \
+	ac_cv_path_INKSCAPE="" \
+	ac_cv_path_XSLTPROC="" \
+	ac_cv_path_XMLCATALOG="" \
+	ac_cv_path_FOP="" \
+	ac_cv_path_XML2CONFIG="$(STAGING_DIR)/usr/bin/xml2-config"
+
+$(eval $(autotools-package))
-- 
2.7.4

[Buildroot] [PATCH 23/34] pkg-download: allow git+ssh URLs

[Gilles Chanteperdrix]

---
 package/pkg-download.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index c86b9ab..f8c5221 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -228,7 +228,7 @@ define DOWNLOAD_INNER
 			scheme="$($(PKG)_SITE_METHOD)" ; \
 		fi ; \
 		case "$$scheme" in \
-			git) $($(3)_GIT) && exit ;; \
+			git|git+ssh) $($(3)_GIT) && exit ;; \
 			svn) $($(3)_SVN) && exit ;; \
 			cvs) $($(3)_CVS) && exit ;; \
 			bzr) $($(3)_BZR) && exit ;; \
-- 
2.7.4

[Buildroot] [PATCH 24/34] grub2: really disable liblzma

[Gilles Chanteperdrix]

---
 boot/grub2/grub2.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/boot/grub2/grub2.mk b/boot/grub2/grub2.mk
index 88a46f4..112a401 100644
--- a/boot/grub2/grub2.mk
+++ b/boot/grub2/grub2.mk
@@ -62,7 +62,7 @@ GRUB2_CONF_OPTS = \
 	--with-platform=$(GRUB2_PLATFORM) \
 	--disable-grub-mkfont \
 	--enable-efiemu=no \
-	--enable-liblzma=no \
+	ac_cv_lib_lzma_lzma_code=no \
 	--enable-device-mapper=no \
 	--enable-libzfs=no \
 	--disable-werror
-- 
2.7.4

[Buildroot] [PATCH 25/34] grub2: allow full install (for installer images)

[Gilles Chanteperdrix]

---
 boot/grub2/Config.in | 10 ++++++++++
 boot/grub2/grub2.mk  |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/boot/grub2/Config.in b/boot/grub2/Config.in
index c465a4d..5e41b89 100644
--- a/boot/grub2/Config.in
+++ b/boot/grub2/Config.in
@@ -139,6 +139,14 @@ config BR2_TARGET_GRUB2_X86_64_EFI
 
 endchoice
 
+config BR2_TARGET_FULL_INSTALL
+	bool "full install on target"
+	help
+	  Select this option to install grub on target rather than
+	  generating images.
+
+if !BR2_TARGET_FULL_INSTALL
+
 if BR2_TARGET_GRUB2_I386_PC
 
 config BR2_TARGET_GRUB2_BOOT_PARTITION
@@ -166,4 +174,6 @@ config BR2_TARGET_GRUB2_BUILTIN_CONFIG
 	  device and other configuration parameters, but however menu
 	  entries cannot be described in this embedded configuration.
 
+endif
+
 endif # BR2_TARGET_GRUB2
diff --git a/boot/grub2/grub2.mk b/boot/grub2/grub2.mk
index 112a401..e2352b6 100644
--- a/boot/grub2/grub2.mk
+++ b/boot/grub2/grub2.mk
@@ -67,6 +67,7 @@ GRUB2_CONF_OPTS = \
 	--enable-libzfs=no \
 	--disable-werror
 
+ifneq ($(BR2_TARGET_FULL_INSTALL),y)
 # We don't want all the native tools and Grub2 modules to be installed
 # in the target. So we in fact install everything into the host
 # directory, and the image generation process (below) will use the
@@ -103,5 +104,6 @@ define GRUB2_EFI_STARTUP_NSH
 endef
 GRUB2_POST_INSTALL_TARGET_HOOKS += GRUB2_EFI_STARTUP_NSH
 endif
+endif
 
 $(eval $(autotools-package))
-- 
2.7.4

[Buildroot] [PATCH 26/34] reproducibility: cpio: avoid embedding file dates in archive

[Gilles Chanteperdrix]

---
 fs/cpio/cpio.mk | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index e82167e..07be2a8 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -27,8 +27,18 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define ROOTFS_CPIO_REPRODUCIBLE_TOUCH
+	find $(TARGET_DIR) -print0 | xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH)
+endef
+else
+define ROOTFS_CPIO_REPRODUCIBLE_TOUCH
+	true
+endef
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	$(ROOTFS_CPIO_REPRODUCIBLE_TOUCH) && cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
 endef
 
 $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
-- 
2.7.4

[Buildroot] [PATCH 27/34] package/cpio: allow generating host-cpio

[Gilles Chanteperdrix]

In order to be sure to have a cpio binar on host which supports
--reproducible.
---
 package/cpio/cpio.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/cpio/cpio.mk b/package/cpio/cpio.mk
index 1576127..87b3df8 100644
--- a/package/cpio/cpio.mk
+++ b/package/cpio/cpio.mk
@@ -18,3 +18,9 @@ CPIO_DEPENDENCIES += argp-standalone
 endif
 
 $(eval $(autotools-package))
+
+HOST_CPIO_DEPENDENCIES =
+
+$(eval $(host-autotools-package))
+
+CPIO_REPRODUCIBLE = $(HOST_DIR)/usr/bin/cpio --reproducible
-- 
2.7.4

[Buildroot] [PATCH 28/34] reproducibility/fs/cpio: generate archive with host-cpio

[Gilles Chanteperdrix]

Using the --reproducible option to generate a reproducible archive.
---
 fs/cpio/cpio.mk | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index 07be2a8..e0e925d 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -27,19 +27,19 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
-ifeq ($(BR2_REPRODUCIBLE),y)
-define ROOTFS_CPIO_REPRODUCIBLE_TOUCH
-	find $(TARGET_DIR) -print0 | xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH)
+ifneq ($(BR2_REPRODUCIBLE),y)
+define ROOTFS_CPIO_CMD
+	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
 endef
 else
-define ROOTFS_CPIO_REPRODUCIBLE_TOUCH
-	true
-endef
-endif
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
 
 define ROOTFS_CPIO_CMD
-	$(ROOTFS_CPIO_REPRODUCIBLE_TOUCH) && cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	find $(TARGET_DIR) -print0 | \
+	xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH) && \
+	cd $(TARGET_DIR) && find . | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
 endef
+endif
 
 $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
 	$(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
-- 
2.7.4

[Buildroot] [PATCH 29/34] cdrkit: add patch allowing to set creation date

[Gilles Chanteperdrix]

The patch was posted here:
https://lists.gnu.org/archive/html/bug-cpio/2014-08/msg00000.html

And allows generating reproducible iso images with genisoimage.
---
 ...-reproducible-allow-setting-creation-date.patch | 140 +++++++++++++++++++++
 1 file changed, 140 insertions(+)
 create mode 100644 package/cdrkit/0003-reproducible-allow-setting-creation-date.patch

diff --git a/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch b/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch
new file mode 100644
index 0000000..f5ed465
--- /dev/null
+++ b/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch
@@ -0,0 +1,140 @@
+diff --git a/genisoimage/.eltorito.c.swp b/genisoimage/.eltorito.c.swp
+new file mode 100644
+index 0000000..a49fcbd
+Binary files /dev/null and b/genisoimage/.eltorito.c.swp differ
+diff --git a/genisoimage/genisoimage.1 b/genisoimage/genisoimage.1
+index d05b24a..d69a1d4 100644
+--- a/genisoimage/genisoimage.1
++++ b/genisoimage/genisoimage.1
+@@ -976,6 +976,12 @@ in the
+ .I .genisoimagerc
+ file.
+ .TP
++.BI \-creation-date " epoch"
++Specifies the date to be used as creation, modification and effective
++date in the volume descriptor and for files and relocations created
++on the fly. Specified as a number of second since
++1970-01-01 00:00:00 +0000 (UTC); if 0, the current time is used.
++.TP
+ .B \-print\-size
+ Print estimated filesystem size in multiples of the sector size (2048 bytes)
+ and exit. This option is needed for
+diff --git a/genisoimage/genisoimage.c b/genisoimage/genisoimage.c
+index cfd079a..58397e9 100644
+--- a/genisoimage/genisoimage.c
++++ b/genisoimage/genisoimage.c
+@@ -169,6 +169,7 @@ char	*abstract = ABSTRACT_DEFAULT;
+ char	*volset_id = VOLSET_ID_DEFAULT;
+ char	*volume_id = VOLUME_ID_DEFAULT;
+ char	*system_id = SYSTEM_ID_DEFAULT;
++time_t  creation_date = 0;
+ char	*boot_catalog = BOOT_CATALOG_DEFAULT;
+ char	*boot_image = BOOT_IMAGE_DEFAULT;
+ char	*genboot_image = BOOT_IMAGE_DEFAULT;
+@@ -405,6 +406,8 @@ struct ld_option {
+ #define	OPTION_ALLOW_LEADING_DOTS	1070
+ #define	OPTION_PUBLISHER		1071
+ 
++#define OPTION_CREATION_DATE            1072
++
+ #ifdef		JIGDO_TEMPLATE
+ #define	OPTION_JTT_OUTPUT		1101
+ #define	OPTION_JTJ_OUTPUT		1102
+@@ -522,6 +525,8 @@ static const struct ld_option ld_options[] =
+ 	'\0', "FILE", "Check all ISO9660 names from previous session", ONE_DASH},
+ 	{{"copyright", required_argument, NULL, OPTION_COPYRIGHT},
+ 	'\0', "FILE", "Set Copyright filename", ONE_DASH},
++	{{"creation-date", required_argument, NULL, OPTION_CREATION_DATE},
++	'\0', NULL, "Set volume creation date", ONE_DASH},
+ 	{{"debug", no_argument, NULL, OPTION_DEBUG},
+ 	'\0', NULL, "Set debug flag", ONE_DASH},
+ 	{{"eltorito-boot", required_argument, NULL, 'b'},
+@@ -1721,6 +1726,22 @@ int main(int argc, char *argv[])
+ #endif
+ 			}
+ 			break;
++		case OPTION_CREATION_DATE:
++		{
++			char	*end = 0;
++
++			creation_date = strtol(optarg, &end, 10);
++			if (!end || *end != 0) {
++#ifdef	USE_LIBSCHILY
++				comerrno(EX_BAD, "Bad epoch for -creation-date\n");
++#else
++				fprintf(stderr, "Bad epoch for -creation-date\n");
++				exit(1);
++#endif
++			}
++			break;
++		}
++
+ 		case OPTION_DEBUG:
+ 			debug++;
+ 			break;
+diff --git a/genisoimage/genisoimage.h b/genisoimage/genisoimage.h
+index bbedfb0..c49576c 100644
+--- a/genisoimage/genisoimage.h
++++ b/genisoimage/genisoimage.h
+@@ -650,6 +650,7 @@ extern char	*appid;
+ extern char	*volset_id;
+ extern char	*system_id;
+ extern char	*volume_id;
++extern time_t	creation_date;
+ extern char	*boot_catalog;
+ extern char	*boot_image;
+ extern char	*genboot_image;
+diff --git a/genisoimage/tree.c b/genisoimage/tree.c
+index 7805888..f17a662 100644
+--- a/genisoimage/tree.c
++++ b/genisoimage/tree.c
+@@ -783,7 +783,11 @@ generate_reloc_directory()
+ 	struct directory_entry *s_entry;
+ 
+ 	/* Create an  entry for our internal tree */
+-	time(&current_time);
++	if (creation_date == 0) {
++		time(&current_time);
++	} else {
++		current_time = creation_date;
++	}
+ 	reloc_dir = (struct directory *)
+ 		e_malloc(sizeof (struct directory));
+ 	memset(reloc_dir, 0, sizeof (struct directory));
+@@ -2680,7 +2684,11 @@ init_fstatbuf()
+ 	time_t	current_time;
+ 
+ 	if (fstatbuf.st_ctime == 0) {
+-		time(&current_time);
++		if (creation_date == 0) {
++			time(&current_time);
++		} else {
++			current_time = creation_date;
++		}
+ 		if (rationalize_uid)
+ 			fstatbuf.st_uid = uid_to_use;
+ 		else
+diff --git a/genisoimage/write.c b/genisoimage/write.c
+index a423ab1..f63507c 100644
+--- a/genisoimage/write.c
++++ b/genisoimage/write.c
+@@ -1885,12 +1885,17 @@ pvd_write(FILE *outfile)
+ 	int		should_write;
+ 	struct tm	local;
+ 	struct tm	gmt;
++	time_t		pvd_date;
+ 
+ 
+ 	time(&begun);
+ 
+-	local = *localtime(&begun);
+-	gmt = *gmtime(&begun);
++	if (creation_date == 0) {
++		creation_date = begun;
++	}
++
++	local = *localtime(&creation_date);
++	gmt = *gmtime(&creation_date);
+ 
+ 	/*
+ 	 * There was a comment here about breaking in the year 2000.
-- 
2.7.4

[Buildroot] [PATCH 30/34] reproducibility/fs/iso9660: reproducible iso images

[Gilles Chanteperdrix]

When reproducibility is required, we:
- fix the iso files timestamps to avoid depending on the build date
- run genisoimage with the -creation-date option to avoid using the
  build date for files in the iso image automatically created by
  genisoimage (such as the catalog)
- disable the rockridge extension, as it stores each file "ctime", which
  can not be changed easily on the host filesystem
- force using an initrd, as a trying and booting an iso image without
  the rockridge extensions will not work
- use isohybrid -i option to avoid embedding a random number in the
  generated iso.
---
 fs/iso9660/Config.in  |  9 +++++++++
 fs/iso9660/iso9660.mk | 22 ++++++++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/fs/iso9660/Config.in b/fs/iso9660/Config.in
index 2482e44..bb82d03 100644
--- a/fs/iso9660/Config.in
+++ b/fs/iso9660/Config.in
@@ -71,6 +71,15 @@ config BR2_TARGET_ROOTFS_ISO9660_INITRD
 	  contain a kernel image, an initrd image (unless an initramfs
 	  linked into the kernel is used) and the bootloader.
 
+config BR2_TARGET_ROOTFS_ISO9660_REPRODUCIBLE
+       def_bool BR2_REPRODUCIBLE
+       select BR2_TARGET_ROOTFS_ISO9660_INITRD
+
+if BR2_REPRODUCIBLE
+comment "A reproducible build requires disabling the RockRidge extensions"
+comment "Which makes using an initrd the only sane solution"
+endif
+
 config BR2_TARGET_ROOTFS_ISO9660_HYBRID
 	bool "Build hybrid image"
 	depends on BR2_TARGET_ROOTFS_ISO9660_ISOLINUX
diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index 32d79d7..5e36ed1 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -143,15 +143,33 @@ ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_DISABLE_EXTERNAL_INITRD
 endif # ROOTFS_ISO9660_USE_INITRD
 
 
+ifneq ($(BR2_REPRODUCIBLE),y)
+define ROOTFS_ISO9660_GENISOIMAGE
+	$(HOST_DIR)/usr/bin/genisoimage -R
+endef
+define ROOTFS_ISO9660_ISOHYBRID
+	$(HOST_DIR)/usr/bin/isohybrid
+endef
+else
+define ROOTFS_ISO9660_GENISOIMAGE
+	find $(ROOTFS_ISO9660_TARGET_DIR) -print0 | \
+	xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH) && \
+	$(HOST_DIR)/usr/bin/genisoimage -creation-date $(SOURCE_DATE_EPOCH)
+endef
+define ROOTFS_ISO9660_ISOHYBRID
+	$(HOST_DIR)/usr/bin/isohybrid -i $(SOURCE_DATE_EPOCH)
+endef
+endif
+
 define ROOTFS_ISO9660_CMD
-	$(HOST_DIR)/usr/bin/genisoimage -J -R -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
+	$(ROOTFS_ISO9660_GENISOIMAGE) -J -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
 		-no-emul-boot -boot-load-size 4 -boot-info-table \
 		-o $@ $(ROOTFS_ISO9660_TARGET_DIR)
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
 define ROOTFS_ISO9660_GEN_HYBRID
-	$(HOST_DIR)/usr/bin/isohybrid -t 0x96 $@
+	$(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
 endef
 
 ROOTFS_ISO9660_POST_GEN_HOOKS += ROOTFS_ISO9660_GEN_HYBRID
-- 
2.7.4

[Buildroot] [PATCH 31/34] reproducible/syslinux: make syslinux build reproducible

[Gilles Chanteperdrix]

Build with the target toolchain so that the binaries are identical with
different host toolchains.
Sort files lists in order to get deterministic link order.
Build with HEXDATE set to the source date epoch.
---
 boot/syslinux/0001-fixed-build-order.patch | 37 +++++++++++++++++++++++
 boot/syslinux/syslinux.mk                  | 47 +++++++++++++++++++++++++-----
 fs/iso9660/iso9660.mk                      |  4 +--
 3 files changed, 79 insertions(+), 9 deletions(-)
 create mode 100644 boot/syslinux/0001-fixed-build-order.patch

diff --git a/boot/syslinux/0001-fixed-build-order.patch b/boot/syslinux/0001-fixed-build-order.patch
new file mode 100644
index 0000000..4eda186
--- /dev/null
+++ b/boot/syslinux/0001-fixed-build-order.patch
@@ -0,0 +1,37 @@
+--- syslinux-6.03/core/Makefile~	2014-10-06 18:27:44.000000000 +0200
++++ syslinux-6.03/core/Makefile	2016-03-24 14:36:38.207391899 +0100
+@@ -41,9 +41,9 @@
+ # All primary source files for the main syslinux files
+ NASMSRC	 := $(wildcard $(SRC)/*.asm)
+ NASMHDR  := $(wildcard $(SRC)/*.inc)
+-CSRC	 := $(shell find $(SRC) -name '*.c' -print)
+-SSRC	 := $(shell find $(SRC) -name '*.S' -print)
+-CHDR	 := $(shell find $(SRC) -name '*.h' -print)
++CSRC	 := $(shell find $(SRC) -name '*.c' -print | sort)
++SSRC	 := $(shell find $(SRC) -name '*.S' -print | sort)
++CHDR	 := $(shell find $(SRC) -name '*.h' -print | sort)
+ OTHERSRC := keywords
+ ALLSRC    = $(NASMSRC) $(NASMHDR) $(CSRC) $(SSRC) $(CHDR) $(OTHERSRC)
+ 
+@@ -56,18 +56,18 @@
+ 	$(addprefix $(SRC)/fs/pxe/, dhcp_option.c pxe.c tftp.c urlparse.c bios.c)
+ 
+ LPXELINUX_CSRC = $(CORE_PXE_CSRC) \
+-	$(shell find $(SRC)/lwip -name '*.c' -print) \
++	$(shell find $(SRC)/lwip -name '*.c' -print | sort) \
+ 	$(addprefix $(SRC)/fs/pxe/, \
+ 		core.c dnsresolv.c ftp.c ftp_readdir.c gpxeurl.c http.c \
+ 		http_readdir.c idle.c isr.c tcp.c)
+ 
+ PXELINUX_CSRC = $(CORE_PXE_CSRC) \
+-	$(shell find $(SRC)/legacynet -name '*.c' -print)
++	$(shell find $(SRC)/legacynet -name '*.c' -print | sort)
+ 
+ LPXELINUX_OBJS = $(subst $(SRC)/,,$(LPXELINUX_CSRC:%.c=%.o))
+ PXELINUX_OBJS  = $(subst $(SRC)/,,$(PXELINUX_CSRC:%.c=%.o))
+ 
+-UNITTEST_CSRC = $(shell find $(SRC) -path '*/tests/*.c' -print)
++UNITTEST_CSRC = $(shell find $(SRC) -path '*/tests/*.c' -print | sort)
+ UNITTEST_OBJS = $(subst $(SRC)/,,$(UNITTEST_CSRC:%.c=%.o))
+ 
+ # Don't include console and network stack specific objects or unit tests
diff --git a/boot/syslinux/syslinux.mk b/boot/syslinux/syslinux.mk
index 0868896..4486e95 100644
--- a/boot/syslinux/syslinux.mk
+++ b/boot/syslinux/syslinux.mk
@@ -13,7 +13,7 @@ SYSLINUX_LICENSE_FILES = COPYING
 
 SYSLINUX_INSTALL_IMAGES = YES
 
-SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx
+SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx host-perl host-python host-xz
 
 ifeq ($(BR2_TARGET_SYSLINUX_LEGACY_BIOS),y)
 SYSLINUX_TARGET = bios
@@ -47,12 +47,35 @@ define SYSLINUX_CLEANUP
 endef
 SYSLINUX_POST_PATCH_HOOKS += SYSLINUX_CLEANUP
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define SYSLINUX_REPRODUCIBLE
+	HEXDATE="`printf "0x%x" $(SOURCE_DATE_EPOCH)`"
+endef
+endif
+
+define SYSLINUX_MAKE
+	$(TARGET_MAKE_ENV) $(MAKE1) \
+		$(SYSLINUX_REPRODUCIBLE) \
+		NASM=$(HOST_DIR)/usr/bin/nasm \
+		PERL=$(HOST_DIR)/usr/bin/perl \
+		PYTHON=$(HOST_DIR)/usr/bin/python \
+		UPX=$(HOST_DIR)/usr/bin/upx \
+		CC="$(TARGET_CC)" \
+		LD="$(TARGET_LD) -m elf_i386" \
+		OBJDUMP="$(TARGET_OBJDUMP)" \
+		OBJCOPY="$(TARGET_OBJCOPY)" \
+		STRIP="$(TARGET_STRIP)" \
+		AR="$(TARGET_AR)" \
+		NM="$(TARGET_NM)" \
+		RANLIB="$(TARGET_RANLIB)" \
+		XZ=$(HOST_DIR)/usr/bin/xz $(SYSLINUX_EFI_ARGS)
+endef
+
 # syslinux build system has no convenient way to pass CFLAGS,
 # and the internal zlib should take precedence so -I shouldn't
 # be used.
 define SYSLINUX_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
-		AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) -C $(@D) $(SYSLINUX_TARGET)
+	$(SYSLINUX_MAKE) -C $(@D) $(SYSLINUX_TARGET)
 endef
 
 # While the actual bootloader is compiled for the target, several
@@ -61,8 +84,7 @@ endef
 # Repeat CC and AR, since syslinux really wants to check them at
 # install time
 define SYSLINUX_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
-		AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) INSTALLROOT=$(HOST_DIR) \
+	$(SYSLINUX_MAKE) INSTALLROOT=$(@D)/inst \
 		-C $(@D) $(SYSLINUX_TARGET) install
 endef
 
@@ -80,10 +102,21 @@ define SYSLINUX_INSTALL_IMAGES_CMDS
 	for i in $(SYSLINUX_IMAGES-y); do \
 		$(INSTALL) -D -m 0755 $(@D)/$$i $(BINARIES_DIR)/syslinux/$${i##*/}; \
 	done
-	for i in $(SYSLINUX_C32); do \
-		$(INSTALL) -D -m 0755 $(HOST_DIR)/usr/share/syslinux/$${i} \
+	for i in $(SYSLINUX_C32) ldlinux.c32; do \
+		$(INSTALL) -D -m 0755 $(@D)/inst/usr/share/syslinux/$${i} \
 			$(BINARIES_DIR)/syslinux/$${i}; \
 	done
 endef
 
+define HOST_SYSLINUX_BUILD_CMDS
+       $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
+-               AR="$(HOSTAR)" -C $(@D) bios
+endef
+
+define HOST_SYSLINUX_INSTALL_CMDS
+       $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
+-               AR="$(HOSTAR)" -C $(@D) INSTALLROOT=$(HOST_DIR) bios install
+endef
+
 $(eval $(generic-package))
+$(eval $(host-generic-package))
diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index 5e36ed1..899d8e8 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -70,8 +70,6 @@ ROOTFS_ISO9660_BOOT_IMAGE = isolinux/isolinux.bin
 define ROOTFS_ISO9660_INSTALL_BOOTLOADER
 	$(INSTALL) -D -m 0644 $(BINARIES_DIR)/syslinux/* \
 		$(ROOTFS_ISO9660_TARGET_DIR)/isolinux/
-	$(INSTALL) -D -m 0644 $(HOST_DIR)/usr/share/syslinux/ldlinux.c32 \
-		$(ROOTFS_ISO9660_TARGET_DIR)/isolinux/ldlinux.c32
 endef
 endif
 
@@ -168,6 +166,8 @@ define ROOTFS_ISO9660_CMD
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
+ROOTFS_ISO9660_DEPENDENCIES += host-syslinux
+
 define ROOTFS_ISO9660_GEN_HYBRID
 	$(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
 endef
-- 
2.7.4

[Buildroot] [PATCH 32/34] reproducible/iso9660: generate reproducible isos

[Gilles Chanteperdrix]

Sort the files list to get a deterministic file order
---
 fs/iso9660/iso9660.mk | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index 899d8e8..bfc056f 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -142,29 +142,30 @@ endif # ROOTFS_ISO9660_USE_INITRD
 
 
 ifneq ($(BR2_REPRODUCIBLE),y)
-define ROOTFS_ISO9660_GENISOIMAGE
-	$(HOST_DIR)/usr/bin/genisoimage -R
+define ROOTFS_ISO9660_CMD
+	$(HOST_DIR)/usr/bin/genisoimage -R -J -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
+		-no-emul-boot -boot-load-size 4 -boot-info-table \
+		-o $@ $(ROOTFS_ISO9660_TARGET_DIR)
 endef
 define ROOTFS_ISO9660_ISOHYBRID
 	$(HOST_DIR)/usr/bin/isohybrid
 endef
 else
-define ROOTFS_ISO9660_GENISOIMAGE
-	find $(ROOTFS_ISO9660_TARGET_DIR) -print0 | \
-	xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH) && \
-	$(HOST_DIR)/usr/bin/genisoimage -creation-date $(SOURCE_DATE_EPOCH)
+define ROOTFS_ISO9660_CMD
+	find $(ROOTFS_ISO9660_TARGET_DIR) -print | \
+	xargs touch -h -d @$(SOURCE_DATE_EPOCH) && \
+	find $(ROOTFS_ISO9660_TARGET_DIR) -! -type d -o -empty | sort | \
+	sed "s%$(ROOTFS_ISO9660_TARGET_DIR)"'\\(/.*\\)%\\1=&%' | \
+	$(HOST_DIR)/usr/bin/genisoimage -J -creation-date $(SOURCE_DATE_EPOCH) \
+		-b $(ROOTFS_ISO9660_BOOT_IMAGE) \
+		-no-emul-boot -boot-load-size 4 -boot-info-table \
+		-o $@ -graft-points -path-list -
 endef
 define ROOTFS_ISO9660_ISOHYBRID
 	$(HOST_DIR)/usr/bin/isohybrid -i $(SOURCE_DATE_EPOCH)
 endef
 endif
 
-define ROOTFS_ISO9660_CMD
-	$(ROOTFS_ISO9660_GENISOIMAGE) -J -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
-		-no-emul-boot -boot-load-size 4 -boot-info-table \
-		-o $@ $(ROOTFS_ISO9660_TARGET_DIR)
-endef
-
 ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
 ROOTFS_ISO9660_DEPENDENCIES += host-syslinux
 
-- 
2.7.4

[Buildroot] [PATCH 33/34] reproducible/cpio: generate reproducible cpio archives

[Gilles Chanteperdrix]

Sort the file list to get a deterministic file order.
---
 fs/cpio/cpio.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index e0e925d..81018a4 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -35,9 +35,9 @@ else
 ROOTFS_CPIO_DEPENDENCIES += host-cpio
 
 define ROOTFS_CPIO_CMD
-	find $(TARGET_DIR) -print0 | \
-	xargs -0 touch -h -d @$(SOURCE_DATE_EPOCH) && \
-	cd $(TARGET_DIR) && find . | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
+	find $(TARGET_DIR) | \
+	xargs touch -h -d @$(SOURCE_DATE_EPOCH) && \
+	cd $(TARGET_DIR) && find . | sort | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
 endef
 endif
 
-- 
2.7.4

[Buildroot] [PATCH 34/34] reproducible/grub2: get a reproducible grub2 full installation

[Gilles Chanteperdrix]

Get a reproducible modules dependencies file by fixing the awk script to
avoid depending on awk hash order, in order to get the same result
independently of the particular variant of awk used.
Compile grub2 with the target toolchain in the "full install" case, so
that the generated binaries do not depend on the host toolchain used.
---
 boot/grub2/grub2-deterministic-awk.patch | 68 ++++++++++++++++++++++++++++++++
 boot/grub2/grub2.mk                      | 34 ++++++++++------
 2 files changed, 91 insertions(+), 11 deletions(-)
 create mode 100644 boot/grub2/grub2-deterministic-awk.patch

diff --git a/boot/grub2/grub2-deterministic-awk.patch b/boot/grub2/grub2-deterministic-awk.patch
new file mode 100644
index 0000000..7f4d886
--- /dev/null
+++ b/boot/grub2/grub2-deterministic-awk.patch
@@ -0,0 +1,68 @@
+--- grub2-2.00/grub-core/genmoddep.awk~	2016-03-27 15:36:25.138050578 +0200
++++ grub2-2.00/grub-core/genmoddep.awk	2016-03-27 15:39:33.704636947 +0200
+@@ -15,17 +15,21 @@
+ BEGIN {
+   error = 0
+   lineno = 0;
++  i = 1;
+   while (getline <"/dev/stdin") {
+     lineno++;
+     if ($1 == "defined") {
+       symtab[$3] = $2;
+-      modtab[$2] = "" modtab[$2]
++      if (!modtab_index[$2])
++        modtab_index[$2]=i++
+     } else if ($1 == "undefined") {
++      if (!modtab_index[$2])
++        modtab_index[$2]=i++
+       if ($3 in symtab)
+-	modtab[$2] = modtab[$2] " " symtab[$3];
++        modtab[modtab_index[$2]] = modtab[modtab_index[$2]] " " symtab[$3];
+       else if ($3 != "__gnu_local_gp") {
+-	printf "%s in %s is not defined\n", $3, $2 >"/dev/stderr";
+-	error++;
++        printf "%s in %s is not defined\n", $3, $2 >"/dev/stderr";
++        error++;
+       }
+     }
+     else {
+@@ -41,24 +45,28 @@ END {
+   if (error >= 1)
+     exit 1;
+ 
+-  for (mod in modtab) {
++  for (j = 1; j < i; j++) {
++    for (mod in modtab_index)
++      if (modtab_index[mod] == j)
++        break;
+     # Remove duplications.
+-    split(modtab[mod], depmods, " ");
++    len=split(modtab[j], depmods, " ");
+     for (depmod in uniqmods) {
+       delete uniqmods[depmod];
+     }
+-    for (i in depmods) {
+-      depmod = depmods[i];
+-      # Ignore kernel, as always loaded.
+-      if (depmod != "kernel" && depmod != mod)
+-	uniqmods[depmod] = 1;
+-    }
+     modlist = ""
+-    for (depmod in uniqmods) {
+-      modlist = modlist " " depmod;
++    for (k = 1; k <= len; k++) {
++      depmod = depmods[k]
++      if (uniqmods[depmod])
++        continue
++      # Ignore kernel, as always loaded.
++      if (depmod != "kernel" && depmod != mod) {
++        uniqmods[depmod] = 1;
++        modlist = modlist " " depmod;
++      }
+     }
+     if (mod == "all_video") {
+-	continue;
++       continue;
+     }
+     printf "%s:%s\n", mod, modlist;
+   }
diff --git a/boot/grub2/grub2.mk b/boot/grub2/grub2.mk
index e2352b6..e94f524 100644
--- a/boot/grub2/grub2.mk
+++ b/boot/grub2/grub2.mk
@@ -46,17 +46,6 @@ endif
 # bootloader itself; none of these are used to build the native
 # tools.
 
-GRUB2_CONF_ENV = \
-	$(HOST_CONFIGURE_OPTS) \
-	CPP="$(HOSTCC) -E" \
-	TARGET_CC="$(TARGET_CC)" \
-	TARGET_CFLAGS="$(TARGET_CFLAGS)" \
-	TARGET_CPPFLAGS="$(TARGET_CPPFLAGS)" \
-	TARGET_LDFLAGS="$(TARGET_LDFLAGS)" \
-	NM="$(TARGET_NM)" \
-	OBJCOPY="$(TARGET_OBJCOPY)" \
-	STRIP="$(TARGET_STRIP)"
-
 GRUB2_CONF_OPTS = \
 	--target=$(GRUB2_TARGET) \
 	--with-platform=$(GRUB2_PLATFORM) \
@@ -68,6 +57,17 @@ GRUB2_CONF_OPTS = \
 	--disable-werror
 
 ifneq ($(BR2_TARGET_FULL_INSTALL),y)
+GRUB2_CONF_ENV = \
+	$(HOST_CONFIGURE_OPTS) \
+	CPP="$(HOSTCC) -E" \
+	TARGET_CC="$(TARGET_CC)" \
+	TARGET_CFLAGS="$(TARGET_CFLAGS)" \
+	TARGET_CPPFLAGS="$(TARGET_CPPFLAGS)" \
+	TARGET_LDFLAGS="$(TARGET_LDFLAGS)" \
+	NM="$(TARGET_NM)" \
+	OBJCOPY="$(TARGET_OBJCOPY)" \
+	STRIP="$(TARGET_STRIP)"
+
 # We don't want all the native tools and Grub2 modules to be installed
 # in the target. So we in fact install everything into the host
 # directory, and the image generation process (below) will use the
@@ -104,6 +104,18 @@ define GRUB2_EFI_STARTUP_NSH
 endef
 GRUB2_POST_INSTALL_TARGET_HOOKS += GRUB2_EFI_STARTUP_NSH
 endif
+
+else
+GRUB2_CONF_ENV = \
+	$(TARGET_CONFIGURE_OPTS) \
+	CPP="$(TARGET_CC) -E" \
+	TARGET_CC="$(TARGET_CC)" \
+	TARGET_CFLAGS="$(TARGET_CFLAGS)" \
+	TARGET_CPPFLAGS="$(TARGET_CPPFLAGS)" \
+	TARGET_LDFLAGS="$(TARGET_LDFLAGS)" \
+	NM="$(TARGET_NM)" \
+	OBJCOPY="$(TARGET_OBJCOPY)" \
+	STRIP="$(TARGET_STRIP)"
 endif
 
 $(eval $(autotools-package))
-- 
2.7.4

[Buildroot] Reproducible builds

[Thomas Petazzoni]

Hello Gilles,

On Sat, 30 Apr 2016 09:43:58 +0200, Gilles Chanteperdrix wrote:

> some time ago, I worked on getting reproducible builds for
> buildroot. I take "reproducible builds" in a broader sense than what
> Debian does for instance: in the case of buildroot, we would like
> the built binaries to be as independent of the build environment as
> possible, so that in addition to getting identical binaries when
> building twice on the same machine, we also get identical binaries
> when building on a different distribution.

First of all, thanks a lot for working on this! It's definitely an
interesting and useful topic.

> So far, I got
> reproducible builds for a (relatively reduced) subset of buildroot
> packages, on a Debian jessie and Slackware 14.1, with as many
> differences in the distributions configuration as I could find
> (timezone, locale, date, time, etc... even the awk flavor used). The
> only thing I have to get identical is the build path, this is what
> Debian does, a patch has been proposed a long time ago to allow gcc
> to avoid the dependence between the __FILE__ macro and the build
> path, but it does not seem to have been merged:
> https://mail-index.netbsd.org/tech-toolchain/2009/02/17/msg000577.html
> 
> I expected to be able to clean up the patches before submission (as
> many patches do the same thing, like overriding the __DATE__ and
> __TIME__ macros in order to avoid depending on the build time, so
> this could probably be made generic by buildroot core), but I am not
> going to have time soon to do that, so here come the patches as they
> are, in order to spark discussion and gather remarks, before I have
> time to do more. The patches are based on the 2015.11.1 release.

Indeed, as you say yourself, some of your patches are not mergeable
as-is. However, several of them are not related, or directly related to
reproducible builds (some of your patches add new packages, etc.).

One first thing that is missing is your Signed-off-by line on the
patches, which we require for all Buildroot contributions (exactly like
for the Linux kernel).

Another thing that worries me is all the patches/modifications needed
by each package to make their build "reproducible". Do you expect such
patches to be merged in the respective upstream projects?

Finally, I'm also concerned by the testability of the reproducible
build feature. To make sure such a feature work, we would need to do
some automated build testing of the same configuration in various
environments/configurations and ensure they provide the same output.
This is clearly not impossible to do, but there's quite some work
involved to set up such an infrastructure.

I'll review some of your patches (when it makes sense). However, I'll
mark them all as "RFC" in our patch tracking system: since none of them
have a Signed-off-by line, we can't even take them. It would be good if
you could resend first with your SoB line the patches that are not
related to the reproducible builds. It would help reduce your stack of
patches.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:48:57 +0200, Gilles Chanteperdrix wrote:
> Making the builds reproducible involves turning off build timestamps,
> which some users may find annoying. So make the reproducible builds
> optional.

Missing SoB, as I said.

Originally, I thought "why make this optional", but I indeed understand
that for some use cases, not having timestamps might be annoying. So I
guess making this optional is the best option.

>  
> +config BR2_REPRODUCIBLE
> +	bool "Try and remove all sources of non-reproducibility from the build"

This should probably be:

	bool "Make the build reproducible (experimental)"
	help
	  This option will remove all sources of non-reproducibility
	  from the build process. For a given Buildroot configuration,
	  this allows to generate exactly identical binaries from one
	  build to the other, including on different machines.

	  This is labeled as an experimental feature, as not all
	  packages behave properly to ensure reproducibility.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 02/34] reproducibility: override locale and timezone

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:48:58 +0200, Gilles Chanteperdrix wrote:
> ---
>  Makefile | 7 +++++++
>  1 file changed, 7 insertions(+)

Missing SoB.

> diff --git a/Makefile b/Makefile
> index ec7dccb..86b2ed2 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -218,6 +218,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
>  -include $(BR2_CONFIG)
>  endif
>  
> +export BR2_REPRODUCIBLE

Why ?

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +export TZ=UTC
> +export LANG=C
> +export LC_ALL=C
> +endif

Regarding overriding TZ, LANG and LC_ALL, I'm wondering if we shouldn't
do it unconditionally, regardless of the value of BR2_REPRODUCIBLE.
What is the benefit of leaving those variable to their value from the
build environment?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:48:59 +0200, Gilles Chanteperdrix wrote:
> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH

wanted -> requested

generates -> generate

> environment variable which contains either the date of buildroot last
> commit if running from a git repository, or the latest release date.
> 
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
> may not have changed in fact.

Missing SoB.

> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)

Yirk. I'm wondering if we shouldn't simply have a BR2_RELEASE_DATE
variable next to BR2_VERSION in the top level Makefile that gives the
release date.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 04/34] reproducibility: make rootfs.tar reproducible

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:00 +0200, Gilles Chanteperdrix wrote:

> +define ROOTFS_TAR_CMD
> +	cd $(TARGET_DIR) && { \
> +		find . -\( -! -type d -o -empty -\) -print0 | \
> +		sort -z | \
> +		tar --null -T - -c$(TAR_OPTS)f $@ --mtime=@$(SOURCE_DATE_EPOCH) --numeric-owner; \
> +	}

We normally write such constructs as:

	(cd $(TARGET_DIR) && \
		foo ....)

However, this raises the question of what's needed for all the other
filesystem formats. Will they all have to implement a different
ROOTFS_<foo>_CMD variable ? Or will there be some commonalities that
should be factored out in the common rootfs image infrastructure ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 07/34] reproducibility/libgcrypt: override timestamps

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:03 +0200, Gilles Chanteperdrix wrote:
> with SOURCE_DATE_EPOCH
> ---
>  package/libgcrypt/0003-honor-source-date-epoch.patch | 15 +++++++++++++++
>  package/libgcrypt/libgcrypt.mk                       |  1 +
>  2 files changed, 16 insertions(+)
>  create mode 100644 package/libgcrypt/0003-honor-source-date-epoch.patch
> 
> diff --git a/package/libgcrypt/0003-honor-source-date-epoch.patch b/package/libgcrypt/0003-honor-source-date-epoch.patch
> new file mode 100644
> index 0000000..0f41c1b
> --- /dev/null
> +++ b/package/libgcrypt/0003-honor-source-date-epoch.patch

All patches need a description + Signed-off-by.

Also, has this been submitted upstream? Is it already used by Debian?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Thomas Petazzoni]

Hello,

Cc'ing Gustavo on this, since he has done quite a bit of libglib
packaging work.

On Sat, 30 Apr 2016 09:49:05 +0200, Gilles Chanteperdrix wrote:
> glib2 installs compiled python bytecode in /usr/share/glib-2.0/codegen,
> likely for the gdbus-codegen program.
> 
> This bytecode is not compiled with the buildroot host python, which
> causes a first source of non reproducibility, fix that by using
> buildroot host python.

Which has the annoying side effect of increasing even more the set of
mandatory dependencies to build libglib :-/

So I wonder if the dependency on host-python should be made conditional
on BR2_REPRODUCIBLE.

> But this is not sufficient, compiling the python bytecode with the
> same interpreter in different environments yields different binaries,
> so, since buildroot users are unlikely to need the qdbus-codegen
> program on target, add an option to remove it. The option is disabled
> by default.

Since then, Gustavo has already modified the libglib2 package to
remove the /usr/share/glib-2.0/codegen directory. However,
gdbus-codegen is not removed.

See commit 1f1f16e9e5367048faa1cf17237e0c0d422e98d6.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 11/34] reproducibility/vim: make vim reproducible

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:07 +0200, Gilles Chanteperdrix wrote:

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +VIM_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
> +VIM_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
> +
> +VIM_CONF_OPTS += \
> +	--with-compiledby=buildroot \
> +	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(VIM_DIR)/.br_date_override.h"
> +
> +define VIM_OVERRIDE_DATE_HOOK
> +	echo "#define buildroot_stringify(x) #x" > $(VIM_DIR)/.br_date_override.h
> +	echo "#define __DATE__ buildroot_stringify($(VIM_SOURCE_DATE))" >> $(VIM_DIR)/.br_date_override.h
> +	echo "#define __TIME__ buildroot_stringify($(VIM_SOURCE_TIME))" >> $(VIM_DIR)/.br_date_override.h
> +endef
> +
> +VIM_POST_PATCH_HOOKS += VIM_OVERRIDE_DATE_HOOK

As you say yourself, this seems to be needed in many packages, so I
think a more generic solution is needed. At the very least,
the .br_date_override.h file should be generated once in a generic
place.

> +define VIM_FIX_AWK_REFERENCE
> +	sed -i 's,^#!.*awk,#!/usr/bin/awk,' $(TARGET_DIR)/usr/share/vim/vim74/tools/mve.awk
> +endef
> +
> +VIM_POST_INSTALL_TARGET_HOOKS += VIM_FIX_AWK_REFERENCE

Do we even need this awk script installed?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 12/34] reproducibility/wireshark: remove documentation

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:08 +0200, Gilles Chanteperdrix wrote:
> HTML documentation contains a link to a mail address specific to
> the distribution of the build machine. Remove html documentation
> from the rootfs as it is unlikely to be needed. At this chance
> also remove the txt documentation.
> ---
>  package/wireshark/wireshark.mk | 6 ++++++
>  1 file changed, 6 insertions(+)

If this one had your Signed-off-by, I could have applied it. Can you
respin?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 02/34] reproducibility: override locale and timezone

[Arnout Vandecappelle]

On 05/07/16 15:15, Thomas Petazzoni wrote:
> Hello,
>
> On Sat, 30 Apr 2016 09:48:58 +0200, Gilles Chanteperdrix wrote:
>> ---
>>  Makefile | 7 +++++++
>>  1 file changed, 7 insertions(+)
>
> Missing SoB.
>
>> diff --git a/Makefile b/Makefile
>> index ec7dccb..86b2ed2 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -218,6 +218,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
>>  -include $(BR2_CONFIG)
>>  endif
>>
>> +export BR2_REPRODUCIBLE
>
> Why ?

  I guess because it will (or may be) be used in some helper scripts...

>
>> +ifeq ($(BR2_REPRODUCIBLE),y)
>> +export TZ=UTC
>> +export LANG=C
>> +export LC_ALL=C
>> +endif
>
> Regarding overriding TZ, LANG and LC_ALL, I'm wondering if we shouldn't
> do it unconditionally, regardless of the value of BR2_REPRODUCIBLE.
> What is the benefit of leaving those variable to their value from the
> build environment?

  For the record: I had the same comment about this entire series. Most of it 
should be done unconditionally, only the aspects that may be annoying for 
debugging (like the timestamps) should be conditional.

  Unfortunately, that will be a cause for discussion... For instance, LANG=C 
means that compiler error messages will be in English instead of in your native 
language. Do we really want that by default?

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 04/34] reproducibility: make rootfs.tar reproducible

[Arnout Vandecappelle]

On 05/07/16 15:23, Thomas Petazzoni wrote:
> Hello,
>
> On Sat, 30 Apr 2016 09:49:00 +0200, Gilles Chanteperdrix wrote:
>
>> +define ROOTFS_TAR_CMD
>> +	cd $(TARGET_DIR) && { \
>> +		find . -\( -! -type d -o -empty -\) -print0 | \
>> +		sort -z | \
>> +		tar --null -T - -c$(TAR_OPTS)f $@ --mtime=@$(SOURCE_DATE_EPOCH) --numeric-owner; \
>> +	}
>
> We normally write such constructs as:
>
> 	(cd $(TARGET_DIR) && \
> 		foo ....)

  Actually, we don't AFAIK... In general, the parenthesis are not needed so they 
should be removed. So also in this case it should be

	cd $(TARGET_DIR);
		find ....

>
> However, this raises the question of what's needed for all the other
> filesystem formats. Will they all have to implement a different
> ROOTFS_<foo>_CMD variable ? Or will there be some commonalities that
> should be factored out in the common rootfs image infrastructure ?

  Yes, to me it makes more sense to do this in the actual target directory after 
the post-build scripts have been run.


  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 13/34] reproducibility/libtool: fix reproducibility issues

[Thomas Petazzoni]

Hello,

This probably should be several separate patches.

On Sat, 30 Apr 2016 09:49:09 +0200, Gilles Chanteperdrix wrote:
> - remove libtool and libtoolize; they are full of references to the
>   build environment tools, and unlikely to be needed on target;
> - do not use /etc/ld.so.conf contents in libtool search path: they are
>   not relevant on target, and cause a useless dependency of libltdl
>   on the build environment.
> ---
>  package/libtool/0001-fix-cross-compilation.patch | 11 +++++++++++
>  package/libtool/libtool.mk                       |  8 ++++++--
>  2 files changed, 17 insertions(+), 2 deletions(-)
>  create mode 100644 package/libtool/0001-fix-cross-compilation.patch
> 
> diff --git a/package/libtool/0001-fix-cross-compilation.patch b/package/libtool/0001-fix-cross-compilation.patch
> new file mode 100644
> index 0000000..c203371
> --- /dev/null
> +++ b/package/libtool/0001-fix-cross-compilation.patch

This patch should have a description + Signed-off-by.

However, libtool is really a core package, and not a simple one, so I'm
a bit uneasy with the idea of carrying patches against it. What are the
chances of having something like this submitted upstream?

> +   fi
> diff --git a/package/libtool/libtool.mk b/package/libtool/libtool.mk
> index 8688f8b..c177626 100644
> --- a/package/libtool/libtool.mk
> +++ b/package/libtool/libtool.mk
> @@ -8,7 +8,7 @@ LIBTOOL_VERSION = 2.4.6
>  LIBTOOL_SOURCE = libtool-$(LIBTOOL_VERSION).tar.xz
>  LIBTOOL_SITE = $(BR2_GNU_MIRROR)/libtool
>  LIBTOOL_INSTALL_STAGING = YES
> -LIBTOOL_CONF_ENV = HELP2MAN=true
> +LIBTOOL_CONF_OPTS = HELP2MAN=true

Why?

>  LIBTOOL_DEPENDENCIES = host-m4
>  LIBTOOL_LICENSE = GPLv2+
>  LIBTOOL_LICENSE_FILES = COPYING
> @@ -33,9 +33,13 @@ define LIBTOOL_AVOID_AUTORECONF_HOOK
>  	find $(@D) -name configure -exec touch '{}' \;
>  	find $(@D) -name Makefile.in -exec touch '{}' \;
>  endef
> -LIBTOOL_PRE_CONFIGURE_HOOKS += LIBTOOL_AVOID_AUTORECONF_HOOK

Why?

>  HOST_LIBTOOL_PRE_CONFIGURE_HOOKS += LIBTOOL_AVOID_AUTORECONF_HOOK
>  
> +define LIBTOOL_REMOVE_LIBTOOL
> +	rm $(TARGET_DIR)/usr/bin/libtool $(TARGET_DIR)/usr/bin/libtoolize
> +endef
> +LIBTOOL_POST_INSTALL_TARGET_HOOKS += LIBTOOL_REMOVE_LIBTOOL

This part could be a separate patch that would be easy to apply today,
so don't hesitate to resend just this part, with your Signed-off-by
line, of course.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 14/34] reproducibility/python: also remove pyo files

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:10 +0200, Gilles Chanteperdrix wrote:
> when BR2_PACKAGE_PYTHON_PY_ONLY is selected. Make that option the
> default when BR2_REPRODUCIBLE is selected in order to avoid problems
> with timestamps embedded in compiled python files.

The title of the patch says "also remove pyo files", but this patch
does a lot more.

> diff --git a/package/python/Config.in b/package/python/Config.in
> index fa83e77..2c62559 100644
> --- a/package/python/Config.in
> +++ b/package/python/Config.in
> @@ -19,7 +19,8 @@ if BR2_PACKAGE_PYTHON
>  
>  choice
>  	prompt "python module format to install"
> -	default BR2_PACKAGE_PYTHON_PYC_ONLY
> +	default BR2_PACKAGE_PYTHON_PYC_ONLY if !BR2_REPRODUCIBLE
> +	default BR2_PACKAGE_PYTHON_PY_ONLY if BR2_REPRODUCIBLE

If .pyc files are really not suitable for reproducible builds, then the
option of having .pyc files should be completely hidden when
BR2_REPRODUCIBLE=y.

> diff --git a/package/python/python.mk b/package/python/python.mk
> index 80ffbde..5c5ebb8 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -225,7 +225,7 @@ endif
>  
>  ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
>  define PYTHON_FINALIZE_TARGET
> -	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f
> +	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py[co]' -print0 | xargs -0 rm -f

This part is already handled in another patch series I have sent around
Python. See https://patchwork.ozlabs.org/patch/617272/ specifically.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 15/34] reproducibility/gzip: avoid gzip timestamp in rootfs.tar.gz

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:11 +0200, Gilles Chanteperdrix wrote:
> ---
>  fs/common.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/common.mk b/fs/common.mk
> index 528e194..ea248bb 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -48,7 +48,7 @@ ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
>  
>  ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
>  ROOTFS_$(2)_COMPRESS_EXT = .gz
> -ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
> +ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c $(if $(BR2_REPRODUCIBLE),-n)
>  endif
>  ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
>  ROOTFS_$(2)_COMPRESS_EXT = .bz2

Looks good, just resend with your Signed-off-by line.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 16/34] reproducibility/lzo: forbid lzo with reproducible builds

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:12 +0200, Gilles Chanteperdrix wrote:
> The lzo archive contains a timestamp.

Missing Signed-off-by line.

> ---
>  fs/tar/Config.in | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/fs/tar/Config.in b/fs/tar/Config.in
> index 63663ec..7e4b603 100644
> --- a/fs/tar/Config.in
> +++ b/fs/tar/Config.in
> @@ -33,6 +33,7 @@ config BR2_TARGET_ROOTFS_TAR_LZMA
>  
>  config BR2_TARGET_ROOTFS_TAR_LZO
>  	bool "lzo"
> +	depends on !BR2_REPRODUCIBLE
>  	help
>  	  Do compress the tarball with lzop.
>  
> @@ -43,6 +44,12 @@ config BR2_TARGET_ROOTFS_TAR_XZ
>  
>  endchoice
>  
> +if BR2_REPRODUCIBLE
> +
> +comment "lzo compression is only available with non-reproducible builds"
> +
> +endif

When we are in such situations, we normally do:

comment "blablabla"
	depends on BR2_REPRODUCIBLE

however, in this specific case, I would simply not bother adding a
comment. If we start adding a comment for everything that is not
suitable for reproducible builds, we're going to add hundreds of them I
believe.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 17/34] reproducibility/libnspr: override build timestamps

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:13 +0200, Gilles Chanteperdrix wrote:
> with SOURCE_DATE_EPOCH

Missing SoB line here.

> ---
>  package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch | 71 ++++++++++++++++++++++
>  1 file changed, 71 insertions(+)
>  create mode 100644 package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
> 
> diff --git a/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
> new file mode 100644
> index 0000000..1a0fb7c
> --- /dev/null
> +++ b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch

Missing patch description and Signed-off-by line. Also, did you submit
this upstream? Does Debian carry a similar patch for its Reproducible
Builds effort?

Those questions are relevant for every package for which you're adding
a patch.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 19/34] reproducibility/python: make python reproducible

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:15 +0200, Gilles Chanteperdrix wrote:
> python build is not reproducible for two reasons:
> - the libpython library embeds the build timestamps, fix this by
>   overriding the __DATE__ and __TIME__ macro using SOURCE_DATE_EPOCH;
> - _sysconfigdata.py and config/Makefile on target includes references
>   through INSTALL and MKDIR_P variables to host tools, fix this by
>   overriding these variables values with the corresponding paths on
>   buildroot rootfs.

For this second part, are you really aiming at having reproducible
builds with different build directories ? As you say in your cover
letter, that's an even more complicated problem.

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +PYTHON_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
> +PYTHON_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
> +
> +PYTHON_CONF_OPTS += \
> +	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(PYTHON_DIR)/.br_date_override.h"
> +
> +define PYTHON_OVERRIDE_DATE_HOOK
> +	echo "#define buildroot_stringify(x) #x" > $(PYTHON_DIR)/.br_date_override.h
> +	echo "#define __DATE__ buildroot_stringify($(PYTHON_SOURCE_DATE))" >> $(PYTHON_DIR)/.br_date_override.h
> +	echo "#define __TIME__ buildroot_stringify($(PYTHON_SOURCE_TIME))" >> $(PYTHON_DIR)/.br_date_override.h
> +endef
> +
> +PYTHON_POST_PATCH_HOOKS += PYTHON_OVERRIDE_DATE_HOOK
> +endif

This really needs to be factored out somewhere. Maybe our compiler
wrapper should take of this. Arnout?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 21/34] hddtemp: add

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:17 +0200, Gilles Chanteperdrix wrote:
> ---
>  package/Config.in            |  1 +
>  package/hddtemp/Config.in    |  8 ++++++++
>  package/hddtemp/hddtemp.hash |  2 ++
>  package/hddtemp/hddtemp.mk   | 19 +++++++++++++++++++
>  4 files changed, 30 insertions(+)
>  create mode 100644 package/hddtemp/Config.in
>  create mode 100644 package/hddtemp/hddtemp.hash
>  create mode 100644 package/hddtemp/hddtemp.mk

Generally looks good. As usual your SoB line is missing, so we can't
apply for the moment. Some more comments below.

> diff --git a/package/hddtemp/Config.in b/package/hddtemp/Config.in
> new file mode 100644
> index 0000000..0d28250
> --- /dev/null
> +++ b/package/hddtemp/Config.in
> @@ -0,0 +1,8 @@
> +config BR2_PACKAGE_HDDTEMP
> +	bool "hddtemp"

Really no toolchain dependencies? Not using fork() ? Not using
threads ?

> +++ b/package/hddtemp/hddtemp.mk
> @@ -0,0 +1,19 @@
> +################################################################################
> +#
> +# hddtemp
> +#
> +################################################################################
> +
> +HDDTEMP_VERSION = 0.3-beta15
> +HDDTEMP_SOURCE = hddtemp-$(HDDTEMP_VERSION).tar.bz2
> +HDDTEMP_SITE = http://download.savannah.gnu.org/releases/hddtemp
> +HDDTEMP_LICENSE = GPLv2
> +HDDTEMP_LICENSE_FILES = GPL-2
> +
> +define HDDTEMP_REMOVE_MAN
> +	rm -rf $(TARGET_DIR)/usr/man
> +endef

This is not needed usr/man is removed from $(TARGET_DIR) at the end of
the build. See the target-finalize make target in the main Buildroot
Makefile.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 22/34] High availability: add packages

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:18 +0200, Gilles Chanteperdrix wrote:
> cluster-glue
> corosync
> heartbeat
> libqb
> pacemaker

This should be split in a series of patches, with one package per patch.

>  menu "Javascript"
>  	source "package/angularjs/Config.in"
>  	source "package/bootstrap/Config.in"
> diff --git a/package/cluster-glue/0001-fix-cross-compile.patch b/package/cluster-glue/0001-fix-cross-compile.patch
> new file mode 100644
> index 0000000..2f8bd47
> --- /dev/null
> +++ b/package/cluster-glue/0001-fix-cross-compile.patch
> @@ -0,0 +1,26 @@

All patches need a description and Signed-off-by line.


> diff --git a/package/cluster-glue/Config.in b/package/cluster-glue/Config.in
> new file mode 100644
> index 0000000..8b39152
> --- /dev/null
> +++ b/package/cluster-glue/Config.in
> @@ -0,0 +1,9 @@
> +config BR2_PACKAGE_CLUSTER_GLUE
> +	bool "cluster-glue"
> +	select BR2_PACKAGE_UTIL_LINUX
> +	select BR2_PACKAGE_UTIL_LINUX_LIBUUID

This looks wrong, for two reasons:

 (1) Because you are not replicating the dependencies of util-linux
     here.

 (2) Because this package has many more packages in its
     <pkg>_DEPENDENCIES variable than just util-linux. For example,
     libxml2, libglib2, etc. They should all be selected here as well.


> +CLUSTER_GLUE_VERSION = 0a7add1d9996
> +CLUSTER_GLUE_SOURCE = $(CLUSTER_GLUE_VERSION).tar.bz2
> +CLUSTER_GLUE_SITE =  http://hg.linux-ha.org/glue/archive
> +CLUSTER_GLUE_LICENSE = LGPLv2.1+ (library), GPLv2+ (programs)
> +CLUSTER_GLUE_LICENSE_FILES = COPYING COPYING.LIB
> +CLUSTER_GLUE_DEPENDENCIES += host-pkgconf host-python libtool util-linux bzip2 libglib2 libxml2

The dependency on libtool is useless, it's automatically added by
<pkg>_AUTORECONF = YES.

> +
> +CLUSTER_GLUE_INSTALL_STAGING = YES
> +CLUSTER_GLUE_AUTORECONF = YES

Add a comment above saying why it's needed.

> diff --git a/package/corosync/Config.in b/package/corosync/Config.in
> new file mode 100644
> index 0000000..3855136
> --- /dev/null
> +++ b/package/corosync/Config.in
> @@ -0,0 +1,8 @@
> +config BR2_PACKAGE_COROSYNC
> +	bool "corosync"

Same question as usual: no dependency on threads? MMU? Any other
toolchain feature?

Your .mk file has a dependency on libnss, so you must select it.

> +################################################################################
> +#
> +# corosync
> +#
> +################################################################################
> +
> +COROSYNC_VERSION = 1.4.8
> +COROSYNC_SOURCE = corosync-$(COROSYNC_VERSION).tar.gz
> +COROSYNC_SITE = http://build.clusterlabs.org/corosync/releases
> +COROSYNC_LICENSE = BSD-3c except for crypto.[ch], public domain for crypto.[ch]

Should be:

	BSD-3c (except for crypto.[ch]), Public Domain (crypto.[ch])

> +COROSYNC_LICENSE_FILES = LICENSE
> +COROSYNC_INSTALL_STAGING = YES
> +COROSYNC_DEPENDENCIES += host-pkgconf libnss
> +
> +COROSYNC_CONF_OPTS += ac_cv_func_fnmatch_works=yes
> +
> +$(eval $(autotools-package))
> diff --git a/package/heartbeat/0001-fix-cross-compilation.patch b/package/heartbeat/0001-fix-cross-compilation.patch
> new file mode 100644
> index 0000000..4a653f6
> --- /dev/null
> +++ b/package/heartbeat/0001-fix-cross-compilation.patch

Description + SoB and upstream status (ditto for all patches).

> diff --git a/package/heartbeat/Config.in b/package/heartbeat/Config.in
> new file mode 100644
> index 0000000..0f8ee80
> --- /dev/null
> +++ b/package/heartbeat/Config.in
> @@ -0,0 +1,10 @@
> +config BR2_PACKAGE_HEARTBEAT
> +	bool "heartbeat"
> +	select BR2_PACKAGE_UTIL_LINUX
> +	select BR2_PACKAGE_UTIL_LINUX_LIBUUID
> +	select BR2_PACKAGE_IPROUTE2

Replication of dependencies needed here.

> diff --git a/package/heartbeat/heartbeat.mk b/package/heartbeat/heartbeat.mk
> new file mode 100644
> index 0000000..eedf40f
> --- /dev/null
> +++ b/package/heartbeat/heartbeat.mk
> @@ -0,0 +1,37 @@
> +################################################################################
> +#
> +# heartbeat
> +#
> +################################################################################

Nit: missing empty line.

> +HEARTBEAT_VERSION = 958e11be8686
> +HEARTBEAT_SOURCE = $(HEARTBEAT_VERSION).tar.bz2
> +HEARTBEAT_SITE =  http://hg.linux-ha.org/heartbeat-STABLE_3_0/archive
> +HEARTBEAT_LICENSE = LGPLv2.1+ (library), GPLv2+ (programs)
> +HEARTBEAT_LICENSE_FILES = doc/COPYING doc/COPYING.LGPL
> +HEARTBEAT_DEPENDENCIES += host-pkgconf host-python libtool util-linux bzip2 cluster-glue

libtool dependency not needed. Please sync your Config.in with this
dependency line by selecting all relevant packages.

Another question: is Python 2.x absolutely needed on the host, or can
it work with Python 3.x as well ?

[I'm skipping lots of additional code, for which the same comments
appply.]


> +# This program configure script expects the proper -I flags in CFLAGS and not
> +# CPPFLAGS
> +PACEMAKER_CONF_OPTS += \
> +	CFLAGS="$(TARGET_CFLAGS) -I$(STAGING_DIR)/usr/include -I$(STAGING_DIR)/usr/include/heartbeat" \
> +	ac_cv_path_HTML2TXT="" \
> +	ac_cv_path_HELP2MAN="" \
> +	ac_cv_path_POD2MAN="" \
> +	ac_cv_path_ASCIIDOC="" \
> +	ac_cv_path_PUBLICAN="" \
> +	ac_cv_path_INKSCAPE="" \
> +	ac_cv_path_XSLTPROC="" \
> +	ac_cv_path_XMLCATALOG="" \
> +	ac_cv_path_FOP="" \

We usually just assign to the empty value, and it's normally sufficient.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 23/34] pkg-download: allow git+ssh URLs

[Thomas Petazzoni]

Hello,

On Sat, 30 Apr 2016 09:49:19 +0200, Gilles Chanteperdrix wrote:
> ---
>  package/pkg-download.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

This looks good I believe, so please resend with your SoB line. Maybe
an update to the Buildroot manual is also needed.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 05/34] reproducibility/linux: override build timestamp

[Arnout Vandecappelle]

On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> with SOURCE_DATE_EPOCH
> ---
>  linux/linux.mk | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 9507837..d882702 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -74,6 +74,14 @@ LINUX_MAKE_ENV = \
>  	$(TARGET_MAKE_ENV) \
>  	BR_BINARIES_DIR=$(BINARIES_DIR)
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +LINUX_MAKE_ENV += \
> +	KBUILD_BUILD_VERSION=1 \
> +	KBUILD_BUILD_USER=buildroot \
> +	KBUILD_BUILD_HOST=buildroot \

  This is for example something that really should be inside BR2_REPRODUCIBLE, 
because the build user and version gives me useful information while developing.

> +	KBUILD_BUILD_TIMESTAMP="$(shell TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH))"

  I think the TZ LANG LC_ALL things are redundant, because we already exported 
those.

  This timestamp is used in several places, so I think it's useful to save it in 
a global SOURCE_DATE variable.

  Regards,
  Arnout

> +endif
> +
>  # Get the real Linux version, which tells us where kernel modules are
>  # going to be installed in the target filesystem.
>  LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Reproducible builds

[Arnout Vandecappelle]

On 05/07/16 15:11, Thomas Petazzoni wrote:
> Finally, I'm also concerned by the testability of the reproducible
> build feature. To make sure such a feature work, we would need to do
> some automated build testing of the same configuration in various
> environments/configurations and ensure they provide the same output.
> This is clearly not impossible to do, but there's quite some work
> involved to set up such an infrastructure.

  Even if it is not regularly tested, I think we can start including this 
feature. But the help text should clarify that there are no guarantees.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Arnout Vandecappelle]

On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> environment variable which contains either the date of buildroot last
> commit if running from a git repository, or the latest release date.
>
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
> may not have changed in fact.
> ---
>  Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 86b2ed2..c1d2961 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ=UTC
>  export LANG=C
>  export LC_ALL=C
> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))

  I really don't like exporting a variable that is not well-known and that 
doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in 
Buildroot-specific patches, so we can just as well add the BR_ prefix.

  But actually, we don't want to introduce buildroot-specific (and therefore 
unupstreamable) things in patches either. So there is not much point in 
exporting, except if it's used by scripts.


  Regards,
  Arnout

>  endif
>
>  # To put more focus on warnings, be less verbose as default
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 07/34] reproducibility/libgcrypt: override timestamps

[Arnout Vandecappelle]

On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> with SOURCE_DATE_EPOCH
> ---
>  package/libgcrypt/0003-honor-source-date-epoch.patch | 15 +++++++++++++++
>  package/libgcrypt/libgcrypt.mk                       |  1 +
>  2 files changed, 16 insertions(+)
>  create mode 100644 package/libgcrypt/0003-honor-source-date-epoch.patch
>
> diff --git a/package/libgcrypt/0003-honor-source-date-epoch.patch b/package/libgcrypt/0003-honor-source-date-epoch.patch
> new file mode 100644
> index 0000000..0f41c1b
> --- /dev/null
> +++ b/package/libgcrypt/0003-honor-source-date-epoch.patch
> @@ -0,0 +1,15 @@
> +--- libgcrypt-1.6.4/configure.ac.orig	2015-09-08 08:27:26.000000000 +0200
> ++++ libgcrypt-1.6.4/configure.ac	2016-02-23 21:31:19.290079195 +0100
> +@@ -1993,7 +1993,11 @@ changequote([,])dnl
> + BUILD_FILEVERSION="${BUILD_FILEVERSION}mym4_revision_dec"
> + AC_SUBST(BUILD_FILEVERSION)
> +
> +-BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
> ++if test x"${SOURCE_DATE_EPOCH}" = x; then
> ++   	BUILD_TIMESTAMP=`date -u +%Y-%m-%dT%H:%M+0000 2>/dev/null || date`
> ++else
> ++	BUILD_TIMESTAMP=`date -u -d @${SOURCE_DATE_EPOCH} +%Y-%m-%dT%H:%M+0000 2>/dev/null || date -d @${SOURCE_DATE_EPOCH}`

  Seems to me that SOURCE_DATE_EPOCH is something that is too
> ++fi
> + AC_SUBST(BUILD_TIMESTAMP)
> + AC_DEFINE_UNQUOTED(BUILD_TIMESTAMP, "$BUILD_TIMESTAMP",
> +                    [The time this package was configured for a build])
> diff --git a/package/libgcrypt/libgcrypt.mk b/package/libgcrypt/libgcrypt.mk
> index 53d1d6c..768e42f 100644
> --- a/package/libgcrypt/libgcrypt.mk
> +++ b/package/libgcrypt/libgcrypt.mk
> @@ -12,6 +12,7 @@ LIBGCRYPT_SITE = ftp://ftp.gnupg.org/gcrypt/libgcrypt
>  LIBGCRYPT_INSTALL_STAGING = YES
>  LIBGCRYPT_DEPENDENCIES = libgpg-error
>  LIBGCRYPT_CONFIG_SCRIPTS = libgcrypt-config
> +LIBGCRYPT_AUTORECONF = YES

  Isn't it sufficient to do

ifeq ($(BR2_REPRODUCIBLE),y)
LIBGCRYPT_MAKE_OPTS = BUILD_TIMESTAMP=$(SOURCE_DATE)
endif


  Regards,
  Arnout

>
>  LIBGCRYPT_CONF_ENV = \
>  	ac_cv_sys_symbol_underscore=no
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Arnout Vandecappelle]

On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> But this is not sufficient, compiling the python bytecode with the
> same interpreter in different environments yields different binaries,

  Er, this is worrisome... You are saying that we don't have a chance in hell of 
generating reproducible python bytecode?

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 10/34] reproducibility/sysstat: override gcc macros

[Arnout Vandecappelle]

On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> Override the __DATE__ and __TIME__ macros with values obtained from
> the package SOURCE_DATE_EPOCH.
> ---
>  package/sysstat/sysstat.mk | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/package/sysstat/sysstat.mk b/package/sysstat/sysstat.mk
> index 7568a4b..5cc6cd6 100644
> --- a/package/sysstat/sysstat.mk
> +++ b/package/sysstat/sysstat.mk
> @@ -17,6 +17,25 @@ SYSSTAT_DEPENDENCIES += gettext
>  SYSSTAT_MAKE_OPTS += LFLAGS="$(TARGET_LDFLAGS) -lintl"
>  endif
>
> +SYSSTAT_CONF_OPTS += \
> +	sa_lib_dir=/usr/lib/sa
> +
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +SYSSTAT_SOURCE_DATE = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%b %_d %Y'`
> +SYSSTAT_SOURCE_TIME = `TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH) '+%T'`
> +
> +SYSSTAT_CONF_OPTS += \
> +	CFLAGS="$(TARGET_CFLAGS) -Wno-builtin-macro-redefined -include $(SYSSTAT_DIR)/.br_date_override.h"
> +
> +define SYSSTAT_OVERRIDE_DATE_HOOK
> +	echo "#define buildroot_stringify(x) #x" > $(SYSSTAT_DIR)/.br_date_override.h
> +	echo "#define __DATE__ buildroot_stringify($(SYSSTAT_SOURCE_DATE))" >> $(SYSSTAT_DIR)/.br_date_override.h
> +	echo "#define __TIME__ buildroot_stringify($(SYSSTAT_SOURCE_TIME))" >> $(SYSSTAT_DIR)/.br_date_override.h

  Actually, we have the toolchain wrapper that could do this override much more 
elegantly, and globally for all packages.

  Regards,
  Arnout

> +endef
> +
> +SYSSTAT_POST_PATCH_HOOKS += SYSSTAT_OVERRIDE_DATE_HOOK
> +endif
> +
>  # The isag tool is a post processing script that depends on tcl/tk
>  # among other things. So we don't install it.
>  SYSSTAT_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) INSTALL_ISAG=n install
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Reproducible builds

[Thomas Petazzoni]

Hello,

On Sat, 7 May 2016 22:55:35 +0200, Arnout Vandecappelle wrote:

>   Even if it is not regularly tested, I think we can start including this 
> feature. But the help text should clarify that there are no guarantees.

Yes, absolutely. My question was more about what is the long term plan
for this feature. I have no problem with starting to include this
feature before we have such a test infrastructure in place, as long as
the feature is marked experimental.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gustavo Zacarias]

On 07/05/16 10:28, Thomas Petazzoni wrote:

> Which has the annoying side effect of increasing even more the set of
> mandatory dependencies to build libglib :-/
>
> So I wonder if the dependency on host-python should be made conditional
> on BR2_REPRODUCIBLE.

Hi.
Generally, there's no use case of having gdbus-codegen in the target 
since it's used at build time.

> Since then, Gustavo has already modified the libglib2 package to
> remove the /usr/share/glib-2.0/codegen directory. However,
> gdbus-codegen is not removed.
>
> See commit 1f1f16e9e5367048faa1cf17237e0c0d422e98d6.

Exactly, gdbus-codegen is a leftover that i missed, it can be safely 
removed.
Regards.

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Thomas Petazzoni]

Hello,

On Sun, 8 May 2016 09:51:32 -0300, Gustavo Zacarias wrote:

> Exactly, gdbus-codegen is a leftover that i missed, it can be safely 
> removed.

Thanks for confirming. Could you submit a patch doing this?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 13/34] reproducibility/libtool: fix reproducibility issues

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 10:22:31PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> This probably should be several separate patches.
> 
> On Sat, 30 Apr 2016 09:49:09 +0200, Gilles Chanteperdrix wrote:
> > - remove libtool and libtoolize; they are full of references to the
> >   build environment tools, and unlikely to be needed on target;
> > - do not use /etc/ld.so.conf contents in libtool search path: they are
> >   not relevant on target, and cause a useless dependency of libltdl
> >   on the build environment.
> > ---
> >  package/libtool/0001-fix-cross-compilation.patch | 11 +++++++++++
> >  package/libtool/libtool.mk                       |  8 ++++++--
> >  2 files changed, 17 insertions(+), 2 deletions(-)
> >  create mode 100644 package/libtool/0001-fix-cross-compilation.patch
> > 
> > diff --git a/package/libtool/0001-fix-cross-compilation.patch b/package/libtool/0001-fix-cross-compilation.patch
> > new file mode 100644
> > index 0000000..c203371
> > --- /dev/null
> > +++ b/package/libtool/0001-fix-cross-compilation.patch
> 
> This patch should have a description + Signed-off-by.
> 
> However, libtool is really a core package, and not a simple one, so I'm
> a bit uneasy with the idea of carrying patches against it. What are the
> chances of having something like this submitted upstream?
> 
> > +   fi
> > diff --git a/package/libtool/libtool.mk b/package/libtool/libtool.mk
> > index 8688f8b..c177626 100644
> > --- a/package/libtool/libtool.mk
> > +++ b/package/libtool/libtool.mk
> > @@ -8,7 +8,7 @@ LIBTOOL_VERSION = 2.4.6
> >  LIBTOOL_SOURCE = libtool-$(LIBTOOL_VERSION).tar.xz
> >  LIBTOOL_SITE = $(BR2_GNU_MIRROR)/libtool
> >  LIBTOOL_INSTALL_STAGING = YES
> > -LIBTOOL_CONF_ENV = HELP2MAN=true
> > +LIBTOOL_CONF_OPTS = HELP2MAN=true
> 
> Why?

Well, passing autoconf configure arguments on its command line and
not in its environment has been the recommended way for something
like 16 years. It is recommended in the "goat book", published in
2000.

If you pass the argument on the command line, its gets built into
config.status, so that if config.status needs to be run again, the
option gets passed to configure again. On the other hand, if you use
the environment, if config.status is not run in an environment where
the option is defined, the option gets forgotten.

We patch the autotools files and (have to) run autoreconf, and for
a reason of file dates I forgot, when we run make, config.status is
re-run without the LIBTOOL_CONV_ENV, and the HELP2MAN variable gets
forgotten. 

> 
> >  LIBTOOL_DEPENDENCIES = host-m4
> >  LIBTOOL_LICENSE = GPLv2+
> >  LIBTOOL_LICENSE_FILES = COPYING
> > @@ -33,9 +33,13 @@ define LIBTOOL_AVOID_AUTORECONF_HOOK
> >  	find $(@D) -name configure -exec touch '{}' \;
> >  	find $(@D) -name Makefile.in -exec touch '{}' \;
> >  endef
> > -LIBTOOL_PRE_CONFIGURE_HOOKS += LIBTOOL_AVOID_AUTORECONF_HOOK
> 
> Why?

Because the patch touches the autoconf files.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 04/34] reproducibility: make rootfs.tar reproducible

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 09:51:36PM +0200, Arnout Vandecappelle wrote:
> On 05/07/16 15:23, Thomas Petazzoni wrote:
> > Hello,
> >
> > On Sat, 30 Apr 2016 09:49:00 +0200, Gilles Chanteperdrix wrote:
> >
> >> +define ROOTFS_TAR_CMD
> >> +	cd $(TARGET_DIR) && { \
> >> +		find . -\( -! -type d -o -empty -\) -print0 | \
> >> +		sort -z | \
> >> +		tar --null -T - -c$(TAR_OPTS)f $@ --mtime=@$(SOURCE_DATE_EPOCH) --numeric-owner; \
> >> +	}
> >
> > We normally write such constructs as:
> >
> > 	(cd $(TARGET_DIR) && \
> > 		foo ....)
> 
>   Actually, we don't AFAIK... In general, the parenthesis are not needed so they 
> should be removed. So also in this case it should be
> 
> 	cd $(TARGET_DIR);
> 		find ....
> 
> >
> > However, this raises the question of what's needed for all the other
> > filesystem formats. Will they all have to implement a different
> > ROOTFS_<foo>_CMD variable ? Or will there be some commonalities that
> > should be factored out in the common rootfs image infrastructure ?
> 
>   Yes, to me it makes more sense to do this in the actual target directory after 
> the post-build scripts have been run.

I do not understand what you mean. The aim of the command is to sort
the list of files passed to tar, it does not operate on the file
system.

Sorting is indeed needed for most other outputs I have tested (cpio,
isofs).

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 19/34] reproducibility/python: make python reproducible

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 10:29:26PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 30 Apr 2016 09:49:15 +0200, Gilles Chanteperdrix wrote:
> > python build is not reproducible for two reasons:
> > - the libpython library embeds the build timestamps, fix this by
> >   overriding the __DATE__ and __TIME__ macro using SOURCE_DATE_EPOCH;
> > - _sysconfigdata.py and config/Makefile on target includes references
> >   through INSTALL and MKDIR_P variables to host tools, fix this by
> >   overriding these variables values with the corresponding paths on
> >   buildroot rootfs.
> 
> For this second part, are you really aiming at having reproducible
> builds with different build directories ? As you say in your cover
> letter, that's an even more complicated problem.

No, the aim is to get binaries which do not depend on the location
of "install" and "mkdir" on the host file system, but rather get
them to depend on their location on the target file system, since it
is where theses files are going to be used.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 22/34] High availability: add packages

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 10:38:19PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 30 Apr 2016 09:49:18 +0200, Gilles Chanteperdrix wrote:
> > cluster-glue
> > corosync
> > heartbeat
> > libqb
> > pacemaker
> 
> This should be split in a series of patches, with one package per patch.

Please ignore this patch, the packages are not even fully integrated
and will not start automatically on boot. I did not intend to post it.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> > When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> > environment variable which contains either the date of buildroot last
> > commit if running from a git repository, or the latest release date.
> >
> > This means that all packages embedding build dates will appear to
> > have the same build date, so in case of new commit or release, all
> > packages will appear to have been change, even though some of them
> > may not have changed in fact.
> > ---
> >  Makefile | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/Makefile b/Makefile
> > index 86b2ed2..c1d2961 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
> >  export TZ=UTC
> >  export LANG=C
> >  export LC_ALL=C
> > +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> > +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> > +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> > +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> > +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> 
>   I really don't like exporting a variable that is not well-known and that 
> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in 
> Buildroot-specific patches, so we can just as well add the BR_ prefix.

SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
specification:
https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal

> 
>   But actually, we don't want to introduce buildroot-specific (and therefore 
> unupstreamable) things in patches either. So there is not much point in 
> exporting, except if it's used by scripts.

I used it in some patched configure scripts (libnspr, if I remember
correctly). Which is why I exported it.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> > But this is not sufficient, compiling the python bytecode with the
> > same interpreter in different environments yields different binaries,
> 
>   Er, this is worrisome... You are saying that we don't have a chance in hell of 
> generating reproducible python bytecode?

I am not a python specialist, but it seems to me four bytes in the
python generated bytecode are the build timestamp, so unless there
is a way to override it with SOURCE_DATE_EPOCH, I do not see that
possible. 

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 11/34] reproducibility/vim: make vim reproducible

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 03:30:06PM +0200, Thomas Petazzoni wrote:
> > +define VIM_FIX_AWK_REFERENCE
> > +	sed -i 's,^#!.*awk,#!/usr/bin/awk,' $(TARGET_DIR)/usr/share/vim/vim74/tools/mve.awk
> > +endef
> > +
> > +VIM_POST_INSTALL_TARGET_HOOKS += VIM_FIX_AWK_REFERENCE
> 
> Do we even need this awk script installed?

How should I know, I am an emacs user...

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 17/34] reproducibility/libnspr: override build timestamps

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 10:27:07PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 30 Apr 2016 09:49:13 +0200, Gilles Chanteperdrix wrote:
> > with SOURCE_DATE_EPOCH
> 
> Missing SoB line here.
> 
> > ---
> >  package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch | 71 ++++++++++++++++++++++
> >  1 file changed, 71 insertions(+)
> >  create mode 100644 package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
> > 
> > diff --git a/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
> > new file mode 100644
> > index 0000000..1a0fb7c
> > --- /dev/null
> > +++ b/package/libnspr/0003-honor-SOURCE_DATE_EPOCH.patch
> 
> Missing patch description and Signed-off-by line. Also, did you submit
> this upstream?

No, it looks a bit specific to me.

> Does Debian carry a similar patch for its Reproducible Builds effort?

No idea.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 23/34] pkg-download: allow git+ssh URLs

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Sat, 30 Apr 2016 09:49:19 +0200, Gilles Chanteperdrix wrote:
 >> ---
 >> package/pkg-download.mk | 2 +-
 >> 1 file changed, 1 insertion(+), 1 deletion(-)

 > This looks good I believe, so please resend with your SoB line. Maybe
 > an update to the Buildroot manual is also needed.

I'm not sure we want to advertise this feature too much (or at least
mention it is deprecated). I've personally not heard about git+ssh://,
and googling around I didn't find anything right away.

The only reference I have found is in the git source code:

static enum protocol get_protocol(const char *name)
{
	if (!strcmp(name, "ssh"))
		return PROTO_SSH;
	if (!strcmp(name, "git"))
		return PROTO_GIT;
	if (!strcmp(name, "git+ssh")) /* deprecated - do not use */
		return PROTO_SSH;
	if (!strcmp(name, "ssh+git")) /* deprecated - do not use */
		return PROTO_SSH;
	if (!strcmp(name, "file"))
		return PROTO_FILE;
	die("I don't handle protocol '%s'", name);
}

So the question is how long git will continue to support it.

With that said, I don't have a problem with us detecting git+ssh:// (and
perhaps ssh+git://) as git URLs and forwarding it to the download
wrapper - But if we start documenting it as supported we should continue
to do so, so we might end up having to rewrite git+ssh:// -> ssh:// URLs
and so on when/if git stops supporting them natively.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 21/34] hddtemp: add

[Peter Korsgaard]

>>>>> "Gilles" == Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

 > ---
 >  package/Config.in            |  1 +
 >  package/hddtemp/Config.in    |  8 ++++++++
 >  package/hddtemp/hddtemp.hash |  2 ++
 >  package/hddtemp/hddtemp.mk   | 19 +++++++++++++++++++
 >  4 files changed, 30 insertions(+)
 >  create mode 100644 package/hddtemp/Config.in
 >  create mode 100644 package/hddtemp/hddtemp.hash
 >  create mode 100644 package/hddtemp/hddtemp.mk

 > diff --git a/package/Config.in b/package/Config.in
 > index bdc3063..76665e6 100644
 > --- a/package/Config.in
 > +++ b/package/Config.in
 > @@ -1458,6 +1458,7 @@ comment "Utilities"
 >  	source "package/file/Config.in"
 >  	source "package/gnupg/Config.in"
 >  	source "package/gnupg2/Config.in"
 > +	source "package/hddtemp/Config.in"

I think this should rather go under "Hardware handling" like
smartmontools or hdparm.


 >  	source "package/inotify-tools/Config.in"
 >  	source "package/lockfile-progs/Config.in"
 >  	source "package/logrotate/Config.in"
 > diff --git a/package/hddtemp/Config.in b/package/hddtemp/Config.in
 > new file mode 100644
 > index 0000000..0d28250
 > --- /dev/null
 > +++ b/package/hddtemp/Config.in
 > @@ -0,0 +1,8 @@
 > +config BR2_PACKAGE_HDDTEMP
 > +	bool "hddtemp"
 > +	help
 > +	  hddtemp is a small utility that gives you the temperature of your
 > +	  hard drive by reading S.M.A.R.T. informations, for drives that
 > +	  support this feature. 

It also uses iconv (to figure out how to print a degrees celsius sign,
gaah), so you need to add:

       select BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE

And add it to HDDTEMP_DEPENDENCIES. A quick test shows that it forgets
to link with it, so it needs some help:

ifeq ($(BR2_PACKAGE_LIBICONV),y)
HDDTEMP_DEPENDENCIES += libiconv
HDDTEMP_CONF_ENV += LIBS="-liconv"
endif

It also needs a small patch to build without gettext as it
unconditionally calls:

  setlocale (LC_ALL, "");
  bindtextdomain (PACKAGE, LOCALEDIR);
  textdomain (PACKAGE);

Those needs to be inside #ifdef ENABLE_NLS conditionals.

And finally there's code in backtrace.c that breaks when building with
uClibc on i386 (as uClibc also defines __GLIBC__ for compatibility but
doesn't provide backtrace functionality by default):

#if defined(__i386__) && defined(__GLIBC__)

These patches should preferably be upstreamed, but as the latest (beta!)
release is 10 years old by now, the project seems pretty dead.

The savannah status is also listed as Orphaned/Unmaintained.


 > +
 > +	  http://www.guzu.net/linux/hddtemp.php
 > diff --git a/package/hddtemp/hddtemp.hash b/package/hddtemp/hddtemp.hash
 > new file mode 100644
 > index 0000000..c308989
 > --- /dev/null
 > +++ b/package/hddtemp/hddtemp.hash
 > @@ -0,0 +1,2 @@
 > +# Locally calculated after checking pgp signature
 > +sha256	618541584054093d53be8a2d9e81c97174f30f00af91cb8700a97e442d79ef5b  hddtemp-0.3-beta15.tar.bz2
 > diff --git a/package/hddtemp/hddtemp.mk b/package/hddtemp/hddtemp.mk
 > new file mode 100644
 > index 0000000..a51548d
 > --- /dev/null
 > +++ b/package/hddtemp/hddtemp.mk
 > @@ -0,0 +1,19 @@
 > +################################################################################
 > +#
 > +# hddtemp
 > +#
 > +################################################################################
 > +
 > +HDDTEMP_VERSION = 0.3-beta15
 > +HDDTEMP_SOURCE = hddtemp-$(HDDTEMP_VERSION).tar.bz2
 > +HDDTEMP_SITE = http://download.savannah.gnu.org/releases/hddtemp
 > +HDDTEMP_LICENSE = GPLv2

The source files contain the "(at your option) any later version"
wording, so this should be GPLv2+


> +HDDTEMP_LICENSE_FILES = GPL-2
 > +
 > +define HDDTEMP_REMOVE_MAN
 > +	rm -rf $(TARGET_DIR)/usr/man
 > +endef

Like Thomas mentioned, this is already taken care of in target-finalize.


The big question is how to handle hddtemp.db?

The tarball doesn't contain the file, so hddtemp doesn't do anything
sensible:

sudo ./target/usr/sbin/hddtemp /dev/sda
hddtemp: can't open /usr/share/misc/hddtemp.db: No such file or directory

The file is available on the upstream website, but it is not versioned -
So that is not suitable for Buildroot:

http://www.guzu.net/linux/hddtemp.db

Either we:

- Convince upstream to version this file (and preferably include it in
  the tarball) - But as the project seems dead this might be hard

- Apply the Debian patch which includes (an older version of?)
  hddtemp.db. Notice that Debian also patches various things in the
  code, so please check that the changes make sense in the context of
  Buildroot:
  http://http.debian.net/debian/pool/main/h/hddtemp/hddtemp_0.3-beta15-52.diff.gz

- We include hddtemp.db in package/hddtemp. It isn't really nice, but
  the file is relatively small (29K)

Given the missing hddtemp.db, how have you tested this package?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Peter Korsgaard]

>>>>> "Gilles" == Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

 > Making the builds reproducible involves turning off build timestamps,
 > which some users may find annoying. So make the reproducible builds
 > optional.
 > ---
 >  Config.in | 3 +++
 >  1 file changed, 3 insertions(+)

 > diff --git a/Config.in b/Config.in
 > index d795361..3cdd659 100644
 > --- a/Config.in
 > +++ b/Config.in
 > @@ -623,6 +623,9 @@ config BR2_COMPILER_PARANOID_UNSAFE_PATH
 
 >  endmenu
 
 > +config BR2_REPRODUCIBLE
 > +	bool "Try and remove all sources of non-reproducibility from the build"
 > +

Hmm, I'm not sure I like the extra complexity of 2 ways of doing
things. What are the use cases for these timestamps? Are they important
enough to keep?

In any case, as enabling this option makes us do "more" (E.G. pass extra
arguments to tools), we should imho enable it by default to ensure those
code paths gets tested.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Thomas Petazzoni]

Hello,

On Mon, 09 May 2016 09:42:05 +0200, Peter Korsgaard wrote:

> Hmm, I'm not sure I like the extra complexity of 2 ways of doing
> things. What are the use cases for these timestamps? Are they important
> enough to keep?

I use timestamps all the time to verify that I'm indeed running the
version of the code I intend to run. As I work on the kernel, I use the
kernel timestamp, but I really use it all the time, to double check
that I'm running the kernel image I just built and not one that was
left around, built several days/weeks ago.

So timestamps are useful IMO.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Mon, 09 May 2016 09:42:05 +0200, Peter Korsgaard wrote:

 >> Hmm, I'm not sure I like the extra complexity of 2 ways of doing
 >> things. What are the use cases for these timestamps? Are they important
 >> enough to keep?

 > I use timestamps all the time to verify that I'm indeed running the
 > version of the code I intend to run. As I work on the kernel, I use the
 > kernel timestamp, but I really use it all the time, to double check
 > that I'm running the kernel image I just built and not one that was
 > left around, built several days/weeks ago.

Ok, what timestamps exactly? Embedded timestamps like uname -v or
filesystem timestamps of the files?

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 01/34] reproducibility: introduce config knob

[Thomas Petazzoni]

Hello,

On Mon, 09 May 2016 15:01:53 +0200, Peter Korsgaard wrote:

>  > I use timestamps all the time to verify that I'm indeed running the
>  > version of the code I intend to run. As I work on the kernel, I use the
>  > kernel timestamp, but I really use it all the time, to double check
>  > that I'm running the kernel image I just built and not one that was
>  > left around, built several days/weeks ago.  
> 
> Ok, what timestamps exactly? Embedded timestamps like uname -v or
> filesystem timestamps of the files?

The one I use for the kernel is obviously the embedded time stamp,
displayed at boot time (and also in uname -v).

I guess the filesystem time stamps are not that useful, but they are
the "easy" part (i.e they can be fixed globally when generating the
filesystem image). The embedded timestamps all over the place in
different programs/libraries are really the annoying part.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 23/34] pkg-download: allow git+ssh URLs

[Gilles Chanteperdrix]

On Mon, May 09, 2016 at 08:41:09AM +0200, Peter Korsgaard wrote:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Hello,
>  > On Sat, 30 Apr 2016 09:49:19 +0200, Gilles Chanteperdrix wrote:
>  >> ---
>  >> package/pkg-download.mk | 2 +-
>  >> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
>  > This looks good I believe, so please resend with your SoB line. Maybe
>  > an update to the Buildroot manual is also needed.
> 
> I'm not sure we want to advertise this feature too much (or at least
> mention it is deprecated). I've personally not heard about git+ssh://,
> and googling around I didn't find anything right away.

Well, I always used git+ssh://, I did not even know it was
deprecated. The advantage of git+ssh is that the scheme clearly
identifies git over ssh, so we can choose the git helper. If we got
the ssh scheme to do the same and other revision control systems
used ssh URL (I do not even know, I have not been using something
else than git for a long time), it is not going to be possible to
find out which helper to use.

One way or another, I believe it is helpful to be able to use git
over ssh URLs, are they may be used for projects specific to a
company, not public.

Regards.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 21/34] hddtemp: add

[Gilles Chanteperdrix]

On Mon, May 09, 2016 at 09:38:06AM +0200, Peter Korsgaard wrote:
> >>>>> "Gilles" == Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> 
>  > ---
>  >  package/Config.in            |  1 +
>  >  package/hddtemp/Config.in    |  8 ++++++++
>  >  package/hddtemp/hddtemp.hash |  2 ++
>  >  package/hddtemp/hddtemp.mk   | 19 +++++++++++++++++++
>  >  4 files changed, 30 insertions(+)
>  >  create mode 100644 package/hddtemp/Config.in
>  >  create mode 100644 package/hddtemp/hddtemp.hash
>  >  create mode 100644 package/hddtemp/hddtemp.mk
> 
>  > diff --git a/package/Config.in b/package/Config.in
>  > index bdc3063..76665e6 100644
>  > --- a/package/Config.in
>  > +++ b/package/Config.in
>  > @@ -1458,6 +1458,7 @@ comment "Utilities"
>  >  	source "package/file/Config.in"
>  >  	source "package/gnupg/Config.in"
>  >  	source "package/gnupg2/Config.in"
>  > +	source "package/hddtemp/Config.in"
> 
> I think this should rather go under "Hardware handling" like
> smartmontools or hdparm.
> 
> 
>  >  	source "package/inotify-tools/Config.in"
>  >  	source "package/lockfile-progs/Config.in"
>  >  	source "package/logrotate/Config.in"
>  > diff --git a/package/hddtemp/Config.in b/package/hddtemp/Config.in
>  > new file mode 100644
>  > index 0000000..0d28250
>  > --- /dev/null
>  > +++ b/package/hddtemp/Config.in
>  > @@ -0,0 +1,8 @@
>  > +config BR2_PACKAGE_HDDTEMP
>  > +	bool "hddtemp"
>  > +	help
>  > +	  hddtemp is a small utility that gives you the temperature of your
>  > +	  hard drive by reading S.M.A.R.T. informations, for drives that
>  > +	  support this feature. 
> 
> It also uses iconv (to figure out how to print a degrees celsius sign,
> gaah), so you need to add:
> 
>        select BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE
> 
> And add it to HDDTEMP_DEPENDENCIES. A quick test shows that it forgets
> to link with it, so it needs some help:
> 
> ifeq ($(BR2_PACKAGE_LIBICONV),y)
> HDDTEMP_DEPENDENCIES += libiconv
> HDDTEMP_CONF_ENV += LIBS="-liconv"
> endif
> 
> It also needs a small patch to build without gettext as it
> unconditionally calls:
> 
>   setlocale (LC_ALL, "");
>   bindtextdomain (PACKAGE, LOCALEDIR);
>   textdomain (PACKAGE);
> 
> Those needs to be inside #ifdef ENABLE_NLS conditionals.
> 
> And finally there's code in backtrace.c that breaks when building with
> uClibc on i386 (as uClibc also defines __GLIBC__ for compatibility but
> doesn't provide backtrace functionality by default):
> 
> #if defined(__i386__) && defined(__GLIBC__)
> 
> These patches should preferably be upstreamed, but as the latest (beta!)
> release is 10 years old by now, the project seems pretty dead.

Ok, let us forget about this patch, as for the "high-availability"
patch, I did not expect to post this patch with the rest, it landed
here only because I posted the patch list, unsorted.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Arnout Vandecappelle]

On 05/08/16 22:22, Gilles Chanteperdrix wrote:
> On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
>> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
>>> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
>>> environment variable which contains either the date of buildroot last
>>> commit if running from a git repository, or the latest release date.
>>>
>>> This means that all packages embedding build dates will appear to
>>> have the same build date, so in case of new commit or release, all
>>> packages will appear to have been change, even though some of them
>>> may not have changed in fact.
>>> ---
>>>  Makefile | 5 +++++
>>>  1 file changed, 5 insertions(+)
>>>
>>> diff --git a/Makefile b/Makefile
>>> index 86b2ed2..c1d2961 100644
>>> --- a/Makefile
>>> +++ b/Makefile
>>> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
>>>  export TZ=UTC
>>>  export LANG=C
>>>  export LC_ALL=C
>>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>>> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
>>> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
>>> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
>>> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
>>
>>   I really don't like exporting a variable that is not well-known and that
>> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in
>> Buildroot-specific patches, so we can just as well add the BR_ prefix.
>
> SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
> specification:
> https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal

  Ah, good.

  Actually, a better reference is
https://reproducible-builds.org/specs/source-date-epoch/
(actually the same developers wrote this spec, but it's supposedly not 
debian-specific).

  When you repost, could you make sure that the commit message mentions this 
explicitly?

>
>>
>>   But actually, we don't want to introduce buildroot-specific (and therefore
>> unupstreamable) things in patches either. So there is not much point in
>> exporting, except if it's used by scripts.
>
> I used it in some patched configure scripts (libnspr, if I remember
> correctly). Which is why I exported it.

  As I said: used in patches.

  But indeed, it is well-known. So OK to export it.

  BTW, when it is possible to download a debian patch for a package (by setting 
FOO_PATCH = ...) rather than rolling our own, we prefer that.

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 04/34] reproducibility: make rootfs.tar reproducible

[Arnout Vandecappelle]

On 05/08/16 22:17, Gilles Chanteperdrix wrote:
> On Sat, May 07, 2016 at 09:51:36PM +0200, Arnout Vandecappelle wrote:
>> On 05/07/16 15:23, Thomas Petazzoni wrote:
>>> Hello,
>>>
>>> On Sat, 30 Apr 2016 09:49:00 +0200, Gilles Chanteperdrix wrote:
>>>
>>>> +define ROOTFS_TAR_CMD
>>>> +	cd $(TARGET_DIR) && { \
>>>> +		find . -\( -! -type d -o -empty -\) -print0 | \
>>>> +		sort -z | \
>>>> +		tar --null -T - -c$(TAR_OPTS)f $@ --mtime=@$(SOURCE_DATE_EPOCH) --numeric-owner; \
>>>> +	}
>>>
>>> We normally write such constructs as:
>>>
>>> 	(cd $(TARGET_DIR) && \
>>> 		foo ....)
>>
>>   Actually, we don't AFAIK... In general, the parenthesis are not needed so they
>> should be removed. So also in this case it should be
>>
>> 	cd $(TARGET_DIR);
>> 		find ....
>>
>>>
>>> However, this raises the question of what's needed for all the other
>>> filesystem formats. Will they all have to implement a different
>>> ROOTFS_<foo>_CMD variable ? Or will there be some commonalities that
>>> should be factored out in the common rootfs image infrastructure ?
>>
>>   Yes, to me it makes more sense to do this in the actual target directory after
>> the post-build scripts have been run.
>
> I do not understand what you mean. The aim of the command is to sort
> the list of files passed to tar, it does not operate on the file
> system.

  Duh, sorry, I was just talking about the --mtime bit.

  Regards,
  Arnout

>
> Sorting is indeed needed for most other outputs I have tested (cpio,
> isofs).
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Arnout Vandecappelle]

On 05/08/16 22:25, Gilles Chanteperdrix wrote:
> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
>> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
>>> But this is not sufficient, compiling the python bytecode with the
>>> same interpreter in different environments yields different binaries,
>>
>>   Er, this is worrisome... You are saying that we don't have a chance in hell of
>> generating reproducible python bytecode?
>
> I am not a python specialist, but it seems to me four bytes in the
> python generated bytecode are the build timestamp, so unless there
> is a way to override it with SOURCE_DATE_EPOCH, I do not see that
> possible.

  I've checked the docs. What is saved is the timestamp of the .py file, not the 
build time.

  I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all 
files after patching to capture this aspect. This was also proposed for 
Fedora[1] (though there it was only for the .py files). Not sure what happened 
with that proposal in the end.

  Regards,
  Arnout

[1] https://lists.fedoraproject.org/pipermail/python-devel/2014-August/000617.html



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gilles Chanteperdrix]

On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
> On 05/08/16 22:25, Gilles Chanteperdrix wrote:
> > On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
> >> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> >>> But this is not sufficient, compiling the python bytecode with the
> >>> same interpreter in different environments yields different binaries,
> >>
> >>   Er, this is worrisome... You are saying that we don't have a chance in hell of
> >> generating reproducible python bytecode?
> >
> > I am not a python specialist, but it seems to me four bytes in the
> > python generated bytecode are the build timestamp, so unless there
> > is a way to override it with SOURCE_DATE_EPOCH, I do not see that
> > possible.
> 
>   I've checked the docs. What is saved is the timestamp of the .py file, not the 
> build time.

Mmmm I don't remember. I would run a compilation manually, twice at
a different time, to make sure that the problem is only the file
date.

> 
>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all 
> files after patching to capture this aspect. This was also proposed for 
> Fedora[1] (though there it was only for the .py files). Not sure what happened 
> with that proposal in the end.

I think I tried running "touch" before compiling, unfortunately
playing with file dates before running "make" is a bit like playing
with fire. For instance with autotools based projects for which
autoreconf is not run, the project may use versions of the autotools
not installed on the user machine, and because of file dates may
want to rerun autoconf or autmake and whine because the right
versions are not available. Doing this only for .py files is much
more reasonable.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 03/34] reproducibility: generate SOURCE_DATE_EPOCH

[Gilles Chanteperdrix]

On Tue, May 10, 2016 at 01:28:14AM +0200, Arnout Vandecappelle wrote:
> On 05/08/16 22:22, Gilles Chanteperdrix wrote:
> > On Sat, May 07, 2016 at 11:00:11PM +0200, Arnout Vandecappelle wrote:
> >> On 04/30/16 09:48, Gilles Chanteperdrix wrote:
> >>> When reproducibility is wanted, generates a global SOURCE_DATE_EPOCH
> >>> environment variable which contains either the date of buildroot last
> >>> commit if running from a git repository, or the latest release date.
> >>>
> >>> This means that all packages embedding build dates will appear to
> >>> have the same build date, so in case of new commit or release, all
> >>> packages will appear to have been change, even though some of them
> >>> may not have changed in fact.
> >>> ---
> >>>  Makefile | 5 +++++
> >>>  1 file changed, 5 insertions(+)
> >>>
> >>> diff --git a/Makefile b/Makefile
> >>> index 86b2ed2..c1d2961 100644
> >>> --- a/Makefile
> >>> +++ b/Makefile
> >>> @@ -223,6 +223,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
> >>>  export TZ=UTC
> >>>  export LANG=C
> >>>  export LC_ALL=C
> >>> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> >>> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> >>> +	sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> >>> +	LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
> >>> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> >>
> >>   I really don't like exporting a variable that is not well-known and that
> >> doesn't have a BR_ prefix. Where it is used by packages, it will anyway be in
> >> Buildroot-specific patches, so we can just as well add the BR_ prefix.
> >
> > SOURCE_DATE_EPOCH is well-known. Its meaning is defined by a
> > specification:
> > https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal
> 
>   Ah, good.
> 
>   Actually, a better reference is
> https://reproducible-builds.org/specs/source-date-epoch/
> (actually the same developers wrote this spec, but it's supposedly not 
> debian-specific).
> 
>   When you repost, could you make sure that the commit message mentions this 
> explicitly?
> 
> >
> >>
> >>   But actually, we don't want to introduce buildroot-specific (and therefore
> >> unupstreamable) things in patches either. So there is not much point in
> >> exporting, except if it's used by scripts.
> >
> > I used it in some patched configure scripts (libnspr, if I remember
> > correctly). Which is why I exported it.
> 
>   As I said: used in patches.
> 
>   But indeed, it is well-known. So OK to export it.
> 
>   BTW, when it is possible to download a debian patch for a package (by setting 
> FOO_PATCH = ...) rather than rolling our own, we prefer that.

The thing is that trying and understanding what Debian does is not
convenient for me and most of the time a waste of time:
- things that I believe are problems for buildroot may not be
considered problems for Debian (see for instance the libltdl issue,
or the grub2 issue). 
- the Debian wiki has a high noise/signal ratio, and is not always
up to date, and this when I can access it, because the IP addresses
I use for browsing are blacklisted by Debian, so I have to take
special steps to access it.

So, I gave up on Debian pretty early, and I am not interested in
trying that painful exercise again. If someone else wants to do it
though, go ahead.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Arnout Vandecappelle]

On 05/10/16 21:42, Gilles Chanteperdrix wrote:
> On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
>> On 05/08/16 22:25, Gilles Chanteperdrix wrote:
>>> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
>>>> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
>>>>> But this is not sufficient, compiling the python bytecode with the
>>>>> same interpreter in different environments yields different binaries,
>>>>
>>>>   Er, this is worrisome... You are saying that we don't have a chance in hell of
>>>> generating reproducible python bytecode?
>>>
>>> I am not a python specialist, but it seems to me four bytes in the
>>> python generated bytecode are the build timestamp, so unless there
>>> is a way to override it with SOURCE_DATE_EPOCH, I do not see that
>>> possible.
>>
>>   I've checked the docs. What is saved is the timestamp of the .py file, not the
>> build time.
>
> Mmmm I don't remember. I would run a compilation manually, twice at
> a different time, to make sure that the problem is only the file
> date.

  I did that - admittedly with just a few seconds difference. Both in 
python2.7.11 and python3.5, they were identical when compiling a second time, 
and different after touch'ing.

>
>>
>>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all
>> files after patching to capture this aspect. This was also proposed for
>> Fedora[1] (though there it was only for the .py files). Not sure what happened
>> with that proposal in the end.
>
> I think I tried running "touch" before compiling, unfortunately
> playing with file dates before running "make" is a bit like playing
> with fire. For instance with autotools based projects for which
> autoreconf is not run, the project may use versions of the autotools
> not installed on the user machine, and because of file dates may
> want to rerun autoconf or autmake and whine because the right
> versions are not available. Doing this only for .py files is much
> more reasonable.

  I tested this as well, with make-3.81 and make-4.04. Both of them do _not_ 
rebuild if the timestamps are identical. And since the idea is to use touch -d 
@$(SOURCE_DATE_EPOCH), all timestamps will be identical.

  But it does indeed mean that if a package has a generated file with an earlier 
date than the source files, it will now suddenly no longer be rebuilt.

  I don't know how the various other build systems (waf, scons) will behave. 
There is indeed a risk, but I'd say we deal with that when it happens. Actually 
I think we would prefer to detect such issues.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 16/34] reproducibility/lzo: forbid lzo with reproducible builds

[Arnout Vandecappelle]

On 05/07/16 22:25, Thomas Petazzoni wrote:
> Hello,
>
> On Sat, 30 Apr 2016 09:49:12 +0200, Gilles Chanteperdrix wrote:
>> The lzo archive contains a timestamp.
>
> Missing Signed-off-by line.
>
>> ---
>>  fs/tar/Config.in | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/fs/tar/Config.in b/fs/tar/Config.in
>> index 63663ec..7e4b603 100644
>> --- a/fs/tar/Config.in
>> +++ b/fs/tar/Config.in
>> @@ -33,6 +33,7 @@ config BR2_TARGET_ROOTFS_TAR_LZMA
>>
>>  config BR2_TARGET_ROOTFS_TAR_LZO
>>  	bool "lzo"
>> +	depends on !BR2_REPRODUCIBLE
>>  	help
>>  	  Do compress the tarball with lzop.
>>
>> @@ -43,6 +44,12 @@ config BR2_TARGET_ROOTFS_TAR_XZ
>>
>>  endchoice
>>
>> +if BR2_REPRODUCIBLE
>> +
>> +comment "lzo compression is only available with non-reproducible builds"
>> +
>> +endif
>
> When we are in such situations, we normally do:
>
> comment "blablabla"
> 	depends on BR2_REPRODUCIBLE
>
> however, in this specific case, I would simply not bother adding a
> comment. If we start adding a comment for everything that is not
> suitable for reproducible builds, we're going to add hundreds of them I
> believe.

  IMHO, it should be different: things that are not reproducible should not be 
excluded, but instead a comment should be shown to warn the user the build is 
not reproducible. Perhaps like for legacy: a config option that is not 
reproducible selects BR2_NOT_REPRODUCIBLE (a blind option), and at the bottom of 
the top level menu we add a comment that depends on BR2_REPRODUCIBLE && 
BR2_NOT_REPRODUCIBLE. It will be a bit tricky to find out exactly which option 
is the one that triggers this comment, but it's doable by searching for 
BR2_NOT_REPRODUCIBLE and seeing which one in the long list of selectors is y.


  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 10:05:36PM +0200, Arnout Vandecappelle wrote:
> On 05/10/16 21:42, Gilles Chanteperdrix wrote:
> > On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
> >> On 05/08/16 22:25, Gilles Chanteperdrix wrote:
> >>> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
> >>>> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
> >>>>> But this is not sufficient, compiling the python bytecode with the
> >>>>> same interpreter in different environments yields different binaries,
> >>>>
> >>>>   Er, this is worrisome... You are saying that we don't have a chance in hell of
> >>>> generating reproducible python bytecode?
> >>>
> >>> I am not a python specialist, but it seems to me four bytes in the
> >>> python generated bytecode are the build timestamp, so unless there
> >>> is a way to override it with SOURCE_DATE_EPOCH, I do not see that
> >>> possible.
> >>
> >>   I've checked the docs. What is saved is the timestamp of the .py file, not the
> >> build time.
> >
> > Mmmm I don't remember. I would run a compilation manually, twice at
> > a different time, to make sure that the problem is only the file
> > date.
> 
>   I did that - admittedly with just a few seconds difference. Both in 
> python2.7.11 and python3.5, they were identical when compiling a second time, 
> and different after touch'ing.
> 
> >
> >>
> >>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all
> >> files after patching to capture this aspect. This was also proposed for
> >> Fedora[1] (though there it was only for the .py files). Not sure what happened
> >> with that proposal in the end.
> >
> > I think I tried running "touch" before compiling, unfortunately
> > playing with file dates before running "make" is a bit like playing
> > with fire. For instance with autotools based projects for which
> > autoreconf is not run, the project may use versions of the autotools
> > not installed on the user machine, and because of file dates may
> > want to rerun autoconf or autmake and whine because the right
> > versions are not available. Doing this only for .py files is much
> > more reasonable.
> 
>   I tested this as well, with make-3.81 and make-4.04. Both of them do _not_ 
> rebuild if the timestamps are identical. And since the idea is to use touch -d 
> @$(SOURCE_DATE_EPOCH), all timestamps will be identical.

You checked both the pyo and the pyc?

> 
>   But it does indeed mean that if a package has a generated file with an earlier 
> date than the source files, it will now suddenly no longer be
> rebuilt.

Yes, that is another problem. But I tried it, and this is not the
one I had, the one I had was the contrary: the dates made make want
to rebuild some files (the autotools/automake files), whereas the
right versions of autoconf and automake were not installed, this was
a package that did not run autoreconf. The "make" tool is completely
based on file dates, so again, I think messing with file dates
before running make is a bad idea.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Arnout Vandecappelle]

On 05/14/16 15:34, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 10:05:36PM +0200, Arnout Vandecappelle wrote:
>> On 05/10/16 21:42, Gilles Chanteperdrix wrote:
>>> On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
[snip]
>>>>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all
>>>> files after patching to capture this aspect. This was also proposed for
>>>> Fedora[1] (though there it was only for the .py files). Not sure what happened
>>>> with that proposal in the end.
>>>
>>> I think I tried running "touch" before compiling, unfortunately
>>> playing with file dates before running "make" is a bit like playing
>>> with fire. For instance with autotools based projects for which
>>> autoreconf is not run, the project may use versions of the autotools
>>> not installed on the user machine, and because of file dates may
>>> want to rerun autoconf or autmake and whine because the right
>>> versions are not available. Doing this only for .py files is much
>>> more reasonable.
>>
>>   I tested this as well, with make-3.81 and make-4.04. Both of them do _not_
>> rebuild if the timestamps are identical. And since the idea is to use touch -d
>> @$(SOURCE_DATE_EPOCH), all timestamps will be identical.
>
> You checked both the pyo and the pyc?

  Yep.


>>   But it does indeed mean that if a package has a generated file with an earlier
>> date than the source files, it will now suddenly no longer be
>> rebuilt.
>
> Yes, that is another problem. But I tried it, and this is not the
> one I had, the one I had was the contrary: the dates made make want
> to rebuild some files (the autotools/automake files), whereas the
> right versions of autoconf and automake were not installed, this was
> a package that did not run autoreconf. The "make" tool is completely
> based on file dates, so again, I think messing with file dates
> before running make is a bad idea.

  Hm, I don't see how this could happen... My test showed that 'make' would not 
try to rebuild the autotools/automake files when they have the same timestamp as 
the corresponding source file. So what triggers the reconf then?

  I've tried with a random package (bash) and it's not reconfiguring after 
touching all sources. Do you remember for which package you saw this?


  Regards,
  Arnout



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 02/34] reproducibility: override locale and timezone

[Gilles Chanteperdrix]

On Sat, May 07, 2016 at 03:15:50PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 30 Apr 2016 09:48:58 +0200, Gilles Chanteperdrix wrote:
> > ---
> >  Makefile | 7 +++++++
> >  1 file changed, 7 insertions(+)
> 
> Missing SoB.
> 
> > diff --git a/Makefile b/Makefile
> > index ec7dccb..86b2ed2 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -218,6 +218,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
> >  -include $(BR2_CONFIG)
> >  endif
> >  
> > +export BR2_REPRODUCIBLE
> 
> Why ?

So that it can be used in scripts. Like wrappers script. In order to
know if we should take special actions because we are targeting a
reproducible build.

> 
> > +ifeq ($(BR2_REPRODUCIBLE),y)
> > +export TZ=UTC
> > +export LANG=C
> > +export LC_ALL=C
> > +endif
> 
> Regarding overriding TZ, LANG and LC_ALL, I'm wondering if we shouldn't
> do it unconditionally, regardless of the value of BR2_REPRODUCIBLE.
> What is the benefit of leaving those variable to their value from the
> build environment?

Obviously:
- having dates printed for your local timezone (like build timestamps
if you do not want a reproducible build). I do not know about you,
but I would hate to have to convert a build timestamp from UTC to
local timezone in order to see if I am running the binary I think I
am running.
- having compiler messages printed in your locale, if you set the
locale, it is probably what you want.

-- 
					    Gilles.
https://click-hack.org

[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Gilles Chanteperdrix]

On Sun, May 15, 2016 at 01:48:29AM +0200, Arnout Vandecappelle wrote:
> On 05/14/16 15:34, Gilles Chanteperdrix wrote:
> > On Thu, May 12, 2016 at 10:05:36PM +0200, Arnout Vandecappelle wrote:
> >> On 05/10/16 21:42, Gilles Chanteperdrix wrote:
> >>> On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
> [snip]
> >>>>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all
> >>>> files after patching to capture this aspect. This was also proposed for
> >>>> Fedora[1] (though there it was only for the .py files). Not sure what happened
> >>>> with that proposal in the end.
> >>>
> >>> I think I tried running "touch" before compiling, unfortunately
> >>> playing with file dates before running "make" is a bit like playing
> >>> with fire. For instance with autotools based projects for which
> >>> autoreconf is not run, the project may use versions of the autotools
> >>> not installed on the user machine, and because of file dates may
> >>> want to rerun autoconf or autmake and whine because the right
> >>> versions are not available. Doing this only for .py files is much
> >>> more reasonable.
> >>
> >>   I tested this as well, with make-3.81 and make-4.04. Both of them do _not_
> >> rebuild if the timestamps are identical. And since the idea is to use touch -d
> >> @$(SOURCE_DATE_EPOCH), all timestamps will be identical.
> >
> > You checked both the pyo and the pyc?
> 
>   Yep.
> 
> 
> >>   But it does indeed mean that if a package has a generated file with an earlier
> >> date than the source files, it will now suddenly no longer be
> >> rebuilt.
> >
> > Yes, that is another problem. But I tried it, and this is not the
> > one I had, the one I had was the contrary: the dates made make want
> > to rebuild some files (the autotools/automake files), whereas the
> > right versions of autoconf and automake were not installed, this was
> > a package that did not run autoreconf. The "make" tool is completely
> > based on file dates, so again, I think messing with file dates
> > before running make is a bad idea.
> 
>   Hm, I don't see how this could happen... My test showed that 'make' would not 
> try to rebuild the autotools/automake files when they have the same timestamp as 
> the corresponding source file. So what triggers the reconf then?
> 
>   I've tried with a random package (bash) and it's not reconfiguring after 
> touching all sources. Do you remember for which package you saw this?

Actually, I can not remember what I did. I may not have used
SOURCE_DATE_EPOCH. Anyway, I still find touching all the source
files can have unintended effects, and it is not sufficient either:
- some py files may be generated by configure
- some py files may be generated by Makefiles

So, I would suggest to add a python compiler wrapper which touches
the files with SOURCE_DATE_EPOCH before compiling them. That would
cover all the cases and limit the unintended effects. The downside
is that you have to get all packages compiling python files to use
that wrapper instead of the python compiler.

-- 
					    Gilles.
https://click-hack.org