* [Buildroot] Reproducible build v2
@ 2016-06-14 15:29 Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
2016-06-14 15:32 ` [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date Gilles Chanteperdrix
0 siblings, 2 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:29 UTC (permalink / raw)
To: buildroot
Hi,
Please find a second version of the patches, containing all patches
which did not raise objections, or which have been fixed according
to your advices.
The patches follow this mail and are available as a git branch:
https://git.click-hack.org/buildroot.git/log/?h=reproducible
Some modifications have been left out this time around, because as
has been said in answer to the first set, because they need to be
factored rather than fixed again and again for every package.
Namely:
- overriding gcc macros containing build timestamps in packages;
- touching with SOURCE_DATE_EPOCH all files before starting;
archiving (whether with tar, cpio, or genisoimage);
- touching the tarball before compressing it, in order to allow
generating images in format where the compressor does not allow
disabling timestamps (such as lzo);
- sort file list when archiving so that the archive members order do
not depend on the order of files in the directory.
Unfortunately, I have other more urgent issues to work on, so I am
going to abandon this work on buildroot. If anyone is interested in
continuing this work, you can also find my original patches in a git
branch too:
https://git.click-hack.org/buildroot.git/log/?h=reproducible.orig
Regards.
--
Gilles.
https://click-hack.org
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 01/12] reproducibility: introduce config knob
2016-06-14 15:29 [Buildroot] Reproducible build v2 Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
` (9 more replies)
2016-06-14 15:32 ` [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date Gilles Chanteperdrix
1 sibling, 10 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
Making the builds reproducible involves turning off build timestamps,
which some users may find annoying. So make the reproducible builds
optional.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
Config.in | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/Config.in b/Config.in
index 9fa7415..faf4c71 100644
--- a/Config.in
+++ b/Config.in
@@ -724,6 +724,17 @@ config BR2_COMPILER_PARANOID_UNSAFE_PATH
endmenu
+config BR2_REPRODUCIBLE
+ bool "Make the build reproducible (experimental)"
+ help
+ This option will remove all sources of non-reproducibility
+ from the build process. For a given Buildroot configuration,
+ this allows to generate exactly identical binaries from one
+ build to the other, including on different machines.
+
+ This is labeled as an experimental feature, as not all
+ packages behave properly to ensure reproducibility.
+
endmenu
source "toolchain/Config.in"
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:01 ` Yann E. MORIN
2016-07-02 9:58 ` Peter Korsgaard
2016-06-14 15:31 ` [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH Gilles Chanteperdrix
` (8 subsequent siblings)
9 siblings, 2 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
Makefile | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Makefile b/Makefile
index 3f5c3d3..0217ff0 100644
--- a/Makefile
+++ b/Makefile
@@ -208,6 +208,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
-include $(BR2_CONFIG)
endif
+export BR2_REPRODUCIBLE
+ifeq ($(BR2_REPRODUCIBLE),y)
+export TZ=UTC
+export LANG=C
+export LC_ALL=C
+endif
+
# To put more focus on warnings, be less verbose as default
# Use 'make V=1' to see the full commands
ifeq ("$(origin V)", "command line")
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:20 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp Gilles Chanteperdrix
` (7 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
environment variable which contains either the date of buildroot last
commit if running from a git repository, or the latest release date.
This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been change, even though some of them
may not have changed in fact.
The meaning of SOURCE_DATE_EPOCH is specified by the following
specification:
https://reproducible-builds.org/specs/source-date-epoch/
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
Makefile | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/Makefile b/Makefile
index 0217ff0..bba7f25 100644
--- a/Makefile
+++ b/Makefile
@@ -213,6 +213,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
export TZ=UTC
export LANG=C
export LC_ALL=C
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
+ sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
+ LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
endif
# To put more focus on warnings, be less verbose as default
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:34 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps Gilles Chanteperdrix
` (6 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
with SOURCE_DATE_EPOCH
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
linux/linux.mk | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/linux/linux.mk b/linux/linux.mk
index a751da9..2a7310c 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -97,6 +97,14 @@ LINUX_MAKE_ENV = \
$(TARGET_MAKE_ENV) \
BR_BINARIES_DIR=$(BINARIES_DIR)
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+ KBUILD_BUILD_VERSION=1 \
+ KBUILD_BUILD_USER=buildroot \
+ KBUILD_BUILD_HOST=buildroot \
+ KBUILD_BUILD_TIMESTAMP="$(shell TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
# Get the real Linux version, which tells us where kernel modules are
# going to be installed in the target filesystem.
LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (2 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:51 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps Gilles Chanteperdrix
` (5 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
package/busybox/busybox.mk | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 08d8e64..a3d526c 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
$(TARGET_MAKE_ENV) \
CFLAGS="$(BUSYBOX_CFLAGS)" \
CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+ KCONFIG_NOTIMESTAMP=1
+endif
+
BUSYBOX_MAKE_OPTS = \
CC="$(TARGET_CC)" \
ARCH=$(KERNEL_ARCH) \
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (3 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:52 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: " Gilles Chanteperdrix
` (4 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
with SOURCE_DATE_EPOCH
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
package/libgcrypt/libgcrypt.mk | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/package/libgcrypt/libgcrypt.mk b/package/libgcrypt/libgcrypt.mk
index 7c27a22..89fe6a0 100644
--- a/package/libgcrypt/libgcrypt.mk
+++ b/package/libgcrypt/libgcrypt.mk
@@ -18,6 +18,10 @@ LIBGCRYPT_CONF_ENV = \
LIBGCRYPT_CONF_OPTS = \
--with-gpg-error-prefix=$(STAGING_DIR)/usr
+ifeq ($(BR2_REPRODUCIBLE),y)
+LIBGCRYPT_MAKE_OPTS = BUILD_TIMESTAMP=$(SOURCE_DATE)
+endif
+
# Code doesn't build in thumb mode
ifeq ($(BR2_arm),y)
LIBGCRYPT_CONF_ENV += CFLAGS="$(patsubst -mthumb,,$(TARGET_CFLAGS))"
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: override timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (4 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 9:54 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio Gilles Chanteperdrix
` (3 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
with SOURCE_DATE_EPOCH
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
package/libgpg-error/libgpg-error.mk | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/package/libgpg-error/libgpg-error.mk b/package/libgpg-error/libgpg-error.mk
index 52b7058..de232eb 100644
--- a/package/libgpg-error/libgpg-error.mk
+++ b/package/libgpg-error/libgpg-error.mk
@@ -14,4 +14,8 @@ LIBGPG_ERROR_CONFIG_SCRIPTS = gpg-error-config
LIBGPG_ERROR_AUTORECONF = YES
LIBGPG_ERROR_GETTEXTIZE = YES
+ifeq ($(BR2_REPRODUCIBLE),y)
+LIBGPG_ERROR_MAKE_OPTS = BUILD_TIMESTAMP=$(SOURCE_DATE)
+endif
+
$(eval $(autotools-package))
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (5 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: " Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 10:16 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio Gilles Chanteperdrix
` (2 subsequent siblings)
9 siblings, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
In order to be sure to have a cpio binary on host which supports
--reproducible.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
package/cpio/cpio.mk | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/package/cpio/cpio.mk b/package/cpio/cpio.mk
index 1576127..87b3df8 100644
--- a/package/cpio/cpio.mk
+++ b/package/cpio/cpio.mk
@@ -18,3 +18,9 @@ CPIO_DEPENDENCIES += argp-standalone
endif
$(eval $(autotools-package))
+
+HOST_CPIO_DEPENDENCIES =
+
+$(eval $(host-autotools-package))
+
+CPIO_REPRODUCIBLE = $(HOST_DIR)/usr/bin/cpio --reproducible
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (6 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio Gilles Chanteperdrix
@ 2016-06-14 15:31 ` Gilles Chanteperdrix
2016-07-02 10:18 ` Yann E. MORIN
2016-07-02 11:11 ` Arnout Vandecappelle
2016-07-02 9:00 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Yann E. MORIN
2016-07-02 9:26 ` Peter Korsgaard
9 siblings, 2 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:31 UTC (permalink / raw)
To: buildroot
Using the --reproducible option to generate a reproducible archive.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
fs/cpio/cpio.mk | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index e82167e..4da6648 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -27,9 +27,17 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
+ifneq ($(BR2_REPRODUCIBLE),y)
define ROOTFS_CPIO_CMD
cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
endef
+else
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
+
+define ROOTFS_CPIO_CMD
+ cd $(TARGET_DIR) && find . | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
+endef
+endif
$(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
$(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date
2016-06-14 15:29 [Buildroot] Reproducible build v2 Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
@ 2016-06-14 15:32 ` Gilles Chanteperdrix
2016-06-14 15:32 ` [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images Gilles Chanteperdrix
2016-06-14 15:32 ` [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible Gilles Chanteperdrix
1 sibling, 2 replies; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:32 UTC (permalink / raw)
To: buildroot
The patch was posted here:
https://lists.gnu.org/archive/html/bug-cpio/2014-08/msg00000.html
And allows generating reproducible iso images with genisoimage.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
...-reproducible-allow-setting-creation-date.patch | 140 +++++++++++++++++++++
1 file changed, 140 insertions(+)
create mode 100644 package/cdrkit/0003-reproducible-allow-setting-creation-date.patch
diff --git a/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch b/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch
new file mode 100644
index 0000000..f5ed465
--- /dev/null
+++ b/package/cdrkit/0003-reproducible-allow-setting-creation-date.patch
@@ -0,0 +1,140 @@
+diff --git a/genisoimage/.eltorito.c.swp b/genisoimage/.eltorito.c.swp
+new file mode 100644
+index 0000000..a49fcbd
+Binary files /dev/null and b/genisoimage/.eltorito.c.swp differ
+diff --git a/genisoimage/genisoimage.1 b/genisoimage/genisoimage.1
+index d05b24a..d69a1d4 100644
+--- a/genisoimage/genisoimage.1
++++ b/genisoimage/genisoimage.1
+@@ -976,6 +976,12 @@ in the
+ .I .genisoimagerc
+ file.
+ .TP
++.BI \-creation-date " epoch"
++Specifies the date to be used as creation, modification and effective
++date in the volume descriptor and for files and relocations created
++on the fly. Specified as a number of second since
++1970-01-01 00:00:00 +0000 (UTC); if 0, the current time is used.
++.TP
+ .B \-print\-size
+ Print estimated filesystem size in multiples of the sector size (2048 bytes)
+ and exit. This option is needed for
+diff --git a/genisoimage/genisoimage.c b/genisoimage/genisoimage.c
+index cfd079a..58397e9 100644
+--- a/genisoimage/genisoimage.c
++++ b/genisoimage/genisoimage.c
+@@ -169,6 +169,7 @@ char *abstract = ABSTRACT_DEFAULT;
+ char *volset_id = VOLSET_ID_DEFAULT;
+ char *volume_id = VOLUME_ID_DEFAULT;
+ char *system_id = SYSTEM_ID_DEFAULT;
++time_t creation_date = 0;
+ char *boot_catalog = BOOT_CATALOG_DEFAULT;
+ char *boot_image = BOOT_IMAGE_DEFAULT;
+ char *genboot_image = BOOT_IMAGE_DEFAULT;
+@@ -405,6 +406,8 @@ struct ld_option {
+ #define OPTION_ALLOW_LEADING_DOTS 1070
+ #define OPTION_PUBLISHER 1071
+
++#define OPTION_CREATION_DATE 1072
++
+ #ifdef JIGDO_TEMPLATE
+ #define OPTION_JTT_OUTPUT 1101
+ #define OPTION_JTJ_OUTPUT 1102
+@@ -522,6 +525,8 @@ static const struct ld_option ld_options[] =
+ '\0', "FILE", "Check all ISO9660 names from previous session", ONE_DASH},
+ {{"copyright", required_argument, NULL, OPTION_COPYRIGHT},
+ '\0', "FILE", "Set Copyright filename", ONE_DASH},
++ {{"creation-date", required_argument, NULL, OPTION_CREATION_DATE},
++ '\0', NULL, "Set volume creation date", ONE_DASH},
+ {{"debug", no_argument, NULL, OPTION_DEBUG},
+ '\0', NULL, "Set debug flag", ONE_DASH},
+ {{"eltorito-boot", required_argument, NULL, 'b'},
+@@ -1721,6 +1726,22 @@ int main(int argc, char *argv[])
+ #endif
+ }
+ break;
++ case OPTION_CREATION_DATE:
++ {
++ char *end = 0;
++
++ creation_date = strtol(optarg, &end, 10);
++ if (!end || *end != 0) {
++#ifdef USE_LIBSCHILY
++ comerrno(EX_BAD, "Bad epoch for -creation-date\n");
++#else
++ fprintf(stderr, "Bad epoch for -creation-date\n");
++ exit(1);
++#endif
++ }
++ break;
++ }
++
+ case OPTION_DEBUG:
+ debug++;
+ break;
+diff --git a/genisoimage/genisoimage.h b/genisoimage/genisoimage.h
+index bbedfb0..c49576c 100644
+--- a/genisoimage/genisoimage.h
++++ b/genisoimage/genisoimage.h
+@@ -650,6 +650,7 @@ extern char *appid;
+ extern char *volset_id;
+ extern char *system_id;
+ extern char *volume_id;
++extern time_t creation_date;
+ extern char *boot_catalog;
+ extern char *boot_image;
+ extern char *genboot_image;
+diff --git a/genisoimage/tree.c b/genisoimage/tree.c
+index 7805888..f17a662 100644
+--- a/genisoimage/tree.c
++++ b/genisoimage/tree.c
+@@ -783,7 +783,11 @@ generate_reloc_directory()
+ struct directory_entry *s_entry;
+
+ /* Create an entry for our internal tree */
+- time(¤t_time);
++ if (creation_date == 0) {
++ time(¤t_time);
++ } else {
++ current_time = creation_date;
++ }
+ reloc_dir = (struct directory *)
+ e_malloc(sizeof (struct directory));
+ memset(reloc_dir, 0, sizeof (struct directory));
+@@ -2680,7 +2684,11 @@ init_fstatbuf()
+ time_t current_time;
+
+ if (fstatbuf.st_ctime == 0) {
+- time(¤t_time);
++ if (creation_date == 0) {
++ time(¤t_time);
++ } else {
++ current_time = creation_date;
++ }
+ if (rationalize_uid)
+ fstatbuf.st_uid = uid_to_use;
+ else
+diff --git a/genisoimage/write.c b/genisoimage/write.c
+index a423ab1..f63507c 100644
+--- a/genisoimage/write.c
++++ b/genisoimage/write.c
+@@ -1885,12 +1885,17 @@ pvd_write(FILE *outfile)
+ int should_write;
+ struct tm local;
+ struct tm gmt;
++ time_t pvd_date;
+
+
+ time(&begun);
+
+- local = *localtime(&begun);
+- gmt = *gmtime(&begun);
++ if (creation_date == 0) {
++ creation_date = begun;
++ }
++
++ local = *localtime(&creation_date);
++ gmt = *gmtime(&creation_date);
+
+ /*
+ * There was a comment here about breaking in the year 2000.
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images
2016-06-14 15:32 ` [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date Gilles Chanteperdrix
@ 2016-06-14 15:32 ` Gilles Chanteperdrix
2016-07-17 19:29 ` Yann E. MORIN
2016-06-14 15:32 ` [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible Gilles Chanteperdrix
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:32 UTC (permalink / raw)
To: buildroot
When reproducibility is required, we:
- run genisoimage with the -creation-date option to avoid using the
build date for files in the iso image automatically created by
genisoimage (such as the catalog)
- disable the rockridge extension, as it stores each file "ctime", which
can not be changed easily on the host filesystem
- force using an initrd, as a trying and booting an iso image without
the rockridge extensions will not work
- use isohybrid -i option to avoid embedding a random number in the
generated iso.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
fs/iso9660/Config.in | 9 +++++++++
fs/iso9660/iso9660.mk | 20 ++++++++++++++++++--
2 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/fs/iso9660/Config.in b/fs/iso9660/Config.in
index 2482e44..bb82d03 100644
--- a/fs/iso9660/Config.in
+++ b/fs/iso9660/Config.in
@@ -71,6 +71,15 @@ config BR2_TARGET_ROOTFS_ISO9660_INITRD
contain a kernel image, an initrd image (unless an initramfs
linked into the kernel is used) and the bootloader.
+config BR2_TARGET_ROOTFS_ISO9660_REPRODUCIBLE
+ def_bool BR2_REPRODUCIBLE
+ select BR2_TARGET_ROOTFS_ISO9660_INITRD
+
+if BR2_REPRODUCIBLE
+comment "A reproducible build requires disabling the RockRidge extensions"
+comment "Which makes using an initrd the only sane solution"
+endif
+
config BR2_TARGET_ROOTFS_ISO9660_HYBRID
bool "Build hybrid image"
depends on BR2_TARGET_ROOTFS_ISO9660_ISOLINUX
diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index 32d79d7..f97a9d7 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -143,15 +143,31 @@ ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_DISABLE_EXTERNAL_INITRD
endif # ROOTFS_ISO9660_USE_INITRD
+ifneq ($(BR2_REPRODUCIBLE),y)
+define ROOTFS_ISO9660_GENISOIMAGE
+ $(HOST_DIR)/usr/bin/genisoimage -R
+endef
+define ROOTFS_ISO9660_ISOHYBRID
+ $(HOST_DIR)/usr/bin/isohybrid
+endef
+else
+define ROOTFS_ISO9660_GENISOIMAGE
+ $(HOST_DIR)/usr/bin/genisoimage -creation-date $(SOURCE_DATE_EPOCH)
+endef
+define ROOTFS_ISO9660_ISOHYBRID
+ $(HOST_DIR)/usr/bin/isohybrid -i $(SOURCE_DATE_EPOCH)
+endef
+endif
+
define ROOTFS_ISO9660_CMD
- $(HOST_DIR)/usr/bin/genisoimage -J -R -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
+ $(ROOTFS_ISO9660_GENISOIMAGE) -J -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
-no-emul-boot -boot-load-size 4 -boot-info-table \
-o $@ $(ROOTFS_ISO9660_TARGET_DIR)
endef
ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
define ROOTFS_ISO9660_GEN_HYBRID
- $(HOST_DIR)/usr/bin/isohybrid -t 0x96 $@
+ $(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
endef
ROOTFS_ISO9660_POST_GEN_HOOKS += ROOTFS_ISO9660_GEN_HYBRID
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible
2016-06-14 15:32 ` [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date Gilles Chanteperdrix
2016-06-14 15:32 ` [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images Gilles Chanteperdrix
@ 2016-06-14 15:32 ` Gilles Chanteperdrix
2016-07-17 19:44 ` Yann E. MORIN
1 sibling, 1 reply; 27+ messages in thread
From: Gilles Chanteperdrix @ 2016-06-14 15:32 UTC (permalink / raw)
To: buildroot
Build with the target toolchain so that the binaries are identical with
different host toolchains.
Sort files lists in order to get deterministic link order.
Build with HEXDATE set to the source date epoch.
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
boot/syslinux/0001-fixed-build-order.patch | 42 ++++++++++++++++++++++++++
boot/syslinux/syslinux.mk | 47 +++++++++++++++++++++++++-----
fs/iso9660/iso9660.mk | 4 +--
3 files changed, 84 insertions(+), 9 deletions(-)
create mode 100644 boot/syslinux/0001-fixed-build-order.patch
diff --git a/boot/syslinux/0001-fixed-build-order.patch b/boot/syslinux/0001-fixed-build-order.patch
new file mode 100644
index 0000000..3697b74
--- /dev/null
+++ b/boot/syslinux/0001-fixed-build-order.patch
@@ -0,0 +1,42 @@
+Sort source file names in order for the link order not to depend on the order in
+which find return file names.
+
+Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
+--
+--- syslinux-6.03/core/Makefile~ 2014-10-06 18:27:44.000000000 +0200
++++ syslinux-6.03/core/Makefile 2016-03-24 14:36:38.207391899 +0100
+@@ -41,9 +41,9 @@
+ # All primary source files for the main syslinux files
+ NASMSRC := $(wildcard $(SRC)/*.asm)
+ NASMHDR := $(wildcard $(SRC)/*.inc)
+-CSRC := $(shell find $(SRC) -name '*.c' -print)
+-SSRC := $(shell find $(SRC) -name '*.S' -print)
+-CHDR := $(shell find $(SRC) -name '*.h' -print)
++CSRC := $(shell find $(SRC) -name '*.c' -print | sort)
++SSRC := $(shell find $(SRC) -name '*.S' -print | sort)
++CHDR := $(shell find $(SRC) -name '*.h' -print | sort)
+ OTHERSRC := keywords
+ ALLSRC = $(NASMSRC) $(NASMHDR) $(CSRC) $(SSRC) $(CHDR) $(OTHERSRC)
+
+@@ -56,18 +56,18 @@
+ $(addprefix $(SRC)/fs/pxe/, dhcp_option.c pxe.c tftp.c urlparse.c bios.c)
+
+ LPXELINUX_CSRC = $(CORE_PXE_CSRC) \
+- $(shell find $(SRC)/lwip -name '*.c' -print) \
++ $(shell find $(SRC)/lwip -name '*.c' -print | sort) \
+ $(addprefix $(SRC)/fs/pxe/, \
+ core.c dnsresolv.c ftp.c ftp_readdir.c gpxeurl.c http.c \
+ http_readdir.c idle.c isr.c tcp.c)
+
+ PXELINUX_CSRC = $(CORE_PXE_CSRC) \
+- $(shell find $(SRC)/legacynet -name '*.c' -print)
++ $(shell find $(SRC)/legacynet -name '*.c' -print | sort)
+
+ LPXELINUX_OBJS = $(subst $(SRC)/,,$(LPXELINUX_CSRC:%.c=%.o))
+ PXELINUX_OBJS = $(subst $(SRC)/,,$(PXELINUX_CSRC:%.c=%.o))
+
+-UNITTEST_CSRC = $(shell find $(SRC) -path '*/tests/*.c' -print)
++UNITTEST_CSRC = $(shell find $(SRC) -path '*/tests/*.c' -print | sort)
+ UNITTEST_OBJS = $(subst $(SRC)/,,$(UNITTEST_CSRC:%.c=%.o))
+
+ # Don't include console and network stack specific objects or unit tests
diff --git a/boot/syslinux/syslinux.mk b/boot/syslinux/syslinux.mk
index 82890c5..cdd5b3c 100644
--- a/boot/syslinux/syslinux.mk
+++ b/boot/syslinux/syslinux.mk
@@ -13,7 +13,7 @@ SYSLINUX_LICENSE_FILES = COPYING
SYSLINUX_INSTALL_IMAGES = YES
-SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx
+SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx host-perl host-python host-xz
ifeq ($(BR2_TARGET_SYSLINUX_LEGACY_BIOS),y)
SYSLINUX_TARGET += bios
@@ -47,12 +47,35 @@ define SYSLINUX_CLEANUP
endef
SYSLINUX_POST_PATCH_HOOKS += SYSLINUX_CLEANUP
+ifeq ($(BR2_REPRODUCIBLE),y)
+define SYSLINUX_REPRODUCIBLE
+ HEXDATE="`printf "0x%x" $(SOURCE_DATE_EPOCH)`"
+endef
+endif
+
+define SYSLINUX_MAKE
+ $(TARGET_MAKE_ENV) $(MAKE1) \
+ $(SYSLINUX_REPRODUCIBLE) \
+ NASM=$(HOST_DIR)/usr/bin/nasm \
+ PERL=$(HOST_DIR)/usr/bin/perl \
+ PYTHON=$(HOST_DIR)/usr/bin/python \
+ UPX=$(HOST_DIR)/usr/bin/upx \
+ CC="$(TARGET_CC)" \
+ LD="$(TARGET_LD) -m elf_i386" \
+ OBJDUMP="$(TARGET_OBJDUMP)" \
+ OBJCOPY="$(TARGET_OBJCOPY)" \
+ STRIP="$(TARGET_STRIP)" \
+ AR="$(TARGET_AR)" \
+ NM="$(TARGET_NM)" \
+ RANLIB="$(TARGET_RANLIB)" \
+ XZ=$(HOST_DIR)/usr/bin/xz $(SYSLINUX_EFI_ARGS)
+endef
+
# syslinux build system has no convenient way to pass CFLAGS,
# and the internal zlib should take precedence so -I shouldn't
# be used.
define SYSLINUX_BUILD_CMDS
- $(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
- AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) -C $(@D) $(SYSLINUX_TARGET)
+ $(SYSLINUX_MAKE) -C $(@D) $(SYSLINUX_TARGET)
endef
# While the actual bootloader is compiled for the target, several
@@ -61,8 +84,7 @@ endef
# Repeat CC and AR, since syslinux really wants to check them at
# install time
define SYSLINUX_INSTALL_TARGET_CMDS
- $(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
- AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) INSTALLROOT=$(HOST_DIR) \
+ $(SYSLINUX_MAKE) INSTALLROOT=$(@D)/inst \
-C $(@D) $(SYSLINUX_TARGET) install
endef
@@ -80,10 +102,21 @@ define SYSLINUX_INSTALL_IMAGES_CMDS
for i in $(SYSLINUX_IMAGES-y); do \
$(INSTALL) -D -m 0755 $(@D)/$$i $(BINARIES_DIR)/syslinux/$${i##*/}; \
done
- for i in $(SYSLINUX_C32); do \
- $(INSTALL) -D -m 0755 $(HOST_DIR)/usr/share/syslinux/$${i} \
+ for i in $(SYSLINUX_C32) ldlinux.c32; do \
+ $(INSTALL) -D -m 0755 $(@D)/inst/usr/share/syslinux/$${i} \
$(BINARIES_DIR)/syslinux/$${i}; \
done
endef
+define HOST_SYSLINUX_BUILD_CMDS
+ $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
+- AR="$(HOSTAR)" -C $(@D) bios
+endef
+
+define HOST_SYSLINUX_INSTALL_CMDS
+ $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
+- AR="$(HOSTAR)" -C $(@D) INSTALLROOT=$(HOST_DIR) bios install
+endef
+
$(eval $(generic-package))
+$(eval $(host-generic-package))
diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index f97a9d7..db22ca4 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -70,8 +70,6 @@ ROOTFS_ISO9660_BOOT_IMAGE = isolinux/isolinux.bin
define ROOTFS_ISO9660_INSTALL_BOOTLOADER
$(INSTALL) -D -m 0644 $(BINARIES_DIR)/syslinux/* \
$(ROOTFS_ISO9660_TARGET_DIR)/isolinux/
- $(INSTALL) -D -m 0644 $(HOST_DIR)/usr/share/syslinux/ldlinux.c32 \
- $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/ldlinux.c32
endef
endif
@@ -166,6 +164,8 @@ define ROOTFS_ISO9660_CMD
endef
ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
+ROOTFS_ISO9660_DEPENDENCIES += host-syslinux
+
define ROOTFS_ISO9660_GEN_HYBRID
$(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
endef
--
2.8.2
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 01/12] reproducibility: introduce config knob
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (7 preceding siblings ...)
2016-06-14 15:31 ` [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio Gilles Chanteperdrix
@ 2016-07-02 9:00 ` Yann E. MORIN
2016-07-02 9:26 ` Peter Korsgaard
9 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:00 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> Making the builds reproducible involves turning off build timestamps,
> which some users may find annoying. So make the reproducible builds
> optional.
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Quite obviously:
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Note: it may take some time for my reviews on the rest of your series to
come in...
Regards,
Yann E. MORIN.
> ---
> Config.in | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/Config.in b/Config.in
> index 9fa7415..faf4c71 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -724,6 +724,17 @@ config BR2_COMPILER_PARANOID_UNSAFE_PATH
>
> endmenu
>
> +config BR2_REPRODUCIBLE
> + bool "Make the build reproducible (experimental)"
> + help
> + This option will remove all sources of non-reproducibility
> + from the build process. For a given Buildroot configuration,
> + this allows to generate exactly identical binaries from one
> + build to the other, including on different machines.
> +
> + This is labeled as an experimental feature, as not all
> + packages behave properly to ensure reproducibility.
> +
> endmenu
>
> source "toolchain/Config.in"
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
@ 2016-07-02 9:01 ` Yann E. MORIN
2016-07-02 9:58 ` Peter Korsgaard
1 sibling, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:01 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> ---
> Makefile | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 3f5c3d3..0217ff0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -208,6 +208,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
> -include $(BR2_CONFIG)
> endif
>
> +export BR2_REPRODUCIBLE
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +export TZ=UTC
> +export LANG=C
> +export LC_ALL=C
> +endif
> +
> # To put more focus on warnings, be less verbose as default
> # Use 'make V=1' to see the full commands
> ifeq ("$(origin V)", "command line")
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH
2016-06-14 15:31 ` [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH Gilles Chanteperdrix
@ 2016-07-02 9:20 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:20 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
> environment variable which contains either the date of buildroot last
> commit if running from a git repository, or the latest release date.
>
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been change, even though some of them
> may not have changed in fact.
>
> The meaning of SOURCE_DATE_EPOCH is specified by the following
> specification:
> https://reproducible-builds.org/specs/source-date-epoch/
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> Makefile | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 0217ff0..bba7f25 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -213,6 +213,11 @@ ifeq ($(BR2_REPRODUCIBLE),y)
> export TZ=UTC
> export LANG=C
> export LC_ALL=C
> +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
> +SOURCE_DATE_CHANGES = $(shell head -n 1 $(TOPDIR)/CHANGES | \
> + sed 's/^.*Released \(.*\)$$/\1/;s/\(st\|nd\|rd\|th\),//' | \
> + LANG=C LC_ALL=C TZ=UTC xargs -i date -d \{\} +%s)
Why do you need to set TZ, LANG and LC_ALL here, since they are exported
just the lines above?
However, we've just discussed that during the summer camp, and we've
come to the conlusion that parsing the CHANGES file is not the best
solution.
After all, what we need is a reproducible value. The one extracted from
CHANGES is not absolutely right (e.g. in case one grabs a tarball from a
random cset), but is close-enough to the real value that it is OK.
Yet, we think that we could just use the last second of the last minute
of the last hour of the first day of the mont of the last release. (wee!)
So we suggest we use the release version, which is already YYYY-MM (e.g.
in my local tree):
$ make print-version
2016.08-git-00009-g5173895
And it is then trivial to get it to look more like YYYY-MM-01T23:59:59
and the rest is easy to go by... ;-)
What's your opinion on this?
Regards,
Yann E. MORIN.
> +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
> endif
>
> # To put more focus on warnings, be less verbose as default
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 01/12] reproducibility: introduce config knob
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
` (8 preceding siblings ...)
2016-07-02 9:00 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Yann E. MORIN
@ 2016-07-02 9:26 ` Peter Korsgaard
9 siblings, 0 replies; 27+ messages in thread
From: Peter Korsgaard @ 2016-07-02 9:26 UTC (permalink / raw)
To: buildroot
>>>>> "Gilles" == Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> Making the builds reproducible involves turning off build timestamps,
> which some users may find annoying. So make the reproducible builds
> optional.
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Committed, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp
2016-06-14 15:31 ` [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp Gilles Chanteperdrix
@ 2016-07-02 9:34 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:34 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> with SOURCE_DATE_EPOCH
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> linux/linux.mk | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index a751da9..2a7310c 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -97,6 +97,14 @@ LINUX_MAKE_ENV = \
> $(TARGET_MAKE_ENV) \
> BR_BINARIES_DIR=$(BINARIES_DIR)
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +LINUX_MAKE_ENV += \
> + KBUILD_BUILD_VERSION=1 \
The kernel sets it to 0 (zero) as a fallback when there is no .version
file. Should we also set it to 0, too? Note: I'm fine with 1 as well,
so:
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> + KBUILD_BUILD_USER=buildroot \
> + KBUILD_BUILD_HOST=buildroot \
> + KBUILD_BUILD_TIMESTAMP="$(shell TZ=UTC LANG=C LC_ALL=C date -d @$(SOURCE_DATE_EPOCH))"
> +endif
> +
> # Get the real Linux version, which tells us where kernel modules are
> # going to be installed in the target filesystem.
> LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps Gilles Chanteperdrix
@ 2016-07-02 9:51 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:51 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> package/busybox/busybox.mk | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
> index 08d8e64..a3d526c 100644
> --- a/package/busybox/busybox.mk
> +++ b/package/busybox/busybox.mk
> @@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
> $(TARGET_MAKE_ENV) \
> CFLAGS="$(BUSYBOX_CFLAGS)" \
> CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
> +
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +BUSYBOX_MAKE_ENV += \
> + KCONFIG_NOTIMESTAMP=1
This could have been a single line (since it is not too long). Otherwise;
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> +endif
> +
> BUSYBOX_MAKE_OPTS = \
> CC="$(TARGET_CC)" \
> ARCH=$(KERNEL_ARCH) \
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps Gilles Chanteperdrix
@ 2016-07-02 9:52 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:52 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> with SOURCE_DATE_EPOCH
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> package/libgcrypt/libgcrypt.mk | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/package/libgcrypt/libgcrypt.mk b/package/libgcrypt/libgcrypt.mk
> index 7c27a22..89fe6a0 100644
> --- a/package/libgcrypt/libgcrypt.mk
> +++ b/package/libgcrypt/libgcrypt.mk
> @@ -18,6 +18,10 @@ LIBGCRYPT_CONF_ENV = \
> LIBGCRYPT_CONF_OPTS = \
> --with-gpg-error-prefix=$(STAGING_DIR)/usr
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +LIBGCRYPT_MAKE_OPTS = BUILD_TIMESTAMP=$(SOURCE_DATE)
^^^^^^^^^^^
You probably meant SOURCE_DATE_EPOCH here, right?
Regards,
Yann E. MORIN.
> +endif
> +
> # Code doesn't build in thumb mode
> ifeq ($(BR2_arm),y)
> LIBGCRYPT_CONF_ENV += CFLAGS="$(patsubst -mthumb,,$(TARGET_CFLAGS))"
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: override timestamps
2016-06-14 15:31 ` [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: " Gilles Chanteperdrix
@ 2016-07-02 9:54 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 9:54 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> with SOURCE_DATE_EPOCH
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> package/libgpg-error/libgpg-error.mk | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/package/libgpg-error/libgpg-error.mk b/package/libgpg-error/libgpg-error.mk
> index 52b7058..de232eb 100644
> --- a/package/libgpg-error/libgpg-error.mk
> +++ b/package/libgpg-error/libgpg-error.mk
> @@ -14,4 +14,8 @@ LIBGPG_ERROR_CONFIG_SCRIPTS = gpg-error-config
> LIBGPG_ERROR_AUTORECONF = YES
> LIBGPG_ERROR_GETTEXTIZE = YES
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +LIBGPG_ERROR_MAKE_OPTS = BUILD_TIMESTAMP=$(SOURCE_DATE)
^^^^^^^^^^^
You probably meant SOURCE_DATE_EPOCH, right?
Regards,
Yann E. MORIN.
> +endif
> +
> $(eval $(autotools-package))
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
2016-07-02 9:01 ` Yann E. MORIN
@ 2016-07-02 9:58 ` Peter Korsgaard
1 sibling, 0 replies; 27+ messages in thread
From: Peter Korsgaard @ 2016-07-02 9:58 UTC (permalink / raw)
To: buildroot
>>>>> "Gilles" == Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> Makefile | 7 +++++++
> 1 file changed, 7 insertions(+)
> diff --git a/Makefile b/Makefile
> index 3f5c3d3..0217ff0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -208,6 +208,13 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
> -include $(BR2_CONFIG)
> endif
> +export BR2_REPRODUCIBLE
I moved this down with the other exports we do for post-build /
post-image scripts.
> +ifeq ($(BR2_REPRODUCIBLE),y)
And added a comment why we do this.
Committed with that fixed, thanks.
> +export TZ=UTC
> +export LANG=C
> +export LC_ALL=C
> +endif
> +
> # To put more focus on warnings, be less verbose as default
> # Use 'make V=1' to see the full commands
> ifeq ("$(origin V)", "command line")
> --
> 2.8.2
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio
2016-06-14 15:31 ` [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio Gilles Chanteperdrix
@ 2016-07-02 10:16 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 10:16 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> In order to be sure to have a cpio binary on host which supports
> --reproducible.
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> package/cpio/cpio.mk | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/package/cpio/cpio.mk b/package/cpio/cpio.mk
> index 1576127..87b3df8 100644
> --- a/package/cpio/cpio.mk
> +++ b/package/cpio/cpio.mk
> @@ -18,3 +18,9 @@ CPIO_DEPENDENCIES += argp-standalone
> endif
>
> $(eval $(autotools-package))
> +
> +HOST_CPIO_DEPENDENCIES =
> +
> +$(eval $(host-autotools-package))
We usually write all package variables first, and eventually call the
package macros, like:
[...existing stuff...]
HOST_CPIO_DEPENDENCIES =
$(eval $(autotools-package))
$(eval $(host-autotools-package))
> +CPIO_REPRODUCIBLE = $(HOST_DIR)/usr/bin/cpio --reproducible
Since we now have host-cpio, we can alsso add:
CPIO = $(HOST_DIR)/usr/bin/cpio
Regards,
Yann E. MORIN.
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio
2016-06-14 15:31 ` [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio Gilles Chanteperdrix
@ 2016-07-02 10:18 ` Yann E. MORIN
2016-07-02 11:11 ` Arnout Vandecappelle
1 sibling, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-02 10:18 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:31 +0200, Gilles Chanteperdrix spake thusly:
> Using the --reproducible option to generate a reproducible archive.
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> fs/cpio/cpio.mk | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
> index e82167e..4da6648 100644
> --- a/fs/cpio/cpio.mk
> +++ b/fs/cpio/cpio.mk
> @@ -27,9 +27,17 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
>
> ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
>
> +ifneq ($(BR2_REPRODUCIBLE),y)
> define ROOTFS_CPIO_CMD
> cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
> endef
> +else
> +ROOTFS_CPIO_DEPENDENCIES += host-cpio
> +
> +define ROOTFS_CPIO_CMD
> + cd $(TARGET_DIR) && find . | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
> +endef
> +endif
Rather than duplicate the commands, why not something like:
ifeq ($(BR2_REPDUCIBLE),y)
ROOTFS_CPIO_CPIO = $(CPIO)
else
ROOTFS_CPIO_CPIO = $(CPIO_REPRODUCIBLE)
endif
define ROOTFS_CPIO_CMD
cd $(TARGET_DIR) && find . | $(ROOTFS_CPIO_CPIO) --quiet -o -H newc > $@
endef
Regards,
Yann E. MORIN.
> $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
> $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio
2016-06-14 15:31 ` [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio Gilles Chanteperdrix
2016-07-02 10:18 ` Yann E. MORIN
@ 2016-07-02 11:11 ` Arnout Vandecappelle
1 sibling, 0 replies; 27+ messages in thread
From: Arnout Vandecappelle @ 2016-07-02 11:11 UTC (permalink / raw)
To: buildroot
On 14-06-16 17:31, Gilles Chanteperdrix wrote:
> Using the --reproducible option to generate a reproducible archive.
>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> fs/cpio/cpio.mk | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
> index e82167e..4da6648 100644
> --- a/fs/cpio/cpio.mk
> +++ b/fs/cpio/cpio.mk
> @@ -27,9 +27,17 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
>
> ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
>
> +ifneq ($(BR2_REPRODUCIBLE),y)
> define ROOTFS_CPIO_CMD
> cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
> endef
> +else
> +ROOTFS_CPIO_DEPENDENCIES += host-cpio
Instead of unconditionally building host-cpio, it would be better to check if
the installed cpio supports --reproducible. Cfr. what we do for tar, in
support/dependencies/check-host-tar.{sh,mk}.
Regards,
Arnout
> +
> +define ROOTFS_CPIO_CMD
> + cd $(TARGET_DIR) && find . | $(CPIO_REPRODUCIBLE) --quiet -o -H newc > $@
> +endef
> +endif
>
> $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
> $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images
2016-06-14 15:32 ` [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images Gilles Chanteperdrix
@ 2016-07-17 19:29 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-17 19:29 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:32 +0200, Gilles Chanteperdrix spake thusly:
> When reproducibility is required, we:
> - run genisoimage with the -creation-date option to avoid using the
> build date for files in the iso image automatically created by
> genisoimage (such as the catalog)
> - disable the rockridge extension, as it stores each file "ctime", which
> can not be changed easily on the host filesystem
> - force using an initrd, as a trying and booting an iso image without
> the rockridge extensions will not work
> - use isohybrid -i option to avoid embedding a random number in the
> generated iso.
I'm not too fond of having a huge patch in Buildroot, that has no chance
of ever getting upstream (since upstream is virtually dead as there's
been no activity since Oct 2010, and their homepage is now a blank
page).
Instead, when programs can't be told what date to use, I'd prefer we
have a look at libfaketime:
https://github.com/wolfcw/libfaketime
Then one can use the 'faketime' wrapper to force a specific date:
faketime '2016-07-17 21:28:42' genisoimage blablalba
In which case, it might be worth doing it directly in the fs infra so
that all filesystem images are handled once and for all.
Thoughts?
Regards,
Yann E. MORIN.
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ---
> fs/iso9660/Config.in | 9 +++++++++
> fs/iso9660/iso9660.mk | 20 ++++++++++++++++++--
> 2 files changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/fs/iso9660/Config.in b/fs/iso9660/Config.in
> index 2482e44..bb82d03 100644
> --- a/fs/iso9660/Config.in
> +++ b/fs/iso9660/Config.in
> @@ -71,6 +71,15 @@ config BR2_TARGET_ROOTFS_ISO9660_INITRD
> contain a kernel image, an initrd image (unless an initramfs
> linked into the kernel is used) and the bootloader.
>
> +config BR2_TARGET_ROOTFS_ISO9660_REPRODUCIBLE
> + def_bool BR2_REPRODUCIBLE
> + select BR2_TARGET_ROOTFS_ISO9660_INITRD
> +
> +if BR2_REPRODUCIBLE
> +comment "A reproducible build requires disabling the RockRidge extensions"
> +comment "Which makes using an initrd the only sane solution"
> +endif
> +
> config BR2_TARGET_ROOTFS_ISO9660_HYBRID
> bool "Build hybrid image"
> depends on BR2_TARGET_ROOTFS_ISO9660_ISOLINUX
> diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
> index 32d79d7..f97a9d7 100644
> --- a/fs/iso9660/iso9660.mk
> +++ b/fs/iso9660/iso9660.mk
> @@ -143,15 +143,31 @@ ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_DISABLE_EXTERNAL_INITRD
> endif # ROOTFS_ISO9660_USE_INITRD
>
>
> +ifneq ($(BR2_REPRODUCIBLE),y)
> +define ROOTFS_ISO9660_GENISOIMAGE
> + $(HOST_DIR)/usr/bin/genisoimage -R
> +endef
> +define ROOTFS_ISO9660_ISOHYBRID
> + $(HOST_DIR)/usr/bin/isohybrid
> +endef
> +else
> +define ROOTFS_ISO9660_GENISOIMAGE
> + $(HOST_DIR)/usr/bin/genisoimage -creation-date $(SOURCE_DATE_EPOCH)
> +endef
> +define ROOTFS_ISO9660_ISOHYBRID
> + $(HOST_DIR)/usr/bin/isohybrid -i $(SOURCE_DATE_EPOCH)
> +endef
> +endif
> +
> define ROOTFS_ISO9660_CMD
> - $(HOST_DIR)/usr/bin/genisoimage -J -R -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
> + $(ROOTFS_ISO9660_GENISOIMAGE) -J -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
> -no-emul-boot -boot-load-size 4 -boot-info-table \
> -o $@ $(ROOTFS_ISO9660_TARGET_DIR)
> endef
>
> ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
> define ROOTFS_ISO9660_GEN_HYBRID
> - $(HOST_DIR)/usr/bin/isohybrid -t 0x96 $@
> + $(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
> endef
>
> ROOTFS_ISO9660_POST_GEN_HOOKS += ROOTFS_ISO9660_GEN_HYBRID
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible
2016-06-14 15:32 ` [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible Gilles Chanteperdrix
@ 2016-07-17 19:44 ` Yann E. MORIN
0 siblings, 0 replies; 27+ messages in thread
From: Yann E. MORIN @ 2016-07-17 19:44 UTC (permalink / raw)
To: buildroot
Gilles, All,
On 2016-06-14 17:32 +0200, Gilles Chanteperdrix spake thusly:
> Build with the target toolchain so that the binaries are identical with
> different host toolchains.
> Sort files lists in order to get deterministic link order.
> Build with HEXDATE set to the source date epoch.
It looks like those are three different changes, so should have been
three different patches.
Especially the change to use the cross-toolchain should really be
separate (and come first).
Further comments below...
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
[--SNIP--]
> diff --git a/boot/syslinux/0001-fixed-build-order.patch b/boot/syslinux/0001-fixed-build-order.patch
> new file mode 100644
> index 0000000..3697b74
> --- /dev/null
> +++ b/boot/syslinux/0001-fixed-build-order.patch
> @@ -0,0 +1,42 @@
> +Sort source file names in order for the link order not to depend on the order in
> +which find return file names.
> +
> +Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Have you tried submitting this patch upstream?
We do not much like having feature patches in Buildroot, because they
are a pain to maintain when we want to update the package.
Otherwise, this looks pretty simple, I guess upstream will probably like
it. ;-)
[--SNIP--]
> diff --git a/boot/syslinux/syslinux.mk b/boot/syslinux/syslinux.mk
> index 82890c5..cdd5b3c 100644
> --- a/boot/syslinux/syslinux.mk
> +++ b/boot/syslinux/syslinux.mk
> @@ -13,7 +13,7 @@ SYSLINUX_LICENSE_FILES = COPYING
>
> SYSLINUX_INSTALL_IMAGES = YES
>
> -SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx
> +SYSLINUX_DEPENDENCIES = host-nasm host-util-linux host-upx host-perl host-python host-xz
Why are those new host packages needed?
> ifeq ($(BR2_TARGET_SYSLINUX_LEGACY_BIOS),y)
> SYSLINUX_TARGET += bios
> @@ -47,12 +47,35 @@ define SYSLINUX_CLEANUP
> endef
> SYSLINUX_POST_PATCH_HOOKS += SYSLINUX_CLEANUP
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define SYSLINUX_REPRODUCIBLE
> + HEXDATE="`printf "0x%x" $(SOURCE_DATE_EPOCH)`"
> +endef
> +endif
> +
> +define SYSLINUX_MAKE
> + $(TARGET_MAKE_ENV) $(MAKE1) \
> + $(SYSLINUX_REPRODUCIBLE) \
> + NASM=$(HOST_DIR)/usr/bin/nasm \
> + PERL=$(HOST_DIR)/usr/bin/perl \
> + PYTHON=$(HOST_DIR)/usr/bin/python \
Why do we need to specify nasm, perl and python? The PATH as set by
Buildroot already has the host dirs early in the PATH, so they should
be found before the system ones.
> + UPX=$(HOST_DIR)/usr/bin/upx \
> + CC="$(TARGET_CC)" \
> + LD="$(TARGET_LD) -m elf_i386" \
> + OBJDUMP="$(TARGET_OBJDUMP)" \
> + OBJCOPY="$(TARGET_OBJCOPY)" \
> + STRIP="$(TARGET_STRIP)" \
> + AR="$(TARGET_AR)" \
> + NM="$(TARGET_NM)" \
> + RANLIB="$(TARGET_RANLIB)" \
> + XZ=$(HOST_DIR)/usr/bin/xz $(SYSLINUX_EFI_ARGS)
Ditto xz.
You're also adding more variables than were present in the existing
commands; that's why using the cross-toolchain should be a separate
patch: so that we can more easily understand the changes.
> +endef
> +
> # syslinux build system has no convenient way to pass CFLAGS,
> # and the internal zlib should take precedence so -I shouldn't
> # be used.
> define SYSLINUX_BUILD_CMDS
> - $(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
> - AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) -C $(@D) $(SYSLINUX_TARGET)
> + $(SYSLINUX_MAKE) -C $(@D) $(SYSLINUX_TARGET)
> endef
>
> # While the actual bootloader is compiled for the target, several
> @@ -61,8 +84,7 @@ endef
> # Repeat CC and AR, since syslinux really wants to check them at
> # install time
> define SYSLINUX_INSTALL_TARGET_CMDS
> - $(TARGET_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
> - AR="$(HOSTAR)" $(SYSLINUX_EFI_ARGS) INSTALLROOT=$(HOST_DIR) \
> + $(SYSLINUX_MAKE) INSTALLROOT=$(@D)/inst \
> -C $(@D) $(SYSLINUX_TARGET) install
> endef
>
> @@ -80,10 +102,21 @@ define SYSLINUX_INSTALL_IMAGES_CMDS
> for i in $(SYSLINUX_IMAGES-y); do \
> $(INSTALL) -D -m 0755 $(@D)/$$i $(BINARIES_DIR)/syslinux/$${i##*/}; \
> done
> - for i in $(SYSLINUX_C32); do \
> - $(INSTALL) -D -m 0755 $(HOST_DIR)/usr/share/syslinux/$${i} \
> + for i in $(SYSLINUX_C32) ldlinux.c32; do \
> + $(INSTALL) -D -m 0755 $(@D)/inst/usr/share/syslinux/$${i} \
> $(BINARIES_DIR)/syslinux/$${i}; \
> done
> endef
>
> +define HOST_SYSLINUX_BUILD_CMDS
> + $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
> +- AR="$(HOSTAR)" -C $(@D) bios
^
Leading dash here?...
Also, use TABs for indentation of the *_CMDS defines.
But then, you are building the 'bios' stuff with the host compiler.
Doesn't that defeats the very purpose of that patch, and contradicts the
commit log itself (which states that we are now using the target
toolchain) ?
> +endef
> +
> +define HOST_SYSLINUX_INSTALL_CMDS
> + $(HOST_MAKE_ENV) $(MAKE1) CC="$(HOSTCC) -idirafter $(HOST_DIR)/usr/include $(HOST_LDFLAGS)" \
> +- AR="$(HOSTAR)" -C $(@D) INSTALLROOT=$(HOST_DIR) bios install
> +endef
> +
> $(eval $(generic-package))
> +$(eval $(host-generic-package))
Since you submitted this patch, we've changed the way how dependencies
of host packages are handled: they are no longer automatically inherited
from the dependencies of the target variant; you now have to explicitly
define the dependencies of the host variant.
Regards,
Yann E. MORIN.
> diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
> index f97a9d7..db22ca4 100644
> --- a/fs/iso9660/iso9660.mk
> +++ b/fs/iso9660/iso9660.mk
> @@ -70,8 +70,6 @@ ROOTFS_ISO9660_BOOT_IMAGE = isolinux/isolinux.bin
> define ROOTFS_ISO9660_INSTALL_BOOTLOADER
> $(INSTALL) -D -m 0644 $(BINARIES_DIR)/syslinux/* \
> $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/
> - $(INSTALL) -D -m 0644 $(HOST_DIR)/usr/share/syslinux/ldlinux.c32 \
> - $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/ldlinux.c32
> endef
> endif
>
> @@ -166,6 +164,8 @@ define ROOTFS_ISO9660_CMD
> endef
>
> ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
> +ROOTFS_ISO9660_DEPENDENCIES += host-syslinux
> +
> define ROOTFS_ISO9660_GEN_HYBRID
> $(ROOTFS_ISO9660_ISOHYBRID) -t 0x96 $@
> endef
> --
> 2.8.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2016-07-17 19:44 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-14 15:29 [Buildroot] Reproducible build v2 Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Gilles Chanteperdrix
2016-06-14 15:31 ` [Buildroot] [PATCH 02/12] reproducibility: override locale and timezone Gilles Chanteperdrix
2016-07-02 9:01 ` Yann E. MORIN
2016-07-02 9:58 ` Peter Korsgaard
2016-06-14 15:31 ` [Buildroot] [PATCH 03/12] reproducibility: generate SOURCE_DATE_EPOCH Gilles Chanteperdrix
2016-07-02 9:20 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 04/12] reproducibility/linux: override build timestamp Gilles Chanteperdrix
2016-07-02 9:34 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 05/12] reproducibility/busybox: disable build timestamps Gilles Chanteperdrix
2016-07-02 9:51 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 06/12] reproducibility/libgcrypt: override timestamps Gilles Chanteperdrix
2016-07-02 9:52 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 07/12] reproducibility/libgpg-error: " Gilles Chanteperdrix
2016-07-02 9:54 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 08/12] package/cpio: allow generating host-cpio Gilles Chanteperdrix
2016-07-02 10:16 ` Yann E. MORIN
2016-06-14 15:31 ` [Buildroot] [PATCH 09/12] reproducibility/fs/cpio: generate archive with host-cpio Gilles Chanteperdrix
2016-07-02 10:18 ` Yann E. MORIN
2016-07-02 11:11 ` Arnout Vandecappelle
2016-07-02 9:00 ` [Buildroot] [PATCH 01/12] reproducibility: introduce config knob Yann E. MORIN
2016-07-02 9:26 ` Peter Korsgaard
2016-06-14 15:32 ` [Buildroot] [PATCH 10/12] cdrkit: add patch allowing to set creation date Gilles Chanteperdrix
2016-06-14 15:32 ` [Buildroot] [PATCH 11/12] reproducibility/fs/iso9660: reproducible iso images Gilles Chanteperdrix
2016-07-17 19:29 ` Yann E. MORIN
2016-06-14 15:32 ` [Buildroot] [PATCH 12/12] reproducible/syslinux: make syslinux build reproducible Gilles Chanteperdrix
2016-07-17 19:44 ` Yann E. MORIN
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.