[PATCH 1/3 v2] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds for deploy

[PATCH 1/3 v2] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds for deploy

[Richard Purdie]

Using DATETIME means the do_deploy output is not deterministic. Use SDE
when using reproducible builds since it is consistent for given input data.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/image-artifact-names.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image-artifact-names.bbclass b/meta/classes/image-artifact-names.bbclass
index 3ac8dd731a6..67f04e81658 100644
--- a/meta/classes/image-artifact-names.bbclass
+++ b/meta/classes/image-artifact-names.bbclass
@@ -4,7 +4,7 @@
 
 IMAGE_BASENAME ?= "${PN}"
 IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
-IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
+IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
 IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
 IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
 
@@ -13,3 +13,10 @@ IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
 # by default) followed by additional suffices which describe the format (.ext4,
 # .ext4.xz, etc.).
 IMAGE_NAME_SUFFIX ??= ".rootfs"
+
+python () {
+    if bb.data.inherits_class('reproducible_build', d) and bb.data.inherits_class('deploy', d) and d.getVar("IMAGE_VERSION_SUFFIX") == "-${DATETIME}":
+        import datetime
+        d.setVar("IMAGE_VERSION_SUFFIX", "-" + datetime.datetime.fromtimestamp(int(d.getVar("SOURCE_DATE_EPOCH")), datetime.timezone.utc).strftime('%Y%m%d%H%M%S'))
+        d.setVarFlag("IMAGE_VERSION_SUFFIX", "vardepvalue", "")
+}
-- 
2.32.0

[PATCH 2/3 v2] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION

[Richard Purdie]

At this point the hash equivalence and sstate is 'junk' on the autobuilder
unforuntately due to the volume of fixes and also the volume of slighly
not quite right patches tested during the development of the fixes.

In order to try and help any remaining sanity I might have, bump the
version numbers to start with a clean slate so we're working from a known
good baseline rather than risk chasing phantom issues. For those
upgrading, there wouldn't be much reuse anyway after the changes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass                          | 2 +-
 meta/conf/abi_version.conf                           | 2 +-
 meta/recipes-devtools/elfutils/elfutils_0.185.bb     | 1 -
 meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb | 1 -
 4 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 89e9f561787..7f4b1f6804c 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1,4 +1,4 @@
-SSTATE_VERSION = "3"
+SSTATE_VERSION = "5"
 
 SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
 SSTATE_MANFILEPREFIX = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-${PN}"
diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
index 35faef9a368..e84cad1019a 100644
--- a/meta/conf/abi_version.conf
+++ b/meta/conf/abi_version.conf
@@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
 # a reset of the equivalence, for example when reproducibility issues break the
 # existing match data. Distros can also append to this value for the same effect.
 #
-HASHEQUIV_HASH_VERSION  = "5"
+HASHEQUIV_HASH_VERSION  = "8"
diff --git a/meta/recipes-devtools/elfutils/elfutils_0.185.bb b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
index 9ea4de8e407..f4769e36328 100644
--- a/meta/recipes-devtools/elfutils/elfutils_0.185.bb
+++ b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
@@ -34,7 +34,6 @@ SRC_URI[sha256sum] = "dc8d3e74ab209465e7f568e1b3bb9a5a142f8656e2b57d10049a73da2a
 
 # remove at next version upgrade or when output changes
 PR = "r1"
-HASHEQUIV_HASH_VERSION .= ".2"
 
 inherit autotools gettext ptest pkgconfig
 
diff --git a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
index 0dd18d7a068..62aa1b0e877 100644
--- a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
+++ b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
@@ -28,7 +28,6 @@ SRC_URI[sha256sum] = "4441a5d593f85bb6e8d578cf6653fb4ec30f9e8f4a2315a3d8f2d0a8b3
 
 # remove at next version upgrade or when output changes
 PR = "r1"
-HASHEQUIV_HASH_VERSION .= ".1"
 
 RECIPE_NO_UPDATE_REASON = "6.04-pre3 is broken"
 UPSTREAM_CHECK_URI = "https://www.zytor.com/pub/syslinux/"
-- 
2.32.0

[PATCH 3/3] reproducible_build: Work around caching issues

[Richard Purdie]

SOURCE_DATE_EPOCH can be expanded early in the parsing process before
the class extensions are applied. This can mean the directory pointed
to for the SDE can be incorrect until later in parsing. Cache the file
name in the cached value and allow it to dynamically update.

This isn't ideal but avoding expansion of the variable likely isn't
possible and I'm not sure how else to handle this. This works around
the issue until a better solution can be found.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/reproducible_build.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 89f645b8584..7571c116c82 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -115,11 +115,14 @@ EPOCHTASK = "do_deploy_source_date_epoch"
 do_unpack[postfuncs] += "create_source_date_epoch_stamp"
 
 def get_source_date_epoch_value(d):
-    cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
-    if cached:
+    epochfile = d.getVar('SDE_FILE')
+    cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
+    if cached and efile == epochfile:
         return cached
 
-    epochfile = d.getVar('SDE_FILE')
+    if cached and epochfile != efile:
+        bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
+
     source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
     try:
         with open(epochfile, 'r') as f:
@@ -137,7 +140,7 @@ def get_source_date_epoch_value(d):
     except FileNotFoundError:
         bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
 
-    d.setVar('__CACHED_SOURCE_DATE_EPOCH', str(source_date_epoch))
+    d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
     return str(source_date_epoch)
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
-- 
2.32.0