* [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds
@ 2021-10-02 22:17 Richard Purdie
2021-10-02 22:18 ` [PATCH 2/5] sstatesig: Add processing for full build paths in sysroot files Richard Purdie
` (4 more replies)
0 siblings, 5 replies; 10+ messages in thread
From: Richard Purdie @ 2021-10-02 22:17 UTC (permalink / raw)
To: openembedded-core
When reproducible builds are enabled and externalsrc is in use, the
source date epoch function is added. The conditions on the conditional
code removing the unpack task need to match the deltask function, else
the source date epoch function can end up running twice and the functions
can race with each other causing build failures or corruption.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/externalsrc.bbclass | 19 +++++++++----------
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 7f1a760eec5..ad93b2d2abf 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -109,16 +109,15 @@ python () {
if local_srcuri and task in fetch_tasks:
continue
bb.build.deltask(task, d)
-
- if bb.data.inherits_class('reproducible_build', d) and 'do_unpack' in d.getVar("SRCTREECOVEREDTASKS").split():
- # The reproducible_build's create_source_date_epoch_stamp function must
- # be run after the source is available and before the
- # do_deploy_source_date_epoch task. In the normal case, it's attached
- # to do_unpack as a postfuncs, but since we removed do_unpack (above)
- # we need to move the function elsewhere. The easiest thing to do is
- # move it into the prefuncs of the do_deploy_source_date_epoch task.
- # This is safe, as externalsrc runs with the source already unpacked.
- d.prependVarFlag('do_deploy_source_date_epoch', 'prefuncs', 'create_source_date_epoch_stamp ')
+ if bb.data.inherits_class('reproducible_build', d) and task == 'do_unpack':
+ # The reproducible_build's create_source_date_epoch_stamp function must
+ # be run after the source is available and before the
+ # do_deploy_source_date_epoch task. In the normal case, it's attached
+ # to do_unpack as a postfuncs, but since we removed do_unpack (above)
+ # we need to move the function elsewhere. The easiest thing to do is
+ # move it into the prefuncs of the do_deploy_source_date_epoch task.
+ # This is safe, as externalsrc runs with the source already unpacked.
+ d.prependVarFlag('do_deploy_source_date_epoch', 'prefuncs', 'create_source_date_epoch_stamp ')
d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
--
2.32.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 2/5] sstatesig: Add processing for full build paths in sysroot files
2021-10-02 22:17 [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds Richard Purdie
@ 2021-10-02 22:18 ` Richard Purdie
2021-10-02 22:18 ` [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds Richard Purdie
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Richard Purdie @ 2021-10-02 22:18 UTC (permalink / raw)
To: openembedded-core
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/sstate.bbclass | 7 ++++
meta/lib/oe/sstatesig.py | 39 +++++++++++++++++--
meta/recipes-devtools/perl/perl_5.34.0.bb | 7 ++++
meta/recipes-devtools/python/python3_3.9.6.bb | 8 ++++
.../qemu/qemuwrapper-cross_1.0.bb | 1 +
meta/recipes-devtools/rpm/rpm_4.16.1.3.bb | 5 +++
6 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 92a73114bb5..89e9f561787 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -67,6 +67,13 @@ SSTATE_DUPWHITELIST += "${DEPLOY_DIR_IMAGE}/microcode"
SSTATE_SCAN_FILES ?= "*.la *-config *_config postinst-*"
SSTATE_SCAN_CMD ??= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES").split())}" \) -type f'
SSTATE_SCAN_CMD_NATIVE ??= 'grep -Irl -e ${RECIPE_SYSROOT} -e ${RECIPE_SYSROOT_NATIVE} -e ${HOSTTOOLS_DIR} ${SSTATE_BUILDDIR}'
+SSTATE_HASHEQUIV_FILEMAP ?= " \
+ populate_sysroot:*/postinst-useradd-*:${TMPDIR} \
+ populate_sysroot:*/postinst-useradd-*:${COREBASE} \
+ populate_sysroot:*/postinst-useradd-*:regex-\s(PATH|PSEUDO_IGNORE_PATHS|HOME|LOGNAME|OMP_NUM_THREADS|USER)=.*\s \
+ populate_sysroot:*/crossscripts/*:${TMPDIR} \
+ populate_sysroot:*/crossscripts/*:${COREBASE} \
+ "
BB_HASHFILENAME = "False ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}"
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 24577249ffa..0c3b4589c57 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -470,6 +470,8 @@ def OEOuthashBasic(path, sigfile, task, d):
import stat
import pwd
import grp
+ import re
+ import fnmatch
def update_hash(s):
s = s.encode('utf-8')
@@ -479,6 +481,8 @@ def OEOuthashBasic(path, sigfile, task, d):
h = hashlib.sha256()
prev_dir = os.getcwd()
+ corebase = d.getVar("COREBASE")
+ tmpdir = d.getVar("TMPDIR")
include_owners = os.environ.get('PSEUDO_DISABLED') == '0'
if "package_write_" in task or task == "package_qa":
include_owners = False
@@ -489,8 +493,17 @@ def OEOuthashBasic(path, sigfile, task, d):
include_root = False
extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
+ filemaps = {}
+ for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():
+ entry = m.split(":")
+ if len(entry) != 3 or entry[0] != task:
+ continue
+ filemaps.setdefault(entry[1], [])
+ filemaps[entry[1]].append(entry[2])
+
try:
os.chdir(path)
+ basepath = os.path.normpath(path)
update_hash("OEOuthashBasic\n")
if extra_content:
@@ -572,8 +585,13 @@ def OEOuthashBasic(path, sigfile, task, d):
else:
update_hash(" " * 9)
+ filterfile = False
+ for entry in filemaps:
+ if fnmatch.fnmatch(path, entry):
+ filterfile = True
+
update_hash(" ")
- if stat.S_ISREG(s.st_mode):
+ if stat.S_ISREG(s.st_mode) and not filterfile:
update_hash("%10d" % s.st_size)
else:
update_hash(" " * 10)
@@ -582,9 +600,24 @@ def OEOuthashBasic(path, sigfile, task, d):
fh = hashlib.sha256()
if stat.S_ISREG(s.st_mode):
# Hash file contents
- with open(path, 'rb') as d:
- for chunk in iter(lambda: d.read(4096), b""):
+ if filterfile:
+ # Need to ignore paths in crossscripts and postinst-useradd files.
+ with open(path, 'rb') as d:
+ chunk = d.read()
+ chunk = chunk.replace(bytes(basepath, encoding='utf8'), b'')
+ for entry in filemaps:
+ if not fnmatch.fnmatch(path, entry):
+ continue
+ for r in filemaps[entry]:
+ if r.startswith("regex-"):
+ chunk = re.sub(bytes(r[6:], encoding='utf8'), b'', chunk)
+ else:
+ chunk = chunk.replace(bytes(r, encoding='utf8'), b'')
fh.update(chunk)
+ else:
+ with open(path, 'rb') as d:
+ for chunk in iter(lambda: d.read(4096), b""):
+ fh.update(chunk)
update_hash(fh.hexdigest())
else:
update_hash(" " * len(fh.hexdigest()))
diff --git a/meta/recipes-devtools/perl/perl_5.34.0.bb b/meta/recipes-devtools/perl/perl_5.34.0.bb
index 0e0fe7f9854..175db4ee319 100644
--- a/meta/recipes-devtools/perl/perl_5.34.0.bb
+++ b/meta/recipes-devtools/perl/perl_5.34.0.bb
@@ -385,3 +385,10 @@ EOF
chmod 0755 ${SYSROOT_DESTDIR}${bindir}/nativeperl
cat ${SYSROOT_DESTDIR}${bindir}/nativeperl
}
+
+SSTATE_HASHEQUIV_FILEMAP = " \
+ populate_sysroot:*/lib*/perl5/*/*/Config_heavy.pl:${TMPDIR} \
+ populate_sysroot:*/lib*/perl5/*/*/Config_heavy.pl:${COREBASE} \
+ populate_sysroot:*/lib*/perl5/config.sh:${TMPDIR} \
+ populate_sysroot:*/lib*/perl5/config.sh:${COREBASE} \
+ "
diff --git a/meta/recipes-devtools/python/python3_3.9.6.bb b/meta/recipes-devtools/python/python3_3.9.6.bb
index aae7837180a..d09943f891b 100644
--- a/meta/recipes-devtools/python/python3_3.9.6.bb
+++ b/meta/recipes-devtools/python/python3_3.9.6.bb
@@ -195,6 +195,14 @@ do_install:append:class-nativesdk () {
}
SSTATE_SCAN_FILES += "Makefile _sysconfigdata.py"
+SSTATE_HASHEQUIV_FILEMAP = " \
+ populate_sysroot:*/lib*/python3*/_sysconfigdata*.py:${TMPDIR} \
+ populate_sysroot:*/lib*/python3*/_sysconfigdata*.py:${COREBASE} \
+ populate_sysroot:*/lib*/python3*/config-*/Makefile:${TMPDIR} \
+ populate_sysroot:*/lib*/python3*/config-*/Makefile:${COREBASE} \
+ populate_sysroot:*/lib*/python-sysconfigdata/_sysconfigdata.py:${TMPDIR} \
+ populate_sysroot:*/lib*/python-sysconfigdata/_sysconfigdata.py:${COREBASE} \
+ "
PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
py_package_preprocess () {
diff --git a/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb b/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb
index a0448a18039..97b44ad2e57 100644
--- a/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb
+++ b/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb
@@ -18,6 +18,7 @@ do_install () {
cat >> ${D}${bindir_crossscripts}/${MLPREFIX}qemuwrapper << EOF
#!/bin/sh
+# Wrapper script to run binaries under qemu user-mode emulation
set -x
if [ ${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'True', 'False', d)} = False -a "${PN}" != "nativesdk-qemuwrapper-cross" ]; then
diff --git a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
index 60181f26c70..2ff9c2b1122 100644
--- a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
@@ -195,3 +195,8 @@ rpm_package_preprocess () {
sed -i -e 's:--sysroot[^ ]*::g' \
${PKGD}/${libdir}/rpm/macros
}
+
+SSTATE_HASHEQUIV_FILEMAP = " \
+ populate_sysroot:*/rpm/macros:${TMPDIR} \
+ populate_sysroot:*/rpm/macros:${COREBASE} \
+ "
--
2.32.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds
2021-10-02 22:17 [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds Richard Purdie
2021-10-02 22:18 ` [PATCH 2/5] sstatesig: Add processing for full build paths in sysroot files Richard Purdie
@ 2021-10-02 22:18 ` Richard Purdie
2021-10-03 2:09 ` [OE-core] " Peter Kjellerstedt
2021-10-02 22:18 ` [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION Richard Purdie
` (2 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2021-10-02 22:18 UTC (permalink / raw)
To: openembedded-core
Using DATETIME means the deploy output is not deterministic. Use SDE
when using reproducible builds since it is consistent for given input data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/image-artifact-names.bbclass | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image-artifact-names.bbclass b/meta/classes/image-artifact-names.bbclass
index 3ac8dd731a6..df9e17c6b0f 100644
--- a/meta/classes/image-artifact-names.bbclass
+++ b/meta/classes/image-artifact-names.bbclass
@@ -4,7 +4,7 @@
IMAGE_BASENAME ?= "${PN}"
IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
-IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
+IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
@@ -13,3 +13,10 @@ IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
# by default) followed by additional suffices which describe the format (.ext4,
# .ext4.xz, etc.).
IMAGE_NAME_SUFFIX ??= ".rootfs"
+
+python () {
+ if bb.data.inherits_class('reproducible_build', d):
+ import datetime
+ d.setVar("IMAGE_VERSION_SUFFIX", "-" + datetime.datetime.fromtimestamp(int(d.getVar("SOURCE_DATE_EPOCH")), datetime.timezone.utc).strftime('%Y%m%d%H%M%S'))
+ d.setVarFlag("IMAGE_VERSION_SUFFIX", "vardepvalue", "")
+}
--
2.32.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION
2021-10-02 22:17 [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds Richard Purdie
2021-10-02 22:18 ` [PATCH 2/5] sstatesig: Add processing for full build paths in sysroot files Richard Purdie
2021-10-02 22:18 ` [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds Richard Purdie
@ 2021-10-02 22:18 ` Richard Purdie
2021-10-03 9:59 ` [OE-core] " Jose Quaresma
2021-10-02 22:18 ` [PATCH 5/5] python3: Drop broken pyc files Richard Purdie
[not found] ` <16AA56A68DAA9487.23650@lists.openembedded.org>
4 siblings, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2021-10-02 22:18 UTC (permalink / raw)
To: openembedded-core
At this point the hash equivalence and sstate is 'junk' on the autobuilder
unforuntately due to the volume of fixes and also the volume of slighly
not quite right patches tested during the development of the fixes.
In order to try and help any remaining sanity I might have, bump the
version numbers to start with a clean slate so we're working from a known
good baseline rather than risk chasing phantom issues. For those
upgrading, there wouldn't be much reuse anyway after the changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/sstate.bbclass | 2 +-
meta/conf/abi_version.conf | 2 +-
meta/recipes-devtools/elfutils/elfutils_0.185.bb | 1 -
meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb | 1 -
4 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 89e9f561787..06a722c735d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1,4 +1,4 @@
-SSTATE_VERSION = "3"
+SSTATE_VERSION = "4"
SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
SSTATE_MANFILEPREFIX = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-${PN}"
diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
index 35faef9a368..515a6def405 100644
--- a/meta/conf/abi_version.conf
+++ b/meta/conf/abi_version.conf
@@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
# a reset of the equivalence, for example when reproducibility issues break the
# existing match data. Distros can also append to this value for the same effect.
#
-HASHEQUIV_HASH_VERSION = "5"
+HASHEQUIV_HASH_VERSION = "7"
diff --git a/meta/recipes-devtools/elfutils/elfutils_0.185.bb b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
index 9ea4de8e407..f4769e36328 100644
--- a/meta/recipes-devtools/elfutils/elfutils_0.185.bb
+++ b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
@@ -34,7 +34,6 @@ SRC_URI[sha256sum] = "dc8d3e74ab209465e7f568e1b3bb9a5a142f8656e2b57d10049a73da2a
# remove at next version upgrade or when output changes
PR = "r1"
-HASHEQUIV_HASH_VERSION .= ".2"
inherit autotools gettext ptest pkgconfig
diff --git a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
index 0dd18d7a068..62aa1b0e877 100644
--- a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
+++ b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
@@ -28,7 +28,6 @@ SRC_URI[sha256sum] = "4441a5d593f85bb6e8d578cf6653fb4ec30f9e8f4a2315a3d8f2d0a8b3
# remove at next version upgrade or when output changes
PR = "r1"
-HASHEQUIV_HASH_VERSION .= ".1"
RECIPE_NO_UPDATE_REASON = "6.04-pre3 is broken"
UPSTREAM_CHECK_URI = "https://www.zytor.com/pub/syslinux/"
--
2.32.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 5/5] python3: Drop broken pyc files
2021-10-02 22:17 [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds Richard Purdie
` (2 preceding siblings ...)
2021-10-02 22:18 ` [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION Richard Purdie
@ 2021-10-02 22:18 ` Richard Purdie
[not found] ` <16AA56A68DAA9487.23650@lists.openembedded.org>
4 siblings, 0 replies; 10+ messages in thread
From: Richard Purdie @ 2021-10-02 22:18 UTC (permalink / raw)
To: openembedded-core
The underlying py files are editted so delete the now incorrect pyc files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/recipes-devtools/python/python3_3.9.6.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-devtools/python/python3_3.9.6.bb b/meta/recipes-devtools/python/python3_3.9.6.bb
index d09943f891b..8a638b142b5 100644
--- a/meta/recipes-devtools/python/python3_3.9.6.bb
+++ b/meta/recipes-devtools/python/python3_3.9.6.bb
@@ -164,6 +164,7 @@ do_install:append() {
for c in ${D}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
python3 ${WORKDIR}/reformat_sysconfig.py $c
done
+ rm ${D}${libdir}/python${PYTHON_MAJMIN}/__pycache__/_sysconfigdata*.cpython*
mkdir -p ${D}${libdir}/python-sysconfigdata
sysconfigfile=`find ${D} -name _sysconfig*.py`
--
2.32.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* RE: [OE-core] [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds
2021-10-02 22:18 ` [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds Richard Purdie
@ 2021-10-03 2:09 ` Peter Kjellerstedt
0 siblings, 0 replies; 10+ messages in thread
From: Peter Kjellerstedt @ 2021-10-03 2:09 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 3 oktober 2021 00:18
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH
> when making reproducible builds
>
> Using DATETIME means the deploy output is not deterministic. Use SDE
> when using reproducible builds since it is consistent for given input
> data.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes/image-artifact-names.bbclass | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/image-artifact-names.bbclass
> b/meta/classes/image-artifact-names.bbclass
> index 3ac8dd731a6..df9e17c6b0f 100644
> --- a/meta/classes/image-artifact-names.bbclass
> +++ b/meta/classes/image-artifact-names.bbclass
> @@ -4,7 +4,7 @@
>
> IMAGE_BASENAME ?= "${PN}"
> IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
> -IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
> +IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
> IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
>
> @@ -13,3 +13,10 @@ IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
> # by default) followed by additional suffices which describe the format (.ext4,
> # .ext4.xz, etc.).
> IMAGE_NAME_SUFFIX ??= ".rootfs"
> +
> +python () {
> + if bb.data.inherits_class('reproducible_build', d):
This needs to be something like:
if bb.data.inherits_class('reproducible_build', d) and d.getVar("IMAGE_VERSION_SUFFIX", False) == "-${DATETIME}":
otherwise it will be impossible to change IMAGE_VERSION_SUFFIX if one also
has reproducible builds enabled.
//Peter
> + import datetime
> + d.setVar("IMAGE_VERSION_SUFFIX", "-" + datetime.datetime.fromtimestamp(int(d.getVar("SOURCE_DATE_EPOCH")), datetime.timezone.utc).strftime('%Y%m%d%H%M%S'))
> + d.setVarFlag("IMAGE_VERSION_SUFFIX", "vardepvalue", "")
> +}
> --
> 2.32.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION
2021-10-02 22:18 ` [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION Richard Purdie
@ 2021-10-03 9:59 ` Jose Quaresma
2021-10-03 10:28 ` Richard Purdie
0 siblings, 1 reply; 10+ messages in thread
From: Jose Quaresma @ 2021-10-03 9:59 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 3629 bytes --]
Hi Richard,
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia sábado,
2/10/2021 à(s) 23:18:
> At this point the hash equivalence and sstate is 'junk' on the autobuilder
> unforuntately due to the volume of fixes and also the volume of slighly
> not quite right patches tested during the development of the fixes.
>
> In order to try and help any remaining sanity I might have, bump the
> version numbers to start with a clean slate so we're working from a known
> good baseline rather than risk chasing phantom issues. For those
> upgrading, there wouldn't be much reuse anyway after the changes.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes/sstate.bbclass | 2 +-
> meta/conf/abi_version.conf | 2 +-
> meta/recipes-devtools/elfutils/elfutils_0.185.bb | 1 -
> meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb | 1 -
> 4 files changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 89e9f561787..06a722c735d 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -1,4 +1,4 @@
> -SSTATE_VERSION = "3"
> +SSTATE_VERSION = "4"
>
> SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
> SSTATE_MANFILEPREFIX =
> "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-${PN}"
> diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
> index 35faef9a368..515a6def405 100644
> --- a/meta/conf/abi_version.conf
> +++ b/meta/conf/abi_version.conf
> @@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
> # a reset of the equivalence, for example when reproducibility issues
> break the
> # existing match data. Distros can also append to this value for the same
> effect.
> #
> -HASHEQUIV_HASH_VERSION = "5"
> +HASHEQUIV_HASH_VERSION = "7"
>
Why is it bumped to 7 instead of 6 ?
> diff --git a/meta/recipes-devtools/elfutils/elfutils_0.185.bb
> b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
> index 9ea4de8e407..f4769e36328 100644
> --- a/meta/recipes-devtools/elfutils/elfutils_0.185.bb
> +++ b/meta/recipes-devtools/elfutils/elfutils_0.185.bb
> @@ -34,7 +34,6 @@ SRC_URI[sha256sum] =
> "dc8d3e74ab209465e7f568e1b3bb9a5a142f8656e2b57d10049a73da2a
>
> # remove at next version upgrade or when output changes
> PR = "r1"
> -HASHEQUIV_HASH_VERSION .= ".2"
>
> inherit autotools gettext ptest pkgconfig
>
> diff --git a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
> b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
> index 0dd18d7a068..62aa1b0e877 100644
> --- a/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
> +++ b/meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb
> @@ -28,7 +28,6 @@ SRC_URI[sha256sum] =
> "4441a5d593f85bb6e8d578cf6653fb4ec30f9e8f4a2315a3d8f2d0a8b3
>
> # remove at next version upgrade or when output changes
> PR = "r1"
> -HASHEQUIV_HASH_VERSION .= ".1"
>
> RECIPE_NO_UPDATE_REASON = "6.04-pre3 is broken"
> UPSTREAM_CHECK_URI = "https://www.zytor.com/pub/syslinux/"
> --
> 2.32.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156542):
> https://lists.openembedded.org/g/openembedded-core/message/156542
> Mute This Topic: https://lists.openembedded.org/mt/86032667/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
--
Best regards,
José Quaresma
[-- Attachment #2: Type: text/html, Size: 5894 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION
2021-10-03 9:59 ` [OE-core] " Jose Quaresma
@ 2021-10-03 10:28 ` Richard Purdie
2021-10-03 21:47 ` Jose Quaresma
0 siblings, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2021-10-03 10:28 UTC (permalink / raw)
To: Jose Quaresma; +Cc: OE-core
On Sun, 2021-10-03 at 10:59 +0100, Jose Quaresma wrote:
> Hi Richard,
>
> Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia sábado,
> 2/10/2021 à(s) 23:18:
> > At this point the hash equivalence and sstate is 'junk' on the autobuilder
> > unforuntately due to the volume of fixes and also the volume of slighly
> > not quite right patches tested during the development of the fixes.
> >
> > In order to try and help any remaining sanity I might have, bump the
> > version numbers to start with a clean slate so we're working from a known
> > good baseline rather than risk chasing phantom issues. For those
> > upgrading, there wouldn't be much reuse anyway after the changes.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > meta/classes/sstate.bbclass | 2 +-
> > meta/conf/abi_version.conf | 2 +-
> > meta/recipes-devtools/elfutils/elfutils_0.185.bb | 1 -
> > meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb | 1 -
> > 4 files changed, 2 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > index 89e9f561787..06a722c735d 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -1,4 +1,4 @@
> > -SSTATE_VERSION = "3"
> > +SSTATE_VERSION = "4"
> >
> > SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
> > SSTATE_MANFILEPREFIX = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-
> > ${PN}"
> > diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
> > index 35faef9a368..515a6def405 100644
> > --- a/meta/conf/abi_version.conf
> > +++ b/meta/conf/abi_version.conf
> > @@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
> > # a reset of the equivalence, for example when reproducibility issues break
> > the
> > # existing match data. Distros can also append to this value for the same
> > effect.
> > #
> > -HASHEQUIV_HASH_VERSION = "5"
> > +HASHEQUIV_HASH_VERSION = "7"
> >
>
>
> Why is it bumped to 7 instead of 6 ?
I'd already managed to break the 6 namespace in testing on the autobuilder. I
don't have any easy way to remove that namespace from the hash equivalence
server. I could likely do it with a lot of manual work.
Sadly there is some deeper issue going on and current master on the autobuilder
is corrupted and even after these bumps, there is some problem creeping back in
and corrupting[1] builds. I'm trying to narrow it down but it is
proving problematic.
[1] the corruption is bad file timestamps so not something really bad but it
does need to be fixed.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds
[not found] ` <16AA56A68DAA9487.23650@lists.openembedded.org>
@ 2021-10-03 13:35 ` Richard Purdie
0 siblings, 0 replies; 10+ messages in thread
From: Richard Purdie @ 2021-10-03 13:35 UTC (permalink / raw)
To: openembedded-core
On Sat, 2021-10-02 at 23:18 +0100, Richard Purdie via lists.openembedded.org
wrote:
> Using DATETIME means the deploy output is not deterministic. Use SDE
> when using reproducible builds since it is consistent for given input data.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes/image-artifact-names.bbclass | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/image-artifact-names.bbclass b/meta/classes/image-artifact-names.bbclass
> index 3ac8dd731a6..df9e17c6b0f 100644
> --- a/meta/classes/image-artifact-names.bbclass
> +++ b/meta/classes/image-artifact-names.bbclass
> @@ -4,7 +4,7 @@
>
> IMAGE_BASENAME ?= "${PN}"
> IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
> -IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
> +IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
> IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
>
> @@ -13,3 +13,10 @@ IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
> # by default) followed by additional suffices which describe the format (.ext4,
> # .ext4.xz, etc.).
> IMAGE_NAME_SUFFIX ??= ".rootfs"
> +
> +python () {
> + if bb.data.inherits_class('reproducible_build', d):
> + import datetime
> + d.setVar("IMAGE_VERSION_SUFFIX", "-" + datetime.datetime.fromtimestamp(int(d.getVar("SOURCE_DATE_EPOCH")), datetime.timezone.utc).strftime('%Y%m%d%H%M%S'))
> + d.setVarFlag("IMAGE_VERSION_SUFFIX", "vardepvalue", "")
> +}
This patch is incorrect. It breaks images as they have noexec do_unpack tasks.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION
2021-10-03 10:28 ` Richard Purdie
@ 2021-10-03 21:47 ` Jose Quaresma
0 siblings, 0 replies; 10+ messages in thread
From: Jose Quaresma @ 2021-10-03 21:47 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 3438 bytes --]
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia
domingo, 3/10/2021 à(s) 11:28:
> On Sun, 2021-10-03 at 10:59 +0100, Jose Quaresma wrote:
> > Hi Richard,
> >
> > Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia
> sábado,
> > 2/10/2021 à(s) 23:18:
> > > At this point the hash equivalence and sstate is 'junk' on the
> autobuilder
> > > unforuntately due to the volume of fixes and also the volume of slighly
> > > not quite right patches tested during the development of the fixes.
> > >
> > > In order to try and help any remaining sanity I might have, bump the
> > > version numbers to start with a clean slate so we're working from a
> known
> > > good baseline rather than risk chasing phantom issues. For those
> > > upgrading, there wouldn't be much reuse anyway after the changes.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > > meta/classes/sstate.bbclass | 2 +-
> > > meta/conf/abi_version.conf | 2 +-
> > > meta/recipes-devtools/elfutils/elfutils_0.185.bb | 1 -
> > > meta/recipes-devtools/syslinux/syslinux_6.04-pre2.bb | 1 -
> > > 4 files changed, 2 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > > index 89e9f561787..06a722c735d 100644
> > > --- a/meta/classes/sstate.bbclass
> > > +++ b/meta/classes/sstate.bbclass
> > > @@ -1,4 +1,4 @@
> > > -SSTATE_VERSION = "3"
> > > +SSTATE_VERSION = "4"
> > >
> > > SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
> > > SSTATE_MANFILEPREFIX =
> "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-
> > > ${PN}"
> > > diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
> > > index 35faef9a368..515a6def405 100644
> > > --- a/meta/conf/abi_version.conf
> > > +++ b/meta/conf/abi_version.conf
> > > @@ -12,4 +12,4 @@ OELAYOUT_ABI = "14"
> > > # a reset of the equivalence, for example when reproducibility issues
> break
> > > the
> > > # existing match data. Distros can also append to this value for the
> same
> > > effect.
> > > #
> > > -HASHEQUIV_HASH_VERSION = "5"
> > > +HASHEQUIV_HASH_VERSION = "7"
> > >
> >
> >
> > Why is it bumped to 7 instead of 6 ?
>
> I'd already managed to break the 6 namespace in testing on the
> autobuilder. I
> don't have any easy way to remove that namespace from the hash equivalence
> server. I could likely do it with a lot of manual work.
>
> Sadly there is some deeper issue going on and current master on the
> autobuilder
> is corrupted and even after these bumps, there is some problem creeping
> back in
> and corrupting[1] builds. I'm trying to narrow it down but it is
> proving problematic.
>
> [1] the corruption is bad file timestamps so not something really bad but
> it
> does need to be fixed.
>
> Cheers,
>
> Richard
>
>
Thank you for the explanation.
Regarding the corruption of bad file timestamps I have sent a patch that
can be
a little useful, It will show a warning when the timestamp can't be updated.
https://lists.openembedded.org/g/openembedded-core/topic/patch_sstate_bbclass_adds/86052256?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,86052256,previd=1633297133859036611,nextid=1633195100689062135&previd=1633297133859036611&nextid=1633195100689062135
--
Best regards,
José Quaresma
[-- Attachment #2: Type: text/html, Size: 5046 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-10-03 21:47 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-02 22:17 [PATCH 1/5] externalsrc: Fix a source date epoch race in reproducible builds Richard Purdie
2021-10-02 22:18 ` [PATCH 2/5] sstatesig: Add processing for full build paths in sysroot files Richard Purdie
2021-10-02 22:18 ` [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds Richard Purdie
2021-10-03 2:09 ` [OE-core] " Peter Kjellerstedt
2021-10-02 22:18 ` [PATCH 4/5] abi_version/sstate: Bump HASH_VERSION and SSTATE_VERSION Richard Purdie
2021-10-03 9:59 ` [OE-core] " Jose Quaresma
2021-10-03 10:28 ` Richard Purdie
2021-10-03 21:47 ` Jose Quaresma
2021-10-02 22:18 ` [PATCH 5/5] python3: Drop broken pyc files Richard Purdie
[not found] ` <16AA56A68DAA9487.23650@lists.openembedded.org>
2021-10-03 13:35 ` [OE-core] [PATCH 3/5] image-artifact-names: Use SOURCE_DATE_EPOCH when making reproducible builds Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.