* [OE-core][dunfell 1/6] selftest/reproducible: enable world reproducibility test
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 2/6] selftest/reproducible: add an exclusion list for items that are not yet reproducible Steve Sakoman
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
From: Alexander Kanavin <alex.kanavin@gmail.com>
Add systemd and pam distro features, and commercial license flag
to include more recipes into the world set.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ed8f94942c07784f9f6eef5c00b75aa9b398ea5e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index f570958f80..4b60b2e1b0 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -83,7 +83,7 @@ class ReproducibleTests(OESelftestTestCase):
package_classes = ['deb', 'ipk']
# targets are the things we want to test the reproducibility of
- targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline']
+ targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'world']
# sstate targets are things to pull from sstate to potentially cut build/debugging time
sstate_targets = []
save_results = False
@@ -156,6 +156,8 @@ class ReproducibleTests(OESelftestTestCase):
PACKAGE_CLASSES = "{package_classes}"
INHIBIT_PACKAGE_STRIP = "1"
TMPDIR = "{tmpdir}"
+ LICENSE_FLAGS_WHITELIST = "commercial"
+ DISTRO_FEATURES_append = ' systemd pam'
''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes),
tmpdir=tmpdir)
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OE-core][dunfell 2/6] selftest/reproducible: add an exclusion list for items that are not yet reproducible
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 1/6] selftest/reproducible: enable world reproducibility test Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 3/6] selftest/reproducible: adjust exclusion list for dunfell Steve Sakoman
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
From: Alexander Kanavin <alex.kanavin@gmail.com>
Hopefully over time this list will be reduced to an empty one.
Non-reproducible excluded packages are not given to diffoscope and do not cause a
failure, but still saved side-by-side with non-reproducible failing ones to make
investigation easier.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 406bd0d48d8f90e2c836f7d3e204f21d5f13c833)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 82 +++++++++++++++++++-
1 file changed, 79 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 4b60b2e1b0..cf0375c8a1 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -17,6 +17,72 @@ import stat
import os
import datetime
+# For sample packages, see:
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-0t7wr_oo/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-4s9ejwyp/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-haiwdlbr/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-hwds3mcl/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201203-sua0pzvc/
+# (both packages/ and packages-excluded/)
+exclude_packages = [
+ 'acpica-src',
+ 'babeltrace2-ptest',
+ 'bootchart2-doc',
+ 'cups',
+ 'cwautomacros',
+ 'dtc',
+ 'efivar',
+ 'epiphany',
+ 'gcr',
+ 'git',
+ 'glide',
+ 'go-dep',
+ 'go-helloworld',
+ 'go-runtime',
+ 'go_',
+ 'groff',
+ 'gst-devtools',
+ 'gstreamer1.0-python',
+ 'gtk-doc',
+ 'igt-gpu-tools',
+ 'kernel-devsrc',
+ 'libaprutil',
+ 'libcap-ng',
+ 'libhandy-1-src',
+ 'libid3tag',
+ 'libproxy',
+ 'libsecret-dev',
+ 'libsecret-src',
+ 'lttng-tools-dbg',
+ 'lttng-tools-ptest',
+ 'ltp',
+ 'meson',
+ 'ovmf-shell-efi',
+ 'parted-ptest',
+ 'perf',
+ 'python3-cython',
+ 'qemu',
+ 'quilt-ptest',
+ 'rsync',
+ 'ruby',
+ 'spirv-tools-dev',
+ 'swig',
+ 'syslinux-misc',
+ 'systemd-bootchart',
+ 'valgrind-ptest',
+ 'vim',
+ 'watchdog',
+ 'xmlto',
+ 'xorg-minimal-fonts'
+ ]
+
+def is_excluded(package):
+ package_name = os.path.basename(package)
+ for i in exclude_packages:
+ if package_name.startswith(i):
+ return True
+ return False
+
MISSING = 'MISSING'
DIFFERENT = 'DIFFERENT'
SAME = 'SAME'
@@ -39,6 +105,7 @@ class PackageCompareResults(object):
self.total = []
self.missing = []
self.different = []
+ self.different_excluded = []
self.same = []
def add_result(self, r):
@@ -46,7 +113,10 @@ class PackageCompareResults(object):
if r.status == MISSING:
self.missing.append(r)
elif r.status == DIFFERENT:
- self.different.append(r)
+ if is_excluded(r.reference):
+ self.different_excluded.append(r)
+ else:
+ self.different.append(r)
else:
self.same.append(r)
@@ -54,10 +124,11 @@ class PackageCompareResults(object):
self.total.sort()
self.missing.sort()
self.different.sort()
+ self.different_excluded.sort()
self.same.sort()
def __str__(self):
- return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total))
+ return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total))
def compare_file(reference, test, diffutils_sysroot):
result = CompareResult()
@@ -226,6 +297,7 @@ class ReproducibleTests(OESelftestTestCase):
self.write_package_list(package_class, 'missing', result.missing)
self.write_package_list(package_class, 'different', result.different)
+ self.write_package_list(package_class, 'different_excluded', result.different_excluded)
self.write_package_list(package_class, 'same', result.same)
if self.save_results:
@@ -233,8 +305,12 @@ class ReproducibleTests(OESelftestTestCase):
self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
+ for d in result.different_excluded:
+ self.copy_file(d.reference, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.reference)]))
+ self.copy_file(d.test, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.test)]))
+
if result.missing or result.different:
- fails.append("The following %s packages are missing or different: %s" %
+ fails.append("The following %s packages are missing or different and not in exclusion list: %s" %
(c, '\n'.join(r.test for r in (result.missing + result.different))))
# Clean up empty directories
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OE-core][dunfell 3/6] selftest/reproducible: adjust exclusion list for dunfell
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 1/6] selftest/reproducible: enable world reproducibility test Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 2/6] selftest/reproducible: add an exclusion list for items that are not yet reproducible Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 4/6] selftest/reproducible: track unusued entries in the exclusion list Steve Sakoman
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
Signed-off-be: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 32 ++++++--------------
1 file changed, 10 insertions(+), 22 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index cf0375c8a1..c023c92d5d 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -27,11 +27,8 @@ import datetime
exclude_packages = [
'acpica-src',
'babeltrace2-ptest',
+ 'bind',
'bootchart2-doc',
- 'cups',
- 'cwautomacros',
- 'dtc',
- 'efivar',
'epiphany',
'gcr',
'git',
@@ -40,40 +37,31 @@ exclude_packages = [
'go-helloworld',
'go-runtime',
'go_',
- 'groff',
- 'gst-devtools',
'gstreamer1.0-python',
- 'gtk-doc',
- 'igt-gpu-tools',
+ 'hwlatdetect',
'kernel-devsrc',
'libaprutil',
'libcap-ng',
- 'libhandy-1-src',
- 'libid3tag',
+ 'libjson',
'libproxy',
- 'libsecret-dev',
- 'libsecret-src',
+ 'lsb-release',
'lttng-tools-dbg',
'lttng-tools-ptest',
'ltp',
- 'meson',
'ovmf-shell-efi',
'parted-ptest',
'perf',
- 'python3-cython',
+ 'piglit',
+ 'pybootchartgui',
'qemu',
- 'quilt-ptest',
+ "rpm",
'rsync',
'ruby',
- 'spirv-tools-dev',
- 'swig',
- 'syslinux-misc',
+ 'stress-ng',
'systemd-bootchart',
- 'valgrind-ptest',
+ 'systemtap',
'vim',
- 'watchdog',
- 'xmlto',
- 'xorg-minimal-fonts'
+ 'webkitgtk',
]
def is_excluded(package):
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OE-core][dunfell 4/6] selftest/reproducible: track unusued entries in the exclusion list
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
` (2 preceding siblings ...)
2021-04-05 14:53 ` [OE-core][dunfell 3/6] selftest/reproducible: adjust exclusion list for dunfell Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 5/6] selftest/reproducible: Sort the unused " Steve Sakoman
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
From: Alexander Kanavin <alex.kanavin@gmail.com>
This helps with trimming down the list, and towards 100% reproducibility :)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da7a173d7a01524229c8515326465968a845e96f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index c023c92d5d..b331c3dedf 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -68,8 +68,8 @@ def is_excluded(package):
package_name = os.path.basename(package)
for i in exclude_packages:
if package_name.startswith(i):
- return True
- return False
+ return i
+ return None
MISSING = 'MISSING'
DIFFERENT = 'DIFFERENT'
@@ -95,14 +95,17 @@ class PackageCompareResults(object):
self.different = []
self.different_excluded = []
self.same = []
+ self.active_exclusions = set()
def add_result(self, r):
self.total.append(r)
if r.status == MISSING:
self.missing.append(r)
elif r.status == DIFFERENT:
- if is_excluded(r.reference):
+ exclusion = is_excluded(r.reference)
+ if exclusion:
self.different_excluded.append(r)
+ self.active_exclusions.add(exclusion)
else:
self.different.append(r)
else:
@@ -116,7 +119,10 @@ class PackageCompareResults(object):
self.same.sort()
def __str__(self):
- return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total))
+ return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions())
+
+ def unused_exclusions(self):
+ return set(exclude_packages) - self.active_exclusions
def compare_file(reference, test, diffutils_sysroot):
result = CompareResult()
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OE-core][dunfell 5/6] selftest/reproducible: Sort the unused exclusion list
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
` (3 preceding siblings ...)
2021-04-05 14:53 ` [OE-core][dunfell 4/6] selftest/reproducible: track unusued entries in the exclusion list Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-05 14:53 ` [OE-core][dunfell 6/6] diffoscope: Upgrade 136 -> 168 Steve Sakoman
2021-04-06 10:46 ` [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Richard Purdie
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes comparisions between lists easier.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d2c52125d1cdc06c7e08d507ca68f3e4612a4314)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index b331c3dedf..f5e3bff5a4 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -122,7 +122,7 @@ class PackageCompareResults(object):
return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions())
def unused_exclusions(self):
- return set(exclude_packages) - self.active_exclusions
+ return sorted(set(exclude_packages) - self.active_exclusions)
def compare_file(reference, test, diffutils_sysroot):
result = CompareResult()
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OE-core][dunfell 6/6] diffoscope: Upgrade 136 -> 168
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
` (4 preceding siblings ...)
2021-04-05 14:53 ` [OE-core][dunfell 5/6] selftest/reproducible: Sort the unused " Steve Sakoman
@ 2021-04-05 14:53 ` Steve Sakoman
2021-04-06 10:46 ` [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Richard Purdie
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-04-05 14:53 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Needed to support reproducible build testing on autobuilder
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../{diffoscope_136.bb => diffoscope_168.bb} | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
rename meta/recipes-support/diffoscope/{diffoscope_136.bb => diffoscope_168.bb} (46%)
diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_168.bb
similarity index 46%
rename from meta/recipes-support/diffoscope/diffoscope_136.bb
rename to meta/recipes-support/diffoscope/diffoscope_168.bb
index e699d15e90..00a8195c84 100644
--- a/meta/recipes-support/diffoscope/diffoscope_136.bb
+++ b/meta/recipes-support/diffoscope/diffoscope_168.bb
@@ -1,10 +1,5 @@
SUMMARY = "in-depth comparison of files, archives, and directories"
-DESCRIPTION = "Tries to get to the bottom of what makes files or directories \
-different. It will recursively unpack archives of many kinds and transform \
-various binary formats into more human-readable form to compare them. \
-It can compare two tarballs, ISO images, or PDF just as easily."
HOMEPAGE = "https://diffoscope.org/"
-BUGTRACKER = "https://salsa.debian.org/reproducible-builds/diffoscope/-/issues"
LICENSE = "GPL-3.0+"
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
@@ -12,12 +7,18 @@ PYPI_PACKAGE = "diffoscope"
inherit pypi setuptools3
-SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
-SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+SRC_URI[sha256sum] = "c6f1dc3e75b7e2e5ceac4f857fbd2ee0ddb3f0169c2b39ea9187af34208e98de"
RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
# Dependencies don't build for musl
COMPATIBLE_HOST_libc-musl = 'null'
+do_install_append_class-native() {
+ create_wrapper ${D}${bindir}/diffoscope \
+ MAGIC=${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc \
+ RPM_CONFIGDIR=${STAGING_LIBDIR_NATIVE}/rpm \
+ RPM_ETCCONFIGDIR=${STAGING_DIR_NATIVE}
+}
+
BBCLASSEXTEND = "native"
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing
2021-04-05 14:53 [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing Steve Sakoman
` (5 preceding siblings ...)
2021-04-05 14:53 ` [OE-core][dunfell 6/6] diffoscope: Upgrade 136 -> 168 Steve Sakoman
@ 2021-04-06 10:46 ` Richard Purdie
6 siblings, 0 replies; 8+ messages in thread
From: Richard Purdie @ 2021-04-06 10:46 UTC (permalink / raw)
To: Steve Sakoman, openembedded-core
On Mon, 2021-04-05 at 04:53 -1000, Steve Sakoman wrote:
> This patch series enables world reproducibility testing so that we can begin monitoring
> progress on reproducibile builds for dunfell. It also includes a version update for
> diffoscope to better enable debugging failed packages.
>
> The main downside to merging this series is increased autobuilder load to run the world
> tests.
>
> I've been testing this over the past couple of months and have updated the exclusion list for
> dunfell as I've encountered intemittent reproducibility issues (see selftest/reproducible:
> adjust exclusion list for dunfell)
Much as I hate loading up the autobuilder, improving reproducibility in dunfell
where we can relatively easily is a good thing.
It could be we just don't test it on as many targets as in master?
[which means I'm in favour of these patches]
Cheers,
Richard
^ permalink raw reply [flat|nested] 8+ messages in thread