[OE-core][dunfell 0/6] RFC: Enable world reproducibility testing

[OE-core][dunfell 0/6] RFC: Enable world reproducibility testing

[Steve Sakoman]

This patch series enables world reproducibility testing so that we can begin monitoring
progress on reproducibile builds for dunfell. It also includes a version update for
diffoscope to better enable debugging failed packages.

The main downside to merging this series is increased autobuilder load to run the world
tests.

I've been testing this over the past couple of months and have updated the exclusion list for
dunfell as I've encountered intemittent reproducibility issues (see selftest/reproducible:
adjust exclusion list for dunfell)

Most recent passing test of a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2024

Once this patch is merged reproducibility reports will be published at:

https://www.yoctoproject.org/reproducible-build-results/?branch=dunfell

The following changes since commit d044d9c0cb672c499059eb273e399ce4aee17e0d:

  image,populate_sdk_base: move 'func' flag setting for sdk command vars (2021-04-02 04:21:56 -1000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Alexander Kanavin (3):
  selftest/reproducible: enable world reproducibility test
  selftest/reproducible: add an exclusion list for items that are not
    yet reproducible
  selftest/reproducible: track unusued entries in the exclusion list

Richard Purdie (2):
  selftest/reproducible: Sort the unused exclusion list
  diffoscope: Upgrade 136 -> 168

Steve Sakoman (1):
  selftest/reproducible: adjust exclusion list for dunfell

 meta/lib/oeqa/selftest/cases/reproducible.py  | 80 ++++++++++++++++++-
 .../{diffoscope_136.bb => diffoscope_168.bb}  | 15 ++--
 2 files changed, 84 insertions(+), 11 deletions(-)
 rename meta/recipes-support/diffoscope/{diffoscope_136.bb => diffoscope_168.bb} (46%)

-- 
2.25.1

[OE-core][dunfell 1/6] selftest/reproducible: enable world reproducibility test

[Steve Sakoman]

From: Alexander Kanavin <alex.kanavin@gmail.com>

Add systemd and pam distro features, and commercial license flag
to include more recipes into the world set.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ed8f94942c07784f9f6eef5c00b75aa9b398ea5e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index f570958f80..4b60b2e1b0 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -83,7 +83,7 @@ class ReproducibleTests(OESelftestTestCase):
     package_classes = ['deb', 'ipk']
 
     # targets are the things we want to test the reproducibility of
-    targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline']
+    targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'world']
     # sstate targets are things to pull from sstate to potentially cut build/debugging time
     sstate_targets = []
     save_results = False
@@ -156,6 +156,8 @@ class ReproducibleTests(OESelftestTestCase):
             PACKAGE_CLASSES = "{package_classes}"
             INHIBIT_PACKAGE_STRIP = "1"
             TMPDIR = "{tmpdir}"
+            LICENSE_FLAGS_WHITELIST = "commercial"
+            DISTRO_FEATURES_append = ' systemd pam'
             ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes),
                         tmpdir=tmpdir)
 
-- 
2.25.1

[OE-core][dunfell 2/6] selftest/reproducible: add an exclusion list for items that are not yet reproducible

[Steve Sakoman]

From: Alexander Kanavin <alex.kanavin@gmail.com>

Hopefully over time this list will be reduced to an empty one.

Non-reproducible excluded packages are not given to diffoscope and do not cause a
failure, but still saved side-by-side with non-reproducible failing ones to make
investigation easier.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 406bd0d48d8f90e2c836f7d3e204f21d5f13c833)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 82 +++++++++++++++++++-
 1 file changed, 79 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 4b60b2e1b0..cf0375c8a1 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -17,6 +17,72 @@ import stat
 import os
 import datetime
 
+# For sample packages, see:
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-0t7wr_oo/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-4s9ejwyp/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-haiwdlbr/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201127-hwds3mcl/
+# https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20201203-sua0pzvc/
+# (both packages/ and packages-excluded/)
+exclude_packages = [
+	'acpica-src',
+	'babeltrace2-ptest',
+	'bootchart2-doc',
+	'cups',
+	'cwautomacros',
+	'dtc',
+	'efivar',
+	'epiphany',
+	'gcr',
+	'git',
+	'glide',
+	'go-dep',
+	'go-helloworld',
+	'go-runtime',
+	'go_',
+	'groff',
+	'gst-devtools',
+	'gstreamer1.0-python',
+	'gtk-doc',
+	'igt-gpu-tools',
+        'kernel-devsrc',
+	'libaprutil',
+	'libcap-ng',
+	'libhandy-1-src',
+	'libid3tag',
+	'libproxy',
+	'libsecret-dev',
+	'libsecret-src',
+	'lttng-tools-dbg',
+	'lttng-tools-ptest',
+	'ltp',
+	'meson',
+	'ovmf-shell-efi',
+	'parted-ptest',
+	'perf',
+	'python3-cython',
+	'qemu',
+	'quilt-ptest',
+	'rsync',
+	'ruby',
+	'spirv-tools-dev',
+	'swig',
+	'syslinux-misc',
+	'systemd-bootchart',
+	'valgrind-ptest',
+	'vim',
+	'watchdog',
+	'xmlto',
+	'xorg-minimal-fonts'
+	]
+
+def is_excluded(package):
+    package_name = os.path.basename(package)
+    for i in exclude_packages:
+        if package_name.startswith(i):
+            return True
+    return False
+
 MISSING = 'MISSING'
 DIFFERENT = 'DIFFERENT'
 SAME = 'SAME'
@@ -39,6 +105,7 @@ class PackageCompareResults(object):
         self.total = []
         self.missing = []
         self.different = []
+        self.different_excluded = []
         self.same = []
 
     def add_result(self, r):
@@ -46,7 +113,10 @@ class PackageCompareResults(object):
         if r.status == MISSING:
             self.missing.append(r)
         elif r.status == DIFFERENT:
-            self.different.append(r)
+            if is_excluded(r.reference):
+                self.different_excluded.append(r)
+            else:
+                self.different.append(r)
         else:
             self.same.append(r)
 
@@ -54,10 +124,11 @@ class PackageCompareResults(object):
         self.total.sort()
         self.missing.sort()
         self.different.sort()
+        self.different_excluded.sort()
         self.same.sort()
 
     def __str__(self):
-        return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total))
+        return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total))
 
 def compare_file(reference, test, diffutils_sysroot):
     result = CompareResult()
@@ -226,6 +297,7 @@ class ReproducibleTests(OESelftestTestCase):
 
                 self.write_package_list(package_class, 'missing', result.missing)
                 self.write_package_list(package_class, 'different', result.different)
+                self.write_package_list(package_class, 'different_excluded', result.different_excluded)
                 self.write_package_list(package_class, 'same', result.same)
 
                 if self.save_results:
@@ -233,8 +305,12 @@ class ReproducibleTests(OESelftestTestCase):
                         self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
                         self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
 
+                    for d in result.different_excluded:
+                        self.copy_file(d.reference, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.reference)]))
+                        self.copy_file(d.test, '/'.join([save_dir, 'packages-excluded', strip_topdir(d.test)]))
+
                 if result.missing or result.different:
-                    fails.append("The following %s packages are missing or different: %s" %
+                    fails.append("The following %s packages are missing or different and not in exclusion list: %s" %
                             (c, '\n'.join(r.test for r in (result.missing + result.different))))
 
         # Clean up empty directories
-- 
2.25.1

[OE-core][dunfell 3/6] selftest/reproducible: adjust exclusion list for dunfell

[Steve Sakoman]

Signed-off-be: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 32 ++++++--------------
 1 file changed, 10 insertions(+), 22 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index cf0375c8a1..c023c92d5d 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -27,11 +27,8 @@ import datetime
 exclude_packages = [
 	'acpica-src',
 	'babeltrace2-ptest',
+	'bind',
 	'bootchart2-doc',
-	'cups',
-	'cwautomacros',
-	'dtc',
-	'efivar',
 	'epiphany',
 	'gcr',
 	'git',
@@ -40,40 +37,31 @@ exclude_packages = [
 	'go-helloworld',
 	'go-runtime',
 	'go_',
-	'groff',
-	'gst-devtools',
 	'gstreamer1.0-python',
-	'gtk-doc',
-	'igt-gpu-tools',
+	'hwlatdetect',
         'kernel-devsrc',
 	'libaprutil',
 	'libcap-ng',
-	'libhandy-1-src',
-	'libid3tag',
+	'libjson',
 	'libproxy',
-	'libsecret-dev',
-	'libsecret-src',
+	'lsb-release',
 	'lttng-tools-dbg',
 	'lttng-tools-ptest',
 	'ltp',
-	'meson',
 	'ovmf-shell-efi',
 	'parted-ptest',
 	'perf',
-	'python3-cython',
+	'piglit',
+	'pybootchartgui',
 	'qemu',
-	'quilt-ptest',
+	"rpm",
 	'rsync',
 	'ruby',
-	'spirv-tools-dev',
-	'swig',
-	'syslinux-misc',
+	'stress-ng',
 	'systemd-bootchart',
-	'valgrind-ptest',
+	'systemtap',
 	'vim',
-	'watchdog',
-	'xmlto',
-	'xorg-minimal-fonts'
+	'webkitgtk',
 	]
 
 def is_excluded(package):
-- 
2.25.1

[OE-core][dunfell 4/6] selftest/reproducible: track unusued entries in the exclusion list

[Steve Sakoman]

From: Alexander Kanavin <alex.kanavin@gmail.com>

This helps with trimming down the list, and towards 100% reproducibility :)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da7a173d7a01524229c8515326465968a845e96f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index c023c92d5d..b331c3dedf 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -68,8 +68,8 @@ def is_excluded(package):
     package_name = os.path.basename(package)
     for i in exclude_packages:
         if package_name.startswith(i):
-            return True
-    return False
+            return i
+    return None
 
 MISSING = 'MISSING'
 DIFFERENT = 'DIFFERENT'
@@ -95,14 +95,17 @@ class PackageCompareResults(object):
         self.different = []
         self.different_excluded = []
         self.same = []
+        self.active_exclusions = set()
 
     def add_result(self, r):
         self.total.append(r)
         if r.status == MISSING:
             self.missing.append(r)
         elif r.status == DIFFERENT:
-            if is_excluded(r.reference):
+            exclusion = is_excluded(r.reference)
+            if exclusion:
                 self.different_excluded.append(r)
+                self.active_exclusions.add(exclusion)
             else:
                 self.different.append(r)
         else:
@@ -116,7 +119,10 @@ class PackageCompareResults(object):
         self.same.sort()
 
     def __str__(self):
-        return 'same=%i different=%i different_excluded=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total))
+        return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions())
+
+    def unused_exclusions(self):
+        return set(exclude_packages) - self.active_exclusions
 
 def compare_file(reference, test, diffutils_sysroot):
     result = CompareResult()
-- 
2.25.1

[OE-core][dunfell 5/6] selftest/reproducible: Sort the unused exclusion list

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

This makes comparisions between lists easier.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d2c52125d1cdc06c7e08d507ca68f3e4612a4314)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index b331c3dedf..f5e3bff5a4 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -122,7 +122,7 @@ class PackageCompareResults(object):
         return 'same=%i different=%i different_excluded=%i missing=%i total=%i\nunused_exclusions=%s' % (len(self.same), len(self.different), len(self.different_excluded), len(self.missing), len(self.total), self.unused_exclusions())
 
     def unused_exclusions(self):
-        return set(exclude_packages) - self.active_exclusions
+        return sorted(set(exclude_packages) - self.active_exclusions)
 
 def compare_file(reference, test, diffutils_sysroot):
     result = CompareResult()
-- 
2.25.1

[OE-core][dunfell 6/6] diffoscope: Upgrade 136 -> 168

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Needed to support reproducible build testing on autobuilder

Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../{diffoscope_136.bb => diffoscope_168.bb}      | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
 rename meta/recipes-support/diffoscope/{diffoscope_136.bb => diffoscope_168.bb} (46%)

diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_168.bb
similarity index 46%
rename from meta/recipes-support/diffoscope/diffoscope_136.bb
rename to meta/recipes-support/diffoscope/diffoscope_168.bb
index e699d15e90..00a8195c84 100644
--- a/meta/recipes-support/diffoscope/diffoscope_136.bb
+++ b/meta/recipes-support/diffoscope/diffoscope_168.bb
@@ -1,10 +1,5 @@
 SUMMARY = "in-depth comparison of files, archives, and directories"
-DESCRIPTION = "Tries to get to the bottom of what makes files or directories \
-different. It will recursively unpack archives of many kinds and transform \
-various binary formats into more human-readable form to compare them. \
-It can compare two tarballs, ISO images, or PDF just as easily."
 HOMEPAGE = "https://diffoscope.org/"
-BUGTRACKER = "https://salsa.debian.org/reproducible-builds/diffoscope/-/issues"
 LICENSE = "GPL-3.0+"
 LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
 
@@ -12,12 +7,18 @@ PYPI_PACKAGE = "diffoscope"
 
 inherit pypi setuptools3
 
-SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
-SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+SRC_URI[sha256sum] = "c6f1dc3e75b7e2e5ceac4f857fbd2ee0ddb3f0169c2b39ea9187af34208e98de"
 
 RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
 
 # Dependencies don't build for musl
 COMPATIBLE_HOST_libc-musl = 'null'
 
+do_install_append_class-native() {
+	create_wrapper ${D}${bindir}/diffoscope \
+		MAGIC=${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc \
+		RPM_CONFIGDIR=${STAGING_LIBDIR_NATIVE}/rpm \
+		RPM_ETCCONFIGDIR=${STAGING_DIR_NATIVE}
+}
+
 BBCLASSEXTEND = "native"
-- 
2.25.1

Re: [OE-core][dunfell 0/6] RFC: Enable world reproducibility testing

[Richard Purdie]

On Mon, 2021-04-05 at 04:53 -1000, Steve Sakoman wrote:
> This patch series enables world reproducibility testing so that we can begin monitoring
> progress on reproducibile builds for dunfell. It also includes a version update for
> diffoscope to better enable debugging failed packages.
> 
> The main downside to merging this series is increased autobuilder load to run the world
> tests.
> 
> I've been testing this over the past couple of months and have updated the exclusion list for
> dunfell as I've encountered intemittent reproducibility issues (see selftest/reproducible:
> adjust exclusion list for dunfell)

Much as I hate loading up the autobuilder, improving reproducibility in dunfell
where we can relatively easily is a good thing.

It could be we just don't test it on as many targets as in master?

[which means I'm in favour of these patches]

Cheers,

Richard