[PATCH 0/4] oeqa: Run diffoscop on saved output

[PATCH 0/4] oeqa: Run diffoscop on saved output

[Joshua Watt]

Adds recipes to build the diffoscope tool and run it when the OEQA
reproducible build test saves output to a local path. This should make
it much easier to debug reproducible build issues from the autobuilder,
since the published output can be easily viewed on the website.

Joshua Watt (4):
  python: Add libarchive-c recipe
  python: Add magic recipe
  recipes-support: Add diffoscope recipe
  oeqa: reproducible: Run diffoscope on saved output

 meta/conf/distro/include/maintainers.inc      |  3 +++
 meta/lib/oeqa/selftest/cases/reproducible.py  | 24 +++++++++++++------
 .../python/python3-libarchive-c_2.9.bb        | 17 +++++++++++++
 .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++
 .../diffoscope/diffoscope_136.bb              | 15 ++++++++++++
 5 files changed, 71 insertions(+), 7 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
 create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb

-- 
2.23.0

[PATCH 1/4] python: Add libarchive-c recipe

[Joshua Watt]

The libarchive python module is used by diffoscope tool to
make build comparisons.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/distro/include/maintainers.inc        |  1 +
 .../python/python3-libarchive-c_2.9.bb          | 17 +++++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index c68e9edf634..abbdb74a0d3 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
 RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
+RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
 RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
new file mode 100644
index 00000000000..4983ae527a1
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
@@ -0,0 +1,17 @@
+SUMMARY = "Python interface to libarchive"
+DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
+    dynamically load and access the C library."
+HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
+LICENSE = "CC0-1.0"
+LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
+
+PYPI_PACKAGE = "libarchive-c"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
+SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
+
+RDEPENDS_${PN} += "libarchive"
+
+BBCLASSEXTEND = "native"
-- 
2.23.0

[PATCH 2/4] python: Add magic recipe

[Joshua Watt]

The python-magic module is used by diffoscope tool to make
build comparisons.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/distro/include/maintainers.inc      |  1 +
 .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index abbdb74a0d3..171a43615e3 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -580,6 +580,7 @@ RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kr
 RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
+RECIPE_MAINTAINER_pn-python3-magic = "Joshua Watt <JPEWhacker@gmail.com>"
 RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python3-magic_0.4.15.bb b/meta/recipes-devtools/python/python3-magic_0.4.15.bb
new file mode 100644
index 00000000000..698016ba4c2
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-magic_0.4.15.bb
@@ -0,0 +1,19 @@
+SUMMARY = "File type identification using libmagic"
+DESCRIPTION = "This module uses ctypes to access the libmagic file type \
+    identification library. It makes use of the local magic database and supports \
+    both textual and MIME-type output."
+HOMEPAGE = "http://github.com/ahupp/python-magic"
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88"
+
+PYPI_PACKAGE = "python-magic"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "e384c95a47218f66c6501cd6dd45ff59"
+SRC_URI[sha256sum] = "f3765c0f582d2dfc72c15f3b5a82aecfae9498bd29ca840d72f37d7bd38bfcd5"
+
+RDEPENDS_${PN} += "file"
+
+BBCLASSEXTEND = "native"
-- 
2.23.0

[PATCH 3/4] recipes-support: Add diffoscope recipe

[Joshua Watt]

Diffoscope is the universal diff tool, capable of comparing many
different formats.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/distro/include/maintainers.inc          |  1 +
 meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index 171a43615e3..4a267faabce 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <nathan@nathanrossi.com>"
 RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <mark.hatle@windriver.com>"
 RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <alex.kanavin@gmail.com>"
 RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
+RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
 RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_136.bb
new file mode 100644
index 00000000000..a00f281deb0
--- /dev/null
+++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
@@ -0,0 +1,15 @@
+SUMMARY = "in-depth comparison of files, archives, and directories"
+HOMEPAGE = "https://diffoscope.org/"
+LICENSE = "GPL-3.0+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+
+PYPI_PACKAGE = "diffoscope"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
+SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+
+RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
+
+BBCLASSEXTEND = "native"
-- 
2.23.0

[PATCH 4/4] oeqa: reproducible: Run diffoscope on saved output

[Joshua Watt]

If there are differing packages and they are being saved for review,
automatically run diffoscope on them and include the output in the saved
output. The output is currently done in HTML format since these are
typically published on a webpage by the autobuilder.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 24 ++++++++++++++------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index f6433c9a02b..d3fd8c392b8 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -1,7 +1,7 @@
 #
 # SPDX-License-Identifier: MIT
 #
-# Copyright 2019 by Garmin Ltd. or its subsidiaries
+# Copyright 2019-2020 by Garmin Ltd. or its subsidiaries
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars
@@ -167,10 +167,16 @@ class ReproducibleTests(OESelftestTestCase):
         return d
 
     def test_reproducible_builds(self):
+        def strip_topdir(s):
+            if s.startswith(self.topdir):
+                return s[len(self.topdir):]
+            return s
+
         # Build native utilities
         self.write_config('')
-        bitbake("diffutils-native -c addto_recipe_sysroot")
+        bitbake("diffoscope-native diffutils-native -c addto_recipe_sysroot")
         diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native")
+        diffoscope_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffoscope-native")
 
         if self.save_results:
             os.makedirs(self.save_results, exist_ok=True)
@@ -206,18 +212,22 @@ class ReproducibleTests(OESelftestTestCase):
 
                 if self.save_results:
                     for d in result.different:
-                        self.copy_file(d.reference, '/'.join([save_dir, d.reference]))
-                        self.copy_file(d.test, '/'.join([save_dir, d.test]))
+                        self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
+                        self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
 
                 if result.missing or result.different:
                     fails.append("The following %s packages are missing or different: %s" %
                             (c, '\n'.join(r.test for r in (result.missing + result.different))))
 
-        if fails:
-            self.fail('\n'.join(fails))
-
         # Clean up empty directories
         if self.save_results:
             if not os.listdir(save_dir):
                 os.rmdir(save_dir)
+            else:
+                self.logger.info('Running diffoscope')
+                runCmd(['diffoscope', '--no-default-limits', '--exclude-directory-metadata', '--html-dir', 'diff-html', 'reproducibleA', 'reproducibleB'],
+                        native_sysroot=diffoscope_sysroot, ignore_status=True, cwd=os.path.join(save_dir, 'packages'))
+
+        if fails:
+            self.fail('\n'.join(fails))
 
-- 
2.23.0

Re: [PATCH 0/4] oeqa: Run diffoscop on saved output

[akuster808]

On 2/11/20 7:14 PM, Joshua Watt wrote:
> Adds recipes to build the diffoscope tool and run it when the OEQA
> reproducible build test saves output to a local path. This should make
> it much easier to debug reproducible build issues from the autobuilder,
> since the published output can be easily viewed on the website.\

Thanks
>
> Joshua Watt (4):
>   python: Add libarchive-c recipe
>   python: Add magic recipe
>   recipes-support: Add diffoscope recipe
>   oeqa: reproducible: Run diffoscope on saved output
>
>  meta/conf/distro/include/maintainers.inc      |  3 +++
>  meta/lib/oeqa/selftest/cases/reproducible.py  | 24 +++++++++++++------
>  .../python/python3-libarchive-c_2.9.bb        | 17 +++++++++++++
>  .../python/python3-magic_0.4.15.bb            | 19 +++++++++++++++
>  .../diffoscope/diffoscope_136.bb              | 15 ++++++++++++
>  5 files changed, 71 insertions(+), 7 deletions(-)
>  create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
>  create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
>  create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
>

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

[Khem Raj]

Hi Joshua

I think it needs to be ignored on musl

15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
/mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/diffoscope_136.bb
RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host
aarch64-yoe-linux-musl (not in COMPATIBLE_HOST)

On Tue, Feb 11, 2020 at 7:15 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>
> Diffoscope is the universal diff tool, capable of comparing many
> different formats.
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/conf/distro/include/maintainers.inc          |  1 +
>  meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
>  2 files changed, 16 insertions(+)
>  create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
>
> diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
> index 171a43615e3..4a267faabce 100644
> --- a/meta/conf/distro/include/maintainers.inc
> +++ b/meta/conf/distro/include/maintainers.inc
> @@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <nathan@nathanrossi.com>"
>  RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <mark.hatle@windriver.com>"
>  RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <alex.kanavin@gmail.com>"
>  RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
> +RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
>  RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
>  RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
>  RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
> diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_136.bb
> new file mode 100644
> index 00000000000..a00f281deb0
> --- /dev/null
> +++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
> @@ -0,0 +1,15 @@
> +SUMMARY = "in-depth comparison of files, archives, and directories"
> +HOMEPAGE = "https://diffoscope.org/"
> +LICENSE = "GPL-3.0+"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
> +
> +PYPI_PACKAGE = "diffoscope"
> +
> +inherit pypi setuptools3
> +
> +SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
> +SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
> +
> +RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
> +
> +BBCLASSEXTEND = "native"
> --
> 2.23.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:

> Hi Joshua
>
> I think it needs to be ignored on musl
>
> 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
>
> /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/
> diffoscope_136.bb
> RDEPENDS on or otherwise requires it)
> 15:37:23 squashfs-tools was skipped: incompatible with host
> aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
>

Would an RRECOMMENDS fix it?


> On Tue, Feb 11, 2020 at 7:15 PM Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > Diffoscope is the universal diff tool, capable of comparing many
> > different formats.
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> >  meta/conf/distro/include/maintainers.inc          |  1 +
> >  meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
> >  2 files changed, 16 insertions(+)
> >  create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> b/meta/conf/distro/include/maintainers.inc
> > index 171a43615e3..4a267faabce 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <
> nathan@nathanrossi.com>"
> >  RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <
> mark.hatle@windriver.com>"
> >  RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <
> alex.kanavin@gmail.com>"
> >  RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
> > +RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
> >  RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
> >  RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
> >  RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
> > diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb
> b/meta/recipes-support/diffoscope/diffoscope_136.bb
> > new file mode 100644
> > index 00000000000..a00f281deb0
> > --- /dev/null
> > +++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
> > @@ -0,0 +1,15 @@
> > +SUMMARY = "in-depth comparison of files, archives, and directories"
> > +HOMEPAGE = "https://diffoscope.org/"
> > +LICENSE = "GPL-3.0+"
> > +LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
> > +
> > +PYPI_PACKAGE = "diffoscope"
> > +
> > +inherit pypi setuptools3
> > +
> > +SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
> > +SRC_URI[sha256sum] =
> "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
> > +
> > +RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c
> python3-magic"
> > +
> > +BBCLASSEXTEND = "native"
> > --
> > 2.23.0
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 5639 bytes --]

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

[Richard Purdie]

On Fri, 2020-02-14 at 07:15 -0600, Joshua Watt wrote:
> 
> 
> On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:
> > Hi Joshua
> > 
> > I think it needs to be ignored on musl
> > 
> > 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
> > /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-
> > core/meta/recipes-support/diffoscope/diffoscope_136.bb
> > RDEPENDS on or otherwise requires it)
> > 15:37:23 squashfs-tools was skipped: incompatible with host
> > aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
> 
> Would an RRECOMMENDS fix it?

I merged your patch and one of mine to follow it adding a
COMPATIBLE_HOST entry.

I see Khem is fixing some other software so we may end up not needing
it but we can remove later.

Cheers,

Richard

Re: [PATCH 3/4] recipes-support: Add diffoscope recipe

[Khem Raj]

On Fri, Feb 14, 2020 at 6:18 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2020-02-14 at 07:15 -0600, Joshua Watt wrote:
> >
> >
> > On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:
> > > Hi Joshua
> > >
> > > I think it needs to be ignored on musl
> > >
> > > 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
> > > /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-
> > > core/meta/recipes-support/diffoscope/diffoscope_136.bb
> > > RDEPENDS on or otherwise requires it)
> > > 15:37:23 squashfs-tools was skipped: incompatible with host
> > > aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
> >
> > Would an RRECOMMENDS fix it?
>
> I merged your patch and one of mine to follow it adding a
> COMPATIBLE_HOST entry.
>
> I see Khem is fixing some other software so we may end up not needing
> it but we can remove later.
>

right, we were unnecessarily masking squashfs-tools for musl

> Cheers,
>
> Richard
>

Re: [PATCH 0/4] oeqa: Run diffoscop on saved output

[Richard Purdie]

On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> Adds recipes to build the diffoscope tool and run it when the OEQA
> reproducible build test saves output to a local path. This should
> make
> it much easier to debug reproducible build issues from the
> autobuilder,
> since the published output can be easily viewed on the website.
> 
> Joshua Watt (4):
>   python: Add libarchive-c recipe
>   python: Add magic recipe
>   recipes-support: Add diffoscope recipe
>   oeqa: reproducible: Run diffoscope on saved output

Thanks!

The first production use:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/

:)

I am a bit puzzled/concerned about how this hasn't been removed from
the system yet as it should have been, unless its the hashequiv problem
with timestamps continuing to cause problems. Need to fix my patch...

Cheers,

Richard

Re: [PATCH 0/4] oeqa: Run diffoscop on saved output

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

On Sat, Feb 15, 2020, 5:07 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> > Adds recipes to build the diffoscope tool and run it when the OEQA
> > reproducible build test saves output to a local path. This should
> > make
> > it much easier to debug reproducible build issues from the
> > autobuilder,
> > since the published output can be easily viewed on the website.
> >
> > Joshua Watt (4):
> >   python: Add libarchive-c recipe
> >   python: Add magic recipe
> >   recipes-support: Add diffoscope recipe
> >   oeqa: reproducible: Run diffoscope on saved output
>
> Thanks!
>
> The first production use:
>
>
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
>
> :)
>

That's great!


> I am a bit puzzled/concerned about how this hasn't been removed from
> the system yet as it should have been, unless its the hashequiv problem
> with timestamps continuing to cause problems. Need to fix my patch...
>

It looks like a directory ordering issue to me, since the timestamps are
the same. One way to really flush these out would be to use disorderfs (
https://salsa.debian.org/reproducible-builds/disorderfs) which permutates
the order in which entries are listed in a directory, but I think we should
try to get "wider" coverage (more recipes) before we try that.


> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 2614 bytes --]

Re: [PATCH 0/4] oeqa: Run diffoscop on saved output

[Richard Purdie]

On Sun, 2020-02-16 at 12:27 -0600, Joshua Watt wrote:
> On Sat, Feb 15, 2020, 5:07 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> > > Adds recipes to build the diffoscope tool and run it when the OEQA
> > > reproducible build test saves output to a local path. This should
> > > make
> > > it much easier to debug reproducible build issues from the
> > > autobuilder,
> > > since the published output can be easily viewed on the website.
> > > 
> > > Joshua Watt (4):
> > >   python: Add libarchive-c recipe
> > >   python: Add magic recipe
> > >   recipes-support: Add diffoscope recipe
> > >   oeqa: reproducible: Run diffoscope on saved output
> > 
> > Thanks!
> > 
> > The first production use:
> > 
> > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
> > 
> > :)
> 
> That's great!
> 
> > I am a bit puzzled/concerned about how this hasn't been removed
> > from
> > the system yet as it should have been, unless its the hashequiv
> > problem
> > with timestamps continuing to cause problems. Need to fix my
> > patch...
> 
> It looks like a directory ordering issue to me, since the timestamps
> are the same. One way to really flush these out would be to use
> disorderfs (https://salsa.debian.org/reproducible-builds/disorderfs)
> which permutates the order in which entries are listed in a
> directory, but I think we should try to get "wider" coverage (more
> recipes) before we try that.

Its more complicated unfortunately. We should have fixed the directory
order issue with:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=09c6a41b751d080f0c12fc4172a31d1dbe760b0b

However issues which shows signs of that bug still persisted in the
reproducibility test results.

I theorised that since opkg-utils is in ABISAFE, the change wasn't
accounted for. I therefore merged:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5fd3eb4bda651bddf4e4dc76d456333c70df4ddc

However with current master, we're seeing the above reproducible build
failure. I checked in a build and its using this sstate object for
comparison:

pub/sstate/b0/ca/sstate\:matchbox-config-gtk\:core2-64-poky-linux\:0.2\:r0\:core2-64\:3\:b0ca3b3559d37ba8de8ade8b282baa5082e612ef6590d3ff94fcebe38e9f17b9_package_write_ipk.tgz.siginfo

which is interesting in that its from 2nd Feb (predating both commits
above) and doesn't show the above change to package_ipk.bbclass.

The change to package_ipk means the task's own basehash has to have
changed which means it *has* to have rerun. Somehow hashequiv has then
marked it as equivalent.

I guess the issue may be that in some builds, this did generate output
that was equivalent and things have then raced so this "broken" sstate
object has persisted.

The question then becomes, how to we get this out the cache?! I'm not
entirely sure how we can :(

(obviously I can cleansstate a few things which I'll do now but its not
good this can't recover even through metadata changes)

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] python: Add libarchive-c recipe

[Yu, Mingli]

Hi Joshua,

I encounter below error when use the libarchive module, could you help 
to take a look?

# python3
Python 3.8.2 (default, Jun  8 2020, 06:35:32)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
 >>> import libarchive
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib64/python3.8/site-packages/libarchive/__init__.py", 
line 1, in <module>
     from .entry import ArchiveEntry
   File "/usr/lib64/python3.8/site-packages/libarchive/entry.py", line 
6, in <module>
     from . import ffi
   File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line 
109, in <module>
     errno = ffi('errno', [c_archive_p], c_int)
   File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line 96, 
in ffi
     f = getattr(libarchive, 'archive_'+name)
   File "/usr/lib64/python3.8/ctypes/__init__.py", line 386, in __getattr__
     func = self.__getitem__(name)
   File "/usr/lib64/python3.8/ctypes/__init__.py", line 391, in __getitem__
     func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: archive_errno
 >>>

Thanks,


On 2/12/20 11:14 AM, Joshua Watt wrote:
> The libarchive python module is used by diffoscope tool to
> make build comparisons.
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>   meta/conf/distro/include/maintainers.inc        |  1 +
>   .../python/python3-libarchive-c_2.9.bb          | 17 +++++++++++++++++
>   2 files changed, 18 insertions(+)
>   create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> 
> diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
> index c68e9edf634..abbdb74a0d3 100644
> --- a/meta/conf/distro/include/maintainers.inc
> +++ b/meta/conf/distro/include/maintainers.inc
> @@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
>   RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
>   RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
>   RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> +RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
>   RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
>   RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
>   RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> diff --git a/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> new file mode 100644
> index 00000000000..4983ae527a1
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> @@ -0,0 +1,17 @@
> +SUMMARY = "Python interface to libarchive"
> +DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
> +    dynamically load and access the C library."
> +HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
> +LICENSE = "CC0-1.0"
> +LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
> +
> +PYPI_PACKAGE = "libarchive-c"
> +
> +inherit pypi setuptools3
> +
> +SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
> +SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
> +
> +RDEPENDS_${PN} += "libarchive"
> +
> +BBCLASSEXTEND = "native"
>