* [PATCH 1/4] python: Add libarchive-c recipe
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
@ 2020-02-12 3:14 ` Joshua Watt
2020-06-10 3:17 ` [OE-core] " Yu, Mingli
2020-02-12 3:14 ` [PATCH 2/4] python: Add magic recipe Joshua Watt
` (4 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Joshua Watt @ 2020-02-12 3:14 UTC (permalink / raw)
To: openembedded-core
The libarchive python module is used by diffoscope tool to
make build comparisons.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/conf/distro/include/maintainers.inc | 1 +
.../python/python3-libarchive-c_2.9.bb | 17 +++++++++++++++++
2 files changed, 18 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index c68e9edf634..abbdb74a0d3 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
+RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
new file mode 100644
index 00000000000..4983ae527a1
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
@@ -0,0 +1,17 @@
+SUMMARY = "Python interface to libarchive"
+DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
+ dynamically load and access the C library."
+HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
+LICENSE = "CC0-1.0"
+LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
+
+PYPI_PACKAGE = "libarchive-c"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
+SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
+
+RDEPENDS_${PN} += "libarchive"
+
+BBCLASSEXTEND = "native"
--
2.23.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [OE-core] [PATCH 1/4] python: Add libarchive-c recipe
2020-02-12 3:14 ` [PATCH 1/4] python: Add libarchive-c recipe Joshua Watt
@ 2020-06-10 3:17 ` Yu, Mingli
0 siblings, 0 replies; 14+ messages in thread
From: Yu, Mingli @ 2020-06-10 3:17 UTC (permalink / raw)
To: Joshua Watt; +Cc: openembedded-core
Hi Joshua,
I encounter below error when use the libarchive module, could you help
to take a look?
# python3
Python 3.8.2 (default, Jun 8 2020, 06:35:32)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.8/site-packages/libarchive/__init__.py",
line 1, in <module>
from .entry import ArchiveEntry
File "/usr/lib64/python3.8/site-packages/libarchive/entry.py", line
6, in <module>
from . import ffi
File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line
109, in <module>
errno = ffi('errno', [c_archive_p], c_int)
File "/usr/lib64/python3.8/site-packages/libarchive/ffi.py", line 96,
in ffi
f = getattr(libarchive, 'archive_'+name)
File "/usr/lib64/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/usr/lib64/python3.8/ctypes/__init__.py", line 391, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: archive_errno
>>>
Thanks,
On 2/12/20 11:14 AM, Joshua Watt wrote:
> The libarchive python module is used by diffoscope tool to
> make build comparisons.
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
> meta/conf/distro/include/maintainers.inc | 1 +
> .../python/python3-libarchive-c_2.9.bb | 17 +++++++++++++++++
> 2 files changed, 18 insertions(+)
> create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
>
> diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
> index c68e9edf634..abbdb74a0d3 100644
> --- a/meta/conf/distro/include/maintainers.inc
> +++ b/meta/conf/distro/include/maintainers.inc
> @@ -579,6 +579,7 @@ RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr
> RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> +RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
> RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
> diff --git a/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> new file mode 100644
> index 00000000000..4983ae527a1
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> @@ -0,0 +1,17 @@
> +SUMMARY = "Python interface to libarchive"
> +DESCRIPTION = "A Python interface to libarchive. It uses the standard ctypes module to \
> + dynamically load and access the C library."
> +HOMEPAGE = "https://github.com/Changaco/python-libarchive-c"
> +LICENSE = "CC0-1.0"
> +LIC_FILES_CHKSUM = "file://LICENSE.md;md5=bcab380227a83bc147350b40a81e6ffc"
> +
> +PYPI_PACKAGE = "libarchive-c"
> +
> +inherit pypi setuptools3
> +
> +SRC_URI[md5sum] = "083bd2cb0043c1e22a52cb9a05e31532"
> +SRC_URI[sha256sum] = "9919344cec203f5db6596a29b5bc26b07ba9662925a05e24980b84709232ef60"
> +
> +RDEPENDS_${PN} += "libarchive"
> +
> +BBCLASSEXTEND = "native"
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 2/4] python: Add magic recipe
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
2020-02-12 3:14 ` [PATCH 1/4] python: Add libarchive-c recipe Joshua Watt
@ 2020-02-12 3:14 ` Joshua Watt
2020-02-12 3:14 ` [PATCH 3/4] recipes-support: Add diffoscope recipe Joshua Watt
` (3 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Joshua Watt @ 2020-02-12 3:14 UTC (permalink / raw)
To: openembedded-core
The python-magic module is used by diffoscope tool to make
build comparisons.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/conf/distro/include/maintainers.inc | 1 +
.../python/python3-magic_0.4.15.bb | 19 +++++++++++++++++++
2 files changed, 20 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index abbdb74a0d3..171a43615e3 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -580,6 +580,7 @@ RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kr
RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-iniparse = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-libarchive-c = "Joshua Watt <JPEWhacker@gmail.com>"
+RECIPE_MAINTAINER_pn-python3-magic = "Joshua Watt <JPEWhacker@gmail.com>"
RECIPE_MAINTAINER_pn-python3-mako = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-nose = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python3-magic_0.4.15.bb b/meta/recipes-devtools/python/python3-magic_0.4.15.bb
new file mode 100644
index 00000000000..698016ba4c2
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-magic_0.4.15.bb
@@ -0,0 +1,19 @@
+SUMMARY = "File type identification using libmagic"
+DESCRIPTION = "This module uses ctypes to access the libmagic file type \
+ identification library. It makes use of the local magic database and supports \
+ both textual and MIME-type output."
+HOMEPAGE = "http://github.com/ahupp/python-magic"
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88"
+
+PYPI_PACKAGE = "python-magic"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "e384c95a47218f66c6501cd6dd45ff59"
+SRC_URI[sha256sum] = "f3765c0f582d2dfc72c15f3b5a82aecfae9498bd29ca840d72f37d7bd38bfcd5"
+
+RDEPENDS_${PN} += "file"
+
+BBCLASSEXTEND = "native"
--
2.23.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 3/4] recipes-support: Add diffoscope recipe
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
2020-02-12 3:14 ` [PATCH 1/4] python: Add libarchive-c recipe Joshua Watt
2020-02-12 3:14 ` [PATCH 2/4] python: Add magic recipe Joshua Watt
@ 2020-02-12 3:14 ` Joshua Watt
2020-02-14 6:34 ` Khem Raj
2020-02-12 3:14 ` [PATCH 4/4] oeqa: reproducible: Run diffoscope on saved output Joshua Watt
` (2 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Joshua Watt @ 2020-02-12 3:14 UTC (permalink / raw)
To: openembedded-core
Diffoscope is the universal diff tool, capable of comparing many
different formats.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/conf/distro/include/maintainers.inc | 1 +
meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
2 files changed, 16 insertions(+)
create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index 171a43615e3..4a267faabce 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <nathan@nathanrossi.com>"
RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <mark.hatle@windriver.com>"
RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <alex.kanavin@gmail.com>"
RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
+RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_136.bb
new file mode 100644
index 00000000000..a00f281deb0
--- /dev/null
+++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
@@ -0,0 +1,15 @@
+SUMMARY = "in-depth comparison of files, archives, and directories"
+HOMEPAGE = "https://diffoscope.org/"
+LICENSE = "GPL-3.0+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+
+PYPI_PACKAGE = "diffoscope"
+
+inherit pypi setuptools3
+
+SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
+SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
+
+RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
+
+BBCLASSEXTEND = "native"
--
2.23.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 3/4] recipes-support: Add diffoscope recipe
2020-02-12 3:14 ` [PATCH 3/4] recipes-support: Add diffoscope recipe Joshua Watt
@ 2020-02-14 6:34 ` Khem Raj
2020-02-14 13:15 ` Joshua Watt
0 siblings, 1 reply; 14+ messages in thread
From: Khem Raj @ 2020-02-14 6:34 UTC (permalink / raw)
To: Joshua Watt; +Cc: Patches and discussions about the oe-core layer
Hi Joshua
I think it needs to be ignored on musl
15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
/mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/diffoscope_136.bb
RDEPENDS on or otherwise requires it)
15:37:23 squashfs-tools was skipped: incompatible with host
aarch64-yoe-linux-musl (not in COMPATIBLE_HOST)
On Tue, Feb 11, 2020 at 7:15 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>
> Diffoscope is the universal diff tool, capable of comparing many
> different formats.
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
> meta/conf/distro/include/maintainers.inc | 1 +
> meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
> 2 files changed, 16 insertions(+)
> create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
>
> diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
> index 171a43615e3..4a267faabce 100644
> --- a/meta/conf/distro/include/maintainers.inc
> +++ b/meta/conf/distro/include/maintainers.inc
> @@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <nathan@nathanrossi.com>"
> RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <mark.hatle@windriver.com>"
> RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <alex.kanavin@gmail.com>"
> RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
> +RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
> RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
> RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
> RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
> diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb b/meta/recipes-support/diffoscope/diffoscope_136.bb
> new file mode 100644
> index 00000000000..a00f281deb0
> --- /dev/null
> +++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
> @@ -0,0 +1,15 @@
> +SUMMARY = "in-depth comparison of files, archives, and directories"
> +HOMEPAGE = "https://diffoscope.org/"
> +LICENSE = "GPL-3.0+"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
> +
> +PYPI_PACKAGE = "diffoscope"
> +
> +inherit pypi setuptools3
> +
> +SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
> +SRC_URI[sha256sum] = "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
> +
> +RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c python3-magic"
> +
> +BBCLASSEXTEND = "native"
> --
> 2.23.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/4] recipes-support: Add diffoscope recipe
2020-02-14 6:34 ` Khem Raj
@ 2020-02-14 13:15 ` Joshua Watt
2020-02-14 14:18 ` Richard Purdie
0 siblings, 1 reply; 14+ messages in thread
From: Joshua Watt @ 2020-02-14 13:15 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]
On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:
> Hi Joshua
>
> I think it needs to be ignored on musl
>
> 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
>
> /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-core/meta/recipes-support/diffoscope/
> diffoscope_136.bb
> RDEPENDS on or otherwise requires it)
> 15:37:23 squashfs-tools was skipped: incompatible with host
> aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
>
Would an RRECOMMENDS fix it?
> On Tue, Feb 11, 2020 at 7:15 PM Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > Diffoscope is the universal diff tool, capable of comparing many
> > different formats.
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> > meta/conf/distro/include/maintainers.inc | 1 +
> > meta/recipes-support/diffoscope/diffoscope_136.bb | 15 +++++++++++++++
> > 2 files changed, 16 insertions(+)
> > create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
> >
> > diff --git a/meta/conf/distro/include/maintainers.inc
> b/meta/conf/distro/include/maintainers.inc
> > index 171a43615e3..4a267faabce 100644
> > --- a/meta/conf/distro/include/maintainers.inc
> > +++ b/meta/conf/distro/include/maintainers.inc
> > @@ -146,6 +146,7 @@ RECIPE_MAINTAINER_pn-dejagnu = "Nathan Rossi <
> nathan@nathanrossi.com>"
> > RECIPE_MAINTAINER_pn-depmodwrapper-cross = "Mark Hatle <
> mark.hatle@windriver.com>"
> > RECIPE_MAINTAINER_pn-desktop-file-utils = "Alexander Kanavin <
> alex.kanavin@gmail.com>"
> > RECIPE_MAINTAINER_pn-dhcp = "Hongxu Jia <hongxu.jia@windriver.com>"
> > +RECIPE_MAINTAINER_pn-diffoscope = "Joshua Watt <JPEWhacker@gmail.com>"
> > RECIPE_MAINTAINER_pn-diffstat = "Chen Qi <Qi.Chen@windriver.com>"
> > RECIPE_MAINTAINER_pn-diffutils = "Chen Qi <Qi.Chen@windriver.com>"
> > RECIPE_MAINTAINER_pn-distcc = "Hongxu Jia <hongxu.jia@windriver.com>"
> > diff --git a/meta/recipes-support/diffoscope/diffoscope_136.bb
> b/meta/recipes-support/diffoscope/diffoscope_136.bb
> > new file mode 100644
> > index 00000000000..a00f281deb0
> > --- /dev/null
> > +++ b/meta/recipes-support/diffoscope/diffoscope_136.bb
> > @@ -0,0 +1,15 @@
> > +SUMMARY = "in-depth comparison of files, archives, and directories"
> > +HOMEPAGE = "https://diffoscope.org/"
> > +LICENSE = "GPL-3.0+"
> > +LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
> > +
> > +PYPI_PACKAGE = "diffoscope"
> > +
> > +inherit pypi setuptools3
> > +
> > +SRC_URI[md5sum] = "c84d8d308a40176ba2f5dc4abdbf6f73"
> > +SRC_URI[sha256sum] =
> "0d6486d6eb6e0445ba21fee2e8bdd3a366ce786bfac98e00e5a95038b7815f15"
> > +
> > +RDEPENDS_${PN} += "binutils vim squashfs-tools python3-libarchive-c
> python3-magic"
> > +
> > +BBCLASSEXTEND = "native"
> > --
> > 2.23.0
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
[-- Attachment #2: Type: text/html, Size: 5639 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/4] recipes-support: Add diffoscope recipe
2020-02-14 13:15 ` Joshua Watt
@ 2020-02-14 14:18 ` Richard Purdie
2020-02-14 14:31 ` Khem Raj
0 siblings, 1 reply; 14+ messages in thread
From: Richard Purdie @ 2020-02-14 14:18 UTC (permalink / raw)
To: Joshua Watt, Khem Raj; +Cc: Patches and discussions about the oe-core layer
On Fri, 2020-02-14 at 07:15 -0600, Joshua Watt wrote:
>
>
> On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:
> > Hi Joshua
> >
> > I think it needs to be ignored on musl
> >
> > 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
> > /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-
> > core/meta/recipes-support/diffoscope/diffoscope_136.bb
> > RDEPENDS on or otherwise requires it)
> > 15:37:23 squashfs-tools was skipped: incompatible with host
> > aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
>
> Would an RRECOMMENDS fix it?
I merged your patch and one of mine to follow it adding a
COMPATIBLE_HOST entry.
I see Khem is fixing some other software so we may end up not needing
it but we can remove later.
Cheers,
Richard
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/4] recipes-support: Add diffoscope recipe
2020-02-14 14:18 ` Richard Purdie
@ 2020-02-14 14:31 ` Khem Raj
0 siblings, 0 replies; 14+ messages in thread
From: Khem Raj @ 2020-02-14 14:31 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
On Fri, Feb 14, 2020 at 6:18 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2020-02-14 at 07:15 -0600, Joshua Watt wrote:
> >
> >
> > On Fri, Feb 14, 2020, 12:34 AM Khem Raj <raj.khem@gmail.com> wrote:
> > > Hi Joshua
> > >
> > > I think it needs to be ignored on musl
> > >
> > > 15:37:23 ERROR: Nothing RPROVIDES 'squashfs-tools' (but
> > > /mnt/jenkins/workspace/Yocto-world-musl/sources/openembedded-
> > > core/meta/recipes-support/diffoscope/diffoscope_136.bb
> > > RDEPENDS on or otherwise requires it)
> > > 15:37:23 squashfs-tools was skipped: incompatible with host
> > > aarch64-yoe-linux-musl (not in COMPATIBLE_HOST
> >
> > Would an RRECOMMENDS fix it?
>
> I merged your patch and one of mine to follow it adding a
> COMPATIBLE_HOST entry.
>
> I see Khem is fixing some other software so we may end up not needing
> it but we can remove later.
>
right, we were unnecessarily masking squashfs-tools for musl
> Cheers,
>
> Richard
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 4/4] oeqa: reproducible: Run diffoscope on saved output
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
` (2 preceding siblings ...)
2020-02-12 3:14 ` [PATCH 3/4] recipes-support: Add diffoscope recipe Joshua Watt
@ 2020-02-12 3:14 ` Joshua Watt
2020-02-12 3:28 ` [PATCH 0/4] oeqa: Run diffoscop " akuster808
2020-02-15 11:07 ` Richard Purdie
5 siblings, 0 replies; 14+ messages in thread
From: Joshua Watt @ 2020-02-12 3:14 UTC (permalink / raw)
To: openembedded-core
If there are differing packages and they are being saved for review,
automatically run diffoscope on them and include the output in the saved
output. The output is currently done in HTML format since these are
typically published on a webpage by the autobuilder.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 24 ++++++++++++++------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index f6433c9a02b..d3fd8c392b8 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -1,7 +1,7 @@
#
# SPDX-License-Identifier: MIT
#
-# Copyright 2019 by Garmin Ltd. or its subsidiaries
+# Copyright 2019-2020 by Garmin Ltd. or its subsidiaries
from oeqa.selftest.case import OESelftestTestCase
from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars
@@ -167,10 +167,16 @@ class ReproducibleTests(OESelftestTestCase):
return d
def test_reproducible_builds(self):
+ def strip_topdir(s):
+ if s.startswith(self.topdir):
+ return s[len(self.topdir):]
+ return s
+
# Build native utilities
self.write_config('')
- bitbake("diffutils-native -c addto_recipe_sysroot")
+ bitbake("diffoscope-native diffutils-native -c addto_recipe_sysroot")
diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native")
+ diffoscope_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffoscope-native")
if self.save_results:
os.makedirs(self.save_results, exist_ok=True)
@@ -206,18 +212,22 @@ class ReproducibleTests(OESelftestTestCase):
if self.save_results:
for d in result.different:
- self.copy_file(d.reference, '/'.join([save_dir, d.reference]))
- self.copy_file(d.test, '/'.join([save_dir, d.test]))
+ self.copy_file(d.reference, '/'.join([save_dir, 'packages', strip_topdir(d.reference)]))
+ self.copy_file(d.test, '/'.join([save_dir, 'packages', strip_topdir(d.test)]))
if result.missing or result.different:
fails.append("The following %s packages are missing or different: %s" %
(c, '\n'.join(r.test for r in (result.missing + result.different))))
- if fails:
- self.fail('\n'.join(fails))
-
# Clean up empty directories
if self.save_results:
if not os.listdir(save_dir):
os.rmdir(save_dir)
+ else:
+ self.logger.info('Running diffoscope')
+ runCmd(['diffoscope', '--no-default-limits', '--exclude-directory-metadata', '--html-dir', 'diff-html', 'reproducibleA', 'reproducibleB'],
+ native_sysroot=diffoscope_sysroot, ignore_status=True, cwd=os.path.join(save_dir, 'packages'))
+
+ if fails:
+ self.fail('\n'.join(fails))
--
2.23.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 0/4] oeqa: Run diffoscop on saved output
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
` (3 preceding siblings ...)
2020-02-12 3:14 ` [PATCH 4/4] oeqa: reproducible: Run diffoscope on saved output Joshua Watt
@ 2020-02-12 3:28 ` akuster808
2020-02-15 11:07 ` Richard Purdie
5 siblings, 0 replies; 14+ messages in thread
From: akuster808 @ 2020-02-12 3:28 UTC (permalink / raw)
To: Joshua Watt, openembedded-core
On 2/11/20 7:14 PM, Joshua Watt wrote:
> Adds recipes to build the diffoscope tool and run it when the OEQA
> reproducible build test saves output to a local path. This should make
> it much easier to debug reproducible build issues from the autobuilder,
> since the published output can be easily viewed on the website.\
Thanks
>
> Joshua Watt (4):
> python: Add libarchive-c recipe
> python: Add magic recipe
> recipes-support: Add diffoscope recipe
> oeqa: reproducible: Run diffoscope on saved output
>
> meta/conf/distro/include/maintainers.inc | 3 +++
> meta/lib/oeqa/selftest/cases/reproducible.py | 24 +++++++++++++------
> .../python/python3-libarchive-c_2.9.bb | 17 +++++++++++++
> .../python/python3-magic_0.4.15.bb | 19 +++++++++++++++
> .../diffoscope/diffoscope_136.bb | 15 ++++++++++++
> 5 files changed, 71 insertions(+), 7 deletions(-)
> create mode 100644 meta/recipes-devtools/python/python3-libarchive-c_2.9.bb
> create mode 100644 meta/recipes-devtools/python/python3-magic_0.4.15.bb
> create mode 100644 meta/recipes-support/diffoscope/diffoscope_136.bb
>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 0/4] oeqa: Run diffoscop on saved output
2020-02-12 3:14 [PATCH 0/4] oeqa: Run diffoscop on saved output Joshua Watt
` (4 preceding siblings ...)
2020-02-12 3:28 ` [PATCH 0/4] oeqa: Run diffoscop " akuster808
@ 2020-02-15 11:07 ` Richard Purdie
2020-02-16 18:27 ` Joshua Watt
5 siblings, 1 reply; 14+ messages in thread
From: Richard Purdie @ 2020-02-15 11:07 UTC (permalink / raw)
To: Joshua Watt, openembedded-core
On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> Adds recipes to build the diffoscope tool and run it when the OEQA
> reproducible build test saves output to a local path. This should
> make
> it much easier to debug reproducible build issues from the
> autobuilder,
> since the published output can be easily viewed on the website.
>
> Joshua Watt (4):
> python: Add libarchive-c recipe
> python: Add magic recipe
> recipes-support: Add diffoscope recipe
> oeqa: reproducible: Run diffoscope on saved output
Thanks!
The first production use:
https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
:)
I am a bit puzzled/concerned about how this hasn't been removed from
the system yet as it should have been, unless its the hashequiv problem
with timestamps continuing to cause problems. Need to fix my patch...
Cheers,
Richard
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 0/4] oeqa: Run diffoscop on saved output
2020-02-15 11:07 ` Richard Purdie
@ 2020-02-16 18:27 ` Joshua Watt
2020-02-17 11:02 ` Richard Purdie
0 siblings, 1 reply; 14+ messages in thread
From: Joshua Watt @ 2020-02-16 18:27 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]
On Sat, Feb 15, 2020, 5:07 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> > Adds recipes to build the diffoscope tool and run it when the OEQA
> > reproducible build test saves output to a local path. This should
> > make
> > it much easier to debug reproducible build issues from the
> > autobuilder,
> > since the published output can be easily viewed on the website.
> >
> > Joshua Watt (4):
> > python: Add libarchive-c recipe
> > python: Add magic recipe
> > recipes-support: Add diffoscope recipe
> > oeqa: reproducible: Run diffoscope on saved output
>
> Thanks!
>
> The first production use:
>
>
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
>
> :)
>
That's great!
> I am a bit puzzled/concerned about how this hasn't been removed from
> the system yet as it should have been, unless its the hashequiv problem
> with timestamps continuing to cause problems. Need to fix my patch...
>
It looks like a directory ordering issue to me, since the timestamps are
the same. One way to really flush these out would be to use disorderfs (
https://salsa.debian.org/reproducible-builds/disorderfs) which permutates
the order in which entries are listed in a directory, but I think we should
try to get "wider" coverage (more recipes) before we try that.
> Cheers,
>
> Richard
>
>
[-- Attachment #2: Type: text/html, Size: 2614 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/4] oeqa: Run diffoscop on saved output
2020-02-16 18:27 ` Joshua Watt
@ 2020-02-17 11:02 ` Richard Purdie
0 siblings, 0 replies; 14+ messages in thread
From: Richard Purdie @ 2020-02-17 11:02 UTC (permalink / raw)
To: Joshua Watt; +Cc: OE-core
On Sun, 2020-02-16 at 12:27 -0600, Joshua Watt wrote:
> On Sat, Feb 15, 2020, 5:07 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> > > Adds recipes to build the diffoscope tool and run it when the OEQA
> > > reproducible build test saves output to a local path. This should
> > > make
> > > it much easier to debug reproducible build issues from the
> > > autobuilder,
> > > since the published output can be easily viewed on the website.
> > >
> > > Joshua Watt (4):
> > > python: Add libarchive-c recipe
> > > python: Add magic recipe
> > > recipes-support: Add diffoscope recipe
> > > oeqa: reproducible: Run diffoscope on saved output
> >
> > Thanks!
> >
> > The first production use:
> >
> > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
> >
> > :)
>
> That's great!
>
> > I am a bit puzzled/concerned about how this hasn't been removed
> > from
> > the system yet as it should have been, unless its the hashequiv
> > problem
> > with timestamps continuing to cause problems. Need to fix my
> > patch...
>
> It looks like a directory ordering issue to me, since the timestamps
> are the same. One way to really flush these out would be to use
> disorderfs (https://salsa.debian.org/reproducible-builds/disorderfs)
> which permutates the order in which entries are listed in a
> directory, but I think we should try to get "wider" coverage (more
> recipes) before we try that.
Its more complicated unfortunately. We should have fixed the directory
order issue with:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=09c6a41b751d080f0c12fc4172a31d1dbe760b0b
However issues which shows signs of that bug still persisted in the
reproducibility test results.
I theorised that since opkg-utils is in ABISAFE, the change wasn't
accounted for. I therefore merged:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5fd3eb4bda651bddf4e4dc76d456333c70df4ddc
However with current master, we're seeing the above reproducible build
failure. I checked in a build and its using this sstate object for
comparison:
pub/sstate/b0/ca/sstate\:matchbox-config-gtk\:core2-64-poky-linux\:0.2\:r0\:core2-64\:3\:b0ca3b3559d37ba8de8ade8b282baa5082e612ef6590d3ff94fcebe38e9f17b9_package_write_ipk.tgz.siginfo
which is interesting in that its from 2nd Feb (predating both commits
above) and doesn't show the above change to package_ipk.bbclass.
The change to package_ipk means the task's own basehash has to have
changed which means it *has* to have rerun. Somehow hashequiv has then
marked it as equivalent.
I guess the issue may be that in some builds, this did generate output
that was equivalent and things have then raced so this "broken" sstate
object has persisted.
The question then becomes, how to we get this out the cache?! I'm not
entirely sure how we can :(
(obviously I can cleansstate a few things which I'll do now but its not
good this can't recover even through metadata changes)
Cheers,
Richard
^ permalink raw reply [flat|nested] 14+ messages in thread