All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/5] opkg: Add upstream fixes for empty packages
@ 2019-11-21 20:54 Richard Purdie
  2019-11-21 20:54 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie
                   ` (4 more replies)
  0 siblings, 5 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 20:54 UTC (permalink / raw)
  To: bitbake-devel

An ipk with a zero size data.tar file caused opkg to crash with a
double free abort. Add the upstream fixes for this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../opkg/opkg/open_inner.patch                | 46 ++++++++++++++++
 .../opkg/opkg/opkg_archive.patch              | 54 +++++++++++++++++++
 meta/recipes-devtools/opkg/opkg_0.4.1.bb      |  2 +
 3 files changed, 102 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg/opkg/open_inner.patch
 create mode 100644 meta/recipes-devtools/opkg/opkg/opkg_archive.patch

diff --git a/meta/recipes-devtools/opkg/opkg/open_inner.patch b/meta/recipes-devtools/opkg/opkg/open_inner.patch
new file mode 100644
index 00000000000..278e099e3ac
--- /dev/null
+++ b/meta/recipes-devtools/opkg/opkg/open_inner.patch
@@ -0,0 +1,46 @@
+From alejandro.delcastillo@ni.com Wed Nov 20 22:35:02 2019
+From: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+To: <opkg-devel@googlegroups.com>, <richard.purdie@linuxfoundation.org>
+CC: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+Subject: [opkg][PATCH 2/2] open_inner: add support for empty payloads
+Date: Wed, 20 Nov 2019 16:34:48 -0600
+Message-ID: <20191120223448.26522-3-alejandro.delcastillo@ni.com>
+X-Mailer: git-send-email 2.22.0
+In-Reply-To: <20191120223448.26522-1-alejandro.delcastillo@ni.com>
+References: <20191120223448.26522-1-alejandro.delcastillo@ni.com>
+MIME-Version: 1.0
+Content-Type: text/plain
+Content-Transfer-Encoding: 8bit
+
+Support for empty compressed payloads need to be explicitly enabled on
+libarchive.
+
+Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+
+Upstream-Status: Backport
+---
+ libopkg/opkg_archive.c | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/libopkg/opkg_archive.c b/libopkg/opkg_archive.c
+index 0e9ccea..f19cece 100644
+--- a/libopkg/opkg_archive.c
++++ b/libopkg/opkg_archive.c
+@@ -618,6 +618,13 @@ static struct archive *open_inner(struct archive *outer)
+         goto err_cleanup;
+     }
+ 
++    r = archive_read_support_format_empty(inner);
++    if (r != ARCHIVE_OK) {
++        opkg_msg(ERROR, "Empty format not supported: %s\n",
++                 archive_error_string(inner));
++        goto err_cleanup;
++    }
++
+     r = archive_read_open(inner, data, NULL, inner_read, inner_close);
+     if (r != ARCHIVE_OK) {
+         opkg_msg(ERROR, "Failed to open inner archive: %s\n",
+-- 
+2.22.0
+
+
diff --git a/meta/recipes-devtools/opkg/opkg/opkg_archive.patch b/meta/recipes-devtools/opkg/opkg/opkg_archive.patch
new file mode 100644
index 00000000000..3e1ebae9530
--- /dev/null
+++ b/meta/recipes-devtools/opkg/opkg/opkg_archive.patch
@@ -0,0 +1,54 @@
+From alejandro.delcastillo@ni.com Wed Nov 20 22:35:01 2019
+Return-Path: <richard.purdie+caf_=rpurdie=rpsys.net@linuxfoundation.org>
+From: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+To: <opkg-devel@googlegroups.com>, <richard.purdie@linuxfoundation.org>
+CC: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+Subject: [opkg][PATCH 1/2] opkg_archive.c: avoid double free on uncompress
+ error
+Date: Wed, 20 Nov 2019 16:34:47 -0600
+Message-ID: <20191120223448.26522-2-alejandro.delcastillo@ni.com>
+X-Mailer: git-send-email 2.22.0
+In-Reply-To: <20191120223448.26522-1-alejandro.delcastillo@ni.com>
+References: <20191120223448.26522-1-alejandro.delcastillo@ni.com>
+MIME-Version: 1.0
+Content-Type: text/plain
+Content-Transfer-Encoding: 8bit
+
+The open-inner function calls archive_read_open. On error,
+archive_read_open calls inner_close, which also closes the outter
+archive. On error, return NULL directly to avoid double free.
+
+
+Upstream-Status: Backport
+
+Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
+---
+ libopkg/opkg_archive.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/libopkg/opkg_archive.c b/libopkg/opkg_archive.c
+index 3d87db1..0e9ccea 100644
+--- a/libopkg/opkg_archive.c
++++ b/libopkg/opkg_archive.c
+@@ -622,7 +622,7 @@ static struct archive *open_inner(struct archive *outer)
+     if (r != ARCHIVE_OK) {
+         opkg_msg(ERROR, "Failed to open inner archive: %s\n",
+                  archive_error_string(inner));
+-        goto err_cleanup;
++        return NULL;
+     }
+ 
+     return inner;
+@@ -683,7 +683,7 @@ static struct archive *extract_outer(const char *filename, const char *arname)
+ 
+     inner = open_inner(outer);
+     if (!inner)
+-        goto err_cleanup;
++        return NULL;
+ 
+     return inner;
+ 
+-- 
+2.22.0
+
+
diff --git a/meta/recipes-devtools/opkg/opkg_0.4.1.bb b/meta/recipes-devtools/opkg/opkg_0.4.1.bb
index 149ee3ca190..f0ae8b36bd1 100644
--- a/meta/recipes-devtools/opkg/opkg_0.4.1.bb
+++ b/meta/recipes-devtools/opkg/opkg_0.4.1.bb
@@ -14,6 +14,8 @@ PE = "1"
 SRC_URI = "http://downloads.yoctoproject.org/releases/${BPN}/${BPN}-${PV}.tar.gz \
            file://opkg.conf \
            file://0001-opkg_conf-create-opkg.lock-in-run-instead-of-var-run.patch \
+           file://opkg_archive.patch \
+           file://open_inner.patch \
            file://run-ptest \
 "
 
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation
  2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
@ 2019-11-21 20:54 ` Richard Purdie
  2019-11-22  0:38   ` Khem Raj
  2019-11-21 20:54 ` [PATCH 3/5] sanity: Add check for tar older than 1.28 Richard Purdie
                   ` (3 subsequent siblings)
  4 siblings, 1 reply; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 20:54 UTC (permalink / raw)
  To: bitbake-devel

opkg-build was failing on hosts where tar < 1.28 and reproducibile builds
were enabled but it was doing this silently and generating corrupted
(empty) ipk files. Add a fix for this (submitted upstream).

The fix requires bash but if you're building ipk files this shoudn't be
a problem.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../opkg-utils/opkg-utils/pipefail.patch      | 31 +++++++++++++++++++
 .../opkg-utils/opkg-utils_0.4.1.bb            |  3 ++
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
new file mode 100644
index 00000000000..55ddcc1fd20
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
@@ -0,0 +1,31 @@
+We need opkg-build to fail if for example the tar command is passed invalid 
+options. Without this, we see silently created empty packaged where data.tar
+is zero bytes in size. This creates hard to debug problems.
+
+An example is when reproducible builds are enabled and run on old hosts like
+centos7 which has tar < 1.28:
+
+Subprocess output:tar: unrecognized option '--clamp-mtime'
+Try `tar --help' or `tar --usage' for more information.
+
+Upstream-Status: Pending
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+Index: opkg-utils-0.4.1/opkg-build
+===================================================================
+--- opkg-utils-0.4.1.orig/opkg-build
++++ opkg-utils-0.4.1/opkg-build
+@@ -1,4 +1,4 @@
+-#!/bin/sh
++#!/bin/bash
+ 
+ : <<=cut
+ =head1 NAME
+@@ -12,6 +12,7 @@ opkg-build - construct an .opk from a di
+ #   Updated to work on Familiar Pre0.7rc1, with busybox tar.
+ #   Note it Requires: binutils-ar (since the busybox ar can't create)
+ set -e
++set -o pipefail
+ 
+ version=1.0
+ 
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
index cf1e4670c65..eb6c7a3a6ad 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
@@ -10,6 +10,7 @@ PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtu
 SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \
            file://0001-Switch-all-scripts-to-use-Python-3.x.patch \
            file://0001-opkg-build-clamp-mtimes-to-SOURCE_DATE_EPOCH.patch \
+           file://pipefail.patch \
 "
 UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
 
@@ -19,6 +20,8 @@ SRC_URI[sha256sum] = "9ea9efdd9fe13661ad251e3a2860c1c93045adcfaa6659c3e86d9748ec
 
 TARGET_CC_ARCH += "${LDFLAGS}"
 
+RDEPENDS_${PN} += "bash"
+
 # For native builds we use the host Python
 PYTHONRDEPS = "python3 python3-shell python3-io python3-math python3-crypt python3-logging python3-fcntl python3-pickle python3-compression python3-stringold"
 PYTHONRDEPS_class-native = ""
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 3/5] sanity: Add check for tar older than 1.28
  2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
  2019-11-21 20:54 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie
@ 2019-11-21 20:54 ` Richard Purdie
  2019-11-21 20:54 ` [PATCH 4/5] core-image-full-cmdline: Add less Richard Purdie
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 20:54 UTC (permalink / raw)
  To: bitbake-devel

Older versions break opkg-build when reproducible builds are enabled.
Rather than trying to be selective based on which features are enabled,
lets just make this a minimum version.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sanity.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index a14bf53883c..12ea14796e8 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -523,6 +523,7 @@ def check_wsl(d):
 
 # Tar version 1.24 and onwards handle overwriting symlinks correctly
 # but earlier versions do not; this needs to work properly for sstate
+# Version 1.28 is needed so opkg-build works correctly when reproducibile builds are enabled
 def check_tar_version(sanity_data):
     from distutils.version import LooseVersion
     import subprocess
@@ -532,7 +533,9 @@ def check_tar_version(sanity_data):
         return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output)
     version = result.split()[3]
     if LooseVersion(version) < LooseVersion("1.24"):
-        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar.\n"
+        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n"
+    if LooseVersion(version) < LooseVersion("1.28"):
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtoold-tarball from our last release).\n"
     return None
 
 # We use git parameters and functionality only found in 1.7.8 or later
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 4/5] core-image-full-cmdline: Add less
  2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
  2019-11-21 20:54 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie
  2019-11-21 20:54 ` [PATCH 3/5] sanity: Add check for tar older than 1.28 Richard Purdie
@ 2019-11-21 20:54 ` Richard Purdie
  2019-11-21 20:54 ` [PATCH 5/5] oeqa/selftest/sstatetests: Ensure we don't use hashequiv for sstatesigs tests Richard Purdie
  2019-11-21 21:22 ` [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
  4 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 20:54 UTC (permalink / raw)
  To: bitbake-devel

Less was coming from busybox in these images, add the full version.

[YOCTO #13630]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../packagegroups/packagegroup-core-full-cmdline.bb              | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb b/meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb
index 2d96d1ba385..15a8e6dedc3 100644
--- a/meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb
+++ b/meta/recipes-extended/packagegroups/packagegroup-core-full-cmdline.bb
@@ -81,6 +81,7 @@ RDEPENDS_packagegroup-core-full-cmdline-utils = "\
     gawk \
     gmp \
     grep \
+    less \
     makedevs \
     mc \
     mc-fish \
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 5/5] oeqa/selftest/sstatetests: Ensure we don't use hashequiv for sstatesigs tests
  2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
                   ` (2 preceding siblings ...)
  2019-11-21 20:54 ` [PATCH 4/5] core-image-full-cmdline: Add less Richard Purdie
@ 2019-11-21 20:54 ` Richard Purdie
  2019-11-21 21:22 ` [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
  4 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 20:54 UTC (permalink / raw)
  To: bitbake-devel

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/cases/sstatetests.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 2867cb78abb..6757a0ec689 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -255,6 +255,7 @@ BUILD_ARCH = "x86_64"
 BUILD_OS = "linux"
 SDKMACHINE = "x86_64"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash")
         bitbake("core-image-sato -S none")
@@ -266,6 +267,7 @@ BUILD_ARCH = "i686"
 BUILD_OS = "linux"
 SDKMACHINE = "i686"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash2")
         bitbake("core-image-sato -S none")
@@ -298,6 +300,7 @@ PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash\"
 TCLIBCAPPEND = \"\"
 NATIVELSBSTRING = \"DistroA\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash")
         bitbake("core-image-sato -S none")
@@ -305,6 +308,7 @@ NATIVELSBSTRING = \"DistroA\"
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash2\"
 TCLIBCAPPEND = \"\"
 NATIVELSBSTRING = \"DistroB\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash2")
         bitbake("core-image-sato -S none")
@@ -332,11 +336,13 @@ NATIVELSBSTRING = \"DistroB\"
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash\"
 TCLIBCAPPEND = \"\"
 MACHINE = \"qemux86-64\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """
         configB = """
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash2\"
 TCLIBCAPPEND = \"\"
 MACHINE = \"qemuarm\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """
         self.sstate_allarch_samesigs(configA, configB)
 
@@ -352,6 +358,7 @@ MACHINE = \"qemux86-64\"
 require conf/multilib.conf
 MULTILIBS = \"multilib:lib32\"
 DEFAULTTUNE_virtclass-multilib-lib32 = \"x86\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """
         configB = """
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash2\"
@@ -359,6 +366,7 @@ TCLIBCAPPEND = \"\"
 MACHINE = \"qemuarm\"
 require conf/multilib.conf
 MULTILIBS = \"\"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """
         self.sstate_allarch_samesigs(configA, configB)
 
@@ -404,6 +412,7 @@ MACHINE = \"qemux86\"
 require conf/multilib.conf
 MULTILIBS = "multilib:lib32"
 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash")
         bitbake("world meta-toolchain -S none")
@@ -414,6 +423,7 @@ MACHINE = \"qemux86copy\"
 require conf/multilib.conf
 MULTILIBS = "multilib:lib32"
 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash2")
         bitbake("world meta-toolchain -S none")
@@ -452,6 +462,7 @@ TIME = "111111"
 DATE = "20161111"
 INHERIT_remove = "buildstats-summary buildhistory uninative"
 http_proxy = ""
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash")
         self.track_for_cleanup(self.topdir + "/download1")
@@ -468,6 +479,7 @@ DATE = "20161212"
 INHERIT_remove = "uninative"
 INHERIT += "buildstats-summary buildhistory"
 http_proxy = "http://example.com/"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
 """)
         self.track_for_cleanup(self.topdir + "/tmp-sstatesamehash2")
         self.track_for_cleanup(self.topdir + "/download2")
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH 1/5] opkg: Add upstream fixes for empty packages
  2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
                   ` (3 preceding siblings ...)
  2019-11-21 20:54 ` [PATCH 5/5] oeqa/selftest/sstatetests: Ensure we don't use hashequiv for sstatesigs tests Richard Purdie
@ 2019-11-21 21:22 ` Richard Purdie
  4 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 21:22 UTC (permalink / raw)
  To: bitbake-devel

On Thu, 2019-11-21 at 20:54 +0000, Richard Purdie wrote:
> An ipk with a zero size data.tar file caused opkg to crash with a
> double free abort. Add the upstream fixes for this.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Resent to OE-Core, sorry about the noise.

Cheers,

Richard



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation
  2019-11-21 20:54 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie
@ 2019-11-22  0:38   ` Khem Raj
  2019-11-22  9:03     ` Richard Purdie
  0 siblings, 1 reply; 9+ messages in thread
From: Khem Raj @ 2019-11-22  0:38 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel

On Thu, Nov 21, 2019 at 12:54 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> opkg-build was failing on hosts where tar < 1.28 and reproducibile builds
> were enabled but it was doing this silently and generating corrupted
> (empty) ipk files. Add a fix for this (submitted upstream).
>
> The fix requires bash but if you're building ipk files this shoudn't be
> a problem.

is the problem similar to this one

http://cfajohnson.com/shell/cus-faq-2.html#Q11

does it impact build times when default shell is not bash ?

>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  .../opkg-utils/opkg-utils/pipefail.patch      | 31 +++++++++++++++++++
>  .../opkg-utils/opkg-utils_0.4.1.bb            |  3 ++
>  2 files changed, 34 insertions(+)
>  create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
>
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
> new file mode 100644
> index 00000000000..55ddcc1fd20
> --- /dev/null
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
> @@ -0,0 +1,31 @@
> +We need opkg-build to fail if for example the tar command is passed invalid
> +options. Without this, we see silently created empty packaged where data.tar
> +is zero bytes in size. This creates hard to debug problems.
> +
> +An example is when reproducible builds are enabled and run on old hosts like
> +centos7 which has tar < 1.28:
> +
> +Subprocess output:tar: unrecognized option '--clamp-mtime'
> +Try `tar --help' or `tar --usage' for more information.
> +
> +Upstream-Status: Pending
> +Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> +
> +Index: opkg-utils-0.4.1/opkg-build
> +===================================================================
> +--- opkg-utils-0.4.1.orig/opkg-build
> ++++ opkg-utils-0.4.1/opkg-build
> +@@ -1,4 +1,4 @@
> +-#!/bin/sh
> ++#!/bin/bash
> +
> + : <<=cut
> + =head1 NAME
> +@@ -12,6 +12,7 @@ opkg-build - construct an .opk from a di
> + #   Updated to work on Familiar Pre0.7rc1, with busybox tar.
> + #   Note it Requires: binutils-ar (since the busybox ar can't create)
> + set -e
> ++set -o pipefail
> +
> + version=1.0
> +
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> index cf1e4670c65..eb6c7a3a6ad 100644
> --- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
> @@ -10,6 +10,7 @@ PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtu
>  SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \
>             file://0001-Switch-all-scripts-to-use-Python-3.x.patch \
>             file://0001-opkg-build-clamp-mtimes-to-SOURCE_DATE_EPOCH.patch \
> +           file://pipefail.patch \
>  "
>  UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
>
> @@ -19,6 +20,8 @@ SRC_URI[sha256sum] = "9ea9efdd9fe13661ad251e3a2860c1c93045adcfaa6659c3e86d9748ec
>
>  TARGET_CC_ARCH += "${LDFLAGS}"
>
> +RDEPENDS_${PN} += "bash"
> +
>  # For native builds we use the host Python
>  PYTHONRDEPS = "python3 python3-shell python3-io python3-math python3-crypt python3-logging python3-fcntl python3-pickle python3-compression python3-stringold"
>  PYTHONRDEPS_class-native = ""
> --
> 2.20.1
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation
  2019-11-22  0:38   ` Khem Raj
@ 2019-11-22  9:03     ` Richard Purdie
  0 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-22  9:03 UTC (permalink / raw)
  To: Khem Raj, Alejandro del Castillo; +Cc: bitbake-devel

On Thu, 2019-11-21 at 16:38 -0800, Khem Raj wrote:
> On Thu, Nov 21, 2019 at 12:54 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > opkg-build was failing on hosts where tar < 1.28 and reproducibile
> > builds
> > were enabled but it was doing this silently and generating
> > corrupted
> > (empty) ipk files. Add a fix for this (submitted upstream).
> > 
> > The fix requires bash but if you're building ipk files this
> > shoudn't be
> > a problem.
> 
> is the problem similar to this one
> 
> http://cfajohnson.com/shell/cus-faq-2.html#Q11

Yes.

> does it impact build times when default shell is not bash ?

I doubt very much whether it makes much difference at all. Its a single
execution per package of a script which takes a comparatively long
time. 

There are multiple places pipelines are used in the script and it would
be fairly invasive to try and use tricks like the ones in the link
above.

If it were something which looped tightly or has a short execution
time, it would make a difference but that isn't the case.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation
  2019-11-21 21:22 Richard Purdie
@ 2019-11-21 21:22 ` Richard Purdie
  0 siblings, 0 replies; 9+ messages in thread
From: Richard Purdie @ 2019-11-21 21:22 UTC (permalink / raw)
  To: openembedded-core

opkg-build was failing on hosts where tar < 1.28 and reproducibile builds
were enabled but it was doing this silently and generating corrupted
(empty) ipk files. Add a fix for this (submitted upstream).

The fix requires bash but if you're building ipk files this shoudn't be
a problem.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../opkg-utils/opkg-utils/pipefail.patch      | 31 +++++++++++++++++++
 .../opkg-utils/opkg-utils_0.4.1.bb            |  3 ++
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
new file mode 100644
index 00000000000..55ddcc1fd20
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/pipefail.patch
@@ -0,0 +1,31 @@
+We need opkg-build to fail if for example the tar command is passed invalid 
+options. Without this, we see silently created empty packaged where data.tar
+is zero bytes in size. This creates hard to debug problems.
+
+An example is when reproducible builds are enabled and run on old hosts like
+centos7 which has tar < 1.28:
+
+Subprocess output:tar: unrecognized option '--clamp-mtime'
+Try `tar --help' or `tar --usage' for more information.
+
+Upstream-Status: Pending
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+Index: opkg-utils-0.4.1/opkg-build
+===================================================================
+--- opkg-utils-0.4.1.orig/opkg-build
++++ opkg-utils-0.4.1/opkg-build
+@@ -1,4 +1,4 @@
+-#!/bin/sh
++#!/bin/bash
+ 
+ : <<=cut
+ =head1 NAME
+@@ -12,6 +12,7 @@ opkg-build - construct an .opk from a di
+ #   Updated to work on Familiar Pre0.7rc1, with busybox tar.
+ #   Note it Requires: binutils-ar (since the busybox ar can't create)
+ set -e
++set -o pipefail
+ 
+ version=1.0
+ 
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
index cf1e4670c65..eb6c7a3a6ad 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.1.bb
@@ -10,6 +10,7 @@ PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtu
 SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \
            file://0001-Switch-all-scripts-to-use-Python-3.x.patch \
            file://0001-opkg-build-clamp-mtimes-to-SOURCE_DATE_EPOCH.patch \
+           file://pipefail.patch \
 "
 UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
 
@@ -19,6 +20,8 @@ SRC_URI[sha256sum] = "9ea9efdd9fe13661ad251e3a2860c1c93045adcfaa6659c3e86d9748ec
 
 TARGET_CC_ARCH += "${LDFLAGS}"
 
+RDEPENDS_${PN} += "bash"
+
 # For native builds we use the host Python
 PYTHONRDEPS = "python3 python3-shell python3-io python3-math python3-crypt python3-logging python3-fcntl python3-pickle python3-compression python3-stringold"
 PYTHONRDEPS_class-native = ""
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2019-11-22  9:03 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-21 20:54 [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
2019-11-21 20:54 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie
2019-11-22  0:38   ` Khem Raj
2019-11-22  9:03     ` Richard Purdie
2019-11-21 20:54 ` [PATCH 3/5] sanity: Add check for tar older than 1.28 Richard Purdie
2019-11-21 20:54 ` [PATCH 4/5] core-image-full-cmdline: Add less Richard Purdie
2019-11-21 20:54 ` [PATCH 5/5] oeqa/selftest/sstatetests: Ensure we don't use hashequiv for sstatesigs tests Richard Purdie
2019-11-21 21:22 ` [PATCH 1/5] opkg: Add upstream fixes for empty packages Richard Purdie
2019-11-21 21:22 Richard Purdie
2019-11-21 21:22 ` [PATCH 2/5] opkg-utils: Fix silent empty/broken opkg package creation Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.