[PATCH 1/3] lib/oe/package: Improve filedeprunner subprocess handling

[PATCH 1/3] lib/oe/package: Improve filedeprunner subprocess handling

[Richard Purdie]

Currently the exit code of the spawned program isn't checked so it can
fail and the do_package task will continue merrily upon its way.

Use subprocess.check_output() to ensure we check the exit code and
redirect stderr to stdout so if it fails, we see the error output.

We can then drop the existing exception handling as the subprocess
exception gives a much better error.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oe/package.py | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index fcee389..1e5c3aa 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -176,8 +176,7 @@ def filedeprunner(arg):
 
     def process_deps(pipe, pkg, pkgdest, provides, requires):
         file = None
-        for line in pipe:
-            line = line.decode("utf-8")
+        for line in pipe.split("\n"):
 
             m = file_re.match(line)
             if m:
@@ -226,12 +225,8 @@ def filedeprunner(arg):
 
         return provides, requires
 
-    try:
-        dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles, stdout=subprocess.PIPE)
-        provides, requires = process_deps(dep_popen.stdout, pkg, pkgdest, provides, requires)
-    except OSError as e:
-        bb.error("rpmdeps: '%s' command failed, '%s'" % (shlex.split(rpmdeps) + pkgfiles, e))
-        raise e
+    output = subprocess.check_output(shlex.split(rpmdeps) + pkgfiles, stderr=subprocess.STDOUT).decode("utf-8")
+    provides, requires = process_deps(output, pkg, pkgdest, provides, requires)
 
     return (pkg, provides, requires)
 
-- 
2.7.4

[PATCH 2/3] lib/oe/utils: Handle exceptions in multiprocess_exec

[Richard Purdie]

Currently exceptions that happen in pool commands are ignored. Any errors
would be printed on the console but everything else is silent.

Switch to use pool.map_async which allows for an error_callback which
we can use to detect exceptions and make sure these errors are handled.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oe/utils.py | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 822d0cd..643ab78 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -224,25 +224,30 @@ def multiprocess_exec(commands, function):
     def init_worker():
         signal.signal(signal.SIGINT, signal.SIG_IGN)
 
+    fails = []
+
+    def failures(res):
+        fails.append(res)
+
     nproc = min(multiprocessing.cpu_count(), len(commands))
     pool = bb.utils.multiprocessingpool(nproc, init_worker)
-    imap = pool.imap(function, commands)
 
     try:
-        res = list(imap)
+        mapresult = pool.map_async(function, commands, error_callback=failures)
+
         pool.close()
         pool.join()
-        results = []
-        for result in res:
-            if result is not None:
-                results.append(result)
-        return results
-
+        results = mapresult.get()
     except KeyboardInterrupt:
         pool.terminate()
         pool.join()
         raise
 
+    if fails:
+        raise fails[0]
+
+    return results
+
 def squashspaces(string):
     import re
     return re.sub("\s+", " ", string).strip()
-- 
2.7.4

[PATCH 3/3] staging: Fix a logic error which caused dependency removal

[Richard Purdie]

There was a logic error in the dependency cleanup code which meant
it would remove dependencies which other tasks still depended upon.
Fix the path names so the comparisions work as intended.

This fixes dependencies accidentally disappearing from sysroots
under certain reconfiguration situations.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/staging.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index eb06f39..f409e9d 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -416,8 +416,8 @@ python extend_recipe_sysroot() {
             for l in f:
                 l = l.strip()
                 if l not in installed:
-                    l = depdir + "/" + l
-                    if not os.path.exists(l):
+                    fl = depdir + "/" + l
+                    if not os.path.exists(fl):
                         # Was likely already uninstalled
                         continue
                     potential.append(l)
-- 
2.7.4

Re: [PATCH 1/3] lib/oe/package: Improve filedeprunner subprocess handling

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Richard Purdie
> Sent: den 1 september 2017 16:21
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH 1/3] lib/oe/package: Improve filedeprunner
> subprocess handling
> 
> Currently the exit code of the spawned program isn't checked so it can
> fail and the do_package task will continue merrily upon its way.
> 
> Use subprocess.check_output() to ensure we check the exit code and
> redirect stderr to stdout so if it fails, we see the error output.
> 
> We can then drop the existing exception handling as the subprocess
> exception gives a much better error.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/lib/oe/package.py | 11 +++--------
>  1 file changed, 3 insertions(+), 8 deletions(-)
> 
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index fcee389..1e5c3aa 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -176,8 +176,7 @@ def filedeprunner(arg):
> 
>      def process_deps(pipe, pkg, pkgdest, provides, requires):
>          file = None
> -        for line in pipe:
> -            line = line.decode("utf-8")
> +        for line in pipe.split("\n"):
> 
>              m = file_re.match(line)
>              if m:
> @@ -226,12 +225,8 @@ def filedeprunner(arg):
> 
>          return provides, requires
> 
> -    try:
> -        dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles, stdout=subprocess.PIPE)
> -        provides, requires = process_deps(dep_popen.stdout, pkg, pkgdest, provides, requires)
> -    except OSError as e:
> -        bb.error("rpmdeps: '%s' command failed, '%s'" % (shlex.split(rpmdeps) + pkgfiles, e))
> -        raise e
> +    output = subprocess.check_output(shlex.split(rpmdeps) + pkgfiles, stderr=subprocess.STDOUT).decode("utf-8")
> +    provides, requires = process_deps(output, pkg, pkgdest, provides, requires)
> 
>      return (pkg, provides, requires)
> 
> --
> 2.7.4

Even with these changes applied, I still see texinfo succeed to build 
even when it should not (i.e., without having applied the patch to 
package_deb.bbclass that I just sent).

//Peter

Re: [PATCH 1/3] lib/oe/package: Improve filedeprunner subprocess handling

[Richard Purdie]

On Fri, 2017-09-01 at 16:32 +0000, Peter Kjellerstedt wrote:
> Even with these changes applied, I still see texinfo succeed to
> build 
> even when it should not (i.e., without having applied the patch to 
> package_deb.bbclass that I just sent).

Right, there are some nasty bugs with python's multiprocessing
exception handling in a few places. I've sent some patches for some
others which I ran into with other problems, Leo should have a patch
for this specific issue shortly. Thanks for the colon fix!

Cheers,

Richard