[PATCH] package.py: log rpmdeps call

[PATCH] package.py: log rpmdeps call

[Martin Jansa]

* I've noticed errors like this in log.do_package:

  DEBUG: Executing python function package_do_filedeps
  sh: 1: Syntax error: "(" unexpected
  sh: 1: Syntax error: "(" unexpected
  DEBUG: Python function package_do_filedeps finished

  which are actually caused by some filenames included in package
  containing '()' characters

  Maybe we should change meta/classes/package.bbclass to
  fail when some filedeprunner call fails like this and fix
  filedeprunner to escape '()' and other possibly dangerous chars
  it's called like this:
  processed = list(pool.imap(oe.package.filedeprunner, pkglist))

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/lib/oe/package.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 9a0ddb8..1f78a8d 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -89,7 +89,9 @@ def filedeprunner(arg):
 
         return provides, requires
 
-    dep_pipe = os.popen(rpmdeps + " " + " ".join(pkgfiles))
+    rpmdepscmd = rpmdeps + " " + " ".join(pkgfiles)
+    bb.debug(1, "runrpmdeps: %s" % rpmdepscmd)
+    dep_pipe = os.popen(rpmdepscmd)
 
     provides, requires = process_deps(dep_pipe, pkg, pkgdest, provides, requires)
 
-- 
1.8.5.3

Re: [PATCH] package.py: log rpmdeps call

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Tue, Feb 18, 2014 at 5:27 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

>   Maybe we should change meta/classes/package.bbclass to
>   fail when some filedeprunner call fails like this and fix
>   filedeprunner to escape '()' and other possibly dangerous chars
>   it's called like this:
>   processed = list(pool.imap(oe.package.filedeprunner, pkglist))
>

We'd likely be better off just avoiding os.popen entirely in favor of
subprocess based operations, which generally support specifying the command
arguments as a list, and can bypass the shell entirely, avoiding the need
to escape anything. Alternatively, use os.popen2(), which can accept a list
to bypass the shell as well.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1298 bytes --]

Re: [PATCH] package.py: log rpmdeps call

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Tue, Feb 18, 2014 at 01:27:02PM +0100, Martin Jansa wrote:
> * I've noticed errors like this in log.do_package:
> 
>   DEBUG: Executing python function package_do_filedeps
>   sh: 1: Syntax error: "(" unexpected
>   sh: 1: Syntax error: "(" unexpected
>   DEBUG: Python function package_do_filedeps finished
> 
>   which are actually caused by some filenames included in package
>   containing '()' characters
> 
>   Maybe we should change meta/classes/package.bbclass to
>   fail when some filedeprunner call fails like this and fix
>   filedeprunner to escape '()' and other possibly dangerous chars
>   it's called like this:
>   processed = list(pool.imap(oe.package.filedeprunner, pkglist))

Please don't merge this one yet, It's possible that it's causing
ValueError: insecure string pickle:
http://lists.openembedded.org/pipermail/openembedded-core/2014-February/089805.html

> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/lib/oe/package.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index 9a0ddb8..1f78a8d 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -89,7 +89,9 @@ def filedeprunner(arg):
>  
>          return provides, requires
>  
> -    dep_pipe = os.popen(rpmdeps + " " + " ".join(pkgfiles))
> +    rpmdepscmd = rpmdeps + " " + " ".join(pkgfiles)
> +    bb.debug(1, "runrpmdeps: %s" % rpmdepscmd)
> +    dep_pipe = os.popen(rpmdepscmd)
>  
>      provides, requires = process_deps(dep_pipe, pkg, pkgdest, provides, requires)
>  
> -- 
> 1.8.5.3
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[RFC][PATCH] package.py: use subprocess.Popen for rpmdeps call

[Martin Jansa]

* I've noticed errors like this in log.do_package:

  DEBUG: Executing python function package_do_filedeps
  sh: 1: Syntax error: "(" unexpected
  sh: 1: Syntax error: "(" unexpected
  DEBUG: Python function package_do_filedeps finished

  which are actually caused by some filenames included in package
  containing '()' characters

  Maybe we should change meta/classes/package.bbclass to
  fail when some filedeprunner call fails like this and fix
  filedeprunner to escape '()' and other possibly dangerous chars
  it's called like this:
  processed = list(pool.imap(oe.package.filedeprunner, pkglist))

* don't use shell=True
* show the command when it fails and let do_package task to fail

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/lib/oe/package.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 9a0ddb8..f8b5322 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -54,7 +54,7 @@ def file_translate(file):
     return ft
 
 def filedeprunner(arg):
-    import re
+    import re, subprocess, shlex
 
     (pkg, pkgfiles, rpmdeps, pkgdest) = arg
     provides = {}
@@ -89,8 +89,11 @@ def filedeprunner(arg):
 
         return provides, requires
 
-    dep_pipe = os.popen(rpmdeps + " " + " ".join(pkgfiles))
-
-    provides, requires = process_deps(dep_pipe, pkg, pkgdest, provides, requires)
+    try:
+        dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles, stdout=subprocess.PIPE)
+        provides, requires = process_deps(dep_popen.stdout, pkg, pkgdest, provides, requires)
+    except OSError as e:
+        bb.error("rpmdeps: '%s' command failed, '%s'" % (shlex.split(rpmdeps) + pkgfiles, e))
+        raise e
 
     return (pkg, provides, requires)
-- 
1.9.0