[PATCH 0/9] devtool / recipetool fixes

[PATCH 0/9] devtool / recipetool fixes

[Paul Eggleton]

Very late for M3 I know, but here are some fixes for issues I found
recently in devtool / recipetool as well as support for creating
recipes to package node.js code without using npm.


The following changes since commit 55bb6816aca39bfa25d4f7e2158a57a5f0ac1cca:

  oeqa.buildperf: correct globalres time format (2016-09-06 10:24:00 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool20-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool20-oe

Paul Eggleton (9):
  lib/oe/patch: handle non-UTF8 encoding when reading patches
  devtool: update-recipe: support files with subdir=
  recipetool: create: AX_PKG_SWIG should add dependency on swig-native
  recipetool: create: fix mapping python dependencies to python-dbg package
  recipetool: create: support git submodules
  recipetool: create: add --keep-temp command line option
  recipetool: create: allow license variable handling to be rerun
  recipetool: create: support node.js code outside of npm
  recipetool: create: avoid extra blank lines in output recipe

 meta/classes/npm.bbclass                         |   4 +
 meta/lib/oe/patch.py                             |  28 +++--
 meta/lib/oe/recipeutils.py                       |  12 ++-
 scripts/lib/devtool/standard.py                  |  38 ++++++-
 scripts/lib/recipetool/create.py                 | 129 +++++++++++++----------
 scripts/lib/recipetool/create_buildsys.py        |   2 +-
 scripts/lib/recipetool/create_buildsys_python.py |   4 +
 scripts/lib/recipetool/create_npm.py             | 111 ++++++++++++++++++-
 8 files changed, 258 insertions(+), 70 deletions(-)

-- 
2.5.5

[PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches

[Paul Eggleton]

When extracting patches from a git repository with PATCHTOOL = "git" we
cannot assume that all patches will be UTF-8 formatted, so as with other
places in this module, try latin-1 if utf-8 fails.

This fixes UnicodeDecodeError running devtool update-recipe or devtool
finish on the openssl recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/patch.py | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index cad5015..05e0faa 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -415,16 +415,24 @@ class GitApplyTree(PatchTree):
             out = runcmd(["sh", "-c", " ".join(shellcmd)], tree)
             if out:
                 for srcfile in out.split():
-                    patchlines = []
-                    outfile = None
-                    with open(srcfile, 'r') as f:
-                        for line in f:
-                            if line.startswith(GitApplyTree.patch_line_prefix):
-                                outfile = line.split()[-1].strip()
-                                continue
-                            if line.startswith(GitApplyTree.ignore_commit_prefix):
-                                continue
-                            patchlines.append(line)
+                    for encoding in ['utf-8', 'latin-1']:
+                        patchlines = []
+                        outfile = None
+                        try:
+                            with open(srcfile, 'r', encoding=encoding) as f:
+                                for line in f:
+                                    if line.startswith(GitApplyTree.patch_line_prefix):
+                                        outfile = line.split()[-1].strip()
+                                        continue
+                                    if line.startswith(GitApplyTree.ignore_commit_prefix):
+                                        continue
+                                    patchlines.append(line)
+                        except UnicodeDecodeError:
+                            continue
+                        break
+                    else:
+                        raise PatchError('Unable to find a character encoding to decode %s' % srcfile)
+
                     if not outfile:
                         outfile = os.path.basename(srcfile)
                     with open(os.path.join(outdir, outfile), 'w') as of:
-- 
2.5.5

[PATCH 2/9] devtool: update-recipe: support files with subdir=

[Paul Eggleton]

It's rare but there are recipes that have individual files (as opposed
to archives) in SRC_URI using subdir= to put them under the source tree,
the examples in OE-Core being bzip2 and openssl. This broke devtool
update-recipe (and devtool finish) because the file wasn't unpacked into
the oe-local-files directory and thus when it came time to update the
recipe, the file was assumed to have been deleted by the user and thus
the file was erroneously removed. Add logic to handle these properly so
that this doesn't happen.

(We still have another potential problem in that these files become part
of the initial commit from upstream, which could be confusing because
they didn't come from there - but that's a separate issue and not one
that is trivially solved.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py      | 12 ++++++++++--
 scripts/lib/devtool/standard.py | 38 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index e7dd8af..a0d78dd 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -400,8 +400,16 @@ def get_recipe_local_files(d, patches=False):
                     bb.utils.exec_flat_python_func('patch_path', uri, fetch, '')):
                 continue
             # Skip files that are referenced by absolute path
-            if not os.path.isabs(fetch.ud[uri].basepath):
-                ret[fetch.ud[uri].basepath] = fetch.localpath(uri)
+            fname = fetch.ud[uri].basepath
+            if os.path.isabs(fname):
+                continue
+            # Handle subdir=
+            subdir = fetch.ud[uri].parm.get('subdir', '')
+            if subdir:
+                if os.path.isabs(subdir):
+                    continue
+                fname = os.path.join(subdir, fname)
+            ret[fname] = fetch.localpath(uri)
     return ret
 
 
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 98451da..baef23e 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -307,6 +307,13 @@ def _move_file(src, dst):
         bb.utils.mkdirhier(dst_d)
     shutil.move(src, dst)
 
+def _copy_file(src, dst):
+    """Copy a file. Creates all the directory components of destination path."""
+    dst_d = os.path.dirname(dst)
+    if dst_d:
+        bb.utils.mkdirhier(dst_d)
+    shutil.copy(src, dst)
+
 def _git_ls_tree(repodir, treeish='HEAD', recursive=False):
     """List contents of a git treeish"""
     import bb
@@ -1050,6 +1057,23 @@ def _export_local_files(srctree, rd, destdir):
             elif fname != '.gitignore':
                 added[fname] = None
 
+        workdir = rd.getVar('WORKDIR', True)
+        s = rd.getVar('S', True)
+        if not s.endswith(os.sep):
+            s += os.sep
+
+        if workdir != s:
+            # Handle files where subdir= was specified
+            for fname in list(existing_files.keys()):
+                # FIXME handle both subdir starting with BP and not?
+                fworkpath = os.path.join(workdir, fname)
+                if fworkpath.startswith(s):
+                    fpath = os.path.join(srctree, os.path.relpath(fworkpath, s))
+                    if os.path.exists(fpath):
+                        origpath = existing_files.pop(fname)
+                        if not filecmp.cmp(origpath, fpath):
+                            updated[fpath] = origpath
+
         removed = existing_files
     return (updated, added, removed)
 
@@ -1122,7 +1146,12 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo
             files_dir = _determine_files_dir(rd)
             for basepath, path in upd_f.items():
                 logger.info('Updating file %s' % basepath)
-                _move_file(os.path.join(local_files_dir, basepath), path)
+                if os.path.isabs(basepath):
+                    # Original file (probably with subdir pointing inside source tree)
+                    # so we do not want to move it, just copy
+                    _copy_file(basepath, path)
+                else:
+                    _move_file(os.path.join(local_files_dir, basepath), path)
                 update_srcuri= True
             for basepath, path in new_f.items():
                 logger.info('Adding new file %s' % basepath)
@@ -1205,7 +1234,12 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
             # Update existing files
             for basepath, path in upd_f.items():
                 logger.info('Updating file %s' % basepath)
-                _move_file(os.path.join(local_files_dir, basepath), path)
+                if os.path.isabs(basepath):
+                    # Original file (probably with subdir pointing inside source tree)
+                    # so we do not want to move it, just copy
+                    _copy_file(basepath, path)
+                else:
+                    _move_file(os.path.join(local_files_dir, basepath), path)
                 updatefiles = True
             for basepath, path in upd_p.items():
                 patchfn = os.path.join(patches_dir, basepath)
-- 
2.5.5

[PATCH 3/9] recipetool: create: AX_PKG_SWIG should add dependency on swig-native

[Paul Eggleton]

If AX_PKG_SWIG is found in configure.ac, then what's being looked for is
the swig binary, not swig for the target - so fix the dependency
accordingly.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create_buildsys.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index f784f94..0ad748e 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -570,7 +570,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
             elif keyword == 'AX_LIB_TAGLIB':
                 deps.append('taglib')
             elif keyword == 'AX_PKG_SWIG':
-                deps.append('swig')
+                deps.append('swig-native')
             elif keyword == 'AX_PROG_XSLTPROC':
                 deps.append('libxslt-native')
             elif keyword == 'AX_WITH_CURSES':
-- 
2.5.5

[PATCH 4/9] recipetool: create: fix mapping python dependencies to python-dbg package

[Paul Eggleton]

When trying to map python module dependencies to the packages that
provide them, if we're looking for .so files that satisfy
dependencies then we need to exclude files found under the .debug
directory, otherwise the dependency will get mapped to the python-dbg
package which isn't correct.

For example, this fixes creating a recipe for pyserial and not getting
python-fcntl in RDEPENDS_${PN}, leading to errors when trying to use the
serial module on the target.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create_buildsys_python.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index aff13cf..fb9806d 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -504,8 +504,10 @@ class PythonRecipeHandler(RecipeHandler):
         for dep in scanned_deps:
             mapped = provided_packages.get(dep)
             if mapped:
+                logger.debug('Mapped %s to %s' % (dep, mapped))
                 mapped_deps.add(mapped)
             else:
+                logger.debug('Could not map %s' % dep)
                 unmapped_deps.add(dep)
         return mapped_deps, unmapped_deps
 
@@ -566,6 +568,8 @@ class PythonRecipeHandler(RecipeHandler):
                     continue
 
                 if fn.startswith(dynload_dir + os.sep):
+                    if '/.debug/' in fn:
+                        continue
                     base = os.path.basename(fn)
                     provided = base.split('.', 1)[0]
                     packages[provided] = os.path.basename(pkgdatafile)
-- 
2.5.5

[PATCH 5/9] recipetool: create: support git submodules

[Paul Eggleton]

Ensure we fetch submodules and set SRC_URI correctly when pointing to a
git repository that contains submodules.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index a08352e..7cbd614 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -399,6 +399,11 @@ def create_recipe(args):
                     if '<html' in f.read(100).lower():
                         logger.error('Fetching "%s" returned a single HTML page - check the URL is correct and functional' % fetchuri)
                         sys.exit(1)
+        if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
+            srcuri = 'gitsm://' + srcuri[6:]
+            logger.info('Fetching submodules...')
+            bb.process.run('git submodule init', cwd=srctree)
+            bb.process.run('git submodule update', cwd=srctree)
 
         if is_package(fetchuri):
             tmpfdir = tempfile.mkdtemp(prefix='recipetool-')
@@ -658,8 +663,11 @@ def create_recipe(args):
             # devtool looks for this specific exit code, so don't change it
             sys.exit(15)
         else:
-            if srcuri and srcuri.startswith(('git://', 'hg://', 'svn://')):
-                outfile = '%s_%s.bb' % (pn, srcuri.split(':', 1)[0])
+            if srcuri and srcuri.startswith(('gitsm://', 'git://', 'hg://', 'svn://')):
+                suffix = srcuri.split(':', 1)[0]
+                if suffix == 'gitsm':
+                    suffix = 'git'
+                outfile = '%s_%s.bb' % (pn, suffix)
             elif realpv:
                 outfile = '%s_%s.bb' % (pn, realpv)
             else:
-- 
2.5.5

[PATCH 6/9] recipetool: create: add --keep-temp command line option

[Paul Eggleton]

For debugging it's useful to be able to tell recipetool to keep the
temporary directory.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7cbd614..89ab748 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -765,7 +765,10 @@ def create_recipe(args):
         logger.info('Recipe %s has been created; further editing may be required to make it fully functional' % outfile)
 
     if tempsrc:
-        shutil.rmtree(tempsrc)
+        if args.keep_temp:
+            logger.info('Preserving temporary directory %s' % tempsrc)
+        else:
+            shutil.rmtree(tempsrc)
 
     return 0
 
@@ -1048,5 +1051,6 @@ def register_commands(subparsers):
     parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
     parser_create.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.set_defaults(func=create_recipe)
 
-- 
2.5.5

[PATCH 7/9] recipetool: create: allow license variable handling to be rerun

[Paul Eggleton]

If you make adjustments to the source tree (as create_npm.py will be)
then you will need to re-run the license variable handling code at the
end so that we get all of the files that should go into
LIC_FILES_CHKSUM if nothing else. Split out the license variable
handling to a separate function in order to allow this.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 103 ++++++++++++++++++++-------------------
 1 file changed, 53 insertions(+), 50 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 89ab748..f1ce28c 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -482,51 +482,8 @@ def create_recipe(args):
     # We need a blank line here so that patch_recipe_lines can rewind before the LICENSE comments
     lines_before.append('')
 
-    licvalues = guess_license(srctree_use)
-    lic_files_chksum = []
-    lic_unknown = []
-    if licvalues:
-        licenses = []
-        for licvalue in licvalues:
-            if not licvalue[0] in licenses:
-                licenses.append(licvalue[0])
-            lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1], licvalue[2]))
-            if licvalue[0] == 'Unknown':
-                lic_unknown.append(licvalue[1])
-        lines_before.append('# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is')
-        lines_before.append('# your responsibility to verify that the values are complete and correct.')
-        if len(licvalues) > 1:
-            lines_before.append('#')
-            lines_before.append('# NOTE: multiple licenses have been detected; if that is correct you should separate')
-            lines_before.append('# these in the LICENSE value using & if the multiple licenses all apply, or | if there')
-            lines_before.append('# is a choice between the multiple licenses. If in doubt, check the accompanying')
-            lines_before.append('# documentation to determine which situation is applicable.')
-        if lic_unknown:
-            lines_before.append('#')
-            lines_before.append('# The following license files were not able to be identified and are')
-            lines_before.append('# represented as "Unknown" below, you will need to check them yourself:')
-            for licfile in lic_unknown:
-                lines_before.append('#   %s' % licfile)
-            lines_before.append('#')
-    else:
-        lines_before.append('# Unable to find any files that looked like license statements. Check the accompanying')
-        lines_before.append('# documentation and source headers and set LICENSE and LIC_FILES_CHKSUM accordingly.')
-        lines_before.append('#')
-        lines_before.append('# NOTE: LICENSE is being set to "CLOSED" to allow you to at least start building - if')
-        lines_before.append('# this is not accurate with respect to the licensing of the software being built (it')
-        lines_before.append('# will not be in most cases) you must specify the correct value before using this')
-        lines_before.append('# recipe for anything other than initial testing/development!')
-        licenses = ['CLOSED']
-    pkg_license = extravalues.pop('LICENSE', None)
-    if pkg_license:
-        if licenses == ['Unknown']:
-            lines_before.append('# NOTE: The following LICENSE value was determined from the original package metadata')
-            licenses = [pkg_license]
-        else:
-            lines_before.append('# NOTE: Original package metadata indicates license is: %s' % pkg_license)
-    lines_before.append('LICENSE = "%s"' % ' '.join(licenses))
-    lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n                    '.join(lic_files_chksum))
-    lines_before.append('')
+    handled = []
+    licvalues = handle_license_vars(srctree_use, lines_before, handled, extravalues, tinfoil.config_data)
 
     classes = []
 
@@ -623,9 +580,6 @@ def create_recipe(args):
     handlers = [item[0] for item in handlers]
 
     # Apply the handlers
-    handled = []
-    handled.append(('license', licvalues))
-
     if args.binary:
         classes.append('bin_package')
         handled.append('buildsystem')
@@ -772,6 +726,55 @@ def create_recipe(args):
 
     return 0
 
+def handle_license_vars(srctree, lines_before, handled, extravalues, d):
+    licvalues = guess_license(srctree, d)
+    lic_files_chksum = []
+    lic_unknown = []
+    if licvalues:
+        licenses = []
+        for licvalue in licvalues:
+            if not licvalue[0] in licenses:
+                licenses.append(licvalue[0])
+            lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1], licvalue[2]))
+            if licvalue[0] == 'Unknown':
+                lic_unknown.append(licvalue[1])
+        lines_before.append('# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is')
+        lines_before.append('# your responsibility to verify that the values are complete and correct.')
+        if len(licvalues) > 1:
+            lines_before.append('#')
+            lines_before.append('# NOTE: multiple licenses have been detected; if that is correct you should separate')
+            lines_before.append('# these in the LICENSE value using & if the multiple licenses all apply, or | if there')
+            lines_before.append('# is a choice between the multiple licenses. If in doubt, check the accompanying')
+            lines_before.append('# documentation to determine which situation is applicable.')
+        if lic_unknown:
+            lines_before.append('#')
+            lines_before.append('# The following license files were not able to be identified and are')
+            lines_before.append('# represented as "Unknown" below, you will need to check them yourself:')
+            for licfile in lic_unknown:
+                lines_before.append('#   %s' % licfile)
+            lines_before.append('#')
+    else:
+        lines_before.append('# Unable to find any files that looked like license statements. Check the accompanying')
+        lines_before.append('# documentation and source headers and set LICENSE and LIC_FILES_CHKSUM accordingly.')
+        lines_before.append('#')
+        lines_before.append('# NOTE: LICENSE is being set to "CLOSED" to allow you to at least start building - if')
+        lines_before.append('# this is not accurate with respect to the licensing of the software being built (it')
+        lines_before.append('# will not be in most cases) you must specify the correct value before using this')
+        lines_before.append('# recipe for anything other than initial testing/development!')
+        licenses = ['CLOSED']
+    pkg_license = extravalues.pop('LICENSE', None)
+    if pkg_license:
+        if licenses == ['Unknown']:
+            lines_before.append('# NOTE: The following LICENSE value was determined from the original package metadata')
+            licenses = [pkg_license]
+        else:
+            lines_before.append('# NOTE: Original package metadata indicates license is: %s' % pkg_license)
+    lines_before.append('LICENSE = "%s"' % ' '.join(licenses))
+    lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n                    '.join(lic_files_chksum))
+    lines_before.append('')
+    handled.append(('license', licvalues))
+    return licvalues
+
 def get_license_md5sums(d, static_only=False):
     import bb.utils
     md5sums = {}
@@ -892,9 +895,9 @@ def crunch_license(licfile):
     license = crunched_md5sums.get(md5val, None)
     return license, md5val, lictext
 
-def guess_license(srctree):
+def guess_license(srctree, d):
     import bb
-    md5sums = get_license_md5sums(tinfoil.config_data)
+    md5sums = get_license_md5sums(d)
 
     licenses = []
     licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
-- 
2.5.5

[PATCH 8/9] recipetool: create: support node.js code outside of npm

[Paul Eggleton]

If you have your own node.js application you may not publish it (or at
least not immediately) in an npm registry - it might just be in a
repository on github or on your local machine. Add support to recipetool
create for creating recipes to build such applications - extract their
dependencies, fetch them, and add corresponding npm:// URLs to SRC_URI,
and ensure that LICENSE / LIC_FILES_CHKSUM are updated to match. For
example, you can now run:

  recipetool create https://github.com/diversario/node-ssdp

(I had to borrow some code from bitbake/lib/bb/fetch2/npm.py to
implement this functionality; this should be refactored out but now
isn't the time to do that refactoring.)

Part of the fix for [YOCTO #9537].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/npm.bbclass             |   4 ++
 scripts/lib/recipetool/create_npm.py | 111 ++++++++++++++++++++++++++++++++++-
 2 files changed, 114 insertions(+), 1 deletion(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 55c7c3e..fce4c11 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -16,6 +16,10 @@ def npm_oe_arch_map(target_arch, d):
 NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH', True), d)}"
 
 npm_do_compile() {
+	# Copy in any additionally fetched modules
+	if [ -d ${WORKDIR}/node_modules ] ; then
+		cp -a ${WORKDIR}/node_modules ${S}/
+	fi
 	# changing the home directory to the working directory, the .npmrc will
 	# be created in this directory
 	export HOME=${WORKDIR}
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index e5aaa60..e794614 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -21,7 +21,7 @@ import subprocess
 import tempfile
 import shutil
 import json
-from recipetool.create import RecipeHandler, split_pkg_licenses
+from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars
 
 logger = logging.getLogger('recipetool')
 
@@ -83,6 +83,66 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues['extrafiles']['lockdown.json'] = tmpfile
         lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
 
+    def _handle_dependencies(self, d, deps, lines_before, srctree):
+        import scriptutils
+        # If this isn't a single module we need to get the dependencies
+        # and add them to SRC_URI
+        def varfunc(varname, origvalue, op, newlines):
+            if varname == 'SRC_URI':
+                if not origvalue.startswith('npm://'):
+                    src_uri = origvalue.split()
+                    changed = False
+                    for dep, depdata in deps.items():
+                        version = self.get_node_version(dep, depdata, d)
+                        if version:
+                            url = 'npm://registry.npmjs.org;name=%s;version=%s;subdir=node_modules/%s' % (dep, version, dep)
+                            scriptutils.fetch_uri(d, url, srctree)
+                            src_uri.append(url)
+                            changed = True
+                    if changed:
+                        return src_uri, None, -1, True
+            return origvalue, None, 0, True
+        updated, newlines = bb.utils.edit_metadata(lines_before, ['SRC_URI'], varfunc)
+        if updated:
+            del lines_before[:]
+            for line in newlines:
+                # Hack to avoid newlines that edit_metadata inserts
+                if line.endswith('\n'):
+                    line = line[:-1]
+                lines_before.append(line)
+        return updated
+
+    def _replace_license_vars(self, srctree, lines_before, handled, extravalues, d):
+        for item in handled:
+            if isinstance(item, tuple):
+                if item[0] == 'license':
+                    del item
+                    break
+
+        calledvars = []
+        def varfunc(varname, origvalue, op, newlines):
+            if varname in ['LICENSE', 'LIC_FILES_CHKSUM']:
+                for i, e in enumerate(reversed(newlines)):
+                    if not e.startswith('#'):
+                        stop = i
+                        while stop > 0:
+                            newlines.pop()
+                            stop -= 1
+                        break
+                calledvars.append(varname)
+                if len(calledvars) > 1:
+                    # The second time around, put the new license text in
+                    insertpos = len(newlines)
+                    handle_license_vars(srctree, newlines, handled, extravalues, d)
+                return None, None, 0, True
+            return origvalue, None, 0, True
+        updated, newlines = bb.utils.edit_metadata(lines_before, ['LICENSE', 'LIC_FILES_CHKSUM'], varfunc)
+        if updated:
+            del lines_before[:]
+            lines_before.extend(newlines)
+        else:
+            raise Exception('Did not find license variables')
+
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
         import bb.utils
         import oe
@@ -108,6 +168,12 @@ class NpmRecipeHandler(RecipeHandler):
                 if 'homepage' in data:
                     extravalues['HOMEPAGE'] = data['homepage']
 
+                deps = data.get('dependencies', {})
+                updated = self._handle_dependencies(tinfoil.config_data, deps, lines_before, srctree)
+                if updated:
+                    # We need to redo the license stuff
+                    self._replace_license_vars(srctree, lines_before, handled, extravalues, tinfoil.config_data)
+
                 # Shrinkwrap
                 localfilesdir = tempfile.mkdtemp(prefix='recipetool-npm')
                 self._shrinkwrap(srctree, localfilesdir, extravalues, lines_before)
@@ -148,9 +214,52 @@ class NpmRecipeHandler(RecipeHandler):
                             lines_before[i] = 'LICENSE = "%s"' % ' & '.join(all_licenses)
                             break
 
+                # Need to move S setting after inherit npm
+                for i, line in enumerate(lines_before):
+                    if line.startswith('S ='):
+                        lines_before.pop(i)
+                        lines_after.insert(0, '# Must be set after inherit npm since that itself sets S')
+                        lines_after.insert(1, line)
+                        break
+
                 return True
 
         return False
 
+    # FIXME this is duplicated from lib/bb/fetch2/npm.py
+    def _parse_view(self, output):
+        '''
+        Parse the output of npm view --json; the last JSON result
+        is assumed to be the one that we're interested in.
+        '''
+        pdata = None
+        outdeps = {}
+        datalines = []
+        bracelevel = 0
+        for line in output.splitlines():
+            if bracelevel:
+                datalines.append(line)
+            elif '{' in line:
+                datalines = []
+                datalines.append(line)
+            bracelevel = bracelevel + line.count('{') - line.count('}')
+        if datalines:
+            pdata = json.loads('\n'.join(datalines))
+        return pdata
+
+    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
+    # (split out from _getdependencies())
+    def get_node_version(self, pkg, version, d):
+        import bb.fetch2
+        pkgfullname = pkg
+        if version != '*' and not '/' in version:
+            pkgfullname += "@'%s'" % version
+        logger.debug(2, "Calling getdeps on %s" % pkg)
+        runenv = dict(os.environ, PATH=d.getVar('PATH', True))
+        fetchcmd = "npm view %s --json" % pkgfullname
+        output, _ = bb.process.run(fetchcmd, stderr=subprocess.STDOUT, env=runenv, shell=True)
+        data = self._parse_view(output)
+        return data.get('version', None)
+
 def register_recipe_handlers(handlers):
     handlers.append((NpmRecipeHandler(), 60))
-- 
2.5.5

[PATCH 9/9] recipetool: create: avoid extra blank lines in output recipe

[Paul Eggleton]

If we output extra blank lines (because of some automated editing) then
it makes the output recipe look a bit untidy. You could argue that we
should simply have the editing code not do that, but sometimes we don't
have enough context there for that to be practical. It's simple enough
to just filter out the extra blank lines when writing the file, so just
do it that way.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f1ce28c..b6de999 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -715,7 +715,13 @@ def create_recipe(args):
         sys.stdout.write('\n'.join(outlines) + '\n')
     else:
         with open(outfile, 'w') as f:
-            f.write('\n'.join(outlines) + '\n')
+            lastline = None
+            for line in outlines:
+                if not lastline and not line:
+                    # Skip extra blank lines
+                    continue
+                f.write('%s\n' % line)
+                lastline = line
         logger.info('Recipe %s has been created; further editing may be required to make it fully functional' % outfile)
 
     if tempsrc:
-- 
2.5.5

Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches

[Enrico Scholz]

Paul Eggleton <paul.eggleton-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
writes:

> When extracting patches from a git repository with PATCHTOOL = "git" we
> cannot assume that all patches will be UTF-8 formatted, so as with other
> places in this module, try latin-1 if utf-8 fails.

This will probably not work when patch contains a character between 128
and 159 (which is a blackhole in all locales afaik).

I would read the file as a binary ('rb' instead of 'r') and make the
GitApplyTree.* strings a 'bytes' type.


Enrico

Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches

[Paul Eggleton]

Hi Enrico,

On Tue, 06 Sep 2016 17:50:02 Enrico Scholz wrote:
> Paul Eggleton <paul.eggleton-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> writes:
> > When extracting patches from a git repository with PATCHTOOL = "git" we
> > cannot assume that all patches will be UTF-8 formatted, so as with other
> > places in this module, try latin-1 if utf-8 fails.
> 
> This will probably not work when patch contains a character between 128
> and 159 (which is a blackhole in all locales afaik).

I realise it's by no means perfect - you may even fairly label it a hack, 
since it's only handling two encodings out of many. However I was keen to at 
least restore the ability to handle the majority of patches we have in the 
core, we can always improve it subsequently (even before the release).
 
> I would read the file as a binary ('rb' instead of 'r') and make the
> GitApplyTree.* strings a 'bytes' type.

The code is not just passing the data through, it is actually processing it. 
If we did what you propose wouldn't it make that processing more difficult? 

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 5/9] recipetool: create: support git submodules

[Ola x Nilsson]

> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of Paul Eggleton
> Sent: den 6 september 2016 12:03
> 
> Ensure we fetch submodules and set SRC_URI correctly when pointing to a
> git repository that contains submodules.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  scripts/lib/recipetool/create.py | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
> index a08352e..7cbd614 100644
> --- a/scripts/lib/recipetool/create.py
> +++ b/scripts/lib/recipetool/create.py
> @@ -399,6 +399,11 @@ def create_recipe(args):
>                      if '<html' in f.read(100).lower():
>                          logger.error('Fetching "%s" returned a single HTML page - check
> the URL is correct and functional' % fetchuri)
>                          sys.exit(1)
> +        if os.path.exists(os.path.join(srctree, '.gitmodules')) and
> srcuri.startswith('git://'):
> +            srcuri = 'gitsm://' + srcuri[6:]
> +            logger.info('Fetching submodules...')
> +            bb.process.run('git submodule init', cwd=srctree)
> +            bb.process.run('git submodule update', cwd=srctree)
> 

This is not how the gitsm fetcher does it.

git submodule update --init --recursive

will actually fetch all submodules recursively.

-- 
Ola Nilsson

Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches

[Enrico Scholz]

Paul Eggleton <paul.eggleton@linux.intel.com> writes:

>> I would read the file as a binary ('rb' instead of 'r') and make the
>> GitApplyTree.* strings a 'bytes' type.
>
> The code is not just passing the data through, it is actually processing
> it.  If we did what you propose wouldn't it make that processing more
> difficult?

Should not be very much... You have to write b'foo' instead of 'foo' at
some places, but things like 'file.readline()' or 'line.startswith()'
are still working with 'bytes' resp. binary files.

Only when trying to print them on stdout, things can get nasty.  But
this should not be needed for patches.


Enrico
-- 
SIGMA Chemnitz GmbH       Registergericht:   Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13           Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz

Re: [PATCH 5/9] recipetool: create: support git submodules

[Paul Eggleton]

Hi Ola,

On Wed, 07 Sep 2016 06:23:15 Ola x Nilsson wrote:
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of Paul Eggleton
> > Sent: den 6 september 2016 12:03
> > 
> > Ensure we fetch submodules and set SRC_URI correctly when pointing to a
> > git repository that contains submodules.
> > 
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> > 
> >  scripts/lib/recipetool/create.py | 12 ++++++++++--
> >  1 file changed, 10 insertions(+), 2 deletions(-)
> > 
> > diff --git a/scripts/lib/recipetool/create.py
> > b/scripts/lib/recipetool/create.py index a08352e..7cbd614 100644
> > --- a/scripts/lib/recipetool/create.py
> > +++ b/scripts/lib/recipetool/create.py
> > 
> > @@ -399,6 +399,11 @@ def create_recipe(args):
> >                      if '<html' in f.read(100).lower():
> >                          logger.error('Fetching "%s" returned a single
> >                          HTML page - check
> > 
> > the URL is correct and functional' % fetchuri)
> > 
> >                          sys.exit(1)
> > 
> > +        if os.path.exists(os.path.join(srctree, '.gitmodules')) and
> > srcuri.startswith('git://'):
> > +            srcuri = 'gitsm://' + srcuri[6:]
> > +            logger.info('Fetching submodules...')
> > +            bb.process.run('git submodule init', cwd=srctree)
> > +            bb.process.run('git submodule update', cwd=srctree)
> 
> This is not how the gitsm fetcher does it.
> 
> git submodule update --init --recursive
> 
> will actually fetch all submodules recursively.

Right, we should indeed be doing that. I've pushed a new commit to the branch.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre