[RFC][PATCH v2 0/7] NPM refactoring

[RFC][PATCH v2 0/7] NPM refactoring

[Jean-Marie LEMETAYER]

The current NPM support have several issues:
 - The current NPM fetcher downloads the dependency tree but not the other
   fetchers. The 'subdir' parameter was used to fix this issue.
 - They are multiple issues with package names (uppercase, exotic characters,
   scoped packages) even if they are inside the dependencies.
 - The lockdown file generation have issues. When a package depends on
   multiple version of the same package (all versions have the same checksum).

This patchset refactors the NPM support in Yocto:
 - As the NPM algorithm for dependency management is hard to handle, the new
   NPM fetcher downloads only the package source (and not the dependencies,
   like the other fetchers) (patch submitted in the bitbake-devel list).
 - The NPM class handles the dependencies using NPM (and not manually).
 - The NPM recipe creation is simplified to avoid issues.
 - The lockdown file is no more used as it is no longer relevant compared to the
   latest shrinkwrap file format.

This patchset may remove some features (lockdown file, license management for
dependencies) but fixes the majority of the NPM issues. All of these issues
from the bugzilla.yoctoproject.org are resolved by this patchset:
#10237, #10760, #11028, #11728, #11902, #12534

The fetcher and recipetool are now aware of a 'latest' keyword for the version
which allow to build the latest version available on the registry. This feature
fixes the two issues: #10515, #11029

Moreover the issue #13415 should also be fixed but I cannot test it.

I have tested the recipe creation and builds using a self made example to
generate build failures:
 - https://github.com/savoirfairelinux/node-server-example
 - https://npmjs.com/package/@savoirfairelinux/node-server-example

The test steps are these ones:
  $ source poky/oe-init-build-env
  $ bitbake-layers add-layer ../meta-openembedded/meta-oe
  $ devtool add "npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest"
  $ devtool build savoirfairelinux-node-server-example
  $ bitbake-layers create-layer ../meta-test
  $ bitbake-layers add-layer ../meta-test
  $ devtool finish savoirfairelinux-node-server-example ../meta-test
  $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> conf/local.conf
  $ bitbake core-image-minimal

Also the 'devtool add' url could be one of these:
 - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
   This url uses the new npm fetcher and request the latest version available on
   the registry.
 - npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
   This url uses the new npm fetcher and request a fixed version.
 - git://github.com/savoirfairelinux/node-server-example.git;protocol=https
   This url uses the git fetcher. As the dependencies are managed by the NPM
   class, any fetcher can be used.

When this patchset will be merged, I have planned to update the NPM wiki:
  https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

This patchset is also the base work for the full Angular support in Yocto that I
am preparing. These applications have a huge dependency tree which is ideal to
test the NPM support.

--- V2

 - Add the 'check_network_access' function before each network access to check
   for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.

 - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for
   'oe-selftest' to test the npm recipe creation.

 - Update the 'npm.bbclass' to fetch the dependencies in the 'do_fetch' task.
   The dependencies are cached in a npm cache stored in '${DL_DIR}/npm_cache'
   allowing the dependencies to be saved in the download directory and verify
   on insertion and extraction [1]:
      "All data that passes through the cache is fully verified
       for integrity on both insertion and extraction."

1: https://docs.npmjs.com/cli/cache.html

Jean-Marie LEMETAYER (7):
  npm.bbclass: refactor the npm class
  devtool: update command line options for npm
  recipetool/create_npm.py: refactor the npm recipe creation handler
  devtool/standard.py: update the append file for the npm recipes
  recipetool/create.py: replace 'latest' keyword for npm
  recipetool/create.py: remove the 'noverify' url parameter
  oeqa/selftest/recipetool: add npm recipe creation test

 meta/classes/npm.bbclass                   | 255 ++++++++----
 meta/lib/oeqa/selftest/cases/recipetool.py |  19 +
 scripts/lib/devtool/standard.py            |  22 +-
 scripts/lib/recipetool/create.py           |  16 +-
 scripts/lib/recipetool/create_npm.py       | 443 ++++++++-------------
 5 files changed, 394 insertions(+), 361 deletions(-)

--
2.20.1

[RFC][PATCH v2 1/7] npm.bbclass: refactor the npm class

[Jean-Marie LEMETAYER]

Many issues were related to npm dependencies badly handled: package
names, installation directories, ... In fact npm is using an install
algorithm [1] which is hard to reproduce / anticipate. Moreover some
npm packages use scopes [2] which adds more complexity.

The simplest solution is to let npm do its job. Assuming the fetcher
only get the sources of the package, the class will now run
'npm install' to create a build directory. The build directory is then
copied wisely to the destination.

1: https://docs.npmjs.com/cli/install#algorithm
2: https://docs.npmjs.com/about-scopes

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 255 +++++++++++++++++++++++++++++----------
 1 file changed, 188 insertions(+), 67 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 4b1f0a39f0..1abb0f710a 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -1,19 +1,44 @@
+# Copyright (C) 2019 Savoir-Faire Linux
+#
+# This bbclass builds and installs an npm package to the target. The package
+# sources files should be fetched in the calling recipe by using the SRC_URI
+# variable. The ${S} variable should be updated depending of your fetcher.
+#
+# Usage:
+#  SRC_URI = "..."
+#  inherit npm
+#
+# Optional variables:
+#  NPM_SHRINKWRAP:
+#       Provide a shrinkwrap file [1]. If available a shrinkwrap file in the
+#       sources has priority over the one provided. A shrinkwrap file is
+#       mandatory in order to ensure build reproducibility.
+#       1: https://docs.npmjs.com/files/shrinkwrap.json
+#
+#  NPM_INSTALL_DEV:
+#       Set to 1 to also install devDependencies.
+#
+#  NPM_REGISTRY:
+#       Use the specified registry.
+#
+#  NPM_ARCH:
+#       Override the auto generated npm architecture.
+#
+#  NPM_INSTALL_EXTRA_ARGS:
+#       Add extra arguments to the 'npm install' execution.
+#       Use it at your own risk.
+
 DEPENDS_prepend = "nodejs-native "
 RDEPENDS_${PN}_prepend = "nodejs "
-S = "${WORKDIR}/npmpkg"
 
-def node_pkgname(d):
-    bpn = d.getVar('BPN')
-    if bpn.startswith("node-"):
-        return bpn[5:]
-    return bpn
+NPM_SHRINKWRAP ?= "${THISDIR}/${BPN}/npm-shrinkwrap.json"
 
-NPMPN ?= "${@node_pkgname(d)}"
+NPM_INSTALL_DEV ?= "0"
 
-NPM_INSTALLDIR = "${libdir}/node_modules/${NPMPN}"
+NPM_REGISTRY ?= "http://registry.npmjs.org"
 
 # function maps arch names to npm arch names
-def npm_oe_arch_map(target_arch, d):
+def npm_oe_arch_map(target_arch):
     import re
     if   re.match('p(pc|owerpc)(|64)', target_arch): return 'ppc'
     elif re.match('i.86$', target_arch): return 'ia32'
@@ -21,74 +46,170 @@ def npm_oe_arch_map(target_arch, d):
     elif re.match('arm64$', target_arch): return 'arm'
     return target_arch
 
-NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'), d)}"
-NPM_INSTALL_DEV ?= "0"
+NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'))}"
+
+NPM_INSTALL_EXTRA_ARGS ?= ""
+
+NPM_CACHE ?= "${DL_DIR}/npm_cache"
+
+B = "${WORKDIR}/build"
+
+python do_fetch_append() {
+    import json
+    from bb.fetch2 import check_network_access
+    from bb.fetch2 import runfetchcmd
+
+    basecmd = "npm cache"
+    basecmd += d.expand(" --cache=${NPM_CACHE}")
+    basecmd += d.expand(" --registry=${NPM_REGISTRY}")
+
+    # Parse the shrinkwrap file
+    def get_shrinkwrap():
+        src_shrinkwrap = d.expand("${S}/npm-shrinkwrap.json")
+        src_package_lock = d.expand("${S}/package-lock.json")
+        npm_shrinkwrap = d.getVar("NPM_SHRINKWRAP")
+
+        if os.path.exists(src_shrinkwrap):
+            return src_shrinkwrap
+        elif os.path.exists(src_package_lock):
+            return src_package_lock
+        elif os.path.exists(npm_shrinkwrap):
+            return npm_shrinkwrap
+        else:
+            bb.fatal("No mandatory NPM_SHRINKWRAP file found")
+
+    with open(get_shrinkwrap(), "r") as f:
+        shrinkwrap = json.load(f)
+
+    # Cache dependencies
+    def cache_npm_dependencies(dependencies):
+        for name in dependencies:
+            version = dependencies[name]["version"]
+            cmd = basecmd + " add '{}@{}'".format(name, version)
+            check_network_access(d, cmd, d.getVar("NPM_REGISTRY"))
+            runfetchcmd(cmd, d)
+            cache_npm_dependencies(dependencies[name].get("dependencies", {}))
+
+    cache_npm_dependencies(shrinkwrap.get("dependencies", {}))
+
+    # Verify the cache
+    runfetchcmd(basecmd + " verify", d)
+}
+
+do_fetch[depends] = "nodejs-native:do_populate_sysroot"
+
+npm_install_shrinkwrap() {
+    # This function ensures that there is a shrinkwrap file in the specified
+    # directory. A shrinkwrap file is mandatory to have reproducible builds.
+    # If the shrinkwrap file is not already included in the sources,
+    # the recipe can provide one by using the NPM_SHRINKWRAP option.
+    # This function returns the filename of the installed file (if any).
+    if [ -f ${S}/npm-shrinkwrap.json ]
+    then
+        bbnote "Using the npm-shrinkwrap.json provided in the sources"
+    elif [ -f ${S}/package-lock.json ]
+    then
+        bbnote "Using the package-lock.json provided in the sources"
+    elif [ -f ${NPM_SHRINKWRAP} ]
+    then
+        install -m 644 ${NPM_SHRINKWRAP} ${S}/npm-shrinkwrap.json
+        echo ${S}/npm-shrinkwrap.json
+    else
+        bbfatal "No mandatory NPM_SHRINKWRAP file found"
+    fi
+}
 
 npm_do_compile() {
-	# Copy in any additionally fetched modules
-	if [ -d ${WORKDIR}/node_modules ] ; then
-		cp -a ${WORKDIR}/node_modules ${S}/
-	fi
-	# changing the home directory to the working directory, the .npmrc will
-	# be created in this directory
-	export HOME=${WORKDIR}
-	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-		npm config set dev true
-	else
-		npm config set dev false
-	fi
-	npm set cache ${WORKDIR}/npm_cache
-	# clear cache before every build
-	npm cache clear --force
-	# Install pkg into ${S} without going to the registry
-	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --no-registry install
-	else
-		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
-	fi
+    # This function executes the 'npm install' command which builds and
+    # installs every dependencies needed for the package. All the files are
+    # installed in a build directory ${B} without filtering anything. To do so,
+    # a combination of 'npm pack' and 'npm install' is used to ensure that the
+    # files in ${B} are actual copies instead of symbolic links (which is the
+    # default npm behavior).
+
+    # First ensure that there is a shrinkwrap file in the sources.
+    local NPM_SHRINKWRAP_INSTALLED=$(npm_install_shrinkwrap)
+
+    # Then create a tarball from a npm package whose sources must be in ${S}.
+    local NPM_PACK_FILE=$(cd ${WORKDIR} && npm pack --offline ${S}/)
+
+    # Clean source tree.
+    rm -f ${NPM_SHRINKWRAP_INSTALLED}
+
+    # Remove the build directory.
+    rm -rf ${B}
+
+    # Finally install and build the tarball package in ${B}.
+    local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --cache=${NPM_CACHE}"
+    local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --offline"
+    local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --loglevel silly"
+    local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --prefix=${B}"
+    local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --global"
+
+    if [ "${NPM_INSTALL_DEV}" != 1 ]
+    then
+        local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --production"
+    fi
+
+    local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --arch=${NPM_ARCH}"
+    local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --target_arch=${NPM_ARCH}"
+    local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --release"
+
+    cd ${WORKDIR} && npm install \
+        ${NPM_INSTALL_EXTRA_ARGS} \
+        ${NPM_INSTALL_GYP_ARGS} \
+        ${NPM_INSTALL_ARGS} \
+        ${NPM_PACK_FILE}
 }
 
 npm_do_install() {
-	# changing the home directory to the working directory, the .npmrc will
-	# be created in this directory
-	export HOME=${WORKDIR}
-	mkdir -p ${D}${libdir}/node_modules
-	local NPM_PACKFILE=$(npm pack .)
-	npm install --prefix ${D}${prefix} -g --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry ${NPM_PACKFILE}
-	ln -fs node_modules ${D}${libdir}/node
-	find ${D}${NPM_INSTALLDIR} -type f \( -name "*.a" -o -name "*.d" -o -name "*.o" \) -delete
-	if [ -d ${D}${prefix}/etc ] ; then
-		# This will be empty
-		rmdir ${D}${prefix}/etc
-	fi
-}
+    # This function creates the destination directory from the pre installed
+    # files in the ${B} directory.
+
+    # Copy the entire lib and bin directories from ${B} to ${D}.
+    install -d ${D}/${libdir}
+    cp --no-preserve=ownership --recursive ${B}/lib/. ${D}/${libdir}
+
+    if [ -d "${B}/bin" ]
+    then
+        install -d ${D}/${bindir}
+        cp --no-preserve=ownership --recursive ${B}/bin/. ${D}/${bindir}
+    fi
+
+    # If the package (or its dependencies) uses node-gyp to build native addons,
+    # object files, static libraries or other temporary files can be hidden in
+    # the lib directory. To reduce the package size and to avoid QA issues
+    # (staticdev with static library files) these files must be removed.
+
+    # Remove any node-gyp directory in ${D} to remove temporary build files.
+    for GYP_D_FILE in $(find ${D} -regex ".*/build/Release/[^/]*.node")
+    do
+        local GYP_D_DIR=${GYP_D_FILE%/Release/*}
+
+        rm --recursive --force ${GYP_D_DIR}
+    done
+
+    # Copy only the node-gyp release files from ${B} to ${D}.
+    for GYP_B_FILE in $(find ${B} -regex ".*/build/Release/[^/]*.node")
+    do
+        local GYP_D_FILE=${D}/${prefix}/${GYP_B_FILE#${B}}
+
+        install -d ${GYP_D_FILE%/*}
+        install -m 755 ${GYP_B_FILE} ${GYP_D_FILE}
+    done
+
+    # Remove the shrinkwrap file which does not need to be packed.
+    rm -f ${D}/${libdir}/node_modules/*/npm-shrinkwrap.json
+    rm -f ${D}/${libdir}/node_modules/@*/*/npm-shrinkwrap.json
 
-python populate_packages_prepend () {
-    instdir = d.expand('${D}${NPM_INSTALLDIR}')
-    extrapackages = oe.package.npm_split_package_dirs(instdir)
-    pkgnames = extrapackages.keys()
-    d.prependVar('PACKAGES', '%s ' % ' '.join(pkgnames))
-    for pkgname in pkgnames:
-        pkgrelpath, pdata = extrapackages[pkgname]
-        pkgpath = '${NPM_INSTALLDIR}/' + pkgrelpath
-        # package names can't have underscores but npm packages sometimes use them
-        oe_pkg_name = pkgname.replace('_', '-')
-        expanded_pkgname = d.expand(oe_pkg_name)
-        d.setVar('FILES_%s' % expanded_pkgname, pkgpath)
-        if pdata:
-            version = pdata.get('version', None)
-            if version:
-                d.setVar('PKGV_%s' % expanded_pkgname, version)
-            description = pdata.get('description', None)
-            if description:
-                d.setVar('SUMMARY_%s' % expanded_pkgname, description.replace(u"\u2018", "'").replace(u"\u2019", "'"))
-    d.appendVar('RDEPENDS_%s' % d.getVar('PN'), ' %s' % ' '.join(pkgnames).replace('_', '-'))
+    # node(1) is using /usr/lib/node as default include directory and npm(1) is
+    # using /usr/lib/node_modules as install directory. Let's make both happy.
+    ln -fs node_modules ${D}/${libdir}/node
 }
 
 FILES_${PN} += " \
     ${bindir} \
-    ${libdir}/node \
-    ${NPM_INSTALLDIR} \
+    ${libdir} \
 "
 
 EXPORT_FUNCTIONS do_compile do_install
-- 
2.20.1

[RFC][PATCH v2 2/7] devtool: update command line options for npm

[Jean-Marie LEMETAYER]

This commit renames the '--fetch-dev' option into '--npm-dev' which is
more easily understandable.

It also adds the '--npm-registry' option to allow creating a npm recipe
with a non default npm registry (e.g. if the SRC_URI is using git://).

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/devtool/standard.py  |  9 ++++++---
 scripts/lib/recipetool/create.py | 12 +++++++-----
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8d9c1a3022..7068a02a01 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -145,8 +145,10 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' --src-subdir "%s"' % args.src_subdir
     if args.autorev:
         extracmdopts += ' -a'
-    if args.fetch_dev:
-        extracmdopts += ' --fetch-dev'
+    if args.npm_dev:
+        extracmdopts += ' --npm-dev'
+    if args.npm_registry:
+        extracmdopts += ' --npm-registry "%s"' % args.npm_registry
     if args.mirrors:
         extracmdopts += ' --mirrors'
     if args.srcrev:
@@ -2197,7 +2199,8 @@ def register_commands(subparsers, context):
     group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
-    parser_add.add_argument('--fetch-dev', help='For npm, also fetch devDependencies', action="store_true")
+    parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true")
+    parser_add.add_argument('--npm-registry', help='For npm, use the specified registry', type=str)
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     group = parser_add.add_mutually_exclusive_group()
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 1fb6b55530..932dc3f374 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -716,10 +716,11 @@ def create_recipe(args):
         lines_after.append('INSANE_SKIP_${PN} += "already-stripped"')
         lines_after.append('')
 
-    if args.fetch_dev:
-        extravalues['fetchdev'] = True
-    else:
-        extravalues['fetchdev'] = None
+    if args.npm_dev:
+        extravalues['NPM_INSTALL_DEV'] = 1
+
+    if args.npm_registry:
+        extravalues['NPM_REGISTRY'] = args.npm_registry
 
     # Find all plugins that want to register handlers
     logger.debug('Loading recipe handlers')
@@ -1315,7 +1316,8 @@ def register_commands(subparsers):
     group.add_argument('-S', '--srcrev', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
     parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
-    parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies')
+    parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies')
+    parser_create.add_argument('--npm-registry', help='For npm, use the specified registry', type=str)
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
     parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
     parser_create.set_defaults(func=create_recipe)
-- 
2.20.1

[RFC][PATCH v2 3/7] recipetool/create_npm.py: refactor the npm recipe creation handler

[Jean-Marie LEMETAYER]

This commit refactors the npm recipe creation handler to use the new npm
behavior. The process is kept as simple as possible and only generates
the shrinkwrap file.

To avoid naming issues the recipe name is now extracted from the npm
package name and not directly map.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create_npm.py | 443 ++++++++++-----------------
 1 file changed, 167 insertions(+), 276 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 39429ebad3..2cf67e004b 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -1,321 +1,212 @@
-# Recipe creation tool - node.js NPM module support plugin
-#
 # Copyright (C) 2016 Intel Corporation
+# Copyright (C) 2019 Savoir-Faire Linux
 #
 # SPDX-License-Identifier: GPL-2.0-only
 #
+"""
+    Recipe creation tool - npm module support plugin
+"""
 
+import json
+import logging
 import os
+import re
+import shutil
 import sys
-import logging
-import subprocess
 import tempfile
-import shutil
-import json
-from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars
+import bb
+from bb.fetch2 import runfetchcmd
+from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
 
-
 tinfoil = None
 
 def tinfoil_init(instance):
+    """
+        Initialize tinfoil.
+    """
+
     global tinfoil
     tinfoil = instance
 
-
 class NpmRecipeHandler(RecipeHandler):
-    lockdownpath = None
+    """
+        Class to handle the npm recipe creation
+    """
+
+    @staticmethod
+    def _ensure_npm(d):
+        """
+            Check if the 'npm' command is available in the recipes, then build
+            it and add it to the PATH.
+        """
 
-    def _ensure_npm(self, fixed_setup=False):
         if not tinfoil.recipes_parsed:
             tinfoil.parse_recipes()
         try:
             rd = tinfoil.parse_recipe('nodejs-native')
         except bb.providers.NoProvider:
-            if fixed_setup:
-                msg = 'nodejs-native is required for npm but is not available within this SDK'
-            else:
-                msg = 'nodejs-native is required for npm but is not available - you will likely need to add a layer that provides nodejs'
-            logger.error(msg)
-            return None
+            logger.error("Nothing provides 'nodejs-native' which is required for the build")
+            logger.info("You will likely need to add a layer that provides nodejs")
+            sys.exit(14)
+
         bindir = rd.getVar('STAGING_BINDIR_NATIVE')
         npmpath = os.path.join(bindir, 'npm')
         if not os.path.exists(npmpath):
             tinfoil.build_targets('nodejs-native', 'addto_recipe_sysroot')
             if not os.path.exists(npmpath):
-                logger.error('npm required to process specified source, but nodejs-native did not seem to populate it')
-                return None
-        return bindir
-
-    def _handle_license(self, data):
-        '''
-        Handle the license value from an npm package.json file
-        '''
-        license = None
-        if 'license' in data:
-            license = data['license']
-            if isinstance(license, dict):
-                license = license.get('type', None)
-            if license:
-                if 'OR' in license:
-                    license = license.replace('OR', '|')
-                    license = license.replace('AND', '&')
-                    license = license.replace(' ', '_')
-                    if not license[0] == '(':
-                        license = '(' + license + ')'
-                else:
-                    license = license.replace('AND', '&')
-                    if license[0] == '(':
-                        license = license[1:]
-                    if license[-1] == ')':
-                        license = license[:-1]
-                license = license.replace('MIT/X11', 'MIT')
-                license = license.replace('Public Domain', 'PD')
-                license = license.replace('SEE LICENSE IN EULA',
-                                          'SEE-LICENSE-IN-EULA')
-        return license
-
-    def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before, d):
-        try:
-            runenv = dict(os.environ, PATH=d.getVar('PATH'))
-            bb.process.run('npm shrinkwrap', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        except bb.process.ExecutionError as e:
-            logger.warning('npm shrinkwrap failed:\n%s' % e.stdout)
-            return
+                logger.error("Failed to add 'npm' to sysroot")
+                sys.exit(14)
 
-        tmpfile = os.path.join(localfilesdir, 'npm-shrinkwrap.json')
-        shutil.move(os.path.join(srctree, 'npm-shrinkwrap.json'), tmpfile)
-        extravalues.setdefault('extrafiles', {})
-        extravalues['extrafiles']['npm-shrinkwrap.json'] = tmpfile
-        lines_before.append('NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"')
-
-    def _lockdown(self, srctree, localfilesdir, extravalues, lines_before, d):
-        runenv = dict(os.environ, PATH=d.getVar('PATH'))
-        if not NpmRecipeHandler.lockdownpath:
-            NpmRecipeHandler.lockdownpath = tempfile.mkdtemp('recipetool-npm-lockdown')
-            bb.process.run('npm install lockdown --prefix %s' % NpmRecipeHandler.lockdownpath,
-                           cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        relockbin = os.path.join(NpmRecipeHandler.lockdownpath, 'node_modules', 'lockdown', 'relock.js')
-        if not os.path.exists(relockbin):
-            logger.warning('Could not find relock.js within lockdown directory; skipping lockdown')
-            return
-        try:
-            bb.process.run('node %s' % relockbin, cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        except bb.process.ExecutionError as e:
-            logger.warning('lockdown-relock failed:\n%s' % e.stdout)
-            return
+        d.prependVar("PATH", "{}:".format(bindir))
+
+    @staticmethod
+    def _run_npm_install(d, srctree, registry, development):
+        """
+            Run the 'npm install' command without building the sources (if any).
+            This is only needed to generate the npm-shrinkwrap.json file.
+        """
+
+        cmd = "npm install"
+        cmd += " --ignore-scripts"
+        cmd += d.expand(" --cache=${DL_DIR}/npm_cache")
+        cmd += " --registry {}".format(registry)
+
+        if development is None:
+            cmd += " --production"
+
+        runfetchcmd(cmd, d, workdir=srctree)
+
+    @staticmethod
+    def _run_npm_shrinkwrap(d, srctree, development):
+        """
+            Run the 'npm shrinkwrap' command.
+        """
 
-        tmpfile = os.path.join(localfilesdir, 'lockdown.json')
-        shutil.move(os.path.join(srctree, 'lockdown.json'), tmpfile)
-        extravalues.setdefault('extrafiles', {})
-        extravalues['extrafiles']['lockdown.json'] = tmpfile
-        lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
+        cmd = "npm shrinkwrap"
+
+        if development is not None:
+            cmd += " --development"
+
+        runfetchcmd(cmd, d, workdir=srctree)
+
+    def _generate_shrinkwrap(self, srctree, lines_before, lines_after, extravalues):
+        """
+            Check and generate the npm-shrinkwrap.json file if needed.
+        """
+
+        # Are we using the '--npm-dev' option ?
+        development = extravalues.get("NPM_INSTALL_DEV")
+
+        # Are we using the '--npm-registry' option ?
+        registry_option = extravalues.get("NPM_REGISTRY")
+
+        # Get the registry from the fetch url if using 'npm://registry.url'
+        registry_fetch = None
 
-    def _handle_dependencies(self, d, deps, optdeps, devdeps, lines_before, srctree):
-        import scriptutils
-        # If this isn't a single module we need to get the dependencies
-        # and add them to SRC_URI
         def varfunc(varname, origvalue, op, newlines):
-            if varname == 'SRC_URI':
-                if not origvalue.startswith('npm://'):
-                    src_uri = origvalue.split()
-                    deplist = {}
-                    for dep, depver in optdeps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        if self.check_npm_optional_dependency(depdata):
-                            deplist[dep] = depdata
-                    for dep, depver in devdeps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        if self.check_npm_optional_dependency(depdata):
-                            deplist[dep] = depdata
-                    for dep, depver in deps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        deplist[dep] = depdata
-
-                    extra_urls = []
-                    for dep, depdata in deplist.items():
-                        version = depdata.get('version', None)
-                        if version:
-                            url = 'npm://registry.npmjs.org;name=%s;version=%s;subdir=node_modules/%s' % (dep, version, dep)
-                            extra_urls.append(url)
-                    if extra_urls:
-                        scriptutils.fetch_url(tinfoil, ' '.join(extra_urls), None, srctree, logger)
-                        src_uri.extend(extra_urls)
-                        return src_uri, None, -1, True
+            if varname == "SRC_URI":
+                if origvalue.startswith("npm://"):
+                    nonlocal registry_fetch
+                    registry_fetch = origvalue.replace("npm://", "http://", 1).split(";")[0]
             return origvalue, None, 0, True
-        updated, newlines = bb.utils.edit_metadata(lines_before, ['SRC_URI'], varfunc)
-        if updated:
-            del lines_before[:]
-            for line in newlines:
-                # Hack to avoid newlines that edit_metadata inserts
-                if line.endswith('\n'):
-                    line = line[:-1]
-                lines_before.append(line)
-        return updated
+
+        bb.utils.edit_metadata(lines_before, ["SRC_URI"], varfunc)
+
+        # Compute the proper registry value
+        if registry_fetch is not None:
+            registry = registry_fetch
+
+            if registry_option is not None:
+                logger.warning("The npm registry is specified multiple times")
+                logger.info("Using registry from the fetch url: '{}'".format(registry))
+                extravalues.pop("NPM_REGISTRY", None)
+
+        elif registry_option is not None:
+            registry = registry_option
+
+        else:
+            registry = "http://registry.npmjs.org"
+
+        # Initialize the npm environment
+        d = bb.data.createCopy(tinfoil.config_data)
+        self._ensure_npm(d)
+
+        # Check if a shinkwrap file is already in the source
+        if os.path.exists(os.path.join(srctree, "npm-shrinkwrap.json")):
+            logger.info("Using the npm-shrinkwrap.json provided in the sources")
+            return
+
+        # Generate the 'npm-shrinkwrap.json' file
+        self._run_npm_install(d, srctree, registry, development)
+        self._run_npm_shrinkwrap(d, srctree, development)
+
+        # Save the shrinkwrap file in a temporary location
+        tmpdir = tempfile.mkdtemp(prefix="recipetool-npm")
+        tmpfile = os.path.join(tmpdir, "npm-shrinkwrap.json")
+        shutil.move(os.path.join(srctree, "npm-shrinkwrap.json"), tmpfile)
+
+        # Add the shrinkwrap file as 'extrafiles'
+        extravalues.setdefault("extrafiles", {})
+        extravalues["extrafiles"]["npm-shrinkwrap.json"] = tmpfile
+
+        # Add a line in the recipe to handle the shrinkwrap file
+        lines_after.append("NPM_SHRINKWRAP = \"${THISDIR}/${BPN}/npm-shrinkwrap.json\"")
+
+        # Remove the 'node_modules' directory generated by 'npm install'
+        bb.utils.remove(os.path.join(srctree, "node_modules"), recurse=True)
+
+    @staticmethod
+    def _recipe_name_from_npm(name):
+        """
+            Generate a recipe name based on the npm package name.
+        """
+
+        name = name.lower()
+        name = re.sub("/", "-", name)
+        name = re.sub("[^a-z\-]", "", name)
+        name = name.strip("-")
+        return name
 
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
-        import bb.utils
-        import oe.package
-        from collections import OrderedDict
+        """
+            Handle the npm recipe creation
+        """
 
         if 'buildsystem' in handled:
             return False
 
-        def read_package_json(fn):
-            with open(fn, 'r', errors='surrogateescape') as f:
-                return json.loads(f.read())
+        files = RecipeHandler.checkfiles(srctree, ["package.json"])
 
-        files = RecipeHandler.checkfiles(srctree, ['package.json'])
-        if files:
-            d = bb.data.createCopy(tinfoil.config_data)
-            npm_bindir = self._ensure_npm()
-            if not npm_bindir:
-                sys.exit(14)
-            d.prependVar('PATH', '%s:' % npm_bindir)
-
-            data = read_package_json(files[0])
-            if 'name' in data and 'version' in data:
-                extravalues['PN'] = data['name']
-                extravalues['PV'] = data['version']
-                classes.append('npm')
-                handled.append('buildsystem')
-                if 'description' in data:
-                    extravalues['SUMMARY'] = data['description']
-                if 'homepage' in data:
-                    extravalues['HOMEPAGE'] = data['homepage']
-
-                fetchdev = extravalues['fetchdev'] or None
-                deps, optdeps, devdeps = self.get_npm_package_dependencies(data, fetchdev)
-                self._handle_dependencies(d, deps, optdeps, devdeps, lines_before, srctree)
-
-                # Shrinkwrap
-                localfilesdir = tempfile.mkdtemp(prefix='recipetool-npm')
-                self._shrinkwrap(srctree, localfilesdir, extravalues, lines_before, d)
-
-                # Lockdown
-                self._lockdown(srctree, localfilesdir, extravalues, lines_before, d)
-
-                # Split each npm module out to is own package
-                npmpackages = oe.package.npm_split_package_dirs(srctree)
-                licvalues = None
-                for item in handled:
-                    if isinstance(item, tuple):
-                        if item[0] == 'license':
-                            licvalues = item[1]
-                            break
-                if not licvalues:
-                    licvalues = handle_license_vars(srctree, lines_before, handled, extravalues, d)
-                if licvalues:
-                    # Augment the license list with information we have in the packages
-                    licenses = {}
-                    license = self._handle_license(data)
-                    if license:
-                        licenses['${PN}'] = license
-                    for pkgname, pkgitem in npmpackages.items():
-                        _, pdata = pkgitem
-                        license = self._handle_license(pdata)
-                        if license:
-                            licenses[pkgname] = license
-                    # Now write out the package-specific license values
-                    # We need to strip out the json data dicts for this since split_pkg_licenses
-                    # isn't expecting it
-                    packages = OrderedDict((x,y[0]) for x,y in npmpackages.items())
-                    packages['${PN}'] = ''
-                    pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
-                    all_licenses = list(set([item.replace('_', ' ') for pkglicense in pkglicenses.values() for item in pkglicense]))
-                    if '&' in all_licenses:
-                        all_licenses.remove('&')
-                    extravalues['LICENSE'] = ' & '.join(all_licenses)
-
-                # Need to move S setting after inherit npm
-                for i, line in enumerate(lines_before):
-                    if line.startswith('S ='):
-                        lines_before.pop(i)
-                        lines_after.insert(0, '# Must be set after inherit npm since that itself sets S')
-                        lines_after.insert(1, line)
-                        break
-
-                return True
-
-        return False
-
-    # FIXME this is duplicated from lib/bb/fetch2/npm.py
-    def _parse_view(self, output):
-        '''
-        Parse the output of npm view --json; the last JSON result
-        is assumed to be the one that we're interested in.
-        '''
-        pdata = None
-        outdeps = {}
-        datalines = []
-        bracelevel = 0
-        for line in output.splitlines():
-            if bracelevel:
-                datalines.append(line)
-            elif '{' in line:
-                datalines = []
-                datalines.append(line)
-            bracelevel = bracelevel + line.count('{') - line.count('}')
-        if datalines:
-            pdata = json.loads('\n'.join(datalines))
-        return pdata
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def get_npm_data(self, pkg, version, d):
-        import bb.fetch2
-        pkgfullname = pkg
-        if version != '*' and not '/' in version:
-            pkgfullname += "@'%s'" % version
-        logger.debug(2, "Calling getdeps on %s" % pkg)
-        runenv = dict(os.environ, PATH=d.getVar('PATH'))
-        fetchcmd = "npm view %s --json" % pkgfullname
-        output, _ = bb.process.run(fetchcmd, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        data = self._parse_view(output)
-        return data
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def get_npm_package_dependencies(self, pdata, fetchdev):
-        dependencies = pdata.get('dependencies', {})
-        optionalDependencies = pdata.get('optionalDependencies', {})
-        dependencies.update(optionalDependencies)
-        if fetchdev:
-            devDependencies = pdata.get('devDependencies', {})
-            dependencies.update(devDependencies)
-        else:
-            devDependencies = {}
-        depsfound = {}
-        optdepsfound = {}
-        devdepsfound = {}
-        for dep in dependencies:
-            if dep in optionalDependencies:
-                optdepsfound[dep] = dependencies[dep]
-            elif dep in devDependencies:
-                devdepsfound[dep] = dependencies[dep]
-            else:
-                depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound, devdepsfound
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def check_npm_optional_dependency(self, pdata):
-        pkg_os = pdata.get('os', None)
-        if pkg_os:
-            if not isinstance(pkg_os, list):
-                pkg_os = [pkg_os]
-            blacklist = False
-            for item in pkg_os:
-                if item.startswith('!'):
-                    blacklist = True
-                    break
-            if (not blacklist and 'linux' not in pkg_os) or '!linux' in pkg_os:
-                pkg = pdata.get('name', 'Unnamed package')
-                logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg)
-                return False
-        return True
+        if not files:
+            return False
+
+        with open(files[0], "r", errors="surrogateescape") as f:
+            data = json.load(f)
+
+        if "name" not in data or "version" not in data:
+            return False
 
+        self._generate_shrinkwrap(srctree, lines_before, lines_after, extravalues)
+
+        extravalues["PN"] = self._recipe_name_from_npm(data["name"])
+        extravalues["PV"] = data["version"]
+
+        if "description" in data:
+            extravalues["SUMMARY"] = data["description"]
+
+        if "homepage" in data:
+            extravalues["HOMEPAGE"] = data["homepage"]
+
+        classes.append("npm")
+        handled.append("buildsystem")
+
+        return True
 
 def register_recipe_handlers(handlers):
+    """
+        Register the npm handler
+    """
+
     handlers.append((NpmRecipeHandler(), 60))
-- 
2.20.1

[RFC][PATCH v2 4/7] devtool/standard.py: update the append file for the npm recipes

[Jean-Marie LEMETAYER]

When creating a recipe using devtool, a workspace is created to store
the new recipe, the recipe source and some append files. These append
files are used by devtool to build the recipe using externalsrc (to use
the source which are in the workspace). They can also have some
additional actions according to the class of the recipe.

This commit updates the append file for the npm recipes. The
devtool / externalsrc files are removed in the npm build directory
instead of the install directory.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/devtool/standard.py | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 7068a02a01..2604b79be3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -262,14 +262,11 @@ def add(args, config, basepath, workspace):
                 f.write('}\n')
 
             if bb.data.inherits_class('npm', rd):
-                f.write('do_install_append() {\n')
-                f.write('    # Remove files added to source dir by devtool/externalsrc\n')
-                f.write('    rm -f ${NPM_INSTALLDIR}/singletask.lock\n')
-                f.write('    rm -rf ${NPM_INSTALLDIR}/.git\n')
-                f.write('    rm -rf ${NPM_INSTALLDIR}/oe-local-files\n')
-                f.write('    for symlink in ${EXTERNALSRC_SYMLINKS} ; do\n')
-                f.write('        rm -f ${NPM_INSTALLDIR}/${symlink%%:*}\n')
-                f.write('    done\n')
+                f.write('do_compile_append() {\n')
+                f.write('    rm -rf ${B}/lib/node_modules/*/.git\n')
+                f.write('    rm -rf ${B}/lib/node_modules/@*/*/.git\n')
+                f.write('    rm -f ${B}/lib/node_modules/*/singletask.lock\n')
+                f.write('    rm -f ${B}/lib/node_modules/@*/*/singletask.lock\n')
                 f.write('}\n')
 
         # Check if the new layer provides recipes whose priorities have been
-- 
2.20.1

[RFC][PATCH v2 5/7] recipetool/create.py: replace 'latest' keyword for npm

[Jean-Marie LEMETAYER]

The new npm fetcher allows the 'latest' keyword to be used to download
the latest version on the registry. But the keyword must be replace
as soon as the version is determined to have a stable generated recipe.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 932dc3f374..e49e73ed9b 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -833,6 +833,8 @@ def create_recipe(args):
         elif line.startswith('SRC_URI = '):
             if realpv and not pv_srcpv:
                 line = line.replace(realpv, '${PV}')
+            if scheme == 'npm':
+                line = line.replace('version=latest', 'version=${PV}')
         elif line.startswith('PV = '):
             if realpv:
                 # Replace the first part of the PV value
-- 
2.20.1

[RFC][PATCH v2 6/7] recipetool/create.py: remove the 'noverify' url parameter

[Jean-Marie LEMETAYER]

This commit removes the 'noverify' parameter which was added to the url
to fix warnings with the shrinkwrap / lockdown file generation. This is
not needed anymore with the new npm fetcher.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index e49e73ed9b..b4ab1117b8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -477,8 +477,6 @@ def create_recipe(args):
             storeTagName = params['tag']
             params['nobranch'] = '1'
             del params['tag']
-        if scheme == 'npm':
-            params['noverify'] = '1'
         fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
-- 
2.20.1

[RFC][PATCH v2 7/7] oeqa/selftest/recipetool: add npm recipe creation test

[Jean-Marie LEMETAYER]

This commit adds a recipetool creation test for npm recipe.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index c1562c63b2..71613fd2fb 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -421,6 +421,25 @@ class RecipetoolTests(RecipetoolBase):
         inherits = ['cmake']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+    def test_recipetool_create_npm(self):
+        temprecipe = os.path.join(self.tempdir, 'recipe')
+        os.makedirs(temprecipe)
+        recipefile = os.path.join(temprecipe, 'savoirfairelinux-node-server-example_1.0.0.bb')
+        shrinkwrap = os.path.join(temprecipe, 'savoirfairelinux-node-server-example', 'npm-shrinkwrap.json')
+        srcuri = 'npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0'
+        result = runCmd('recipetool create -o %s \'%s\'' % (temprecipe, srcuri))
+        self.assertTrue(os.path.isfile(recipefile))
+        self.assertTrue(os.path.isfile(shrinkwrap))
+        checkvars = {}
+        checkvars['SUMMARY'] = 'Node Server Example'
+        checkvars['HOMEPAGE'] = 'https://github.com/savoirfairelinux/node-server-example#readme'
+        checkvars['LICENSE'] = set(['MIT'])
+        checkvars['SRC_URI'] = 'npm://registry.npmjs.org/;name=@savoirfairelinux/node-server-example;version=${PV}'
+        checkvars['S'] = '${WORKDIR}/npm'
+        checkvars['NPM_SHRINKWRAP'] = '${THISDIR}/${BPN}/npm-shrinkwrap.json'
+        inherits = ['npm']
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
     def test_recipetool_create_github(self):
         # Basic test to see if github URL mangling works
         temprecipe = os.path.join(self.tempdir, 'recipe')
-- 
2.20.1