[PATCH v4 00/13] NPM refactoring

[PATCH v4 00/13] NPM refactoring

[Jean-Marie LEMETAYER]

Hi folks and happy new year,

For readability here is a link if you want the history of this patchset:
http://lists.openembedded.org/pipermail/openembedded-core/2019-December/290298.html

--- V4

The patches can be found here:
https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v4

 - Two fetchers are available: npm and npmsw

 - The npm fetcher is used to fetch a single package from a registry.

 - The npmsw fetcher is used to fetch the dependencies of a package. It uses a
   npm shrinkwrap file to list the dependencies. The file can be edited
   manually if some dependencies needs to be updated.

 - Both fetcher are independent.

   For example you can have a main package using npm:
     SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
                npmsw://${THISDIR}/npm-shrinkwrap.json"

   The main package can also be fetched another way without changing anything:
     SRC_URI = "git://github.com/foo/bar.git;protocol=https \
                npmsw://${THISDIR}/npm-shrinkwrap.json"

 - The npm class is in charge of executing the 'npm install'. This installation
   is executed completely offline. The package sources can be patched safely
   before the build.

 - The npm class populates a local npm cache during the do_configure task. As
   this step can take time the support of a progress bar have been added.

 - The devtool and recipetool scripts have been updated to match the new
   fetchers and class. The licences are handled and split by dependencies.

 - Both fetcher, the recipe creation and the recipe build are covered by tests.

 - Using devtool I have successfully generated and built recipe for:
   - My test application:
       https://github.com/savoirfairelinux/node-server-example
   - The angular cli which will be used to build angular applications:
       https://cli.angular.io
   - The mozilla iot gateway which is a big project with a lot of dependencies:
       https://github.com/mozilla-iot/gateway
     To build this project you have to manually add this line in the recipe:
       DEPENDS = "nanomsg sqlite3 libuv"

Jean-Marie LEMETAYER (13):
  classes/npm: refactor the npm class
  classes/npm: restrict the build to be offline
  classes/npm: use the local node headers
  classes/npm: use the native python
  classes/npm: force to rebuild the prebuild addons
  devtool: npm: rename npm command line options
  recipetool/create_npm: refactor the npm recipe creation handler
  recipetool/create_npm: handle the licenses of the dependencies
  lib/oe/package: remove unneeded npm_split_package_dirs function
  devtool/standard: npm: update the append file
  recipetool/create: npm: remove the 'noverify' url parameter
  oeqa/selftest/recipetool: add npm recipe creation test
  oeqa/selftest/devtool: add npm recipe build test

 meta/classes/npm.bbclass                   | 360 ++++++++++++---
 meta/lib/oe/package.py                     |  33 --
 meta/lib/oeqa/selftest/cases/devtool.py    |  20 +
 meta/lib/oeqa/selftest/cases/recipetool.py |  25 +
 scripts/lib/devtool/standard.py            |  18 +-
 scripts/lib/recipetool/create.py           |  10 +-
 scripts/lib/recipetool/create_npm.py       | 514 +++++++++------------
 7 files changed, 564 insertions(+), 416 deletions(-)

--
2.20.1

[PATCH v4 01/13] classes/npm: refactor the npm class

[Jean-Marie LEMETAYER]

This commit splits the npm build in three steps:

1. With the new npmsw fetcher, the sources and dependencies of the
   package have been fetched and unpacked. As sources can also be
   patched, a local cache must be configured to use these modified
   sources.

2. Next, the installation process is run using the local cache only.
   Some packages may need to be compiled.

3. The final installation filters the previously installed files to
   avoid unnecessary files.

This new version also fixes multiple issues related to npm dependencies
badly handled: package names, scope packages, installation directories

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 346 ++++++++++++++++++++++++++++++---------
 1 file changed, 271 insertions(+), 75 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 4b1f0a39f0..35b63122df 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -1,94 +1,290 @@
+# Copyright (C) 2020 Savoir-Faire Linux
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# This bbclass builds and installs an npm package to the target. The package
+# sources files should be fetched in the calling recipe by using the SRC_URI
+# variable. The ${S} variable should be updated depending of your fetcher.
+#
+# Usage:
+#  SRC_URI = "..."
+#  inherit npm
+#
+# Optional variables:
+#  NPM_ARCH:
+#       Override the auto generated npm architecture.
+#
+#  NPM_INSTALL_DEV:
+#       Set to 1 to also install devDependencies.
+
 DEPENDS_prepend = "nodejs-native "
 RDEPENDS_${PN}_prepend = "nodejs "
-S = "${WORKDIR}/npmpkg"
 
-def node_pkgname(d):
-    bpn = d.getVar('BPN')
-    if bpn.startswith("node-"):
-        return bpn[5:]
-    return bpn
+NPM_INSTALL_DEV ?= "0"
+
+def npm_target_arch_map(target_arch):
+    """Maps arch names to npm arch names"""
+    import re
+    if re.match("p(pc|owerpc)(|64)", target_arch):
+        return "ppc"
+    elif re.match("i.86$", target_arch):
+        return "ia32"
+    elif re.match("x86_64$", target_arch):
+        return "x64"
+    elif re.match("arm64$", target_arch):
+        return "arm"
+    return target_arch
+
+NPM_ARCH ?= "${@npm_target_arch_map(d.getVar("TARGET_ARCH"))}"
+
+NPM_PACKAGE = "${WORKDIR}/npm-package"
+NPM_CACHE = "${WORKDIR}/npm-cache"
+NPM_BUILD = "${WORKDIR}/npm-build"
 
-NPMPN ?= "${@node_pkgname(d)}"
+def npm_global_configs(d):
+    """Get the npm global configuration"""
+    configs = []
+    # Configure the cache directory
+    configs.append(("cache", d.getVar("NPM_CACHE")))
+    return configs
 
-NPM_INSTALLDIR = "${libdir}/node_modules/${NPMPN}"
+def npm_pack(env, srcdir, workdir):
+    """Run 'npm pack' on a specified directory"""
+    cmd = "npm pack '%s'" % srcdir
+    configs = [("ignore-scripts", "true")]
+    tarball = env.run(cmd, configs=configs, workdir=workdir).strip("\n")
+    return os.path.join(workdir, tarball)
 
-# function maps arch names to npm arch names
-def npm_oe_arch_map(target_arch, d):
+python npm_do_configure() {
+    """
+    Step one: configure the npm cache and the main npm package
+
+    Every dependencies have been fetched and patched in the source directory.
+    They have to be packed (this remove unneeded files) and added to the npm
+    cache to be available for the next step.
+
+    The main package and its associated manifest file and shrinkwrap file have
+    to be configured to take into account these cached dependencies.
+    """
+    import base64
+    import copy
+    import json
     import re
-    if   re.match('p(pc|owerpc)(|64)', target_arch): return 'ppc'
-    elif re.match('i.86$', target_arch): return 'ia32'
-    elif re.match('x86_64$', target_arch): return 'x64'
-    elif re.match('arm64$', target_arch): return 'arm'
-    return target_arch
+    import tempfile
+    from bb.fetch2.npm import NpmEnvironment
+    from bb.fetch2.npm import npm_unpack
+    from bb.fetch2.npmsw import foreach_dependencies
+    from bb.progress import OutOfProgressHandler
 
-NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'), d)}"
-NPM_INSTALL_DEV ?= "0"
+    bb.utils.remove(d.getVar("NPM_CACHE"), recurse=True)
+    bb.utils.remove(d.getVar("NPM_PACKAGE"), recurse=True)
+
+    env = NpmEnvironment(d, configs=npm_global_configs(d))
+
+    def _npm_cache_add(tarball):
+        """Run 'npm cache add' for a specified tarball"""
+        cmd = "npm cache add '%s'" % tarball
+        env.run(cmd)
+
+    def _npm_integrity(tarball):
+        """Return the npm integrity of a specified tarball"""
+        sha512 = bb.utils.sha512_file(tarball)
+        return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
+
+    def _npm_version(tarball):
+        """Return the version of a specified tarball"""
+        regex = r"-(\d+\.\d+\.\d+(-.*)?(\+.*)?)\.tgz"
+        return re.search(regex, tarball).group(1)
+
+    def _npmsw_dependency_dict(orig, deptree):
+        """
+        Return the sub dictionary in the 'orig' dictionary corresponding to the
+        'deptree' dependency tree. This function follows the shrinkwrap file
+        format.
+        """
+        ptr = orig
+        for dep in deptree:
+            if "dependencies" not in ptr:
+                ptr["dependencies"] = {}
+            ptr = ptr["dependencies"]
+            if dep not in ptr:
+                ptr[dep] = {}
+            ptr = ptr[dep]
+        return ptr
+
+    # Manage the manifest file and shrinkwrap files
+    orig_manifest_file = d.expand("${S}/package.json")
+    orig_shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
+    cached_manifest_file = d.expand("${NPM_PACKAGE}/package.json")
+    cached_shrinkwrap_file = d.expand("${NPM_PACKAGE}/npm-shrinkwrap.json")
+
+    with open(orig_manifest_file, "r") as f:
+        orig_manifest = json.load(f)
+
+    cached_manifest = copy.deepcopy(orig_manifest)
+    cached_manifest.pop("dependencies", None)
+    cached_manifest.pop("devDependencies", None)
+
+    with open(orig_shrinkwrap_file, "r") as f:
+        orig_shrinkwrap = json.load(f)
+
+    cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
+    cached_shrinkwrap.pop("dependencies", None)
+
+    # Manage the dependencies
+    progress = OutOfProgressHandler(d, r"^(\d+)/(\d+)$")
+    progress_total = 1 # also count the main package
+    progress_done = 0
+
+    def _count_dependency(name, params, deptree):
+        nonlocal progress_total
+        progress_total += 1
+
+    def _cache_dependency(name, params, deptree):
+        destsubdirs = [os.path.join("node_modules", dep) for dep in deptree]
+        destsuffix = os.path.join(*destsubdirs)
+        with tempfile.TemporaryDirectory() as tmpdir:
+            # Add the dependency to the npm cache
+            destdir = os.path.join(d.getVar("S"), destsuffix)
+            tarball = npm_pack(env, destdir, tmpdir)
+            _npm_cache_add(tarball)
+            # Add its signature to the cached shrinkwrap
+            dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
+            dep["version"] = _npm_version(tarball)
+            dep["integrity"] = _npm_integrity(tarball)
+            if params.get("dev", False):
+                dep["dev"] = True
+            # Display progress
+            nonlocal progress_done
+            progress_done += 1
+            progress.write("%d/%d" % (progress_done, progress_total))
+
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+    foreach_dependencies(orig_shrinkwrap, _count_dependency, dev)
+    foreach_dependencies(orig_shrinkwrap, _cache_dependency, dev)
+
+    # Configure the main package
+    with tempfile.TemporaryDirectory() as tmpdir:
+        tarball = npm_pack(env, d.getVar("S"), tmpdir)
+        npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)
+
+    # Configure the cached manifest file and cached shrinkwrap file
+    def _update_manifest(depkey):
+        for name in orig_manifest.get(depkey, {}):
+            version = cached_shrinkwrap["dependencies"][name]["version"]
+            if depkey not in cached_manifest:
+                cached_manifest[depkey] = {}
+            cached_manifest[depkey][name] = version
 
-npm_do_compile() {
-	# Copy in any additionally fetched modules
-	if [ -d ${WORKDIR}/node_modules ] ; then
-		cp -a ${WORKDIR}/node_modules ${S}/
-	fi
-	# changing the home directory to the working directory, the .npmrc will
-	# be created in this directory
-	export HOME=${WORKDIR}
-	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-		npm config set dev true
-	else
-		npm config set dev false
-	fi
-	npm set cache ${WORKDIR}/npm_cache
-	# clear cache before every build
-	npm cache clear --force
-	# Install pkg into ${S} without going to the registry
-	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --no-registry install
-	else
-		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
-	fi
+    _update_manifest("dependencies")
+
+    if dev:
+        _update_manifest("devDependencies")
+
+    with open(cached_manifest_file, "w") as f:
+        json.dump(cached_manifest, f, indent=2)
+
+    with open(cached_shrinkwrap_file, "w") as f:
+        json.dump(cached_shrinkwrap, f, indent=2)
 }
 
-npm_do_install() {
-	# changing the home directory to the working directory, the .npmrc will
-	# be created in this directory
-	export HOME=${WORKDIR}
-	mkdir -p ${D}${libdir}/node_modules
-	local NPM_PACKFILE=$(npm pack .)
-	npm install --prefix ${D}${prefix} -g --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry ${NPM_PACKFILE}
-	ln -fs node_modules ${D}${libdir}/node
-	find ${D}${NPM_INSTALLDIR} -type f \( -name "*.a" -o -name "*.d" -o -name "*.o" \) -delete
-	if [ -d ${D}${prefix}/etc ] ; then
-		# This will be empty
-		rmdir ${D}${prefix}/etc
-	fi
+python npm_do_compile() {
+    """
+    Step two: install the npm package
+
+    Use the configured main package and the cached dependencies to run the
+    installation process. The installation is done in a directory which is
+    not the destination directory yet.
+
+    A combination of 'npm pack' and 'npm install' is used to ensure that the
+    installed files are actual copies instead of symbolic links (which is the
+    default npm behavior).
+    """
+    import tempfile
+    from bb.fetch2.npm import NpmEnvironment
+
+    bb.utils.remove(d.getVar("NPM_BUILD"), recurse=True)
+
+    env = NpmEnvironment(d, configs=npm_global_configs(d))
+
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+
+    with tempfile.TemporaryDirectory() as tmpdir:
+        configs = []
+
+        if dev:
+            configs.append(("also", "development"))
+        else:
+            configs.append(("only", "production"))
+
+        # Report as many logs as possible for debugging purpose
+        configs.append(("loglevel", "silly"))
+
+        # Configure the installation to be done globally in the build directory
+        configs.append(("global", "true"))
+        configs.append(("prefix", d.getVar("NPM_BUILD")))
+
+        # Add node-gyp configuration
+        configs.append(("arch", d.getVar("NPM_ARCH")))
+        configs.append(("release", "true"))
+
+        # Pack and install the main package
+        tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
+        env.run("npm install '%s'" % tarball, configs=configs)
 }
 
-python populate_packages_prepend () {
-    instdir = d.expand('${D}${NPM_INSTALLDIR}')
-    extrapackages = oe.package.npm_split_package_dirs(instdir)
-    pkgnames = extrapackages.keys()
-    d.prependVar('PACKAGES', '%s ' % ' '.join(pkgnames))
-    for pkgname in pkgnames:
-        pkgrelpath, pdata = extrapackages[pkgname]
-        pkgpath = '${NPM_INSTALLDIR}/' + pkgrelpath
-        # package names can't have underscores but npm packages sometimes use them
-        oe_pkg_name = pkgname.replace('_', '-')
-        expanded_pkgname = d.expand(oe_pkg_name)
-        d.setVar('FILES_%s' % expanded_pkgname, pkgpath)
-        if pdata:
-            version = pdata.get('version', None)
-            if version:
-                d.setVar('PKGV_%s' % expanded_pkgname, version)
-            description = pdata.get('description', None)
-            if description:
-                d.setVar('SUMMARY_%s' % expanded_pkgname, description.replace(u"\u2018", "'").replace(u"\u2019", "'"))
-    d.appendVar('RDEPENDS_%s' % d.getVar('PN'), ' %s' % ' '.join(pkgnames).replace('_', '-'))
+npm_do_install() {
+    # Step three: final install
+    #
+    # The previous installation have to be filtered to remove some extra files.
+
+    rm -rf ${D}
+
+    # Copy the entire lib and bin directories
+    install -d ${D}/${nonarch_libdir}
+    cp --no-preserve=ownership --recursive ${NPM_BUILD}/lib/. ${D}/${nonarch_libdir}
+
+    if [ -d "${NPM_BUILD}/bin" ]
+    then
+        install -d ${D}/${bindir}
+        cp --no-preserve=ownership --recursive ${NPM_BUILD}/bin/. ${D}/${bindir}
+    fi
+
+    # If the package (or its dependencies) uses node-gyp to build native addons,
+    # object files, static libraries or other temporary files can be hidden in
+    # the lib directory. To reduce the package size and to avoid QA issues
+    # (staticdev with static library files) these files must be removed.
+    local GYP_REGEX=".*/build/Release/[^/]*.node"
+
+    # Remove any node-gyp directory in ${D} to remove temporary build files
+    for GYP_D_FILE in $(find ${D} -regex "${GYP_REGEX}")
+    do
+        local GYP_D_DIR=${GYP_D_FILE%/Release/*}
+
+        rm --recursive --force ${GYP_D_DIR}
+    done
+
+    # Copy only the node-gyp release files
+    for GYP_B_FILE in $(find ${NPM_BUILD} -regex "${GYP_REGEX}")
+    do
+        local GYP_D_FILE=${D}/${prefix}/${GYP_B_FILE#${NPM_BUILD}}
+
+        install -d ${GYP_D_FILE%/*}
+        install -m 755 ${GYP_B_FILE} ${GYP_D_FILE}
+    done
+
+    # Remove the shrinkwrap file which does not need to be packed
+    rm -f ${D}/${nonarch_libdir}/node_modules/*/npm-shrinkwrap.json
+    rm -f ${D}/${nonarch_libdir}/node_modules/@*/*/npm-shrinkwrap.json
+
+    # node(1) is using /usr/lib/node as default include directory and npm(1) is
+    # using /usr/lib/node_modules as install directory. Let's make both happy.
+    ln -fs node_modules ${D}/${nonarch_libdir}/node
 }
 
 FILES_${PN} += " \
     ${bindir} \
-    ${libdir}/node \
-    ${NPM_INSTALLDIR} \
+    ${nonarch_libdir} \
 "
 
-EXPORT_FUNCTIONS do_compile do_install
+EXPORT_FUNCTIONS do_configure do_compile do_install
-- 
2.20.1

[PATCH v4 02/13] classes/npm: restrict the build to be offline

[Jean-Marie LEMETAYER]

After the do_fetch task, every other tasks must not access the network.
In order to ensure this point every npm command must use the offline
configuration. In addition setting an invalid proxy is used as a safety.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 35b63122df..7ec58db87b 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -44,6 +44,9 @@ NPM_BUILD = "${WORKDIR}/npm-build"
 def npm_global_configs(d):
     """Get the npm global configuration"""
     configs = []
+    # Ensure no network access is done
+    configs.append(("offline", "true"))
+    configs.append(("proxy", "http://invalid"))
     # Configure the cache directory
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
-- 
2.20.1

[PATCH v4 03/13] classes/npm: use the local node headers

[Jean-Marie LEMETAYER]

When building addons, the node headers are needed to be able to compile
properly. Usually they are downloaded by npm but network access in the
do_compile task are unauthorized. Hopefully the local node headers are
available in the native sysroot so lets use them.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 7ec58db87b..edeef51033 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -230,6 +230,9 @@ python npm_do_compile() {
         # Add node-gyp configuration
         configs.append(("arch", d.getVar("NPM_ARCH")))
         configs.append(("release", "true"))
+        sysroot = d.getVar("RECIPE_SYSROOT_NATIVE")
+        nodedir = os.path.join(sysroot, d.getVar("prefix_native").strip("/"))
+        configs.append(("nodedir", nodedir))
 
         # Pack and install the main package
         tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
-- 
2.20.1

[PATCH v4 04/13] classes/npm: use the native python

[Jean-Marie LEMETAYER]

When building addons, the node-gyp build tool is looking for python. It
is available in the native directory but not directly in the PATH.

This commit configures npm to use the native python executable.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index edeef51033..76321eab06 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -233,6 +233,9 @@ python npm_do_compile() {
         sysroot = d.getVar("RECIPE_SYSROOT_NATIVE")
         nodedir = os.path.join(sysroot, d.getVar("prefix_native").strip("/"))
         configs.append(("nodedir", nodedir))
+        bindir = os.path.join(sysroot, d.getVar("bindir_native").strip("/"))
+        pythondir = os.path.join(bindir, "python-native", "python")
+        configs.append(("python", pythondir))
 
         # Pack and install the main package
         tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
-- 
2.20.1

[PATCH v4 05/13] classes/npm: force to rebuild the prebuild addons

[Jean-Marie LEMETAYER]

This commit forces to rebuild the prebuild addons which are using
node-gyp-build.

  https://www.npmjs.com/package/node-gyp-build

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/classes/npm.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 76321eab06..115c409020 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -213,6 +213,7 @@ python npm_do_compile() {
     dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
 
     with tempfile.TemporaryDirectory() as tmpdir:
+        args = []
         configs = []
 
         if dev:
@@ -237,9 +238,13 @@ python npm_do_compile() {
         pythondir = os.path.join(bindir, "python-native", "python")
         configs.append(("python", pythondir))
 
+        # Add node-pre-gyp configuration
+        args.append(("target_arch", d.getVar("NPM_ARCH")))
+        args.append(("build-from-source", "true"))
+
         # Pack and install the main package
         tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
-        env.run("npm install '%s'" % tarball, configs=configs)
+        env.run("npm install '%s'" % tarball, args=args, configs=configs)
 }
 
 npm_do_install() {
-- 
2.20.1

[PATCH v4 06/13] devtool: npm: rename npm command line options

[Jean-Marie LEMETAYER]

This commit renames the '--fetch-dev' option into '--npm-dev' as it is a
npm only option.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/devtool/standard.py  | 6 +++---
 scripts/lib/recipetool/create.py | 8 +++-----
 2 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 1c0cd8ab51..f33018de3f 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -145,8 +145,8 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' --src-subdir "%s"' % args.src_subdir
     if args.autorev:
         extracmdopts += ' -a'
-    if args.fetch_dev:
-        extracmdopts += ' --fetch-dev'
+    if args.npm_dev:
+        extracmdopts += ' --npm-dev'
     if args.mirrors:
         extracmdopts += ' --mirrors'
     if args.srcrev:
@@ -2197,7 +2197,7 @@ def register_commands(subparsers, context):
     group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
-    parser_add.add_argument('--fetch-dev', help='For npm, also fetch devDependencies', action="store_true")
+    parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     group = parser_add.add_mutually_exclusive_group()
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4c4bbadb4c..7659862f58 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -714,10 +714,8 @@ def create_recipe(args):
         lines_after.append('INSANE_SKIP_${PN} += "already-stripped"')
         lines_after.append('')
 
-    if args.fetch_dev:
-        extravalues['fetchdev'] = True
-    else:
-        extravalues['fetchdev'] = None
+    if args.npm_dev:
+        extravalues['NPM_INSTALL_DEV'] = 1
 
     # Find all plugins that want to register handlers
     logger.debug('Loading recipe handlers')
@@ -1313,7 +1311,7 @@ def register_commands(subparsers):
     group.add_argument('-S', '--srcrev', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
     parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
-    parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies')
+    parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
     parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
     parser_create.set_defaults(func=create_recipe)
-- 
2.20.1

[PATCH v4 07/13] recipetool/create_npm: refactor the npm recipe creation handler

[Jean-Marie LEMETAYER]

This commit refactors the npm recipe creation handler to use the new npm
behavior. The process is kept as simple as possible and only generates
the shrinkwrap file.

To avoid naming issues the recipe name is now extracted from the npm
package name and not directly mapped.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create_npm.py | 468 ++++++++++-----------------
 1 file changed, 178 insertions(+), 290 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 39429ebad3..5eba7dcd7b 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -1,321 +1,209 @@
-# Recipe creation tool - node.js NPM module support plugin
-#
 # Copyright (C) 2016 Intel Corporation
+# Copyright (C) 2020 Savoir-Faire Linux
 #
 # SPDX-License-Identifier: GPL-2.0-only
 #
+"""Recipe creation tool - npm module support plugin"""
 
+import json
 import os
+import re
 import sys
-import logging
-import subprocess
 import tempfile
-import shutil
-import json
-from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars
+import bb
+from bb.fetch2.npm import NpmEnvironment
+from recipetool.create import RecipeHandler
 
-logger = logging.getLogger('recipetool')
+TINFOIL = None
 
+def tinfoil_init(instance):
+    """Initialize tinfoil"""
+    global TINFOIL
+    TINFOIL = instance
 
-tinfoil = None
+class NpmRecipeHandler(RecipeHandler):
+    """Class to handle the npm recipe creation"""
+
+    @staticmethod
+    def _npm_name(name):
+        """Generate a Yocto friendly npm name"""
+        name = re.sub("/", "-", name)
+        name = name.lower()
+        name = re.sub(r"[^\-a-z0-9]", "", name)
+        name = name.strip("-")
+        return name
+
+    @staticmethod
+    def _get_registry(lines):
+        """Get the registry value from the 'npm://registry' url"""
+        registry = None
+
+        def _handle_registry(varname, origvalue, op, newlines):
+            nonlocal registry
+            if origvalue.startswith("npm://"):
+                registry = re.sub(r"^npm://", "http://", origvalue.split(";")[0])
+            return origvalue, None, 0, True
 
-def tinfoil_init(instance):
-    global tinfoil
-    tinfoil = instance
+        bb.utils.edit_metadata(lines, ["SRC_URI"], _handle_registry)
 
+        return registry
 
-class NpmRecipeHandler(RecipeHandler):
-    lockdownpath = None
+    @staticmethod
+    def _ensure_npm():
+        """Check if the 'npm' command is available in the recipes"""
+        if not TINFOIL.recipes_parsed:
+            TINFOIL.parse_recipes()
 
-    def _ensure_npm(self, fixed_setup=False):
-        if not tinfoil.recipes_parsed:
-            tinfoil.parse_recipes()
         try:
-            rd = tinfoil.parse_recipe('nodejs-native')
+            d = TINFOIL.parse_recipe("nodejs-native")
         except bb.providers.NoProvider:
-            if fixed_setup:
-                msg = 'nodejs-native is required for npm but is not available within this SDK'
-            else:
-                msg = 'nodejs-native is required for npm but is not available - you will likely need to add a layer that provides nodejs'
-            logger.error(msg)
-            return None
-        bindir = rd.getVar('STAGING_BINDIR_NATIVE')
-        npmpath = os.path.join(bindir, 'npm')
+            bb.error("Nothing provides 'nodejs-native' which is required for the build")
+            bb.note("You will likely need to add a layer that provides nodejs")
+            sys.exit(14)
+
+        bindir = d.getVar("STAGING_BINDIR_NATIVE")
+        npmpath = os.path.join(bindir, "npm")
+
         if not os.path.exists(npmpath):
-            tinfoil.build_targets('nodejs-native', 'addto_recipe_sysroot')
+            TINFOIL.build_targets("nodejs-native", "addto_recipe_sysroot")
+
             if not os.path.exists(npmpath):
-                logger.error('npm required to process specified source, but nodejs-native did not seem to populate it')
-                return None
+                bb.error("Failed to add 'npm' to sysroot")
+                sys.exit(14)
+
         return bindir
 
-    def _handle_license(self, data):
-        '''
-        Handle the license value from an npm package.json file
-        '''
-        license = None
-        if 'license' in data:
-            license = data['license']
-            if isinstance(license, dict):
-                license = license.get('type', None)
-            if license:
-                if 'OR' in license:
-                    license = license.replace('OR', '|')
-                    license = license.replace('AND', '&')
-                    license = license.replace(' ', '_')
-                    if not license[0] == '(':
-                        license = '(' + license + ')'
-                else:
-                    license = license.replace('AND', '&')
-                    if license[0] == '(':
-                        license = license[1:]
-                    if license[-1] == ')':
-                        license = license[:-1]
-                license = license.replace('MIT/X11', 'MIT')
-                license = license.replace('Public Domain', 'PD')
-                license = license.replace('SEE LICENSE IN EULA',
-                                          'SEE-LICENSE-IN-EULA')
-        return license
-
-    def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before, d):
-        try:
-            runenv = dict(os.environ, PATH=d.getVar('PATH'))
-            bb.process.run('npm shrinkwrap', cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        except bb.process.ExecutionError as e:
-            logger.warning('npm shrinkwrap failed:\n%s' % e.stdout)
-            return
-
-        tmpfile = os.path.join(localfilesdir, 'npm-shrinkwrap.json')
-        shutil.move(os.path.join(srctree, 'npm-shrinkwrap.json'), tmpfile)
-        extravalues.setdefault('extrafiles', {})
-        extravalues['extrafiles']['npm-shrinkwrap.json'] = tmpfile
-        lines_before.append('NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"')
-
-    def _lockdown(self, srctree, localfilesdir, extravalues, lines_before, d):
-        runenv = dict(os.environ, PATH=d.getVar('PATH'))
-        if not NpmRecipeHandler.lockdownpath:
-            NpmRecipeHandler.lockdownpath = tempfile.mkdtemp('recipetool-npm-lockdown')
-            bb.process.run('npm install lockdown --prefix %s' % NpmRecipeHandler.lockdownpath,
-                           cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        relockbin = os.path.join(NpmRecipeHandler.lockdownpath, 'node_modules', 'lockdown', 'relock.js')
-        if not os.path.exists(relockbin):
-            logger.warning('Could not find relock.js within lockdown directory; skipping lockdown')
-            return
-        try:
-            bb.process.run('node %s' % relockbin, cwd=srctree, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        except bb.process.ExecutionError as e:
-            logger.warning('lockdown-relock failed:\n%s' % e.stdout)
-            return
-
-        tmpfile = os.path.join(localfilesdir, 'lockdown.json')
-        shutil.move(os.path.join(srctree, 'lockdown.json'), tmpfile)
-        extravalues.setdefault('extrafiles', {})
-        extravalues['extrafiles']['lockdown.json'] = tmpfile
-        lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
-
-    def _handle_dependencies(self, d, deps, optdeps, devdeps, lines_before, srctree):
-        import scriptutils
-        # If this isn't a single module we need to get the dependencies
-        # and add them to SRC_URI
-        def varfunc(varname, origvalue, op, newlines):
-            if varname == 'SRC_URI':
-                if not origvalue.startswith('npm://'):
-                    src_uri = origvalue.split()
-                    deplist = {}
-                    for dep, depver in optdeps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        if self.check_npm_optional_dependency(depdata):
-                            deplist[dep] = depdata
-                    for dep, depver in devdeps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        if self.check_npm_optional_dependency(depdata):
-                            deplist[dep] = depdata
-                    for dep, depver in deps.items():
-                        depdata = self.get_npm_data(dep, depver, d)
-                        deplist[dep] = depdata
-
-                    extra_urls = []
-                    for dep, depdata in deplist.items():
-                        version = depdata.get('version', None)
-                        if version:
-                            url = 'npm://registry.npmjs.org;name=%s;version=%s;subdir=node_modules/%s' % (dep, version, dep)
-                            extra_urls.append(url)
-                    if extra_urls:
-                        scriptutils.fetch_url(tinfoil, ' '.join(extra_urls), None, srctree, logger)
-                        src_uri.extend(extra_urls)
-                        return src_uri, None, -1, True
-            return origvalue, None, 0, True
-        updated, newlines = bb.utils.edit_metadata(lines_before, ['SRC_URI'], varfunc)
-        if updated:
-            del lines_before[:]
-            for line in newlines:
-                # Hack to avoid newlines that edit_metadata inserts
-                if line.endswith('\n'):
-                    line = line[:-1]
-                lines_before.append(line)
-        return updated
+    @staticmethod
+    def _npm_global_configs(dev):
+        """Get the npm global configuration"""
+        configs = []
+
+        if dev:
+            configs.append(("also", "development"))
+        else:
+            configs.append(("only", "production"))
+
+        configs.append(("save", "false"))
+        configs.append(("package-lock", "false"))
+        configs.append(("shrinkwrap", "false"))
+        return configs
+
+    def _run_npm_install(self, d, srctree, registry, dev):
+        """Run the 'npm install' command without building the addons"""
+        configs = self._npm_global_configs(dev)
+        configs.append(("ignore-scripts", "true"))
+
+        if registry:
+            configs.append(("registry", registry))
+
+        bb.utils.remove(os.path.join(srctree, "node_modules"), recurse=True)
+
+        env = NpmEnvironment(d, configs=configs)
+        env.run("npm install", workdir=srctree)
+
+    def _generate_shrinkwrap(self, d, srctree, dev):
+        """Check and generate the 'npm-shrinkwrap.json' file if needed"""
+        configs = self._npm_global_configs(dev)
+
+        env = NpmEnvironment(d, configs=configs)
+        env.run("npm shrinkwrap", workdir=srctree)
+
+        return os.path.join(srctree, "npm-shrinkwrap.json")
 
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
-        import bb.utils
-        import oe.package
-        from collections import OrderedDict
+        """Handle the npm recipe creation"""
 
-        if 'buildsystem' in handled:
+        if "buildsystem" in handled:
             return False
 
-        def read_package_json(fn):
-            with open(fn, 'r', errors='surrogateescape') as f:
-                return json.loads(f.read())
+        files = RecipeHandler.checkfiles(srctree, ["package.json"])
 
-        files = RecipeHandler.checkfiles(srctree, ['package.json'])
-        if files:
-            d = bb.data.createCopy(tinfoil.config_data)
-            npm_bindir = self._ensure_npm()
-            if not npm_bindir:
-                sys.exit(14)
-            d.prependVar('PATH', '%s:' % npm_bindir)
-
-            data = read_package_json(files[0])
-            if 'name' in data and 'version' in data:
-                extravalues['PN'] = data['name']
-                extravalues['PV'] = data['version']
-                classes.append('npm')
-                handled.append('buildsystem')
-                if 'description' in data:
-                    extravalues['SUMMARY'] = data['description']
-                if 'homepage' in data:
-                    extravalues['HOMEPAGE'] = data['homepage']
-
-                fetchdev = extravalues['fetchdev'] or None
-                deps, optdeps, devdeps = self.get_npm_package_dependencies(data, fetchdev)
-                self._handle_dependencies(d, deps, optdeps, devdeps, lines_before, srctree)
-
-                # Shrinkwrap
-                localfilesdir = tempfile.mkdtemp(prefix='recipetool-npm')
-                self._shrinkwrap(srctree, localfilesdir, extravalues, lines_before, d)
-
-                # Lockdown
-                self._lockdown(srctree, localfilesdir, extravalues, lines_before, d)
-
-                # Split each npm module out to is own package
-                npmpackages = oe.package.npm_split_package_dirs(srctree)
-                licvalues = None
-                for item in handled:
-                    if isinstance(item, tuple):
-                        if item[0] == 'license':
-                            licvalues = item[1]
-                            break
-                if not licvalues:
-                    licvalues = handle_license_vars(srctree, lines_before, handled, extravalues, d)
-                if licvalues:
-                    # Augment the license list with information we have in the packages
-                    licenses = {}
-                    license = self._handle_license(data)
-                    if license:
-                        licenses['${PN}'] = license
-                    for pkgname, pkgitem in npmpackages.items():
-                        _, pdata = pkgitem
-                        license = self._handle_license(pdata)
-                        if license:
-                            licenses[pkgname] = license
-                    # Now write out the package-specific license values
-                    # We need to strip out the json data dicts for this since split_pkg_licenses
-                    # isn't expecting it
-                    packages = OrderedDict((x,y[0]) for x,y in npmpackages.items())
-                    packages['${PN}'] = ''
-                    pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
-                    all_licenses = list(set([item.replace('_', ' ') for pkglicense in pkglicenses.values() for item in pkglicense]))
-                    if '&' in all_licenses:
-                        all_licenses.remove('&')
-                    extravalues['LICENSE'] = ' & '.join(all_licenses)
-
-                # Need to move S setting after inherit npm
-                for i, line in enumerate(lines_before):
-                    if line.startswith('S ='):
-                        lines_before.pop(i)
-                        lines_after.insert(0, '# Must be set after inherit npm since that itself sets S')
-                        lines_after.insert(1, line)
-                        break
-
-                return True
-
-        return False
-
-    # FIXME this is duplicated from lib/bb/fetch2/npm.py
-    def _parse_view(self, output):
-        '''
-        Parse the output of npm view --json; the last JSON result
-        is assumed to be the one that we're interested in.
-        '''
-        pdata = None
-        outdeps = {}
-        datalines = []
-        bracelevel = 0
-        for line in output.splitlines():
-            if bracelevel:
-                datalines.append(line)
-            elif '{' in line:
-                datalines = []
-                datalines.append(line)
-            bracelevel = bracelevel + line.count('{') - line.count('}')
-        if datalines:
-            pdata = json.loads('\n'.join(datalines))
-        return pdata
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def get_npm_data(self, pkg, version, d):
-        import bb.fetch2
-        pkgfullname = pkg
-        if version != '*' and not '/' in version:
-            pkgfullname += "@'%s'" % version
-        logger.debug(2, "Calling getdeps on %s" % pkg)
-        runenv = dict(os.environ, PATH=d.getVar('PATH'))
-        fetchcmd = "npm view %s --json" % pkgfullname
-        output, _ = bb.process.run(fetchcmd, stderr=subprocess.STDOUT, env=runenv, shell=True)
-        data = self._parse_view(output)
-        return data
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def get_npm_package_dependencies(self, pdata, fetchdev):
-        dependencies = pdata.get('dependencies', {})
-        optionalDependencies = pdata.get('optionalDependencies', {})
-        dependencies.update(optionalDependencies)
-        if fetchdev:
-            devDependencies = pdata.get('devDependencies', {})
-            dependencies.update(devDependencies)
-        else:
-            devDependencies = {}
-        depsfound = {}
-        optdepsfound = {}
-        devdepsfound = {}
-        for dep in dependencies:
-            if dep in optionalDependencies:
-                optdepsfound[dep] = dependencies[dep]
-            elif dep in devDependencies:
-                devdepsfound[dep] = dependencies[dep]
-            else:
-                depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound, devdepsfound
-
-    # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
-    # (split out from _getdependencies())
-    def check_npm_optional_dependency(self, pdata):
-        pkg_os = pdata.get('os', None)
-        if pkg_os:
-            if not isinstance(pkg_os, list):
-                pkg_os = [pkg_os]
-            blacklist = False
-            for item in pkg_os:
-                if item.startswith('!'):
-                    blacklist = True
-                    break
-            if (not blacklist and 'linux' not in pkg_os) or '!linux' in pkg_os:
-                pkg = pdata.get('name', 'Unnamed package')
-                logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg)
-                return False
-        return True
+        if not files:
+            return False
+
+        with open(files[0], "r") as f:
+            data = json.load(f)
 
+        if "name" not in data or "version" not in data:
+            return False
+
+        extravalues["PN"] = self._npm_name(data["name"])
+        extravalues["PV"] = data["version"]
+
+        if "description" in data:
+            extravalues["SUMMARY"] = data["description"]
+
+        if "homepage" in data:
+            extravalues["HOMEPAGE"] = data["homepage"]
+
+        dev = bb.utils.to_boolean(str(extravalues.get("NPM_INSTALL_DEV", "0")), False)
+        registry = self._get_registry(lines_before)
+
+        bb.note("Checking if npm is available ...")
+        # The native npm is used here (and not the host one) to ensure that the
+        # npm version is high enough to ensure an efficient dependency tree
+        # resolution and avoid issue with the shrinkwrap file format.
+        # Moreover the native npm is mandatory for the build.
+        bindir = self._ensure_npm()
+
+        d = bb.data.createCopy(TINFOIL.config_data)
+        d.prependVar("PATH", bindir + ":")
+        d.setVar("S", srctree)
+
+        bb.note("Generating shrinkwrap file ...")
+        # To generate the shrinkwrap file the dependencies have to be installed
+        # first. During the generation process some files may be updated /
+        # deleted. By default devtool tracks the diffs in the srctree and raises
+        # errors when finishing the recipe if some diffs are found.
+        git_exclude_file = os.path.join(srctree, ".git", "info", "exclude")
+        if os.path.exists(git_exclude_file):
+            with open(git_exclude_file, "r+") as f:
+                lines = f.readlines()
+                for line in ["/node_modules/", "/npm-shrinkwrap.json"]:
+                    if line not in lines:
+                        f.write(line + "\n")
+
+        lock_file = os.path.join(srctree, "package-lock.json")
+        lock_copy = lock_file + ".copy"
+        if os.path.exists(lock_file):
+            bb.utils.copyfile(lock_file, lock_copy)
+
+        self._run_npm_install(d, srctree, registry, dev)
+        shrinkwrap_file = self._generate_shrinkwrap(d, srctree, dev)
+
+        if os.path.exists(lock_copy):
+            bb.utils.movefile(lock_copy, lock_file)
+
+        # Add the shrinkwrap file as 'extrafiles'
+        shrinkwrap_copy = shrinkwrap_file + ".copy"
+        bb.utils.copyfile(shrinkwrap_file, shrinkwrap_copy)
+        extravalues.setdefault("extrafiles", {})
+        extravalues["extrafiles"]["npm-shrinkwrap.json"] = shrinkwrap_copy
+
+        url_local = "npmsw://%s" % shrinkwrap_file
+        url_recipe= "npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json"
+
+        if dev:
+            url_local += ";dev=1"
+            url_recipe += ";dev=1"
+
+        # Add the npmsw url in the SRC_URI of the generated recipe
+        def _handle_srcuri(varname, origvalue, op, newlines):
+            """Update the version value and add the 'npmsw://' url"""
+            value = origvalue.replace("version=" + data["version"], "version=${PV}")
+            value = origvalue.replace("version=latest", "version=${PV}")
+            values = [line.strip() for line in value.strip('\n').splitlines()]
+            values.append(url_recipe)
+            return values, None, 4, False
+
+        (_, newlines) = bb.utils.edit_metadata(lines_before, ["SRC_URI"], _handle_srcuri)
+        lines_before[:] = [line.rstrip('\n') for line in newlines]
+
+        classes.append("npm")
+        handled.append("buildsystem")
+
+        return True
 
 def register_recipe_handlers(handlers):
+    """Register the npm handler"""
     handlers.append((NpmRecipeHandler(), 60))
-- 
2.20.1

[PATCH v4 08/13] recipetool/create_npm: handle the licenses of the dependencies

[Jean-Marie LEMETAYER]

As usual the 'LICENSE' and the 'LIC_FILES_CHKSUM' values reflects all
the license files discovered in the source tree (including the
dependencies).

For npm recipes the 'LIC_FILES_CHKSUM' value contains also the status of
the 'package.json' file of every packages as it contains license
informations.

Finally each package has a separate 'LICENSE_${PN}-package-name' value
which describes its license.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create_npm.py | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 5eba7dcd7b..38852bbe9a 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -12,7 +12,10 @@ import sys
 import tempfile
 import bb
 from bb.fetch2.npm import NpmEnvironment
+from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
+from recipetool.create import guess_license
+from recipetool.create import split_pkg_licenses
 
 TINFOIL = None
 
@@ -110,6 +113,36 @@ class NpmRecipeHandler(RecipeHandler):
 
         return os.path.join(srctree, "npm-shrinkwrap.json")
 
+    def _handle_licenses(self, srctree, shrinkwrap_file, dev):
+        """Return the extra license files and the list of packages"""
+        licfiles = []
+        packages = {}
+
+        def _licfiles_append(licfile):
+            """Append 'licfile' to the license files list"""
+            licfilepath = os.path.join(srctree, licfile)
+            licmd5 = bb.utils.md5_file(licfilepath)
+            licfiles.append("file://%s;md5=%s" % (licfile, licmd5))
+
+        # Handle the parent package
+        _licfiles_append("package.json")
+        packages["${PN}"] = ""
+
+        # Handle the dependencies
+        def _handle_dependency(name, params, deptree):
+            suffix = "-".join([self._npm_name(dep) for dep in deptree])
+            destdirs = [os.path.join("node_modules", dep) for dep in deptree]
+            destdir = os.path.join(*destdirs)
+            _licfiles_append(os.path.join(destdir, "package.json"))
+            packages["${PN}-" + suffix] = destdir
+
+        with open(shrinkwrap_file, "r") as f:
+            shrinkwrap = json.load(f)
+
+        foreach_dependencies(shrinkwrap, _handle_dependency, dev)
+
+        return licfiles, packages
+
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
         """Handle the npm recipe creation"""
 
@@ -199,6 +232,19 @@ class NpmRecipeHandler(RecipeHandler):
         (_, newlines) = bb.utils.edit_metadata(lines_before, ["SRC_URI"], _handle_srcuri)
         lines_before[:] = [line.rstrip('\n') for line in newlines]
 
+        # In order to generate correct licence checksums in the recipe the
+        # dependencies have to be fetched again using the npmsw url
+        bb.note("Fetching npm dependencies ...")
+        bb.utils.remove(os.path.join(srctree, "node_modules"), recurse=True)
+        fetcher = bb.fetch2.Fetch([url_local], d)
+        fetcher.download()
+        fetcher.unpack(srctree)
+
+        bb.note("Handling licences ...")
+        (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, dev)
+        extravalues["LIC_FILES_CHKSUM"] = licfiles
+        split_pkg_licenses(guess_license(srctree, d), packages, lines_after, [])
+
         classes.append("npm")
         handled.append("buildsystem")
 
-- 
2.20.1

[PATCH v4 09/13] lib/oe/package: remove unneeded npm_split_package_dirs function

[Jean-Marie LEMETAYER]

The npm_split_package_dirs function was used by the recipetool when
creating npm recipes. This is not the case anymore.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/lib/oe/package.py | 33 ---------------------------------
 1 file changed, 33 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index b8585d4253..dd700cbb0c 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -283,36 +283,3 @@ def read_shlib_providers(d):
                         shlib_provider[s[0]] = {}
                     shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
     return shlib_provider
-
-
-def npm_split_package_dirs(pkgdir):
-    """
-    Work out the packages fetched and unpacked by BitBake's npm fetcher
-    Returns a dict of packagename -> (relpath, package.json) ordered
-    such that it is suitable for use in PACKAGES and FILES
-    """
-    from collections import OrderedDict
-    import json
-    packages = {}
-    for root, dirs, files in os.walk(pkgdir):
-        if os.path.basename(root) == 'node_modules':
-            for dn in dirs:
-                relpth = os.path.relpath(os.path.join(root, dn), pkgdir)
-                pkgitems = ['${PN}']
-                for pathitem in relpth.split('/'):
-                    if pathitem == 'node_modules':
-                        continue
-                    pkgitems.append(pathitem)
-                pkgname = '-'.join(pkgitems).replace('_', '-')
-                pkgname = pkgname.replace('@', '')
-                pkgfile = os.path.join(root, dn, 'package.json')
-                data = None
-                if os.path.exists(pkgfile):
-                    with open(pkgfile, 'r') as f:
-                        data = json.loads(f.read())
-                    packages[pkgname] = (relpth, data)
-    # We want the main package for a module sorted *after* its subpackages
-    # (so that it doesn't otherwise steal the files for the subpackage), so
-    # this is a cheap way to do that whilst still having an otherwise
-    # alphabetical sort
-    return OrderedDict((key, packages[key]) for key in sorted(packages, key=lambda pkg: pkg + '~'))
-- 
2.20.1

[PATCH v4 10/13] devtool/standard: npm: update the append file

[Jean-Marie LEMETAYER]

When creating a recipe using devtool, a workspace is created to store
the new recipe, the recipe source and some append files. These append
files are used by devtool to build the recipe using externalsrc (to use
the source which are in the workspace). They can also have some
additional actions according to the class of the recipe.

This commit updates the append file for the npm recipes. The
devtool / externalsrc files are removed in the npm package directory
instead of the install directory.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/devtool/standard.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f33018de3f..52efeae938 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -260,14 +260,10 @@ def add(args, config, basepath, workspace):
                 f.write('}\n')
 
             if bb.data.inherits_class('npm', rd):
-                f.write('do_install_append() {\n')
-                f.write('    # Remove files added to source dir by devtool/externalsrc\n')
-                f.write('    rm -f ${NPM_INSTALLDIR}/singletask.lock\n')
-                f.write('    rm -rf ${NPM_INSTALLDIR}/.git\n')
-                f.write('    rm -rf ${NPM_INSTALLDIR}/oe-local-files\n')
-                f.write('    for symlink in ${EXTERNALSRC_SYMLINKS} ; do\n')
-                f.write('        rm -f ${NPM_INSTALLDIR}/${symlink%%:*}\n')
-                f.write('    done\n')
+                f.write('python do_configure_append() {\n')
+                f.write('    pkgdir = d.getVar("NPM_PACKAGE")\n')
+                f.write('    lockfile = os.path.join(pkgdir, "singletask.lock")\n')
+                f.write('    bb.utils.remove(lockfile)\n')
                 f.write('}\n')
 
         # Check if the new layer provides recipes whose priorities have been
-- 
2.20.1

[PATCH v4 11/13] recipetool/create: npm: remove the 'noverify' url parameter

[Jean-Marie LEMETAYER]

This commit removes the 'noverify' parameter which was added to the url
to fix warnings with the shrinkwrap / lockdown file generation. This is
not needed anymore with the new npm fetcher.

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 scripts/lib/recipetool/create.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7659862f58..6cbf4de674 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -477,8 +477,6 @@ def create_recipe(args):
             storeTagName = params['tag']
             params['nobranch'] = '1'
             del params['tag']
-        if scheme == 'npm':
-            params['noverify'] = '1'
         fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
-- 
2.20.1

[PATCH v4 12/13] oeqa/selftest/recipetool: add npm recipe creation test

[Jean-Marie LEMETAYER]

This commit adds a recipetool creation test for npm recipe:

 - recipetool.RecipetoolTests.test_recipetool_create_npm

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 25 ++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index c1562c63b2..6bfe8f177f 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -421,6 +421,31 @@ class RecipetoolTests(RecipetoolBase):
         inherits = ['cmake']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+    def test_recipetool_create_npm(self):
+        temprecipe = os.path.join(self.tempdir, 'recipe')
+        os.makedirs(temprecipe)
+        recipefile = os.path.join(temprecipe, 'savoirfairelinux-node-server-example_1.0.0.bb')
+        shrinkwrap = os.path.join(temprecipe, 'savoirfairelinux-node-server-example', 'npm-shrinkwrap.json')
+        srcuri = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+        result = runCmd('recipetool create -o %s \'%s\'' % (temprecipe, srcuri))
+        self.assertTrue(os.path.isfile(recipefile))
+        self.assertTrue(os.path.isfile(shrinkwrap))
+        checkvars = {}
+        checkvars['SUMMARY'] = 'Node Server Example'
+        checkvars['HOMEPAGE'] = 'https://github.com/savoirfairelinux/node-server-example#readme'
+        checkvars['LICENSE'] = set(['MIT', 'ISC', 'Unknown'])
+        urls = []
+        urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
+        urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
+        checkvars['SRC_URI'] = set(urls)
+        checkvars['S'] = '${WORKDIR}/npm'
+        checkvars['LICENSE_${PN}'] = 'MIT'
+        checkvars['LICENSE_${PN}-base64'] = 'Unknown'
+        checkvars['LICENSE_${PN}-accepts'] = 'MIT'
+        checkvars['LICENSE_${PN}-inherits'] = 'ISC'
+        inherits = ['npm']
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
     def test_recipetool_create_github(self):
         # Basic test to see if github URL mangling works
         temprecipe = os.path.join(self.tempdir, 'recipe')
-- 
2.20.1

[PATCH v4 13/13] oeqa/selftest/devtool: add npm recipe build test

[Jean-Marie LEMETAYER]

This commit adds a devtool build test for npm recipe:

 - devtool.DevtoolAddTests.test_devtool_add_npm

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index 57e6662e4a..b6b14914bd 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -511,6 +511,26 @@ class DevtoolAddTests(DevtoolBase):
         checkvars['SRC_URI'] = url.replace(testver, '${PV}')
         self._test_recipe_contents(recipefile, checkvars, [])
 
+    def test_devtool_add_npm(self):
+        pn = 'savoirfairelinux-node-server-example'
+        pv = '1.0.0'
+        url = 'npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=' + pv
+        # Test devtool add
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool add \'%s\'' % url)
+        self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created')
+        self.assertExists(os.path.join(self.workspacedir, 'recipes', pn, '%s_%s.bb' % (pn, pv)), 'Recipe not created')
+        self.assertExists(os.path.join(self.workspacedir, 'recipes', pn, pn, 'npm-shrinkwrap.json'), 'Shrinkwrap not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(pn, result.output)
+        # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
+        bitbake('%s -c cleansstate' % pn)
+        # Test devtool build
+        result = runCmd('devtool build %s' % pn)
+
 class DevtoolModifyTests(DevtoolBase):
 
     def test_devtool_modify(self):
-- 
2.20.1