All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/6] npm.bbclass: work with nodejs 16
@ 2022-05-19 10:05 Enrico Scholz
  2022-05-19 10:05 ` [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf' Enrico Scholz
                   ` (7 more replies)
  0 siblings, 8 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

nodejs 16 changed internal caching significantly which breaks the
existing npm.bblcass.

Simulate parts of the npm registry and cache data in the way as
expected.

This patchset requires an additional one in the oe-meta layer.

Enrico Scholz (6):
  npm: replace 'npm pack' call by 'tar czf'
  npm: return content of 'package.json' in 'npm_pack'
  npm: take 'version' directly from 'package.json'
  npm: disable 'audit' + 'fund'
  lib:npm_registry: initial checkin
  npm: use npm_registry to cache package

 meta/classes/npm.bbclass    |  65 +++++++++-----
 meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
 2 files changed, 213 insertions(+), 21 deletions(-)
 create mode 100644 meta/lib/oe/npm_registry.py

-- 
2.36.1



^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf'
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 10:25   ` [OE-core] " Zoltan Boszormenyi
  2022-05-19 10:05 ` [PATCH 2/6] npm: return content of 'package.json' in 'npm_pack' Enrico Scholz
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

'npm pack' is a maintainer tool which tries to execute 'prepare'
and similar scripts.  This fails usually in OE because it requires
completely installed 'node_modules'.

Earlier nodejs versions supported an undocumented 'ignore-scripts'
option.  This has been removed in nodejs 16.

We could patch 'package.json' and remove the unwanted scripts.  But
this might complicate local workflows (applying patches) and installed
packages will contain the modified 'package.json'.

Instead of, package it manually by 'tar czf'.  As a sideeffect,
'do_configure' is running much faster now.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index ba50fcac20..91f8f36b0d 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -57,13 +57,36 @@ def npm_global_configs(d):
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
 
+## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for
+## 'ignore-scripts' which prevents this behavior has been removed
+## from nodejs 16.  Use simple 'tar' instead of.
 def npm_pack(env, srcdir, workdir):
-    """Run 'npm pack' on a specified directory"""
-    import shlex
-    cmd = "npm pack %s" % shlex.quote(srcdir)
-    args = [("ignore-scripts", "true")]
-    tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
-    return os.path.join(workdir, tarball)
+    """Emulate 'npm pack' on a specified directory"""
+    import subprocess
+    import os
+    import json
+
+    src = os.path.join(srcdir, 'package.json')
+    with open(src) as f:
+        j = json.load(f)
+
+    # base does not really matter and is for documentation purposes
+    # only.  But the 'version' part must exist because other parts of
+    # the bbclass rely on it.
+    base = j['name'].split('/')[-1]
+    tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version']));
+
+    # TODO: real 'npm pack' does not include directories while 'tar'
+    # does.  But this does not seem to matter...
+    subprocess.run(['tar', 'czf', tarball,
+                    '--exclude', './node-modules',
+                    '--exclude-vcs',
+                    '--transform', 's,^\./,package/,',
+                    '--mtime', '1985-10-26T08:15:00.000Z',
+                    '.'],
+                   check = True, cwd = srcdir)
+
+    return tarball
 
 python npm_do_configure() {
     """
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 2/6] npm: return content of 'package.json' in 'npm_pack'
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
  2022-05-19 10:05 ` [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf' Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 10:05 ` [PATCH 3/6] npm: take 'version' directly from 'package.json' Enrico Scholz
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

We have to read 'package.json' to calculate the name of the tarball.
This content is interesting for later patches.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 91f8f36b0d..014f793450 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -86,7 +86,7 @@ def npm_pack(env, srcdir, workdir):
                     '.'],
                    check = True, cwd = srcdir)
 
-    return tarball
+    return (tarball, j)
 
 python npm_do_configure() {
     """
@@ -186,7 +186,7 @@ python npm_do_configure() {
         with tempfile.TemporaryDirectory() as tmpdir:
             # Add the dependency to the npm cache
             destdir = os.path.join(d.getVar("S"), destsuffix)
-            tarball = npm_pack(env, destdir, tmpdir)
+            (tarball, pkg) = npm_pack(env, destdir, tmpdir)
             _npm_cache_add(tarball)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
@@ -207,7 +207,7 @@ python npm_do_configure() {
 
     # Configure the main package
     with tempfile.TemporaryDirectory() as tmpdir:
-        tarball = npm_pack(env, d.getVar("S"), tmpdir)
+        (tarball, _) = npm_pack(env, d.getVar("S"), tmpdir)
         npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)
 
     # Configure the cached manifest file and cached shrinkwrap file
@@ -280,7 +280,7 @@ python npm_do_compile() {
         args.append(("build-from-source", "true"))
 
         # Pack and install the main package
-        tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
+        (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
         cmd = "npm install %s %s" % (shlex.quote(tarball), d.getVar("EXTRA_OENPM"))
         env.run(cmd, args=args)
 }
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 3/6] npm: take 'version' directly from 'package.json'
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
  2022-05-19 10:05 ` [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf' Enrico Scholz
  2022-05-19 10:05 ` [PATCH 2/6] npm: return content of 'package.json' in 'npm_pack' Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 10:05 ` [PATCH 4/6] npm: disable 'audit' + 'fund' Enrico Scholz
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

We know the content of 'package.json' from earlier patches; there is
no need to parse the tarball name to extract the version.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 014f793450..11c80a738e 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -125,11 +125,6 @@ python npm_do_configure() {
         sha512 = bb.utils.sha512_file(tarball)
         return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
 
-    def _npm_version(tarball):
-        """Return the version of a specified tarball"""
-        regex = r"-(\d+\.\d+\.\d+(-.*)?(\+.*)?)\.tgz"
-        return re.search(regex, tarball).group(1)
-
     def _npmsw_dependency_dict(orig, deptree):
         """
         Return the sub dictionary in the 'orig' dictionary corresponding to the
@@ -190,7 +185,7 @@ python npm_do_configure() {
             _npm_cache_add(tarball)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
-            dep["version"] = _npm_version(tarball)
+            dep["version"] = pkg['version']
             dep["integrity"] = _npm_integrity(tarball)
             if params.get("dev", False):
                 dep["dev"] = True
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 4/6] npm: disable 'audit' + 'fund'
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
                   ` (2 preceding siblings ...)
  2022-05-19 10:05 ` [PATCH 3/6] npm: take 'version' directly from 'package.json' Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 13:56   ` [OE-core] " Luca Ceresoli
  2022-05-19 14:06   ` [PATCH 4/6, v2] " Enrico Scholz
  2022-05-19 10:05 ` [PATCH 5/6] lib:npm_registry: initial checkin Enrico Scholz
                   ` (3 subsequent siblings)
  7 siblings, 2 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

'audit' can cause extra network traffic; 'fund' is not needed.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 11c80a738e..1e41072116 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -53,6 +53,8 @@ def npm_global_configs(d):
     # Ensure no network access is done
     configs.append(("offline", "true"))
     configs.append(("proxy", "http://invalid"))
+    configs.append(("funds", False))
+    configs.append(("audit", False))
     # Configure the cache directory
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 5/6] lib:npm_registry: initial checkin
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
                   ` (3 preceding siblings ...)
  2022-05-19 10:05 ` [PATCH 4/6] npm: disable 'audit' + 'fund' Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 10:05 ` [PATCH 6/6] npm: use npm_registry to cache package Enrico Scholz
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

Helper module to:

- generate meta information from package.json content.  This data has
  a format as provided by https://registry.npmjs.org

- put this meta information and the corresponding tarball in the
  nodejs cache.  This uses an external, nodejs version specific helper
  script (oe-npm-cache) shipped in oe-meta

To avoid further nodejs version dependencies, future versions of this
module might omit the caching completely and serve meta information
and tarball by an http server.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
 1 file changed, 169 insertions(+)
 create mode 100644 meta/lib/oe/npm_registry.py

diff --git a/meta/lib/oe/npm_registry.py b/meta/lib/oe/npm_registry.py
new file mode 100644
index 0000000000..96c0affb45
--- /dev/null
+++ b/meta/lib/oe/npm_registry.py
@@ -0,0 +1,169 @@
+import bb
+import json
+import subprocess
+
+_ALWAYS_SAFE = frozenset('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+                         'abcdefghijklmnopqrstuvwxyz'
+                         '0123456789'
+                         '_.-~')
+
+MISSING_OK = object()
+
+REGISTRY = "https://registry.npmjs.org"
+
+# we can not use urllib.parse here because npm expects lowercase
+# hex-chars but urllib generates uppercase ones
+def uri_quote(s, safe = '/'):
+    res = ""
+    safe_set = set(safe)
+    for c in s:
+        if c in _ALWAYS_SAFE or c in safe_set:
+            res += c
+        else:
+            res += '%%%02x' % ord(c)
+    return res
+
+class PackageJson:
+    def __init__(self, spec):
+        self.__spec = spec
+
+    @property
+    def name(self):
+        return self.__spec['name']
+
+    @property
+    def version(self):
+        return self.__spec['version']
+
+    @property
+    def empty_manifest(self):
+        return {
+            'name': self.name,
+            'description': self.__spec.get('description', ''),
+            'versions': {},
+        }
+
+    def base_filename(self):
+        return uri_quote(self.name, safe = '@')
+
+    def as_manifest_entry(self, tarball_uri):
+        res = {}
+
+        ## NOTE: 'npm install' requires more than basic meta information;
+        ## e.g. it takes 'bin' from this manifest entry but not the actual
+        ## 'package.json'
+        for (idx,dflt) in [('name', None),
+                           ('description', ""),
+                           ('version', None),
+                           ('bin', MISSING_OK),
+                           ('man', MISSING_OK),
+                           ('scripts', MISSING_OK),
+                           ('directories', MISSING_OK),
+                           ('dependencies', MISSING_OK),
+                           ('devDependencies', MISSING_OK),
+                           ('optionalDependencies', MISSING_OK),
+                           ('license', "unknown")]:
+            if idx in self.__spec:
+                res[idx] = self.__spec[idx]
+            elif dflt == MISSING_OK:
+                pass
+            elif dflt != None:
+                res[idx] = dflt
+            else:
+                raise Exception("%s-%s: missing key %s" % (self.name,
+                                                           self.version,
+                                                           idx))
+
+        res['dist'] = {
+            'tarball': tarball_uri,
+        }
+
+        return res
+
+class ManifestImpl:
+    def __init__(self, base_fname, spec):
+        self.__base = base_fname
+        self.__spec = spec
+
+    def load(self):
+        try:
+            with open(self.filename, "r") as f:
+                res = json.load(f)
+        except IOError:
+            res = self.__spec.empty_manifest
+
+        return res
+
+    def save(self, meta):
+        with open(self.filename, "w") as f:
+            json.dump(meta, f, indent = 2)
+
+    @property
+    def filename(self):
+        return self.__base + ".meta"
+
+class Manifest:
+    def __init__(self, base_fname, spec):
+        self.__base = base_fname
+        self.__spec = spec
+        self.__lockf = None
+        self.__impl = None
+
+    def __enter__(self):
+        self.__lockf = bb.utils.lockfile(self.__base + ".lock")
+        self.__impl  = ManifestImpl(self.__base, self.__spec)
+        return self.__impl
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        bb.utils.unlockfile(self.__lockf)
+
+class NpmCache:
+    def __init__(self, cache):
+        self.__cache = cache
+
+    @property
+    def path(self):
+        return self.__cache
+
+    def run(self, type, key, fname):
+        subprocess.run(['oe-npm-cache', self.__cache, type, key, fname],
+                       check = True)
+
+class NpmRegistry:
+    def __init__(self, path, cache):
+        self.__path = path
+        self.__cache = NpmCache(cache + '/_cacache')
+        bb.utils.mkdirhier(self.__path)
+        bb.utils.mkdirhier(self.__cache.path)
+
+    @staticmethod
+    ## This function is critical and must match nodejs expectations
+    def _meta_uri(spec):
+        return REGISTRY + '/' + uri_quote(spec.name, safe = '@')
+
+    @staticmethod
+    ## Exact return value does not matter; just make it look like a
+    ## usual registry url
+    def _tarball_uri(spec):
+        return '%s/%s/-/%s-%s.tgz' % (REGISTRY,
+                                      uri_quote(spec.name, safe = '@'),
+                                      uri_quote(spec.name, safe = '@/'),
+                                      spec.version)
+
+    def add_pkg(self, tarball, pkg_json):
+        pkg_json = PackageJson(pkg_json)
+        base = os.path.join(self.__path, pkg_json.base_filename())
+
+        with Manifest(base, pkg_json) as manifest:
+            meta = manifest.load()
+            tarball_uri = self._tarball_uri(pkg_json)
+
+            meta['versions'][pkg_json.version] = pkg_json.as_manifest_entry(tarball_uri)
+
+            manifest.save(meta)
+
+            ## Cache entries are a little bit dependent on the nodejs
+            ## version; version specific cache implementation must
+            ## mitigate differences
+            self.__cache.run('meta', self._meta_uri(pkg_json), manifest.filename);
+            self.__cache.run('tgz',  tarball_uri, tarball);
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH 6/6] npm: use npm_registry to cache package
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
                   ` (4 preceding siblings ...)
  2022-05-19 10:05 ` [PATCH 5/6] lib:npm_registry: initial checkin Enrico Scholz
@ 2022-05-19 10:05 ` Enrico Scholz
  2022-05-19 10:38 ` [OE-core] [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
  2022-05-23  7:23 ` Christian Eggers
  7 siblings, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:05 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

With nodejs 16, the simple 'npm cache add' approach does not work
anymore because its fetcher implementation downloads also meta
information from the registry.

We have to generate these information and add them to the cache.
There is no direct support in 'npm' for task so we have to implement
it manually.

This implementation consists of a openembedded python module (in
oe-core) and a nodejs version specific helper (in oe-meta).

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 1e41072116..f3c3e2f5b7 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -19,7 +19,7 @@
 
 inherit python3native
 
-DEPENDS:prepend = "nodejs-native "
+DEPENDS:prepend = "nodejs-native nodejs-oe-cache-native "
 RDEPENDS:${PN}:append:class-target = " nodejs"
 
 EXTRA_OENPM = ""
@@ -46,6 +46,7 @@ NPM_ARCH ?= "${@npm_target_arch_map(d.getVar("TARGET_ARCH"))}"
 NPM_PACKAGE = "${WORKDIR}/npm-package"
 NPM_CACHE = "${WORKDIR}/npm-cache"
 NPM_BUILD = "${WORKDIR}/npm-build"
+NPM_REGISTRY = "${WORKDIR}/npm-registry"
 
 def npm_global_configs(d):
     """Get the npm global configuration"""
@@ -111,16 +112,18 @@ python npm_do_configure() {
     from bb.fetch2.npm import npm_unpack
     from bb.fetch2.npmsw import foreach_dependencies
     from bb.progress import OutOfProgressHandler
+    from oe.npm_registry import NpmRegistry
 
     bb.utils.remove(d.getVar("NPM_CACHE"), recurse=True)
     bb.utils.remove(d.getVar("NPM_PACKAGE"), recurse=True)
 
     env = NpmEnvironment(d, configs=npm_global_configs(d))
+    registry = NpmRegistry(d.getVar('NPM_REGISTRY'), d.getVar('NPM_CACHE'))
 
-    def _npm_cache_add(tarball):
-        """Run 'npm cache add' for a specified tarball"""
-        cmd = "npm cache add %s" % shlex.quote(tarball)
-        env.run(cmd)
+    def _npm_cache_add(tarball, pkg):
+        """Add tarball to local registry and register it in the
+           cache"""
+        registry.add_pkg(tarball, pkg)
 
     def _npm_integrity(tarball):
         """Return the npm integrity of a specified tarball"""
@@ -184,7 +187,7 @@ python npm_do_configure() {
             # Add the dependency to the npm cache
             destdir = os.path.join(d.getVar("S"), destsuffix)
             (tarball, pkg) = npm_pack(env, destdir, tmpdir)
-            _npm_cache_add(tarball)
+            _npm_cache_add(tarball, pkg)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
             dep["version"] = pkg['version']
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [OE-core] [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf'
  2022-05-19 10:05 ` [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf' Enrico Scholz
@ 2022-05-19 10:25   ` Zoltan Boszormenyi
  0 siblings, 0 replies; 12+ messages in thread
From: Zoltan Boszormenyi @ 2022-05-19 10:25 UTC (permalink / raw)
  To: enrico.scholz, openembedded-core

2022. 05. 19. 12:05 keltezéssel, Enrico Scholz via lists.openembedded.org írta:
> 'npm pack' is a maintainer tool which tries to execute 'prepare'
> and similar scripts.  This fails usually in OE because it requires
> completely installed 'node_modules'.
> 
> Earlier nodejs versions supported an undocumented 'ignore-scripts'
> option.  This has been removed in nodejs 16.
> 
> We could patch 'package.json' and remove the unwanted scripts.  But
> this might complicate local workflows (applying patches) and installed
> packages will contain the modified 'package.json'.
> 
> Instead of, package it manually by 'tar czf'.  As a sideeffect,
> 'do_configure' is running much faster now.

\o/ Hurray!

> 
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>   meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------
>   1 file changed, 29 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
> index ba50fcac20..91f8f36b0d 100644
> --- a/meta/classes/npm.bbclass
> +++ b/meta/classes/npm.bbclass
> @@ -57,13 +57,36 @@ def npm_global_configs(d):
>       configs.append(("cache", d.getVar("NPM_CACHE")))
>       return configs
>   
> +## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for
> +## 'ignore-scripts' which prevents this behavior has been removed
> +## from nodejs 16.  Use simple 'tar' instead of.
>   def npm_pack(env, srcdir, workdir):
> -    """Run 'npm pack' on a specified directory"""
> -    import shlex
> -    cmd = "npm pack %s" % shlex.quote(srcdir)
> -    args = [("ignore-scripts", "true")]
> -    tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
> -    return os.path.join(workdir, tarball)
> +    """Emulate 'npm pack' on a specified directory"""
> +    import subprocess
> +    import os
> +    import json
> +
> +    src = os.path.join(srcdir, 'package.json')
> +    with open(src) as f:
> +        j = json.load(f)
> +
> +    # base does not really matter and is for documentation purposes
> +    # only.  But the 'version' part must exist because other parts of
> +    # the bbclass rely on it.
> +    base = j['name'].split('/')[-1]
> +    tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version']));
> +
> +    # TODO: real 'npm pack' does not include directories while 'tar'
> +    # does.  But this does not seem to matter...
> +    subprocess.run(['tar', 'czf', tarball,
> +                    '--exclude', './node-modules',
> +                    '--exclude-vcs',
> +                    '--transform', 's,^\./,package/,',
> +                    '--mtime', '1985-10-26T08:15:00.000Z',
> +                    '.'],
> +                   check = True, cwd = srcdir)
> +
> +    return tarball
>   
>   python npm_do_configure() {
>       """
> 
> 
> 
> 
> 


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [OE-core] [PATCH 0/6] npm.bbclass: work with nodejs 16
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
                   ` (5 preceding siblings ...)
  2022-05-19 10:05 ` [PATCH 6/6] npm: use npm_registry to cache package Enrico Scholz
@ 2022-05-19 10:38 ` Enrico Scholz
  2022-05-23  7:23 ` Christian Eggers
  7 siblings, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 10:38 UTC (permalink / raw)
  To: openembedded-core

"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> writes:

> This patchset requires an additional one in the oe-meta layer.

for reference: https://lists.openembedded.org/g/openembedded-devel/message/97186


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [OE-core] [PATCH 4/6] npm: disable 'audit' + 'fund'
  2022-05-19 10:05 ` [PATCH 4/6] npm: disable 'audit' + 'fund' Enrico Scholz
@ 2022-05-19 13:56   ` Luca Ceresoli
  2022-05-19 14:06   ` [PATCH 4/6, v2] " Enrico Scholz
  1 sibling, 0 replies; 12+ messages in thread
From: Luca Ceresoli @ 2022-05-19 13:56 UTC (permalink / raw)
  To: Enrico Scholz via lists.openembedded.org; +Cc: enrico.scholz, openembedded-core

Il giorno Thu, 19 May 2022 12:05:54 +0200
"Enrico Scholz via lists.openembedded.org"
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> ha scritto:

> 'audit' can cause extra network traffic; 'fund' is not needed.
> 
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>  meta/classes/npm.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
> index 11c80a738e..1e41072116 100644
> --- a/meta/classes/npm.bbclass
> +++ b/meta/classes/npm.bbclass
> @@ -53,6 +53,8 @@ def npm_global_configs(d):
>      # Ensure no network access is done
>      configs.append(("offline", "true"))
>      configs.append(("proxy", "http://invalid"))
> +    configs.append(("funds", False))

'fund' or 'funds'?



-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH 4/6, v2] npm: disable 'audit' + 'fund'
  2022-05-19 10:05 ` [PATCH 4/6] npm: disable 'audit' + 'fund' Enrico Scholz
  2022-05-19 13:56   ` [OE-core] " Luca Ceresoli
@ 2022-05-19 14:06   ` Enrico Scholz
  1 sibling, 0 replies; 12+ messages in thread
From: Enrico Scholz @ 2022-05-19 14:06 UTC (permalink / raw)
  To: openembedded-core; +Cc: Enrico Scholz

'audit' can cause extra network traffic; 'fund' is not needed.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/classes/npm.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 11c80a738e..b613c8e8f5 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -53,6 +53,8 @@ def npm_global_configs(d):
     # Ensure no network access is done
     configs.append(("offline", "true"))
     configs.append(("proxy", "http://invalid"))
+    configs.append(("fund", False))
+    configs.append(("audit", False))
     # Configure the cache directory
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [OE-core] [PATCH 0/6] npm.bbclass: work with nodejs 16
  2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
                   ` (6 preceding siblings ...)
  2022-05-19 10:38 ` [OE-core] [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
@ 2022-05-23  7:23 ` Christian Eggers
  7 siblings, 0 replies; 12+ messages in thread
From: Christian Eggers @ 2022-05-23  7:23 UTC (permalink / raw)
  To: openembedded-core, Stefan Herbrechtsmeier; +Cc: Enrico Scholz

This seems to solve my problem with the quicktype recipe [1]. Thank
you very much for working in this.

[1] https://lore.kernel.org/all/CANNYZj-V8O1-USTdtOYKi5hB80QjxQZH2d2=YbJ-4Y2sm5BjoA@mail.gmail.com/T/


On Thursday, 19 May 2022, 12:05:50 CEST, Enrico Scholz via lists.openembedded.org wrote:
> nodejs 16 changed internal caching significantly which breaks the
> existing npm.bblcass.
> 
> Simulate parts of the npm registry and cache data in the way as
> expected.
> 
> This patchset requires an additional one in the oe-meta layer.
> 
> Enrico Scholz (6):
>   npm: replace 'npm pack' call by 'tar czf'
>   npm: return content of 'package.json' in 'npm_pack'
>   npm: take 'version' directly from 'package.json'
>   npm: disable 'audit' + 'fund'
>   lib:npm_registry: initial checkin
>   npm: use npm_registry to cache package
> 
>  meta/classes/npm.bbclass    |  65 +++++++++-----
>  meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 213 insertions(+), 21 deletions(-)
>  create mode 100644 meta/lib/oe/npm_registry.py
> 
> 






^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2022-05-23  7:23 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-19 10:05 [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
2022-05-19 10:05 ` [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf' Enrico Scholz
2022-05-19 10:25   ` [OE-core] " Zoltan Boszormenyi
2022-05-19 10:05 ` [PATCH 2/6] npm: return content of 'package.json' in 'npm_pack' Enrico Scholz
2022-05-19 10:05 ` [PATCH 3/6] npm: take 'version' directly from 'package.json' Enrico Scholz
2022-05-19 10:05 ` [PATCH 4/6] npm: disable 'audit' + 'fund' Enrico Scholz
2022-05-19 13:56   ` [OE-core] " Luca Ceresoli
2022-05-19 14:06   ` [PATCH 4/6, v2] " Enrico Scholz
2022-05-19 10:05 ` [PATCH 5/6] lib:npm_registry: initial checkin Enrico Scholz
2022-05-19 10:05 ` [PATCH 6/6] npm: use npm_registry to cache package Enrico Scholz
2022-05-19 10:38 ` [OE-core] [PATCH 0/6] npm.bbclass: work with nodejs 16 Enrico Scholz
2022-05-23  7:23 ` Christian Eggers

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.