From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mail.openembedded.org (Postfix) with ESMTP id 2532F7F8AA for ; Thu, 21 Nov 2019 06:36:51 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Nov 2019 22:36:53 -0800 X-IronPort-AV: E=Sophos;i="5.69,224,1571727600"; d="scan'208,217";a="209996449" Received: from nvkremer-mobl1.amr.corp.intel.com ([10.252.133.123]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Nov 2019 22:36:52 -0800 From: Tim Orling Message-Id: <315444F4-D824-4698-B0EB-B20309CD6809@linux.intel.com> Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Date: Wed, 20 Nov 2019 22:36:52 -0800 In-Reply-To: <549C1F29-84E4-43A3-9DD2-FBED13A92C37@linux.intel.com> To: Jean-Marie LEMETAYER References: <20191120093358.11622-1-jean-marie.lemetayer@savoirfairelinux.com> <549C1F29-84E4-43A3-9DD2-FBED13A92C37@linux.intel.com> X-Mailer: Apple Mail (2.3445.104.11) Cc: paul.eggleton@linux.intel.com, openembedded-core@lists.openembedded.org, rennes@savoirfairelinux.com, bunk@stusta.de, jonaskgandersson@gmail.com Subject: Re: [PATCH v3 00/17] NPM refactoring X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2019 06:36:52 -0000 Content-Type: multipart/alternative; boundary="Apple-Mail=_E4F395CE-454C-4D30-8DEB-D313FDD10975" --Apple-Mail=_E4F395CE-454C-4D30-8DEB-D313FDD10975 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 20, 2019, at 10:27 PM, Tim Orling = wrote: >=20 >=20 >=20 >> On Nov 20, 2019, at 1:33 AM, Jean-Marie LEMETAYER = > wrote: >>=20 >> The current NPM support have several issues: >> - The current NPM fetcher downloads the dependency tree but not the = other >> fetchers. The 'subdir' parameter was used to fix this issue. >> - They are multiple issues with package names (uppercase, exotic = characters, >> scoped packages) even if they are inside the dependencies. >> - The lockdown file generation have issues. When a package depends on >> multiple version of the same package (all versions have the same = checksum). >>=20 >> This patchset refactors the NPM support in Yocto: >> - As the NPM algorithm for dependency management is hard to handle, = the new >> NPM fetcher downloads only the package source (and not the = dependencies, >> like the other fetchers) (patch submitted in the bitbake-devel = list). >> - The NPM class handles the dependencies using NPM (and not = manually). >> - The NPM recipe creation is simplified to avoid issues. >> - The lockdown file is no more used as it is no longer relevant = compared to the >> latest shrinkwrap file format. >>=20 >> This patchset may remove some features (lockdown file, license = management for >> dependencies) but fixes the majority of the NPM issues. All of these = issues >> from the bugzilla.yoctoproject.org = are resolved by this patchset: >> #10237, #10760, #11028, #11728, #11902, #12534 >>=20 >> The fetcher and recipetool are now aware of a 'latest' keyword for = the version >> which allow to build the latest version available on the registry. = This feature >> fixes the two issues: #10515, #11029 >>=20 >> Moreover the issue #13415 should also be fixed but I cannot test it. >>=20 >> I have tested the recipe creation and builds using a self made = example to >> generate build failures: >> - https://github.com/savoirfairelinux/node-server-example = >> - https://npmjs.com/package/@savoirfairelinux/node-server-example = >>=20 >> The test steps are these ones: >> $ source poky/oe-init-build-env >> $ bitbake-layers add-layer ../meta-openembedded/meta-oe >> $ devtool add = "npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-example;ver= sion=3Dlatest = =E2=80=9D >=20 > I noticed this before but forgot to report it: >=20 > ERROR: Nothing PROVIDES 'c-ares-native' (but = virtual:native:/build/meta-openembedded/meta-oe/recipes-devtools/nodejs/no= dejs_10.17.0.bb DEPENDS on or otherwise requires it). ... > http://layers.openembedded.org/layerindex/recipe/50318/ = > $ bitbake-layers add-layer ../meta-openembedded/meta-networking > Which requires > $ bitbake-layers add-layer ../meta-openembedded/meta-python >=20 >=20 > Using tip of bitbake with V3, openembedded-core with V3 and = meta-openembedded master, somehow nodejs-native do_compile broke (at = least on qemux86-64) since the last time I built it shortly after = ELC-E=E2=80=A6 > So now I=E2=80=99ll have to git bisect that=E2=80=A6more to come=E2=80=A6= See https://errors.yoctoproject.org/Errors/Details/274939/ = The same error I am seeing. >=20 > FWIW I had previously successfully run your instructions and the tests = for both bitbake and oe-selftest for V2 (distroless, e.g. not with = poky). Something unrelated has broken I suspect. >=20 >> $ devtool build savoirfairelinux-node-server-example >> $ bitbake-layers create-layer ../meta-test >> $ bitbake-layers add-layer ../meta-test >> $ devtool finish savoirfairelinux-node-server-example ../meta-test >> $ echo IMAGE_INSTALL_append =3D '" = savoirfairelinux-node-server-example"' >> conf/local.conf >> $ bitbake core-image-minimal >>=20 >> Also the 'devtool add' url could be one of these: >> - = npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-example;vers= ion=3Dlatest = >> This url uses the new npm fetcher and request the latest version = available on >> the registry. >> - = npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-example;vers= ion=3D1.0.0 = >> This url uses the new npm fetcher and request a fixed version. >> - = git://github.com/savoirfairelinux/node-server-example.git;protocol=3Dhttps= = >> This url uses the git fetcher. As the dependencies are managed by = the NPM >> class, any fetcher can be used. >>=20 >> When this patchset will be merged, I have planned to update the NPM = wiki: >> https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM = >>=20 >> This patchset is also the base work for the full Angular support in = Yocto that I >> am preparing. These applications have a huge dependency tree which is = ideal to >> test the NPM support. >>=20 >> --- V2 >>=20 >> - Add the 'check_network_access' function before each network access = to check >> for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables. >>=20 >> - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test = case for >> 'oe-selftest' to test the npm recipe creation. >>=20 >> - Update the 'npm.bbclass' to fetch the dependencies in the = 'do_fetch' task. >> The dependencies are cached in a npm cache stored in = '${DL_DIR}/npm_cache' >> allowing the dependencies to be saved in the download directory and = verify >> on insertion and extraction [1]: >> "All data that passes through the cache is fully verified >> for integrity on both insertion and extraction." >>=20 >> 1: https://docs.npmjs.com/cli/cache.html = >>=20 >> --- V3 >>=20 >> - Add a test regarding the 'devtool add' and 'devtool build' command: >> - devtool.DevtoolAddTests.test_devtool_add_npm >>=20 >> - Add the license management for dependencies. >>=20 >> - Split the npm workflow to use the bitbake default tasks (do_fetch, = do_unpack, >> do_compile, do_install). Make sure that only the do_fetch task = requires >> network access. >>=20 >> - Split the commits for better understanding. >>=20 >> - Force the rebuild of prebuild addons. >>=20 >> - Use ${nonarch_libdir} instead of ${libdir} in the do_install task. >>=20 >> - These patches can be found here: >> - = https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring= -v3 = >> - = https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v3 = >>=20 >> Jean-Marie LEMETAYER (17): >> devtool: npm: update command line options >> npm.bbclass: refactor the npm class >> recipetool/create_npm.py: refactor the npm recipe creation handler >> lib/oe/package.py: remove unneeded npm_split_package_dirs function >> devtool/standard.py: npm: update the append file >> recipetool/create.py: npm: remove the 'noverify' url parameter >> recipetool/create.py: npm: replace the 'latest' keyword >> npm.bbclass: split the do_compile task >> devtool/standard.py: npm: exclude the node_modules directory >> recipetool/create_npm.py: handle the licenses of the dependencies >> npm.bbclass: restrict the build to be offline >> npm.bbclass: use the local node headers >> recipetool/create_npm.py: convert the shrinkwrap file >> npm.bbclass: force to rebuild the prebuild addons >> devtool/standard.py: npm: configure the NPM_CACHE_DIR to be = persistent >> oeqa/selftest/recipetool: add npm recipe creation test >> oeqa/selftest/devtool: add npm recipe build test >>=20 >> meta/classes/npm.bbclass | 224 +++++--- >> meta/lib/oe/package.py | 33 -- >> meta/lib/oeqa/selftest/cases/devtool.py | 20 + >> meta/lib/oeqa/selftest/cases/recipetool.py | 21 + >> scripts/lib/devtool/standard.py | 31 +- >> scripts/lib/recipetool/create.py | 16 +- >> scripts/lib/recipetool/create_npm.py | 565 = +++++++++++---------- >> 7 files changed, 510 insertions(+), 400 deletions(-) >>=20 >> -- >> 2.20.1 >>=20 >=20 > --=20 > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org = > http://lists.openembedded.org/mailman/listinfo/openembedded-core = --Apple-Mail=_E4F395CE-454C-4D30-8DEB-D313FDD10975 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Nov 20, 2019, at 10:27 PM, Tim Orling <timothy.t.orling@linux.intel.com> wrote:



On Nov 20, 2019, at 1:33 AM, = Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> = wrote:

The current NPM support have several issues:
- = The current NPM fetcher downloads the dependency tree but not the = other
  fetchers. The 'subdir' parameter was = used to fix this issue.
- They are multiple issues with = package names (uppercase, exotic characters,
  scoped packages) even if they are inside the = dependencies.
- The lockdown file generation have issues. = When a package depends on
  multiple version of = the same package (all versions have the same checksum).

This patchset refactors the NPM support in Yocto:
- As the NPM algorithm for dependency management is hard to = handle, the new
  NPM fetcher downloads only the = package source (and not the dependencies,
  like = the other fetchers) (patch submitted in the bitbake-devel list).
- The NPM class handles the dependencies using NPM (and not = manually).
- The NPM recipe creation is simplified to = avoid issues.
- The lockdown file is no more used as it is = no longer relevant compared to the
  latest = shrinkwrap file format.

This patchset may = remove some features (lockdown file, license management for
dependencies) but fixes the majority of the NPM issues. All = of these issues
from the bugzilla.yoctoproject.org are resolved by this = patchset:
#10237, #10760, #11028, #11728, #11902, = #12534

The fetcher and recipetool are now = aware of a 'latest' keyword for the version
which allow to = build the latest version available on the registry. This feature
fixes the two issues: #10515, #11029

Moreover the issue #13415 should also be fixed but I cannot = test it.

I have tested the recipe creation = and builds using a self made example to
generate build = failures:
- https://github.com/savoirfairelinux/node-server-example
- https://npmjs.com/package/@savoirfairelinux/node-server-example=

The test steps are these ones:
 $ source poky/oe-init-build-env
 $ = bitbake-layers add-layer ../meta-openembedded/meta-oe
 $ devtool add "npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-e= xample;version=3Dlatest=E2=80=9D

I noticed this before = but forgot to report it:

ERROR: = Nothing PROVIDES 'c-ares-native' (but = virtual:native:/build/meta-openembedded/meta-oe/recipes-devtools/nodejs/no= dejs_10.17.0.bb DEPENDS on or otherwise requires it). ...
http://layers.openembedded.org/layerindex/recipe/50318/
$ bitbake-layers add-layer = ../meta-openembedded/meta-networking
Which requires
$ = bitbake-layers add-layer ../meta-openembedded/meta-python


Using tip of bitbake with V3, = openembedded-core with V3 and meta-openembedded master, somehow = nodejs-native do_compile broke (at least on qemux86-64) since the last = time I built it shortly after ELC-E=E2=80=A6
So = now I=E2=80=99ll have to git bisect that=E2=80=A6more to = come=E2=80=A6

See https://errors.yoctoproject.org/Errors/Details/274939/
The same error I am seeing.


FWIW = I had previously successfully run your instructions and the tests for = both bitbake and oe-selftest for V2 (distroless, e.g. not with poky). = Something unrelated has broken I suspect.

 $ devtool build savoirfairelinux-node-server-example
 $ bitbake-layers create-layer ../meta-test
 $ bitbake-layers add-layer ../meta-test
 $ devtool finish savoirfairelinux-node-server-example = ../meta-test
 $ echo IMAGE_INSTALL_append =3D '" = savoirfairelinux-node-server-example"' >> conf/local.conf
 $ bitbake core-image-minimal

Also the 'devtool add' url could be one of these:
- npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-e= xample;version=3Dlatest
  This url uses the = new npm fetcher and request the latest version available on
  the registry.
- npm://registry.npmjs.org;name=3D@savoirfairelinux/node-server-e= xample;version=3D1.0.0
  This url uses the = new npm fetcher and request a fixed version.
- git://github.com/savoirfairelinux/node-server-example.git;proto= col=3Dhttps
  This url uses the git fetcher. = As the dependencies are managed by the NPM
  class, any fetcher can be used.

When this patchset will be merged, I have planned to update = the NPM wiki:
 https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

This patchset is also the base work for the = full Angular support in Yocto that I
am preparing. These = applications have a huge dependency tree which is ideal to
test the NPM support.

--- V2

- Add the 'check_network_access' function = before each network access to check
  for = 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.

- Add a = 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for
  'oe-selftest' to test the npm recipe creation.

- Update the 'npm.bbclass' to fetch the = dependencies in the 'do_fetch' task.
  The = dependencies are cached in a npm cache stored in = '${DL_DIR}/npm_cache'
  allowing the = dependencies to be saved in the download directory and verify
  on insertion and extraction [1]:
     "All data that passes through = the cache is fully verified
      for integrity on both = insertion and extraction."

1: https://docs.npmjs.com/cli/cache.html

--- V3

- Add a test regarding = the 'devtool add' and 'devtool build' command:
    - = devtool.DevtoolAddTests.test_devtool_add_npm

- Add the license management for dependencies.

- Split the npm workflow to use the bitbake = default tasks (do_fetch, do_unpack,
  do_compile, = do_install). Make sure that only the do_fetch task requires
  network access.

- = Split the commits for better understanding.

-= Force the rebuild of prebuild addons.

- = Use ${nonarch_libdir} instead of ${libdir} in the do_install task.

 - These patches can be found here:
    - https://github.com/savoirfairelinux/openembedded-core/tree/npm-= refactoring-v3
    - https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v= 3

Jean-Marie LEMETAYER (17):
 devtool: npm: update command line options
 npm.bbclass: refactor the npm class
 recipetool/create_npm.py: refactor the npm recipe = creation handler
 lib/oe/package.py: remove unneeded = npm_split_package_dirs function
 devtool/standard.py: = npm: update the append file
 recipetool/create.py: = npm: remove the 'noverify' url parameter
 recipetool/create.py: npm: replace the 'latest' = keyword
 npm.bbclass: split the do_compile task
 devtool/standard.py: npm: exclude the node_modules = directory
 recipetool/create_npm.py: handle the = licenses of the dependencies
 npm.bbclass: restrict = the build to be offline
 npm.bbclass: use the local = node headers
 recipetool/create_npm.py: convert the = shrinkwrap file
 npm.bbclass: force to rebuild the = prebuild addons
 devtool/standard.py: npm: configure = the NPM_CACHE_DIR to be persistent
 oeqa/selftest/recipetool: add npm recipe creation = test
 oeqa/selftest/devtool: add npm recipe build = test

meta/classes/npm.bbclass =             &n= bsp;     | 224 +++++---
meta/lib/oe/package.py =             &n= bsp;       |  33 --
meta/lib/oeqa/selftest/cases/devtool.py    | =  20 +
meta/lib/oeqa/selftest/cases/recipetool.py | =  21 +
scripts/lib/devtool/standard.py =            | =  31 +-
scripts/lib/recipetool/create.py =           |  16 = +-
scripts/lib/recipetool/create_npm.py =       | 565 +++++++++++----------
7 files changed, 510 insertions(+), 400 deletions(-)

--
2.20.1


-- 
_______________________________________________
Openembedded-core mailing = list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-cor= e

= --Apple-Mail=_E4F395CE-454C-4D30-8DEB-D313FDD10975--