Re: [bitbake-devel] Improving npm(sw) fetcher & integration within Bitbake

[Alexander Kanavin <alex.kanavin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5605 bytes --]

Hello Jasper,

I want to invite Stefan to this discussion because he's been facing similar
issues in integrating npm code with Yocto. He's been working on some
improvements to npm fetcher and class, and is most qualified to comment, so
maybe you guys can work out a plan to make it shine (or at least not be too
horrible).

To the best of my knowledge, no other companies at the moment are using
Yocto to integrate npm-based items into a product.

Alex

On Thu, 4 Nov 2021 at 13:29, Jasper Orschulko <
Jasper.Orschulko@iris-sensing.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Dear Bitbake developers,
>
> recently we have been looking at the npmsw fetcher and discovered some
> challenges regarding the integration into the developer workflow as
> well as the build times within Bitbake. We believe that we found a
> mechanism which would integrate well into Bitbake's existing project
> structure and drastically improve the situation.
>
>
> But first, what are the issues with the current npmsw fetcher?
>
> 1. Let's have a look at a typical npm-based project. You'd typically
> have your package-lock.json (aka shrinkwrap file) stored within the git
> repository containing your source code. Developers will rely on this
> package-lock file on a daily basis during the development cycle.
> Unfortunately, the current npmsw fetcher only supports shrinkwrap files
> stored within the meta layer or within an npm registry. This is not
> ideal, as changes to the file might be made within the project repo,
> which then need to be manually applied to the lock file within the meta
> repo. An ideal npmsw fetcher therefore would support using the lock
> file directly from the source code repo.
>
> 2. The current implementation of the npm class uses multiple shellouts
> per npm module in order to add these to the npm cache. This is done, as
> the `npm install` command is not called within the do_fetch, but at the
> end of the do_configure step. This drastically increases the time
> Bitbake spends in the do_configure step for a npm based recipe. In our
> case (we have a relatively small project with approx. 600 npm packages
> in total, including recursive packages) this takes ~100 minutes to
> complete. What makes things worse, every change to the recipe and/or
> lock file will cause a complete rerun of the do_configure job.
>
> As a result, the npm fetcher currently is not really usable for
> production workloads.
>
>
> So how can we address these issues?
>
> We plan to implement a "sub-fetcher" for npmsw (a concept which might
> also be recyclable for similar use-cases). This would take the
> form of e.g.:
>
> SRC_URI = "npmsw+git://git-uri.git;npm-topdir=path_to_npm_project;..."
>
> The idea is, that the npsw fetcher would then call an arbitrary sub-
> fetcher (in this case git, however any fetcher will be supported) and
> after the sub-fetcher has extracted the source code into the DL_DIR,
> the npm fetcher will create a secondary download folder as a copy of
> the sub-fetchers download folder. Within this copy, the npm fetcher
> will call `npm ci`, effectively downloading the npm packages by doing a
> clean-install on the basis of the package.json and the package-
> lock.json files within the npmsw download dir. This results in a much
> faster build, as it removes the need for seperate handling of the
> individual node packages, as well as streamlining the developers
> workflow with the build process within Bitbake.
>
> As this fetcher would be implemented separately from the current npmsw
> fetcher, this will not cause any breaking changes for existing setups.
>
> Additionally, we plan on writing a separate npmsw.bbclass, which will
> parse the package.json for each node module for an automated Bitbake
> license manifest generation, which will resolve the current challenge
> of having to maintain these manually, as currently described at
>
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#npm-using-the-registry-modules-method
> .
>
> If this is something you see as a worthwhile goal, we will provide a
> set of patch files within the coming weeks.
>
> - --
> With best regards
>
> Jasper Orschulko
> DevOps Engineer
>
> Tel. +49 30 58 58 14 265
> Fax +49 30 58 58 14 999
> Jasper.Orschulko@iris-sensing.com
>
> • • • • • • • • • • • • • • • • • • • • • • • • • •
>
> iris-GmbH
> infrared & intelligent sensors
> Schnellerstraße 1-5 | 12439 Berlin
>
> https://iris-sensing.com/
>
>
>
>
> -----BEGIN PGP SIGNATURE-----
>
> iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGD0jsACgkQYgqew07V
> MNVa1Qf+MwrlwXeS+FI8JrtHdTJ5CNJ64DkiTe0Tgqb7SQVkawXlm6KPezBYFIZb
> HmAruV8vpQpUHkyKXpwuH4X0A2CO3jJ9v20H3sdRfL+33gSCOY9+LDOADlRf1MhT
> fkV+OqwYufwG02ZnOpF3YRcTiDfG9UEzM+lzArOzhY6GjMp4FxvQZ/xLjEvdGVJ1
> l3h3NPmsedqsJnal022wkPi2gN2ZCQPyIw11EJL929wqwPHudvqj8OX2q1JhDUn5
> vjRpN2l4yg6g8bpF1If+5YkT/ZVjrZqcleL9pYPpVOuqk+j/iWbPSTmXjvzsrKDL
> 1UWqTBtRmWMYH+xkjKxD7Spjz1scRA==
> =qjQA
> -----END PGP SIGNATURE-----
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12875):
> https://lists.openembedded.org/g/bitbake-devel/message/12875
> Mute This Topic: https://lists.openembedded.org/mt/86814331/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 7006 bytes --]