Re: Improve npm support to run build scripts

[Stefan Herbrechtsmeier <stefan.herbrechtsmeier-oss@weidmueller.com>]

Hi Richard,

Am 05.10.2021 um 13:09 schrieb Richard Purdie:
> Thanks for bring this up, I've been wondering on the status of our npm support.

Do you know any npm users?

> On Tue, 2021-10-05 at 12:04 +0200, Stefan Herbrechtsmeier wrote:
>> I must improve the npm support for our use cases but need some
>> fundamental decisions to proceed. I need to build express and angular
>> applications from git repositories.
>>
>> I have the following changes in my pipeline until now:
>> - Support npm packages with missing execute mode for directories
>> - Reduce command calls in npm run
>> - Add support for local tarball and local link sources
>> - Rework crunch license to recognize more variants
>> - Move known license checksums to CSV file
>> - Pass README as fallback to guess license if a license is missing
>> - Add support to pass build arguments to npm install
>> - Add support to run build scripts with native packages
>> - Parallelize the populate of the npm cache
>>
>> The build (install) of npm packages leads to problems because of old
>> versions in the dependency tree. For example, older versions of packages
>> are incompatible with the Node.js version and older versions of node-gyp
>> require Python 2.
> 
> Hopefully over time the python2 issue should reduce and fade at least. The
> node.js version is trickier.

The current implementation works like an project with embedded external 
dependencies. I both cases I have to patch the embedded dependencies or 
replace them with external DEPENDS.

>> I see three solutions to integrate npm packages:
>> - Build npm packages from the npm project outside of OE and only fetch
>> the data.
>> - Fork the npm project repository and fix everything inside the fork.
>> Use the fork direct or create patches from the fork to build inside OE.
>> - Create recipes per project and incompatible version to fix everything
>> in OE and reduce redundancy.
>>
>> The last solution keeps everything in OE and benefit from the existing
>> OE functions like download proxy as well as license, CVE and version
>> check. But it increases the initial costs and makes only sense if there
>> is an interest in an meta-nodejs layer.
> 
> The latter solution sounds like the only really viable option from an OE
> perspective since we do need the proxy/CVE/license handling, it is a huge part
> of our value and without it, it isn't really OE.

The problem are the many different packages. An Angular project could 
have more than 1000 different project. Thereby some dependencies only 
differ in the version or needed only for development like a development 
server or a unit test framework.

> Creating a meta-nodejs layer isn't an issue if there are people willing to look
> after it.

Is a layer with more more than 1000 recipes thinkable?

Regards
   Stefan