Improve npm support to run build scripts

Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Jean-Marie and Richard,

I must improve the npm support for our use cases but need some 
fundamental decisions to proceed. I need to build express and angular 
applications from git repositories.

I have the following changes in my pipeline until now:
- Support npm packages with missing execute mode for directories
- Reduce command calls in npm run
- Add support for local tarball and local link sources
- Rework crunch license to recognize more variants
- Move known license checksums to CSV file
- Pass README as fallback to guess license if a license is missing
- Add support to pass build arguments to npm install
- Add support to run build scripts with native packages
- Parallelize the populate of the npm cache

The build (install) of npm packages leads to problems because of old 
versions in the dependency tree. For example, older versions of packages 
are incompatible with the Node.js version and older versions of node-gyp 
require Python 2.

I see three solutions to integrate npm packages:
- Build npm packages from the npm project outside of OE and only fetch 
the data.
- Fork the npm project repository and fix everything inside the fork. 
Use the fork direct or create patches from the fork to build inside OE.
- Create recipes per project and incompatible version to fix everything 
in OE and reduce redundancy.

The last solution keeps everything in OE and benefit from the existing 
OE functions like download proxy as well as license, CVE and version 
check. But it increases the initial costs and makes only sense if there 
is an interest in an meta-nodejs layer.

Regards
   Stefan

Re: Improve npm support to run build scripts

[Richard Purdie]

Hi Stefan,

Thanks for bring this up, I've been wondering on the status of our npm support.

On Tue, 2021-10-05 at 12:04 +0200, Stefan Herbrechtsmeier wrote:
> I must improve the npm support for our use cases but need some 
> fundamental decisions to proceed. I need to build express and angular 
> applications from git repositories.
> 
> I have the following changes in my pipeline until now:
> - Support npm packages with missing execute mode for directories
> - Reduce command calls in npm run
> - Add support for local tarball and local link sources
> - Rework crunch license to recognize more variants
> - Move known license checksums to CSV file
> - Pass README as fallback to guess license if a license is missing
> - Add support to pass build arguments to npm install
> - Add support to run build scripts with native packages
> - Parallelize the populate of the npm cache
> 
> The build (install) of npm packages leads to problems because of old 
> versions in the dependency tree. For example, older versions of packages 
> are incompatible with the Node.js version and older versions of node-gyp 
> require Python 2.

Hopefully over time the python2 issue should reduce and fade at least. The
node.js version is trickier.

> 
> I see three solutions to integrate npm packages:
> - Build npm packages from the npm project outside of OE and only fetch 
> the data.
> - Fork the npm project repository and fix everything inside the fork. 
> Use the fork direct or create patches from the fork to build inside OE.
> - Create recipes per project and incompatible version to fix everything 
> in OE and reduce redundancy.
> 
> The last solution keeps everything in OE and benefit from the existing 
> OE functions like download proxy as well as license, CVE and version 
> check. But it increases the initial costs and makes only sense if there 
> is an interest in an meta-nodejs layer.

The latter solution sounds like the only really viable option from an OE
perspective since we do need the proxy/CVE/license handling, it is a huge part
of our value and without it, it isn't really OE.

Creating a meta-nodejs layer isn't an issue if there are people willing to look
after it.

Cheers,

Richard

Re: Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Richard,

Am 05.10.2021 um 13:09 schrieb Richard Purdie:
> Thanks for bring this up, I've been wondering on the status of our npm support.

Do you know any npm users?

> On Tue, 2021-10-05 at 12:04 +0200, Stefan Herbrechtsmeier wrote:
>> I must improve the npm support for our use cases but need some
>> fundamental decisions to proceed. I need to build express and angular
>> applications from git repositories.
>>
>> I have the following changes in my pipeline until now:
>> - Support npm packages with missing execute mode for directories
>> - Reduce command calls in npm run
>> - Add support for local tarball and local link sources
>> - Rework crunch license to recognize more variants
>> - Move known license checksums to CSV file
>> - Pass README as fallback to guess license if a license is missing
>> - Add support to pass build arguments to npm install
>> - Add support to run build scripts with native packages
>> - Parallelize the populate of the npm cache
>>
>> The build (install) of npm packages leads to problems because of old
>> versions in the dependency tree. For example, older versions of packages
>> are incompatible with the Node.js version and older versions of node-gyp
>> require Python 2.
> 
> Hopefully over time the python2 issue should reduce and fade at least. The
> node.js version is trickier.

The current implementation works like an project with embedded external 
dependencies. I both cases I have to patch the embedded dependencies or 
replace them with external DEPENDS.

>> I see three solutions to integrate npm packages:
>> - Build npm packages from the npm project outside of OE and only fetch
>> the data.
>> - Fork the npm project repository and fix everything inside the fork.
>> Use the fork direct or create patches from the fork to build inside OE.
>> - Create recipes per project and incompatible version to fix everything
>> in OE and reduce redundancy.
>>
>> The last solution keeps everything in OE and benefit from the existing
>> OE functions like download proxy as well as license, CVE and version
>> check. But it increases the initial costs and makes only sense if there
>> is an interest in an meta-nodejs layer.
> 
> The latter solution sounds like the only really viable option from an OE
> perspective since we do need the proxy/CVE/license handling, it is a huge part
> of our value and without it, it isn't really OE.

The problem are the many different packages. An Angular project could 
have more than 1000 different project. Thereby some dependencies only 
differ in the version or needed only for development like a development 
server or a unit test framework.

> Creating a meta-nodejs layer isn't an issue if there are people willing to look
> after it.

Is a layer with more more than 1000 recipes thinkable?

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3766 bytes --]

I think the supposed workflow with the new npm.class was to use 'devtool
create' which would run some npm magic to to create a single recipe that
has all of the npm-fetched dependencies inside SRC_URI. Have you tried that?

A layer with thousands of recipes seems totally intractable.

Alex

On Tue, 5 Oct 2021 at 14:48, Stefan Herbrechtsmeier <
stefan.herbrechtsmeier-oss@weidmueller.com> wrote:

> Hi Richard,
>
> Am 05.10.2021 um 13:09 schrieb Richard Purdie:
> > Thanks for bring this up, I've been wondering on the status of our npm
> support.
>
> Do you know any npm users?
>
> > On Tue, 2021-10-05 at 12:04 +0200, Stefan Herbrechtsmeier wrote:
> >> I must improve the npm support for our use cases but need some
> >> fundamental decisions to proceed. I need to build express and angular
> >> applications from git repositories.
> >>
> >> I have the following changes in my pipeline until now:
> >> - Support npm packages with missing execute mode for directories
> >> - Reduce command calls in npm run
> >> - Add support for local tarball and local link sources
> >> - Rework crunch license to recognize more variants
> >> - Move known license checksums to CSV file
> >> - Pass README as fallback to guess license if a license is missing
> >> - Add support to pass build arguments to npm install
> >> - Add support to run build scripts with native packages
> >> - Parallelize the populate of the npm cache
> >>
> >> The build (install) of npm packages leads to problems because of old
> >> versions in the dependency tree. For example, older versions of packages
> >> are incompatible with the Node.js version and older versions of node-gyp
> >> require Python 2.
> >
> > Hopefully over time the python2 issue should reduce and fade at least.
> The
> > node.js version is trickier.
>
> The current implementation works like an project with embedded external
> dependencies. I both cases I have to patch the embedded dependencies or
> replace them with external DEPENDS.
>
> >> I see three solutions to integrate npm packages:
> >> - Build npm packages from the npm project outside of OE and only fetch
> >> the data.
> >> - Fork the npm project repository and fix everything inside the fork.
> >> Use the fork direct or create patches from the fork to build inside OE.
> >> - Create recipes per project and incompatible version to fix everything
> >> in OE and reduce redundancy.
> >>
> >> The last solution keeps everything in OE and benefit from the existing
> >> OE functions like download proxy as well as license, CVE and version
> >> check. But it increases the initial costs and makes only sense if there
> >> is an interest in an meta-nodejs layer.
> >
> > The latter solution sounds like the only really viable option from an OE
> > perspective since we do need the proxy/CVE/license handling, it is a
> huge part
> > of our value and without it, it isn't really OE.
>
> The problem are the many different packages. An Angular project could
> have more than 1000 different project. Thereby some dependencies only
> differ in the version or needed only for development like a development
> server or a unit test framework.
>
> > Creating a meta-nodejs layer isn't an issue if there are people willing
> to look
> > after it.
>
> Is a layer with more more than 1000 recipes thinkable?
>
> Regards
>    Stefan
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156644):
> https://lists.openembedded.org/g/openembedded-core/message/156644
> Mute This Topic: https://lists.openembedded.org/mt/86089523/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5058 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4086 bytes --]

'devtool add' of course. Test case is in
meta/lib/oeqa/selftest/cases/devtool.py

Alex

On Tue, 5 Oct 2021 at 15:10, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> I think the supposed workflow with the new npm.class was to use 'devtool
> create' which would run some npm magic to to create a single recipe that
> has all of the npm-fetched dependencies inside SRC_URI. Have you tried that?
>
> A layer with thousands of recipes seems totally intractable.
>
> Alex
>
> On Tue, 5 Oct 2021 at 14:48, Stefan Herbrechtsmeier <
> stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
>> Hi Richard,
>>
>> Am 05.10.2021 um 13:09 schrieb Richard Purdie:
>> > Thanks for bring this up, I've been wondering on the status of our npm
>> support.
>>
>> Do you know any npm users?
>>
>> > On Tue, 2021-10-05 at 12:04 +0200, Stefan Herbrechtsmeier wrote:
>> >> I must improve the npm support for our use cases but need some
>> >> fundamental decisions to proceed. I need to build express and angular
>> >> applications from git repositories.
>> >>
>> >> I have the following changes in my pipeline until now:
>> >> - Support npm packages with missing execute mode for directories
>> >> - Reduce command calls in npm run
>> >> - Add support for local tarball and local link sources
>> >> - Rework crunch license to recognize more variants
>> >> - Move known license checksums to CSV file
>> >> - Pass README as fallback to guess license if a license is missing
>> >> - Add support to pass build arguments to npm install
>> >> - Add support to run build scripts with native packages
>> >> - Parallelize the populate of the npm cache
>> >>
>> >> The build (install) of npm packages leads to problems because of old
>> >> versions in the dependency tree. For example, older versions of
>> packages
>> >> are incompatible with the Node.js version and older versions of
>> node-gyp
>> >> require Python 2.
>> >
>> > Hopefully over time the python2 issue should reduce and fade at least.
>> The
>> > node.js version is trickier.
>>
>> The current implementation works like an project with embedded external
>> dependencies. I both cases I have to patch the embedded dependencies or
>> replace them with external DEPENDS.
>>
>> >> I see three solutions to integrate npm packages:
>> >> - Build npm packages from the npm project outside of OE and only fetch
>> >> the data.
>> >> - Fork the npm project repository and fix everything inside the fork.
>> >> Use the fork direct or create patches from the fork to build inside OE.
>> >> - Create recipes per project and incompatible version to fix everything
>> >> in OE and reduce redundancy.
>> >>
>> >> The last solution keeps everything in OE and benefit from the existing
>> >> OE functions like download proxy as well as license, CVE and version
>> >> check. But it increases the initial costs and makes only sense if there
>> >> is an interest in an meta-nodejs layer.
>> >
>> > The latter solution sounds like the only really viable option from an OE
>> > perspective since we do need the proxy/CVE/license handling, it is a
>> huge part
>> > of our value and without it, it isn't really OE.
>>
>> The problem are the many different packages. An Angular project could
>> have more than 1000 different project. Thereby some dependencies only
>> differ in the version or needed only for development like a development
>> server or a unit test framework.
>>
>> > Creating a meta-nodejs layer isn't an issue if there are people willing
>> to look
>> > after it.
>>
>> Is a layer with more more than 1000 recipes thinkable?
>>
>> Regards
>>    Stefan
>>
>>
>>
>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156646):
> https://lists.openembedded.org/g/openembedded-core/message/156646
> Mute This Topic: https://lists.openembedded.org/mt/86089523/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5693 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Alex,

Am 05.10.2021 um 15:10 schrieb Alexander Kanavin:
> I think the supposed workflow with the new npm.class was to use 'devtool 
> create' which would run some npm magic to to create a single recipe that 
> has all of the npm-fetched dependencies inside SRC_URI.

There is no magic inside npm. It parse json recipes, calculate a 
dependency tree, download archives, checks the archives, unpack the 
archives and run some commands from the json recipes.

The 'devtool add' create a recipe with over 1000 dependencies. A lot of 
the dependencies have an unknown license. I have improve createtool to 
detect more licenses and fallback to the readme if no license file exist.

The npm class doesn't support run of build scripts. The build of an 
application (ex. angular) need native binaries inside its npm dependencies.

The existing solution have the same problems like embedded dependencies 
in other recipes (ex. multiple version of the same project, unneeded 
dependencies, unforeseeable downloads, pre-builds, on the fly binary 
builds, ...)

You have to fix a bug in different versions of the same project at 
different positions in the dependency tree inside multiple recipes.

The main different between a normal (ex C++) and npm project is that npm 
use many small packages as dependency.

> Have you tried that?

Sure. Otherwise I haven't spot the limitations and work on improvements.

> A layer with thousands of recipes seems totally intractable.

What is your concern? The high number of dependencies or to handle it 
via OE?

We have the high number of dependencies in any case and need to handle 
the proxy/CVE/license problem. Could we reuse OE or should we use 
different software for the same problems?

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Robert Berger]

Hi Stefan,

On 05/10/2021 15:48, Stefan Herbrechtsmeier wrote:
> 
> Is a layer with more more than 1000 recipes thinkable?

Did you have a look at this[1]?

It says: "For instance I currently am using around 10 NPM-packages, 
those culminated into (currently) 937 recipes that are required to build 
these."

[1] 
https://bitbakesoda.blogspot.com/2020/05/a-different-approach-to-npm-with-yocto.html

> 
> Regards
>    Stefan

Regards,

Robert

Re: [OE-core] Improve npm support to run build scripts

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier <
stefan.herbrechtsmeier-oss@weidmueller.com> wrote:

>
> > A layer with thousands of recipes seems totally intractable.
>
> What is your concern? The high number of dependencies or to handle it
> via OE?
>

Generating recipes with a tool means the tool will be custom and
non-standard, and chances of anyone outside of your company
understanding, using and contributing to it are rather small. There's also
the problem of needing multiple versions of the same thing for different
consumer items, which oe doesn't easily support. The link Robert provided
already exposes some of the pain points with that approach.

I tend to think the best way forward (or least painful at least) is to
gradually improve what there already is, which is the npm class
and devtool, and send patches to various upstream njs projects when they're
using outdated dependencies or otherwise need changing.

Alex

[-- Attachment #2: Type: text/html, Size: 1388 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Konrad Weihmann]

On 05.10.21 21:17, Alexander Kanavin wrote:
> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier 
> <stefan.herbrechtsmeier-oss@weidmueller.com 
> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> 
> 
>      > A layer with thousands of recipes seems totally intractable.
> 
>     What is your concern? The high number of dependencies or to handle it
>     via OE?
> 
> 
> Generating recipes with a tool means the tool will be custom and 
> non-standard, and chances of anyone outside of your company
> understanding, using and contributing to it are rather small. There's 
> also the problem of needing multiple versions of the same thing for 
> different
> consumer items, which oe doesn't easily support. The link Robert 
> provided already exposes some of the pain points with that approach.
> 
> I tend to think the best way forward (or least painful at least) is to 
> gradually improve what there already is, which is the npm class
> and devtool, and send patches to various upstream njs projects when 
> they're using outdated dependencies or otherwise need changing.
> 
> Alex

Just to chime in :-), I like to question this approach of having 
multiple versions of the same in an image.
As already outlined npm is horrible in many ways, but using the lockfile 
approach multiplies that even more.
But I tend to agree that using the currently available oe-core code 
would be suited best for a broader audience - in your own layer you 
simply can do whatever you like.

While personally I think in the long run, every npm dependency has to be 
provided as a recipe of its own (even I know the costs of that pretty 
well)... esp when CVE checking and basic packaging hygiene should be 
enforced.

Nevertheless for oe-core I wouldn't want to change a lot right now, as 
there is simply a valid set of consumers missing that could enable us to 
do some proper testing. But yeah fixes/improvements for devtool are very 
welcome (also the available docu might need a touch)

> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156656): https://lists.openembedded.org/g/openembedded-core/message/156656
> Mute This Topic: https://lists.openembedded.org/mt/86089523/3647476
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] Improve npm support to run build scripts

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On Tue, Oct 5, 2021 at 12:29 PM Konrad Weihmann <kweihmann@outlook.com>
wrote:

>
> While personally I think in the long run, every npm dependency has to be
> provided as a recipe of its own (even I know the costs of that pretty
> well)... esp when CVE checking and basic packaging hygiene should be
> enforced.
>

Emphatically agree. The "stuff it all into one recipe" npm approach is very
broken.

..Ch:W..


-- 
*"Perfection must be reached by degrees; she requires the slow hand of
time." - Voltaire*

[-- Attachment #2: Type: text/html, Size: 1050 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Robert,

Am 05.10.2021 um 20:18 schrieb Robert Berger:
> On 05/10/2021 15:48, Stefan Herbrechtsmeier wrote:
>>
>> Is a layer with more more than 1000 recipes thinkable?
> 
> Did you have a look at this[1]?
> 
> It says: "For instance I currently am using around 10 NPM-packages, 
> those culminated into (currently) 937 recipes that are required to build 
> these."
> 
> [1] 
> https://bitbakesoda.blogspot.com/2020/05/a-different-approach-to-npm-with-yocto.html 

thanks for the link. It matches with my idea.

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Alex,

Am 05.10.2021 um 21:17 schrieb Alexander Kanavin:
> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier 
> <stefan.herbrechtsmeier-oss@weidmueller.com 
> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> 
> 
>      > A layer with thousands of recipes seems totally intractable.
> 
>     What is your concern? The high number of dependencies or to handle it
>     via OE?
> 
> 
> Generating recipes with a tool means the tool will be custom and 
> non-standard, and chances of anyone outside of your company
> understanding, using and contributing to it are rather small.

Why can't we extend devtool add / createtool to generate multiple 
recipes? As bonus we can check if a recipe already exist?

> There's 
> also the problem of needing multiple versions of the same thing for 
> different
> consumer items, which oe doesn't easily support.

The common approach in OE is to add the (partial) version to the recipe 
name. On the other side the major version of an npm package doesn't 
indicate the real API breakage. The major version is also increased if a 
support for an old Node.js version is dropped.

Furthermore I would extend the approach from the link with a symbolic 
link inside the node_modules of the package itself. This allows a recipe 
to use a specific version. It is also possible to track a map between 
npm (package.json) version and OE name so that the generator could 
select the correct OE DEPENDS. But I haven't the experience how often 
you really need an older version. In my test most of the maintainer 
doesn't update the dependencies or the package is orphan.

> The link Robert 
> provided already exposes some of the pain points with that approach.

It also exposes the pain points with the current approach.

> I tend to think the best way forward (or least painful at least) is to 
> gradually improve what there already is, which is the npm class
> and devtool, and send patches to various upstream njs projects when 
> they're using outdated dependencies or otherwise need changing.

How many patches have you send to upstream npm packages and how many 
were successful?

My feeling is that the chance of success isn't high especially deeper in 
the dependency tree.

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Konrad,

Am 05.10.2021 um 21:29 schrieb Konrad Weihmann:
> 
> 
> On 05.10.21 21:17, Alexander Kanavin wrote:
>> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier 
>> <stefan.herbrechtsmeier-oss@weidmueller.com 
>> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
>>
>>
>>      > A layer with thousands of recipes seems totally intractable.
>>
>>     What is your concern? The high number of dependencies or to handle it
>>     via OE?
>>
>>
>> Generating recipes with a tool means the tool will be custom and 
>> non-standard, and chances of anyone outside of your company
>> understanding, using and contributing to it are rather small. There's 
>> also the problem of needing multiple versions of the same thing for 
>> different
>> consumer items, which oe doesn't easily support. The link Robert 
>> provided already exposes some of the pain points with that approach.
>>
>> I tend to think the best way forward (or least painful at least) is to 
>> gradually improve what there already is, which is the npm class
>> and devtool, and send patches to various upstream njs projects when 
>> they're using outdated dependencies or otherwise need changing.
>>
>> Alex
> 
> Just to chime in :-), I like to question this approach of having 
> multiple versions of the same in an image.
> As already outlined npm is horrible in many ways, but using the lockfile 
> approach multiplies that even more.
> But I tend to agree that using the currently available oe-core code 
> would be suited best for a broader audience

But this audience loose the some basic features of OE (ex. version and 
CVE check) and have to manual fix or ignore the licenses.

> - in your own layer you 
> simply can do whatever you like.

But this make it impossible to share recipes. Why don't we create a 
meta-nodejs with a different approach to avoid doing the same thing 
multiple times.

> While personally I think in the long run, every npm dependency has to be 
> provided as a recipe of its own (even I know the costs of that pretty 
> well)... esp when CVE checking and basic packaging hygiene should be 
> enforced.

Why should OE provide a solution with doesn't support this? I think this 
is a main feature of OE.

> Nevertheless for oe-core I wouldn't want to change a lot right now, as 
> there is simply a valid set of consumers missing that could enable us to 
> do some proper testing. But yeah fixes/improvements for devtool are very 
> welcome (also the available docu might need a touch)

My problem is that the current approach doesn't support the build of an 
express or angular application. Does it makes sense to add this feature 
if we know that some basic OE features are missing (CVE and version 
check) and fixing bugs is a nightmare?

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Konrad Weihmann]

On 06.10.21 11:18, Stefan Herbrechtsmeier wrote:
> Hi Konrad,
> 
> Am 05.10.2021 um 21:29 schrieb Konrad Weihmann:
>>
>>
>> On 05.10.21 21:17, Alexander Kanavin wrote:
>>> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier 
>>> <stefan.herbrechtsmeier-oss@weidmueller.com 
>>> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
>>>
>>>
>>>      > A layer with thousands of recipes seems totally intractable.
>>>
>>>     What is your concern? The high number of dependencies or to 
>>> handle it
>>>     via OE?
>>>
>>>
>>> Generating recipes with a tool means the tool will be custom and 
>>> non-standard, and chances of anyone outside of your company
>>> understanding, using and contributing to it are rather small. There's 
>>> also the problem of needing multiple versions of the same thing for 
>>> different
>>> consumer items, which oe doesn't easily support. The link Robert 
>>> provided already exposes some of the pain points with that approach.
>>>
>>> I tend to think the best way forward (or least painful at least) is 
>>> to gradually improve what there already is, which is the npm class
>>> and devtool, and send patches to various upstream njs projects when 
>>> they're using outdated dependencies or otherwise need changing.
>>>
>>> Alex
>>
>> Just to chime in :-), I like to question this approach of having 
>> multiple versions of the same in an image.
>> As already outlined npm is horrible in many ways, but using the 
>> lockfile approach multiplies that even more.
>> But I tend to agree that using the currently available oe-core code 
>> would be suited best for a broader audience
> 
> But this audience loose the some basic features of OE (ex. version and 
> CVE check) and have to manual fix or ignore the licenses.
> 
>> - in your own layer you simply can do whatever you like.
> 
> But this make it impossible to share recipes. Why don't we create a 
> meta-nodejs with a different approach to avoid doing the same thing 
> multiple times.

We could, but we have to think about the costs and ways to properly 
maintain it - and that seems to be the main reason.
You could come up with a repo of your own and see who is interested in 
joining the development (as I for instance did with a repo for ruby 
gems) - still this would be hosted and maintained outside of core I 
think, due to the limited time and resources available.

And **most important** the repo has to provide its own means of testing/CI.

BTW I remember I had a talk with someone last year about it and putting 
a bunch of angular based apps into an image, with all npms as separate 
recipes resulted in ~100k of files.

so CI and testing has to be extra beefy to make it work - which maybe 
requires a good portion of automation, which then brings me back to 
Alex's suggestion to provide some patches to devtool, if there is a need 
for that.

> 
>> While personally I think in the long run, every npm dependency has to 
>> be provided as a recipe of its own (even I know the costs of that 
>> pretty well)... esp when CVE checking and basic packaging hygiene 
>> should be enforced.
> 
> Why should OE provide a solution with doesn't support this? I think this 
> is a main feature of OE.

It actually does, but having this lockfile based approach, basically 
crams up all into a single recipe, which makes it a bit harder to check 
and whitelist any false positives or n.a. items, as you might have to do 
it for more than one recipe - but yeah, the basic CVE checking is part 
of core and works as expected

> 
>> Nevertheless for oe-core I wouldn't want to change a lot right now, as 
>> there is simply a valid set of consumers missing that could enable us 
>> to do some proper testing. But yeah fixes/improvements for devtool are 
>> very welcome (also the available docu might need a touch)
> 
> My problem is that the current approach doesn't support the build of an 
> express or angular application. Does it makes sense to add this feature 
> if we know that some basic OE features are missing (CVE and version 
> check) and fixing bugs is a nightmare?

Just my personal opinion would be, to open up a layer on your own and 
let it grow and mature first before getting it in into core - but 
patches to devtool are very welcome
> 
> Regards
>    Stefan

Re: [OE-core] Improve npm support to run build scripts

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5130 bytes --]

The OE landscape is littered with unmaintained or barely maintained layers.
By the looks of things, no one wants to maintain this proposed meta-nodejs
layer either, which involves thousands of recipes, (so far) non-existent
tooling to create and update them in scalable ways, and rigorous testing.
Or such a layer would have already happened.

It's for this reason that I think the shrinkwrapped, self-contained recipe
approach is more viable (even if it comes with its own pain points) and
also matches how node.js community operates  - we can fix the CVE stuff as
well to look into dependencies if that's a concern.

Alex

On Wed, 6 Oct 2021 at 11:43, Konrad Weihmann <kweihmann@outlook.com> wrote:

>
>
> On 06.10.21 11:18, Stefan Herbrechtsmeier wrote:
> > Hi Konrad,
> >
> > Am 05.10.2021 um 21:29 schrieb Konrad Weihmann:
> >>
> >>
> >> On 05.10.21 21:17, Alexander Kanavin wrote:
> >>> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier
> >>> <stefan.herbrechtsmeier-oss@weidmueller.com
> >>> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> >>>
> >>>
> >>>      > A layer with thousands of recipes seems totally intractable.
> >>>
> >>>     What is your concern? The high number of dependencies or to
> >>> handle it
> >>>     via OE?
> >>>
> >>>
> >>> Generating recipes with a tool means the tool will be custom and
> >>> non-standard, and chances of anyone outside of your company
> >>> understanding, using and contributing to it are rather small. There's
> >>> also the problem of needing multiple versions of the same thing for
> >>> different
> >>> consumer items, which oe doesn't easily support. The link Robert
> >>> provided already exposes some of the pain points with that approach.
> >>>
> >>> I tend to think the best way forward (or least painful at least) is
> >>> to gradually improve what there already is, which is the npm class
> >>> and devtool, and send patches to various upstream njs projects when
> >>> they're using outdated dependencies or otherwise need changing.
> >>>
> >>> Alex
> >>
> >> Just to chime in :-), I like to question this approach of having
> >> multiple versions of the same in an image.
> >> As already outlined npm is horrible in many ways, but using the
> >> lockfile approach multiplies that even more.
> >> But I tend to agree that using the currently available oe-core code
> >> would be suited best for a broader audience
> >
> > But this audience loose the some basic features of OE (ex. version and
> > CVE check) and have to manual fix or ignore the licenses.
> >
> >> - in your own layer you simply can do whatever you like.
> >
> > But this make it impossible to share recipes. Why don't we create a
> > meta-nodejs with a different approach to avoid doing the same thing
> > multiple times.
>
> We could, but we have to think about the costs and ways to properly
> maintain it - and that seems to be the main reason.
> You could come up with a repo of your own and see who is interested in
> joining the development (as I for instance did with a repo for ruby
> gems) - still this would be hosted and maintained outside of core I
> think, due to the limited time and resources available.
>
> And **most important** the repo has to provide its own means of testing/CI.
>
> BTW I remember I had a talk with someone last year about it and putting
> a bunch of angular based apps into an image, with all npms as separate
> recipes resulted in ~100k of files.
>
> so CI and testing has to be extra beefy to make it work - which maybe
> requires a good portion of automation, which then brings me back to
> Alex's suggestion to provide some patches to devtool, if there is a need
> for that.
>
> >
> >> While personally I think in the long run, every npm dependency has to
> >> be provided as a recipe of its own (even I know the costs of that
> >> pretty well)... esp when CVE checking and basic packaging hygiene
> >> should be enforced.
> >
> > Why should OE provide a solution with doesn't support this? I think this
> > is a main feature of OE.
>
> It actually does, but having this lockfile based approach, basically
> crams up all into a single recipe, which makes it a bit harder to check
> and whitelist any false positives or n.a. items, as you might have to do
> it for more than one recipe - but yeah, the basic CVE checking is part
> of core and works as expected
>
> >
> >> Nevertheless for oe-core I wouldn't want to change a lot right now, as
> >> there is simply a valid set of consumers missing that could enable us
> >> to do some proper testing. But yeah fixes/improvements for devtool are
> >> very welcome (also the available docu might need a touch)
> >
> > My problem is that the current approach doesn't support the build of an
> > express or angular application. Does it makes sense to add this feature
> > if we know that some basic OE features are missing (CVE and version
> > check) and fixing bugs is a nightmare?
>
> Just my personal opinion would be, to open up a layer on your own and
> let it grow and mature first before getting it in into core - but
> patches to devtool are very welcome
> >
> > Regards
> >    Stefan
>

[-- Attachment #2: Type: text/html, Size: 6506 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 6025 bytes --]

On Wed, Oct 6, 2021 at 2:48 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> The OE landscape is littered with unmaintained or barely maintained
> layers. By the looks of things, no one wants to maintain this proposed
> meta-nodejs layer either, which involves thousands of recipes, (so far)
> non-existent tooling to create and update them in scalable ways, and
> rigorous testing. Or such a layer would have already happened.
>
> It's for this reason that I think the shrinkwrapped, self-contained recipe
> approach is more viable (even if it comes with its own pain points) and
> also matches how node.js community operates  - we can fix the CVE stuff as
> well to look into dependencies if that's a concern.
>
> Alex
>
> On Wed, 6 Oct 2021 at 11:43, Konrad Weihmann <kweihmann@outlook.com>
> wrote:
>
>>
>>
>> On 06.10.21 11:18, Stefan Herbrechtsmeier wrote:
>> > Hi Konrad,
>> >
>> > Am 05.10.2021 um 21:29 schrieb Konrad Weihmann:
>> >>
>> >>
>> >> On 05.10.21 21:17, Alexander Kanavin wrote:
>> >>> On Tue, 5 Oct 2021 at 19:44, Stefan Herbrechtsmeier
>> >>> <stefan.herbrechtsmeier-oss@weidmueller.com
>> >>> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
>> >>>
>> >>>
>> >>>      > A layer with thousands of recipes seems totally intractable.
>> >>>
>> >>>     What is your concern? The high number of dependencies or to
>> >>> handle it
>> >>>     via OE?
>> >>>
>> >>>
>> >>> Generating recipes with a tool means the tool will be custom and
>> >>> non-standard, and chances of anyone outside of your company
>> >>> understanding, using and contributing to it are rather small. There's
>> >>> also the problem of needing multiple versions of the same thing for
>> >>> different
>> >>> consumer items, which oe doesn't easily support. The link Robert
>> >>> provided already exposes some of the pain points with that approach.
>> >>>
>> >>> I tend to think the best way forward (or least painful at least) is
>> >>> to gradually improve what there already is, which is the npm class
>> >>> and devtool, and send patches to various upstream njs projects when
>> >>> they're using outdated dependencies or otherwise need changing.
>> >>>
>> >>> Alex
>> >>
>> >> Just to chime in :-), I like to question this approach of having
>> >> multiple versions of the same in an image.
>> >> As already outlined npm is horrible in many ways, but using the
>> >> lockfile approach multiplies that even more.
>> >> But I tend to agree that using the currently available oe-core code
>> >> would be suited best for a broader audience
>> >
>> > But this audience loose the some basic features of OE (ex. version and
>> > CVE check) and have to manual fix or ignore the licenses.
>> >
>> >> - in your own layer you simply can do whatever you like.
>> >
>> > But this make it impossible to share recipes. Why don't we create a
>> > meta-nodejs with a different approach to avoid doing the same thing
>> > multiple times.
>>
>> We could, but we have to think about the costs and ways to properly
>> maintain it - and that seems to be the main reason.
>> You could come up with a repo of your own and see who is interested in
>> joining the development (as I for instance did with a repo for ruby
>> gems) - still this would be hosted and maintained outside of core I
>> think, due to the limited time and resources available.
>>
>> And **most important** the repo has to provide its own means of
>> testing/CI.
>>
>> BTW I remember I had a talk with someone last year about it and putting
>> a bunch of angular based apps into an image, with all npms as separate
>> recipes resulted in ~100k of files.
>>
>
We have discussed the severe impact on parsing time with 100k recipes. I
personally worked on a node.js project with that many dependencies (but not
an OE/Yocto deployment). Parsing time is already a pain point for 1000s of
recipes.

>
>> so CI and testing has to be extra beefy to make it work - which maybe
>> requires a good portion of automation, which then brings me back to
>> Alex's suggestion to provide some patches to devtool, if there is a need
>> for that.
>>
>> >
>> >> While personally I think in the long run, every npm dependency has to
>> >> be provided as a recipe of its own (even I know the costs of that
>> >> pretty well)... esp when CVE checking and basic packaging hygiene
>> >> should be enforced.
>> >
>> > Why should OE provide a solution with doesn't support this? I think
>> this
>> > is a main feature of OE.
>>
>> It actually does, but having this lockfile based approach, basically
>> crams up all into a single recipe, which makes it a bit harder to check
>> and whitelist any false positives or n.a. items, as you might have to do
>> it for more than one recipe - but yeah, the basic CVE checking is part
>> of core and works as expected
>>
>> >
>> >> Nevertheless for oe-core I wouldn't want to change a lot right now, as
>> >> there is simply a valid set of consumers missing that could enable us
>> >> to do some proper testing. But yeah fixes/improvements for devtool are
>> >> very welcome (also the available docu might need a touch)
>> >
>> > My problem is that the current approach doesn't support the build of an
>> > express or angular application. Does it makes sense to add this feature
>> > if we know that some basic OE features are missing (CVE and version
>> > check) and fixing bugs is a nightmare?
>>
>> Just my personal opinion would be, to open up a layer on your own and
>> let it grow and mature first before getting it in into core - but
>> patches to devtool are very welcome
>> >
>> > Regards
>> >    Stefan
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156681):
> https://lists.openembedded.org/g/openembedded-core/message/156681
> Mute This Topic: https://lists.openembedded.org/mt/86089523/924729
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 8556 bytes --]

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi Tim,

Am 08.10.2021 um 07:27 schrieb Tim Orling:
> We have discussed the severe impact on parsing time with 100k recipes. I 
> personally worked on a node.js project with that many dependencies (but 
> not an OE/Yocto deployment). Parsing time is already a pain point for 
> 1000s of recipes.

How do you track the version updates, check the CVE status and track the 
licenses?

How do you update a package inside the dependency tree?

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Stefan Herbrechtsmeier]

Hi,

Am 06.10.2021 um 11:47 schrieb Alexander Kanavin:
> The OE landscape is littered with unmaintained or barely maintained 
> layers. By the looks of things, no one wants to maintain this proposed 
> meta-nodejs layer either, which involves thousands of recipes, (so far) 
> non-existent tooling to create and update them in scalable ways, and 
> rigorous testing. Or such a layer would have already happened.
> 
> It's for this reason that I think the shrinkwrapped, self-contained 
> recipe approach is more viable (even if it comes with its own pain 
> points) and also matches how node.js community operates - we can fix the 
> CVE stuff as well to look into dependencies if that's a concern.

How should we run build scripts which need native npm packages and how 
should we handle licenses of build and runtime packages?

A simplified node.js flow looks like the following

cd ${S}
rm -rf node_modules
npm install --arch=$NPM_BUILD_ARCH --dev
npm run-script --arch=$NPM_BUILD_ARCH build
rm -rf node_modules
npm install --arch=$NPM_TARGET_ARCH --global <OR> cp -r dist/* ${D}

Should we split it into two recipes or should we handle everything in a 
single recipe?

Is it accepted to use the output of a native recipe as source for an 
target recipe?

Regards
   Stefan

Re: [OE-core] Improve npm support to run build scripts

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

On Tue, 19 Oct 2021 at 14:52, Stefan Herbrechtsmeier <
stefan.herbrechtsmeier-oss@weidmueller.com> wrote:

> How should we run build scripts which need native npm packages and how
> should we handle licenses of build and runtime packages?
>
> A simplified node.js flow looks like the following
>
> cd ${S}
> rm -rf node_modules
> npm install --arch=$NPM_BUILD_ARCH --dev
> npm run-script --arch=$NPM_BUILD_ARCH build
> rm -rf node_modules
> npm install --arch=$NPM_TARGET_ARCH --global <OR> cp -r dist/* ${D}
>
> Should we split it into two recipes or should we handle everything in a
> single recipe?
>
> Is it accepted to use the output of a native recipe as source for an
> target recipe?
>

If that output is placed in a reasonable place in sysroot-destdir/ and
pulled in via DEPENDS, then yes, that's fine. But it's also okay to do the
build in two steps like above from a single recipe - there are more than a
few recipes where native helper binaries are built to aid with the target
build, typically the build system itself is able to use BUILD_CC for it, or
a custom task is written where CC is set to BUILD_CC explicitly.

I think you should experiment with both options.

Alex

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]