npm offline build for yocto

npm offline build for yocto

[Jan Kiszka]

Hi Michael,

I found your progress report (and unanswered question) on that topic in 
http://lists.openembedded.org/pipermail/openembedded-core/2018-April/268639.html. 
Did this effort go further?

I just ran into it again while updating https://github.com/siemens/meta-iot2000 
to Node.js >v6. I also played with prefilling the cache, but likely in the wrong 
way.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [OE-core] npm offline build for yocto

[Josef Holzmayr]

Hi Jan,

On Thu, Mar 07, 2019 at 10:37:26AM +0100, Jan Kiszka wrote:
> Hi Michael,
> 
> I found your progress report (and unanswered question) on that topic in http://lists.openembedded.org/pipermail/openembedded-core/2018-April/268639.html.
> Did this effort go further?
> 
> I just ran into it again while updating
> https://github.com/siemens/meta-iot2000 to Node.js >v6. I also played with
> prefilling the cache, but likely in the wrong way.

We've had our share of fun ourselves by now, especially with the
event-stream module being hijacked and removed. To my knowledge none of
the existing nodejs package managers (npm, yarn, ...) do support pure
fetching neither explicit caching. The only thing that seems to be
mostly working fine is using an explicit caching proxy like lazy-npm to
externally cache the remote repositories.

Greetz
-- 
———————————————
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548 

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

Re: [OE-core] npm offline build for yocto

[Jan Kiszka]

On 07.03.19 11:25, Josef Holzmayr wrote:
> Hi Jan,
> 
> On Thu, Mar 07, 2019 at 10:37:26AM +0100, Jan Kiszka wrote:
>> Hi Michael,
>>
>> I found your progress report (and unanswered question) on that topic in http://lists.openembedded.org/pipermail/openembedded-core/2018-April/268639.html.
>> Did this effort go further?
>>
>> I just ran into it again while updating
>> https://github.com/siemens/meta-iot2000 to Node.js >v6. I also played with
>> prefilling the cache, but likely in the wrong way.
> 
> We've had our share of fun ourselves by now, especially with the
> event-stream module being hijacked and removed. To my knowledge none of

A mess...

> the existing nodejs package managers (npm, yarn, ...) do support pure
> fetching neither explicit caching. The only thing that seems to be
> mostly working fine is using an explicit caching proxy like lazy-npm to
> externally cache the remote repositories.

Fetching is already implemented by bitbake. Now the question is if we can use 
native npm to install offline.

My understanding was that "npm cache add <tarball>" could be a path, though 
there were at least some issues in the past: 
https://github.com/nodejs/help/issues/944. What only I tried so far was "npm 
cache add <folder>" but that didn't work with npm 6.4.1. Maybe I should also 
check the tarball path. But that would mean unpacking the dependencies into 
${S}/node_modules becomes pointless (well, it already is).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: npm offline build for yocto

[Davis, Michael]

Unfortunately no.  I could never get the cache to prefill correctly with node 8 and haven't tried any of the newer versions.  The way they did the sha1/256/512 sums made it impossible to prefill the cache, and the import of tars just didn’t work.

I ended up using yarn and building the app outside of bitbake, though with yarn the offline problem would become much easier.   It has much better support for offline builds than node / npm does.  If I had to start again I would start with adding support for yarn and go from there.  Node appears to be a dead end unless / until they fix the offline build.


Simple guide I followed to get offline yarn working.  It shouldn't be hard to translate into a system much like the npm part of bitbake.
https://yarnpkg.com/blog/2016/11/24/offline-mirror/

Mike
-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Thursday, March 07, 2019 3:37 AM
To: Davis, Michael <michael.davis@essvote.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: npm offline build for yocto

Hi Michael,

I found your progress report (and unanswered question) on that topic in http://lists.openembedded.org/pipermail/openembedded-core/2018-April/268639.html. 
Did this effort go further?

I just ran into it again while updating https://github.com/siemens/meta-iot2000
to Node.js >v6. I also played with prefilling the cache, but likely in the wrong way.

Thanks,
Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux

Re: npm offline build for yocto

[Jan Kiszka]

On 07.03.19 15:22, Davis, Michael wrote:
> Unfortunately no.  I could never get the cache to prefill correctly with node 8 and haven't tried any of the newer versions.  The way they did the sha1/256/512 sums made it impossible to prefill the cache, and the import of tars just didn’t work.

I suspect the cache does not match packages coming in via different channels 
(add tarball vs. fetch url).

> 
> I ended up using yarn and building the app outside of bitbake, though with yarn the offline problem would become much easier.   It has much better support for offline builds than node / npm does.  If I had to start again I would start with adding support for yarn and go from there.  Node appears to be a dead end unless / until they fix the offline build.
> 
> 
> Simple guide I followed to get offline yarn working.  It shouldn't be hard to translate into a system much like the npm part of bitbake.
> https://yarnpkg.com/blog/2016/11/24/offline-mirror/
> 

Sounds reasonable. Let me play with that, maybe try an ad-hoc integration into 
our layer first.

Thanks,
Jan

> Mike
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Thursday, March 07, 2019 3:37 AM
> To: Davis, Michael <michael.davis@essvote.com>
> Cc: OE-core <openembedded-core@lists.openembedded.org>
> Subject: npm offline build for yocto
> 
> Hi Michael,
> 
> I found your progress report (and unanswered question) on that topic in http://lists.openembedded.org/pipermail/openembedded-core/2018-April/268639.html.
> Did this effort go further?
> 
> I just ran into it again while updating https://github.com/siemens/meta-iot2000
> to Node.js >v6. I also played with prefilling the cache, but likely in the wrong way.
> 
> Thanks,
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux