Yocto and NPM issues

Yocto and NPM issues

[Svein Seldal]

Hi

I'm attempting to package a npm-based js application into a Rocko image, 
and I'm having some issues. For testing I use the official poky repo and 
the meta-openembedded repo (for installing nodejs 8.4.0). I'm building 
for qemux86-64.


Issue 1: NPM builds not working in Rocko

If I'm following the examples outlined in 
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM, and run devtool 
add "npm://registry.npmjs.org;name=cute-files;version=1.0.2" and then 
bitbake cute-file. It fails on an npm error seen below. I found that 
this patch fixes the problem: 
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/classes/npm.bbclass?id=d38e1e2c2ea4646b34ea6282d3d7620df5b0374b

Will there be more releases of rocko? Can I request that this patch is 
backported to the rocko branch please?


Issue 2: Workspace npm recipe does not work after bitbake -c cleanall

If using the devtool/workspace for the above recipe, and run bitbake -c 
cleanall cute-files, then try to rerun bitbake cute-files, it will warn 
on missing LICENSE files for the npm packages and fail on 
LIC_FILES_CHKSUM point to an invalid file QA error. E.g. something in 
line of:

WARNING: cute-files-1.0.2-r0 do_populate_lic: Could not copy license 
file 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md 
to 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/license-destdir/cute-files/license.md: 
[Errno 2] No such file or directory: 
'/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md'
ERROR: cute-files-1.0.2-r0 do_populate_lic: QA Issue: cute-files: 
LIC_FILES_CHKSUM points to an invalid file: 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/accepts/LICENSE 
[license-checksum]


Issue 3: npm recipe fails to download npm images after -c cleanall

If the recipe is copied into a normal layer and run without externalsrc, 
and runned after a -c cleanall, the following error is returned:

WARNING: cute-files-1.0.2-r0 do_fetch: Checksum failure encountered with 
download of npm://registry.npmjs.org;name=cute-files;version=1.0.2 - 
will attempt other sources if available
ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure: Checksum mismatch!
File: 'commander-2.15.1.tgz' has sha1 checksum 
df46e867d0fc2aec66a34662b406a9ccafff5b0f when * was expected
ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure for URL: 
'npm://registry.npmjs.org;name=cute-files;version=1.0.2'. Unable to 
fetch URL from any source.
ERROR: cute-files-1.0.2-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in: 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_fetch.20513
ERROR: Task 
(/home/common/yocto/yocto-poky/meta-foobar/recipes/cute-files/cute-files_1.0.2.bb:do_fetch) 
failed with exit code '1'


The only way to circumvent issue 2 and 3 after a -c cleanall is to use 
the devtool to download the packages, and then build the package as 
normal.


Best regards,
Svein




FAILURE Building cute-files:

ERROR: cute-files-1.0.2-r0 do_compile: Function failed: do_compile (log 
file is located at 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)
ERROR: Logfile of failure stored in: 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536
Log data follows:
| DEBUG: Executing python function externalsrc_compile_prefunc
| NOTE: cute-files: compiling from external source tree 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files
| DEBUG: Python function externalsrc_compile_prefunc finished
| DEBUG: Executing shell function do_compile
| npm ERR! As of npm@5, the npm cache self-heals from corruption issues 
and data extracted from the cache is guaranteed to be valid. If you want 
to make sure everything is consistent, use 'npm cache verify' instead.
| npm ERR!
| npm ERR! If you're sure you want to delete the entire cache, rerun 
this command with --force.
|
| npm ERR! A complete log of this run can be found in:
| npm ERR!     
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/npm_cache/_logs/2018-03-21T12_36_48_034Z-debug.log
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)
ERROR: Task 
(/home/common/yocto/yocto-poky/build/workspace/recipes/cute-files/cute-files_1.0.2.bb:do_compile) 
failed with exit code '1'

Re: Yocto and NPM issues

[Alexander Kanavin]

On 03/21/2018 03:09 PM, Svein Seldal wrote:
> Hi
> 
> I'm attempting to package a npm-based js application into a Rocko image, 
> and I'm having some issues. For testing I use the official poky repo and 
> the meta-openembedded repo (for installing nodejs 8.4.0). I'm building 
> for qemux86-64.

I know this is not an answer to your specific issues, but npm is a bit 
of maintenance nightmare as its workflow (downloading random stuff off 
the internet as an integral part of the build) clashes badly with how 
bitbake, and embedded world in general handle builds. We've discussed 
what to do about it, with no clear outcome, but we would like to have 
something that doesn't require constant fixing and hacks:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html


Alex

Re: Yocto and NPM issues

[Alexander Kanavin]

On 03/21/2018 03:54 PM, Alexander Kanavin wrote:

>> I'm attempting to package a npm-based js application into a Rocko 
>> image, and I'm having some issues. For testing I use the official poky 
>> repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm 
>> building for qemux86-64.
> 
> I know this is not an answer to your specific issues, but npm is a bit 
> of maintenance nightmare as its workflow (downloading random stuff off 
> the internet as an integral part of the build) clashes badly with how 
> bitbake, and embedded world in general handle builds. We've discussed 
> what to do about it, with no clear outcome, but we would like to have 
> something that doesn't require constant fixing and hacks:
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html 

Oh, and you might have more success using meta-nodejs, as it doesn't try 
to implement a custom fetcher, and just runs npm directly during the 
build. If you don't mind that that makes the build outcome essentially 
random from one invocation to the next.

https://github.com/imyller/meta-nodejs


Alex

Re: Yocto and NPM issues

[Svein Seldal]

On 21.03.2018 14:54, Alexander Kanavin wrote:
>> I'm attempting to package a npm-based js application into a Rocko 
>> image, and I'm having some issues. For testing I use the official poky 
>> repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm 
>> building for qemux86-64.
> 
> I know this is not an answer to your specific issues, but npm is a bit 
> of maintenance nightmare as its workflow (downloading random stuff off 
> the internet as an integral part of the build) clashes badly with how 
> bitbake, and embedded world in general handle builds. We've discussed 
> what to do about it, with no clear outcome, but we would like to have 
> something that doesn't require constant fixing and hacks:
> 
> http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html 

I haven't read the post in full yet, but the principle of npm is 
inherently different from the philosophies of Yocto, granted.

One approach that we've been discussing as an alternative to the 
yocto+npm approach is to install the js app on a target machine and 
snapshot the whole resulting directory hierarchy. The product validation 
is then made on the modules as a whole. The recipe would need to be a 
simple file-installer.

The problem with that is that one needs a target deployment system that 
runs npm install on the actual target in order to make the correct files 
for the target. The output from that must be plugged into the oe build 
system somehow, so there will be many moving parts in this scheme too.

 > Oh, and you might have more success using meta-nodejs, as it doesn't try
 > to implement a custom fetcher, and just runs npm directly during the
 > build. If you don't mind that that makes the build outcome essentially
 > random from one invocation to the next.
 >
 > https://github.com/imyller/meta-nodejs

The meta-nodejs repo seems to be abandoned, as it does support any newer 
versions of nodejs. I asked on #yocto and I was suggested that this 
layer wasn't really necessary as meta-openembedded supports nodejs.


Best regards,
Svein

Re: Yocto and NPM issues

[Alexander Kanavin]

On 03/21/2018 04:44 PM, Svein Seldal wrote:

>  > Oh, and you might have more success using meta-nodejs, as it doesn't try
>  > to implement a custom fetcher, and just runs npm directly during the
>  > build. If you don't mind that that makes the build outcome essentially
>  > random from one invocation to the next.
>  >
>  > https://github.com/imyller/meta-nodejs
> 
> The meta-nodejs repo seems to be abandoned, as it does support any newer 
> versions of nodejs. I asked on #yocto and I was suggested that this 
> layer wasn't really necessary as meta-openembedded supports nodejs.

You have seen first-hand what 'supports' means in practice :-) As long 
as it's not regularly, automatically tested against a wide set of target 
npm packages, it just isn't going to work well.

I'd suggest you could take over the maintenance of meta-nodejs maybe?

Alex