[Buildroot] [Bug 9531] New: NPM fails to build embedded modules

[Buildroot] [Bug 9531] New: NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

            Bug ID: 9531
           Summary: NPM fails to build embedded modules
           Product: buildroot
           Version: 2016.11
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: Other
          Assignee: unassigned at buildroot.uclibc.org
          Reporter: bilge at scriptfusion.com
                CC: buildroot at uclibc.org
  Target Milestone: ---

In this example I try to build ghost at 0.11.3. Its first dependency is
sqlite3 at 3.1.8, which fails immediately. I was able to build Ghost 0.5.10 two
years ago, which also had the sqlite dependency, and have run my blog on it
ever since (https://github.com/Docker-nano/Ghost). It should be possible to
make it work again.

> sqlite3 at 3.1.8 install /root/buildroot-2016.11/output/target/usr/lib/node_modules/ghost/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

sh: 1: node-pre-gyp: Permission denied
npm info lifecycle sqlite3 at 3.1.8~install: Failed to exec install script
/root/buildroot-2016.11/output/target/usr/lib
`-- (empty)

npm ERR! Linux 3.13.0-68-generic
npm ERR! argv "/root/buildroot-2016.11/output/host/usr/bin/node"
"/root/buildroot-2016.11/output/host/usr/bin/npm" "install" "-d" "-g"
"--build-from-source" "ghost at 0.11.3"
npm ERR! node v6.7.0
npm ERR! npm  v3.10.3
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn

npm ERR! sqlite3 at 3.1.8 install: `node-pre-gyp install --fallback-to-build`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the sqlite3 at 3.1.8 install script 'node-pre-gyp install
--fallback-to-build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sqlite3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /root/buildroot-2016.11/npm-debug.log
npm ERR! code 1
package/pkg-generic.mk:297: recipe for target
'/root/buildroot-2016.11/output/build/nodejs-6.7.0/.stamp_target_installed'
failed
make: ***
[/root/buildroot-2016.11/output/build/nodejs-6.7.0/.stamp_target_installed]
Error 1

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #1 from bilge at scriptfusion.com ---
Although you may observe the npm command has been modified to include "-d" and
"--build-from-source" switches, these make no difference to the error output.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #2 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
Please include the Buildroot version, and a minimal Buildroot .config file that
allows to reproduce the issue. Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #3 from bilge at scriptfusion.com ---
The version is specified in the ticket (2016.11).

Here is the config.

BR2_x86_64=y
BR2_WGET="wget --passive-ftp -nd -t 3 -nv"
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_PATH="$(HOME)/toolchain"
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-nano-linux-uclibc"
BR2_TOOLCHAIN_EXTERNAL_GCC_5=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_3=y
BR2_TOOLCHAIN_EXTERNAL_WCHAR=y
BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_TARGET_GENERIC_HOSTNAME="Docker-nano"
BR2_TARGET_GENERIC_ISSUE="Docker nano"
BR2_INIT_NONE=y
# BR2_TARGET_GENERIC_GETTY is not set
BR2_ROOTFS_OVERLAY="rootfs_overlay"
BR2_ROOTFS_POST_BUILD_SCRIPT="./post_build.sh"
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_NODEJS=y
BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="ghost at 0.11.3"
BR2_PACKAGE_OPENSSL=y
BR2_TARGET_ROOTFS_TAR_XZ=y

This uses the toolchain available from
https://github.com/Docker-nano/crosstool-NG/releases/download/2.0.0/x86_64-nano-linux-uclibc.tar.xz

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #4 from bilge at scriptfusion.com ---
Further investigation reveals the culprit is the `-g` switch in the line,
`$(NPM) install -g $(NODEJS_MODULES_LIST)` in package/nodejs/nodejs.mk.

Removing the `-g` switch allows the module and all its dependencies, including
sqlite3, to build successfully. However, since they are now built in a
different location, they are not included in the rootfs which defeats the
purpose of building them!

My knowledge of Buildroot is too poor to know why an access denied violation
occurs with `-g` or how to include the modules in the rootfs when the `-g`
option is omitted, however I know the 2014.08 release of Buildroot did not rely
on `-g` and this is the last known release version I could get this module
working on.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #5 from Peter Korsgaard <jacmet@uclibc.org> ---
I cannot reproduce it here. The error message refers to node-pre-grep which
also uses /usr/bin/env, so presumably this is related to #9526?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #6 from bilge at scriptfusion.com ---
Hi Peter, unfortunately I cannot stop producing this error. I do not believe
this is related to `env` on the target at all because I just spent the entire
day upgrading my toolchain and rebuilding everything to find exactly the same
error is produced even when `env` is successfully built and installed to the
target prior to running the NPM module build process.

NPM tries to run the following command:

PATH="/root/buildroot-2016.11/output/host/bin:/root/buildroot-2016.11/output/host/sbin:/root/buildroot-2016.11/output/host/usr/bin:/root/buildroot-2016.11/output/host/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
AR="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-ar"
AS="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-as"
LD="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-ld"
NM="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-nm"
CC="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-gcc"
GCC="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-gcc"
CPP="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-cpp"
CXX="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-g++"
FC="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-gfortran"
F77="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-gfortran"
RANLIB="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-ranlib"
READELF="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-readelf"
STRIP="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-strip"
OBJCOPY="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-objcopy"
OBJDUMP="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-objdump"
AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as"
CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc"
CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld"
CPPFLAGS_FOR_BUILD="-I/root/buildroot-2016.11/output/host/usr/include"
CFLAGS_FOR_BUILD="-O2 -I/root/buildroot-2016.11/output/host/usr/include"
CXXFLAGS_FOR_BUILD="-O2 -I/root/buildroot-2016.11/output/host/usr/include"
LDFLAGS_FOR_BUILD="-L/root/buildroot-2016.11/output/host/lib
-L/root/buildroot-2016.11/output/host/usr/lib
-Wl,-rpath,/root/buildroot-2016.11/output/host/usr/lib" FCFLAGS_FOR_BUILD=""
DEFAULT_ASSEMBLER="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-as"
DEFAULT_LINKER="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-ld"
CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os "
CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os
" LDFLAGS="" FCFLAGS=" -Os " FFLAGS=" -Os "
PKG_CONFIG="/root/buildroot-2016.11/output/host/usr/bin/pkg-config"
STAGING_DIR="/root/buildroot-2016.11/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot"
INTLTOOL_PERL=/usr/bin/perl
LD="/root/buildroot-2016.11/output/host/usr/bin/x86_64-nano-linux-uclibc-g++"
npm_config_arch=x64 npm_config_target_arch=x64
npm_config_build_from_source=true
npm_config_nodedir=/root/buildroot-2016.11/output/build/nodejs-6.9.2
npm_config_prefix=/root/buildroot-2016.11/output/target/usr
/root/buildroot-2016.11/output/host/usr/bin/npm install -g ghost at 0.11.3

Which spits out the same errors mentioned previously:

npm WARN lifecycle ghost at 0.11.3~preinstall: cannot run in wd %s %s (wd=%s)
ghost at 0.11.3 node core/server/utils/npm/preinstall.js
/root/buildroot-2016.11/output/target/usr/lib/node_modules/.staging/ghost-88d37318

> sqlite3 at 3.1.8 install /root/buildroot-2016.11/output/target/usr/lib/node_modules/ghost/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

sh: 1: node-pre-gyp: Permission denied
/root/buildroot-2016.11/output/target/usr/lib
`-- (empty)

npm ERR! Linux 3.13.0-68-generic
npm ERR! argv "/root/buildroot-2016.11/output/host/usr/bin/node"
"/root/buildroot-2016.11/output/host/usr/bin/npm" "install" "-g" "ghost at 0.11.3"
npm ERR! node v6.9.2
npm ERR! npm  v3.10.9
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn

npm ERR! sqlite3 at 3.1.8 install: `node-pre-gyp install --fallback-to-build`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the sqlite3 at 3.1.8 install script 'node-pre-gyp install
--fallback-to-build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sqlite3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /root/buildroot-2016.11/package/nodejs/npm-debug.log
npm ERR! code 1

As before, simply removing the `-g` flag results in a successful build in
whatever the working directory happens to be. This is true for literally any
directory because I'm root and I can write everywhere, so I do not understand
where the "sh: 1: node-pre-gyp: Permission denied" error comes from when the
`-g` flag is specified.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #7 from bilge at scriptfusion.com ---
It seems this issue has been extensively documented at
https://github.com/mapbox/node-sqlite3/issues/612 over the past 9 months. The
solution, when building as root, is simply to specify the --unsafe-perm option
to NPM. I include a patch, below.

+++ package/nodejs/nodejs.mk
@@ -160,7 +160,7 @@
        # If you're having trouble with module installation, adding -d to the
        # npm install call below and setting npm_config_rollback=false can both
        # help in diagnosing the problem.
-       $(NPM) install -g $(NODEJS_MODULES_LIST)
+       $(NPM) install -g --unsafe-perm $(NODEJS_MODULES_LIST)
 endef
 endif

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998 at free.fr

--- Comment #8 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Bilge,

Nothing in Buildroot requires building as root. Building as root is not
supported (at least, not officially).

I'm not sure we want to take this patch. Playing with permissions (as
the name seems to imply) is not very clean IMHO. What does the doc about
--unsafe-perms says exactly?

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #9 from bilge at scriptfusion.com ---
If you don't want it, that's your prerogative, I'm just trying to help.

The documentation is here: https://docs.npmjs.com/misc/config#unsafe-perm
Copied in for completeness:

unsafe-perm

    Default: false if running as root, true otherwise
    Type: Boolean

Set to true to suppress the UID/GID switching when running package scripts. If
set explicitly to false, then installing as a non-root user will fail.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #10 from Peter Korsgaard <jacmet@uclibc.org> ---
(In reply to bilge from comment #7)

Why are you building as root in the first place? Is that some kind of side
effect of docker?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #11 from bilge at scriptfusion.com ---
It has nothing to do with Docker; you can think of Docker like any other
virtual machine. I build as root (a) because I do not care about users or
permissions in a VM and (b) because I had no reason to believe Buildroot would
not work when running as root.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #12 from Peter Korsgaard <jacmet@uclibc.org> ---
(In reply to bilge from comment #11)

Ok, but I would strongly suggest not building as root. Several packages behave
differently when built as root (E.G. your npm issues), and Buildroot is
specifically made to not require root permissions and be used from normal
users. E.G. from the manual:

Important: you can and should build everything as a normal user. There is no
need to be root to configure and use Buildroot. By running all commands as a
regular user, you protect your system against packages behaving badly during
compilation and installation.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

--- Comment #13 from Martin <martin@barkynet.com> ---
I tested this and setting BR2_PACKAGE_NODEJS_MODULES_ADDITIONAL="ghost at 0.11.3"
builds fine when building as a non-root user.

What is happening here is npm is dropping it's root privileges before running
node-pre-gyp which is why you see "node-pre-gyp: Permission denied" because it
can't access the files under /root.  --unsafe-perm stops this behaviour which
is why it then works.

I agree with Peter, building as root should be avoided.  I would recommend you
modify your Dockerfile to build as a non-root user.

The one change which might be worth making is a recommendation against building
as root in the manual and/or the Makefile.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 9531] NPM fails to build embedded modules

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=9531

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998 at free.fr
         Resolution|---                         |INVALID
             Status|NEW                         |RESOLVED

--- Comment #14 from Yann E. MORIN <yann.morin.1998@free.fr> ---
(In reply to Martin from comment #13)

There is already a blurb in the manual that says that running as root is not
recommended:

    https://buildroot.org/downloads/manual/manual.html#_buildroot_quick_start

Closing this bug, as it is not a bug in Buildroot but in the environment.

Thanks!

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.