Fix some NPM related problems

Fix some NPM related problems

[Zoltan Boszormenyi]

When trying to create recipes for Angular so that "ng build"
can be run, I ran into some problems. Two of them I was able fix.

The first two problems are in the same class and boil down to
the fact that some node module archives are created by an idiotic
tar client (probably on Windows) and contain wrong permissions.

Problem 1) Directories have no "x" permission bit

This cannot be worked around as tar creates the directory as is
but creating the first file in the directory will fail.

tar exits with an error and there is no option for it to
override directory permissions.

license-webpack-plugin 2.3.1, a dependency for Angular 11 is
such a contender.

Problem 2) package.json have no "w" permission bit

This can be worked around with
[PATCH 1/2] npm.bbclass: Fix file permissions before opening it for writing

Problem 3) Recipes with "inherit npm" install the /usr/lib/node symlink

I needed to create a complete set of Angular recipes where "ng build"
can run. But when trying to use them in DEPENDS for an Angular based
project, this error kicked in from staging.bbclass:

bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))

This is fixed by ignoring the sole /usr/lib/node symlink in staging.bbclass
when checking for identical files with

[PATCH 2/2] staging.bbclass: Ignore installing multiple symlinks of the same name

I discovered this issue in Yocto 3.3.

I cc-ed relevant parties who edited either bbclass around where
I just modified them.

Best regards,
Zoltán Böszörményi

[PATCH 1/2] npm.bbclass: Fix file permissions before opening it for writing

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Some node module archives in npmjs.org contain wrong permissions.
I found a case with package.json in the archive being r-xr-xr-x
for which open(..., "w") fails. Set the manifest file permissions
to 0666 just in case.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/npm.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index ba50fcac20..7f52ec6061 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -202,6 +202,7 @@ python npm_do_configure() {
         if has_shrinkwrap_file:
             _update_manifest("devDependencies")
 
+    os.chmod(cached_manifest_file, 0o666)
     with open(cached_manifest_file, "w") as f:
         json.dump(cached_manifest, f, indent=2)
 
-- 
2.36.1

[PATCH 2/2] staging.bbclass: Ignore installing multiple symlinks of the same name

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Recipes using "inherit npm" install the same /usr/lib/node symlink
that points to node_modules. When a recipe DEPENDS on multiple
npm based recipe, do_prepare_recipe_sysroot() fails.

Ignore this symlink in a way that the set of ignored symlinks
can be extended later if needed.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/staging.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 9fc8f4f283..f7b6056219 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -528,6 +528,7 @@ python extend_recipe_sysroot() {
 
             with open(manifest, "r") as f:
                 manifests[dep] = manifest
+                ignored_dests = [ ''.join([d.getVar('nonarch_libdir'), '/node' ]) ]
                 for l in f:
                     l = l.strip()
                     if l.endswith("/fixmepath"):
@@ -545,7 +546,10 @@ python extend_recipe_sysroot() {
                     hashname = targetdir + dest
                     if not hashname.endswith("/"):
                         if hashname in fileset:
-                            bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
+                            if os.path.islink(hashname) and  dest in ignored_dests:
+                                bb.note("The symlink '%s' %s wants to install is already installed by %s, ignoring" % (dest, c, fileset[hashname]))
+                            else:
+                                bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
                         else:
                             fileset[hashname] = c
 
-- 
2.36.1

Re: [OE-core] [PATCH 1/2] npm.bbclass: Fix file permissions before opening it for writing

[Alexander Kanavin]

Apologies, 666 is not right. You shouldn't grant rights to 'others'.
Besides, 6 is 110, which is rw-, no?

Alex

On Fri, 13 May 2022 at 15:49, Zoltan Boszormenyi via
lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>
> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> Some node module archives in npmjs.org contain wrong permissions.
> I found a case with package.json in the archive being r-xr-xr-x
> for which open(..., "w") fails. Set the manifest file permissions
> to 0666 just in case.
>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/npm.bbclass | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
> index ba50fcac20..7f52ec6061 100644
> --- a/meta/classes/npm.bbclass
> +++ b/meta/classes/npm.bbclass
> @@ -202,6 +202,7 @@ python npm_do_configure() {
>          if has_shrinkwrap_file:
>              _update_manifest("devDependencies")
>
> +    os.chmod(cached_manifest_file, 0o666)
>      with open(cached_manifest_file, "w") as f:
>          json.dump(cached_manifest, f, indent=2)
>
> --
> 2.36.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#165565): https://lists.openembedded.org/g/openembedded-core/message/165565
> Mute This Topic: https://lists.openembedded.org/mt/91080992/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH 2/2] staging.bbclass: Ignore installing multiple symlinks of the same name

[Alexander Kanavin]

Apologies, but special-casing this in staging class is not right
either. The problem should be addressed at the making of
sysroot-destdir/ step from the npm class.

Alex


On Fri, 13 May 2022 at 15:49, Zoltan Boszormenyi via
lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>
> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> Recipes using "inherit npm" install the same /usr/lib/node symlink
> that points to node_modules. When a recipe DEPENDS on multiple
> npm based recipe, do_prepare_recipe_sysroot() fails.
>
> Ignore this symlink in a way that the set of ignored symlinks
> can be extended later if needed.
>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/staging.bbclass | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
> index 9fc8f4f283..f7b6056219 100644
> --- a/meta/classes/staging.bbclass
> +++ b/meta/classes/staging.bbclass
> @@ -528,6 +528,7 @@ python extend_recipe_sysroot() {
>
>              with open(manifest, "r") as f:
>                  manifests[dep] = manifest
> +                ignored_dests = [ ''.join([d.getVar('nonarch_libdir'), '/node' ]) ]
>                  for l in f:
>                      l = l.strip()
>                      if l.endswith("/fixmepath"):
> @@ -545,7 +546,10 @@ python extend_recipe_sysroot() {
>                      hashname = targetdir + dest
>                      if not hashname.endswith("/"):
>                          if hashname in fileset:
> -                            bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
> +                            if os.path.islink(hashname) and  dest in ignored_dests:
> +                                bb.note("The symlink '%s' %s wants to install is already installed by %s, ignoring" % (dest, c, fileset[hashname]))
> +                            else:
> +                                bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
>                          else:
>                              fileset[hashname] = c
>
> --
> 2.36.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#165566): https://lists.openembedded.org/g/openembedded-core/message/165566
> Mute This Topic: https://lists.openembedded.org/mt/91080993/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH 1/2] npm.bbclass: Fix file permissions before opening it for writing

[Alexander Kanavin]

I think you should just add the needed permissions, not re-set them
fully: u+rw or similar.

Alex

On Fri, 13 May 2022 at 15:58, Böszörményi Zoltán <zboszor@gmail.com> wrote:
>
> 2022. 05. 13. 15:52 keltezéssel, Alexander Kanavin írta:
> > Apologies, 666 is not right. You shouldn't grant rights to 'others'.
> > Besides, 6 is 110, which is rw-, no?
>
> Reading the file is also important so the r and w permissions
> should be there.
>
> But 664 or 644 is definitely better security-wise.
>
> I will re-send a v2.
>
> >
> > Alex
> >
> > On Fri, 13 May 2022 at 15:49, Zoltan Boszormenyi via
> > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >> From: Zoltán Böszörményi <zboszor@gmail.com>
> >>
> >> Some node module archives in npmjs.org contain wrong permissions.
> >> I found a case with package.json in the archive being r-xr-xr-x
> >> for which open(..., "w") fails. Set the manifest file permissions
> >> to 0666 just in case.
> >>
> >> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> >> ---
> >>   meta/classes/npm.bbclass | 1 +
> >>   1 file changed, 1 insertion(+)
> >>
> >> diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
> >> index ba50fcac20..7f52ec6061 100644
> >> --- a/meta/classes/npm.bbclass
> >> +++ b/meta/classes/npm.bbclass
> >> @@ -202,6 +202,7 @@ python npm_do_configure() {
> >>           if has_shrinkwrap_file:
> >>               _update_manifest("devDependencies")
> >>
> >> +    os.chmod(cached_manifest_file, 0o666)
> >>       with open(cached_manifest_file, "w") as f:
> >>           json.dump(cached_manifest, f, indent=2)
> >>
> >> --
> >> 2.36.1
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#165565): https://lists.openembedded.org/g/openembedded-core/message/165565
> >> Mute This Topic: https://lists.openembedded.org/mt/91080992/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>

Re: [OE-core] [PATCH 2/2] staging.bbclass: Ignore installing multiple symlinks of the same name

[Zoltan Boszormenyi]

2022. 05. 13. 15:55 keltezéssel, Alexander Kanavin írta:
> Apologies, but special-casing this in staging class is not right
> either. The problem should be addressed at the making of
> sysroot-destdir/ step from the npm class.

npm.bbclass creates the symlink in sysroot-destdir for recipe A

But then it's the job of staging.bbclass to populate a recipe B's
recipe-sysroot and recipe-sysroot-native from recipe A.

How can this be worked around from npm.bbclass?

> 
> Alex
> 
> 
> On Fri, 13 May 2022 at 15:49, Zoltan Boszormenyi via
> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>
>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>
>> Recipes using "inherit npm" install the same /usr/lib/node symlink
>> that points to node_modules. When a recipe DEPENDS on multiple
>> npm based recipe, do_prepare_recipe_sysroot() fails.
>>
>> Ignore this symlink in a way that the set of ignored symlinks
>> can be extended later if needed.
>>
>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>> ---
>>   meta/classes/staging.bbclass | 6 +++++-
>>   1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
>> index 9fc8f4f283..f7b6056219 100644
>> --- a/meta/classes/staging.bbclass
>> +++ b/meta/classes/staging.bbclass
>> @@ -528,6 +528,7 @@ python extend_recipe_sysroot() {
>>
>>               with open(manifest, "r") as f:
>>                   manifests[dep] = manifest
>> +                ignored_dests = [ ''.join([d.getVar('nonarch_libdir'), '/node' ]) ]
>>                   for l in f:
>>                       l = l.strip()
>>                       if l.endswith("/fixmepath"):
>> @@ -545,7 +546,10 @@ python extend_recipe_sysroot() {
>>                       hashname = targetdir + dest
>>                       if not hashname.endswith("/"):
>>                           if hashname in fileset:
>> -                            bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
>> +                            if os.path.islink(hashname) and  dest in ignored_dests:
>> +                                bb.note("The symlink '%s' %s wants to install is already installed by %s, ignoring" % (dest, c, fileset[hashname]))
>> +                            else:
>> +                                bb.fatal("The file %s is installed by both %s and %s, aborting" % (dest, c, fileset[hashname]))
>>                           else:
>>                               fileset[hashname] = c
>>
>> --
>> 2.36.1
>>
>>
>>
>>
>>
>>
>> 
>>

Re: [OE-core] [PATCH 2/2] staging.bbclass: Ignore installing multiple symlinks of the same name

[richard.purdie]

On Fri, 2022-05-13 at 16:02 +0200, Zoltan Boszormenyi via
lists.openembedded.org wrote:
> 2022. 05. 13. 15:55 keltezéssel, Alexander Kanavin írta:
> > Apologies, but special-casing this in staging class is not right
> > either. The problem should be addressed at the making of
> > sysroot-destdir/ step from the npm class.
> 
> npm.bbclass creates the symlink in sysroot-destdir for recipe A
> 
> But then it's the job of staging.bbclass to populate a recipe B's
> recipe-sysroot and recipe-sysroot-native from recipe A.
> 
> How can this be worked around from npm.bbclass?

You could move the line from npm.bbclass into the node recipe so it is
only created once?

Cheers,

Richard