[PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Mattias Hansson]

do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
which sometimes causes a build error when building a recipe that adds
groups.

This issue only occurs when executing do_prepare_recipe_sysroot for a
recipe that adds groups before finishing a task that depends on pseudo
for a recipe that doesn't add groups.

Signed-off-by: Mattias Hansson <mattihn@axis.com>
---
 meta/classes/base.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 0c8a4b2862..0576b110c9 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -480,6 +480,7 @@ python () {
     # If we're building a target package we need to use fakeroot (pseudo)
     # in order to capture permissions, owners, groups and special files
     if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
+        d.setVarFlag('do_prepare_recipe_sysroot', 'fakeroot', '1')
         d.setVarFlag('do_unpack', 'umask', '022')
         d.setVarFlag('do_configure', 'umask', '022')
         d.setVarFlag('do_compile', 'umask', '022')
-- 
2.11.0

Re: [PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Mattias Hansson]

On Fri, Aug 16, 2019 at 11:13:29AM +0200, Mattias Hansson wrote:
> do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
> However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
> which sometimes causes a build error when building a recipe that adds
> groups.
> 
> This issue only occurs when executing do_prepare_recipe_sysroot for a
> recipe that adds groups before finishing a task that depends on pseudo
> for a recipe that doesn't add groups.
> 
> Signed-off-by: Mattias Hansson <mattihn@axis.com>
> ---
>  meta/classes/base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 0c8a4b2862..0576b110c9 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -480,6 +480,7 @@ python () {
>      # If we're building a target package we need to use fakeroot (pseudo)
>      # in order to capture permissions, owners, groups and special files
>      if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
> +        d.setVarFlag('do_prepare_recipe_sysroot', 'fakeroot', '1')
>          d.setVarFlag('do_unpack', 'umask', '022')
>          d.setVarFlag('do_configure', 'umask', '022')
>          d.setVarFlag('do_compile', 'umask', '022')
> -- 
> 2.11.0
> 

Ping. Just a friendly reminder for someone to take a look at this.

/Mattias

Re: [PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Richard Purdie]

On Fri, 2019-08-16 at 11:13 +0200, Mattias Hansson wrote:
> do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
> However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
> which sometimes causes a build error when building a recipe that adds
> groups.
> 
> This issue only occurs when executing do_prepare_recipe_sysroot for a
> recipe that adds groups before finishing a task that depends on pseudo
> for a recipe that doesn't add groups.
> 
> Signed-off-by: Mattias Hansson <mattihn@axis.com>
> ---
>  meta/classes/base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 0c8a4b2862..0576b110c9 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -480,6 +480,7 @@ python () {
>      # If we're building a target package we need to use fakeroot (pseudo)
>      # in order to capture permissions, owners, groups and special files
>      if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
> +        d.setVarFlag('do_prepare_recipe_sysroot', 'fakeroot', '1')
>          d.setVarFlag('do_unpack', 'umask', '022')
>          d.setVarFlag('do_configure', 'umask', '022')
>          d.setVarFlag('do_compile', 'umask', '022')

This basically forces all target recipes prepare-recipe sysroot to run
under pseudo "just in case", with all the performance overhead that
entails. prepare_recipe_sysroot does a lot of file accesses so this is
significant. It will also increase the pseudo database sizes
everywhere.

We'll need to find a better way to handle this I'm afraid.

Cheers,

Richard

Re: [PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 30 augusti 2019 18:50
> To: Mattias Hansson <Mattias.Hansson@axis.com>; openembedded-
> core@lists.openembedded.org
> Cc: Mattias Hansson <Mattias.Hansson@axis.com>
> Subject: Re: [OE-core] [PATCH] base.bbclass: add dependency on pseudo
> from do_prepare_recipe_sysroot
> 
> On Fri, 2019-08-16 at 11:13 +0200, Mattias Hansson wrote:
> > do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
> > However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
> > which sometimes causes a build error when building a recipe that adds
> > groups.
> >
> > This issue only occurs when executing do_prepare_recipe_sysroot for a
> > recipe that adds groups before finishing a task that depends on pseudo
> > for a recipe that doesn't add groups.
> >
> > Signed-off-by: Mattias Hansson <mattihn@axis.com>
> > ---
> >  meta/classes/base.bbclass | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> > index 0c8a4b2862..0576b110c9 100644
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -480,6 +480,7 @@ python () {
> >      # If we're building a target package we need to use fakeroot (pseudo)
> >      # in order to capture permissions, owners, groups and special files
> >      if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
> > +        d.setVarFlag('do_prepare_recipe_sysroot', 'fakeroot', '1')
> >          d.setVarFlag('do_unpack', 'umask', '022')
> >          d.setVarFlag('do_configure', 'umask', '022')
> >          d.setVarFlag('do_compile', 'umask', '022')
> 
> This basically forces all target recipes prepare-recipe sysroot to run
> under pseudo "just in case", with all the performance overhead that
> entails. prepare_recipe_sysroot does a lot of file accesses so this is
> significant. It will also increase the pseudo database sizes
> everywhere.
> 
> We'll need to find a better way to handle this I'm afraid.
> 
> Cheers,
> 
> Richard

What do you prefer then, that we add this to useradd.bbclass instead?

python () {
    # This corresponds to similar code in base.bbclass, but is added here as it
    # is only needed for recipes that add users/groups.
    if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
        d.setVarFlag('do_prepare_recipe_sysroot', 'fakeroot', '1')}
}

That way it should only affect recipes that manipulate users/groups.

//Peter

Re: [PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Mattias Hansson]

On Tue, Oct 15, 2019 at 10:27:27AM +0200, Richard Purdie wrote:
> On Tue, 2019-10-15 at 10:00 +0200, Mattias Hansson wrote:
> > do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
> > However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
> > which sometimes causes a build error when building a recipe that adds
> > groups.
> > 
> > This issue only occurs when executing do_prepare_recipe_sysroot for a
> > recipe that adds groups before finishing a task that depends on pseudo
> > for a recipe that doesn't add groups.
> > 
> > Signed-off-by: Mattias Hansson <mattihn@axis.com>
> > ---
> >  meta/classes/base.bbclass | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> > index d3184ecf7b..1cea3a2213 100644
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -482,6 +482,7 @@ python () {
> >      # If we're building a target package we need to use fakeroot (pseudo)
> >      # in order to capture permissions, owners, groups and special files
> >      if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
> > +        d.appendVarFlag('do_prepare_recipe_sysroot', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
> >          d.setVarFlag('do_unpack', 'umask', '022')
> >          d.setVarFlag('do_configure', 'umask', '022')
> >          d.setVarFlag('do_compile', 'umask', '022')
> 
> This was posted once already and the answer is still no, we do not want
> to add pseudo as a dependency for all sysroot construction. IO under
> pseudo is much slower and this is a very IO intensive task so it would
> hurt performance and isn't necessary in most cases.
> 
> I appreciate its harder but we need to find a better more targeted way
> to solve this and some of the other related pseudo issues for inter
> dependent user/groups.
> 
> Cheers,
> 
> RIchard
> 

Sorry for the confusion, but this is a different patch. The previous
forced do_prepare_recipe_sysroot to run in a fakerooted environment.
This caused a performance degrade and was therefore rejected, if I
understood you correctly.

This patch will only declare fakeroot-native as a dependency and only
cause pseudo to be available during do_prepare_recipe_sysroot. This
should not cause the task to run in a fakerooted environment, is this a
valid assumption?

/Mattias

Re: [PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Richard Purdie]

On Tue, 2019-10-15 at 10:00 +0200, Mattias Hansson wrote:
> do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
> However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
> which sometimes causes a build error when building a recipe that adds
> groups.
> 
> This issue only occurs when executing do_prepare_recipe_sysroot for a
> recipe that adds groups before finishing a task that depends on pseudo
> for a recipe that doesn't add groups.
> 
> Signed-off-by: Mattias Hansson <mattihn@axis.com>
> ---
>  meta/classes/base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index d3184ecf7b..1cea3a2213 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -482,6 +482,7 @@ python () {
>      # If we're building a target package we need to use fakeroot (pseudo)
>      # in order to capture permissions, owners, groups and special files
>      if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
> +        d.appendVarFlag('do_prepare_recipe_sysroot', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
>          d.setVarFlag('do_unpack', 'umask', '022')
>          d.setVarFlag('do_configure', 'umask', '022')
>          d.setVarFlag('do_compile', 'umask', '022')

This was posted once already and the answer is still no, we do not want
to add pseudo as a dependency for all sysroot construction. IO under
pseudo is much slower and this is a very IO intensive task so it would
hurt performance and isn't necessary in most cases.

I appreciate its harder but we need to find a better more targeted way
to solve this and some of the other related pseudo issues for inter
dependent user/groups.

Cheers,

RIchard

[PATCH] base.bbclass: add dependency on pseudo from do_prepare_recipe_sysroot

[Mattias Hansson]

do_prepare_recipe_sysroot may perform groupadd, which requires pseudo.
However, do_prepare_recipe_sysroot does not depend on pseudo explicitly,
which sometimes causes a build error when building a recipe that adds
groups.

This issue only occurs when executing do_prepare_recipe_sysroot for a
recipe that adds groups before finishing a task that depends on pseudo
for a recipe that doesn't add groups.

Signed-off-by: Mattias Hansson <mattihn@axis.com>
---
 meta/classes/base.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index d3184ecf7b..1cea3a2213 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -482,6 +482,7 @@ python () {
     # If we're building a target package we need to use fakeroot (pseudo)
     # in order to capture permissions, owners, groups and special files
     if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
+        d.appendVarFlag('do_prepare_recipe_sysroot', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
         d.setVarFlag('do_unpack', 'umask', '022')
         d.setVarFlag('do_configure', 'umask', '022')
         d.setVarFlag('do_compile', 'umask', '022')
-- 
2.11.0