[PATCH][RFC] sstate: allow specifying indirect dependencies to exclude from sysroot

[PATCH][RFC] sstate: allow specifying indirect dependencies to exclude from sysroot

[André Draszik]

From: André Draszik <andre.draszik@jci.com>

Currently, a dependency on any -native recipe will pull in
all dependencies of that -native recipe in the recipe
sysroot. This behaviour might not always be wanted, e.g.
when that -native recipe depends on build-tools that are
not relevant for the current recipe.

This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
which will be evaluated for such recursive dependencies to
be excluded. The idea is similar to
   http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html
except that the list is not hard-coded anymore.

SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
expressions of recipe and dependency to ignore, e.g. in
the above flex-native / bison-native use-case, one would
specify

    SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"

in layer.conf.

The existing special handling of "-initial" as well as
"base-passwd" and "shadow-sysroot" could also be
streamlined:

    SSTATE_EXCLUDEDEPS_SYSROOT += "\
        .*->.*-initial.* \
        .*(base-passwd|shadow-sysroot)->.* \
    "

Another anticipated user is meta-java, where certain newer
JDKs can only be bootstrapped (built) using older JDKs,
but it doesn't make much sense to copy all those older
JDKs and their own build tools (ant, etc.) into the
sysroot of recipes wanting to be built using the newer JDK
(only), e.g.:

    SSTATE_EXCLUDEDEPS_SYSROOT += "\
        openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \
    "

Signed-off-by: André Draszik <andre.draszik@jci.com>
---
 meta/classes/sstate.bbclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 68089421f5..62e784e497 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -901,6 +901,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
     # task is included in taskdependees too
     # Return - False - We need this dependency
     #        - True - We can skip this dependency
+    import re
 
     def logit(msg, log):
         if log is not None:
@@ -961,6 +962,18 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
             # Nothing need depend on libc-initial/gcc-cross-initial
             if "-initial" in taskdependees[task][0]:
                 continue
+            # Allow excluding certain recursive dependencies. If a recipe needs it should add a
+            # specific dependency itself, rather than relying on one of its dependees to pull
+            # them in.
+            # See also http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html
+            not_needed = False
+            for excl in (d.getVar('SSTATE_EXCLUDEDEPS_SYSROOT') or "").split():
+                if re.match(excl.split('->', 1)[0], taskdependees[dep][0]):
+                    if re.match(excl.split('->', 1)[1], taskdependees[task][0]):
+                        not_needed = True
+                        break
+            if not_needed:
+                continue
             # For meta-extsdk-toolchain we want all sysroot dependencies
             if taskdependees[dep][0] == 'meta-extsdk-toolchain':
                 return False
-- 
2.16.2

Re: [PATCH][RFC] sstate: allow specifying indirect dependencies to exclude from sysroot

[André Draszik]

I realise that this is basically working around the fact that -native
recipes don't get any RDEPENDS treatment AFAICS, except that RDEPENDS ==
DEPENDS, which is what I want to avoid, at least in certain cases...

I guess if computing proper RDEPENDS for -native recipes was easy, it would
have been done already?


Cheers,
Andre'


On Wed, 2018-04-25 at 15:12 +0100, André Draszik wrote:
> From: André Draszik <andre.draszik@jci.com>
> 
> Currently, a dependency on any -native recipe will pull in
> all dependencies of that -native recipe in the recipe
> sysroot. This behaviour might not always be wanted, e.g.
> when that -native recipe depends on build-tools that are
> not relevant for the current recipe.
> 
> This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
> which will be evaluated for such recursive dependencies to
> be excluded. The idea is similar to
>    http://lists.openembedded.org/pipermail/openembedded-core/2018-January/
> 146324.html
> except that the list is not hard-coded anymore.
> 
> SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
> expressions of recipe and dependency to ignore, e.g. in
> the above flex-native / bison-native use-case, one would
> specify
> 
>     SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"
> 
> in layer.conf.
> 
> The existing special handling of "-initial" as well as
> "base-passwd" and "shadow-sysroot" could also be
> streamlined:
> 
>     SSTATE_EXCLUDEDEPS_SYSROOT += "\
>         .*->.*-initial.* \
>         .*(base-passwd|shadow-sysroot)->.* \
>     "
> 
> Another anticipated user is meta-java, where certain newer
> JDKs can only be bootstrapped (built) using older JDKs,
> but it doesn't make much sense to copy all those older
> JDKs and their own build tools (ant, etc.) into the
> sysroot of recipes wanting to be built using the newer JDK
> (only), e.g.:
> 
>     SSTATE_EXCLUDEDEPS_SYSROOT += "\
>         openjdk-8-native->(ant-native|attr-native|coreutils-
> native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-
> native|unzip-native) \
>     "
> 
> Signed-off-by: André Draszik <andre.draszik@jci.com>
> ---
>  meta/classes/sstate.bbclass | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 68089421f5..62e784e497 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -901,6 +901,7 @@ def setscene_depvalid(task, taskdependees, notneeded,
> d, log=None):
>      # task is included in taskdependees too
>      # Return - False - We need this dependency
>      #        - True - We can skip this dependency
> +    import re
>  
>      def logit(msg, log):
>          if log is not None:
> @@ -961,6 +962,18 @@ def setscene_depvalid(task, taskdependees, notneeded,
> d, log=None):
>              # Nothing need depend on libc-initial/gcc-cross-initial
>              if "-initial" in taskdependees[task][0]:
>                  continue
> +            # Allow excluding certain recursive dependencies. If a recipe
> needs it should add a
> +            # specific dependency itself, rather than relying on one of
> its dependees to pull
> +            # them in.
> +            # See also http://lists.openembedded.org/pipermail/openembedd
> ed-core/2018-January/146324.html
> +            not_needed = False
> +            for excl in (d.getVar('SSTATE_EXCLUDEDEPS_SYSROOT') or
> "").split():
> +                if re.match(excl.split('->', 1)[0],
> taskdependees[dep][0]):
> +                    if re.match(excl.split('->', 1)[1],
> taskdependees[task][0]):
> +                        not_needed = True
> +                        break
> +            if not_needed:
> +                continue
>              # For meta-extsdk-toolchain we want all sysroot dependencies
>              if taskdependees[dep][0] == 'meta-extsdk-toolchain':
>                  return False