[PATCHv3 0/1] Use fixed DISTRO_FEATURES for native

[PATCHv3 0/1] Use fixed DISTRO_FEATURES for native

[Jussi Kukkonen]

Avoid signifant native task churn when DISTRO_FEATURES is changed
by defining (almost) fixed DISTRO_FEATURES for native recipes.

Add DISTRO_FEATURES_FILTER_NATIVE and DISTRO_FEATURES_NATIVE to
limit the changes to DISTRO_FEATURES when building native recipes.


Changes since V3:
* Use both DISTRO_FEATURES_FILTER_NATIVE (from v2) and
  DISTRO_FEATURES_NATIVE (from v1): see commit message for details
* Do the same thing for nativesdk
* Remove the x11 patches and solve the problem by always adding
  x11 to DISTRO_FEATURES_NATIVE and DISTRO_FEATURES_NATIVESDK
  (so it's used even when target DISTRO_FEATURES does not have it)


Thanks,
  Jussi

The following changes since commit 0793c758b15e5007b4dd4b146987a0e74d6f6cba:

  Revert "staging: Fix sysroot problem with populate_sysroot dependencies on do_fetch" (2017-04-11 00:57:03 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib jku/native-distro-features
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=jku/native-distro-features

Jussi Kukkonen (1):
  native/nativesdk: Use fixed DISTRO_FEATURES

 meta/classes/native.bbclass    | 14 ++++++++++----
 meta/classes/nativesdk.bbclass |  6 ++++++
 meta/conf/bitbake.conf         |  9 +++++++++
 3 files changed, 25 insertions(+), 4 deletions(-)

-- 
2.1.4

[PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES

[Jussi Kukkonen]

There seems to be little advantage to letting distro features affect
native builds. There is a significant disadvantage: a change to
DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In a
test like this:
  $ bitbake core-image-minimal
  # append " systemd" to DISTRO_FEATURES
  $ bitbake core-image-minimal
The latter build takes 44 minutes (28%) of cpu-time less with this
patch (skipping 135 native tasks). Sadly wall clock time was not
affected as glibc remains the bottleneck.

Set native distro features to DISTRO_FEATURES_NATIVE appended with
an intersection of DISTRO_FEATURES and DISTRO_FEATURES_FILTER_NATIVE.
Current default values (baitbake.conf) are
* DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" (as gtk-doc-native
has much less dependencies when built without it)
* DISTRO_FEATURES_NATIVE ?= "x11" (to enable native UIs even if target
does not containe them)

Do the variable setting in native_virtclass_handler() because otherwise
it could still be overridden by appends and the feature backfilling.
Shuffle the early returns so DISTRO_FEATURES gets set as long as
the packagename ends with "-native".

Add similar variables for nativesdk.

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
---
 meta/classes/native.bbclass    | 14 ++++++++++----
 meta/classes/nativesdk.bbclass |  6 ++++++
 meta/conf/bitbake.conf         |  9 +++++++++
 3 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/meta/classes/native.bbclass b/meta/classes/native.bbclass
index 1919fbc..aec1087 100644
--- a/meta/classes/native.bbclass
+++ b/meta/classes/native.bbclass
@@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-intercept:"
 SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
 
 python native_virtclass_handler () {
-    classextend = e.data.getVar('BBCLASSEXTEND') or ""
-    if "native" not in classextend:
-        return
-
     pn = e.data.getVar("PN")
     if not pn.endswith("-native"):
         return
 
+    # Set features here to prevent appends and distro features backfill
+    # from modifying native distro features
+    features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
+    filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
+    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
+
+    classextend = e.data.getVar('BBCLASSEXTEND') or ""
+    if "native" not in classextend:
+        return
+
     def map_dependencies(varname, d, suffix = ""):
         if suffix:
             varname = varname + "_" + suffix
diff --git a/meta/classes/nativesdk.bbclass b/meta/classes/nativesdk.bbclass
index ebcfb2d..655b456 100644
--- a/meta/classes/nativesdk.bbclass
+++ b/meta/classes/nativesdk.bbclass
@@ -70,6 +70,12 @@ python nativesdk_virtclass_handler () {
     if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")):
         return
 
+    # Set features here to prevent appends and distro features backfill
+    # from modifying nativesdk distro features
+    features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split())
+    filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split())
+    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
+
     e.data.setVar("MLPREFIX", "nativesdk-")
     e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-nativesdk", "").replace("nativesdk-", ""))
     e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) + ":virtclass-nativesdk")
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 5e98d45..dca6bd4 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -789,6 +789,15 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
 EXTRA_IMAGE_FEATURES ??= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
 
+# Native distro features (will always be used for -native, even if they
+# are not enabled for target)
+DISTRO_FEATURES_NATIVE ?= "x11"
+DISTRO_FEATURES_NATIVESDK ?= "x11"
+# Normally target distro features will not be applied to native builds:
+# Native distro features on this list will use the target feature value
+DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation"
+DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation"
+
 DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-introspection-data ldconfig"
 MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
 
-- 
2.1.4

Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Jussi Kukkonen
> Sent: den 11 april 2017 16:36
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixed
> DISTRO_FEATURES
> 
> There seems to be little advantage to letting distro features affect
> native builds. There is a significant disadvantage: a change to
> DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In a
> test like this:
>   $ bitbake core-image-minimal
>   # append " systemd" to DISTRO_FEATURES
>   $ bitbake core-image-minimal
> The latter build takes 44 minutes (28%) of cpu-time less with this
> patch (skipping 135 native tasks). Sadly wall clock time was not
> affected as glibc remains the bottleneck.
> 
> Set native distro features to DISTRO_FEATURES_NATIVE appended with
> an intersection of DISTRO_FEATURES and DISTRO_FEATURES_FILTER_NATIVE.
> Current default values (baitbake.conf) are
> * DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" (as gtk-doc-
> native
> has much less dependencies when built without it)
> * DISTRO_FEATURES_NATIVE ?= "x11" (to enable native UIs even if target
> does not containe them)
> 
> Do the variable setting in native_virtclass_handler() because otherwise
> it could still be overridden by appends and the feature backfilling.
> Shuffle the early returns so DISTRO_FEATURES gets set as long as
> the packagename ends with "-native".
> 
> Add similar variables for nativesdk.
> 
> Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
> ---
>  meta/classes/native.bbclass    | 14 ++++++++++----
>  meta/classes/nativesdk.bbclass |  6 ++++++
>  meta/conf/bitbake.conf         |  9 +++++++++
>  3 files changed, 25 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes/native.bbclass b/meta/classes/native.bbclass
> index 1919fbc..aec1087 100644
> --- a/meta/classes/native.bbclass
> +++ b/meta/classes/native.bbclass
> @@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-
> intercept:"
>  SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
> 
>  python native_virtclass_handler () {
> -    classextend = e.data.getVar('BBCLASSEXTEND') or ""
> -    if "native" not in classextend:
> -        return
> -
>      pn = e.data.getVar("PN")
>      if not pn.endswith("-native"):
>          return
> 
> +    # Set features here to prevent appends and distro features
> backfill
> +    # from modifying native distro features
> +    features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
> +    filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
> +    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))

You should sort the list of features to make it deterministic.

> +
> +    classextend = e.data.getVar('BBCLASSEXTEND') or ""
> +    if "native" not in classextend:
> +        return
> +
>      def map_dependencies(varname, d, suffix = ""):
>          if suffix:
>              varname = varname + "_" + suffix
> diff --git a/meta/classes/nativesdk.bbclass
> b/meta/classes/nativesdk.bbclass
> index ebcfb2d..655b456 100644
> --- a/meta/classes/nativesdk.bbclass
> +++ b/meta/classes/nativesdk.bbclass
> @@ -70,6 +70,12 @@ python nativesdk_virtclass_handler () {
>      if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")):
>          return
> 
> +    # Set features here to prevent appends and distro features
> backfill
> +    # from modifying nativesdk distro features
> +    features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split())
> +    filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split())
> +    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))

Sort this one as well.

> +
>      e.data.setVar("MLPREFIX", "nativesdk-")
>      e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-
> nativesdk", "").replace("nativesdk-", ""))
>      e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) +
> ":virtclass-nativesdk")
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 5e98d45..dca6bd4 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -789,6 +789,15 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
>  EXTRA_IMAGE_FEATURES ??= ""
>  IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
> 
> +# Native distro features (will always be used for -native, even if
> they
> +# are not enabled for target)
> +DISTRO_FEATURES_NATIVE ?= "x11"
> +DISTRO_FEATURES_NATIVESDK ?= "x11"
> +# Normally target distro features will not be applied to native
> builds:
> +# Native distro features on this list will use the target feature
> value
> +DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation"
> +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation"
> +
>  DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-
> introspection-data ldconfig"
>  MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
> 
> --
> 2.1.4

//Peter

Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES

[Richard Purdie]

On Tue, 2017-04-11 at 15:32 +0000, Peter Kjellerstedt wrote:
> > 
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of
> > Jussi Kukkonen
> > Sent: den 11 april 2017 16:36
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixed
> > DISTRO_FEATURES
> > 
> > There seems to be little advantage to letting distro features
> > affect
> > native builds. There is a significant disadvantage: a change to
> > DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In
> > a
> > test like this:
> >   $ bitbake core-image-minimal
> >   # append " systemd" to DISTRO_FEATURES
> >   $ bitbake core-image-minimal
> > The latter build takes 44 minutes (28%) of cpu-time less with this
> > patch (skipping 135 native tasks). Sadly wall clock time was not
> > affected as glibc remains the bottleneck.
> > 
> > Set native distro features to DISTRO_FEATURES_NATIVE appended with
> > an intersection of DISTRO_FEATURES and
> > DISTRO_FEATURES_FILTER_NATIVE.
> > Current default values (baitbake.conf) are
> > * DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" (as gtk-doc-
> > native
> > has much less dependencies when built without it)
> > * DISTRO_FEATURES_NATIVE ?= "x11" (to enable native UIs even if
> > target
> > does not containe them)
> > 
> > Do the variable setting in native_virtclass_handler() because
> > otherwise
> > it could still be overridden by appends and the feature
> > backfilling.
> > Shuffle the early returns so DISTRO_FEATURES gets set as long as
> > the packagename ends with "-native".
> > 
> > Add similar variables for nativesdk.
> > 
> > Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
> > ---
> >  meta/classes/native.bbclass    | 14 ++++++++++----
> >  meta/classes/nativesdk.bbclass |  6 ++++++
> >  meta/conf/bitbake.conf         |  9 +++++++++
> >  3 files changed, 25 insertions(+), 4 deletions(-)
> > 
> > diff --git a/meta/classes/native.bbclass
> > b/meta/classes/native.bbclass
> > index 1919fbc..aec1087 100644
> > --- a/meta/classes/native.bbclass
> > +++ b/meta/classes/native.bbclass
> > @@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-
> > intercept:"
> >  SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
> > 
> >  python native_virtclass_handler () {
> > -    classextend = e.data.getVar('BBCLASSEXTEND') or ""
> > -    if "native" not in classextend:
> > -        return
> > -
> >      pn = e.data.getVar("PN")
> >      if not pn.endswith("-native"):
> >          return
> > 
> > +    # Set features here to prevent appends and distro features
> > backfill
> > +    # from modifying native distro features
> > +    features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
> > +    filtered = set(bb.utils.filter("DISTRO_FEATURES",
> > d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
> > +    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
> You should sort the list of features to make it deterministic.

Do we sort DISTRO_FEATURES anywhere else?

I thought we only accessed DISTRO_FEATURES with functions which have
support in bitbake (contains/filter) which means that should be
unnecessary?

Cheers,

Richard

Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: den 11 april 2017 19:16
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Jussi Kukkonen
> <jussi.kukkonen@intel.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixed
> DISTRO_FEATURES
> 
> On Tue, 2017-04-11 at 15:32 +0000, Peter Kjellerstedt wrote:
> > >
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org
> > > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > > Of
> > > Jussi Kukkonen
> > > Sent: den 11 april 2017 16:36
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixed
> > > DISTRO_FEATURES
> > >
> > > There seems to be little advantage to letting distro features
> > > affect
> > > native builds. There is a significant disadvantage: a change to
> > > DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In
> > > a
> > > test like this:
> > >   $ bitbake core-image-minimal
> > >   # append " systemd" to DISTRO_FEATURES
> > >   $ bitbake core-image-minimal
> > > The latter build takes 44 minutes (28%) of cpu-time less with this
> > > patch (skipping 135 native tasks). Sadly wall clock time was not
> > > affected as glibc remains the bottleneck.
> > >
> > > Set native distro features to DISTRO_FEATURES_NATIVE appended with
> > > an intersection of DISTRO_FEATURES and
> > > DISTRO_FEATURES_FILTER_NATIVE.
> > > Current default values (baitbake.conf) are
> > > * DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" (as gtk-doc-
> > > native
> > > has much less dependencies when built without it)
> > > * DISTRO_FEATURES_NATIVE ?= "x11" (to enable native UIs even if
> > > target
> > > does not containe them)
> > >
> > > Do the variable setting in native_virtclass_handler() because
> > > otherwise
> > > it could still be overridden by appends and the feature
> > > backfilling.
> > > Shuffle the early returns so DISTRO_FEATURES gets set as long as
> > > the packagename ends with "-native".
> > >
> > > Add similar variables for nativesdk.
> > >
> > > Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
> > > ---
> > >  meta/classes/native.bbclass    | 14 ++++++++++----
> > >  meta/classes/nativesdk.bbclass |  6 ++++++
> > >  meta/conf/bitbake.conf         |  9 +++++++++
> > >  3 files changed, 25 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/meta/classes/native.bbclass
> > > b/meta/classes/native.bbclass
> > > index 1919fbc..aec1087 100644
> > > --- a/meta/classes/native.bbclass
> > > +++ b/meta/classes/native.bbclass
> > > @@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-
> > > intercept:"
> > >  SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
> > >
> > >  python native_virtclass_handler () {
> > > -    classextend = e.data.getVar('BBCLASSEXTEND') or ""
> > > -    if "native" not in classextend:
> > > -        return
> > > -
> > >      pn = e.data.getVar("PN")
> > >      if not pn.endswith("-native"):
> > >          return
> > >
> > > +    # Set features here to prevent appends and distro features
> > > backfill
> > > +    # from modifying native distro features
> > > +    features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
> > > +    filtered = set(bb.utils.filter("DISTRO_FEATURES",
> > > d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
> > > +    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
> > You should sort the list of features to make it deterministic.
> 
> Do we sort DISTRO_FEATURES anywhere else?

No, but it is never worked upon via set() as these are here...

> I thought we only accessed DISTRO_FEATURES with functions which have
> support in bitbake (contains/filter) which means that should be
> unnecessary?

Will the lack of sorting not affect the task hashes (and bitbake -e) if 
these are set via set() operations, which may result in the order of 
the features listed in the final DISTRO_VARIABLE from varying based on 
how Python happens to pull the features out of the set()?

> Cheers,
> 
> Richard

//Peter

Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES

[Richard Purdie]

On Tue, 2017-04-11 at 18:06 +0000, Peter Kjellerstedt wrote:
> > 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Se
> > On Tue, 2017-04-11 at 15:32 +0000, Peter Kjellerstedt wrote:
> > > 
> > > > +    # Set features here to prevent appends and distro features
> > > > backfill
> > > > +    # from modifying native distro features
> > > > +    features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
> > > > +    filtered = set(bb.utils.filter("DISTRO_FEATURES",
> > > > d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
> > > > +    d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
> > > You should sort the list of features to make it deterministic.
> > Do we sort DISTRO_FEATURES anywhere else?
>
> No, but it is never worked upon via set() as these are here...
> 
> > 
> > I thought we only accessed DISTRO_FEATURES with functions which
> > have
> > support in bitbake (contains/filter) which means that should be
> > unnecessary?
>
> Will the lack of sorting not affect the task hashes (and bitbake -e)
> if these are set via set() operations, which may result in the order
> of the features listed in the final DISTRO_VARIABLE from varying
> based on how Python happens to pull the features out of the set()?

setVar operations don't actually affect task hashes directly. We track
functions which use variables through getVar/getVarFlag and only if
something uses it does its value affect the hash.

We can therefore safely setVar this, its only users of
getVar/getVarFlag which would see its value and with DISTRO_FEATURES,
we only obtain the value through contains/filter which have special
handling in bitbake (similar to getVar).

So I still think sorting is unnecessary, at least in theory.

Cheers,

Richard