[PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Mike Crowe]

Give recipes and classes the ability to opt out of EXTRA_OEMAKE
containing the legacy value without removing other recipe-specific or
local additions.

The default value of EXTRA_OEMAKE="-e MAKEFLAGS=" is unfortunate. It
causes breakage and unintended behaviour in various recipes.

It is particularly toxic when variables are passed on the command line
to make since they then don't survive calling into a submake.

In particular this breaks building modules when LD=gold since the
overridden LD=${KERNEL_LD} is lost by the time the kernel's Makefile is
invoked.

This solution isn't pretty either, but it may be a good small first step
towards a future utopia where EXTRA_OEMAKE needn't contain "-e
MAKEFLAGS=" by default at all.

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Acked-by: Phil Blundell <pb@pbcl.net>
---
 meta/classes/module.bbclass | 4 ++++
 meta/conf/bitbake.conf      | 3 ++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/module.bbclass b/meta/classes/module.bbclass
index 0952c0c..4913aac 100644
--- a/meta/classes/module.bbclass
+++ b/meta/classes/module.bbclass
@@ -4,6 +4,10 @@ addtask make_scripts after do_patch before do_compile
 do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
 do_make_scripts[depends] += "virtual/kernel:do_shared_workdir"
 
+# -e MAKEFLAGS= is toxic when building modules since it will cause
+# none of the variables passed to make on the command line to make it
+# through to submakes.
+EXTRA_OEMAKE_LEGACY = ""
 EXTRA_OEMAKE += "KERNEL_SRC=${STAGING_KERNEL_DIR}"
 
 module_do_compile() {
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 06971da..93ab02a 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -477,7 +477,8 @@ export BUILD_STRIP = "${BUILD_PREFIX}strip"
 export BUILD_NM = "${BUILD_PREFIX}nm"
 
 export MAKE = "make"
-EXTRA_OEMAKE = "-e MAKEFLAGS="
+EXTRA_OEMAKE_LEGACY = "-e MAKEFLAGS="
+EXTRA_OEMAKE = "${EXTRA_OEMAKE_LEGACY}"
 EXTRA_OECONF = ""
 export LC_ALL = "C"
 
-- 
2.1.4

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]


> On Nov 5, 2015, at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> 
> export MAKE = "make"
> -EXTRA_OEMAKE = "-e MAKEFLAGS="
> +EXTRA_OEMAKE_LEGACY = "-e MAKEFLAGS="
> +EXTRA_OEMAKE = "${EXTRA_OEMAKE_LEGACY}"

may be this should be a weak assignment.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Thu, Nov 5, 2015 at 9:23 AM, Khem Raj <raj.khem@gmail.com> wrote:

> > On Nov 5, 2015, at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> >
> > export MAKE = "make"
> > -EXTRA_OEMAKE = "-e MAKEFLAGS="
> > +EXTRA_OEMAKE_LEGACY = "-e MAKEFLAGS="
> > +EXTRA_OEMAKE = "${EXTRA_OEMAKE_LEGACY}"
>
> may be this should be a weak assignment.


That's the habit, but this is bitbake.conf. Anything a recipe does will be
long after this.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1076 bytes --]

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Khem Raj]

On Thu, Nov 5, 2015 at 8:27 AM, Christopher Larson <clarson@kergoth.com> wrote:
>>
>> >
>> > export MAKE = "make"
>> > -EXTRA_OEMAKE = "-e MAKEFLAGS="
>> > +EXTRA_OEMAKE_LEGACY = "-e MAKEFLAGS="
>> > +EXTRA_OEMAKE = "${EXTRA_OEMAKE_LEGACY}"
>>
>> may be this should be a weak assignment.
>
>
> That's the habit, but this is bitbake.conf. Anything a recipe does will be
> long after this.

True, however ?= will convey that its overridable.

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Andre McCurdy]

On Thu, Nov 5, 2015 at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> Give recipes and classes the ability to opt out of EXTRA_OEMAKE
> containing the legacy value without removing other recipe-specific or
> local additions.

Isn't this possible already from within a recipe or class by using

  EXTRA_OEMAKE = ...

instead of

  EXTRA_OEMAKE += ...

ie what autotools.bbclass, kernel.bbclass and many recipes do already.

For the specific case of module.bbclass, changing the EXTRA_OEMAKE
assignment to '=' might require some recipes to be tweaked to so that
they "inherit module" before adding their own options to EXTRA_OEMAKE,
but it seems like a cleaner solution?


> The default value of EXTRA_OEMAKE="-e MAKEFLAGS=" is unfortunate. It
> causes breakage and unintended behaviour in various recipes.
>
> It is particularly toxic when variables are passed on the command line
> to make since they then don't survive calling into a submake.
>
> In particular this breaks building modules when LD=gold since the
> overridden LD=${KERNEL_LD} is lost by the time the kernel's Makefile is
> invoked.
>
> This solution isn't pretty either, but it may be a good small first step
> towards a future utopia where EXTRA_OEMAKE needn't contain "-e
> MAKEFLAGS=" by default at all.
>
> Signed-off-by: Mike Crowe <mac@mcrowe.com>
> Acked-by: Phil Blundell <pb@pbcl.net>
> ---
>  meta/classes/module.bbclass | 4 ++++
>  meta/conf/bitbake.conf      | 3 ++-
>  2 files changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/module.bbclass b/meta/classes/module.bbclass
> index 0952c0c..4913aac 100644
> --- a/meta/classes/module.bbclass
> +++ b/meta/classes/module.bbclass
> @@ -4,6 +4,10 @@ addtask make_scripts after do_patch before do_compile
>  do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
>  do_make_scripts[depends] += "virtual/kernel:do_shared_workdir"
>
> +# -e MAKEFLAGS= is toxic when building modules since it will cause
> +# none of the variables passed to make on the command line to make it
> +# through to submakes.
> +EXTRA_OEMAKE_LEGACY = ""
>  EXTRA_OEMAKE += "KERNEL_SRC=${STAGING_KERNEL_DIR}"
>
>  module_do_compile() {
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 06971da..93ab02a 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -477,7 +477,8 @@ export BUILD_STRIP = "${BUILD_PREFIX}strip"
>  export BUILD_NM = "${BUILD_PREFIX}nm"
>
>  export MAKE = "make"
> -EXTRA_OEMAKE = "-e MAKEFLAGS="
> +EXTRA_OEMAKE_LEGACY = "-e MAKEFLAGS="
> +EXTRA_OEMAKE = "${EXTRA_OEMAKE_LEGACY}"
>  EXTRA_OECONF = ""
>  export LC_ALL = "C"
>
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Mike Crowe]

On Friday 06 November 2015 at 01:16:46 -0800, Andre McCurdy wrote:
> On Thu, Nov 5, 2015 at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> > Give recipes and classes the ability to opt out of EXTRA_OEMAKE
> > containing the legacy value without removing other recipe-specific or
> > local additions.
> 
> Isn't this possible already from within a recipe or class by using
> 
>   EXTRA_OEMAKE = ...
> 
> instead of
> 
>   EXTRA_OEMAKE += ...
> 
> ie what autotools.bbclass, kernel.bbclass and many recipes do already.
> 
> For the specific case of module.bbclass, changing the EXTRA_OEMAKE
> assignment to '=' might require some recipes to be tweaked to so that
> they "inherit module" before adding their own options to EXTRA_OEMAKE,
> but it seems like a cleaner solution?

It would be, but I was afraid of what I might break. I suspect that there
are many unseen third-party and local recipes that inherit module.bbclass.

It would be great to get to the point that EXTRA_OEMAKE is empty by default
but I imagine that the experts are already aware of the difficulties with
doing this which is why the current value has lasted so long.

Mike.

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

On Fri, Nov 06, 2015 at 10:30:04AM +0000, Mike Crowe wrote:
> On Friday 06 November 2015 at 01:16:46 -0800, Andre McCurdy wrote:
> > On Thu, Nov 5, 2015 at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> > > Give recipes and classes the ability to opt out of EXTRA_OEMAKE
> > > containing the legacy value without removing other recipe-specific or
> > > local additions.
> > 
> > Isn't this possible already from within a recipe or class by using
> > 
> >   EXTRA_OEMAKE = ...
> > 
> > instead of
> > 
> >   EXTRA_OEMAKE += ...
> > 
> > ie what autotools.bbclass, kernel.bbclass and many recipes do already.
> > 
> > For the specific case of module.bbclass, changing the EXTRA_OEMAKE
> > assignment to '=' might require some recipes to be tweaked to so that
> > they "inherit module" before adding their own options to EXTRA_OEMAKE,
> > but it seems like a cleaner solution?
> 
> It would be, but I was afraid of what I might break. I suspect that there
> are many unseen third-party and local recipes that inherit module.bbclass.
> 
> It would be great to get to the point that EXTRA_OEMAKE is empty by default
> but I imagine that the experts are already aware of the difficulties with
> doing this which is why the current value has lasted so long.

Is it really good goal to get rid of "-e"?

I know that the environment used in bitbake tasks is already well
defined and controlled, but I still find a bit more control with -e to
be useful in many cases.

I know I'll be able to return -e where useful, but what's the main
advantage of removing it from default?

Regards,

> 
> Mike.
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2669 bytes --]

On Fri, Nov 6, 2015 at 6:18 AM, Martin Jansa <martin.jansa@gmail.com> wrote:

> On Fri, Nov 06, 2015 at 10:30:04AM +0000, Mike Crowe wrote:
> > On Friday 06 November 2015 at 01:16:46 -0800, Andre McCurdy wrote:
> > > On Thu, Nov 5, 2015 at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> > > > Give recipes and classes the ability to opt out of EXTRA_OEMAKE
> > > > containing the legacy value without removing other recipe-specific or
> > > > local additions.
> > >
> > > Isn't this possible already from within a recipe or class by using
> > >
> > >   EXTRA_OEMAKE = ...
> > >
> > > instead of
> > >
> > >   EXTRA_OEMAKE += ...
> > >
> > > ie what autotools.bbclass, kernel.bbclass and many recipes do already.
> > >
> > > For the specific case of module.bbclass, changing the EXTRA_OEMAKE
> > > assignment to '=' might require some recipes to be tweaked to so that
> > > they "inherit module" before adding their own options to EXTRA_OEMAKE,
> > > but it seems like a cleaner solution?
> >
> > It would be, but I was afraid of what I might break. I suspect that there
> > are many unseen third-party and local recipes that inherit
> module.bbclass.
> >
> > It would be great to get to the point that EXTRA_OEMAKE is empty by
> default
> > but I imagine that the experts are already aware of the difficulties with
> > doing this which is why the current value has lasted so long.
>
> Is it really good goal to get rid of "-e"?
>
> I know that the environment used in bitbake tasks is already well
> defined and controlled, but I still find a bit more control with -e to
> be useful in many cases.
>
> I know I'll be able to return -e where useful, but what's the main
> advantage of removing it from default?


The original goal of the default EXTRA_OEMAKE was to let us keep our
recipes as minimal as possible and have as many "Just Work" out of the box
as possible. It succeeded in this goal. The problem is the corner cases,
and more importantly, it encourages people creating recipes for custom
make-based buildsystems to just try building it and hack at it till it
works, rather than reading the makefiles, determining which variables to
pass in, in what form, and customizing EXTRA_OEMAKE to explicitly pass
what's needed in the appropriate ways.

That's my biggest concern with it, other than the aforementioned occasional
breakage. It's implicit, automatic, rather than explicit, and tacitly
encourages ignorance of the buildsystem in question.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3404 bytes --]

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3094 bytes --]

On Fri, Nov 06, 2015 at 07:59:32AM -0700, Christopher Larson wrote:
> On Fri, Nov 6, 2015 at 6:18 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> 
> > On Fri, Nov 06, 2015 at 10:30:04AM +0000, Mike Crowe wrote:
> > > On Friday 06 November 2015 at 01:16:46 -0800, Andre McCurdy wrote:
> > > > On Thu, Nov 5, 2015 at 6:47 AM, Mike Crowe <mac@mcrowe.com> wrote:
> > > > > Give recipes and classes the ability to opt out of EXTRA_OEMAKE
> > > > > containing the legacy value without removing other recipe-specific or
> > > > > local additions.
> > > >
> > > > Isn't this possible already from within a recipe or class by using
> > > >
> > > >   EXTRA_OEMAKE = ...
> > > >
> > > > instead of
> > > >
> > > >   EXTRA_OEMAKE += ...
> > > >
> > > > ie what autotools.bbclass, kernel.bbclass and many recipes do already.
> > > >
> > > > For the specific case of module.bbclass, changing the EXTRA_OEMAKE
> > > > assignment to '=' might require some recipes to be tweaked to so that
> > > > they "inherit module" before adding their own options to EXTRA_OEMAKE,
> > > > but it seems like a cleaner solution?
> > >
> > > It would be, but I was afraid of what I might break. I suspect that there
> > > are many unseen third-party and local recipes that inherit
> > module.bbclass.
> > >
> > > It would be great to get to the point that EXTRA_OEMAKE is empty by
> > default
> > > but I imagine that the experts are already aware of the difficulties with
> > > doing this which is why the current value has lasted so long.
> >
> > Is it really good goal to get rid of "-e"?
> >
> > I know that the environment used in bitbake tasks is already well
> > defined and controlled, but I still find a bit more control with -e to
> > be useful in many cases.
> >
> > I know I'll be able to return -e where useful, but what's the main
> > advantage of removing it from default?
> 
> 
> The original goal of the default EXTRA_OEMAKE was to let us keep our
> recipes as minimal as possible and have as many "Just Work" out of the box
> as possible. It succeeded in this goal. The problem is the corner cases,
> and more importantly, it encourages people creating recipes for custom
> make-based buildsystems to just try building it and hack at it till it
> works, rather than reading the makefiles, determining which variables to
> pass in, in what form, and customizing EXTRA_OEMAKE to explicitly pass
> what's needed in the appropriate ways.
> 
> That's my biggest concern with it, other than the aforementioned occasional
> breakage. It's implicit, automatic, rather than explicit, and tacitly
> encourages ignorance of the buildsystem in question.

I'm sorry I was reading it backwards (I should never reply on e-mails
before first morning coffee).

Removing -e from default value is good goal and I like it :)

e.g. in qmake5_base.bbclass it saved me a lot of headaches with generated
Makefiles reading variables from env which were supposed to be set
correctly by qmake.

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Mike Crowe]

On Friday 06 November 2015 at 10:30:04 +0000, Mike Crowe wrote:
> It would be great to get to the point that EXTRA_OEMAKE is empty by default
> but I imagine that the experts are already aware of the difficulties with
> doing this which is why the current value has lasted so long.

I had a go at this. Fewer recipes than I expected seem to be affected.

With bitbake.conf containing EXTRA_OEMAKE = "", the following recipes
needed to be changed to explicitly set EXTRA_OEMAKE = "-e MAKEFLAGS=":

 meta/recipes-bsp/pciutils/pciutils_3.3.1.bb
 meta/recipes-connectivity/openssl/openssl.inc
 meta/recipes-core/busybox/busybox.inc
 meta/recipes-devtools/perl/perl_5.22.0.bb

for core-image-minimal to build successfully[1].

I tried core-image-sato but got stuck when apr-util-native wouldn't compile
for me[2], even with stock master.

So, perhaps the situation isn't quite as bad as I feared and it might be
possible to change the default value of EXTRA_OEMAKE.

Mike.

[1] I suspect that perl isn't required by core-image-minimal. That list is
    my complete set of changes from trying to get core-image-sato to build.

[2] x86_64-linux-libtool: compile: unable to infer tagged configuration

Re: [PATCH] bitbake.conf, module.bbclass: Support opting out of legacy EXTRA_OEMAKE

[Andre McCurdy]

On Thu, Nov 12, 2015 at 4:10 AM, Mike Crowe <mac@mcrowe.com> wrote:
> On Friday 06 November 2015 at 10:30:04 +0000, Mike Crowe wrote:
>> It would be great to get to the point that EXTRA_OEMAKE is empty by default
>> but I imagine that the experts are already aware of the difficulties with
>> doing this which is why the current value has lasted so long.
>
> I had a go at this. Fewer recipes than I expected seem to be affected.
>
> With bitbake.conf containing EXTRA_OEMAKE = "", the following recipes
> needed to be changed to explicitly set EXTRA_OEMAKE = "-e MAKEFLAGS=":
>
>  meta/recipes-bsp/pciutils/pciutils_3.3.1.bb
>  meta/recipes-connectivity/openssl/openssl.inc
>  meta/recipes-core/busybox/busybox.inc

Busybox doesn't need make -e if you have my busybox cleanup patches
(posted to the list recently, but not merged to master or mut yet).

>  meta/recipes-devtools/perl/perl_5.22.0.bb
>
> for core-image-minimal to build successfully[1].
>
> I tried core-image-sato but got stuck when apr-util-native wouldn't compile
> for me[2], even with stock master.
>
> So, perhaps the situation isn't quite as bad as I feared and it might be
> possible to change the default value of EXTRA_OEMAKE.
>
> Mike.
>
> [1] I suspect that perl isn't required by core-image-minimal. That list is
>     my complete set of changes from trying to get core-image-sato to build.
>
> [2] x86_64-linux-libtool: compile: unable to infer tagged configuration
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core