Question about OVERRIDES precedence

Question about OVERRIDES precedence

[Maupin, Chase]

All,

I have been looking into the OVERRIDES setting and it appears to me that the OVERRIDES are being handled in least specific to most specific order.  Meaning that if I have an override for a particular machine set that it has a lower priority than the override for a distro.

My OVERRIDES variable is actually set to evaluate:
# OVERRIDES=local:${MACHINE}:${DISTRO}:${TARGET_OS}:${TARGET_ARCH}:build-${BUILD_OS}:fail-fast:pn-${PN}:${FEED_ARCH}:${SOC_FAMILY}:libc-glibc${OVERRIDE_THUMB}${OVERRIDE_INTERWORK}

Which translates in my build to:
OVERRIDES="local:am37x-evm:angstrom:linux-gnueabi:arm:build-linux:fail-fast:pn-task-arago-toolchain-tisdk-multimedia-host:armv7a:omap3:libc-glibc:thumb-interwork"

I would expect that if I set an override for am37x-evm that this MACHINE specific override should trump the DISTRO specific override or the SOC_FAMILY override.

Likewise I would expect that the local override would have the highest precedence because I am trying to override something specifically for my local copy and that the "thumb-interwork" override would have the lowest precedence.

Is my expectation incorrect?  

If so then can we just reverse the list of overrides in the update_data method of <bitbake>/lib/bb/data.py, around line 500 in my version.  My bitbake is on version 1.10.  In the master this would be in the finalize function of the DataSmart class.

Thanks in advance for you input.  

Sincerely,
Chase Maupin
Software Applications
Catalog DSP Products
e-mail: chase.maupin@ti.com
phone: (281) 274-3285

For support:
Forums - http://community.ti.com/forums/
Wiki - http://wiki.davincidsp.com/

Re: Question about OVERRIDES precedence

[Chris Larson]

On Wed, Oct 13, 2010 at 1:29 PM, Maupin, Chase <chase.maupin@ti.com> wrote:

> I have been looking into the OVERRIDES setting and it appears to me that
> the OVERRIDES are being handled in least specific to most specific order.
>  Meaning that if I have an override for a particular machine set that it has
> a lower priority than the override for a distro.
>
> My OVERRIDES variable is actually set to evaluate:
> #
> OVERRIDES=local:${MACHINE}:${DISTRO}:${TARGET_OS}:${TARGET_ARCH}:build-${BUILD_OS}:fail-fast:pn-${PN}:${FEED_ARCH}:${SOC_FAMILY}:libc-glibc${OVERRIDE_THUMB}${OVERRIDE_INTERWORK}
>
> Which translates in my build to:
>
> OVERRIDES="local:am37x-evm:angstrom:linux-gnueabi:arm:build-linux:fail-fast:pn-task-arago-toolchain-tisdk-multimedia-host:armv7a:omap3:libc-glibc:thumb-interwork"
>
> I would expect that if I set an override for am37x-evm that this MACHINE
> specific override should trump the DISTRO specific override or the
> SOC_FAMILY override.
>
> Likewise I would expect that the local override would have the highest
> precedence because I am trying to override something specifically for my
> local copy and that the "thumb-interwork" override would have the lowest
> precedence.
>
> Is my expectation incorrect?
>

Yes, your expectation is correct.  OVERRIDES is intended to essentially give
us a layering mechanism for the metadata, ensuring that the most specific
available information is always used.  So machine should always have higher
priority than target architecture, for example, and local is highest of all.

I assume in your testing you've found that this is not behaving the way it
should?

 If so then can we just reverse the list of overrides in the update_data
> method of <bitbake>/lib/bb/data.py, around line 500 in my version.  My
> bitbake is on version 1.10.  In the master this would be in the finalize
> function of the DataSmart class.
>

I think you're correct, yes, from a quick re-read of the code -- it seems to
be evaluating them from highest to lowest priority, so the latter may
override the former, unless I'm missing something.  This seems like it would
have been an obvious thing -- if overrides aren't working as intended, I'd
have expected it to have been caught before now :\  This code hasn't changed
since the original implementation, it seems -- see
d8b28d544cd7ba8e9c303e72f8ce2ed04ee71b3a.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Question about OVERRIDES precedence

[Maupin, Chase]

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Chris Larson
> Sent: Wednesday, October 13, 2010 4:22 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Question about OVERRIDES precedence
> 
> On Wed, Oct 13, 2010 at 1:29 PM, Maupin, Chase <chase.maupin@ti.com>
> wrote:
> 
> > I have been looking into the OVERRIDES setting and it appears to me that
> > the OVERRIDES are being handled in least specific to most specific order.
> >  Meaning that if I have an override for a particular machine set that it
> has
> > a lower priority than the override for a distro.
> >
> > My OVERRIDES variable is actually set to evaluate:
> > #
> > OVERRIDES=local:${MACHINE}:${DISTRO}:${TARGET_OS}:${TARGET_ARCH}:build-
> ${BUILD_OS}:fail-fast:pn-${PN}:${FEED_ARCH}:${SOC_FAMILY}:libc-
> glibc${OVERRIDE_THUMB}${OVERRIDE_INTERWORK}
> >
> > Which translates in my build to:
> >
> > OVERRIDES="local:am37x-evm:angstrom:linux-gnueabi:arm:build-linux:fail-
> fast:pn-task-arago-toolchain-tisdk-multimedia-host:armv7a:omap3:libc-
> glibc:thumb-interwork"
> >
> > I would expect that if I set an override for am37x-evm that this MACHINE
> > specific override should trump the DISTRO specific override or the
> > SOC_FAMILY override.
> >
> > Likewise I would expect that the local override would have the highest
> > precedence because I am trying to override something specifically for my
> > local copy and that the "thumb-interwork" override would have the lowest
> > precedence.
> >
> > Is my expectation incorrect?
> >
> 
> Yes, your expectation is correct.  OVERRIDES is intended to essentially
> give
> us a layering mechanism for the metadata, ensuring that the most specific
> available information is always used.  So machine should always have
> higher
> priority than target architecture, for example, and local is highest of
> all.
> 
> I assume in your testing you've found that this is not behaving the way it
> should?
> 
>  If so then can we just reverse the list of overrides in the update_data
> > method of <bitbake>/lib/bb/data.py, around line 500 in my version.  My
> > bitbake is on version 1.10.  In the master this would be in the finalize
> > function of the DataSmart class.
> >
> 
> I think you're correct, yes, from a quick re-read of the code -- it seems
> to
> be evaluating them from highest to lowest priority, so the latter may
> override the former, unless I'm missing something.  This seems like it
> would
> have been an obvious thing -- if overrides aren't working as intended, I'd
> have expected it to have been caught before now :\  This code hasn't
> changed
> since the original implementation, it seems -- see
> d8b28d544cd7ba8e9c303e72f8ce2ed04ee71b3a.

Chris,

That was my thought as well but I can consistently cause this issue.  I have tested a quick change that seems to fix the issue by reversing the list before it is used in the for loop.  My change was:

diff --git a/lib/bb/data.py b/lib/bb/data.py
index f424ac7..b3bd8b9 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -480,7 +480,7 @@ def update_data(d):
     #print "Append/Prepend %s" % d._special_values
     #print "Overrides      %s" % d._seen_overrides
 
-    overrides = (getVar('OVERRIDES', d, 1) or "").split(':') or []
+    overrides = (getVar('OVERRIDES', d, 1) or "").split(':')[::-1] or []
 
     #
     # Well let us see what breaks here. We used to iterate

If you agree with this change I'll submit a patch.  Should I submit it to this list or another one?

> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: Question about OVERRIDES precedence

[Chris Larson]

On Wed, Oct 13, 2010 at 2:30 PM, Maupin, Chase <chase.maupin@ti.com> wrote:

> That was my thought as well but I can consistently cause this issue.  I
> have tested a quick change that seems to fix the issue by reversing the list
> before it is used in the for loop.  My change was:


From discussion on IRC, we think we should just reverse the definition of
OVERRIDES in OE, this avoids compatibility issues associated with changing
the implementation.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Question about OVERRIDES precedence

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13-10-10 23:38, Chris Larson wrote:
> On Wed, Oct 13, 2010 at 2:30 PM, Maupin, Chase <chase.maupin@ti.com> wrote:
> 
>> That was my thought as well but I can consistently cause this issue.  I
>> have tested a quick change that seems to fix the issue by reversing the list
>> before it is used in the for loop.  My change was:
> 
> 
> From discussion on IRC, we think we should just reverse the definition of
> OVERRIDES in OE, this avoids compatibility issues associated with changing
> the implementation.

Isn't the definition covered in the bitbake docs? I'm in favour of
fixing bitbake instead working around it in OE.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMtwRCMkyGM64RGpERAhyDAJ0ZKsVJEzs2Fo8iXR7F32ijePz+lwCeMuHT
vfpaqzI6gh8aKRBwE5h1WA4=
=9m/w
-----END PGP SIGNATURE-----

Re: Question about OVERRIDES precedence

[Chris Larson]

On Thu, Oct 14, 2010 at 6:23 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 13-10-10 23:38, Chris Larson wrote:
> > On Wed, Oct 13, 2010 at 2:30 PM, Maupin, Chase <chase.maupin@ti.com>
> wrote:
> >
> >> That was my thought as well but I can consistently cause this issue.  I
> >> have tested a quick change that seems to fix the issue by reversing the
> list
> >> before it is used in the for loop.  My change was:
> >
> >
> > From discussion on IRC, we think we should just reverse the definition of
> > OVERRIDES in OE, this avoids compatibility issues associated with
> changing
> > the implementation.
>
> Isn't the definition covered in the bitbake docs? I'm in favour of
> fixing bitbake instead working around it in OE.
>

As far as I can tell:
1. The bitbake docs don't cover the ordering.
2. The behavior isn't a bug, it's a mismatch between OE's definition and
bitbake's implementation, and bitbake has been this way since 2003.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Question about OVERRIDES precedence

[Phil Blundell]

On Thu, 2010-10-14 at 06:59 -0700, Chris Larson wrote:
> On Thu, Oct 14, 2010 at 6:23 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:
> > Isn't the definition covered in the bitbake docs? I'm in favour of
> > fixing bitbake instead working around it in OE.
> >
> 
> As far as I can tell:
> 1. The bitbake docs don't cover the ordering.
> 2. The behavior isn't a bug, it's a mismatch between OE's definition and
> bitbake's implementation, and bitbake has been this way since 2003.

Agreed.  And there is plenty of existing code in OE which depends on the
current ordering.  OE's bitbake.conf even has a big comment describing
the behaviour that it expects:

# Overrides are processed left to right, so the ones that are named later take precedence.
# You generally want them to go from least to most specific.
# 
# This means that an envionment variable named '<foo>_arm' overrides an
# environment variable '<foo>' (when ${TARGET_ARCH} is arm).
# an environment variable '<foo>_ramses' overrides '<foo>' but doesn't override
# '<foo>_arm' when ${MACHINE} is 'ramses'. 
# If you use combination ie '<foo>_arm_ramses', then '<foo>_arm_ramses' will override 
# '<foo>_arm' and then '<foo>' will be overriden with that value from '<foo>_arm'.
# And finally '<foo>_local' overrides anything, but with lowest priority.

So, changing the bitbake behaviour at this late stage seems like a very
bad idea.  If the documentation calls for something else then I think
the documentation should be fixed to match the reality, not vice versa.

p.

Re: Question about OVERRIDES precedence

[Maupin, Chase]

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Chris Larson
> Sent: Wednesday, October 13, 2010 4:39 PM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Question about OVERRIDES precedence
> 
> On Wed, Oct 13, 2010 at 2:30 PM, Maupin, Chase <chase.maupin@ti.com>
> wrote:
> 
> > That was my thought as well but I can consistently cause this issue.  I
> > have tested a quick change that seems to fix the issue by reversing the
> list
> > before it is used in the for loop.  My change was:
> 
> 
> From discussion on IRC, we think we should just reverse the definition of
> OVERRIDES in OE, this avoids compatibility issues associated with changing
> the implementation.

Chris,

So are you saying that we are just going to change the order of how OVERRIDES is appended to in OE like Denys was suggesting?  That would still lead to issues like local not being the highest priority unless we would also change bitbake.conf to define OVERRIDES as "${TARGET_ARCH}:${TARGET_OS}:${MACHINE}:local".  That way we could prepend OVERRIDES in OE and still have local be the last one evaluated.

Or are you saying that we should build the whole OVERRIDES list and then reverse it in the OE configuration files?

If that is the case I tried adding a line like:

REVERSE_OVERRIDES=${@d.setVar("OVERRIDES", ':'.join(bb.data.getVar('OVERRIDES', d, 1).split(':')[::-1]))}

to the end of my local.conf file.  When I do a bitbake -e to dump my environment I see that OVERRIDES is now defined to:

OVERRIDES="thumb-interwork:libc-glibc:omap3:armv7a:pn-task-arago-toolchain-tisdk-multimedia-host:fail-fast:build-linux:arm:linux-gnueabi:angstrom:am37x-evm:local"

But while that order looks right it seems like maybe that is being interpreted to late because my variables that use overrides are not picking up the right values.  i.e. my MACHINE override is not taking precedence over my SOC_FAMILY override.

Any ideas why this does not work?


> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: Question about OVERRIDES precedence

[Chris Larson]

On Thu, Oct 14, 2010 at 8:29 AM, Maupin, Chase <chase.maupin@ti.com> wrote:

> So are you saying that we are just going to change the order of how
> OVERRIDES is appended to in OE like Denys was suggesting?  That would still
> lead to issues like local not being the highest priority unless we would
> also change bitbake.conf to define OVERRIDES as
> "${TARGET_ARCH}:${TARGET_OS}:${MACHINE}:local".  That way we could prepend
> OVERRIDES in OE and still have local be the last one evaluated.
>

No.  What I'm saying is, the definition of OVERRIDES in bitbake.conf will be
reversed.  Not that difficult to understand.  I don't know how I could
possibly be any clearer here.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Question about OVERRIDES precedence

[Maupin, Chase]

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Chris Larson
> Sent: Thursday, October 14, 2010 10:34 AM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Question about OVERRIDES precedence
> 
> On Thu, Oct 14, 2010 at 8:29 AM, Maupin, Chase <chase.maupin@ti.com>
> wrote:
> 
> > So are you saying that we are just going to change the order of how
> > OVERRIDES is appended to in OE like Denys was suggesting?  That would
> still
> > lead to issues like local not being the highest priority unless we would
> > also change bitbake.conf to define OVERRIDES as
> > "${TARGET_ARCH}:${TARGET_OS}:${MACHINE}:local".  That way we could
> prepend
> > OVERRIDES in OE and still have local be the last one evaluated.
> >
> 
> No.  What I'm saying is, the definition of OVERRIDES in bitbake.conf will
> be
> reversed.  Not that difficult to understand.  I don't know how I could
> possibly be any clearer here.

Great.  I assume this will be on the 1.10 branch as well as the master branch?  Is there anything I can do to help here?

I guess with the reversal of the definition in bitbake.conf we can then change the appends to prepends in OE as well as reversing the definitions there.  That way the OVERRIDES list should be built up in the proper order.


> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: Question about OVERRIDES precedence

[Chris Larson]

On Thu, Oct 14, 2010 at 9:37 AM, Maupin, Chase <chase.maupin@ti.com> wrote:

> > -----Original Message-----
> > From: openembedded-devel-bounces@lists.openembedded.org
> > [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> > Chris Larson
> > Sent: Thursday, October 14, 2010 10:34 AM
> > To: openembedded-devel@lists.openembedded.org
> > Subject: Re: [oe] Question about OVERRIDES precedence
> >
> > On Thu, Oct 14, 2010 at 8:29 AM, Maupin, Chase <chase.maupin@ti.com>
> > wrote:
> >
> > > So are you saying that we are just going to change the order of how
> > > OVERRIDES is appended to in OE like Denys was suggesting?  That would
> > still
> > > lead to issues like local not being the highest priority unless we
> would
> > > also change bitbake.conf to define OVERRIDES as
> > > "${TARGET_ARCH}:${TARGET_OS}:${MACHINE}:local".  That way we could
> > prepend
> > > OVERRIDES in OE and still have local be the last one evaluated.
> > >
> >
> > No.  What I'm saying is, the definition of OVERRIDES in bitbake.conf will
> > be
> > reversed.  Not that difficult to understand.  I don't know how I could
> > possibly be any clearer here.
>
> Great.  I assume this will be on the 1.10 branch as well as the master
> branch?  Is there anything I can do to help here?
>

There is no 1.10 branch in OE, so no.  Again, we're changing it in OE, not
bitbake.

I guess with the reversal of the definition in bitbake.conf we can then
> change the appends to prepends in OE as well as reversing the definitions
> there.  That way the OVERRIDES list should be built up in the proper order.


Yes, indeed, though I suspect that the existing prepend/appends were done by
people not thinking about the order, so they may not be  correct the way
they are now anyway .. :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Question about OVERRIDES precedence

[Maupin, Chase]

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf Of
> Chris Larson
> Sent: Thursday, October 14, 2010 11:42 AM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Question about OVERRIDES precedence
> 
> On Thu, Oct 14, 2010 at 9:37 AM, Maupin, Chase <chase.maupin@ti.com>
> wrote:
> 
> > > -----Original Message-----
> > > From: openembedded-devel-bounces@lists.openembedded.org
> > > [mailto:openembedded-devel-bounces@lists.openembedded.org] On Behalf
> Of
> > > Chris Larson
> > > Sent: Thursday, October 14, 2010 10:34 AM
> > > To: openembedded-devel@lists.openembedded.org
> > > Subject: Re: [oe] Question about OVERRIDES precedence
> > >
> > > On Thu, Oct 14, 2010 at 8:29 AM, Maupin, Chase <chase.maupin@ti.com>
> > > wrote:
> > >
> > > > So are you saying that we are just going to change the order of how
> > > > OVERRIDES is appended to in OE like Denys was suggesting?  That
> would
> > > still
> > > > lead to issues like local not being the highest priority unless we
> > would
> > > > also change bitbake.conf to define OVERRIDES as
> > > > "${TARGET_ARCH}:${TARGET_OS}:${MACHINE}:local".  That way we could
> > > prepend
> > > > OVERRIDES in OE and still have local be the last one evaluated.
> > > >
> > >
> > > No.  What I'm saying is, the definition of OVERRIDES in bitbake.conf
> will
> > > be
> > > reversed.  Not that difficult to understand.  I don't know how I could
> > > possibly be any clearer here.
> >
> > Great.  I assume this will be on the 1.10 branch as well as the master
> > branch?  Is there anything I can do to help here?
> >
> 
> There is no 1.10 branch in OE, so no.  Again, we're changing it in OE, not
> bitbake.

Chris,

I just realized what you were talking about.  I was looking at the bitbake.conf file in the bitbake repository, not the one in the OE repository.  Now what you are saying makes more sense.  Sorry for the confusion.

> 
> I guess with the reversal of the definition in bitbake.conf we can then
> > change the appends to prepends in OE as well as reversing the
> definitions
> > there.  That way the OVERRIDES list should be built up in the proper
> order.
> 
> 
> Yes, indeed, though I suspect that the existing prepend/appends were done
> by
> people not thinking about the order, so they may not be  correct the way
> they are now anyway .. :)
> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel