Does support for external toolchains working in current OE?

Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

For last few months I am working on fixing Linaro binary cross
toolchains and their support in OpenEmbedded. Got to point when they 
work (both AArch64 and ARMv7a ones) for single packages but problem 
starts when I want to create image...

11:14 hrw@puchatek:build$ bitbake core-image-minimal -g
Parsing of 1302 .bb files complete (1294 cached, 8 parsed). 1670 targets, 45 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: multiple providers are available for virtual/gettext (proxy-libintl, gettext)
NOTE: consider defining a PREFERRED_PROVIDER entry to match virtual/gettext
NOTE: Preparing runqueue
ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
 This usually means one provides something the other doesn't and should.
ERROR: Multiple .bb files are due to be built which each provide virtual/arm-linux-gnueabihf-libc-for-gcc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
 This usually means one provides something the other doesn't and should.
ERROR: Multiple .bb files are due to be built which each provide virtual/libiconv (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
 This usually means one provides something the other doesn't and should.
NOTE: PN build list saved to 'pn-buildlist'
NOTE: PN dependencies saved to 'pn-depends.dot'
NOTE: Package dependencies saved to 'package-depends.dot'
NOTE: Task dependencies saved to 'task-depends.dot'

Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

All three virtual/ are provided by external-linaro-toolchain recipe... 

Is there any external toolchain which allows to create image with current
OpenEmbedded layers? 

TCMODE="external-sourcery" gives exactly same problem.

Re: Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

https://bugzilla.yoctoproject.org/show_bug.cgi?id=3393 is about same issue.

Re: Does support for external toolchains working in current OE?

[Richard Purdie]

On Fri, 2013-04-05 at 11:20 +0200, Marcin Juszkiewicz wrote:
> For last few months I am working on fixing Linaro binary cross
> toolchains and their support in OpenEmbedded. Got to point when they 
> work (both AArch64 and ARMv7a ones) for single packages but problem 
> starts when I want to create image...
> 
> 11:14 hrw@puchatek:build$ bitbake core-image-minimal -g
> Parsing of 1302 .bb files complete (1294 cached, 8 parsed). 1670 targets, 45 skipped, 0 masked, 0 errors.
> NOTE: Resolving any missing task queue dependencies
> NOTE: multiple providers are available for virtual/gettext (proxy-libintl, gettext)
> NOTE: consider defining a PREFERRED_PROVIDER entry to match virtual/gettext
> NOTE: Preparing runqueue
> ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
>  This usually means one provides something the other doesn't and should.
> ERROR: Multiple .bb files are due to be built which each provide virtual/arm-linux-gnueabihf-libc-for-gcc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
>  This usually means one provides something the other doesn't and should.
> ERROR: Multiple .bb files are due to be built which each provide virtual/libiconv (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
>  This usually means one provides something the other doesn't and should.
> NOTE: PN build list saved to 'pn-buildlist'
> NOTE: PN dependencies saved to 'pn-depends.dot'
> NOTE: Package dependencies saved to 'package-depends.dot'
> NOTE: Task dependencies saved to 'task-depends.dot'
> 
> Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
> 
> All three virtual/ are provided by external-linaro-toolchain recipe... 
> 
> Is there any external toolchain which allows to create image with current
> OpenEmbedded layers? 
> 
> TCMODE="external-sourcery" gives exactly same problem.

This means that virtual/libc is providing something which the external
toolchain recipes are not. Can you share the "bitbake -DDDv" output for
the above command somewhere (I know there will be a lot of it). Buried
somewhere in there will be the answer as it does tell you what is going
on if you search for the right things.

Cheers,

Richard

Re: Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

W dniu 05.04.2013 11:54, Richard Purdie pisze:
> This means that virtual/libc is providing something which the external
> toolchain recipes are not. Can you share the "bitbake -DDDv" output for
> the above command somewhere (I know there will be a lot of it). Buried
> somewhere in there will be the answer as it does tell you what is going
> on if you search for the right things.

http://tygrysek.juszkiewicz.com.pl/~hrw/tmp/rp-log.txt 3.4MB

Re: Does support for external toolchains working in current OE?

[Richard Purdie]

On Fri, 2013-04-05 at 11:56 +0200, Marcin Juszkiewicz wrote:
> W dniu 05.04.2013 11:54, Richard Purdie pisze:
> > This means that virtual/libc is providing something which the external
> > toolchain recipes are not. Can you share the "bitbake -DDDv" output for
> > the above command somewhere (I know there will be a lot of it). Buried
> > somewhere in there will be the answer as it does tell you what is going
> > on if you search for the right things.
> 
> http://tygrysek.juszkiewicz.com.pl/~hrw/tmp/rp-log.txt 3.4MB

Some key bits of the log:

DEBUG: Added runtime dependency eglibc
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-dbg
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-dev
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-utils
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-thread-db
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-localedata-i18n
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-ibm850
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-cp1252
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-iso8859-1
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-iso8859-15
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: providers for eglibc-dev are: ['eglibc']
DEBUG: adding
'/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc-dev'
DEBUG: providers for eglibc are: ['eglibc']
DEBUG: adding
'/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc'
DEBUG: providers for eglibc-utils are: ['eglibc']

So where does this come from? packagegroup-core-standalone-sdk-target.bb
leads us to LIBC_DEPENDENCIES which leads to:

$ cat conf/distro/include/tclibc-eglibc.inc

LIBC_DEPENDENCIES = "libsegfault \
		     eglibc \
		     eglibc-dbg \
		     eglibc-dev \
		     eglibc-utils \
		     eglibc-thread-db \
		     ${@get_libc_locales_dependencies(d)}"

LIBC_LOCALE_DEPENDENCIES = "\
	eglibc-localedata-i18n \
	eglibc-gconv-ibm850 \
	eglibc-gconv-cp1252 \
	eglibc-gconv-iso8859-1 \
	eglibc-gconv-iso8859-15"

so I suspect you either need to provide some of these things, or reset
those variables...

Cheers,

Richard

Re: Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

W dniu 05.04.2013 12:11, Richard Purdie pisze:
> On Fri, 2013-04-05 at 11:56 +0200, Marcin Juszkiewicz wrote:
>> http://tygrysek.juszkiewicz.com.pl/~hrw/tmp/rp-log.txt 3.4MB

> Some key bits of the log:
> 
> DEBUG: Added runtime dependency eglibc
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-dbg
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-dev
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-utils
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-thread-db
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-localedata-i18n
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-gconv-ibm850
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-gconv-cp1252
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-gconv-iso8859-1
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: Added runtime dependency eglibc-gconv-iso8859-15
> for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
> DEBUG: providers for eglibc-dev are: ['eglibc']
> DEBUG: adding
> '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc-dev'
> DEBUG: providers for eglibc are: ['eglibc']
> DEBUG: adding
> '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc'
> DEBUG: providers for eglibc-utils are: ['eglibc']
> 
> So where does this come from? packagegroup-core-standalone-sdk-target.bb
> leads us to LIBC_DEPENDENCIES which leads to:
> 
> $ cat conf/distro/include/tclibc-eglibc.inc
> 
> LIBC_DEPENDENCIES = "libsegfault \
> 		     eglibc \
> 		     eglibc-dbg \
> 		     eglibc-dev \
> 		     eglibc-utils \
> 		     eglibc-thread-db \
> 		     ${@get_libc_locales_dependencies(d)}"


> LIBC_LOCALE_DEPENDENCIES = "\
> 	eglibc-localedata-i18n \
> 	eglibc-gconv-ibm850 \
> 	eglibc-gconv-cp1252 \
> 	eglibc-gconv-iso8859-1 \
> 	eglibc-gconv-iso8859-15"


> so I suspect you either need to provide some of these things, 

> or reset those variables...

They can not be reset cause TCMODE is loaded before TCLIBC.

But LIBC_LOCALE_DEPENDENCIES can be ignored when TCMODE include does not
have "libc-locale" in DISTRO_FEATURES_LIBC (which is how it is done in
external-linaro toolchain).

LIBC_DEPENDENCIES left... External Linaro toolchain uses
eglibc-package.inc to get RPROVIDES for "glibc*" things so I added also
"eglibc*" ones:

RPROVIDES_${PN}            += " eglibc"
RPROVIDES_${PN}-dev        += " eglibc-dev"
RPROVIDES_${PN}-doc        += " eglibc-doc"
RPROVIDES_${PN}-dbg        += " eglibc-dbg"
RPROVIDES_${PN}-pic        += " eglibc-pic"
RPROVIDES_${PN}-utils      += " eglibc-utils"
RPROVIDES_${PN}-gconv      += " eglibc-gconv"
RPROVIDES_${PN}-pcprofile  += " eglibc-pcprofile"
RPROVIDES_eglibc-extra-nss += " eglibc-extra-nss"
RPROVIDES_eglibc-thread-db += " eglibc-thread-db"

Last two lines does not have any sense from BitBake point of view so are
ignored.

Resulting packages are still wrong... So let's hack tclibc-eglibc.inc to
use 'glibc*' in LIBC_DEPENDENCIES instead of 'eglibc*'. No luck again:

Provides: glibc-doc, eglibc-doc
Provides: glibc-pcprofile, eglibc-pcprofile
Provides: glibc-staticdev, libc-staticdev
Provides: eglibc-dbg, libc-dbg, glibc-dbg
Provides: virtual-libc-dev, libc6-dev, glibc-dev, libc-dev, eglibc-dev
Provides: rtld(GNU_HASH), glibc, eglibc
Provides: glibc-thread-db

But I do not think that LIBC_DEPENDENCIES are reason here. When reset
them in tclibc-eglibc.inc file then I get the same error about multiple
recipes ;(

Re: Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

W dniu 05.04.2013 11:20, Marcin Juszkiewicz pisze:

> ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
>  This usually means one provides something the other doesn't and should.

Let's enable debug:

DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']
NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

And now - let's dig deep into BitBake code. We want two files:

- lib/bb/providers.py
- lib/bb/taskdata.py

1. Bitbake jumps into taskdata/add_provider_internal(). 
2. Then asks bb.providers.filterProviders() 
   "is there any provider for 'virtual/libc'"?
3. In providers/_filterProviders this output is printed:
   "DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']"
4. providers/filterProviders() (note lack of "_") checks for
   PREFERRED_PROVIDER_virtual/libc and outputs:
   NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
   sets foundUnique to True, outputs:
   DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
   and returns all providers + foundUnique
5. We are back at taskdata/add_provider_internal() and we
   have more then one entry in "eligible" array plus
   foundUnique set to True.
6. Here we are happy of results and skips two "if" checks.
7. Then we add each entry as good one with this output:
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

The question is - why providers/filterProviders() returns more 
then one entry when it founds that one of them is preferred?

So I got with this change:

diff --git a/lib/bb/providers.py b/lib/bb/providers.py
index fcee6dc..0b37c44 100644
--- a/lib/bb/providers.py
+++ b/lib/bb/providers.py
@@ -296,8 +296,7 @@ def filterProviders(providers, item, cfgData, dataCache):
             pn = dataCache.pkg_fn[p]
             if dataCache.preferred[item] == pn:
                 logger.verbose("selecting %s to satisfy %s due to PREFERRED_PROVIDERS", pn, item)
-                eligible.remove(p)
-                eligible = [p] + eligible
+                eligible = [p]
                 foundUnique = True
                 break

And when I added "eglibc" into PROVIDES in 
external-linaro-toolchain.bb I got bitbake pass!!!
NO MULTIPLE RECIPES!!! :D

Re: Does support for external toolchains working in current OE?

[Marcin Juszkiewicz]

W dniu 05.04.2013 14:21, Marcin Juszkiewicz pisze:

> So I got with this change:
> 
> diff --git a/lib/bb/providers.py b/lib/bb/providers.py
> index fcee6dc..0b37c44 100644
> --- a/lib/bb/providers.py
> +++ b/lib/bb/providers.py
> @@ -296,8 +296,7 @@ def filterProviders(providers, item, cfgData, dataCache):
>              pn = dataCache.pkg_fn[p]
>              if dataCache.preferred[item] == pn:
>                  logger.verbose("selecting %s to satisfy %s due to PREFERRED_PROVIDERS", pn, item)
> -                eligible.remove(p)
> -                eligible = [p] + eligible
> +                eligible = [p]
>                  foundUnique = True
>                  break

Ignore patch.

> And when I added "eglibc" into PROVIDES in external-linaro-toolchain.bb

This was a source of issues.

Re: Does support for external toolchains working in current OE?

[Richard Purdie]

On Fri, 2013-04-05 at 14:21 +0200, Marcin Juszkiewicz wrote:
> W dniu 05.04.2013 11:20, Marcin Juszkiewicz pisze:
> 
> > ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
> >  This usually means one provides something the other doesn't and should.
> 
> Let's enable debug:
> 
> DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']
> NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
> DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc
> 
> And now - let's dig deep into BitBake code. We want two files:
> 
> - lib/bb/providers.py
> - lib/bb/taskdata.py
> 
> 1. Bitbake jumps into taskdata/add_provider_internal(). 
> 2. Then asks bb.providers.filterProviders() 
>    "is there any provider for 'virtual/libc'"?
> 3. In providers/_filterProviders this output is printed:
>    "DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']"
> 4. providers/filterProviders() (note lack of "_") checks for
>    PREFERRED_PROVIDER_virtual/libc and outputs:
>    NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
>    sets foundUnique to True, outputs:
>    DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
>    and returns all providers + foundUnique
> 5. We are back at taskdata/add_provider_internal() and we
>    have more then one entry in "eligible" array plus
>    foundUnique set to True.
> 6. Here we are happy of results and skips two "if" checks.
> 7. Then we add each entry as good one with this output:
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
> DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc
> 
> The question is - why providers/filterProviders() returns more 
> then one entry when it founds that one of them is preferred?

This is primarily due to the way bitbake once worked which was to
execute A, then stop and figure out what it should execute next.

When we enabled multiple tasks in parallel, we had to start computing
the dependency tree in advance. The trouble is I was told at the time we
had to continue to support the "if X is unavailable, try building with
Y" in --continue mode. The only way to do this is to be able to
recompute dependency trees upon a build failure.

Over time we've decided deterministic builds are actually a good think
and we shouldn't change behaviour upon task failures so we now default
to not recomputing dependencies upon failure.

The code therefore has lists for each provider sorted by priority.

So that is the history lesson and the code does it for a reason.

That said, I'm unsure how it manages to build with the code change you
mention without warnings, that is rather puzzling as I'd not have
expected that.

Cheers,

Richard

Re: Does support for external toolchains working in current OE?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

On Fri, Apr 05, 2013 at 11:20:44AM +0200, Marcin Juszkiewicz wrote:
> For last few months I am working on fixing Linaro binary cross
> toolchains and their support in OpenEmbedded. Got to point when they 
> work (both AArch64 and ARMv7a ones) for single packages but problem 
> starts when I want to create image...

Today I got different king of issue with external toolchain.

systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
but host prefix in OE is not the same as in binary toolchain (different TARGET_VENDOR)
so it looks for configure:14545: checking for arm-foo-linux-gnueabi-objcopy
while binary toolchain has only               arm-bar-linux-gnueabi-objcopy

and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
configure:14588: checking for objcopy
configure:14606: found /usr/bin/objcopy
and bam a lot later when it tries objcopy from host on some arm lib

Interesting that systemd is first recipe in our image where I've noticed
issues like this

Any idea how to easily resolve this without checking every configure.ac
how it's looking for e.g. objcopy?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: Does support for external toolchains working in current OE?

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

On Fri, Apr 5, 2013 at 2:20 PM, Martin Jansa <martin.jansa@gmail.com> wrote:

> On Fri, Apr 05, 2013 at 11:20:44AM +0200, Marcin Juszkiewicz wrote:
> > For last few months I am working on fixing Linaro binary cross
> > toolchains and their support in OpenEmbedded. Got to point when they
> > work (both AArch64 and ARMv7a ones) for single packages but problem
> > starts when I want to create image...
>
> Today I got different king of issue with external toolchain.
>
> systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
> but host prefix in OE is not the same as in binary toolchain (different
> TARGET_VENDOR)
> so it looks for configure:14545: checking for arm-foo-linux-gnueabi-objcopy
> while binary toolchain has only               arm-bar-linux-gnueabi-objcopy
>
> and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
> configure:14588: checking for objcopy
> configure:14606: found /usr/bin/objcopy
> and bam a lot later when it tries objcopy from host on some arm lib
>
> Interesting that systemd is first recipe in our image where I've noticed
> issues like this
>
> Any idea how to easily resolve this without checking every configure.ac
> how it's looking for e.g. objcopy?


export OBJCOPY in the environment to the one based on TARGET_PREFIX rather
than TARGET_SYS, and it'll use that.
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 1816 bytes --]

Re: Does support for external toolchains working in current OE?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

I've already tried that, systemd's configure does not respect that.


On Sat, Apr 6, 2013 at 12:08 AM, Chris Larson <clarson@kergoth.com> wrote:

>
> On Fri, Apr 5, 2013 at 2:20 PM, Martin Jansa <martin.jansa@gmail.com>wrote:
>
>> On Fri, Apr 05, 2013 at 11:20:44AM +0200, Marcin Juszkiewicz wrote:
>> > For last few months I am working on fixing Linaro binary cross
>> > toolchains and their support in OpenEmbedded. Got to point when they
>> > work (both AArch64 and ARMv7a ones) for single packages but problem
>> > starts when I want to create image...
>>
>> Today I got different king of issue with external toolchain.
>>
>> systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
>> but host prefix in OE is not the same as in binary toolchain (different
>> TARGET_VENDOR)
>> so it looks for configure:14545: checking for
>> arm-foo-linux-gnueabi-objcopy
>> while binary toolchain has only
>> arm-bar-linux-gnueabi-objcopy
>>
>> and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
>> configure:14588: checking for objcopy
>> configure:14606: found /usr/bin/objcopy
>> and bam a lot later when it tries objcopy from host on some arm lib
>>
>> Interesting that systemd is first recipe in our image where I've noticed
>> issues like this
>>
>> Any idea how to easily resolve this without checking every configure.ac
>> how it's looking for e.g. objcopy?
>
>
> export OBJCOPY in the environment to the one based on TARGET_PREFIX rather
> than TARGET_SYS, and it'll use that.
> --
> Christopher Larson
>

[-- Attachment #2: Type: text/html, Size: 2352 bytes --]

Re: Does support for external toolchains working in current OE?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3679 bytes --]

On Sat, Apr 06, 2013 at 12:10:01AM +0200, Martin Jansa wrote:
> I've already tried that, systemd's configure does not respect that.

I've patch for systemd which "fixes" this:

diff --git a/meta/recipes-core/systemd/systemd_199.bb b/meta/recipes-core/systemd/systemd_199.bb
index e574548..2bb9566 100644
--- a/meta/recipes-core/systemd/systemd_199.bb
+++ b/meta/recipes-core/systemd/systemd_199.bb
@@ -76,6 +76,15 @@ EXTRA_OECONF_append_libc-uclibc = " --disable-myhostname "
 do_configure_prepend() {
        export CPP="${HOST_PREFIX}cpp ${TOOLCHAIN_OPTIONS} ${HOST_CC_ARCH}"
 
+       # Allow to override them from shell variables with right prefix
+       # in some cases (external toolchain) HOST_PREFIX is not the same as HOST_SYS
+       sed -i "s#AC_PATH_TOOL(OBJCOPY,#AC_CHECK_TOOL(OBJCOPY,#g; \
+               s#AC_PATH_TOOL(STRINGS,#AC_CHECK_TOOL(STRINGS,#g; \
+               s#AC_PATH_TOOL(GPERF,#AC_CHECK_TOOL(GPERF,#g" ${S}/configure.ac
+       # OBJCOPY is already exported by default
+       export STRINGS="${HOST_PREFIX}strings"
+       export GPERF="${HOST_PREFIX}gperf"
+
        sed -i -e 's:=/root:=${ROOT_HOME}:g' units/*.service*
 }
 
But I'm still looking for some more global solution, what can go wrong when
external toolchain overwrites HOST_SYS to match overwritten TARGET_PREFIX?

e.g.
tcmode-external-sourcery.inc:TARGET_PREFIX = "${CSL_TARGET_SYS}-"

bitbake.conf:
HOST_ARCH = "${TARGET_ARCH}"
HOST_OS = "${TARGET_OS}"
HOST_VENDOR = "${TARGET_VENDOR}"
HOST_SYS = "${HOST_ARCH}${HOST_VENDOR}-${HOST_OS}"
HOST_PREFIX = "${TARGET_PREFIX}"

TARGET_ARCH = "${TUNE_ARCH}"
TARGET_OS = "linux${LIBCEXTENSION}${ABIEXTENSION}"
TARGET_VENDOR = "-oe"
TARGET_SYS = "${TARGET_ARCH}${TARGET_VENDOR}${@['-' + d.getVar('TARGET_OS', True), ''][d.getVar('TARGET_OS', True) == ('' or 'custom')]}"
TARGET_PREFIX = "${TARGET_SYS}-"

So when external-tc changes TARGET_PREFIX, it modifies HOST_PREFIX and 
both are inconsistent with TARGET_SYS/HOST_SYS which is used by autoconf 
for AC_PATH_TOOL.

> On Sat, Apr 6, 2013 at 12:08 AM, Chris Larson <clarson@kergoth.com> wrote:
> 
> >
> > On Fri, Apr 5, 2013 at 2:20 PM, Martin Jansa <martin.jansa@gmail.com>wrote:
> >
> >> On Fri, Apr 05, 2013 at 11:20:44AM +0200, Marcin Juszkiewicz wrote:
> >> > For last few months I am working on fixing Linaro binary cross
> >> > toolchains and their support in OpenEmbedded. Got to point when they
> >> > work (both AArch64 and ARMv7a ones) for single packages but problem
> >> > starts when I want to create image...
> >>
> >> Today I got different king of issue with external toolchain.
> >>
> >> systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
> >> but host prefix in OE is not the same as in binary toolchain (different
> >> TARGET_VENDOR)
> >> so it looks for configure:14545: checking for
> >> arm-foo-linux-gnueabi-objcopy
> >> while binary toolchain has only
> >> arm-bar-linux-gnueabi-objcopy
> >>
> >> and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
> >> configure:14588: checking for objcopy
> >> configure:14606: found /usr/bin/objcopy
> >> and bam a lot later when it tries objcopy from host on some arm lib
> >>
> >> Interesting that systemd is first recipe in our image where I've noticed
> >> issues like this
> >>
> >> Any idea how to easily resolve this without checking every configure.ac
> >> how it's looking for e.g. objcopy?
> >
> >
> > export OBJCOPY in the environment to the one based on TARGET_PREFIX rather
> > than TARGET_SYS, and it'll use that.
> > --
> > Christopher Larson
> >

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]