OE gcc-cross with builtin sysroot, BUG?

OE gcc-cross with builtin sysroot, BUG?

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]

hi there,

first of all, we are building dylan branch with oe-core +
meta-linaro-toolchain, not using external toolchain, hence the cross post
on OE and linaro lists.

we've been debugging some nasty build issues over the last few days on our
Jenkins server. and i'd like to bring our findings up to the list... i
suspect there could be a bug in OE gcc-cross, but i'd like to hear from you
about that.

our bug appeared because we were 'wrongly' unsetting the CFLAGS from OE in
one of our recipe, so we would loose the following from our compilation
command line

-march=armv7-a
-marm
-mthumb-interwork
-mfloat-abi=hard
-mfpu=neon
--sysroot=/work/rdk/build-genericarmv7a/tmp/sysroots/genericarmv7a

The compilation error was that stdlib.h was not found.

as a matter of fact, we build for several different machines (.conf file),
*but* we share the same sstate and downloads for all machines. We don't do
parallel builds, each machine is built sequentially in its own build folder
(.../workspace/machines/<MACHINE>/build), and all build folder are deleted
before each build (of course sstate and downloads aren't deleted).

What we noticed is that we end up pulling the compiler (gcc-cross) from
sstate as expected. However we pull the same 'blob' from sstate for
gcc-cross for *all* machines we build. It does seem that the compiler does
not use $MACHINE for the sstate checksum.

When the gcc-cross package was built and pushed in sstate, it was being
built for a specific machine (let's say MACHINE-A), hence the compiler has
the 'builtin' sysroot set to 'tmp/sysroots/MACHINE-A'.

Now when building our image for MACHINE-B, we pull gcc-cross from sstate,
and we get the default builtin sysroot for MACHINE-A!! That's why our build
failed, because we tried to build MACHINE-B before MACHINE-A, so stdlib.h
wasn't there yet on Jenkins..

To me the problem is that gcc-cross 'embedds' some $MACHINE data in its
package, but it is not marked as 'machine specific, but arch specific.  So
several machines will end up sharing the same gcc-cross package.

So, of course we shouldn't ignore the CFLAGS from OE, and the CFLAGS would
set the right sysroot, and of course we fixed our software so that we don't
ignore CFLAGS anymore... but that still looks like a bug to me.


cheers

nico

[-- Attachment #2: Type: text/html, Size: 2825 bytes --]

Re: OE gcc-cross with builtin sysroot, BUG?

[Tomas Frydrych]

Hi,

On 11/09/13 11:29, Nicolas Dechesne wrote:
> To me the problem is that gcc-cross 'embedds' some $MACHINE data in its
> package, but it is not marked as 'machine specific, but arch specific.
>  So several machines will end up sharing the same gcc-cross package.

The cross compiler should not be machine specific, that makes no sense.
It seems to me that your problem comes entirely from the introduction of
the machine element into the location of the *architecture* sysroot,
which is not how the OE system is designed to work -- if your sysroot
(i.e., the tmp dir) is in different place for different builds, then
they are entirely separate builds, you should not share sstate between
them. If you want to share the sstate, you need to share the sysroot.

But it might be worth having some sort of sanity checker to catch this
kind of misconfiguration.

Tomas


-- 
http://sleepfive.com

Re: OE gcc-cross with builtin sysroot, BUG?

[Richard Purdie]

On Wed, 2013-09-11 at 12:29 +0200, Nicolas Dechesne wrote:
> our bug appeared because we were 'wrongly' unsetting the CFLAGS from
> OE in one of our recipe, so we would loose the following from our
> compilation command line
>
> -march=armv7-a
> -marm
> -mthumb-interwork
> -mfloat-abi=hard
> -mfpu=neon
> --sysroot=/work/rdk/build-genericarmv7a/tmp/sysroots/genericarmv7a

>
> The compilation error was that stdlib.h was not found. 
> 
> 
> as a matter of fact, we build for several different machines (.conf
> file), *but* we share the same sstate and downloads for all machines.
> We don't do parallel builds, each machine is built sequentially in its
> own build folder (.../workspace/machines/<MACHINE>/build), and all
> build folder are deleted before each build (of course sstate and
> downloads aren't deleted).
> 
> 
> What we noticed is that we end up pulling the compiler (gcc-cross)
> from sstate as expected. However we pull the same 'blob' from sstate
> for gcc-cross for *all* machines we build. It does seem that the
> compiler does not use $MACHINE for the sstate checksum. 
> 
> 
> When the gcc-cross package was built and pushed in sstate, it was
> being built for a specific machine (let's say MACHINE-A), hence the
> compiler has the 'builtin' sysroot set to 'tmp/sysroots/MACHINE-A'. 
>
> Now when building our image for MACHINE-B, we pull gcc-cross from
> sstate, and we get the default builtin sysroot for MACHINE-A!! That's
> why our build failed, because we tried to build MACHINE-B before
> MACHINE-A, so stdlib.h wasn't there yet on Jenkins..
>
> To me the problem is that gcc-cross 'embedds' some $MACHINE data in
> its package, but it is not marked as 'machine specific, but arch
> specific.  So several machines will end up sharing the same gcc-cross
> package.
>
> So, of course we shouldn't ignore the CFLAGS from OE, and the CFLAGS
> would set the right sysroot, and of course we fixed our software so
> that we don't ignore CFLAGS anymore... but that still looks like a bug
> to me.
> 
Not really, its actually intentionally designed like this since its
pointless rebuilding gcc-cross multiple time just because we want to use
it with a different sysroot. We therefore just pass in the arguments to
the compiler to ensure it uses the right one. If you remove them, you
hit the problems you describe.

We should probably compile in a bogus sysroot so it never works and
makes this kind of issue more visible.

Cheers,

Richard

Re: OE gcc-cross with builtin sysroot, BUG?

[Richard Purdie]

On Wed, 2013-09-11 at 12:33 +0100, Tomas Frydrych wrote:
> Hi,
> 
> On 11/09/13 11:29, Nicolas Dechesne wrote:
> > To me the problem is that gcc-cross 'embedds' some $MACHINE data in its
> > package, but it is not marked as 'machine specific, but arch specific.
> >  So several machines will end up sharing the same gcc-cross package.
> 
> The cross compiler should not be machine specific, that makes no sense.
> It seems to me that your problem comes entirely from the introduction of
> the machine element into the location of the *architecture* sysroot,
> which is not how the OE system is designed to work -- if your sysroot
> (i.e., the tmp dir) is in different place for different builds, then
> they are entirely separate builds, you should not share sstate between
> them. If you want to share the sstate, you need to share the sysroot.

Nearly, but not quite. The sstate is designed to be able to relocate to
any tmpdir. We play with the RPATHs and other things to ensure this
works. In the gcc-cross case, we need to pass in the option to the right
sysroot but we do that. If you stop passing in the right options, you
might get lucky or you might not...

> But it might be worth having some sort of sanity checker to catch this
> kind of misconfiguration.

Poisoning the defaults for gcc-cross would seem to be the best way of
avoiding confusion.

Cheers,

Richard

Re: OE gcc-cross with builtin sysroot, BUG?

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

On Wed, Sep 11, 2013 at 2:00 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> > So, of course we shouldn't ignore the CFLAGS from OE, and the CFLAGS
> > would set the right sysroot, and of course we fixed our software so
> > that we don't ignore CFLAGS anymore... but that still looks like a bug
> > to me.
> >
> Not really, its actually intentionally designed like this since its
> pointless rebuilding gcc-cross multiple time just because we want to use
> it with a different sysroot. We therefore just pass in the arguments to
> the compiler to ensure it uses the right one. If you remove them, you
> hit the problems you describe.
>
> We should probably compile in a bogus sysroot so it never works and
> makes this kind of issue more visible.
>

hmm. ok. i see. it would certainly help to have a better 'error' message...
i suppose that not having a builtin sysroot at all, would be an even bigger
problem as it would default to /usr/include...

[-- Attachment #2: Type: text/html, Size: 1478 bytes --]

Re: OE gcc-cross with builtin sysroot, BUG?

[Richard Purdie]

On Wed, 2013-09-11 at 15:01 +0200, Nicolas Dechesne wrote:
> 
> On Wed, Sep 11, 2013 at 2:00 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:

>         Not really, its actually intentionally designed like this
>         since its
>         pointless rebuilding gcc-cross multiple time just because we
>         want to use
>         it with a different sysroot. We therefore just pass in the
>         arguments to
>         the compiler to ensure it uses the right one. If you remove
>         them, you
>         hit the problems you describe.
>         
>         We should probably compile in a bogus sysroot so it never
>         works and
>         makes this kind of issue more visible.
> 
> hmm. ok. i see. it would certainly help to have a better 'error'
> message... i suppose that not having a builtin sysroot at all, would
> be an even bigger problem as it would default to /usr/include... 
> 
> 
Right, that would be even worse than a poisoned one...

Cheers,

Richard
>