Adding an external toolchain to oe-core

Adding an external toolchain to oe-core

[Tom Parkin]

Hi folks,

We have been working to support a Codesourcery Lite MIPS toolchain
recently.  RP__ (on IRC) asked me to write up the process such that
others may benefit from our experiences.

Aside: I think this information would probably be best hosted on a
wiki, so once we've made any required changes I will seek to post this
on the OpenEmbedded wiki.  Let me know where I should put it.

Toolchain addition writeup follows:



We wanted to add support in oe-core for a Codesourcery Lite MIPS
toolchain.  The toolchain we targetted was a legacy release based
around gcc-4.2.  Toolchain binaries and release notes can be obtained
from the Codesourcery website:

http://www.codesourcery.com/sgpp/lite/mips/portal/subscription3130


# Step 0: install the toolchain on your development host

The recipes we create for the external toolchain expect that the
toolchain has been downloaded from the Codesourcery website and
installed onto the development host.

We downloaded the tarball version of the toolchain and installed it
under /usr.


# Step 1: create recipes for the new toolchain

Taking the existing recipes for the CSL 2008q3 as a template, we
cloned the following files:

   cp meta/conf/distro/include/tcmode-external-csl2008q3.inc \
      meta/conf/distro/include/tcmode-external-csl-mips-4.2-177.inc

   cp meta/recipes-core/meta/external-csl-toolchain_2008q3-72.bb \
      meta/recipes-core/meta/external-csl-toolchain_mips-4.2-177.bb

Unlike OpenEmbedded (or Poky), oe-core doesn't directly make use of
the "distro" concept; and so toolchain and libc choices are controlled
directly from local.conf.  To use the new MIPS toolchain, add the
following to local.conf:

   TCMODE = "external-csl-mips-4.2-177"
   TCLIBC = "glibc"


# Step 2: fix up toolchain recipe bitrot

The CSL 2008q3 recipe we cloned seems to have bitrotted a bit relative
to the current oe-core tree.  We had to make the following changes to
allow our new mips-4.2 recipes to work:

   *  Rework the do_configure_prepend() function in
      libc-package.bbclass as a task added to the queue before
      do_configure().  This allowed us to override the function in the
      toolchain recipe which inherits from libc-package.

   *  Add a LIC_FILES_CHECKSUM variable to the toolchain recipe.

   *  Replace hardcoded target prefixes with TARGET_PREFIX.

   *  Add libgcc and compilerlibs to the toolchain PROVIDES list.


# Step 3: tell OE that the toolchain provides locale information

The CSL MIPS toolchain provides locale information out of the box, so
we modified the toolchain recipe to suit by setting
GLIBC_INTERNAL_USE_BINARY_LOCALE to "precompiled".  You have to also
set ENABLE_BINARY_LOCALE_GENERATION to "0" in local.conf to prevent
this setting getting overridden, though!


# Step 4: hack around the non-standard toolchain naming convention

OE expects toolchain naming to align with the target prefix, e.g.

   arch-vendor-os-

The CSL toolchain is named slightly differently:

   mips-linux-gnu-

To work around this we set the variables TARGET_VENDOR and TARGET_OS
appropriately in the toolchain include file:

   TARGET_VENDOR = ""
   TARGET_OS = "linux-gnu"

To allow us to do this without tclibc-glibc.inc overriding the setting
of TARGET_OS we had to change the assignment of TARGET_OS in
tclibc-glibc.inc (to use "?=" rather than "=").

In order to have the rest of OE recognise our new target prefix, we
had to modify two lookup tables: one in siteinfo.bbclass, and one in
insane.bbclass.


# Step 5: fix up packages which depend on features CSL doesn't have

The CSL MIPS toolchain is built around sanitised kernel headers from a
2.6.24 kernel.  Various packages in the oe-core tree play poorly with
these headers, so we had to make a series of fixes for these issues.

   *  Update to util-linux-2.19.1 for changes to better deal with
      kernels which don't support fallocate.

   *  Add extra patches to the OpenSSL configure script to support a
      linux-gnu-mips target.

   *  Add a missing header include in glib for targets which have no
      inline asm "atomic" implementation in the glib tree.  MIPS is
      such a target!

   *  Use busybox mdev rather than udev.  Udev wants libc to support
      O_CLOEXEC, which isn't the case for the libc headers included in
      the CSL toolchain.


# Step 6: endianness specification

Typically cross-compiling MIPS toolchains reflect which endianness
they support by default in their naming, e.g. mipsel for little endian
targets, and mips for big endian targets.

The CSL MIPS toolchain supports both little and big endian targets
with the same toolchain with the "mips" prefix.  Endianness is
specified using the -EL and -EB arguments to gcc.

OE doesn't expect this to be the case, so we had to make some
modifications to ensure that the endianness arguments were correctly
provided to gcc throughout the build.

-- 
Tom Parkin
Senior Software Engineer
Pace Plc

Bringing Technology Home

Victoria Road, Saltaire, West Yorkshire, BD18 3LF, United Kingdom. www.pace.com

Think before you print! Please consider the environment before printing this e-mail.


This message has been scanned for viruses by BlackSpider MailControl. - www.blackspider.com

Re: Adding an external toolchain to oe-core

[Richard Purdie]

Hi,

On Thu, 2011-06-30 at 18:26 +0100, Tom Parkin wrote:
> We have been working to support a Codesourcery Lite MIPS toolchain
> recently.  RP__ (on IRC) asked me to write up the process such that
> others may benefit from our experiences.
> 
> Aside: I think this information would probably be best hosted on a
> wiki, so once we've made any required changes I will seek to post this
> on the OpenEmbedded wiki.  Let me know where I should put it.

I really appreciate this writeup, thanks!

Before we put this on the wiki do you think you'd be able to help us
clean up this process as I think we can make this easier for people?

If you are willing, I'd like to propose we drop the old 2008q3 .bb files
and replace them with ones from the toolchain you've worked with so its
a working example.

I'd then like to find ways to allow this to work "out the box" rather
than needing as much hacking to OE-Core such as the TARGET_OS change.

> Toolchain addition writeup follows:
> 
> 
> 
> We wanted to add support in oe-core for a Codesourcery Lite MIPS
> toolchain.  The toolchain we targetted was a legacy release based
> around gcc-4.2.  Toolchain binaries and release notes can be obtained
> from the Codesourcery website:
> 
> http://www.codesourcery.com/sgpp/lite/mips/portal/subscription3130
> 
> 
> # Step 0: install the toolchain on your development host
> 
> The recipes we create for the external toolchain expect that the
> toolchain has been downloaded from the Codesourcery website and
> installed onto the development host.
> 
> We downloaded the tarball version of the toolchain and installed it
> under /usr.
> 
> 
> # Step 1: create recipes for the new toolchain
> 
> Taking the existing recipes for the CSL 2008q3 as a template, we
> cloned the following files:
> 
>    cp meta/conf/distro/include/tcmode-external-csl2008q3.inc \
>       meta/conf/distro/include/tcmode-external-csl-mips-4.2-177.inc
> 
>    cp meta/recipes-core/meta/external-csl-toolchain_2008q3-72.bb \
>       meta/recipes-core/meta/external-csl-toolchain_mips-4.2-177.bb
> 
> Unlike OpenEmbedded (or Poky), oe-core doesn't directly make use of
> the "distro" concept; and so toolchain and libc choices are controlled
> directly from local.conf.  To use the new MIPS toolchain, add the
> following to local.conf:
> 
>    TCMODE = "external-csl-mips-4.2-177"
>    TCLIBC = "glibc"
> 
> 
> # Step 2: fix up toolchain recipe bitrot
> 
> The CSL 2008q3 recipe we cloned seems to have bitrotted a bit relative
> to the current oe-core tree.  We had to make the following changes to
> allow our new mips-4.2 recipes to work:
> 
>    *  Rework the do_configure_prepend() function in
>       libc-package.bbclass as a task added to the queue before
>       do_configure().  This allowed us to override the function in the
>       toolchain recipe which inherits from libc-package.
> 
>    *  Add a LIC_FILES_CHECKSUM variable to the toolchain recipe.
> 
>    *  Replace hardcoded target prefixes with TARGET_PREFIX.
> 
>    *  Add libgcc and compilerlibs to the toolchain PROVIDES list.

These issues should get addressed if you were able to provide new
toolchain recipes?

> 
> # Step 3: tell OE that the toolchain provides locale information
> 
> The CSL MIPS toolchain provides locale information out of the box, so
> we modified the toolchain recipe to suit by setting
> GLIBC_INTERNAL_USE_BINARY_LOCALE to "precompiled".  You have to also
> set ENABLE_BINARY_LOCALE_GENERATION to "0" in local.conf to prevent
> this setting getting overridden, though!

Ouch, we should find a way for the local.conf setting not to be needed.

> 
> # Step 4: hack around the non-standard toolchain naming convention
> 
> OE expects toolchain naming to align with the target prefix, e.g.
> 
>    arch-vendor-os-
> 
> The CSL toolchain is named slightly differently:
> 
>    mips-linux-gnu-
> 
> To work around this we set the variables TARGET_VENDOR and TARGET_OS
> appropriately in the toolchain include file:
> 
>    TARGET_VENDOR = ""
>    TARGET_OS = "linux-gnu"
> 
> To allow us to do this without tclibc-glibc.inc overriding the setting
> of TARGET_OS we had to change the assignment of TARGET_OS in
> tclibc-glibc.inc (to use "?=" rather than "=").

This is probably going to be the hardest issue to fix. I would like to
find a way to do it but that variable is tricky to handle for various
reasons...

> In order to have the rest of OE recognise our new target prefix, we
> had to modify two lookup tables: one in siteinfo.bbclass, and one in
> insane.bbclass.

I wonder if we can use some wildcards or some other mechanism to avoid
that.

> # Step 5: fix up packages which depend on features CSL doesn't have
> 
> The CSL MIPS toolchain is built around sanitised kernel headers from a
> 2.6.24 kernel.  Various packages in the oe-core tree play poorly with
> these headers, so we had to make a series of fixes for these issues.
> 
>    *  Update to util-linux-2.19.1 for changes to better deal with
>       kernels which don't support fallocate.
> 
>    *  Add extra patches to the OpenSSL configure script to support a
>       linux-gnu-mips target.
> 
>    *  Add a missing header include in glib for targets which have no
>       inline asm "atomic" implementation in the glib tree.  MIPS is
>       such a target!
> 
>    *  Use busybox mdev rather than udev.  Udev wants libc to support
>       O_CLOEXEC, which isn't the case for the libc headers included in
>       the CSL toolchain.

Its certainly useful to document these but since they're
version/toolchain specific, I think these are less interesting from an
OE-Core perspective.

Could you get away with using more recent kernel headers or does that
cause issues within the toolchain? I guess that ultimately depends what
kernel version you're using for your target too.

> # Step 6: endianness specification
> 
> Typically cross-compiling MIPS toolchains reflect which endianness
> they support by default in their naming, e.g. mipsel for little endian
> targets, and mips for big endian targets.
> 
> The CSL MIPS toolchain supports both little and big endian targets
> with the same toolchain with the "mips" prefix.  Endianness is
> specified using the -EL and -EB arguments to gcc.
> 
> OE doesn't expect this to be the case, so we had to make some
> modifications to ensure that the endianness arguments were correctly
> provided to gcc throughout the build.

Did you do this by changing TOOLCHAIN_OPTIONS or TARGET_CC_ARCH out of
interest?

Cheers,

Richard

Re: Adding an external toolchain to oe-core

[Tom Parkin]

Hi Richard,

Thanks for your feedback.  I've added specific comments inline below
-- but basically, ACK to the idea of submitting the updated CSL
mips-4.2 recipes for inclusion.  I think some of the specifics
discussed below will be clearer when you see the patches.

I have an upcoming demo in the pipeline which is demanding most of my
attention at the moment, but I will endeavor to get a patchset
submitted for early review this week.

On 30 June 2011 23:27, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> If you are willing, I'd like to propose we drop the old 2008q3 .bb files
> and replace them with ones from the toolchain you've worked with so its
> a working example.

That sounds like a good idea.  I would caution that the recipes we've
developed to date are merely "more working" than the 2008q3 versions,
but I think they still represent a better starting place for new
external toolchains.

> I'd then like to find ways to allow this to work "out the box" rather
> than needing as much hacking to OE-Core such as the TARGET_OS change.

Again, I agree.

Since looking at the mips 4.2 CSL toolchain I've gone on to try the
same integration process with a few toolchains we have from silicon
vendors -- this is my actual goal, as our projects typically have to
make use of vendor-provided toolchains for support reasons.

My experience from this process suggests that each new toolchain
requires a new quota of recipe hacks across the stack.  The nature of
the hacks depends on whether the specific kernel header / libc
combination included in the toolchain gels well with the selection of
packages in oe-core at the time.

So I think any effort to support a new pre-built toolchain will
require some effort on the part of the developer.  But anything that
makes external toolchain support easier across the board would be
useful.

>> The CSL MIPS toolchain provides locale information out of the box, so
>> we modified the toolchain recipe to suit by setting
>> GLIBC_INTERNAL_USE_BINARY_LOCALE to "precompiled".  You have to also
>> set ENABLE_BINARY_LOCALE_GENERATION to "0" in local.conf to prevent
>> this setting getting overridden, though!
>
> Ouch, we should find a way for the local.conf setting not to be needed.

Yes.  I'll look into how we can fix this properly.

>> To allow us to do this without tclibc-glibc.inc overriding the setting
>> of TARGET_OS we had to change the assignment of TARGET_OS in
>> tclibc-glibc.inc (to use "?=" rather than "=").
>
> This is probably going to be the hardest issue to fix. I would like to
> find a way to do it but that variable is tricky to handle for various
> reasons...

It's not clear to me whether the correct approach is:

 * to subtly abuse the variables as described previously and then fix
up the side-effects
or:
 * to modify oe-core to give it a more flexible rendering of the
arch/target/libc triplet.

FWIW, with the hacks we added to support the CSL toolchain we have
been able to add support for a couple of other MIPS toolchains without
issue, so perhaps the approach we followed was appropriate.

>> In order to have the rest of OE recognise our new target prefix, we
>> had to modify two lookup tables: one in siteinfo.bbclass, and one in
>> insane.bbclass.
>
> I wonder if we can use some wildcards or some other mechanism to avoid
> that.

Quite possibly -- I'm sure more experienced bitbake/oe devs would be
able to give pointers once we post patches.

>> # Step 5: fix up packages which depend on features CSL doesn't have
> Its certainly useful to document these but since they're
> version/toolchain specific, I think these are less interesting from an
> OE-Core perspective.

Agreed.  It is probably useful to document the types of issue one
might expect to face in supporting a new external toolchain, but the
specifics of the changes will vary from toolchain to toolchain as
discussed.

> Could you get away with using more recent kernel headers or does that
> cause issues within the toolchain? I guess that ultimately depends what
> kernel version you're using for your target too.

It's really a case of what features your libc supports since that's
what userspace is linking against at runtime.  In some cases one can
make use of kernel functionality that libc doesn't know about by means
of direct syscall()s in code, but that's pretty ugly.

As a rule I'm more comfortable to take the toolchain as-is, sanitised
kernel headers and all.

>> # Step 6: endianness specification
> Did you do this by changing TOOLCHAIN_OPTIONS or TARGET_CC_ARCH out of
> interest?

We did this by means of appending -EL to CXX and LD in the toolchain
recipe.  We then added bit of hackery in siteinfo_get_endianess() to
look at CLAGS for the "mips" case to decide whether the target is
little or big endian...

Thanks,
Tom
-- 
Tom Parkin
www.thesecretdogproject.com
Morality, like art, means drawing a line someplace /Wilde/