External toolchain support

External toolchain support

[Paul Barker]

Hi all,

As discussed on the Yocto Project call today it's great to see the ARM
toolchain recipes moving forward under the new meta-arm layers. I've
had some issues in the past with the external pre-built toolchain
which really need fixing to properly integrate this toolchain into
projects. I'm raising them here in the hopes that these can now be
fixed.

1) Maintaining GPL compliance while using an external toolchain

The Yocto Project has an archiver class which can be used to collect
the source code of all packages built by bitbake which is a
requirement for meeting the terms of the GPL. However for the external
toolchain recipes the "source" is the pre-built toolchain archive
downloaded from the ARM website. As the pre-built toolchain includes
glibc (which is then distributed as part of the images we build) what
we need is the actual source code used to build the various toolchain
components.

The toolchain recipe needs to include some link to the actual upstream
sources and the archiver class in oe-core probably needs extending to
support this.

2) Building an SDK/ESDK

If an SDK is generated for an image built with the pre-built toolchain
then there can be some confusion about what exactly should provide the
gcc binary and various headers. In the past I've had to hack
components out of the SDK after it is built and then tell customers to
install the pre-built toolchain alongside the SDK in order to get a
working cross-compiler. Ideally everything should actually get
packaged into the SDK so that only one download is needed and no
hard-coded paths are used.

Additionally, running the populate_sdk task often fails due to missing
packages (libatomic-dev right now) and it looks like the recipe
bitrots fairly quickly after each fix. SDK generation needs to be
regularly tested and it needs to be confirmed that these SDKs can
actually cross-compile working binaries.

3) Toolchain recipe is effectively split between meta-arm & meta-arago layers

The bbappends carried in meta-arago fix items which are missing from
the external-arm-toolcahin recipe in meta-arg (e.g. the installation
of the ldconfig binary). That effectively splits the recipe over
multiple layers as these things are pretty fundamental parts of the
external-arm-toolchain recipe. An effort needs to be made to merge all
this together and have the base recipe well tested.

4) No example configurations or visible test matrix

The meta-arm-toolchain layer lacks a README or other file providing
some examples of working build configurations which use the pre-built
toolchain. It's unclear which MACHINEs and DISTROs are regularly
tested and are well supported. The best way to solve this would be to
actually store the test scripts and configurations in this layer and
use a publicly visible CI system like Travis, GitLab CI, etc to run
the tests. That would increase confidence in the pre-built toolchain
and make it clear what is supported and how frequently it's tested.

5) LICENSE is incorrectly stated in the external-arm-toolchain recipe

The full license of the toolchain package needs to be specified. Right
now all packages are marked as having the MIT license which is
obviously not correct for most of them.

Apologies for the long rant but I hope this is useful to highlight
some of the issues seen when trying to practically use the pre-built
ARM toolchain.

Thanks,

-- 
Paul Barker
Konsulko Group

Re: [meta-arm] External toolchain support

[Denys Dmytriyenko]

Hi, Paul,

We have had an extensive discussion on this topic over on irc earlier today. 
Yet, I see you went ahead with your rant without any of the clarifications we 
disucssed on irc. If it is all about simply bashing external toolchains, as 
Ross suggested, then knock yourself out - I won't mind as long as you don't 
drag me into it... :) Thanks.

Denys


On Tue, May 19, 2020 at 09:03:53PM +0100, Paul Barker wrote:
> Hi all,
> 
> As discussed on the Yocto Project call today it's great to see the ARM
> toolchain recipes moving forward under the new meta-arm layers. I've
> had some issues in the past with the external pre-built toolchain
> which really need fixing to properly integrate this toolchain into
> projects. I'm raising them here in the hopes that these can now be
> fixed.
> 
> 1) Maintaining GPL compliance while using an external toolchain
> 
> The Yocto Project has an archiver class which can be used to collect
> the source code of all packages built by bitbake which is a
> requirement for meeting the terms of the GPL. However for the external
> toolchain recipes the "source" is the pre-built toolchain archive
> downloaded from the ARM website. As the pre-built toolchain includes
> glibc (which is then distributed as part of the images we build) what
> we need is the actual source code used to build the various toolchain
> components.
> 
> The toolchain recipe needs to include some link to the actual upstream
> sources and the archiver class in oe-core probably needs extending to
> support this.
> 
> 2) Building an SDK/ESDK
> 
> If an SDK is generated for an image built with the pre-built toolchain
> then there can be some confusion about what exactly should provide the
> gcc binary and various headers. In the past I've had to hack
> components out of the SDK after it is built and then tell customers to
> install the pre-built toolchain alongside the SDK in order to get a
> working cross-compiler. Ideally everything should actually get
> packaged into the SDK so that only one download is needed and no
> hard-coded paths are used.
> 
> Additionally, running the populate_sdk task often fails due to missing
> packages (libatomic-dev right now) and it looks like the recipe
> bitrots fairly quickly after each fix. SDK generation needs to be
> regularly tested and it needs to be confirmed that these SDKs can
> actually cross-compile working binaries.
> 
> 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
> 
> The bbappends carried in meta-arago fix items which are missing from
> the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> of the ldconfig binary). That effectively splits the recipe over
> multiple layers as these things are pretty fundamental parts of the
> external-arm-toolchain recipe. An effort needs to be made to merge all
> this together and have the base recipe well tested.
> 
> 4) No example configurations or visible test matrix
> 
> The meta-arm-toolchain layer lacks a README or other file providing
> some examples of working build configurations which use the pre-built
> toolchain. It's unclear which MACHINEs and DISTROs are regularly
> tested and are well supported. The best way to solve this would be to
> actually store the test scripts and configurations in this layer and
> use a publicly visible CI system like Travis, GitLab CI, etc to run
> the tests. That would increase confidence in the pre-built toolchain
> and make it clear what is supported and how frequently it's tested.
> 
> 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
> 
> The full license of the toolchain package needs to be specified. Right
> now all packages are marked as having the MIT license which is
> obviously not correct for most of them.
> 
> Apologies for the long rant but I hope this is useful to highlight
> some of the issues seen when trying to practically use the pre-built
> ARM toolchain.
> 
> Thanks,
> 
> -- 
> Paul Barker
> Konsulko Group

> 

Re: [meta-arm] External toolchain support

[William Mills]

Denys/Paul,

On 5/20/20 1:18 AM, Denys Dmytriyenko wrote:
> Hi, Paul,
> 
> We have had an extensive discussion on this topic over on irc earlier today. 
> Yet, I see you went ahead with your rant without any of the clarifications we 
> disucssed on irc. If it is all about simply bashing external toolchains, as 
> Ross suggested, then knock yourself out - I won't mind as long as you don't 
> drag me into it... :) Thanks.
> 
> Denys
> 


It would be great if any clarifications actually made it to the list
here.  My reading of the below does not come off as bashing to me.
meta-arm should be usable w/o meta-ti or meta-arago and our integration
scripts.  If there are inaccuracies in the below I would like to know
what they are.

> 
> On Tue, May 19, 2020 at 09:03:53PM +0100, Paul Barker wrote:
>> Hi all,
>>
>> As discussed on the Yocto Project call today it's great to see the ARM
>> toolchain recipes moving forward under the new meta-arm layers. I've
>> had some issues in the past with the external pre-built toolchain
>> which really need fixing to properly integrate this toolchain into
>> projects. I'm raising them here in the hopes that these can now be
>> fixed.
>>
>> 1) Maintaining GPL compliance while using an external toolchain
>>
>> The Yocto Project has an archiver class which can be used to collect
>> the source code of all packages built by bitbake which is a
>> requirement for meeting the terms of the GPL. However for the external
>> toolchain recipes the "source" is the pre-built toolchain archive
>> downloaded from the ARM website. As the pre-built toolchain includes
>> glibc (which is then distributed as part of the images we build) what
>> we need is the actual source code used to build the various toolchain
>> components.
>>
>> The toolchain recipe needs to include some link to the actual upstream
>> sources and the archiver class in oe-core probably needs extending to
>> support this.
>>
>> 2) Building an SDK/ESDK
>>
>> If an SDK is generated for an image built with the pre-built toolchain
>> then there can be some confusion about what exactly should provide the
>> gcc binary and various headers. In the past I've had to hack
>> components out of the SDK after it is built and then tell customers to
>> install the pre-built toolchain alongside the SDK in order to get a
>> working cross-compiler. Ideally everything should actually get
>> packaged into the SDK so that only one download is needed and no
>> hard-coded paths are used.
>>
>> Additionally, running the populate_sdk task often fails due to missing
>> packages (libatomic-dev right now) and it looks like the recipe
>> bitrots fairly quickly after each fix. SDK generation needs to be
>> regularly tested and it needs to be confirmed that these SDKs can
>> actually cross-compile working binaries.
>>
>> 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
>>
>> The bbappends carried in meta-arago fix items which are missing from
>> the external-arm-toolcahin recipe in meta-arg (e.g. the installation
>> of the ldconfig binary). That effectively splits the recipe over
>> multiple layers as these things are pretty fundamental parts of the
>> external-arm-toolchain recipe. An effort needs to be made to merge all
>> this together and have the base recipe well tested.
>>
>> 4) No example configurations or visible test matrix
>>
>> The meta-arm-toolchain layer lacks a README or other file providing
>> some examples of working build configurations which use the pre-built
>> toolchain. It's unclear which MACHINEs and DISTROs are regularly
>> tested and are well supported. The best way to solve this would be to
>> actually store the test scripts and configurations in this layer and
>> use a publicly visible CI system like Travis, GitLab CI, etc to run
>> the tests. That would increase confidence in the pre-built toolchain
>> and make it clear what is supported and how frequently it's tested.
>>
>> 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
>>
>> The full license of the toolchain package needs to be specified. Right
>> now all packages are marked as having the MIT license which is
>> obviously not correct for most of them.
>>
>> Apologies for the long rant but I hope this is useful to highlight
>> some of the issues seen when trying to practically use the pre-built
>> ARM toolchain.
>>
>> Thanks,
>>
>> -- 
>> Paul Barker
>> Konsulko Group
> 
>>
> 
> 
> 
> 

Re: [meta-arm] External toolchain support

[Sumit Garg]

Hi Paul,

I think first we need to understand the implementation of this recipe.
The recipe itself does *not* fetch the pre-built toolchain by itself
but it is rather dependent on the user to download and install
toolchain on the host machine and thereby provide toolchain install
path via variable "EXTERNAL_TOOLCHAIN".

In order to use any of pre-built Arm toolchain versions (8.2, 8.3 and
9.2), a user just needs to specify following in conf/local.conf:

TCMODE = "external-arm"
EXTERNAL_TOOLCHAIN = "<path-to-the-toolchain>"

- Eg. for AArch64 (eg. qemuarm64 machine in poky distro)
EXTERNAL_TOOLCHAIN =
"<installation-path>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu"

- Eg. for AArch32 (eg. qemuarm machine in poky distro)
EXTERNAL_TOOLCHAIN =
"<installation-path>/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf"

So with the above framework, the toolchain recipe isn't tied to a
single version of pre-built toolchain (or single source for which
toolchain is built) but rather is flexible to support various
pre-built toolchains which can differ in following aspects:
- Host machine.
- Arm 32bit or 64bit toolchain.
- Multiple toolchain versions.

On Wed, 20 May 2020 at 01:34, Paul Barker <pbarker@konsulko.com> wrote:
>
> Hi all,
>
> As discussed on the Yocto Project call today it's great to see the ARM
> toolchain recipes moving forward under the new meta-arm layers. I've
> had some issues in the past with the external pre-built toolchain
> which really need fixing to properly integrate this toolchain into
> projects. I'm raising them here in the hopes that these can now be
> fixed.
>
> 1) Maintaining GPL compliance while using an external toolchain
>
> The Yocto Project has an archiver class which can be used to collect
> the source code of all packages built by bitbake which is a
> requirement for meeting the terms of the GPL. However for the external
> toolchain recipes the "source" is the pre-built toolchain archive
> downloaded from the ARM website. As the pre-built toolchain includes
> glibc (which is then distributed as part of the images we build) what
> we need is the actual source code used to build the various toolchain
> components.
>
> The toolchain recipe needs to include some link to the actual upstream
> sources and the archiver class in oe-core probably needs extending to
> support this.

As I mentioned above, it's not the recipe which fetches the pre-built
toolchain but the user provides a path to the pre-installed toolchain.
And the user can easily navigate through the sources from where the
toolchain is built on the download page (see section "Sources" here
[1]) itself.

[1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

>
> 2) Building an SDK/ESDK
>
> If an SDK is generated for an image built with the pre-built toolchain
> then there can be some confusion about what exactly should provide the
> gcc binary and various headers. In the past I've had to hack
> components out of the SDK after it is built and then tell customers to
> install the pre-built toolchain alongside the SDK in order to get a
> working cross-compiler. Ideally everything should actually get
> packaged into the SDK so that only one download is needed and no
> hard-coded paths are used.

As I mentioned above a hardcoded path is needed to provide flexibility
and to avoid having duplicate recipes for every pre-built toolchain
variant.

AFAIK, with SDK all the headers and libraries should be pre-installed.
It's only the cross-compiler path which is derived from
EXTERNAL_TOOLCHAIN that needs to be specified in conf/local.conf.

>
> Additionally, running the populate_sdk task often fails due to missing
> packages (libatomic-dev right now) and it looks like the recipe
> bitrots fairly quickly after each fix. SDK generation needs to be
> regularly tested and it needs to be confirmed that these SDKs can
> actually cross-compile working binaries.

Did you give external-arm-toolchain a try? Simply do "bitbake
external-arm-toolchain" and you could see libatomic-dev package:

tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/libatomic-dev

>
> 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
>
> The bbappends carried in meta-arago fix items which are missing from
> the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> of the ldconfig binary). That effectively splits the recipe over
> multiple layers as these things are pretty fundamental parts of the
> external-arm-toolchain recipe. An effort needs to be made to merge alls
> this together and have the base recipe well tested.
>

I am not sure from where you got the reference that meta-arago is
needed to build external-arm-toolchain. external-arm-toolchain is an
independent recipe and any fixes should be pushed to meta-arm only.

Regarding ldconfig, did you give external-arm-toolchain a try? I could
see it being packaged here:

tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/ldconfig

> 4) No example configurations or visible test matrix
>
> The meta-arm-toolchain layer lacks a README or other file providing
> some examples of working build configurations which use the pre-built
> toolchain. It's unclear which MACHINEs and DISTROs are regularly
> tested and are well supported. The best way to solve this would be to
> actually store the test scripts and configurations in this layer and
> use a publicly visible CI system like Travis, GitLab CI, etc to run
> the tests. That would increase confidence in the pre-built toolchain
> and make it clear what is supported and how frequently it's tested.

I agree with you here that we atleast need a CI system to test every
fix that goes in external-arm-toolchain in order to maintain quality.

It's only two variables "TCMODE" and "EXTERNAL_TOOLCHAIN" for a
particular build to use a pre-built toolchain. We can have a README to
describe their purpose as well.

Regarding MACHINEs and DISTROs testing, I would like to point to ARM
GCC 9.2 integration release announcement [1] where you could find the
test status just prior to when we made this recipe import from
meta-linaro to meta-arm.

[1] https://lists.yoctoproject.org/g/yocto/message/48094

>
> 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
>
> The full license of the toolchain package needs to be specified. Right
> now all packages are marked as having the MIT license which is
> obviously not correct for most of them.

I guess you missed below comment:

# License applies to this recipe code, not the toolchain itself.

How do you envision to have a separate license file for every package
provided by a single recipe "external-arm-toolchain.bb"? Also, given
there is single "EULA" file available from the downloads page [1].

[1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

-Sumit

>
> Apologies for the long rant but I hope this is useful to highlight
> some of the issues seen when trying to practically use the pre-built
> ARM toolchain.
>
> Thanks,
>
> --
> Paul Barker
> Konsulko Group
> 

Re: [meta-arm] External toolchain support

[Paul Barker]

(Re-sending to the list as I got "500 Internal error" from groups.io
on the first attempt)

On Wed, 20 May 2020 at 16:33, Sumit Garg <sumit.garg@linaro.org> wrote:
>
> Hi Paul,
>
> I think first we need to understand the implementation of this recipe.
> The recipe itself does *not* fetch the pre-built toolchain by itself
> but it is rather dependent on the user to download and install
> toolchain on the host machine and thereby provide toolchain install
> path via variable "EXTERNAL_TOOLCHAIN".
>
> In order to use any of pre-built Arm toolchain versions (8.2, 8.3 and
> 9.2), a user just needs to specify following in conf/local.conf:
>
> TCMODE = "external-arm"
> EXTERNAL_TOOLCHAIN = "<path-to-the-toolchain>"
>
> - Eg. for AArch64 (eg. qemuarm64 machine in poky distro)
> EXTERNAL_TOOLCHAIN =
> "<installation-path>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu"
>
> - Eg. for AArch32 (eg. qemuarm machine in poky distro)
> EXTERNAL_TOOLCHAIN =
> "<installation-path>/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf"
>
> So with the above framework, the toolchain recipe isn't tied to a
> single version of pre-built toolchain (or single source for which
> toolchain is built) but rather is flexible to support various
> pre-built toolchains which can differ in following aspects:
> - Host machine.
> - Arm 32bit or 64bit toolchain.
> - Multiple toolchain versions.

This is fine, but the recipe still needs to allow common Yocto Project
tasks to work such as using the archiver, keeping track of the
licenses of the software that goes into an image, generating an SDK,
etc.

>
> On Wed, 20 May 2020 at 01:34, Paul Barker <pbarker@konsulko.com> wrote:
> >
> > Hi all,
> >
> > As discussed on the Yocto Project call today it's great to see the ARM
> > toolchain recipes moving forward under the new meta-arm layers. I've
> > had some issues in the past with the external pre-built toolchain
> > which really need fixing to properly integrate this toolchain into
> > projects. I'm raising them here in the hopes that these can now be
> > fixed.
> >
> > 1) Maintaining GPL compliance while using an external toolchain
> >
> > The Yocto Project has an archiver class which can be used to collect
> > the source code of all packages built by bitbake which is a
> > requirement for meeting the terms of the GPL. However for the external
> > toolchain recipes the "source" is the pre-built toolchain archive
> > downloaded from the ARM website. As the pre-built toolchain includes
> > glibc (which is then distributed as part of the images we build) what
> > we need is the actual source code used to build the various toolchain
> > components.
> >
> > The toolchain recipe needs to include some link to the actual upstream
> > sources and the archiver class in oe-core probably needs extending to
> > support this.
>
> As I mentioned above, it's not the recipe which fetches the pre-built
> toolchain but the user provides a path to the pre-installed toolchain.
> And the user can easily navigate through the sources from where the
> toolchain is built on the download page (see section "Sources" here
> [1]) itself.
>
> [1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

This needs to be supported by the recipe though - even if the source
location is given as a path and the developer is responsible for
downloading the toolchain source before running bitbake.

The recipe does place GPL licensed components into an image so it
really needs to work with the archiver to allow users to achieve GPL
compliance.

>
> >
> > 2) Building an SDK/ESDK
> >
> > If an SDK is generated for an image built with the pre-built toolchain
> > then there can be some confusion about what exactly should provide the
> > gcc binary and various headers. In the past I've had to hack
> > components out of the SDK after it is built and then tell customers to
> > install the pre-built toolchain alongside the SDK in order to get a
> > working cross-compiler. Ideally everything should actually get
> > packaged into the SDK so that only one download is needed and no
> > hard-coded paths are used.
>
> As I mentioned above a hardcoded path is needed to provide flexibility
> and to avoid having duplicate recipes for every pre-built toolchain
> variant.
>
> AFAIK, with SDK all the headers and libraries should be pre-installed.
> It's only the cross-compiler path which is derived from
> EXTERNAL_TOOLCHAIN that needs to be specified in conf/local.conf.
>
> >
> > Additionally, running the populate_sdk task often fails due to missing
> > packages (libatomic-dev right now) and it looks like the recipe
> > bitrots fairly quickly after each fix. SDK generation needs to be
> > regularly tested and it needs to be confirmed that these SDKs can
> > actually cross-compile working binaries.
>
> Did you give external-arm-toolchain a try? Simply do "bitbake
> external-arm-toolchain" and you could see libatomic-dev package:
>
> tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/libatomic-dev
>

I'm interested in building a real image (say core-image-base) then
building an SDK (bitbake core-image-base -c populate_sdk) and then
confirming that the SDK can be installed on a new host machine and
used to cross-compile working binaries for the target. Is that
supported and regularly tested for the pre-built toolchain?

> >
> > 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
> >
> > The bbappends carried in meta-arago fix items which are missing from
> > the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> > of the ldconfig binary). That effectively splits the recipe over
> > multiple layers as these things are pretty fundamental parts of the
> > external-arm-toolchain recipe. An effort needs to be made to merge alls
> > this together and have the base recipe well tested.
> >
>
> I am not sure from where you got the reference that meta-arago is
> needed to build external-arm-toolchain. external-arm-toolchain is an
> independent recipe and any fixes should be pushed to meta-arm only.
>
> Regarding ldconfig, did you give external-arm-toolchain a try? I could
> see it being packaged here:
>
> tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/ldconfig

As there's no README file or anything else to suggest how to use the
pre-built toolchain, I have only tested the toolchain in the Arago
distro where the toolchain is enabled by default. If some
documentation is added which lists machines, distros and images that
are supported then I'd be willing to try a build and report back on
what I find.

>
> > 4) No example configurations or visible test matrix
> >
> > The meta-arm-toolchain layer lacks a README or other file providing
> > some examples of working build configurations which use the pre-built
> > toolchain. It's unclear which MACHINEs and DISTROs are regularly
> > tested and are well supported. The best way to solve this would be to
> > actually store the test scripts and configurations in this layer and
> > use a publicly visible CI system like Travis, GitLab CI, etc to run
> > the tests. That would increase confidence in the pre-built toolchain
> > and make it clear what is supported and how frequently it's tested.
>
> I agree with you here that we atleast need a CI system to test every
> fix that goes in external-arm-toolchain in order to maintain quality.
>
> It's only two variables "TCMODE" and "EXTERNAL_TOOLCHAIN" for a
> particular build to use a pre-built toolchain. We can have a README to
> describe their purpose as well.
>
> Regarding MACHINEs and DISTROs testing, I would like to point to ARM
> GCC 9.2 integration release announcement [1] where you could find the
> test status just prior to when we made this recipe import from
> meta-linaro to meta-arm.
>
> [1] https://lists.yoctoproject.org/g/yocto/message/48094

This information belongs in the repository in a readme file not just
in an old email.

>
> >
> > 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
> >
> > The full license of the toolchain package needs to be specified. Right
> > now all packages are marked as having the MIT license which is
> > obviously not correct for most of them.
>
> I guess you missed below comment:
>
> # License applies to this recipe code, not the toolchain itself.
>
> How do you envision to have a separate license file for every package
> provided by a single recipe "external-arm-toolchain.bb"? Also, given
> there is single "EULA" file available from the downloads page [1].
>
> [1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

The LICENSE field is explicitly there to record the license of the
packages generated from the recipe, not of the recipe code itself.
There are many examples in openembedded-core and other layers of using
LICENSE_subpackage to set the license of "subpackage" within the
recipe. See https://www.yoctoproject.org/docs/3.1/mega-manual/mega-manual.html#var-LICENSE
for more info.

Regards,

-- 
Paul Barker
Konsulko Group

Re: [meta-arm] External toolchain support

[Sumit Garg]

On Wed, 20 May 2020 at 21:32, Paul Barker <pbarker@konsulko.com> wrote:
>
> (Re-sending to the list as I got "500 Internal error" from groups.io
> on the first attempt)
>
> On Wed, 20 May 2020 at 16:33, Sumit Garg <sumit.garg@linaro.org> wrote:
> >
> > Hi Paul,
> >
> > I think first we need to understand the implementation of this recipe.
> > The recipe itself does *not* fetch the pre-built toolchain by itself
> > but it is rather dependent on the user to download and install
> > toolchain on the host machine and thereby provide toolchain install
> > path via variable "EXTERNAL_TOOLCHAIN".
> >
> > In order to use any of pre-built Arm toolchain versions (8.2, 8.3 and
> > 9.2), a user just needs to specify following in conf/local.conf:
> >
> > TCMODE = "external-arm"
> > EXTERNAL_TOOLCHAIN = "<path-to-the-toolchain>"
> >
> > - Eg. for AArch64 (eg. qemuarm64 machine in poky distro)
> > EXTERNAL_TOOLCHAIN =
> > "<installation-path>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu"
> >
> > - Eg. for AArch32 (eg. qemuarm machine in poky distro)
> > EXTERNAL_TOOLCHAIN =
> > "<installation-path>/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf"
> >
> > So with the above framework, the toolchain recipe isn't tied to a
> > single version of pre-built toolchain (or single source for which
> > toolchain is built) but rather is flexible to support various
> > pre-built toolchains which can differ in following aspects:
> > - Host machine.
> > - Arm 32bit or 64bit toolchain.
> > - Multiple toolchain versions.
>
> This is fine, but the recipe still needs to allow common Yocto Project
> tasks to work such as using the archiver, keeping track of the
> licenses of the software that goes into an image, generating an SDK,
> etc.
>

See my response below.

> >
> > On Wed, 20 May 2020 at 01:34, Paul Barker <pbarker@konsulko.com> wrote:
> > >
> > > Hi all,
> > >
> > > As discussed on the Yocto Project call today it's great to see the ARM
> > > toolchain recipes moving forward under the new meta-arm layers. I've
> > > had some issues in the past with the external pre-built toolchain
> > > which really need fixing to properly integrate this toolchain into
> > > projects. I'm raising them here in the hopes that these can now be
> > > fixed.
> > >
> > > 1) Maintaining GPL compliance while using an external toolchain
> > >
> > > The Yocto Project has an archiver class which can be used to collect
> > > the source code of all packages built by bitbake which is a
> > > requirement for meeting the terms of the GPL. However for the external
> > > toolchain recipes the "source" is the pre-built toolchain archive
> > > downloaded from the ARM website. As the pre-built toolchain includes
> > > glibc (which is then distributed as part of the images we build) what
> > > we need is the actual source code used to build the various toolchain
> > > components.
> > >
> > > The toolchain recipe needs to include some link to the actual upstream
> > > sources and the archiver class in oe-core probably needs extending to
> > > support this.
> >
> > As I mentioned above, it's not the recipe which fetches the pre-built
> > toolchain but the user provides a path to the pre-installed toolchain.
> > And the user can easily navigate through the sources from where the
> > toolchain is built on the download page (see section "Sources" here
> > [1]) itself.
> >
> > [1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads
>
> This needs to be supported by the recipe though - even if the source
> location is given as a path and the developer is responsible for
> downloading the toolchain source before running bitbake.
>
> The recipe does place GPL licensed components into an image so it
> really needs to work with the archiver to allow users to achieve GPL
> compliance.
>

See my response below.

> >
> > >
> > > 2) Building an SDK/ESDK
> > >
> > > If an SDK is generated for an image built with the pre-built toolchain
> > > then there can be some confusion about what exactly should provide the
> > > gcc binary and various headers. In the past I've had to hack
> > > components out of the SDK after it is built and then tell customers to
> > > install the pre-built toolchain alongside the SDK in order to get a
> > > working cross-compiler. Ideally everything should actually get
> > > packaged into the SDK so that only one download is needed and no
> > > hard-coded paths are used.
> >
> > As I mentioned above a hardcoded path is needed to provide flexibility
> > and to avoid having duplicate recipes for every pre-built toolchain
> > variant.
> >
> > AFAIK, with SDK all the headers and libraries should be pre-installed.
> > It's only the cross-compiler path which is derived from
> > EXTERNAL_TOOLCHAIN that needs to be specified in conf/local.conf.
> >
> > >
> > > Additionally, running the populate_sdk task often fails due to missing
> > > packages (libatomic-dev right now) and it looks like the recipe
> > > bitrots fairly quickly after each fix. SDK generation needs to be
> > > regularly tested and it needs to be confirmed that these SDKs can
> > > actually cross-compile working binaries.
> >
> > Did you give external-arm-toolchain a try? Simply do "bitbake
> > external-arm-toolchain" and you could see libatomic-dev package:
> >
> > tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/libatomic-dev
> >
>
> I'm interested in building a real image (say core-image-base)

This is well tested and supported using external-arm-toolchain.

> then
> building an SDK (bitbake core-image-base -c populate_sdk) and then
> confirming that the SDK can be installed on a new host machine and
> used to cross-compile working binaries for the target.

Building an SDK isn't supported and tested as of now. So I gave it a
try and it seems to be minor dependency issues which we should be able
to fix. I will try to find some time next week in order to fix those
but in case you already have corresponding fixes then I would be happy
to review them too.

> Is that
> supported and regularly tested for the pre-built toolchain?
>
> > >
> > > 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
> > >
> > > The bbappends carried in meta-arago fix items which are missing from
> > > the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> > > of the ldconfig binary). That effectively splits the recipe over
> > > multiple layers as these things are pretty fundamental parts of the
> > > external-arm-toolchain recipe. An effort needs to be made to merge alls
> > > this together and have the base recipe well tested.
> > >
> >
> > I am not sure from where you got the reference that meta-arago is
> > needed to build external-arm-toolchain. external-arm-toolchain is an
> > independent recipe and any fixes should be pushed to meta-arm only.
> >
> > Regarding ldconfig, did you give external-arm-toolchain a try? I could
> > see it being packaged here:
> >
> > tmp/work/aarch64-poky-linux/external-arm-toolchain/2019.12-r0/pkgdata/runtime/ldconfig
>
> As there's no README file or anything else to suggest how to use the
> pre-built toolchain, I have only tested the toolchain in the Arago
> distro where the toolchain is enabled by default. If some
> documentation is added which lists machines, distros and images that
> are supported then I'd be willing to try a build and report back on
> what I find.
>

external-arm-toolchain is meant to be distro and machine agnostic as
long as you are cross-compiling for arm/arm64 architectures. It is
just a replacement of the OE toolchain built from source with a
pre-built toolchain.

But yes I agree with you that we need a README for
external-arm-toolchain to list its usage and supported features. Will
add it.

> >
> > > 4) No example configurations or visible test matrix
> > >
> > > The meta-arm-toolchain layer lacks a README or other file providing
> > > some examples of working build configurations which use the pre-built
> > > toolchain. It's unclear which MACHINEs and DISTROs are regularly
> > > tested and are well supported. The best way to solve this would be to
> > > actually store the test scripts and configurations in this layer and
> > > use a publicly visible CI system like Travis, GitLab CI, etc to run
> > > the tests. That would increase confidence in the pre-built toolchain
> > > and make it clear what is supported and how frequently it's tested.
> >
> > I agree with you here that we atleast need a CI system to test every
> > fix that goes in external-arm-toolchain in order to maintain quality.
> >
> > It's only two variables "TCMODE" and "EXTERNAL_TOOLCHAIN" for a
> > particular build to use a pre-built toolchain. We can have a README to
> > describe their purpose as well.
> >
> > Regarding MACHINEs and DISTROs testing, I would like to point to ARM
> > GCC 9.2 integration release announcement [1] where you could find the
> > test status just prior to when we made this recipe import from
> > meta-linaro to meta-arm.
> >
> > [1] https://lists.yoctoproject.org/g/yocto/message/48094
>
> This information belongs in the repository in a readme file not just
> in an old email.
>
> >
> > >
> > > 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
> > >
> > > The full license of the toolchain package needs to be specified. Right
> > > now all packages are marked as having the MIT license which is
> > > obviously not correct for most of them.
> >
> > I guess you missed below comment:
> >
> > # License applies to this recipe code, not the toolchain itself.
> >
> > How do you envision to have a separate license file for every package
> > provided by a single recipe "external-arm-toolchain.bb"? Also, given
> > there is single "EULA" file available from the downloads page [1].
> >
> > [1] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads
>
> The LICENSE field is explicitly there to record the license of the
> packages generated from the recipe, not of the recipe code itself.
> There are many examples in openembedded-core and other layers of using
> LICENSE_subpackage to set the license of "subpackage" within the
> recipe. See https://www.yoctoproject.org/docs/3.1/mega-manual/mega-manual.html#var-LICENSE
> for more info.
>

I think your above and this concern should be addressed with a
license.inc file listing licenses for all sub-packages. If yes, then I
will post a patch for the same.

-Sumit

> Regards,
>
> --
> Paul Barker
> Konsulko Group
> 

Re: [meta-arm] External toolchain support

[Denys Dmytriyenko]

All,

I know I said I wasn't going to participate in this discussion, but seeing 
where things are heading, I wanted to set some records straight... A brief 
history, that should explain few things the way they are now and I'll try to 
be brief and humble :) Also, I mean no disrespect or offense to anybody and am 
trying to be as polite and diplomatic as possible!

Around 2007-2008 we migrated our Linux product to OpenEmbedded (Classic at the 
time) and we called it the Arago Project - one of the original requirements 
was the use of a prebuilt cross-compile toolchain. Back then CodeSourcery Lite 
was the most popular and free ARM cross-compile toolchain. So I had to add 
support for it to OpenEmbedded and maintain it for few years since 2009 [1]. 
I used a basic stub from Poky as a starting point, but most of it was my own, 
like a single recipe supporting different toolchain versions, etc. I also had 
recipes for different OE-based Distro toolchains, like Angstrom...

Late 2010, early 2011 saw 2 relevant major events happening - CodeSourcery was 
acquired by Mentor and eventually discontinued the free Lite tier; and the 
announcement of the Yocto Project and reaching agreement with OpenEmbedded 
(yeah, I was there :) While everyone started creating their own meta layers 
(meta-ti for our BSP and a pair of meta-arago for Apps and Distro), there was 
a void for a free prebuilt ARM cross-compile toolchain. That resulted in a 
public Arago toolchain release [2] that I built and maintained for few years 
(and the corresponding recipe). BTW, back then I had the correct per-component 
Licenses set based on the version [3].

We had additional requirements for our products to support on-target native 
compilation (external runtime + internal compiler), as well as SDK 
cross-compiling while re-using the same external toolchain (with the option of 
getting standalone toolchain separately from SDK). A lot of extras has been 
added on top of the initial external-toolchain recipe. Most of that was inside 
the Arago Project code base, as that's where it originated. And I have given 
several presentations around that time and following years at conferences on 
the use of external toolchains in OpenEmbedded, using same external toolchains 
in SDKs, dealing with relocation issues, etc. [4] Those weren't very popular 
topics among OE/Yocto crowd, as that camp prefers building toolchains from 
sources :) But our non-OE developers and users demanded prebuilt toolchains 
and maximum re-use of corresponding components, while being flexible shipping 
OE SDK w/o the toolchain pre-integrated due to license restrictions...

And while Linaro already had their gcc team working on ARM enhancements, they 
didn't have any releases for few years. Thanks to Bill and others, by 2013 
that got corrected and OE recipes based on existing old work were added to 
support the new Linaro toolchain releases. But it only addressed the basic 
cross-compiling of target binaries - no SDK support, no native on-target 
compilation, no extras. Still, it was time to switch to the new Linaro 
toolchain releases. And meta-linaro was quite open and receptive to patches 
from outside, so I started submitting fixes and enhancements I had accumulated 
in meta-arago over the years. I also had an open communication channel with 
Koen, so that was also helpful to push changes in.

But in recent years things weren't as rosy as they were before - toolchain 
release ownership has changed, there was no clear owner for the recipe to go 
along, so I had to step up my game [5], although I raised the question few 
times whether I should just take over and take the recipe back...

Also, 2 major downsides surfaced with meta-linaro process itself - lack of 
testing of external-toolchain use in OE (besides basics) and no public review 
of Linaro-originated changes by using internal gerrit instead of public 
mailing list. The last one I didn't realize until later time. Often I would 
get an unnoticed run-time breakage introduced by some recent change in the 
recipe that didn't account for all our extensive downstream use cases. As 
there were no patches on the list, no reviews, no heads-up, I would scramble 
around in order to fix new breakages and in most cases would try to submit the 
fix upstream. Only to get them broken later due to the mentioned process 
downsides. I did mention couple of such instances recently [6][7]...

So, when earlier this year I started working with meta-arm, I was very pleased 
with the open process, where all internal and external patches go to the list 
for review. And I was happy to see OPTEE and ARM toolchain recipes migrating 
there as well [8], so in the past several months I cleaned up and upstreamed 
many hacks and enhancements there from meta-arago [9]. Of course, few were 
still remaining, but the plan was always to get everything upstream. 
Unfortunately, Paul wasn't willing to wait much longer... :)

That "activated" Sumit :) and I understand his position very well and 
appreciate his energy! He's been given a problem and is trying to solve it, 
but he's changing some fundamentals as he's not bound by many use cases. 
Changing those fundamentals actually breaks bunch of my use cases that I'm 
bound to by our product. So, on one hand he's changing things upstream and 
I'm downstream and am supposed to adapt. But on the other hand, I've been 
maintaining and supporting this for 11-12 years and I have tons of baggage in 
form of existing products and multiple use cases he is not considering. Plus, 
it was never clear if there are any other users or consumers of 
external-toolchain, especially to the extent we use it in the Arago Project as 
part of TI SDK products, which means changing things in incompatible way 
breaks the only major consumer of this feature. I fear if we cannot find a 
reasonable technical and architectural consensus for these changes, it may 
result in the opposite outcome from what Paul was seeking - a major fork with 
2 separate external-toolchain supports. And I'm not trying to make any 
excuses, but the fact that I'm severely burned out right now only exacerbates 
the whole situation, sorry...


[1] https://git.openembedded.org/openembedded/log/recipes/meta/external-toolchain-csl.bb
[2] http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/index_FDS.html
[3] http://arago-project.org/git/?p=meta-arago.git;a=blob;f=meta-arago-extras/recipes-core/meta/external-arago-toolchain.inc;h=da0d9abd77a17f2334cdea8b5f05ef43a550eaaf;hb=90dc157478439b2facc87a47d8ba8abb596edc6e
[4] https://elinux.org/images/c/c8/ExternalToolchainsInYocto.pdf
[5] https://git.linaro.org/openembedded/meta-linaro.git/log/meta-linaro-toolchain/recipes-devtools/external-arm-toolchain/external-arm-toolchain.bb
[6] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=efd07731659f2e8e8970f1a26e8dacfbd19ce832
[7] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=7765d89009f9ace12e600d9e6650fa8f019ba034
[8] https://lists.yoctoproject.org/g/meta-arm/message/13
[9] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/log/meta-arm-toolchain


On Tue, May 19, 2020 at 09:03:53PM +0100, Paul Barker wrote:
> Hi all,
> 
> As discussed on the Yocto Project call today it's great to see the ARM
> toolchain recipes moving forward under the new meta-arm layers. I've
> had some issues in the past with the external pre-built toolchain
> which really need fixing to properly integrate this toolchain into
> projects. I'm raising them here in the hopes that these can now be
> fixed.
> 
> 1) Maintaining GPL compliance while using an external toolchain
> 
> The Yocto Project has an archiver class which can be used to collect
> the source code of all packages built by bitbake which is a
> requirement for meeting the terms of the GPL. However for the external
> toolchain recipes the "source" is the pre-built toolchain archive
> downloaded from the ARM website. As the pre-built toolchain includes
> glibc (which is then distributed as part of the images we build) what
> we need is the actual source code used to build the various toolchain
> components.
> 
> The toolchain recipe needs to include some link to the actual upstream
> sources and the archiver class in oe-core probably needs extending to
> support this.
> 
> 2) Building an SDK/ESDK
> 
> If an SDK is generated for an image built with the pre-built toolchain
> then there can be some confusion about what exactly should provide the
> gcc binary and various headers. In the past I've had to hack
> components out of the SDK after it is built and then tell customers to
> install the pre-built toolchain alongside the SDK in order to get a
> working cross-compiler. Ideally everything should actually get
> packaged into the SDK so that only one download is needed and no
> hard-coded paths are used.
> 
> Additionally, running the populate_sdk task often fails due to missing
> packages (libatomic-dev right now) and it looks like the recipe
> bitrots fairly quickly after each fix. SDK generation needs to be
> regularly tested and it needs to be confirmed that these SDKs can
> actually cross-compile working binaries.
> 
> 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
> 
> The bbappends carried in meta-arago fix items which are missing from
> the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> of the ldconfig binary). That effectively splits the recipe over
> multiple layers as these things are pretty fundamental parts of the
> external-arm-toolchain recipe. An effort needs to be made to merge all
> this together and have the base recipe well tested.
> 
> 4) No example configurations or visible test matrix
> 
> The meta-arm-toolchain layer lacks a README or other file providing
> some examples of working build configurations which use the pre-built
> toolchain. It's unclear which MACHINEs and DISTROs are regularly
> tested and are well supported. The best way to solve this would be to
> actually store the test scripts and configurations in this layer and
> use a publicly visible CI system like Travis, GitLab CI, etc to run
> the tests. That would increase confidence in the pre-built toolchain
> and make it clear what is supported and how frequently it's tested.
> 
> 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
> 
> The full license of the toolchain package needs to be specified. Right
> now all packages are marked as having the MIT license which is
> obviously not correct for most of them.
> 
> Apologies for the long rant but I hope this is useful to highlight
> some of the issues seen when trying to practically use the pre-built
> ARM toolchain.
> 
> Thanks,
> 
> -- 
> Paul Barker
> Konsulko Group

> 

Re: [meta-arm] External toolchain support

[Paul Barker]

On Thu, 11 Jun 2020 at 03:01, Denys Dmytriyenko <denis@denix.org> wrote:
>
> All,
>
> I know I said I wasn't going to participate in this discussion, but seeing
> where things are heading, I wanted to set some records straight... A brief
> history, that should explain few things the way they are now and I'll try to
> be brief and humble :) Also, I mean no disrespect or offense to anybody and am
> trying to be as polite and diplomatic as possible!
>
> Around 2007-2008 we migrated our Linux product to OpenEmbedded (Classic at the
> time) and we called it the Arago Project - one of the original requirements
> was the use of a prebuilt cross-compile toolchain. Back then CodeSourcery Lite
> was the most popular and free ARM cross-compile toolchain. So I had to add
> support for it to OpenEmbedded and maintain it for few years since 2009 [1].
> I used a basic stub from Poky as a starting point, but most of it was my own,
> like a single recipe supporting different toolchain versions, etc. I also had
> recipes for different OE-based Distro toolchains, like Angstrom...
>
> Late 2010, early 2011 saw 2 relevant major events happening - CodeSourcery was
> acquired by Mentor and eventually discontinued the free Lite tier; and the
> announcement of the Yocto Project and reaching agreement with OpenEmbedded
> (yeah, I was there :) While everyone started creating their own meta layers
> (meta-ti for our BSP and a pair of meta-arago for Apps and Distro), there was
> a void for a free prebuilt ARM cross-compile toolchain. That resulted in a
> public Arago toolchain release [2] that I built and maintained for few years
> (and the corresponding recipe). BTW, back then I had the correct per-component
> Licenses set based on the version [3].
>
> We had additional requirements for our products to support on-target native
> compilation (external runtime + internal compiler), as well as SDK
> cross-compiling while re-using the same external toolchain (with the option of
> getting standalone toolchain separately from SDK). A lot of extras has been
> added on top of the initial external-toolchain recipe. Most of that was inside
> the Arago Project code base, as that's where it originated. And I have given
> several presentations around that time and following years at conferences on
> the use of external toolchains in OpenEmbedded, using same external toolchains
> in SDKs, dealing with relocation issues, etc. [4] Those weren't very popular
> topics among OE/Yocto crowd, as that camp prefers building toolchains from
> sources :) But our non-OE developers and users demanded prebuilt toolchains
> and maximum re-use of corresponding components, while being flexible shipping
> OE SDK w/o the toolchain pre-integrated due to license restrictions...
>
> And while Linaro already had their gcc team working on ARM enhancements, they
> didn't have any releases for few years. Thanks to Bill and others, by 2013
> that got corrected and OE recipes based on existing old work were added to
> support the new Linaro toolchain releases. But it only addressed the basic
> cross-compiling of target binaries - no SDK support, no native on-target
> compilation, no extras. Still, it was time to switch to the new Linaro
> toolchain releases. And meta-linaro was quite open and receptive to patches
> from outside, so I started submitting fixes and enhancements I had accumulated
> in meta-arago over the years. I also had an open communication channel with
> Koen, so that was also helpful to push changes in.
>
> But in recent years things weren't as rosy as they were before - toolchain
> release ownership has changed, there was no clear owner for the recipe to go
> along, so I had to step up my game [5], although I raised the question few
> times whether I should just take over and take the recipe back...
>
> Also, 2 major downsides surfaced with meta-linaro process itself - lack of
> testing of external-toolchain use in OE (besides basics) and no public review
> of Linaro-originated changes by using internal gerrit instead of public
> mailing list. The last one I didn't realize until later time. Often I would
> get an unnoticed run-time breakage introduced by some recent change in the
> recipe that didn't account for all our extensive downstream use cases. As
> there were no patches on the list, no reviews, no heads-up, I would scramble
> around in order to fix new breakages and in most cases would try to submit the
> fix upstream. Only to get them broken later due to the mentioned process
> downsides. I did mention couple of such instances recently [6][7]...
>
> So, when earlier this year I started working with meta-arm, I was very pleased
> with the open process, where all internal and external patches go to the list
> for review. And I was happy to see OPTEE and ARM toolchain recipes migrating
> there as well [8], so in the past several months I cleaned up and upstreamed
> many hacks and enhancements there from meta-arago [9]. Of course, few were
> still remaining, but the plan was always to get everything upstream.
> Unfortunately, Paul wasn't willing to wait much longer... :)
>
> That "activated" Sumit :) and I understand his position very well and
> appreciate his energy! He's been given a problem and is trying to solve it,
> but he's changing some fundamentals as he's not bound by many use cases.
> Changing those fundamentals actually breaks bunch of my use cases that I'm
> bound to by our product. So, on one hand he's changing things upstream and
> I'm downstream and am supposed to adapt. But on the other hand, I've been
> maintaining and supporting this for 11-12 years and I have tons of baggage in
> form of existing products and multiple use cases he is not considering. Plus,
> it was never clear if there are any other users or consumers of
> external-toolchain, especially to the extent we use it in the Arago Project as
> part of TI SDK products, which means changing things in incompatible way
> breaks the only major consumer of this feature. I fear if we cannot find a
> reasonable technical and architectural consensus for these changes, it may
> result in the opposite outcome from what Paul was seeking - a major fork with
> 2 separate external-toolchain supports. And I'm not trying to make any
> excuses, but the fact that I'm severely burned out right now only exacerbates
> the whole situation, sorry...
>
>
> [1] https://git.openembedded.org/openembedded/log/recipes/meta/external-toolchain-csl.bb
> [2] http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/index_FDS.html
> [3] http://arago-project.org/git/?p=meta-arago.git;a=blob;f=meta-arago-extras/recipes-core/meta/external-arago-toolchain.inc;h=da0d9abd77a17f2334cdea8b5f05ef43a550eaaf;hb=90dc157478439b2facc87a47d8ba8abb596edc6e
> [4] https://elinux.org/images/c/c8/ExternalToolchainsInYocto.pdf
> [5] https://git.linaro.org/openembedded/meta-linaro.git/log/meta-linaro-toolchain/recipes-devtools/external-arm-toolchain/external-arm-toolchain.bb
> [6] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=efd07731659f2e8e8970f1a26e8dacfbd19ce832
> [7] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=7765d89009f9ace12e600d9e6650fa8f019ba034
> [8] https://lists.yoctoproject.org/g/meta-arm/message/13
> [9] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/log/meta-arm-toolchain
>

Denys,

Thanks for the context here, it's really valuable to understand how
things have developed. I'd prefer that things improve slowly than that
we have a breaking fork.

Can we identify which of the proposed changes can be made with low
risk? I think fixing the readme and licensing at least should be
straightforward.

I'll look at the archiver support myself and see if I can find a
solution for that.

Thanks,

-- 
Paul Barker
Konsulko Group

Re: [meta-arm] External toolchain support

[Sumit Garg]

On Thu, 11 Jun 2020 at 07:31, Denys Dmytriyenko <denis@denix.org> wrote:
>
> All,
>
> I know I said I wasn't going to participate in this discussion, but seeing
> where things are heading, I wanted to set some records straight... A brief
> history, that should explain few things the way they are now and I'll try to
> be brief and humble :) Also, I mean no disrespect or offense to anybody and am
> trying to be as polite and diplomatic as possible!
>
> Around 2007-2008 we migrated our Linux product to OpenEmbedded (Classic at the
> time) and we called it the Arago Project - one of the original requirements
> was the use of a prebuilt cross-compile toolchain. Back then CodeSourcery Lite
> was the most popular and free ARM cross-compile toolchain. So I had to add
> support for it to OpenEmbedded and maintain it for few years since 2009 [1].
> I used a basic stub from Poky as a starting point, but most of it was my own,
> like a single recipe supporting different toolchain versions, etc. I also had
> recipes for different OE-based Distro toolchains, like Angstrom...
>
> Late 2010, early 2011 saw 2 relevant major events happening - CodeSourcery was
> acquired by Mentor and eventually discontinued the free Lite tier; and the
> announcement of the Yocto Project and reaching agreement with OpenEmbedded
> (yeah, I was there :) While everyone started creating their own meta layers
> (meta-ti for our BSP and a pair of meta-arago for Apps and Distro), there was
> a void for a free prebuilt ARM cross-compile toolchain. That resulted in a
> public Arago toolchain release [2] that I built and maintained for few years
> (and the corresponding recipe). BTW, back then I had the correct per-component
> Licenses set based on the version [3].

Thanks for sharing this development history.

>
> We had additional requirements for our products to support on-target native
> compilation (external runtime + internal compiler), as well as SDK
> cross-compiling while re-using the same external toolchain (with the option of
> getting standalone toolchain separately from SDK). A lot of extras has been
> added on top of the initial external-toolchain recipe. Most of that was inside
> the Arago Project code base, as that's where it originated. And I have given
> several presentations around that time and following years at conferences on
> the use of external toolchains in OpenEmbedded, using same external toolchains
> in SDKs, dealing with relocation issues, etc. [4] Those weren't very popular
> topics among OE/Yocto crowd, as that camp prefers building toolchains from
> sources :) But our non-OE developers and users demanded prebuilt toolchains
> and maximum re-use of corresponding components, while being flexible shipping
> OE SDK w/o the toolchain pre-integrated due to license restrictions...
>

Okay, I see the reasoning behind toolchain being distributed separately.

> And while Linaro already had their gcc team working on ARM enhancements, they
> didn't have any releases for few years. Thanks to Bill and others, by 2013
> that got corrected and OE recipes based on existing old work were added to
> support the new Linaro toolchain releases. But it only addressed the basic
> cross-compiling of target binaries - no SDK support, no native on-target
> compilation, no extras.

Yes, external-{linaro/arm}-toolchain only provided cross-compilation
of target binaries and corresponding run-time environment as a
replacement of OE toolchain built from source until now when I started
looking at SDK support.

Although, I agree that your downstream project had the complete
feature-set that only re-uses external-{linaro/arm}-toolchain recipe.

> Still, it was time to switch to the new Linaro
> toolchain releases. And meta-linaro was quite open and receptive to patches
> from outside, so I started submitting fixes and enhancements I had accumulated
> in meta-arago over the years. I also had an open communication channel with
> Koen, so that was also helpful to push changes in.

Thanks for all your contributions.

>
> But in recent years things weren't as rosy as they were before - toolchain
> release ownership has changed, there was no clear owner for the recipe to go
> along, so I had to step up my game [5], although I raised the question few
> times whether I should just take over and take the recipe back...
>ipe
> Also, 2 major downsides surfaced with meta-linaro process itself - lack of
> testing of external-toolchain use in OE (besides basics)

I would like to understand what exactly was missing from the testing
perspective regarding supported features?

If you are talking about bits [1] that you added to support native and
SDK compilation in your downstream project then I think they were
essentially broken/partial features from standalone external-toolchain
perspective which can't be tested until one uses your downstream
project.

[1] https://git.linaro.org/openembedded/meta-linaro.git/commit/?id=91ea4d017bf0598e49944e76c889e66d58c066ce

> and no public review
> of Linaro-originated changes by using internal gerrit instead of public
> mailing list. The last one I didn't realize until later time. Often I would
> get an unnoticed run-time breakage introduced by some recent change in the
> recipe that didn't account for all our extensive downstream use cases. As
> there were no patches on the list, no reviews, no heads-up, I would scramble
> around in order to fix new breakages and in most cases would try to submit the
> fix upstream. Only to get them broken later due to the mentioned process
> downsides. I did mention couple of such instances recently [6][7]...

I guess it would have been better if you had added a complete feature
in upstream (say to support SDK or native compilation) instead which
could be tested standalone using external-toolchain recipe rather than
fragile dependencies that could only be tested with your downstream
project.

>
> So, when earlier this year I started working with meta-arm, I was very pleased
> with the open process, where all internal and external patches go to the list
> for review. And I was happy to see OPTEE and ARM toolchain recipes migrating
> there as well [8], so in the past several months I cleaned up and upstreamed
> many hacks and enhancements there from meta-arago [9]. Of course, few were
> still remaining, but the plan was always to get everything upstream.
> Unfortunately, Paul wasn't willing to wait much longer... :)
>
> That "activated" Sumit :) and I understand his position very well and
> appreciate his energy! He's been given a problem and is trying to solve it,
> but he's changing some fundamentals as he's not bound by many use cases.
> Changing those fundamentals actually breaks bunch of my use cases that I'm
> bound to by our product. So, on one hand he's changing things upstream and
> I'm downstream and am supposed to adapt. But on the other hand, I've been
> maintaining and supporting this for 11-12 years and I have tons of baggage in
> form of existing products and multiple use cases he is not considering.

For sure we wouldn't like to break downstream use-cases but I would
like to understand the fundamentals you are referring too and
use-cases which are affected by mine changes.

Patch #1, after discussions from v1, it seems to be already part of
downstream which we can always get rid-off once we have proper glibc
locale packaging support. Is there anything else I missed?
Patch #2, I think it just corrects packaging of libraries and headers
in intended packages rather than in external-arm-toolchain-dev
package.
Patch #3, new patch in v2, is it the contentious one? Is your
downstream project expecting libraries/headers packaged corresponding
to EAT_TARGET_SYS? If yes then I think in order to make it a
compatible change we could try to package corresponding to both
EAT_TARGET_SYS and TARGET_SYS?
Patch #4 and #5, I think they shouldn't be affecting any of your
use-cases, correct?

> Plus,
> it was never clear if there are any other users or consumers of
> external-toolchain, especially to the extent we use it in the Arago Project as
> part of TI SDK products, which means changing things in incompatible way
> breaks the only major consumer of this feature. I fear if we cannot find a
> reasonable technical and architectural consensus for these changes,

I am very much in favour of making compatible and gradual changes. So
I think we should atleast try to reach a consensus prior to predicting
outcomes.

-Sumit

> it may
> result in the opposite outcome from what Paul was seeking - a major fork with
> 2 separate external-toolchain supports. And I'm not trying to make any
> excuses, but the fact that I'm severely burned out right now only exacerbates
> the whole situation, sorry...
>
>
> [1] https://git.openembedded.org/openembedded/log/recipes/meta/external-toolchain-csl.bb
> [2] http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/index_FDS.html
> [3] http://arago-project.org/git/?p=meta-arago.git;a=blob;f=meta-arago-extras/recipes-core/meta/external-arago-toolchain.inc;h=da0d9abd77a17f2334cdea8b5f05ef43a550eaaf;hb=90dc157478439b2facc87a47d8ba8abb596edc6e
> [4] https://elinux.org/images/c/c8/ExternalToolchainsInYocto.pdf
> [5] https://git.linaro.org/openembedded/meta-linaro.git/log/meta-linaro-toolchain/recipes-devtools/external-arm-toolchain/external-arm-toolchain.bb
> [6] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=efd07731659f2e8e8970f1a26e8dacfbd19ce832
> [7] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/commit/meta-arm-toolchain?id=7765d89009f9ace12e600d9e6650fa8f019ba034
> [8] https://lists.yoctoproject.org/g/meta-arm/message/13
> [9] https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/log/meta-arm-toolchain
>
>
> On Tue, May 19, 2020 at 09:03:53PM +0100, Paul Barker wrote:
> > Hi all,
> >
> > As discussed on the Yocto Project call today it's great to see the ARM
> > toolchain recipes moving forward under the new meta-arm layers. I've
> > had some issues in the past with the external pre-built toolchain
> > which really need fixing to properly integrate this toolchain into
> > projects. I'm raising them here in the hopes that these can now be
> > fixed.
> >
> > 1) Maintaining GPL compliance while using an external toolchain
> >
> > The Yocto Project has an archiver class which can be used to collect
> > the source code of all packages built by bitbake which is a
> > requirement for meeting the terms of the GPL. However for the external
> > toolchain recipes the "source" is the pre-built toolchain archive
> > downloaded from the ARM website. As the pre-built toolchain includes
> > glibc (which is then distributed as part of the images we build) what
> > we need is the actual source code used to build the various toolchain
> > components.
> >
> > The toolchain recipe needs to include some link to the actual upstream
> > sources and the archiver class in oe-core probably needs extending to
> > support this.
> >
> > 2) Building an SDK/ESDK
> >
> > If an SDK is generated for an image built with the pre-built toolchain
> > then there can be some confusion about what exactly should provide the
> > gcc binary and various headers. In the past I've had to hack
> > components out of the SDK after it is built and then tell customers to
> > install the pre-built toolchain alongside the SDK in order to get a
> > working cross-compiler. Ideally everything should actually get
> > packaged into the SDK so that only one download is needed and no
> > hard-coded paths are used.
> >
> > Additionally, running the populate_sdk task often fails due to missing
> > packages (libatomic-dev right now) and it looks like the recipe
> > bitrots fairly quickly after each fix. SDK generation needs to be
> > regularly tested and it needs to be confirmed that these SDKs can
> > actually cross-compile working binaries.
> >
> > 3) Toolchain recipe is effectively split between meta-arm & meta-arago layers
> >
> > The bbappends carried in meta-arago fix items which are missing from
> > the external-arm-toolcahin recipe in meta-arg (e.g. the installation
> > of the ldconfig binary). That effectively splits the recipe over
> > multiple layers as these things are pretty fundamental parts of the
> > external-arm-toolchain recipe. An effort needs to be made to merge all
> > this together and have the base recipe well tested.
> >
> > 4) No example configurations or visible test matrix
> >
> > The meta-arm-toolchain layer lacks a README or other file providing
> > some examples of working build configurations which use the pre-built
> > toolchain. It's unclear which MACHINEs and DISTROs are regularly
> > tested and are well supported. The best way to solve this would be to
> > actually store the test scripts and configurations in this layer and
> > use a publicly visible CI system like Travis, GitLab CI, etc to run
> > the tests. That would increase confidence in the pre-built toolchain
> > and make it clear what is supported and how frequently it's tested.
> >
> > 5) LICENSE is incorrectly stated in the external-arm-toolchain recipe
> >
> > The full license of the toolchain package needs to be specified. Right
> > now all packages are marked as having the MIT license which is
> > obviously not correct for most of them.
> >
> > Apologies for the long rant but I hope this is useful to highlight
> > some of the issues seen when trying to practically use the pre-built
> > ARM toolchain.
> >
> > Thanks,
> >
> > --
> > Paul Barker
> > Konsulko Group
>
> >
>
> 

Re: [meta-arm] External toolchain support

[Denys Dmytriyenko]

On Thu, Jun 11, 2020 at 10:27:50AM +0100, Paul Barker wrote:
> On Thu, 11 Jun 2020 at 03:01, Denys Dmytriyenko <denis@denix.org> wrote:
> >
> > All,
> 
> Denys,
> 
> Thanks for the context here, it's really valuable to understand how
> things have developed. I'd prefer that things improve slowly than that
> we have a breaking fork.

Thank your for understanding!


> Can we identify which of the proposed changes can be made with low
> risk? I think fixing the readme and licensing at least should be
> straightforward.

I guess those can be considered low-impact.


> I'll look at the archiver support myself and see if I can find a
> solution for that.

That would be nice, thanks

-- 
Denys

Re: [meta-arm] External toolchain support

[Denys Dmytriyenko]

On Thu, Jun 11, 2020 at 07:44:26PM +0530, Sumit Garg wrote:
> On Thu, 11 Jun 2020 at 07:31, Denys Dmytriyenko <denis@denix.org> wrote:
> >
> > We had additional requirements for our products to support on-target native
> > compilation (external runtime + internal compiler), as well as SDK
> > cross-compiling while re-using the same external toolchain (with the option of
> > getting standalone toolchain separately from SDK). A lot of extras has been
> > added on top of the initial external-toolchain recipe. Most of that was inside
> > the Arago Project code base, as that's where it originated. And I have given
> > several presentations around that time and following years at conferences on
> > the use of external toolchains in OpenEmbedded, using same external toolchains
> > in SDKs, dealing with relocation issues, etc. [4] Those weren't very popular
> > topics among OE/Yocto crowd, as that camp prefers building toolchains from
> > sources :) But our non-OE developers and users demanded prebuilt toolchains
> > and maximum re-use of corresponding components, while being flexible shipping
> > OE SDK w/o the toolchain pre-integrated due to license restrictions...
> 
> Okay, I see the reasoning behind toolchain being distributed separately.

Correct. While runtime is mostly fine (L/GPLv2 for glibc and RLE exception for 
libgcc/libstdc++, which makes GPLv3 bearable) it is not the case for gcc, gdb 
and binutils binaries, which are mostly GPLv3 now. Originally we were avoiding 
distributing GPLv3 components, but eventually that got sorted out. Hence the 
requirement to be able to distribute OE SDK w/o pre-integrated toolchain. That 
way customers can download the toolchain on their own to use with that SDK.


> > And while Linaro already had their gcc team working on ARM enhancements, they
> > didn't have any releases for few years. Thanks to Bill and others, by 2013
> > that got corrected and OE recipes based on existing old work were added to
> > support the new Linaro toolchain releases. But it only addressed the basic
> > cross-compiling of target binaries - no SDK support, no native on-target
> > compilation, no extras.
> 
> Yes, external-{linaro/arm}-toolchain only provided cross-compilation
> of target binaries and corresponding run-time environment as a
> replacement of OE toolchain built from source until now when I started
> looking at SDK support.
> 
> Although, I agree that your downstream project had the complete
> feature-set that only re-uses external-{linaro/arm}-toolchain recipe.
> 
> > But in recent years things weren't as rosy as they were before - toolchain
> > release ownership has changed, there was no clear owner for the recipe to go
> > along, so I had to step up my game [5], although I raised the question few
> > times whether I should just take over and take the recipe back...
> >ipe
> > Also, 2 major downsides surfaced with meta-linaro process itself - lack of
> > testing of external-toolchain use in OE (besides basics)
> 
> I would like to understand what exactly was missing from the testing
> perspective regarding supported features?

Well, supported basic features were fine with gcc9 upgrade, it was much worse 
with gcc8 upgrade, for example:
https://git.linaro.org/openembedded/meta-linaro.git/commit/?id=a8ce3dc47f3be9f98abc6ac98a849f918386cf9f


> If you are talking about bits [1] that you added to support native and
> SDK compilation in your downstream project then I think they were
> essentially broken/partial features from standalone external-toolchain
> perspective which can't be tested until one uses your downstream
> project.
> 
> [1] https://git.linaro.org/openembedded/meta-linaro.git/commit/?id=91ea4d017bf0598e49944e76c889e66d58c066ce
> 
> > and no public review
> > of Linaro-originated changes by using internal gerrit instead of public
> > mailing list. The last one I didn't realize until later time. Often I would
> > get an unnoticed run-time breakage introduced by some recent change in the
> > recipe that didn't account for all our extensive downstream use cases. As
> > there were no patches on the list, no reviews, no heads-up, I would scramble
> > around in order to fix new breakages and in most cases would try to submit the
> > fix upstream. Only to get them broken later due to the mentioned process
> > downsides. I did mention couple of such instances recently [6][7]...
> 
> I guess it would have been better if you had added a complete feature
> in upstream (say to support SDK or native compilation) instead which
> could be tested standalone using external-toolchain recipe rather than
> fragile dependencies that could only be tested with your downstream
> project.

Well, it's not like I didn't try... :)

Some things were more straightforward and I had those upstreamed for the most 
part. But other things were more controversial, when I was discussing those 
back in the day, like external runtime + internal compiler or OE-built SDK + 
external compiler. From OE perspective external toolchains are evil (or at 
least a very foreign concept), so once you start mixing things, people get 
upset, hence the pushback. That made those features scattered between upstream 
and downstream, unfortunately.

Plus I had some setbacks with aforementioned breakages...

But I still plan to contribute and if we find a consensus, I'd be happy to get 
the rest of the features upstreamed.


> > That "activated" Sumit :) and I understand his position very well and
> > appreciate his energy! He's been given a problem and is trying to solve it,
> > but he's changing some fundamentals as he's not bound by many use cases.
> > Changing those fundamentals actually breaks bunch of my use cases that I'm
> > bound to by our product. So, on one hand he's changing things upstream and
> > I'm downstream and am supposed to adapt. But on the other hand, I've been
> > maintaining and supporting this for 11-12 years and I have tons of baggage in
> > form of existing products and multiple use cases he is not considering.
> 
> For sure we wouldn't like to break downstream use-cases but I would
> like to understand the fundamentals you are referring too and
> use-cases which are affected by mine changes.
> 
> Patch #1, after discussions from v1, it seems to be already part of
> downstream which we can always get rid-off once we have proper glibc
> locale packaging support. Is there anything else I missed?
> Patch #2, I think it just corrects packaging of libraries and headers
> in intended packages rather than in external-arm-toolchain-dev
> package.
> Patch #3, new patch in v2, is it the contentious one? Is your
> downstream project expecting libraries/headers packaged corresponding
> to EAT_TARGET_SYS? If yes then I think in order to make it a
> compatible change we could try to package corresponding to both
> EAT_TARGET_SYS and TARGET_SYS?
> Patch #4 and #5, I think they shouldn't be affecting any of your
> use-cases, correct?

Yeah, #3 will be the most problematic - I did provide more details to the 
patch itself.


> > Plus,
> > it was never clear if there are any other users or consumers of
> > external-toolchain, especially to the extent we use it in the Arago Project as
> > part of TI SDK products, which means changing things in incompatible way
> > breaks the only major consumer of this feature. I fear if we cannot find a
> > reasonable technical and architectural consensus for these changes,
> 
> I am very much in favour of making compatible and gradual changes. So
> I think we should atleast try to reach a consensus prior to predicting
> outcomes.

Sorry, didn't mean to be so dramatic. The main goal of my post was to add 
perspective and highlight my use cases. I was worried my review comments and 
any negative feedback could have been considered baseless. Hopefully this now 
clarifies few things and provides previously missing reasoning and context. 
And I'd be happy to answer any remaining questions.

-- 
Denys

Re: [meta-arm] External toolchain support

[Denys Dmytriyenko]

On Fri, Jun 12, 2020 at 01:34:02AM -0400, Denys Dmytriyenko wrote:
> On Thu, Jun 11, 2020 at 07:44:26PM +0530, Sumit Garg wrote:
> > On Thu, 11 Jun 2020 at 07:31, Denys Dmytriyenko <denis@denix.org> wrote:
> > >
> > > and no public review
> > > of Linaro-originated changes by using internal gerrit instead of public
> > > mailing list. The last one I didn't realize until later time. Often I would
> > > get an unnoticed run-time breakage introduced by some recent change in the
> > > recipe that didn't account for all our extensive downstream use cases. As
> > > there were no patches on the list, no reviews, no heads-up, I would scramble
> > > around in order to fix new breakages and in most cases would try to submit the
> > > fix upstream. Only to get them broken later due to the mentioned process
> > > downsides. I did mention couple of such instances recently [6][7]...
> > 
> > I guess it would have been better if you had added a complete feature
> > in upstream (say to support SDK or native compilation) instead which
> > could be tested standalone using external-toolchain recipe rather than
> > fragile dependencies that could only be tested with your downstream
> > project.
> 
> Well, it's not like I didn't try... :)
> 
> Some things were more straightforward and I had those upstreamed for the most 
> part. But other things were more controversial, when I was discussing those 
> back in the day, like external runtime + internal compiler or OE-built SDK + 
> external compiler. From OE perspective external toolchains are evil (or at 
> least a very foreign concept), so once you start mixing things, people get 
> upset, hence the pushback. That made those features scattered between upstream 
> and downstream, unfortunately.

Oh, forgot to mention! To support those extra features, changes had to be made 
outside of external-toolchain recipe, which may have contributed to them being 
considered controversial...

For example, external-arm-toolchain recipe only packages runtimes (headers, 
libs, stubs, target binaries), but not the host/cross binaries, as it 
basically extends glibc-package.inc from OE-Core. I have an additional recipe 
called external-arm-sdk-toolchain (not the best name), that packages prebuilt 
gcc, gdb, binutils binaries from EAT into proper *-cross-canadian-$ARCH 
packages, which could then be easily pulled into SDK.

We've been using meta-toolchain way of building SDKs from the early days, 
before "<rootfs> -c populate_sdk" was introduced. Some of the toolchain/SDK 
interaction and processing was added there, like packaging SDK w/o the 
toolchain, or establishing some symlinks between OE compiler and EAT runtime. 
Those don't make sense for the rootfs, hence meta-toolchain based SDK remained 
in use for us. Plus we build multiple rootfs images with different content, 
but a single complete SDK to cover them all, so "<rootfs> -c populate_sdk" to 
match the rootfs didn't make sense for our product...

I don't mean this to block a more appropriate re-implementation upstream, if 
needed, of course. I'm open to collaboration and re-design, but would like 
this to be a two-way road. Thanks.


> Plus I had some setbacks with aforementioned breakages...
> 
> But I still plan to contribute and if we find a consensus, I'd be happy to get 
> the rest of the features upstreamed.

-- 
Denys

Re: [meta-arm] External toolchain support

[Sumit Garg]

On Fri, 12 Jun 2020 at 11:04, Denys Dmytriyenko <denis@denix.org> wrote:
>
> On Thu, Jun 11, 2020 at 07:44:26PM +0530, Sumit Garg wrote:
> > On Thu, 11 Jun 2020 at 07:31, Denys Dmytriyenko <denis@denix.org> wrote:
> > >
> > > We had additional requirements for our products to support on-target native
> > > compilation (external runtime + internal compiler), as well as SDK
> > > cross-compiling while re-using the same external toolchain (with the option of
> > > getting standalone toolchain separately from SDK). A lot of extras has been
> > > added on top of the initial external-toolchain recipe. Most of that was inside
> > > the Arago Project code base, as that's where it originated. And I have given
> > > several presentations around that time and following years at conferences on
> > > the use of external toolchains in OpenEmbedded, using same external toolchains
> > > in SDKs, dealing with relocation issues, etc. [4] Those weren't very popular
> > > topics among OE/Yocto crowd, as that camp prefers building toolchains from
> > > sources :) But our non-OE developers and users demanded prebuilt toolchains
> > > and maximum re-use of corresponding components, while being flexible shipping
> > > OE SDK w/o the toolchain pre-integrated due to license restrictions...
> >
> > Okay, I see the reasoning behind toolchain being distributed separately.
>
> Correct. While runtime is mostly fine (L/GPLv2 for glibc and RLE exception for
> libgcc/libstdc++, which makes GPLv3 bearable) it is not the case for gcc, gdb
> and binutils binaries, which are mostly GPLv3 now. Originally we were avoiding
> distributing GPLv3 components, but eventually that got sorted out. Hence the
> requirement to be able to distribute OE SDK w/o pre-integrated toolchain. That
> way customers can download the toolchain on their own to use with that SDK.
>

Sounds reasonable.

>
> > > And while Linaro already had their gcc team working on ARM enhancements, they
> > > didn't have any releases for few years. Thanks to Bill and others, by 2013
> > > that got corrected and OE recipes based on existing old work were added to
> > > support the new Linaro toolchain releases. But it only addressed the basic
> > > cross-compiling of target binaries - no SDK support, no native on-target
> > > compilation, no extras.
> >
> > Yes, external-{linaro/arm}-toolchain only provided cross-compilation
> > of target binaries and corresponding run-time environment as a
> > replacement of OE toolchain built from source until now when I started
> > looking at SDK support.
> >
> > Although, I agree that your downstream project had the complete
> > feature-set that only re-uses external-{linaro/arm}-toolchain recipe.
> >
> > > But in recent years things weren't as rosy as they were before - toolchain
> > > release ownership has changed, there was no clear owner for the recipe to go
> > > along, so I had to step up my game [5], although I raised the question few
> > > times whether I should just take over and take the recipe back...
> > >ipe
> > > Also, 2 major downsides surfaced with meta-linaro process itself - lack of
> > > testing of external-toolchain use in OE (besides basics)
> >
> > I would like to understand what exactly was missing from the testing
> > perspective regarding supported features?
>
> Well, supported basic features were fine with gcc9 upgrade, it was much worse
> with gcc8 upgrade, for example:
> https://git.linaro.org/openembedded/meta-linaro.git/commit/?id=a8ce3dc47f3be9f98abc6ac98a849f918386cf9f
>

Although, I wasn't involved during the gcc8 upgrade but I agree with
you that it was fragile. AFAIK, the main reason for this gcc8 upgrade
state was due to transfer of toolchain releases ownership from Linaro
to Arm which in turn led to drastic changes to how toolchain
components were packaged making reuse of existing
external-linaro-toolchain difficult.

And my involvement started with gcc9 upgrade with a mission to do
fixups and cleanups in order to provide a well tested (for supported
features) external-arm-toolchain recipe.

>
> > If you are talking about bits [1] that you added to support native and
> > SDK compilation in your downstream project then I think they were
> > essentially broken/partial features from standalone external-toolchain
> > perspective which can't be tested until one uses your downstream
> > project.
> >
> > [1] https://git.linaro.org/openembedded/meta-linaro.git/commit/?id=91ea4d017bf0598e49944e76c889e66d58c066ce
> >
> > > and no public review
> > > of Linaro-originated changes by using internal gerrit instead of public
> > > mailing list. The last one I didn't realize until later time. Often I would
> > > get an unnoticed run-time breakage introduced by some recent change in the
> > > recipe that didn't account for all our extensive downstream use cases. As
> > > there were no patches on the list, no reviews, no heads-up, I would scramble
> > > around in order to fix new breakages and in most cases would try to submit the
> > > fix upstream. Only to get them broken later due to the mentioned process
> > > downsides. I did mention couple of such instances recently [6][7]...
> >
> > I guess it would have been better if you had added a complete feature
> > in upstream (say to support SDK or native compilation) instead which
> > could be tested standalone using external-toolchain recipe rather than
> > fragile dependencies that could only be tested with your downstream
> > project.
>
> Well, it's not like I didn't try... :)
>
> Some things were more straightforward and I had those upstreamed for the most
> part. But other things were more controversial, when I was discussing those
> back in the day, like external runtime + internal compiler or OE-built SDK +
> external compiler. From OE perspective external toolchains are evil (or at
> least a very foreign concept), so once you start mixing things, people get
> upset, hence the pushback. That made those features scattered between upstream
> and downstream, unfortunately.
>
> Plus I had some setbacks with aforementioned breakages...
>
> But I still plan to contribute and if we find a consensus, I'd be happy to get
> the rest of the features upstreamed.
>
>
> > > That "activated" Sumit :) and I understand his position very well and
> > > appreciate his energy! He's been given a problem and is trying to solve it,
> > > but he's changing some fundamentals as he's not bound by many use cases.
> > > Changing those fundamentals actually breaks bunch of my use cases that I'm
> > > bound to by our product. So, on one hand he's changing things upstream and
> > > I'm downstream and am supposed to adapt. But on the other hand, I've been
> > > maintaining and supporting this for 11-12 years and I have tons of baggage in
> > > form of existing products and multiple use cases he is not considering.
> >
> > For sure we wouldn't like to break downstream use-cases but I would
> > like to understand the fundamentals you are referring too and
> > use-cases which are affected by mine changes.
> >
> > Patch #1, after discussions from v1, it seems to be already part of
> > downstream which we can always get rid-off once we have proper glibc
> > locale packaging support. Is there anything else I missed?
> > Patch #2, I think it just corrects packaging of libraries and headers
> > in intended packages rather than in external-arm-toolchain-dev
> > package.
> > Patch #3, new patch in v2, is it the contentious one? Is your
> > downstream project expecting libraries/headers packaged corresponding
> > to EAT_TARGET_SYS? If yes then I think in order to make it a
> > compatible change we could try to package corresponding to both
> > EAT_TARGET_SYS and TARGET_SYS?
> > Patch #4 and #5, I think they shouldn't be affecting any of your
> > use-cases, correct?
>
> Yeah, #3 will be the most problematic - I did provide more details to the
> patch itself.

As per our communication in the other thread, I guess we are reaching
consensus for an additional change to support your use-cases.

>
>
> > > Plus,
> > > it was never clear if there are any other users or consumers of
> > > external-toolchain, especially to the extent we use it in the Arago Project as
> > > part of TI SDK products, which means changing things in incompatible way
> > > breaks the only major consumer of this feature. I fear if we cannot find a
> > > reasonable technical and architectural consensus for these changes,
> >
> > I am very much in favour of making compatible and gradual changes. So
> > I think we should atleast try to reach a consensus prior to predicting
> > outcomes.
>
> Sorry, didn't mean to be so dramatic. The main goal of my post was to add
> perspective and highlight my use cases. I was worried my review comments and
> any negative feedback could have been considered baseless. Hopefully this now
> clarifies few things and provides previously missing reasoning and context.
> And I'd be happy to answer any remaining questions.
>

No worries :). And yes this thread provided meaningful insights into
your downstream use-cases.

-Sumit

> --
> Denys

Re: [meta-arm] External toolchain support

[Sumit Garg]

On Fri, 12 Jun 2020 at 12:36, Denys Dmytriyenko <denis@denix.org> wrote:
>
> On Fri, Jun 12, 2020 at 01:34:02AM -0400, Denys Dmytriyenko wrote:
> > On Thu, Jun 11, 2020 at 07:44:26PM +0530, Sumit Garg wrote:
> > > On Thu, 11 Jun 2020 at 07:31, Denys Dmytriyenko <denis@denix.org> wrote:
> > > >
> > > > and no public review
> > > > of Linaro-originated changes by using internal gerrit instead of public
> > > > mailing list. The last one I didn't realize until later time. Often I would
> > > > get an unnoticed run-time breakage introduced by some recent change in the
> > > > recipe that didn't account for all our extensive downstream use cases. As
> > > > there were no patches on the list, no reviews, no heads-up, I would scramble
> > > > around in order to fix new breakages and in most cases would try to submit the
> > > > fix upstream. Only to get them broken later due to the mentioned process
> > > > downsides. I did mention couple of such instances recently [6][7]...
> > >
> > > I guess it would have been better if you had added a complete feature
> > > in upstream (say to support SDK or native compilation) instead which
> > > could be tested standalone using external-toolchain recipe rather than
> > > fragile dependencies that could only be tested with your downstream
> > > project.
> >
> > Well, it's not like I didn't try... :)
> >
> > Some things were more straightforward and I had those upstreamed for the most
> > part. But other things were more controversial, when I was discussing those
> > back in the day, like external runtime + internal compiler or OE-built SDK +
> > external compiler. From OE perspective external toolchains are evil (or at
> > least a very foreign concept), so once you start mixing things, people get
> > upset, hence the pushback. That made those features scattered between upstream
> > and downstream, unfortunately.
>
> Oh, forgot to mention! To support those extra features, changes had to be made
> outside of external-toolchain recipe, which may have contributed to them being
> considered controversial...
>
> For example, external-arm-toolchain recipe only packages runtimes (headers,
> libs, stubs, target binaries), but not the host/cross binaries, as it
> basically extends glibc-package.inc from OE-Core. I have an additional recipe
> called external-arm-sdk-toolchain (not the best name), that packages prebuilt
> gcc, gdb, binutils binaries from EAT into proper *-cross-canadian-$ARCH
> packages, which could then be easily pulled into SDK.
>
> We've been using meta-toolchain way of building SDKs from the early days,
> before "<rootfs> -c populate_sdk" was introduced. Some of the toolchain/SDK
> interaction and processing was added there, like packaging SDK w/o the
> toolchain, or establishing some symlinks between OE compiler and EAT runtime.
> Those don't make sense for the rootfs, hence meta-toolchain based SDK remained
> in use for us. Plus we build multiple rootfs images with different content,
> but a single complete SDK to cover them all, so "<rootfs> -c populate_sdk" to
> match the rootfs didn't make sense for our product...
>
> I don't mean this to block a more appropriate re-implementation upstream, if
> needed, of course. I'm open to collaboration and re-design, but would like
> this to be a two-way road. Thanks.
>

Thanks for the detailed insights on this. Yes I think we should be
able to support both ways of SDK packaging.

-Sumit

>
> > Plus I had some setbacks with aforementioned breakages...
> >
> > But I still plan to contribute and if we find a consensus, I'd be happy to get
> > the rest of the features upstreamed.
>
> --
> Denys