[Buildroot] [FYI] Bootlin external toolchains

[Buildroot] [FYI] Bootlin external toolchains

[Vadim Kochan]

Hi All,

Just in case if someone will be interested. I started Buildroot external
project which provides external toolchains from the Bootlin:

    https://toolchains.bootlin.com

The git repo can be found at:

    https://github.com/vkochan/buildroot-external-toolchain-bootlin

Regards,
Vadim Kochan

[Buildroot] [FYI] Bootlin external toolchains

[Thomas Petazzoni]

Hello,

+Romain Naour in Cc.

On Mon, 2 Dec 2019 02:47:15 +0200
Vadim Kochan <vadim4j@gmail.com> wrote:

> Just in case if someone will be interested. I started Buildroot external
> project which provides external toolchains from the Bootlin:
> 
>     https://toolchains.bootlin.com
> 
> The git repo can be found at:
> 
>     https://github.com/vkochan/buildroot-external-toolchain-bootlin

An effort was started to package the Bootlin external toolchains in
Buildroot itself, some time ago by engineers at Smile. We had some
off-list discussion about this, but I don't think it was ever posted to
the list. The idea back then was to have a script that generates all
the packages, but the design was to have one package for each
toolchain, so it really added a lot of packages.

But I think we should really support these toolchains in Buildroot
proper instead of having an external for them.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [FYI] Bootlin external toolchains

[Romain Naour]

Hi Thomas, Vadim,

+Arthur Courtel in Cc.

Le 02/12/2019 ? 15:26, Thomas Petazzoni a ?crit?:
> Hello,
> 
> +Romain Naour in Cc.
> 
> On Mon, 2 Dec 2019 02:47:15 +0200
> Vadim Kochan <vadim4j@gmail.com> wrote:
> 
>> Just in case if someone will be interested. I started Buildroot external
>> project which provides external toolchains from the Bootlin:
>>
>>     https://toolchains.bootlin.com
>>
>> The git repo can be found at:
>>
>>     https://github.com/vkochan/buildroot-external-toolchain-bootlin
> 
> An effort was started to package the Bootlin external toolchains in
> Buildroot itself, some time ago by engineers at Smile. We had some
> off-list discussion about this, but I don't think it was ever posted to
> the list. The idea back then was to have a script that generates all
> the packages, but the design was to have one package for each
> toolchain, so it really added a lot of packages.
> 
> But I think we should really support these toolchains in Buildroot
> proper instead of having an external for them.

Agree.

Arthur will take a look at the script but it's very complicated script...
Maybe we need to discuss what we really want.

Best regards,
Romain


> 
> Best regards,
> 
> Thomas
> 

[Buildroot] [FYI] Bootlin external toolchains

[Vadim Kochan]

Hi All,

On Tue, Dec 3, 2019 at 11:59 AM Romain Naour <romain.naour@smile.fr> wrote:
>
> Hi Thomas, Vadim,
>
> +Arthur Courtel in Cc.
>
> Le 02/12/2019 ? 15:26, Thomas Petazzoni a ?crit :
> > Hello,
> >
> > +Romain Naour in Cc.
> >
> > On Mon, 2 Dec 2019 02:47:15 +0200
> > Vadim Kochan <vadim4j@gmail.com> wrote:
> >
> >> Just in case if someone will be interested. I started Buildroot external
> >> project which provides external toolchains from the Bootlin:
> >>
> >>     https://toolchains.bootlin.com
> >>
> >> The git repo can be found at:
> >>
> >>     https://github.com/vkochan/buildroot-external-toolchain-bootlin
> >
> > An effort was started to package the Bootlin external toolchains in
> > Buildroot itself, some time ago by engineers at Smile. We had some
> > off-list discussion about this, but I don't think it was ever posted to
> > the list. The idea back then was to have a script that generates all
> > the packages, but the design was to have one package for each
> > toolchain, so it really added a lot of packages.
> >
> > But I think we should really support these toolchains in Buildroot
> > proper instead of having an external for them.
>
> Agree.
>
> Arthur will take a look at the script but it's very complicated script...
> Maybe we need to discuss what we really want.
>
> Best regards,
> Romain

May be it would be good to start with generic script which generates
br2_fragment by specifying the path to the toolchain ?

Regards,
Vadim Kochan

[Buildroot] [FYI] Bootlin external toolchains

[Thomas Petazzoni]

On Tue, 3 Dec 2019 12:07:51 +0200
Vadim Kochan <vadim4j@gmail.com> wrote:

> May be it would be good to start with generic script which generates
> br2_fragment by specifying the path to the toolchain ?

Such fragments already exist for each toolchain, for example:

  https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--stable-2018.11-1.frag

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [FYI] Bootlin external toolchains

[Vadim Kochan]

Hi Thomas, All

On Tue, Dec 3, 2019 at 1:00 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Tue, 3 Dec 2019 12:07:51 +0200
> Vadim Kochan <vadim4j@gmail.com> wrote:
>
> > May be it would be good to start with generic script which generates
> > br2_fragment by specifying the path to the toolchain ?
>
> Such fragments already exist for each toolchain, for example:
>
>   https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--stable-2018.11-1.frag
>

Yes, I already used them) I meant, it would be good to have such
script in Buildroot which allows to generate such fragments for any
toolchain.
The existing script which validates the toolchain (during the build
before the installation into the host part) uses some set of checks -
rpc, wchar, threads supporting.
So, it would be good to use such approach for the toolchain fragment
generation. Or such script already exist ?

> Best regards,
>
> Thomas

Regards,
Vadim Kochan

[Buildroot] [FYI] Bootlin external toolchains

[Thomas Petazzoni]

Hello,

On Tue, 3 Dec 2019 13:06:18 +0200
Vadim Kochan <vadim4j@gmail.com> wrote:

> > Such fragments already exist for each toolchain, for example:
> >
> >   https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--stable-2018.11-1.frag
> >  
> 
> Yes, I already used them) I meant, it would be good to have such
> script in Buildroot which allows to generate such fragments for any
> toolchain.

Well, the script you're talking about, and the script that generates
the above fragment are very different.

The fragment above is generated together with the toolchain, when we
know what is the Buildroot configuration used to build the toolchain.

If I understand correctly, what you propose is instead a script that
given some random toolchain (generated by Buildroot, crosstool-ng, or
any other mean), will output a snippet of Buildroot configuration that
matches this toolchain.

We did have such a proposal some time ago on the mailing list, and
there was some discussion around it. I can't remember if it was
abandoned because nobody cared enough, or because there were some real
technical issues. Unfortunately, I was not able to quickly locate the
patch I'm talking about :-/

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [FYI] Bootlin external toolchains

[Arnout Vandecappelle]

On 03/12/2019 14:57, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 3 Dec 2019 13:06:18 +0200
> Vadim Kochan <vadim4j@gmail.com> wrote:
> 
>>> Such fragments already exist for each toolchain, for example:
>>>
>>>   https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/fragments/aarch64--glibc--stable-2018.11-1.frag
>>>  
>>
>> Yes, I already used them) I meant, it would be good to have such
>> script in Buildroot which allows to generate such fragments for any
>> toolchain.
> 
> Well, the script you're talking about, and the script that generates
> the above fragment are very different.
> 
> The fragment above is generated together with the toolchain, when we
> know what is the Buildroot configuration used to build the toolchain.

 I don't know if it is what Vadim proposed, but what _I_ propose is that we add
a script that generates such a fragment from the buildroot .config. And I would
even invoke that script unconditionally from toolchain-buildroot.mk, and put the
resulting fragment in $(HOST_DIR)/share/buildroot/toolchain.config-frag.

 The script exists already: function make_br_fragment in [1]. It should just
move to the Buildroot repo.

> 
> If I understand correctly, what you propose is instead a script that
> given some random toolchain (generated by Buildroot, crosstool-ng, or
> any other mean), will output a snippet of Buildroot configuration that
> matches this toolchain.
> 
> We did have such a proposal some time ago on the mailing list, and
> there was some discussion around it. 

 Hm, I don't remember that discussion at all, or I would have added the
statement above...

 Regards,
 Arnout

> I can't remember if it was
> abandoned because nobody cared enough, or because there were some real
> technical issues. Unfortunately, I was not able to quickly locate the
> patch I'm talking about :-/
> 
> Thomas

[1] https://gitlab.com/bootlin/toolchains-builder/blob/master/build.sh

[Buildroot] [FYI] Bootlin external toolchains

[Arnout Vandecappelle]

On 03/12/2019 10:59, Romain Naour wrote:
> Hi Thomas, Vadim,
> 
> +Arthur Courtel in Cc.
> 
> Le 02/12/2019 ? 15:26, Thomas Petazzoni a ?crit?:
>> Hello,
>>
>> +Romain Naour in Cc.
>>
>> On Mon, 2 Dec 2019 02:47:15 +0200
>> Vadim Kochan <vadim4j@gmail.com> wrote:
>>
>>> Just in case if someone will be interested. I started Buildroot external
>>> project which provides external toolchains from the Bootlin:
>>>
>>>     https://toolchains.bootlin.com
>>>
>>> The git repo can be found at:
>>>
>>>     https://github.com/vkochan/buildroot-external-toolchain-bootlin
>>
>> An effort was started to package the Bootlin external toolchains in
>> Buildroot itself, some time ago by engineers at Smile. We had some
>> off-list discussion about this, but I don't think it was ever posted to
>> the list. The idea back then was to have a script that generates all
>> the packages, but the design was to have one package for each
>> toolchain, so it really added a lot of packages.
>>
>> But I think we should really support these toolchains in Buildroot
>> proper instead of having an external for them.
> 
> Agree.
> 
> Arthur will take a look at the script but it's very complicated script...
> Maybe we need to discuss what we really want.

 I think this should be like scancpan/scanpypi, i.e. something that generates
the toolchain-external-bootlin package but it's up to the user to fix up the
final details.

 And then we can work on toolchains-builder to make sure it generates something
that is easier for this scanbootlin script to parse.

 Alternatively, maybe even better, we can just update build.sh in
toolchains-builder to generate all the pieces and then add an additional
pipeline stage that collects the pieces into a package.

 Most importantly, though, I think we should take this gradually. Start with a
manually constructed package, it doesn't even need to support all the different
toolchains. We can bikeshed a bit on what that package looks like, and use that
as input for how to script it.

 To kick off the bikeshedding, here's my opinion on what the package should look
like.

* It should be a single package, toolchain-external-bootlin. Maybe two (-stable
and -bleeding-edge) but I think there'll be enough shared infrastructure to
warrant creating a single package.

* It should offer a choice in Config.in.options for libc and stable/bleeding-edge.

* For now, it should be limited to *exactly* the same architecture suboptions as
what the toolchain was built for. Later we can decide how to expand this to
compatible options (e.g. the core-i7 toolchain will also work for core-avx2) -
but I'm pretty sure that that is something that will require help from
toolchain-builder itself.


 Regards,
 Arnout

[Buildroot] [FYI] Bootlin external toolchains

[Romain Naour]

Hi Arnout, All,

Le 03/12/2019 ? 22:15, Arnout Vandecappelle a ?crit?:
> 
> 
> On 03/12/2019 10:59, Romain Naour wrote:
>> Hi Thomas, Vadim,
>>
>> +Arthur Courtel in Cc.
>>
>> Le 02/12/2019 ? 15:26, Thomas Petazzoni a ?crit?:
>>> Hello,
>>>
>>> +Romain Naour in Cc.
>>>
>>> On Mon, 2 Dec 2019 02:47:15 +0200
>>> Vadim Kochan <vadim4j@gmail.com> wrote:
>>>
>>>> Just in case if someone will be interested. I started Buildroot external
>>>> project which provides external toolchains from the Bootlin:
>>>>
>>>>     https://toolchains.bootlin.com
>>>>
>>>> The git repo can be found at:
>>>>
>>>>     https://github.com/vkochan/buildroot-external-toolchain-bootlin
>>>
>>> An effort was started to package the Bootlin external toolchains in
>>> Buildroot itself, some time ago by engineers at Smile. We had some
>>> off-list discussion about this, but I don't think it was ever posted to
>>> the list. The idea back then was to have a script that generates all
>>> the packages, but the design was to have one package for each
>>> toolchain, so it really added a lot of packages.
>>>
>>> But I think we should really support these toolchains in Buildroot
>>> proper instead of having an external for them.
>>
>> Agree.
>>
>> Arthur will take a look at the script but it's very complicated script...
>> Maybe we need to discuss what we really want.
> 
>  I think this should be like scancpan/scanpypi, i.e. something that generates
> the toolchain-external-bootlin package but it's up to the user to fix up the
> final details.
> 
>  And then we can work on toolchains-builder to make sure it generates something
> that is easier for this scanbootlin script to parse.

The initial goal was to use toolchain bootlin as prebuilt external toolchain
when no other prebuilt toolchain is available (Linaro, Mentor etc...).
I would like to avoid letting the user fixing the final details.

> 
>  Alternatively, maybe even better, we can just update build.sh in
> toolchains-builder to generate all the pieces and then add an additional
> pipeline stage that collects the pieces into a package.

The script evoked by Thomas showed that it was too difficult/complex to generate
a toolchain packages from the files currently provided by toolchain-builder.

So, I'm agree that it would be better if toolchain-builder can provide such package.

> 
>  Most importantly, though, I think we should take this gradually. Start with a
> manually constructed package, it doesn't even need to support all the different
> toolchains. We can bikeshed a bit on what that package looks like, and use that
> as input for how to script it.

At least aarch64, arm and x86_64 should be included in the manually constructed
package.

> 
>  To kick off the bikeshedding, here's my opinion on what the package should look
> like.
> 
> * It should be a single package, toolchain-external-bootlin. Maybe two (-stable
> and -bleeding-edge) but I think there'll be enough shared infrastructure to
> warrant creating a single package.

The most difficult part is to generate the dependencies (depends on) of the
toolchain and its properties (select). That is why our experimental script
create one package for each toolchain provided by toolchain Builder.

Also if we can use a single a single package for the Bootlin toolchain package,
it mean that we can also use a single package for the
Linaro/ARM/Codescape/Codesourcery packages.

> 
> * It should offer a choice in Config.in.options for libc and stable/bleeding-edge.
> 
> * For now, it should be limited to *exactly* the same architecture suboptions as
> what the toolchain was built for. Later we can decide how to expand this to
> compatible options (e.g. the core-i7 toolchain will also work for core-avx2) -
> but I'm pretty sure that that is something that will require help from
> toolchain-builder itself.

Indeed.

Best regards,
Romain

> 
> 
>  Regards,
>  Arnout
> 

[Buildroot] [FYI] Bootlin external toolchains

[Thomas Petazzoni]

Hello,

On Tue, 3 Dec 2019 22:15:47 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

>  To kick off the bikeshedding, here's my opinion on what the package should look
> like.
> 
> * It should be a single package, toolchain-external-bootlin. Maybe two (-stable
> and -bleeding-edge) but I think there'll be enough shared infrastructure to
> warrant creating a single package.

Agreed. My initial start of work in this area looked exactly like this:
a single package. See
https://github.com/tpetazzoni/buildroot/commit/58413ab16c6b203dca48db182e74e59ae73c958a.
Note: this commit is far from being something that works. If I had
something that worked, I would have submitted it.

> * It should offer a choice in Config.in.options for libc and stable/bleeding-edge.

See commit above :-)

> * For now, it should be limited to *exactly* the same architecture suboptions as
> what the toolchain was built for. Later we can decide how to expand this to
> compatible options (e.g. the core-i7 toolchain will also work for core-avx2) -
> but I'm pretty sure that that is something that will require help from
> toolchain-builder itself.

Pretty much what I did, yes :-)

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [FYI] Bootlin external toolchains

[Arnout Vandecappelle]

On 04/12/2019 22:52, Romain Naour wrote:
> Hi Arnout, All,
> 
> Le 03/12/2019 ? 22:15, Arnout Vandecappelle a ?crit?:
>>
>>
>> On 03/12/2019 10:59, Romain Naour wrote:
>>> Hi Thomas, Vadim,
>>>
>>> +Arthur Courtel in Cc.
>>>
>>> Le 02/12/2019 ? 15:26, Thomas Petazzoni a ?crit?:
>>>> Hello,
>>>>
>>>> +Romain Naour in Cc.
>>>>
>>>> On Mon, 2 Dec 2019 02:47:15 +0200
>>>> Vadim Kochan <vadim4j@gmail.com> wrote:
>>>>
>>>>> Just in case if someone will be interested. I started Buildroot external
>>>>> project which provides external toolchains from the Bootlin:
>>>>>
>>>>>     https://toolchains.bootlin.com
>>>>>
>>>>> The git repo can be found at:
>>>>>
>>>>>     https://github.com/vkochan/buildroot-external-toolchain-bootlin
>>>>
>>>> An effort was started to package the Bootlin external toolchains in
>>>> Buildroot itself, some time ago by engineers at Smile. We had some
>>>> off-list discussion about this, but I don't think it was ever posted to
>>>> the list. The idea back then was to have a script that generates all
>>>> the packages, but the design was to have one package for each
>>>> toolchain, so it really added a lot of packages.
>>>>
>>>> But I think we should really support these toolchains in Buildroot
>>>> proper instead of having an external for them.
>>>
>>> Agree.
>>>
>>> Arthur will take a look at the script but it's very complicated script...
>>> Maybe we need to discuss what we really want.
>>
>>  I think this should be like scancpan/scanpypi, i.e. something that generates
>> the toolchain-external-bootlin package but it's up to the user to fix up the
>> final details.
>>
>>  And then we can work on toolchains-builder to make sure it generates something
>> that is easier for this scanbootlin script to parse.
> 
> The initial goal was to use toolchain bootlin as prebuilt external toolchain
> when no other prebuilt toolchain is available (Linaro, Mentor etc...).
> I would like to avoid letting the user fixing the final details.

 Er, looks like we have a bit of a disconnect here...

 In my view, the bootlin toolchains should be made available exactly like the
other external toolchains (Linaro, Mentor, ...): as a menuconfig choice. But we
can build a script that creates the corresponding Config.in and .mk file, and
commit the result into git. That script doesn't need to be fully automated, the
output can be manually adapted before committing.


>>  Alternatively, maybe even better, we can just update build.sh in
>> toolchains-builder to generate all the pieces and then add an additional
>> pipeline stage that collects the pieces into a package.
> 
> The script evoked by Thomas showed that it was too difficult/complex to generate
> a toolchain packages from the files currently provided by toolchain-builder.
> 
> So, I'm agree that it would be better if toolchain-builder can provide such package.
> 
>>
>>  Most importantly, though, I think we should take this gradually. Start with a
>> manually constructed package, it doesn't even need to support all the different
>> toolchains. We can bikeshed a bit on what that package looks like, and use that
>> as input for how to script it.
> 
> At least aarch64, arm and x86_64 should be included in the manually constructed
> package.
> 
>>
>>  To kick off the bikeshedding, here's my opinion on what the package should look
>> like.
>>
>> * It should be a single package, toolchain-external-bootlin. Maybe two (-stable
>> and -bleeding-edge) but I think there'll be enough shared infrastructure to
>> warrant creating a single package.
> 
> The most difficult part is to generate the dependencies (depends on) of the
> toolchain and its properties (select). That is why our experimental script
> create one package for each toolchain provided by toolchain Builder.

 That's why starting with something that is constructed manually is a good idea.
We can reason about how to solve the issues with hidden helper symbols etc.


> Also if we can use a single a single package for the Bootlin toolchain package,
> it mean that we can also use a single package for the
> Linaro/ARM/Codescape/Codesourcery packages.

 Er, no. We've factored all the commonalities between those packages into
external-toolchain-package.

 Or maybe you mean that the 3 arm toolchains and the 4 linaro toolchains could
be merged? Yes, that's probably true.


>> * It should offer a choice in Config.in.options for libc and stable/bleeding-edge.
>>
>> * For now, it should be limited to *exactly* the same architecture suboptions as
>> what the toolchain was built for. Later we can decide how to expand this to
>> compatible options (e.g. the core-i7 toolchain will also work for core-avx2) -
>> but I'm pretty sure that that is something that will require help from
>> toolchain-builder itself.
> 
> Indeed.


 Regards,
 Arnout