[Buildroot] Adding support for the Rust programming language

[Buildroot] Adding support for the Rust programming language

[Eric Le Bihan]

Hi all!

I've posted a series of articles about using the Rust programming
language in a Buildroot environment [1].

Rust [2] is a modern, compiled, programming language, oriented towards
safety, memory control and concurrency. Its performances are comparable
to C++.

Rust's features make it a good candidate for writing programs for
embedded systems.

This series will hopefully lead to some patches for properly supporting
Rust in Buildroot.

As explained in one article [3], the Rust compiler is based on LLVM
[4]. When configuring the build, the targeted architecture is given by
its GNU triplet (e.g. "arm-buildroot-linux-gnueabihf"). By default, the
compiler is built with support for x86, ARM and MIPS support
(unfortunately, there is no option to narrow the selection). Then the
Rust standard library is cross-compiled for the configured architecture.

Support for the target is achieved by providing two files:

- the target configuration file: a Makefile fragment where, among
  others, the name of of the cross-compiler is given.
- the target specification file: a JSON file describing the
  architecture (CPU type, OS, etc).

So, in the future patch series that will properly add support for Rust
in Buildroot:

1. Would it be acceptable to provide a script to generate these files
from the Buildroot configuration file?

2. Would Python be a suitable language for this script? It is better
suited for playing with complex structures and JSON than pure shell.

Comments welcomed.

[1] http://www.elebihan.com/categories/rust.html
[2] https://rust-lang.org/
[3] http://www.elebihan.com/posts/using-rust-with-buildroot-full-build.html
[4] https://llvm.org/

Best regards,
--
ELB

[Buildroot] Adding support for the Rust programming language

[Arnout Vandecappelle]

On 04/08/16 22:45, Eric Le Bihan wrote:
> Hi all!
>
> I've posted a series of articles about using the Rust programming
> language in a Buildroot environment [1].
>
> Rust [2] is a modern, compiled, programming language, oriented towards
> safety, memory control and concurrency. Its performances are comparable
> to C++.
>
> Rust's features make it a good candidate for writing programs for
> embedded systems.
>
> This series will hopefully lead to some patches for properly supporting
> Rust in Buildroot.
>
> As explained in one article [3], the Rust compiler is based on LLVM
> [4]. When configuring the build, the targeted architecture is given by
> its GNU triplet (e.g. "arm-buildroot-linux-gnueabihf"). By default, the
> compiler is built with support for x86, ARM and MIPS support
> (unfortunately, there is no option to narrow the selection). Then the
> Rust standard library is cross-compiled for the configured architecture.
>
> Support for the target is achieved by providing two files:
>
> - the target configuration file: a Makefile fragment where, among
>    others, the name of of the cross-compiler is given.
> - the target specification file: a JSON file describing the
>    architecture (CPU type, OS, etc).
>
> So, in the future patch series that will properly add support for Rust
> in Buildroot:
>
> 1. Would it be acceptable to provide a script to generate these files
> from the Buildroot configuration file?

  Obviously yes, we already do that kind of thing for some packages (e.g. cmake).

>
> 2. Would Python be a suitable language for this script? It is better
> suited for playing with complex structures and JSON than pure shell.

  We try to avoid requiring python for a normal build, and use it only for 
specific scripts at the moment (like graph-depends). So I think the answer is no 
here. Anyway, as far as I can see from [3], you don't really need a full json 
generator, it should be enough to do some sed magic on a template file.

  When you submit patches, it's probably a good idea to start with a minimal 
patch that e.g. gives rust a hard dependency on little-endian ARM v6, and add 
more features in subsequent patches. I expect it will be something difficult to 
review.

  Regards,
  Arnout

>
> Comments welcomed.
>
> [1] http://www.elebihan.com/categories/rust.html
> [2] https://rust-lang.org/
> [3] http://www.elebihan.com/posts/using-rust-with-buildroot-full-build.html
> [4] https://llvm.org/
>
> Best regards,
> --
> ELB
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Adding support for the Rust programming language

[Thomas Petazzoni]

Hello,

On Sun, 10 Apr 2016 17:57:45 +0200, Arnout Vandecappelle wrote:

>   We try to avoid requiring python for a normal build, and use it only for 
> specific scripts at the moment (like graph-depends).

Right, but on the other hand, support/dependencies/dependencies.sh
verifies unconditionally that there is a python interpreter available
on the build machine. So there is not really anything that prevents
from using python stuff during the build. Unless of course this Python
logic starts to use modules not in the Python standard library, of
course.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Adding support for the Rust programming language

[Arnout Vandecappelle]

On 04/11/16 10:32, Thomas Petazzoni wrote:
> Hello,
>
> On Sun, 10 Apr 2016 17:57:45 +0200, Arnout Vandecappelle wrote:
>
>>    We try to avoid requiring python for a normal build, and use it only for
>> specific scripts at the moment (like graph-depends).
>
> Right, but on the other hand, support/dependencies/dependencies.sh
> verifies unconditionally that there is a python interpreter available
> on the build machine. So there is not really anything that prevents
> from using python stuff during the build. Unless of course this Python
> logic starts to use modules not in the Python standard library, of
> course.

  Well, for the fix-rpath stuff we asked Samuel to do in in shell instead of 
python, with the argument that we want to avoid python to be in the critical 
path of a normal build. That said, this would only be for a rust build, not for 
all builds, so it's less critical.

  BTW note that e.g. for waf-based packages, we always use host-python, not 
system python. But that's probably because waf is not python3 compatible.

  So, if python is used, please make sure it runs on any python (with full 
standard library) from python2.4 (RHEL5) to python3.5. That means that using the 
json standard library module is not an option, since it was introduced only in 
python 2.6.


  Regards,
  Arnout

-- 
Arnout Vandecappelle      arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Adding support for the Rust programming language

[Arnout Vandecappelle]

On 04/11/16 11:12, Arnout Vandecappelle wrote:
>
>
> On 04/11/16 10:32, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Sun, 10 Apr 2016 17:57:45 +0200, Arnout Vandecappelle wrote:
>>
>>>    We try to avoid requiring python for a normal build, and use it only for
>>> specific scripts at the moment (like graph-depends).
>>
>> Right, but on the other hand, support/dependencies/dependencies.sh
>> verifies unconditionally that there is a python interpreter available
>> on the build machine. So there is not really anything that prevents
>> from using python stuff during the build. Unless of course this Python
>> logic starts to use modules not in the Python standard library, of
>> course.
>
>   Well, for the fix-rpath stuff we asked Samuel to do in in shell instead of
> python, with the argument that we want to avoid python to be in the critical
> path of a normal build. That said, this would only be for a rust build, not for
> all builds, so it's less critical.
>
>   BTW note that e.g. for waf-based packages, we always use host-python, not
> system python. But that's probably because waf is not python3 compatible.
>
>   So, if python is used, please make sure it runs on any python (with full
> standard library) from python2.4 (RHEL5) to python3.5. That means that using the
> json standard library module is not an option, since it was introduced only in
> python 2.6.

  Unless of course you add a dependency on host-python or host-python3 (make 
sure you select the version that corresponds to the selected target python, to 
avoid unnecessarily building both). Since we can probably consider rust to be a 
big package, adding an addition host-python dependency is acceptable IMHO.

  Regards,
  Arnout

-- 
Arnout Vandecappelle      arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Adding support for the Rust programming language

[Thomas Petazzoni]

Hello,

On Mon, 11 Apr 2016 11:13:45 +0200, Arnout Vandecappelle wrote:

>   Unless of course you add a dependency on host-python or host-python3 (make 
> sure you select the version that corresponds to the selected target python, to 
> avoid unnecessarily building both). Since we can probably consider rust to be a 
> big package, adding an addition host-python dependency is acceptable IMHO.

Indeed, adding host-python as a dependency is reasonable for something
big like Rust. Of course, if we can do without Python, fine. But I
really prefer a clean Python script that uses a proper json library,
rather than horrible sed expressions in an obfuscated shell script.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Adding support for the Rust programming language

[Baruch Siach]

Hi Arnout,

On Mon, Apr 11, 2016 at 11:12:07AM +0200, Arnout Vandecappelle wrote:
>  So, if python is used, please make sure it runs on any python (with full
> standard library) from python2.4 (RHEL5) to python3.5. That means that using
> the json standard library module is not an option, since it was introduced
> only in python 2.6.

The manual (http://nightly.buildroot.org/manual.html#requirement-mandatory) 
currently requires Python 2.6 or newer.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] Adding support for the Rust programming language

[Arnout Vandecappelle]

On 04/11/16 11:15, Baruch Siach wrote:
> Hi Arnout,
>
> On Mon, Apr 11, 2016 at 11:12:07AM +0200, Arnout Vandecappelle wrote:
>>   So, if python is used, please make sure it runs on any python (with full
>> standard library) from python2.4 (RHEL5) to python3.5. That means that using
>> the json standard library module is not an option, since it was introduced
>> only in python 2.6.
>
> The manual (http://nightly.buildroot.org/manual.html#requirement-mandatory)
> currently requires Python 2.6 or newer.

  I don't think that that's accurate. Thomas, Jan, I believe you're using RHEL5 
with an older python, no?

  Regards,
  Arnout

-- 
Arnout Vandecappelle      arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Adding support for the Rust programming language

[Thomas De Schampheleire]

Hi,

On Mon, Apr 11, 2016 at 11:29 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 04/11/16 11:15, Baruch Siach wrote:
>>
>> Hi Arnout,
>>
>> On Mon, Apr 11, 2016 at 11:12:07AM +0200, Arnout Vandecappelle wrote:
>>>
>>>   So, if python is used, please make sure it runs on any python (with
>>> full
>>> standard library) from python2.4 (RHEL5) to python3.5. That means that
>>> using
>>> the json standard library module is not an option, since it was
>>> introduced
>>> only in python 2.6.
>>
>>
>> The manual
>> (http://nightly.buildroot.org/manual.html#requirement-mandatory)
>> currently requires Python 2.6 or newer.
>
>
>  I don't think that that's accurate. Thomas, Jan, I believe you're using
> RHEL5 with an older python, no?


We have machines on RHEL 5 and stock Python (2.4), yes. I have not
received reports of this not building for us.
But then again, which python dependency do we really have for standard builds?
Note also that we have a restricted set of packages we build, if the
python dependency is due to a specific package, then we may not see
the problem.

To generate the manual, (see docs/manual/manual.mk), there is a check
on availability of argparse, which is only standard on python 2.7 but
can be downloaded separately for python >= 2.3.

/Thomas

[Buildroot] Adding support for the Rust programming language

[Eric Le Bihan]

On Sun, 10 Apr 2016 17:57:45 +0200,
Arnout Vandecappelle <arnout@mind.be> wrote :

> > 1. Would it be acceptable to provide a script to generate these
> > files from the Buildroot configuration file?  
> 
>   Obviously yes, we already do that kind of thing for some packages
> (e.g. cmake).

So, assuming the script is to be named `rust-target-gen`, it should be
stored in support/scripts, right?

> > 2. Would Python be a suitable language for this script? It is better
> > suited for playing with complex structures and JSON than pure
> > shell.  
> 
>   We try to avoid requiring python for a normal build, and use it
> only for specific scripts at the moment (like graph-depends). So I
> think the answer is no here. Anyway, as far as I can see from [3],
> you don't really need a full json generator, it should be enough to
> do some sed magic on a template file.

Correct. The target specification file is not a complex JSON file.
So I'll go for a Bash shell script (compatible with Bash 3.2, i.e. no
use of associative arrays).

Anyway, there will be a need for building host-python, in order to
build Cargo, the official Rust package manager, as stated in [1].

>   When you submit patches, it's probably a good idea to start with a
> minimal patch that e.g. gives rust a hard dependency on little-endian
> ARM v6, and add more features in subsequent patches. I expect it will
> be something difficult to review.

This looks sensible. I intend to provide the following patches:

- one to add the package "host-rust" (compiler)
- one to add the package "host-cargo" (package manager)
- one to add "package/pkg-cargo.mk" (Rust package infrastructure)

Thanks for your suggestions.

[1] http://www.elebihan.com/posts/using-cargo-with-buildroot-full-build.html

--
ELB

[Buildroot] Adding support for the Rust programming language

[Arnout Vandecappelle]

On 04/11/16 21:03, Eric Le Bihan wrote:
> On Sun, 10 Apr 2016 17:57:45 +0200,
> Arnout Vandecappelle <arnout@mind.be> wrote :
>
>>> 1. Would it be acceptable to provide a script to generate these
>>> files from the Buildroot configuration file?
>>
>>    Obviously yes, we already do that kind of thing for some packages
>> (e.g. cmake).
>
> So, assuming the script is to be named `rust-target-gen`, it should be
> stored in support/scripts, right?

  It's only going to be used by the rust package itself, right? Not by any other 
package that happens to be written in rust? In that case, I'd put the script in 
the package/rust directory.

>
>>> 2. Would Python be a suitable language for this script? It is better
>>> suited for playing with complex structures and JSON than pure
>>> shell.
>>
>>    We try to avoid requiring python for a normal build, and use it
>> only for specific scripts at the moment (like graph-depends). So I
>> think the answer is no here. Anyway, as far as I can see from [3],
>> you don't really need a full json generator, it should be enough to
>> do some sed magic on a template file.
>
> Correct. The target specification file is not a complex JSON file.
> So I'll go for a Bash shell script (compatible with Bash 3.2, i.e. no
> use of associative arrays).
>
> Anyway, there will be a need for building host-python, in order to
> build Cargo, the official Rust package manager, as stated in [1].

  In that case, by all means use a python script that relies on host-python. 
Just make sure it runs on python3 as well.

>
>>    When you submit patches, it's probably a good idea to start with a
>> minimal patch that e.g. gives rust a hard dependency on little-endian
>> ARM v6, and add more features in subsequent patches. I expect it will
>> be something difficult to review.
>
> This looks sensible. I intend to provide the following patches:
>
> - one to add the package "host-rust" (compiler)
> - one to add the package "host-cargo" (package manager)
> - one to add "package/pkg-cargo.mk" (Rust package infrastructure)

  Don't add the package infrastructure unless you have at least one package that 
uses it, so we understand how it works.

  Also, I repeat: if supporting all possible arches/endianness/fp/... 
combinations makes it complicated, start out with a limited set of supported 
arch/... combinations and add others in later patches.

  Regards,
  Arnout

>
> Thanks for your suggestions.
>
> [1] http://www.elebihan.com/posts/using-cargo-with-buildroot-full-build.html
>
> --
> ELB
>
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF