Linux kernel in-tree Rust support

Linux kernel in-tree Rust support

[Nick Desaulniers]

Hello folks,
I'm working on putting together an LLVM "Micro Conference" for the
upcoming Linux Plumbers Conf
(https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
solidified yet, but I would really like to run a session on support
for Rust "in tree."  I suspect we could cover technical aspects of
what that might look like (I have a prototype of that, was trivial to
wire up KBuild support), but also a larger question of "should we do
this?" or "how might we place limits on where this can be used?"

Question to folks explicitly in To:, are you planning on attending plumbers?

If so, would this be an interesting topic that you'd participate in?
-- 
Thanks,
~Nick Desaulniers

Re: Linux kernel in-tree Rust support

[Miguel Ojeda]

Hi Nick,

On Thu, Jul 9, 2020 at 8:42 PM Nick Desaulniers <ndesaulniers@google.com> wrote:
>
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree."  I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to

Agreed, I wrote one ~half a year ago that compiled TUs through rustc
mimicking what we do with cc to some degree (i.e. no cargo etc.), and
it was quite pleasant.

> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"

Indeed, I would say that is going to be the major topic, not so much
the technical bits.

> Question to folks explicitly in To:, are you planning on attending plumbers?

I wasn't, but it definitely is an interesting topic that I wouldn't
want to miss!

Cheers,
Miguel

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree."  I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
> 
> Question to folks explicitly in To:, are you planning on attending plumbers?
> 
> If so, would this be an interesting topic that you'd participate in?

I hadn't planned to attend the virtual event, but this sounds like a
topic I absolutely have to attend. Please follow up if this proposal
gets accepted.

I'd love to see a path to incorporating Rust into the kernel, as long as
we can ensure that:
- There are appropriate Rustic interfaces that are natural and safe to
  use (not just C FFI, and not *just* trivial transformations like
  slices instead of buffer+len pairs).
- Those Rustic interfaces are easy to maintain and evolve with the kernel.
- We provide compelling use cases that go beyond just basic safety, such
  as concurrency checking, or lifetimes for object ownership.
- We make Rust fit naturally into the kernel's norms and standards,
  while also introducing some of Rust's norms and standards where they
  make sense. (We want to fit into the kernel, and at the same time, we
  don't want to hastily saw off all the corners that don't immediately
  fit, because some of those corners provide value. Let's take our
  time.)
- We move slowly and carefully, making sure it's a gradual introduction,
  and give people time to incorporate the Rust toolchain into their
  kernel workflows.

Also, with my "Rust language team lead" hat on, I'd be happy to have the
Linux kernel feeding into Rust language development priorities. If
building Rustic interfaces within the kernel requires some additional
language features, we should see what enhancements to the language would
best serve those requirements. I've often seen the sentiment that
co-evolving Linux and a C compiler would be beneficial for both; I think
the same would be true of Linux and the Rust compiler.

Re: Linux kernel in-tree Rust support

[Greg KH]

On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree."  I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
> 
> Question to folks explicitly in To:, are you planning on attending plumbers?
> 
> If so, would this be an interesting topic that you'd participate in?

Yes, I'll be there.

thanks,

greg k-h

Re: Linux kernel in-tree Rust support

[Christian Brauner]

On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > Hello folks,
> > I'm working on putting together an LLVM "Micro Conference" for the
> > upcoming Linux Plumbers Conf
> > (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> > solidified yet, but I would really like to run a session on support
> > for Rust "in tree."  I suspect we could cover technical aspects of
> > what that might look like (I have a prototype of that, was trivial to
> > wire up KBuild support), but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> > 
> > Question to folks explicitly in To:, are you planning on attending plumbers?
> > 
> > If so, would this be an interesting topic that you'd participate in?
> 
> Yes, I'll be there.

We actually had this dicussion a while back and there were some more
people interested in this. I'd be interested to attend this and I've
spoken with Kees and a few others about this topic at last Plumbers (I
think Greg might have been around for this informal discussion as well.
But I might be imagining things.).

Christian

Re: Linux kernel in-tree Rust support

[Kees Cook]

On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote:
> On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > > Hello folks,
> > > I'm working on putting together an LLVM "Micro Conference" for the
> > > upcoming Linux Plumbers Conf
> > > (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> > > solidified yet, but I would really like to run a session on support
> > > for Rust "in tree."  I suspect we could cover technical aspects of
> > > what that might look like (I have a prototype of that, was trivial to
> > > wire up KBuild support), but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > > 
> > > Question to folks explicitly in To:, are you planning on attending plumbers?
> > > 
> > > If so, would this be an interesting topic that you'd participate in?
> > 
> > Yes, I'll be there.
> 
> We actually had this dicussion a while back and there were some more
> people interested in this. I'd be interested to attend this and I've
> spoken with Kees and a few others about this topic at last Plumbers (I
> think Greg might have been around for this informal discussion as well.
> But I might be imagining things.).

I'm quite interested in this topic still, yes. :) (And will be attending
Plumbers.)

We had interesting discussions with the LSS 2019 "Rust in the kernel"[1]
speakers, and I know Nick has reviewed their work[2] too. Perhaps we should
reach out to them as well?

-Kees

[1] https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf
[2] https://github.com/fishinabarrel/linux-kernel-module-rust

-- 
Kees Cook

Re: Linux kernel in-tree Rust support

[Kees Cook]

On Fri, Jul 10, 2020 at 12:28:30PM -0400, Alex Gaynor wrote:
> :wave:,
> 
> Hey Kees, that's Geoffrey Thomas and I, we're both on this thread :-)

*face palm* Hello! I swear I can read. Though perhaps not well enough
before lunch. :)

-- 
Kees Cook

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote:
> On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > > Hello folks,
> > > I'm working on putting together an LLVM "Micro Conference" for the
> > > upcoming Linux Plumbers Conf
> > > (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> > > solidified yet, but I would really like to run a session on support
> > > for Rust "in tree."  I suspect we could cover technical aspects of
> > > what that might look like (I have a prototype of that, was trivial to
> > > wire up KBuild support), but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > > 
> > > Question to folks explicitly in To:, are you planning on attending plumbers?
> > > 
> > > If so, would this be an interesting topic that you'd participate in?
> > 
> > Yes, I'll be there.
> 
> We actually had this dicussion a while back and there were some more
> people interested in this. I'd be interested to attend this and I've
> spoken with Kees and a few others about this topic at last Plumbers (I
> think Greg might have been around for this informal discussion as well.
> But I might be imagining things.).

I was around for one of the informal conversations with Greg and Kees
and others.

As I recall, Greg's biggest condition for initial introduction of this
was to do the same kind of "turn this Kconfig option on and turn an
option under it off" trick that LTO uses, so that neither "make
allnoconfig" nor "make allyesconfig" would require Rust until we've had
plenty of time to experiment with it. And that seems entirely
reasonable to me too.

- Josh

Re: Linux kernel in-tree Rust support

[Linus Torvalds]

On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <josh@joshtriplett.org> wrote:
>
> As I recall, Greg's biggest condition for initial introduction of this
> was to do the same kind of "turn this Kconfig option on and turn an
> option under it off" trick that LTO uses, so that neither "make
> allnoconfig" nor "make allyesconfig" would require Rust until we've had
> plenty of time to experiment with it.

No, please make it a "is rust available" automatic config option. The
exact same way we already do the compiler versions and check for
various availability of compiler flags at config time.

See init/Kconfig for things like

  config LD_IS_LLD
          def_bool $(success,$(LD) -v | head -n 1 | grep -q LLD)

and the rust support should be similar. Something like

  config RUST_IS_AVAILABLE
          def_bool $(success,$(RUST) ..sometest..)

because I _don't_ want us to be in the situation where any new rust
support isn't even build-tested by default.

Quite the reverse. I'd want the first rust driver (or whatever) to be
introduced in such a simple format that failures will be obvious and
simple.

The _worst_ situation to be in is that s (small) group of people start
testing their very special situation, and do bad and crazy things
because "nobody else cares, it's hidden".

No, thank you.

             Linus

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Fri, Jul 10, 2020 at 04:54:11PM -0700, Linus Torvalds wrote:
> On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <josh@joshtriplett.org> wrote:
> > As I recall, Greg's biggest condition for initial introduction of this
> > was to do the same kind of "turn this Kconfig option on and turn an
> > option under it off" trick that LTO uses, so that neither "make
> > allnoconfig" nor "make allyesconfig" would require Rust until we've had
> > plenty of time to experiment with it.
> 
> No, please make it a "is rust available" automatic config option. The
> exact same way we already do the compiler versions and check for
> various availability of compiler flags at config time.

That sounds even better, and will definitely allow for more testing.

We just need to make sure that any kernel CI infrastructure tests that
right away, then, so that failures don't get introduced by a patch from
someone without a Rust toolchain and not noticed until someone with a
Rust toolchain tests it.

- Josh

Re: Linux kernel in-tree Rust support

[Pavel Machek]

Hi!

> > No, please make it a "is rust available" automatic config option. The
> > exact same way we already do the compiler versions and check for
> > various availability of compiler flags at config time.
> 
> That sounds even better, and will definitely allow for more testing.
> 
> We just need to make sure that any kernel CI infrastructure tests that
> right away, then, so that failures don't get introduced by a patch from
> someone without a Rust toolchain and not noticed until someone with a
> Rust toolchain tests it.

So... I'm fan of Rust, but while trying to use it one thing was obvious: it
takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.

Kernel is quite big project, can CI infrastructure handle additional load?

Will developers see significantly longer compile times when Rust is widespread?

Best regards,
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote:
> > We just need to make sure that any kernel CI infrastructure tests that
> > right away, then, so that failures don't get introduced by a patch from
> > someone without a Rust toolchain and not noticed until someone with a
> > Rust toolchain tests it.
> 
> So... I'm fan of Rust, but while trying to use it one thing was obvious: it
> takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
> 
> Kernel is quite big project, can CI infrastructure handle additional load?
> 
> Will developers see significantly longer compile times when Rust is widespread?

I wouldn't expect the addition of Rust to the kernel to substantially
impact overall build time; on balance, I'd expect the major bottleneck
in kernel builds to continue to be linking and other serialized steps,
not compiling and other highly parallel steps.

There are also *many* things that can be done to improve Rust build time
in a project. And I don't expect that in-kernel Rust will have many
dependencies on third-party crates (since they'd need to be checked into
the tree).

Re: Linux kernel in-tree Rust support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

On Tue 2020-07-28 23:34:17, Josh Triplett wrote:
> On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote:
> > > We just need to make sure that any kernel CI infrastructure tests that
> > > right away, then, so that failures don't get introduced by a patch from
> > > someone without a Rust toolchain and not noticed until someone with a
> > > Rust toolchain tests it.
> > 
> > So... I'm fan of Rust, but while trying to use it one thing was obvious: it
> > takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
> > 
> > Kernel is quite big project, can CI infrastructure handle additional load?
> > 
> > Will developers see significantly longer compile times when Rust is widespread?
> 
> I wouldn't expect the addition of Rust to the kernel to substantially
> impact overall build time; on balance, I'd expect the major bottleneck
> in kernel builds to continue to be linking and other serialized steps,
> not compiling and other highly parallel steps.

Well.. not everyone has 32 cores in their notebook.

> There are also *many* things that can be done to improve Rust build time
> in a project. And I don't expect that in-kernel Rust will have many
> dependencies on third-party crates (since they'd need to be checked into

Okay. I did some refactoring recently and I really wished kernel was
in Rust (and not in C)... so lets see what happens.

Best regards,

  	       	  	      	     	    	   	      	      Pavel
-- 
http://www.livejournal.com/~pavelmachek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Linux kernel in-tree Rust support

[Miguel Ojeda]

On Thu, Dec 17, 2020 at 10:45 PM Pavel Machek <pavel@ucw.cz> wrote:
>
> Well.. not everyone has 32 cores in their notebook.

It is true that complex Rust, like complex C++, does have high
compilation times. But it all depends on how much one relies on
certain language features. Straightforward Rust code is quick to
compile.

For reference, some quick stats :-)

On a given machine, building v5.10 with a minimal config under -j3
takes 3 minutes. With Rust support enabled and 4 trivial Rust modules,
it takes 50 seconds more. "A big increase!", you may indeed claim, but
those 50 seconds are basically all spent on the shared Rust support.
The actual Rust modules compile very quickly afterwards. For example,
touching either `drivers/char/mem.c` or one of the trivial Rust
modules takes the same time in that machine: 10 seconds.

This is for a minimal config -- when you start dealing with
`allmodconfig` builds, or when you start having a hundred Rust modules
instead of 4, the upfront cost becomes very small per Rust module.

> Okay. I did some refactoring recently and I really wished kernel was
> in Rust (and not in C)... so lets see what happens.

Indeed, I think it is definitely worth it.

Cheers,
Miguel

Re: Linux kernel in-tree Rust support

[Geoffrey Thomas]

On Thu, 9 Jul 2020, Nick Desaulniers wrote:

> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree."  I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?

Like Alex - I hadn't decided yet but I'll definitely attend for this!

-- 
Geoffrey Thomas
https://ldpreload.com
geofft@ldpreload.com

Re: Linux kernel in-tree Rust support

[Adrian Bunk]

On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
>...
> but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>...

I won't attend, but I do have a topic that should be covered:

Firefox always depends on recent Rust, which forces distributions to 
update Rust in stable releases.

As an example:
Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.

It would not sound good to me if security updates of distribution
kernels might additionally end up using a different version of the
Rust compiler - the toolchain for the kernel should be stable.

Would Rust usage in the kernel require distributions to ship
a "Rust for Firefox" and a "Rust for the kernel"?

> Thanks,
> ~Nick Desaulniers

cu
Adrian

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> >...
> > but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >...
> 
> I won't attend, but I do have a topic that should be covered:
> 
> Firefox always depends on recent Rust, which forces distributions to 
> update Rust in stable releases.
> 
> As an example:
> Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> 
> It would not sound good to me if security updates of distribution
> kernels might additionally end up using a different version of the
> Rust compiler - the toolchain for the kernel should be stable.
> 
> Would Rust usage in the kernel require distributions to ship
> a "Rust for Firefox" and a "Rust for the kernel"?

Rust has hard stability guarantees when upgrading from one stable
version to the next. If code compiles with a given stable version of
Rust, it'll compile with a newer stable version of Rust. Given that, a
stable distribution will just need a single sufficiently up-to-date Rust
that meets the minimum version requirements of both Firefox and Linux.

(That would not apply if the kernel used nightly Rust, since
nightly-only features are allowed to change before becoming stable;
that's one reason why we should use stable Rust, and try to get Firefox
to stick to stable Rust.)

Re: Linux kernel in-tree Rust support

[Linus Torvalds]

On Sun, Jul 12, 2020 at 12:39 PM Josh Triplett <josh@joshtriplett.org> wrote:
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next.

I think the worry is more about actual compiler bugs, not the set of
exposed features.

That's always been the biggest pain point. Compiler bugs are very
rare, but they are so incredibly hard to debug when they happen that
they end up being extra special.

Random "we need this compiler for this feature" is actually fairly
rare. Yes, the most recent case of me just saying "let's use 4.9
rather than 4.8" was due to that, but honestly, that's the exception
rather than the rule, and is to occasionally simplify the code (and
the test coverage).

The most common case of compiler version checks are due to "compiler
XYZ is known to mis-compile ABC on target IDK".

                    Linus

Re: Linux kernel in-tree Rust support

[Adrian Bunk]

On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > >...
> > > but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >...
> > 
> > I won't attend, but I do have a topic that should be covered:
> > 
> > Firefox always depends on recent Rust, which forces distributions to 
> > update Rust in stable releases.
> > 
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> > 
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> > 
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
> 
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>...

API stability avoids problems that are visible early as build errors.

Rust cannot offer a hard stability guarantee that there will never be 
a code generation regression on any platform.

Rust gets updated frequently.
Sometimes this also changes the LLVM version used by Rust.
Debian stable supports targets like ARMv5 and 32bit MIPS.
Distribution kernel updates are often automatically installed
on user hardware.

cu
Adrian

RE: Linux kernel in-tree Rust support

[David Laight]

From: Adrian Bunk
> Sent: 12 July 2020 21:45
....
> Rust gets updated frequently.
> Sometimes this also changes the LLVM version used by Rust.
> Debian stable supports targets like ARMv5 and 32bit MIPS.
> Distribution kernel updates are often automatically installed
> on user hardware.

This reminds me of why I never want to use an online compiler
service - never mind how hard companies push them.

If I need to do a bug-fix build of something that was released
2 (or more) years ago I want to use exactly the same toolchain
(warts and all) that was used for the original build.

If the compiler has changed I need to do a full test - just
in case it compiles some 'dodgy' code differently.
With the same compiler I only need to test the fix.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux kernel in-tree Rust support

[Arnd Bergmann]

On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <josh@joshtriplett.org> wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> >
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> >
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> >
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>
> (That would not apply if the kernel used nightly Rust, since
> nightly-only features are allowed to change before becoming stable;
> that's one reason why we should use stable Rust, and try to get Firefox
> to stick to stable Rust.)

I would expect we'd want a fairly tight coupling between kernel
releases and minimum rust releases at first. Whatever is the latest
stable rust version during the kernel's merge window might be
assumed to be the minimum version for the life of that kernel, but
an LTS release would not suddenly start relying on features
from a newer compiler (thought it might warn about known bugs).

This might mean that linux-next requires a beta version of rust, if
the release is expected before the merge window and it contains
an important change. Staying with fairly recent versions of clang
certainly helped in getting clang and the kernel to co-evolve and
get to the point we are now in using it as an alternative to gcc.

While Linux used to build with 12 year old compilers (4.1 until
2018), we now require a 6 year old gcc (4.9) or 1 year old
clang/llvm. I don't know whether these will fully converge over
time but it seems sensible that the minimum rust frontend version
we require for a new kernel release would eventually also fall
in that range, requiring a compiler that is no more than a few
years old, but not requiring the latest stable release.

      Arnd

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote:
> On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <josh@joshtriplett.org> wrote:
> > On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> > >
> > > As an example:
> > > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> > >
> > > It would not sound good to me if security updates of distribution
> > > kernels might additionally end up using a different version of the
> > > Rust compiler - the toolchain for the kernel should be stable.
> > >
> > > Would Rust usage in the kernel require distributions to ship
> > > a "Rust for Firefox" and a "Rust for the kernel"?
> >
> > Rust has hard stability guarantees when upgrading from one stable
> > version to the next. If code compiles with a given stable version of
> > Rust, it'll compile with a newer stable version of Rust. Given that, a
> > stable distribution will just need a single sufficiently up-to-date Rust
> > that meets the minimum version requirements of both Firefox and Linux.
> >
> > (That would not apply if the kernel used nightly Rust, since
> > nightly-only features are allowed to change before becoming stable;
> > that's one reason why we should use stable Rust, and try to get Firefox
> > to stick to stable Rust.)
> 
> I would expect we'd want a fairly tight coupling between kernel
> releases and minimum rust releases at first. Whatever is the latest
> stable rust version during the kernel's merge window might be
> assumed to be the minimum version for the life of that kernel, but
> an LTS release would not suddenly start relying on features
> from a newer compiler (thought it might warn about known bugs).

Agreed; we should be careful that any backported fix to an LTS kernel
does not increase the required Rust toolchain.

> While Linux used to build with 12 year old compilers (4.1 until
> 2018), we now require a 6 year old gcc (4.9) or 1 year old
> clang/llvm. I don't know whether these will fully converge over
> time but it seems sensible that the minimum rust frontend version
> we require for a new kernel release would eventually also fall
> in that range, requiring a compiler that is no more than a few
> years old, but not requiring the latest stable release.

I expect in the short term that we will likely have a need for features
from recent stable releases, especially when those features were added
specifically to support the kernel or similar, but the rate at which we
need new features will slow over time, and eventually we'll go from
"latest stable" to "within a year or so".

Re: Linux kernel in-tree Rust support

[Adrian Bunk]

On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote:
> 
> I would expect we'd want a fairly tight coupling between kernel
> releases and minimum rust releases at first. Whatever is the latest
> stable rust version during the kernel's merge window might be
> assumed to be the minimum version for the life of that kernel, but
> an LTS release would not suddenly start relying on features
> from a newer compiler (thought it might warn about known bugs).
> 
> This might mean that linux-next requires a beta version of rust, if
> the release is expected before the merge window and it contains
> an important change.

I would expect this is absolutely not wanted,
it would make testing recent kernels very hard.

If you want to keep a tool that tightly to the kernel,
please bundle it with the kernel and build it as part
of the kernel build.

I would suggest to start with a proper design/specification what the 
kernel wants to use, so that you are confident that a compiler 
implementing this will be sufficient for the next 5 years.

As a secondary benefit, starting with a proper design often brings
a better result than adding permanent features piece by piece.

As a tertiary benefit, it would avoid tying the kernel to one specific 
compiler implementation. A compiler like mrustc or a hypothetical Rust 
frontend for gcc could then implement a superset of what the kernel 
needs.

> Staying with fairly recent versions of clang
> certainly helped in getting clang and the kernel to co-evolve and
> get to the point we are now in using it as an alternative to gcc.

The main difference is between an alternative to an existing tool,
and a mandatory new tool.

> While Linux used to build with 12 year old compilers (4.1 until
> 2018), we now require a 6 year old gcc (4.9) or 1 year old
> clang/llvm. I don't know whether these will fully converge over
> time but it seems sensible that the minimum rust frontend version
> we require for a new kernel release would eventually also fall
> in that range, requiring a compiler that is no more than a few
> years old, but not requiring the latest stable release.

The correct range for a mandatory tool are the 6 to 12 years for gcc.

Debian stable and Ubuntu LTS are providing (different) mechanisms
for installing the kernel from the next stable/LTS release 2 years
later[1] for supporting new hardware.
If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with
the 2019 gcc 9 there would be pain downstream.

In the embedded world spreads far wider than these 3 years are common.
I would currently have a real-life usecase for compiling a recent
kernel with a gcc 4.0 (sic) toolchain.
Properly supporting 15 year old toolchains would be painful upstream,
supporting 6 year old toolchains is a reasonable compromise between
not being too painful upstream while rarely causing pain downstream.

What applies to gcc does also apply to other external tools used
during the kernel build.

>       Arnd

cu
Adrian

[1] following a new upstream kernel stable branch every 6 months (Ubuntu)
    or the latest upstream stable kernels (Debian) until this is reached

RE: Linux kernel in-tree Rust support

[David Laight]

From: Adrian Bunk
> Sent: 19 July 2020 19:19
...
> The correct range for a mandatory tool are the 6 to 12 years for gcc.
> 
> Debian stable and Ubuntu LTS are providing (different) mechanisms
> for installing the kernel from the next stable/LTS release 2 years
> later[1] for supporting new hardware.
> If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with
> the 2019 gcc 9 there would be pain downstream.

We have customers that are (still) using RHEL6 (2.6.32 era).
Since we have to build C++ programs that will run on those
systems we have systems with very old toolchains.
(Yes, you can build in a chroot or an a VM but that needs
setting up quite carefully - easier to keep the old machine.)

It is rather a PITA when kernel builds deprecate the installed gcc.
Now I'm not sure a 5.x kernel will run with the userspace from RHEL6.
But I know it runs with that from Ubuntu 13.04.
In a few years I'll be able to use the 13.04 system for release builds
but it is still far too new for that.
However I can no longer build 'current' custom kernels on it.
I've other (newer) systems but they aren't any (maybe much) faster.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Linux kernel in-tree Rust support

[Adrian Bunk]

On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote:
>...
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust.
>...

In librsvg, breakages with more recent Rust versions in the past year
required updates of two vendored crates:
https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902
https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d

For updating Rust in Debian stable for the next Firefox ESR update it 
would actually be useful if these violations of the "hard stability 
guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds 
again with the latest Rust.

It also makes me wonder how such regressions slip into Rust releases.

cu
Adrian

Re: Linux kernel in-tree Rust support

[Geoffrey Thomas]

On Mon, 24 Aug 2020, Adrian Bunk wrote:

> In librsvg, breakages with more recent Rust versions in the past year
> required updates of two vendored crates:

Interesting!

> https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902

Looks like this was, for a while, a warning and not an error:

https://github.com/servo/rust-cssparser/commit/3c98d22c5de3b696bf1fde2b6c90069812312aa6

     = warning: this error has been downgraded to a warning for backwards compatibility with previous releases
     = warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future

> https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d

Same here, and it looks like the same warning/error, too:

https://github.com/dimforge/nalgebra/issues/561

     = warning: this error has been downgraded to a warning for backwards compatibility with previous releases
     = warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future

Following some links in that second issue, I see that this seems to be a 
summary of what's going on:

https://github.com/rust-lang/rust/issues/59159

in particular this paragraph: "at the time that NLL [non-lexical 
lifetimes] was stabilized, the compiler's acceptance of this borrowing 
pattern was categorized by the NLL team as a 'known bug'. The NLL team 
assumed that, as a bug fix, the compiler would be allowed to start 
rejecting the pattern in the future."

That is, both of these cases were code that should never have been 
accepted by the compiler because it is unsound, but the initial 
implementation of NLL was not clever enough to detect it. They later 
turned it into a warning, and later turned that into an error.

There is a link to this policy document which explains their process for 
breaking existing code in the case where it's necessary to fix a compiler 
bug or similar:

https://github.com/rust-lang/rfcs/blob/master/text/1589-rustc-bug-fix-procedure.md

There is also a link to this comment about why they decided to take this 
approach:

https://github.com/rust-lang/rust/pull/58739#issuecomment-476387184

which includes the followup task, "We do a retrospective and look to ways 
to alter our processes to better prevent this in the future." That is, it 
seems to me that the Rust team sees it as a mistake that they ended up in 
this situation.

Josh, do you know if that retrospective has happened? (I see some mumbling 
about NLL -> Polonius, can we be confident something similar won't happen 
with that? :) )

> For updating Rust in Debian stable for the next Firefox ESR update it
> would actually be useful if these violations of the "hard stability
> guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds
> again with the latest Rust.
>
> It also makes me wonder how such regressions slip into Rust releases.

I do conceptually agree with this, even though it is not technically a 
"regression" (because it was really the old compiler that was buggy, not 
the new one). If I understand it right, neither of these lines of code 
were valid with the pre-NLL implementation either. They were only accepted 
by the initial NLL implementation. While the purpose of NLL was to enable 
writing (valid/sound) code that wouldn't have been accepted by the 
previous, simpler implementation of lifetimes, it seems like it should 
have been possible to opt out of it while there were "known bugs" 
precisely to prevent these sort of effective-regressions. (And I suspect 
that it was in fact possible to do so, but perhaps not documented clearly 
/ perhaps there wasn't an awareness that this was a thing that users who 
deeply value stability would want to do.)

And yeah, I can see the value of a --accept-previously-accepted-buggy-code 
flag from the distros' point of view (even if I can also see why Rust 
upstream would not be super excited about it :) ). After all, if the 
choice is between the distro not taking _any_ bug fixes in rustc and the 
distro taking all but one, the latter seems like a better option.

The other takeaway, I think, is that you should regularly compile with 
warnings turned into hard errors. The policy document above (Rust RFC 
1589) says that all such changes need to be a warning and not a hard error 
for at least one compiler reversion. That happened in this case, and both 
fixes were applied in the vendored crates while they were still warnings. 
For any kernel Rust use, I'd strongly advocate for running CI on every 
stable branch after each compiler release, preferably after each compiler 
beta release, and shaking out any warnings - even if they are not warnings 
the compiler will turn into errors on its own, they may still point to 
logic bugs in the code.

(The Rust folks have some automation named "crater" for running these 
sorts of tests across the Rust ecosystem, which the document mentions. I'd 
expect that anything shipped in major distros, including librsvg + its 
dependencies as well as any Rust in the kernel, should be included in 
crater.)

-- 
Geoffrey Thomas
https://ldpreload.com
geofft@ldpreload.com

Re: Linux kernel in-tree Rust support

[Eric W. Biederman]

Nick Desaulniers <ndesaulniers@google.com> writes:

> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree."  I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?

I have two big concerns about actually using rust.

1) How large is the rust language support, and will each rust module
   need to duplicate it.  I seem to remember someone mentioning it is
   noticable in size.

2) What is rust usable for?  The rust type system will not admit
   doubly linked lists (or anything where two pointers point at the
   same memory) unless you are using an unsafe block.

   Now maybe all of that can be wrapped up in libraries written in
   C that Rust can just call, so rust might be useful for building
   drivers.

   What I am certain of is that in the core kernel where I tend to spend
   my time not being able to use doubly linked lists looks like a
   non-starter.

Eric

Re: Linux kernel in-tree Rust support

[Josh Triplett]

On Mon, Jul 13, 2020 at 01:11:13PM -0500, Eric W. Biederman wrote:
> Nick Desaulniers <ndesaulniers@google.com> writes:
> 
> > Hello folks,
> > I'm working on putting together an LLVM "Micro Conference" for the
> > upcoming Linux Plumbers Conf
> > (https://www.linuxplumbersconf.org/event/7/page/47-attend).  It's not
> > solidified yet, but I would really like to run a session on support
> > for Rust "in tree."  I suspect we could cover technical aspects of
> > what that might look like (I have a prototype of that, was trivial to
> > wire up KBuild support), but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >
> > Question to folks explicitly in To:, are you planning on attending plumbers?
> >
> > If so, would this be an interesting topic that you'd participate in?
> 
> I have two big concerns about actually using rust.
> 
> 1) How large is the rust language support, and will each rust module
>    need to duplicate it.  I seem to remember someone mentioning it is
>    noticable in size.

There should only be a single copy, which could either be in the kernel
(if there's Rust code compiled into the kernel) or be in a "rust.ko"
support module. As long as you use the same version of Rust for the
kernel and all modules, you can supply symbols dynamically.

There are a few other steps we can take to control code size, as well.
In particular, there are tradeoffs between performance and size, such as
the amount of code in generics vs non-generics. (We also need to get
some further optimizations into Rust on that front, such as tail
merging.)

As for the size overall, given that we'll just be providing the portions
of "core" and "alloc" that the built code actually uses, and likely not
providing "std" at all, I would expect the size to remain relatively
small.

I very much care about overall kernel size, and I'm happy to help make
sure it remains reasonable.

> 2) What is rust usable for?  The rust type system will not admit
>    doubly linked lists (or anything where two pointers point at the
>    same memory) unless you are using an unsafe block.

There are libraries like intrusive-collections which implement
kernel-style structures with potentially multiple nodes in the structure
that put them into multiple lists at once. I would expect us to use
those (or in some cases implement our own). They don't need to be
written in C, just unsafe Rust that's wrapped up in a safe interface.

Just as the kernel has a variety of higher-level interfaces and data
structures that make working in the kernel sometimes *easier* than the
average C program, I'd expect that we'll end up with common Rust
structures that do what people need, rather than people reimplementing
their own.

- Josh Triplett