Hello folks, I'm working on putting together an LLVM "Micro Conference" for the upcoming Linux Plumbers Conf (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not solidified yet, but I would really like to run a session on support for Rust "in tree." I suspect we could cover technical aspects of what that might look like (I have a prototype of that, was trivial to wire up KBuild support), but also a larger question of "should we do this?" or "how might we place limits on where this can be used?" Question to folks explicitly in To:, are you planning on attending plumbers? If so, would this be an interesting topic that you'd participate in? -- Thanks, ~Nick Desaulniers
Hi Nick, On Thu, Jul 9, 2020 at 8:42 PM Nick Desaulniers <ndesaulniers@google.com> wrote: > > Hello folks, > I'm working on putting together an LLVM "Micro Conference" for the > upcoming Linux Plumbers Conf > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not > solidified yet, but I would really like to run a session on support > for Rust "in tree." I suspect we could cover technical aspects of > what that might look like (I have a prototype of that, was trivial to Agreed, I wrote one ~half a year ago that compiled TUs through rustc mimicking what we do with cc to some degree (i.e. no cargo etc.), and it was quite pleasant. > wire up KBuild support), but also a larger question of "should we do > this?" or "how might we place limits on where this can be used?" Indeed, I would say that is going to be the major topic, not so much the technical bits. > Question to folks explicitly in To:, are you planning on attending plumbers? I wasn't, but it definitely is an interesting topic that I wouldn't want to miss! Cheers, Miguel
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
I hadn't planned to attend the virtual event, but this sounds like a
topic I absolutely have to attend. Please follow up if this proposal
gets accepted.
I'd love to see a path to incorporating Rust into the kernel, as long as
we can ensure that:
- There are appropriate Rustic interfaces that are natural and safe to
use (not just C FFI, and not *just* trivial transformations like
slices instead of buffer+len pairs).
- Those Rustic interfaces are easy to maintain and evolve with the kernel.
- We provide compelling use cases that go beyond just basic safety, such
as concurrency checking, or lifetimes for object ownership.
- We make Rust fit naturally into the kernel's norms and standards,
while also introducing some of Rust's norms and standards where they
make sense. (We want to fit into the kernel, and at the same time, we
don't want to hastily saw off all the corners that don't immediately
fit, because some of those corners provide value. Let's take our
time.)
- We move slowly and carefully, making sure it's a gradual introduction,
and give people time to incorporate the Rust toolchain into their
kernel workflows.
Also, with my "Rust language team lead" hat on, I'd be happy to have the
Linux kernel feeding into Rust language development priorities. If
building Rustic interfaces within the kernel requires some additional
language features, we should see what enhancements to the language would
best serve those requirements. I've often seen the sentiment that
co-evolving Linux and a C compiler would be beneficial for both; I think
the same would be true of Linux and the Rust compiler.
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
Yes, I'll be there.
thanks,
greg k-h
On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > Hello folks,
> > I'm working on putting together an LLVM "Micro Conference" for the
> > upcoming Linux Plumbers Conf
> > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > solidified yet, but I would really like to run a session on support
> > for Rust "in tree." I suspect we could cover technical aspects of
> > what that might look like (I have a prototype of that, was trivial to
> > wire up KBuild support), but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >
> > Question to folks explicitly in To:, are you planning on attending plumbers?
> >
> > If so, would this be an interesting topic that you'd participate in?
>
> Yes, I'll be there.
We actually had this dicussion a while back and there were some more
people interested in this. I'd be interested to attend this and I've
spoken with Kees and a few others about this topic at last Plumbers (I
think Greg might have been around for this informal discussion as well.
But I might be imagining things.).
Christian
On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote: > On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote: > > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote: > > > Hello folks, > > > I'm working on putting together an LLVM "Micro Conference" for the > > > upcoming Linux Plumbers Conf > > > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not > > > solidified yet, but I would really like to run a session on support > > > for Rust "in tree." I suspect we could cover technical aspects of > > > what that might look like (I have a prototype of that, was trivial to > > > wire up KBuild support), but also a larger question of "should we do > > > this?" or "how might we place limits on where this can be used?" > > > > > > Question to folks explicitly in To:, are you planning on attending plumbers? > > > > > > If so, would this be an interesting topic that you'd participate in? > > > > Yes, I'll be there. > > We actually had this dicussion a while back and there were some more > people interested in this. I'd be interested to attend this and I've > spoken with Kees and a few others about this topic at last Plumbers (I > think Greg might have been around for this informal discussion as well. > But I might be imagining things.). I'm quite interested in this topic still, yes. :) (And will be attending Plumbers.) We had interesting discussions with the LSS 2019 "Rust in the kernel"[1] speakers, and I know Nick has reviewed their work[2] too. Perhaps we should reach out to them as well? -Kees [1] https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf [2] https://github.com/fishinabarrel/linux-kernel-module-rust -- Kees Cook
On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote:
> On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > > Hello folks,
> > > I'm working on putting together an LLVM "Micro Conference" for the
> > > upcoming Linux Plumbers Conf
> > > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > > solidified yet, but I would really like to run a session on support
> > > for Rust "in tree." I suspect we could cover technical aspects of
> > > what that might look like (I have a prototype of that, was trivial to
> > > wire up KBuild support), but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >
> > > Question to folks explicitly in To:, are you planning on attending plumbers?
> > >
> > > If so, would this be an interesting topic that you'd participate in?
> >
> > Yes, I'll be there.
>
> We actually had this dicussion a while back and there were some more
> people interested in this. I'd be interested to attend this and I've
> spoken with Kees and a few others about this topic at last Plumbers (I
> think Greg might have been around for this informal discussion as well.
> But I might be imagining things.).
I was around for one of the informal conversations with Greg and Kees
and others.
As I recall, Greg's biggest condition for initial introduction of this
was to do the same kind of "turn this Kconfig option on and turn an
option under it off" trick that LTO uses, so that neither "make
allnoconfig" nor "make allyesconfig" would require Rust until we've had
plenty of time to experiment with it. And that seems entirely
reasonable to me too.
- Josh
On Fri, Jul 10, 2020 at 12:28:30PM -0400, Alex Gaynor wrote:
> :wave:,
>
> Hey Kees, that's Geoffrey Thomas and I, we're both on this thread :-)
*face palm* Hello! I swear I can read. Though perhaps not well enough
before lunch. :)
--
Kees Cook
On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <josh@joshtriplett.org> wrote:
>
> As I recall, Greg's biggest condition for initial introduction of this
> was to do the same kind of "turn this Kconfig option on and turn an
> option under it off" trick that LTO uses, so that neither "make
> allnoconfig" nor "make allyesconfig" would require Rust until we've had
> plenty of time to experiment with it.
No, please make it a "is rust available" automatic config option. The
exact same way we already do the compiler versions and check for
various availability of compiler flags at config time.
See init/Kconfig for things like
config LD_IS_LLD
def_bool $(success,$(LD) -v | head -n 1 | grep -q LLD)
and the rust support should be similar. Something like
config RUST_IS_AVAILABLE
def_bool $(success,$(RUST) ..sometest..)
because I _don't_ want us to be in the situation where any new rust
support isn't even build-tested by default.
Quite the reverse. I'd want the first rust driver (or whatever) to be
introduced in such a simple format that failures will be obvious and
simple.
The _worst_ situation to be in is that s (small) group of people start
testing their very special situation, and do bad and crazy things
because "nobody else cares, it's hidden".
No, thank you.
Linus
On Thu, 9 Jul 2020, Nick Desaulniers wrote: > Hello folks, > I'm working on putting together an LLVM "Micro Conference" for the > upcoming Linux Plumbers Conf > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not > solidified yet, but I would really like to run a session on support > for Rust "in tree." I suspect we could cover technical aspects of > what that might look like (I have a prototype of that, was trivial to > wire up KBuild support), but also a larger question of "should we do > this?" or "how might we place limits on where this can be used?" > > Question to folks explicitly in To:, are you planning on attending plumbers? > > If so, would this be an interesting topic that you'd participate in? Like Alex - I hadn't decided yet but I'll definitely attend for this! -- Geoffrey Thomas https://ldpreload.com geofft@ldpreload.com
On Fri, Jul 10, 2020 at 04:54:11PM -0700, Linus Torvalds wrote:
> On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <josh@joshtriplett.org> wrote:
> > As I recall, Greg's biggest condition for initial introduction of this
> > was to do the same kind of "turn this Kconfig option on and turn an
> > option under it off" trick that LTO uses, so that neither "make
> > allnoconfig" nor "make allyesconfig" would require Rust until we've had
> > plenty of time to experiment with it.
>
> No, please make it a "is rust available" automatic config option. The
> exact same way we already do the compiler versions and check for
> various availability of compiler flags at config time.
That sounds even better, and will definitely allow for more testing.
We just need to make sure that any kernel CI infrastructure tests that
right away, then, so that failures don't get introduced by a patch from
someone without a Rust toolchain and not noticed until someone with a
Rust toolchain tests it.
- Josh
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote: >... > but also a larger question of "should we do > this?" or "how might we place limits on where this can be used?" >... I won't attend, but I do have a topic that should be covered: Firefox always depends on recent Rust, which forces distributions to update Rust in stable releases. As an example: Ubuntu LTS releases upgrade to a new Rust version every 1-2 months. Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0. It would not sound good to me if security updates of distribution kernels might additionally end up using a different version of the Rust compiler - the toolchain for the kernel should be stable. Would Rust usage in the kernel require distributions to ship a "Rust for Firefox" and a "Rust for the kernel"? > Thanks, > ~Nick Desaulniers cu Adrian
On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> >...
> > but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >...
>
> I won't attend, but I do have a topic that should be covered:
>
> Firefox always depends on recent Rust, which forces distributions to
> update Rust in stable releases.
>
> As an example:
> Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
>
> It would not sound good to me if security updates of distribution
> kernels might additionally end up using a different version of the
> Rust compiler - the toolchain for the kernel should be stable.
>
> Would Rust usage in the kernel require distributions to ship
> a "Rust for Firefox" and a "Rust for the kernel"?
Rust has hard stability guarantees when upgrading from one stable
version to the next. If code compiles with a given stable version of
Rust, it'll compile with a newer stable version of Rust. Given that, a
stable distribution will just need a single sufficiently up-to-date Rust
that meets the minimum version requirements of both Firefox and Linux.
(That would not apply if the kernel used nightly Rust, since
nightly-only features are allowed to change before becoming stable;
that's one reason why we should use stable Rust, and try to get Firefox
to stick to stable Rust.)
On Sun, Jul 12, 2020 at 12:39 PM Josh Triplett <josh@joshtriplett.org> wrote:
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next.
I think the worry is more about actual compiler bugs, not the set of
exposed features.
That's always been the biggest pain point. Compiler bugs are very
rare, but they are so incredibly hard to debug when they happen that
they end up being extra special.
Random "we need this compiler for this feature" is actually fairly
rare. Yes, the most recent case of me just saying "let's use 4.9
rather than 4.8" was due to that, but honestly, that's the exception
rather than the rule, and is to occasionally simplify the code (and
the test coverage).
The most common case of compiler version checks are due to "compiler
XYZ is known to mis-compile ABC on target IDK".
Linus
On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > >...
> > > but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >...
> >
> > I won't attend, but I do have a topic that should be covered:
> >
> > Firefox always depends on recent Rust, which forces distributions to
> > update Rust in stable releases.
> >
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> >
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> >
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>...
API stability avoids problems that are visible early as build errors.
Rust cannot offer a hard stability guarantee that there will never be
a code generation regression on any platform.
Rust gets updated frequently.
Sometimes this also changes the LLVM version used by Rust.
Debian stable supports targets like ARMv5 and 32bit MIPS.
Distribution kernel updates are often automatically installed
on user hardware.
cu
Adrian
Nick Desaulniers <ndesaulniers@google.com> writes:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
I have two big concerns about actually using rust.
1) How large is the rust language support, and will each rust module
need to duplicate it. I seem to remember someone mentioning it is
noticable in size.
2) What is rust usable for? The rust type system will not admit
doubly linked lists (or anything where two pointers point at the
same memory) unless you are using an unsafe block.
Now maybe all of that can be wrapped up in libraries written in
C that Rust can just call, so rust might be useful for building
drivers.
What I am certain of is that in the core kernel where I tend to spend
my time not being able to use doubly linked lists looks like a
non-starter.
Eric
On Mon, Jul 13, 2020 at 01:11:13PM -0500, Eric W. Biederman wrote: > Nick Desaulniers <ndesaulniers@google.com> writes: > > > Hello folks, > > I'm working on putting together an LLVM "Micro Conference" for the > > upcoming Linux Plumbers Conf > > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not > > solidified yet, but I would really like to run a session on support > > for Rust "in tree." I suspect we could cover technical aspects of > > what that might look like (I have a prototype of that, was trivial to > > wire up KBuild support), but also a larger question of "should we do > > this?" or "how might we place limits on where this can be used?" > > > > Question to folks explicitly in To:, are you planning on attending plumbers? > > > > If so, would this be an interesting topic that you'd participate in? > > I have two big concerns about actually using rust. > > 1) How large is the rust language support, and will each rust module > need to duplicate it. I seem to remember someone mentioning it is > noticable in size. There should only be a single copy, which could either be in the kernel (if there's Rust code compiled into the kernel) or be in a "rust.ko" support module. As long as you use the same version of Rust for the kernel and all modules, you can supply symbols dynamically. There are a few other steps we can take to control code size, as well. In particular, there are tradeoffs between performance and size, such as the amount of code in generics vs non-generics. (We also need to get some further optimizations into Rust on that front, such as tail merging.) As for the size overall, given that we'll just be providing the portions of "core" and "alloc" that the built code actually uses, and likely not providing "std" at all, I would expect the size to remain relatively small. I very much care about overall kernel size, and I'm happy to help make sure it remains reasonable. > 2) What is rust usable for? The rust type system will not admit > doubly linked lists (or anything where two pointers point at the > same memory) unless you are using an unsafe block. There are libraries like intrusive-collections which implement kernel-style structures with potentially multiple nodes in the structure that put them into multiple lists at once. I would expect us to use those (or in some cases implement our own). They don't need to be written in C, just unsafe Rust that's wrapped up in a safe interface. Just as the kernel has a variety of higher-level interfaces and data structures that make working in the kernel sometimes *easier* than the average C program, I'd expect that we'll end up with common Rust structures that do what people need, rather than people reimplementing their own. - Josh Triplett
From: Adrian Bunk > Sent: 12 July 2020 21:45 .... > Rust gets updated frequently. > Sometimes this also changes the LLVM version used by Rust. > Debian stable supports targets like ARMv5 and 32bit MIPS. > Distribution kernel updates are often automatically installed > on user hardware. This reminds me of why I never want to use an online compiler service - never mind how hard companies push them. If I need to do a bug-fix build of something that was released 2 (or more) years ago I want to use exactly the same toolchain (warts and all) that was used for the original build. If the compiler has changed I need to do a full test - just in case it compiles some 'dodgy' code differently. With the same compiler I only need to test the fix. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <josh@joshtriplett.org> wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> >
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> >
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> >
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>
> (That would not apply if the kernel used nightly Rust, since
> nightly-only features are allowed to change before becoming stable;
> that's one reason why we should use stable Rust, and try to get Firefox
> to stick to stable Rust.)
I would expect we'd want a fairly tight coupling between kernel
releases and minimum rust releases at first. Whatever is the latest
stable rust version during the kernel's merge window might be
assumed to be the minimum version for the life of that kernel, but
an LTS release would not suddenly start relying on features
from a newer compiler (thought it might warn about known bugs).
This might mean that linux-next requires a beta version of rust, if
the release is expected before the merge window and it contains
an important change. Staying with fairly recent versions of clang
certainly helped in getting clang and the kernel to co-evolve and
get to the point we are now in using it as an alternative to gcc.
While Linux used to build with 12 year old compilers (4.1 until
2018), we now require a 6 year old gcc (4.9) or 1 year old
clang/llvm. I don't know whether these will fully converge over
time but it seems sensible that the minimum rust frontend version
we require for a new kernel release would eventually also fall
in that range, requiring a compiler that is no more than a few
years old, but not requiring the latest stable release.
Arnd
On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote: > On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <josh@joshtriplett.org> wrote: > > On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote: > > > > > > As an example: > > > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months. > > > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0. > > > > > > It would not sound good to me if security updates of distribution > > > kernels might additionally end up using a different version of the > > > Rust compiler - the toolchain for the kernel should be stable. > > > > > > Would Rust usage in the kernel require distributions to ship > > > a "Rust for Firefox" and a "Rust for the kernel"? > > > > Rust has hard stability guarantees when upgrading from one stable > > version to the next. If code compiles with a given stable version of > > Rust, it'll compile with a newer stable version of Rust. Given that, a > > stable distribution will just need a single sufficiently up-to-date Rust > > that meets the minimum version requirements of both Firefox and Linux. > > > > (That would not apply if the kernel used nightly Rust, since > > nightly-only features are allowed to change before becoming stable; > > that's one reason why we should use stable Rust, and try to get Firefox > > to stick to stable Rust.) > > I would expect we'd want a fairly tight coupling between kernel > releases and minimum rust releases at first. Whatever is the latest > stable rust version during the kernel's merge window might be > assumed to be the minimum version for the life of that kernel, but > an LTS release would not suddenly start relying on features > from a newer compiler (thought it might warn about known bugs). Agreed; we should be careful that any backported fix to an LTS kernel does not increase the required Rust toolchain. > While Linux used to build with 12 year old compilers (4.1 until > 2018), we now require a 6 year old gcc (4.9) or 1 year old > clang/llvm. I don't know whether these will fully converge over > time but it seems sensible that the minimum rust frontend version > we require for a new kernel release would eventually also fall > in that range, requiring a compiler that is no more than a few > years old, but not requiring the latest stable release. I expect in the short term that we will likely have a need for features from recent stable releases, especially when those features were added specifically to support the kernel or similar, but the rate at which we need new features will slow over time, and eventually we'll go from "latest stable" to "within a year or so".
On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote: > > I would expect we'd want a fairly tight coupling between kernel > releases and minimum rust releases at first. Whatever is the latest > stable rust version during the kernel's merge window might be > assumed to be the minimum version for the life of that kernel, but > an LTS release would not suddenly start relying on features > from a newer compiler (thought it might warn about known bugs). > > This might mean that linux-next requires a beta version of rust, if > the release is expected before the merge window and it contains > an important change. I would expect this is absolutely not wanted, it would make testing recent kernels very hard. If you want to keep a tool that tightly to the kernel, please bundle it with the kernel and build it as part of the kernel build. I would suggest to start with a proper design/specification what the kernel wants to use, so that you are confident that a compiler implementing this will be sufficient for the next 5 years. As a secondary benefit, starting with a proper design often brings a better result than adding permanent features piece by piece. As a tertiary benefit, it would avoid tying the kernel to one specific compiler implementation. A compiler like mrustc or a hypothetical Rust frontend for gcc could then implement a superset of what the kernel needs. > Staying with fairly recent versions of clang > certainly helped in getting clang and the kernel to co-evolve and > get to the point we are now in using it as an alternative to gcc. The main difference is between an alternative to an existing tool, and a mandatory new tool. > While Linux used to build with 12 year old compilers (4.1 until > 2018), we now require a 6 year old gcc (4.9) or 1 year old > clang/llvm. I don't know whether these will fully converge over > time but it seems sensible that the minimum rust frontend version > we require for a new kernel release would eventually also fall > in that range, requiring a compiler that is no more than a few > years old, but not requiring the latest stable release. The correct range for a mandatory tool are the 6 to 12 years for gcc. Debian stable and Ubuntu LTS are providing (different) mechanisms for installing the kernel from the next stable/LTS release 2 years later[1] for supporting new hardware. If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with the 2019 gcc 9 there would be pain downstream. In the embedded world spreads far wider than these 3 years are common. I would currently have a real-life usecase for compiling a recent kernel with a gcc 4.0 (sic) toolchain. Properly supporting 15 year old toolchains would be painful upstream, supporting 6 year old toolchains is a reasonable compromise between not being too painful upstream while rarely causing pain downstream. What applies to gcc does also apply to other external tools used during the kernel build. > Arnd cu Adrian [1] following a new upstream kernel stable branch every 6 months (Ubuntu) or the latest upstream stable kernels (Debian) until this is reached
From: Adrian Bunk > Sent: 19 July 2020 19:19 ... > The correct range for a mandatory tool are the 6 to 12 years for gcc. > > Debian stable and Ubuntu LTS are providing (different) mechanisms > for installing the kernel from the next stable/LTS release 2 years > later[1] for supporting new hardware. > If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with > the 2019 gcc 9 there would be pain downstream. We have customers that are (still) using RHEL6 (2.6.32 era). Since we have to build C++ programs that will run on those systems we have systems with very old toolchains. (Yes, you can build in a chroot or an a VM but that needs setting up quite carefully - easier to keep the old machine.) It is rather a PITA when kernel builds deprecate the installed gcc. Now I'm not sure a 5.x kernel will run with the userspace from RHEL6. But I know it runs with that from Ubuntu 13.04. In a few years I'll be able to use the 13.04 system for release builds but it is still far too new for that. However I can no longer build 'current' custom kernels on it. I've other (newer) systems but they aren't any (maybe much) faster. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Hi! > > No, please make it a "is rust available" automatic config option. The > > exact same way we already do the compiler versions and check for > > various availability of compiler flags at config time. > > That sounds even better, and will definitely allow for more testing. > > We just need to make sure that any kernel CI infrastructure tests that > right away, then, so that failures don't get introduced by a patch from > someone without a Rust toolchain and not noticed until someone with a > Rust toolchain tests it. So... I'm fan of Rust, but while trying to use it one thing was obvious: it takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM. Kernel is quite big project, can CI infrastructure handle additional load? Will developers see significantly longer compile times when Rust is widespread? Best regards, Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote:
> > We just need to make sure that any kernel CI infrastructure tests that
> > right away, then, so that failures don't get introduced by a patch from
> > someone without a Rust toolchain and not noticed until someone with a
> > Rust toolchain tests it.
>
> So... I'm fan of Rust, but while trying to use it one thing was obvious: it
> takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
>
> Kernel is quite big project, can CI infrastructure handle additional load?
>
> Will developers see significantly longer compile times when Rust is widespread?
I wouldn't expect the addition of Rust to the kernel to substantially
impact overall build time; on balance, I'd expect the major bottleneck
in kernel builds to continue to be linking and other serialized steps,
not compiling and other highly parallel steps.
There are also *many* things that can be done to improve Rust build time
in a project. And I don't expect that in-kernel Rust will have many
dependencies on third-party crates (since they'd need to be checked into
the tree).
On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote: >... > Rust has hard stability guarantees when upgrading from one stable > version to the next. If code compiles with a given stable version of > Rust, it'll compile with a newer stable version of Rust. >... In librsvg, breakages with more recent Rust versions in the past year required updates of two vendored crates: https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902 https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d For updating Rust in Debian stable for the next Firefox ESR update it would actually be useful if these violations of the "hard stability guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds again with the latest Rust. It also makes me wonder how such regressions slip into Rust releases. cu Adrian
On Mon, 24 Aug 2020, Adrian Bunk wrote: > In librsvg, breakages with more recent Rust versions in the past year > required updates of two vendored crates: Interesting! > https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902 Looks like this was, for a while, a warning and not an error: https://github.com/servo/rust-cssparser/commit/3c98d22c5de3b696bf1fde2b6c90069812312aa6 = warning: this error has been downgraded to a warning for backwards compatibility with previous releases = warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future > https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d Same here, and it looks like the same warning/error, too: https://github.com/dimforge/nalgebra/issues/561 = warning: this error has been downgraded to a warning for backwards compatibility with previous releases = warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future Following some links in that second issue, I see that this seems to be a summary of what's going on: https://github.com/rust-lang/rust/issues/59159 in particular this paragraph: "at the time that NLL [non-lexical lifetimes] was stabilized, the compiler's acceptance of this borrowing pattern was categorized by the NLL team as a 'known bug'. The NLL team assumed that, as a bug fix, the compiler would be allowed to start rejecting the pattern in the future." That is, both of these cases were code that should never have been accepted by the compiler because it is unsound, but the initial implementation of NLL was not clever enough to detect it. They later turned it into a warning, and later turned that into an error. There is a link to this policy document which explains their process for breaking existing code in the case where it's necessary to fix a compiler bug or similar: https://github.com/rust-lang/rfcs/blob/master/text/1589-rustc-bug-fix-procedure.md There is also a link to this comment about why they decided to take this approach: https://github.com/rust-lang/rust/pull/58739#issuecomment-476387184 which includes the followup task, "We do a retrospective and look to ways to alter our processes to better prevent this in the future." That is, it seems to me that the Rust team sees it as a mistake that they ended up in this situation. Josh, do you know if that retrospective has happened? (I see some mumbling about NLL -> Polonius, can we be confident something similar won't happen with that? :) ) > For updating Rust in Debian stable for the next Firefox ESR update it > would actually be useful if these violations of the "hard stability > guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds > again with the latest Rust. > > It also makes me wonder how such regressions slip into Rust releases. I do conceptually agree with this, even though it is not technically a "regression" (because it was really the old compiler that was buggy, not the new one). If I understand it right, neither of these lines of code were valid with the pre-NLL implementation either. They were only accepted by the initial NLL implementation. While the purpose of NLL was to enable writing (valid/sound) code that wouldn't have been accepted by the previous, simpler implementation of lifetimes, it seems like it should have been possible to opt out of it while there were "known bugs" precisely to prevent these sort of effective-regressions. (And I suspect that it was in fact possible to do so, but perhaps not documented clearly / perhaps there wasn't an awareness that this was a thing that users who deeply value stability would want to do.) And yeah, I can see the value of a --accept-previously-accepted-buggy-code flag from the distros' point of view (even if I can also see why Rust upstream would not be super excited about it :) ). After all, if the choice is between the distro not taking _any_ bug fixes in rustc and the distro taking all but one, the latter seems like a better option. The other takeaway, I think, is that you should regularly compile with warnings turned into hard errors. The policy document above (Rust RFC 1589) says that all such changes need to be a warning and not a hard error for at least one compiler reversion. That happened in this case, and both fixes were applied in the vendored crates while they were still warnings. For any kernel Rust use, I'd strongly advocate for running CI on every stable branch after each compiler release, preferably after each compiler beta release, and shaking out any warnings - even if they are not warnings the compiler will turn into errors on its own, they may still point to logic bugs in the code. (The Rust folks have some automation named "crater" for running these sorts of tests across the Rust ecosystem, which the document mentions. I'd expect that anything shipped in major distros, including librsvg + its dependencies as well as any Rust in the kernel, should be included in crater.) -- Geoffrey Thomas https://ldpreload.com geofft@ldpreload.com
[-- Attachment #1: Type: text/plain, Size: 1531 bytes --] On Tue 2020-07-28 23:34:17, Josh Triplett wrote: > On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote: > > > We just need to make sure that any kernel CI infrastructure tests that > > > right away, then, so that failures don't get introduced by a patch from > > > someone without a Rust toolchain and not noticed until someone with a > > > Rust toolchain tests it. > > > > So... I'm fan of Rust, but while trying to use it one thing was obvious: it > > takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM. > > > > Kernel is quite big project, can CI infrastructure handle additional load? > > > > Will developers see significantly longer compile times when Rust is widespread? > > I wouldn't expect the addition of Rust to the kernel to substantially > impact overall build time; on balance, I'd expect the major bottleneck > in kernel builds to continue to be linking and other serialized steps, > not compiling and other highly parallel steps. Well.. not everyone has 32 cores in their notebook. > There are also *many* things that can be done to improve Rust build time > in a project. And I don't expect that in-kernel Rust will have many > dependencies on third-party crates (since they'd need to be checked into Okay. I did some refactoring recently and I really wished kernel was in Rust (and not in C)... so lets see what happens. Best regards, Pavel -- http://www.livejournal.com/~pavelmachek [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --]
On Thu, Dec 17, 2020 at 10:45 PM Pavel Machek <pavel@ucw.cz> wrote: > > Well.. not everyone has 32 cores in their notebook. It is true that complex Rust, like complex C++, does have high compilation times. But it all depends on how much one relies on certain language features. Straightforward Rust code is quick to compile. For reference, some quick stats :-) On a given machine, building v5.10 with a minimal config under -j3 takes 3 minutes. With Rust support enabled and 4 trivial Rust modules, it takes 50 seconds more. "A big increase!", you may indeed claim, but those 50 seconds are basically all spent on the shared Rust support. The actual Rust modules compile very quickly afterwards. For example, touching either `drivers/char/mem.c` or one of the trivial Rust modules takes the same time in that machine: 10 seconds. This is for a minimal config -- when you start dealing with `allmodconfig` builds, or when you start having a hundred Rust modules instead of 4, the upfront cost becomes very small per Rust module. > Okay. I did some refactoring recently and I really wished kernel was > in Rust (and not in C)... so lets see what happens. Indeed, I think it is definitely worth it. Cheers, Miguel