[TECH TOPIC] Rust

[TECH TOPIC] Rust

[Miguel Ojeda]

The effort to add Rust support to the kernel is ongoing. There has
been progress in different areas during the last year, and there are
several topics that could benefit from discussion:

  - Dividing the `kernel` crate into pieces, dependency management
between internal crates, writing crates in the rest of the kernel
tree, etc.

  - Whether to allow dependencies on external crates and vendoring of
useful third-party crates.

  - Toolchain requirements in the future and status of Rust unstable features.

  - The future of GCC builds: upcoming compilers, their status and
ETAs, adding the kernel as a testing case for them...

  - Steps needed for further integration in the different kernel CIs,
running tests, etc.

  - Documentation setup on kernel.org and integration between
Sphinx/kernel-doc and rustdoc (this can be part of the documentation
tech topic submitted earlier by Jon).

  - Discussion with prospective maintainers that want to use Rust for
their subsystem.

Cheers,
Miguel

Re: [TECH TOPIC] Rust

[Laurent Pinchart]

Hi Miguel,

On Sat, Jun 18, 2022 at 10:33:01PM +0200, Miguel Ojeda wrote:
> The effort to add Rust support to the kernel is ongoing. There has
> been progress in different areas during the last year, and there are
> several topics that could benefit from discussion:
> 
>   - Dividing the `kernel` crate into pieces, dependency management
> between internal crates, writing crates in the rest of the kernel
> tree, etc.
> 
>   - Whether to allow dependencies on external crates and vendoring of
> useful third-party crates.
> 
>   - Toolchain requirements in the future and status of Rust unstable features.
> 
>   - The future of GCC builds: upcoming compilers, their status and
> ETAs, adding the kernel as a testing case for them...
> 
>   - Steps needed for further integration in the different kernel CIs,
> running tests, etc.
> 
>   - Documentation setup on kernel.org and integration between
> Sphinx/kernel-doc and rustdoc (this can be part of the documentation
> tech topic submitted earlier by Jon).
> 
>   - Discussion with prospective maintainers that want to use Rust for
> their subsystem.

All previous topics but this one are technical. To restore the balance a
bit (and also because I believe it's important to the success of this
project :-)), I'd like to also discuss the impact on the kernel
maintenance at large, beyond just the maintainers who opt in to be early
adopters of rust in their subsystem.

-- 
Regards,

Laurent Pinchart

[Ksummit-discuss] [TECH TOPIC] Rust

[Laurent Pinchart]

Hi Miguel,

On Sat, Jun 18, 2022 at 10:33:01PM +0200, Miguel Ojeda wrote:
> The effort to add Rust support to the kernel is ongoing. There has
> been progress in different areas during the last year, and there are
> several topics that could benefit from discussion:
> 
>   - Dividing the `kernel` crate into pieces, dependency management
> between internal crates, writing crates in the rest of the kernel
> tree, etc.
> 
>   - Whether to allow dependencies on external crates and vendoring of
> useful third-party crates.
> 
>   - Toolchain requirements in the future and status of Rust unstable features.
> 
>   - The future of GCC builds: upcoming compilers, their status and
> ETAs, adding the kernel as a testing case for them...
> 
>   - Steps needed for further integration in the different kernel CIs,
> running tests, etc.
> 
>   - Documentation setup on kernel.org and integration between
> Sphinx/kernel-doc and rustdoc (this can be part of the documentation
> tech topic submitted earlier by Jon).
> 
>   - Discussion with prospective maintainers that want to use Rust for
> their subsystem.

All previous topics but this one are technical. To restore the balance a
bit (and also because I believe it's important to the success of this
project :-)), I'd like to also discuss the impact on the kernel
maintenance at large, beyond just the maintainers who opt in to be early
adopters of rust in their subsystem.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Rust

[Laurent Pinchart]

On Sat, Jun 18, 2022 at 11:42:09PM +0300, Laurent Pinchart wrote:
> Hi Miguel,
> 
> On Sat, Jun 18, 2022 at 10:33:01PM +0200, Miguel Ojeda wrote:
> > The effort to add Rust support to the kernel is ongoing. There has
> > been progress in different areas during the last year, and there are
> > several topics that could benefit from discussion:
> > 
> >   - Dividing the `kernel` crate into pieces, dependency management
> > between internal crates, writing crates in the rest of the kernel
> > tree, etc.
> > 
> >   - Whether to allow dependencies on external crates and vendoring of
> > useful third-party crates.
> > 
> >   - Toolchain requirements in the future and status of Rust unstable features.
> > 
> >   - The future of GCC builds: upcoming compilers, their status and
> > ETAs, adding the kernel as a testing case for them...
> > 
> >   - Steps needed for further integration in the different kernel CIs,
> > running tests, etc.
> > 
> >   - Documentation setup on kernel.org and integration between
> > Sphinx/kernel-doc and rustdoc (this can be part of the documentation
> > tech topic submitted earlier by Jon).
> > 
> >   - Discussion with prospective maintainers that want to use Rust for
> > their subsystem.
> 
> All previous topics but this one are technical. To restore the balance a
> bit (and also because I believe it's important to the success of this
> project :-)), I'd like to also discuss the impact on the kernel
> maintenance at large, beyond just the maintainers who opt in to be early
> adopters of rust in their subsystem.

This may be better addressed as part of the maintainers summit (I've
just seen your proposal for that event), but I'm concerned that the
audience there will be very restricted due to the size of the event.

-- 
Regards,

Laurent Pinchart

[Ksummit-discuss] [TECH TOPIC] Rust

[Laurent Pinchart]

On Sat, Jun 18, 2022 at 11:42:09PM +0300, Laurent Pinchart wrote:
> Hi Miguel,
> 
> On Sat, Jun 18, 2022 at 10:33:01PM +0200, Miguel Ojeda wrote:
> > The effort to add Rust support to the kernel is ongoing. There has
> > been progress in different areas during the last year, and there are
> > several topics that could benefit from discussion:
> > 
> >   - Dividing the `kernel` crate into pieces, dependency management
> > between internal crates, writing crates in the rest of the kernel
> > tree, etc.
> > 
> >   - Whether to allow dependencies on external crates and vendoring of
> > useful third-party crates.
> > 
> >   - Toolchain requirements in the future and status of Rust unstable features.
> > 
> >   - The future of GCC builds: upcoming compilers, their status and
> > ETAs, adding the kernel as a testing case for them...
> > 
> >   - Steps needed for further integration in the different kernel CIs,
> > running tests, etc.
> > 
> >   - Documentation setup on kernel.org and integration between
> > Sphinx/kernel-doc and rustdoc (this can be part of the documentation
> > tech topic submitted earlier by Jon).
> > 
> >   - Discussion with prospective maintainers that want to use Rust for
> > their subsystem.
> 
> All previous topics but this one are technical. To restore the balance a
> bit (and also because I believe it's important to the success of this
> project :-)), I'd like to also discuss the impact on the kernel
> maintenance at large, beyond just the maintainers who opt in to be early
> adopters of rust in their subsystem.

This may be better addressed as part of the maintainers summit (I've
just seen your proposal for that event), but I'm concerned that the
audience there will be very restricted due to the size of the event.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Rust

[Miguel Ojeda]

On Sat, Jun 18, 2022 at 10:42 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> All previous topics but this one are technical. To restore the balance a
> bit (and also because I believe it's important to the success of this
> project :-)), I'd like to also discuss the impact on the kernel
> maintenance at large, beyond just the maintainers who opt in to be early
> adopters of rust in their subsystem.

Definitely, I think it is a fair topic (there was some discussion on
this last year, but it may be good to revisit it).

Cheers,
Miguel

Re: [TECH TOPIC] Rust

[Christoph Hellwig]

On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> All previous topics but this one are technical. To restore the balance a
> bit (and also because I believe it's important to the success of this
> project :-)), I'd like to also discuss the impact on the kernel
> maintenance at large, beyond just the maintainers who opt in to be early
> adopters of rust in their subsystem.

Yes.  That is I think the most important point.

I've played around with rust a bit for userspace project, and there is
things I like a lot like strict type safety and the ownership model, and
things I detest like verbose and unlogic syntax, the cargo cult of
vendoring libraries instead of a sane shared library and versioning
model (althought that should be largerly irrelevant for the kernel),
and compared to many other languages it seems overall rather sane.

But I'm really worried about the impact on the kernel, as interface
between languages are a real pain, and so far I'm not convinced at
all that this pain is worth the gain, even if that could change in
the future.  It doesn't help that the showcase seems to be binder,
code that should have been in userspace from the start.

Btw, a lot of these aspects also apply to the whole drivers in eBPF
discussion.

[Ksummit-discuss] [TECH TOPIC] Rust

[Christoph Hellwig]

On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> All previous topics but this one are technical. To restore the balance a
> bit (and also because I believe it's important to the success of this
> project :-)), I'd like to also discuss the impact on the kernel
> maintenance at large, beyond just the maintainers who opt in to be early
> adopters of rust in their subsystem.

Yes.  That is I think the most important point.

I've played around with rust a bit for userspace project, and there is
things I like a lot like strict type safety and the ownership model, and
things I detest like verbose and unlogic syntax, the cargo cult of
vendoring libraries instead of a sane shared library and versioning
model (althought that should be largerly irrelevant for the kernel),
and compared to many other languages it seems overall rather sane.

But I'm really worried about the impact on the kernel, as interface
between languages are a real pain, and so far I'm not convinced at
all that this pain is worth the gain, even if that could change in
the future.  It doesn't help that the showcase seems to be binder,
code that should have been in userspace from the start.

Btw, a lot of these aspects also apply to the whole drivers in eBPF
discussion.

Re: [TECH TOPIC] Rust

[Laurent Pinchart]

On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > All previous topics but this one are technical. To restore the balance a
> > bit (and also because I believe it's important to the success of this
> > project :-)), I'd like to also discuss the impact on the kernel
> > maintenance at large, beyond just the maintainers who opt in to be early
> > adopters of rust in their subsystem.
> 
> Yes.  That is I think the most important point.
> 
> I've played around with rust a bit for userspace project, and there is
> things I like a lot like strict type safety and the ownership model, and
> things I detest like verbose and unlogic syntax, the cargo cult of
> vendoring libraries instead of a sane shared library and versioning
> model (althought that should be largerly irrelevant for the kernel),
> and compared to many other languages it seems overall rather sane.
> 
> But I'm really worried about the impact on the kernel, as interface
> between languages are a real pain, and so far I'm not convinced at
> all that this pain is worth the gain, even if that could change in
> the future.

One point I'd like to explicitly address, as it seems to be very
relevant to me when it comes to whether or not the pain is worth the
gain, is how we'll deal with the fact that the rust compiler will tell a
non-negligible [*] part of the kernel developers that they don't have
the skills to write kernel code. Will we have a credible option to offer
there to help people improve their skills set, and how will we deal with
the fact that some people will be left on the side of the road ? Or
would we acknowledge what may be the elephant in the room that this
would actually be a good thing for the kernel code quality ? What about
the impact on a community that is already overworked and prone to
burn-out ?

[*] I have no way to quantify this at the moment, maybe I'm overly
pessimistic, and the number will likely vary depending on areas,
probably impacting BSPs differently than mainline.

> It doesn't help that the showcase seems to be binder,
> code that should have been in userspace from the start.
> 
> Btw, a lot of these aspects also apply to the whole drivers in eBPF
> discussion.

-- 
Regards,

Laurent Pinchart

[Ksummit-discuss] [TECH TOPIC] Rust

[Laurent Pinchart]

On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > All previous topics but this one are technical. To restore the balance a
> > bit (and also because I believe it's important to the success of this
> > project :-)), I'd like to also discuss the impact on the kernel
> > maintenance at large, beyond just the maintainers who opt in to be early
> > adopters of rust in their subsystem.
> 
> Yes.  That is I think the most important point.
> 
> I've played around with rust a bit for userspace project, and there is
> things I like a lot like strict type safety and the ownership model, and
> things I detest like verbose and unlogic syntax, the cargo cult of
> vendoring libraries instead of a sane shared library and versioning
> model (althought that should be largerly irrelevant for the kernel),
> and compared to many other languages it seems overall rather sane.
> 
> But I'm really worried about the impact on the kernel, as interface
> between languages are a real pain, and so far I'm not convinced at
> all that this pain is worth the gain, even if that could change in
> the future.

One point I'd like to explicitly address, as it seems to be very
relevant to me when it comes to whether or not the pain is worth the
gain, is how we'll deal with the fact that the rust compiler will tell a
non-negligible [*] part of the kernel developers that they don't have
the skills to write kernel code. Will we have a credible option to offer
there to help people improve their skills set, and how will we deal with
the fact that some people will be left on the side of the road ? Or
would we acknowledge what may be the elephant in the room that this
would actually be a good thing for the kernel code quality ? What about
the impact on a community that is already overworked and prone to
burn-out ?

[*] I have no way to quantify this at the moment, maybe I'm overly
pessimistic, and the number will likely vary depending on areas,
probably impacting BSPs differently than mainline.

> It doesn't help that the showcase seems to be binder,
> code that should have been in userspace from the start.
> 
> Btw, a lot of these aspects also apply to the whole drivers in eBPF
> discussion.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Rust

[James Bottomley]

On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > All previous topics but this one are technical. To restore the
> > > balance a bit (and also because I believe it's important to the
> > > success of this project :-)), I'd like to also discuss the impact
> > > on the kernel maintenance at large, beyond just the maintainers
> > > who opt in to be early adopters of rust in their subsystem.
> > 
> > Yes.  That is I think the most important point.
> > 
> > I've played around with rust a bit for userspace project, and there
> > is things I like a lot like strict type safety and the ownership
> > model, and things I detest like verbose and unlogic syntax, the
> > cargo cult of vendoring libraries instead of a sane shared library
> > and versioning model (althought that should be largerly irrelevant
> > for the kernel), and compared to many other languages it seems
> > overall rather sane.

This is more or less my assessment from playing with rust as well.

> > But I'm really worried about the impact on the kernel, as interface
> > between languages are a real pain, and so far I'm not convinced at
> > all that this pain is worth the gain, even if that could change in
> > the future.
> 
> One point I'd like to explicitly address, as it seems to be very
> relevant to me when it comes to whether or not the pain is worth the
> gain, is how we'll deal with the fact that the rust compiler will
> tell a non-negligible [*] part of the kernel developers that they
> don't have the skills to write kernel code. Will we have a credible
> option to offer there to help people improve their skills set, and
> how will we deal with the fact that some people will be left on the
> side of the road ? Or would we acknowledge what may be the elephant
> in the room that this would actually be a good thing for the kernel
> code quality ? What about the impact on a community that is already
> overworked and prone to burn-out ?
> 
> [*] I have no way to quantify this at the moment, maybe I'm overly
> pessimistic, and the number will likely vary depending on areas,
> probably impacting BSPs differently than mainline.

I don't think that's a huge concern.  After all rust isn't going to
penetrate every subsystem all at once ... and realistically it will
only penetrate some subsystems if there's the ability to review it.

I really doubt anyone at the maintainer or reviewer level of the kernel
doesn't have the ability to learn rust (now whether they have the
desire to is quite another matter) and I'd be surprised if we can find
any kernel developer who only speaks C.  I think the biggest problem
with rust is that there's definitely an anti-C bias in the language. 
What I mean by that is if you look at Java they chose to be as close to
C where possible to make the language easier to learn.  Rust seems to
have taken the opposite view and picked a lot of things which could
have been C like (typing, function calls, returns, mutability, etc.)
and done them differently just because they wanted to be different from
C.  That does cause issues for C people because you tend to trip over
the anti-C biases in the language and are constantly having to look the
basic syntax up.  This really increases the learning curve going from C
to rust, which gets annoying very fast, but it's not insurmountable.

All that said, I expect once we start getting rust based patches, it
will quickly become apparent where the rust receptive subsystems are
and I would expect reviewers in those subsystems will quickly become
rust fluent.  This doesn't answer the pain vs gain aspect, but I
suspect we won't know that until we actually try.

James

[Ksummit-discuss] [TECH TOPIC] Rust

[James Bottomley]

On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > All previous topics but this one are technical. To restore the
> > > balance a bit (and also because I believe it's important to the
> > > success of this project :-)), I'd like to also discuss the impact
> > > on the kernel maintenance at large, beyond just the maintainers
> > > who opt in to be early adopters of rust in their subsystem.
> > 
> > Yes.  That is I think the most important point.
> > 
> > I've played around with rust a bit for userspace project, and there
> > is things I like a lot like strict type safety and the ownership
> > model, and things I detest like verbose and unlogic syntax, the
> > cargo cult of vendoring libraries instead of a sane shared library
> > and versioning model (althought that should be largerly irrelevant
> > for the kernel), and compared to many other languages it seems
> > overall rather sane.

This is more or less my assessment from playing with rust as well.

> > But I'm really worried about the impact on the kernel, as interface
> > between languages are a real pain, and so far I'm not convinced at
> > all that this pain is worth the gain, even if that could change in
> > the future.
> 
> One point I'd like to explicitly address, as it seems to be very
> relevant to me when it comes to whether or not the pain is worth the
> gain, is how we'll deal with the fact that the rust compiler will
> tell a non-negligible [*] part of the kernel developers that they
> don't have the skills to write kernel code. Will we have a credible
> option to offer there to help people improve their skills set, and
> how will we deal with the fact that some people will be left on the
> side of the road ? Or would we acknowledge what may be the elephant
> in the room that this would actually be a good thing for the kernel
> code quality ? What about the impact on a community that is already
> overworked and prone to burn-out ?
> 
> [*] I have no way to quantify this at the moment, maybe I'm overly
> pessimistic, and the number will likely vary depending on areas,
> probably impacting BSPs differently than mainline.

I don't think that's a huge concern.  After all rust isn't going to
penetrate every subsystem all at once ... and realistically it will
only penetrate some subsystems if there's the ability to review it.

I really doubt anyone at the maintainer or reviewer level of the kernel
doesn't have the ability to learn rust (now whether they have the
desire to is quite another matter) and I'd be surprised if we can find
any kernel developer who only speaks C.  I think the biggest problem
with rust is that there's definitely an anti-C bias in the language. 
What I mean by that is if you look at Java they chose to be as close to
C where possible to make the language easier to learn.  Rust seems to
have taken the opposite view and picked a lot of things which could
have been C like (typing, function calls, returns, mutability, etc.)
and done them differently just because they wanted to be different from
C.  That does cause issues for C people because you tend to trip over
the anti-C biases in the language and are constantly having to look the
basic syntax up.  This really increases the learning curve going from C
to rust, which gets annoying very fast, but it's not insurmountable.

All that said, I expect once we start getting rust based patches, it
will quickly become apparent where the rust receptive subsystems are
and I would expect reviewers in those subsystems will quickly become
rust fluent.  This doesn't answer the pain vs gain aspect, but I
suspect we won't know that until we actually try.

James

Re: [TECH TOPIC] Rust

[Jens Axboe]

On 6/19/22 6:56 AM, James Bottomley wrote:
>> One point I'd like to explicitly address, as it seems to be very
>> relevant to me when it comes to whether or not the pain is worth the
>> gain, is how we'll deal with the fact that the rust compiler will
>> tell a non-negligible [*] part of the kernel developers that they
>> don't have the skills to write kernel code. Will we have a credible
>> option to offer there to help people improve their skills set, and
>> how will we deal with the fact that some people will be left on the
>> side of the road ? Or would we acknowledge what may be the elephant
>> in the room that this would actually be a good thing for the kernel
>> code quality ? What about the impact on a community that is already
>> overworked and prone to burn-out ?
>>
>> [*] I have no way to quantify this at the moment, maybe I'm overly
>> pessimistic, and the number will likely vary depending on areas,
>> probably impacting BSPs differently than mainline.
> 
> I don't think that's a huge concern.  After all rust isn't going to
> penetrate every subsystem all at once ... and realistically it will
> only penetrate some subsystems if there's the ability to review it.
> 
> I really doubt anyone at the maintainer or reviewer level of the kernel
> doesn't have the ability to learn rust (now whether they have the
> desire to is quite another matter) and I'd be surprised if we can find
> any kernel developer who only speaks C.  I think the biggest problem
> with rust is that there's definitely an anti-C bias in the language. 
> What I mean by that is if you look at Java they chose to be as close to
> C where possible to make the language easier to learn.  Rust seems to
> have taken the opposite view and picked a lot of things which could
> have been C like (typing, function calls, returns, mutability, etc.)
> and done them differently just because they wanted to be different from
> C.  That does cause issues for C people because you tend to trip over
> the anti-C biases in the language and are constantly having to look the
> basic syntax up.  This really increases the learning curve going from C
> to rust, which gets annoying very fast, but it's not insurmountable.
> 
> All that said, I expect once we start getting rust based patches, it
> will quickly become apparent where the rust receptive subsystems are
> and I would expect reviewers in those subsystems will quickly become
> rust fluent.  This doesn't answer the pain vs gain aspect, but I
> suspect we won't know that until we actually try.

FWIW, I agree with you here, James. And I have to say that the original
sentiment expressed here by Laurent is weirdly elitist, and not
something that seems positive if that's the case. I'd suggest
re-calibrating that mindset...

Some subsystem maintainers are pretty much spread as thinly as they can
be in terms of workloads, while other subsystems are less active and
supposedly have more spare cycles on their hands. Some handle multiple
subsystems, some subsystems are tiny, some are huge. All of this will
impact both the motivation abd ability to adopt rust.

-- 
Jens Axboe

[Ksummit-discuss] [TECH TOPIC] Rust

[Jens Axboe]

On 6/19/22 6:56 AM, James Bottomley wrote:
>> One point I'd like to explicitly address, as it seems to be very
>> relevant to me when it comes to whether or not the pain is worth the
>> gain, is how we'll deal with the fact that the rust compiler will
>> tell a non-negligible [*] part of the kernel developers that they
>> don't have the skills to write kernel code. Will we have a credible
>> option to offer there to help people improve their skills set, and
>> how will we deal with the fact that some people will be left on the
>> side of the road ? Or would we acknowledge what may be the elephant
>> in the room that this would actually be a good thing for the kernel
>> code quality ? What about the impact on a community that is already
>> overworked and prone to burn-out ?
>>
>> [*] I have no way to quantify this at the moment, maybe I'm overly
>> pessimistic, and the number will likely vary depending on areas,
>> probably impacting BSPs differently than mainline.
> 
> I don't think that's a huge concern.  After all rust isn't going to
> penetrate every subsystem all at once ... and realistically it will
> only penetrate some subsystems if there's the ability to review it.
> 
> I really doubt anyone at the maintainer or reviewer level of the kernel
> doesn't have the ability to learn rust (now whether they have the
> desire to is quite another matter) and I'd be surprised if we can find
> any kernel developer who only speaks C.  I think the biggest problem
> with rust is that there's definitely an anti-C bias in the language. 
> What I mean by that is if you look at Java they chose to be as close to
> C where possible to make the language easier to learn.  Rust seems to
> have taken the opposite view and picked a lot of things which could
> have been C like (typing, function calls, returns, mutability, etc.)
> and done them differently just because they wanted to be different from
> C.  That does cause issues for C people because you tend to trip over
> the anti-C biases in the language and are constantly having to look the
> basic syntax up.  This really increases the learning curve going from C
> to rust, which gets annoying very fast, but it's not insurmountable.
> 
> All that said, I expect once we start getting rust based patches, it
> will quickly become apparent where the rust receptive subsystems are
> and I would expect reviewers in those subsystems will quickly become
> rust fluent.  This doesn't answer the pain vs gain aspect, but I
> suspect we won't know that until we actually try.

FWIW, I agree with you here, James. And I have to say that the original
sentiment expressed here by Laurent is weirdly elitist, and not
something that seems positive if that's the case. I'd suggest
re-calibrating that mindset...

Some subsystem maintainers are pretty much spread as thinly as they can
be in terms of workloads, while other subsystems are less active and
supposedly have more spare cycles on their hands. Some handle multiple
subsystems, some subsystems are tiny, some are huge. All of this will
impact both the motivation abd ability to adopt rust.

-- 
Jens Axboe

Re: [TECH TOPIC] Rust

[Laurent Pinchart]

On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
> On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> > On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > > All previous topics but this one are technical. To restore the
> > > > balance a bit (and also because I believe it's important to the
> > > > success of this project :-)), I'd like to also discuss the impact
> > > > on the kernel maintenance at large, beyond just the maintainers
> > > > who opt in to be early adopters of rust in their subsystem.
> > > 
> > > Yes.  That is I think the most important point.
> > > 
> > > I've played around with rust a bit for userspace project, and there
> > > is things I like a lot like strict type safety and the ownership
> > > model, and things I detest like verbose and unlogic syntax, the
> > > cargo cult of vendoring libraries instead of a sane shared library
> > > and versioning model (althought that should be largerly irrelevant
> > > for the kernel), and compared to many other languages it seems
> > > overall rather sane.
> 
> This is more or less my assessment from playing with rust as well.
> 
> > > But I'm really worried about the impact on the kernel, as interface
> > > between languages are a real pain, and so far I'm not convinced at
> > > all that this pain is worth the gain, even if that could change in
> > > the future.
> > 
> > One point I'd like to explicitly address, as it seems to be very
> > relevant to me when it comes to whether or not the pain is worth the
> > gain, is how we'll deal with the fact that the rust compiler will
> > tell a non-negligible [*] part of the kernel developers that they
> > don't have the skills to write kernel code. Will we have a credible
> > option to offer there to help people improve their skills set, and
> > how will we deal with the fact that some people will be left on the
> > side of the road ? Or would we acknowledge what may be the elephant
> > in the room that this would actually be a good thing for the kernel
> > code quality ? What about the impact on a community that is already
> > overworked and prone to burn-out ?
> > 
> > [*] I have no way to quantify this at the moment, maybe I'm overly
> > pessimistic, and the number will likely vary depending on areas,
> > probably impacting BSPs differently than mainline.
> 
> I don't think that's a huge concern.  After all rust isn't going to
> penetrate every subsystem all at once ... and realistically it will
> only penetrate some subsystems if there's the ability to review it.
> 
> I really doubt anyone at the maintainer or reviewer level of the kernel
> doesn't have the ability to learn rust (now whether they have the
> desire to is quite another matter) and I'd be surprised if we can find
> any kernel developer who only speaks C.  I think the biggest problem
> with rust is that there's definitely an anti-C bias in the language. 
> What I mean by that is if you look at Java they chose to be as close to
> C where possible to make the language easier to learn.  Rust seems to
> have taken the opposite view and picked a lot of things which could
> have been C like (typing, function calls, returns, mutability, etc.)
> and done them differently just because they wanted to be different from
> C.  That does cause issues for C people because you tend to trip over
> the anti-C biases in the language and are constantly having to look the
> basic syntax up.  This really increases the learning curve going from C
> to rust, which gets annoying very fast, but it's not insurmountable.

Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
understand the syntax is different from C because it was inspired by
other languages, not out of a desire to be anti-C. I could be wrong
though, and it doesn't really matter anyway, I doubt there's anything we
could do there.

> All that said, I expect once we start getting rust based patches, it
> will quickly become apparent where the rust receptive subsystems are
> and I would expect reviewers in those subsystems will quickly become
> rust fluent.  This doesn't answer the pain vs gain aspect, but I
> suspect we won't know that until we actually try.

I'm not too concerned about maintainers and reviewers when it comes to
learning rust (beside possibly the fact that it will be yet another task
on the plate of a scarce resource, but the real answer to that is likely
to address the maintainers and reviewers shortage). As you mentioned, it
will be opt-in to start with. My concern is more about the longer term,
when rust will become a requirement to contribute to some subsystems.

Writing rust code that compiles (without wrapping everything in unsafe)
requires understanding of life time management of objects. This is also
required to write safe C code, but as can be seen through the kernel
it's more often that not completely ignored, especially in drivers (just
look at how many drivers that expose a chardev to userspace are happy
using devm_kzalloc() to allocate their private data structure that is
accessible through userspace calls). It may be caused by a combination
of both the compiler an the kernel allowing bad practices (the fact that
devm_kzalloc() exists without a huge flashing red warning around it
shows that we either don't understand the problem or don't care), and
developers not willing to invest the time required to do things right
(which is understandable, given the amount of work involved, in one case
I know about too well it would require rewriting the whole core of a
subsystem). Or it could be that the technical problem is complex, and
will require a big amount of time and skill to fix properly.

This being said, if the subsystem I mentioned were to accept rust
drivers, it would need to be rewritten first, which would be a really
big gain. I however have no words to describe the pain.

There's also the question of how to handle tree-wide changes, or
subsystem core changes that end up having to touch rust code in drivers.
That I think will affect the transition period, and we need a good
strategy.

-- 
Regards,

Laurent Pinchart

[Ksummit-discuss] [TECH TOPIC] Rust

[Laurent Pinchart]

On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
> On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> > On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > > All previous topics but this one are technical. To restore the
> > > > balance a bit (and also because I believe it's important to the
> > > > success of this project :-)), I'd like to also discuss the impact
> > > > on the kernel maintenance at large, beyond just the maintainers
> > > > who opt in to be early adopters of rust in their subsystem.
> > > 
> > > Yes.  That is I think the most important point.
> > > 
> > > I've played around with rust a bit for userspace project, and there
> > > is things I like a lot like strict type safety and the ownership
> > > model, and things I detest like verbose and unlogic syntax, the
> > > cargo cult of vendoring libraries instead of a sane shared library
> > > and versioning model (althought that should be largerly irrelevant
> > > for the kernel), and compared to many other languages it seems
> > > overall rather sane.
> 
> This is more or less my assessment from playing with rust as well.
> 
> > > But I'm really worried about the impact on the kernel, as interface
> > > between languages are a real pain, and so far I'm not convinced at
> > > all that this pain is worth the gain, even if that could change in
> > > the future.
> > 
> > One point I'd like to explicitly address, as it seems to be very
> > relevant to me when it comes to whether or not the pain is worth the
> > gain, is how we'll deal with the fact that the rust compiler will
> > tell a non-negligible [*] part of the kernel developers that they
> > don't have the skills to write kernel code. Will we have a credible
> > option to offer there to help people improve their skills set, and
> > how will we deal with the fact that some people will be left on the
> > side of the road ? Or would we acknowledge what may be the elephant
> > in the room that this would actually be a good thing for the kernel
> > code quality ? What about the impact on a community that is already
> > overworked and prone to burn-out ?
> > 
> > [*] I have no way to quantify this at the moment, maybe I'm overly
> > pessimistic, and the number will likely vary depending on areas,
> > probably impacting BSPs differently than mainline.
> 
> I don't think that's a huge concern.  After all rust isn't going to
> penetrate every subsystem all at once ... and realistically it will
> only penetrate some subsystems if there's the ability to review it.
> 
> I really doubt anyone at the maintainer or reviewer level of the kernel
> doesn't have the ability to learn rust (now whether they have the
> desire to is quite another matter) and I'd be surprised if we can find
> any kernel developer who only speaks C.  I think the biggest problem
> with rust is that there's definitely an anti-C bias in the language. 
> What I mean by that is if you look at Java they chose to be as close to
> C where possible to make the language easier to learn.  Rust seems to
> have taken the opposite view and picked a lot of things which could
> have been C like (typing, function calls, returns, mutability, etc.)
> and done them differently just because they wanted to be different from
> C.  That does cause issues for C people because you tend to trip over
> the anti-C biases in the language and are constantly having to look the
> basic syntax up.  This really increases the learning curve going from C
> to rust, which gets annoying very fast, but it's not insurmountable.

Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
understand the syntax is different from C because it was inspired by
other languages, not out of a desire to be anti-C. I could be wrong
though, and it doesn't really matter anyway, I doubt there's anything we
could do there.

> All that said, I expect once we start getting rust based patches, it
> will quickly become apparent where the rust receptive subsystems are
> and I would expect reviewers in those subsystems will quickly become
> rust fluent.  This doesn't answer the pain vs gain aspect, but I
> suspect we won't know that until we actually try.

I'm not too concerned about maintainers and reviewers when it comes to
learning rust (beside possibly the fact that it will be yet another task
on the plate of a scarce resource, but the real answer to that is likely
to address the maintainers and reviewers shortage). As you mentioned, it
will be opt-in to start with. My concern is more about the longer term,
when rust will become a requirement to contribute to some subsystems.

Writing rust code that compiles (without wrapping everything in unsafe)
requires understanding of life time management of objects. This is also
required to write safe C code, but as can be seen through the kernel
it's more often that not completely ignored, especially in drivers (just
look at how many drivers that expose a chardev to userspace are happy
using devm_kzalloc() to allocate their private data structure that is
accessible through userspace calls). It may be caused by a combination
of both the compiler an the kernel allowing bad practices (the fact that
devm_kzalloc() exists without a huge flashing red warning around it
shows that we either don't understand the problem or don't care), and
developers not willing to invest the time required to do things right
(which is understandable, given the amount of work involved, in one case
I know about too well it would require rewriting the whole core of a
subsystem). Or it could be that the technical problem is complex, and
will require a big amount of time and skill to fix properly.

This being said, if the subsystem I mentioned were to accept rust
drivers, it would need to be rewritten first, which would be a really
big gain. I however have no words to describe the pain.

There's also the question of how to handle tree-wide changes, or
subsystem core changes that end up having to touch rust code in drivers.
That I think will affect the transition period, and we need a good
strategy.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Rust

[Laurent Pinchart]

On Sun, Jun 19, 2022 at 07:19:38AM -0600, Jens Axboe wrote:
> On 6/19/22 6:56 AM, James Bottomley wrote:
> >> One point I'd like to explicitly address, as it seems to be very
> >> relevant to me when it comes to whether or not the pain is worth the
> >> gain, is how we'll deal with the fact that the rust compiler will
> >> tell a non-negligible [*] part of the kernel developers that they
> >> don't have the skills to write kernel code. Will we have a credible
> >> option to offer there to help people improve their skills set, and
> >> how will we deal with the fact that some people will be left on the
> >> side of the road ? Or would we acknowledge what may be the elephant
> >> in the room that this would actually be a good thing for the kernel
> >> code quality ? What about the impact on a community that is already
> >> overworked and prone to burn-out ?
> >>
> >> [*] I have no way to quantify this at the moment, maybe I'm overly
> >> pessimistic, and the number will likely vary depending on areas,
> >> probably impacting BSPs differently than mainline.
> > 
> > I don't think that's a huge concern.  After all rust isn't going to
> > penetrate every subsystem all at once ... and realistically it will
> > only penetrate some subsystems if there's the ability to review it.
> > 
> > I really doubt anyone at the maintainer or reviewer level of the kernel
> > doesn't have the ability to learn rust (now whether they have the
> > desire to is quite another matter) and I'd be surprised if we can find
> > any kernel developer who only speaks C.  I think the biggest problem
> > with rust is that there's definitely an anti-C bias in the language. 
> > What I mean by that is if you look at Java they chose to be as close to
> > C where possible to make the language easier to learn.  Rust seems to
> > have taken the opposite view and picked a lot of things which could
> > have been C like (typing, function calls, returns, mutability, etc.)
> > and done them differently just because they wanted to be different from
> > C.  That does cause issues for C people because you tend to trip over
> > the anti-C biases in the language and are constantly having to look the
> > basic syntax up.  This really increases the learning curve going from C
> > to rust, which gets annoying very fast, but it's not insurmountable.
> > 
> > All that said, I expect once we start getting rust based patches, it
> > will quickly become apparent where the rust receptive subsystems are
> > and I would expect reviewers in those subsystems will quickly become
> > rust fluent.  This doesn't answer the pain vs gain aspect, but I
> > suspect we won't know that until we actually try.
> 
> FWIW, I agree with you here, James. And I have to say that the original
> sentiment expressed here by Laurent is weirdly elitist, and not
> something that seems positive if that's the case. I'd suggest
> re-calibrating that mindset...

I'm not sure about the elitist part, but it's certainly not a positive
statement. I'm concerned about the pain that rust in the kernel would
inflict. Whether it's worth the gain or not is not for me to decide, but
I'm certainly concerned that it could lead to a catastrophe if we don't
carefully think about the issues, acknowledge them, and find ways to
solve them. I don't think winging it is a real option here, but I'd be
more than happy to be shown that my concerns are not founded :-)

> Some subsystem maintainers are pretty much spread as thinly as they can
> be in terms of workloads, while other subsystems are less active and
> supposedly have more spare cycles on their hands. Some handle multiple
> subsystems, some subsystems are tiny, some are huge. All of this will
> impact both the motivation abd ability to adopt rust.

-- 
Regards,

Laurent Pinchart

[Ksummit-discuss] [TECH TOPIC] Rust

[Laurent Pinchart]

On Sun, Jun 19, 2022 at 07:19:38AM -0600, Jens Axboe wrote:
> On 6/19/22 6:56 AM, James Bottomley wrote:
> >> One point I'd like to explicitly address, as it seems to be very
> >> relevant to me when it comes to whether or not the pain is worth the
> >> gain, is how we'll deal with the fact that the rust compiler will
> >> tell a non-negligible [*] part of the kernel developers that they
> >> don't have the skills to write kernel code. Will we have a credible
> >> option to offer there to help people improve their skills set, and
> >> how will we deal with the fact that some people will be left on the
> >> side of the road ? Or would we acknowledge what may be the elephant
> >> in the room that this would actually be a good thing for the kernel
> >> code quality ? What about the impact on a community that is already
> >> overworked and prone to burn-out ?
> >>
> >> [*] I have no way to quantify this at the moment, maybe I'm overly
> >> pessimistic, and the number will likely vary depending on areas,
> >> probably impacting BSPs differently than mainline.
> > 
> > I don't think that's a huge concern.  After all rust isn't going to
> > penetrate every subsystem all at once ... and realistically it will
> > only penetrate some subsystems if there's the ability to review it.
> > 
> > I really doubt anyone at the maintainer or reviewer level of the kernel
> > doesn't have the ability to learn rust (now whether they have the
> > desire to is quite another matter) and I'd be surprised if we can find
> > any kernel developer who only speaks C.  I think the biggest problem
> > with rust is that there's definitely an anti-C bias in the language. 
> > What I mean by that is if you look at Java they chose to be as close to
> > C where possible to make the language easier to learn.  Rust seems to
> > have taken the opposite view and picked a lot of things which could
> > have been C like (typing, function calls, returns, mutability, etc.)
> > and done them differently just because they wanted to be different from
> > C.  That does cause issues for C people because you tend to trip over
> > the anti-C biases in the language and are constantly having to look the
> > basic syntax up.  This really increases the learning curve going from C
> > to rust, which gets annoying very fast, but it's not insurmountable.
> > 
> > All that said, I expect once we start getting rust based patches, it
> > will quickly become apparent where the rust receptive subsystems are
> > and I would expect reviewers in those subsystems will quickly become
> > rust fluent.  This doesn't answer the pain vs gain aspect, but I
> > suspect we won't know that until we actually try.
> 
> FWIW, I agree with you here, James. And I have to say that the original
> sentiment expressed here by Laurent is weirdly elitist, and not
> something that seems positive if that's the case. I'd suggest
> re-calibrating that mindset...

I'm not sure about the elitist part, but it's certainly not a positive
statement. I'm concerned about the pain that rust in the kernel would
inflict. Whether it's worth the gain or not is not for me to decide, but
I'm certainly concerned that it could lead to a catastrophe if we don't
carefully think about the issues, acknowledge them, and find ways to
solve them. I don't think winging it is a real option here, but I'd be
more than happy to be shown that my concerns are not founded :-)

> Some subsystem maintainers are pretty much spread as thinly as they can
> be in terms of workloads, while other subsystems are less active and
> supposedly have more spare cycles on their hands. Some handle multiple
> subsystems, some subsystems are tiny, some are huge. All of this will
> impact both the motivation abd ability to adopt rust.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Rust

[James Bottomley]

On Sun, 2022-06-19 at 16:53 +0300, Laurent Pinchart wrote:
> On Sun, Jun 19, 2022 at 07:19:38AM -0600, Jens Axboe wrote:
> > On 6/19/22 6:56 AM, James Bottomley wrote:
[...]
> > > All that said, I expect once we start getting rust based patches,
> > > it will quickly become apparent where the rust receptive
> > > subsystems are and I would expect reviewers in those subsystems
> > > will quickly become rust fluent.  This doesn't answer the pain vs
> > > gain aspect, but I suspect we won't know that until we actually
> > > try.
> > 
> > FWIW, I agree with you here, James. And I have to say that the
> > original sentiment expressed here by Laurent is weirdly elitist,
> > and not something that seems positive if that's the case. I'd
> > suggest re-calibrating that mindset...
> 
> I'm not sure about the elitist part, but it's certainly not a
> positive statement. I'm concerned about the pain that rust in the
> kernel would inflict.

I get that, and I think it's a serious concern.  I just also think we
would waste more time second guessing this issue that we would consume
if we just tried it.  It's not like allowing Rust into the kernel is
some sort of Rubicon that once crossed we can never go back.  I think
we won't know the pain vs gain until at least a year into the
experiment of accepting Rust.  Even at that late stage, if everyone
agrees it caused more problems than it solves, I bet it could all be
quickly excised.

So the bottom line is I think while we have a community of volunteers
willing to try it we let them on the understanding the results will be
evaluated at some point.

>  Whether it's worth the gain or not is not for me to decide,
> but I'm certainly concerned that it could lead to a catastrophe if we
> don't carefully think about the issues, acknowledge them, and find
> ways to solve them. I don't think winging it is a real option here,
> but I'd be more than happy to be shown that my concerns are not
> founded :-)

Have more faith in the community and open source process. We've screwed
up many times (devfs anyone ...) learned from the mistake and fixed it.
I'm happy to bet that accepting Rust will be no different from all the
other screwups we allowed in and later fixed.  So I don't think there
will be a catastrophe.  Either the rust experiment works or it will
become fairly quickly apparent if it doesn't and it will get removed. 
The worst case, I suppose, is that the benefit is marginal in which
case there's no consensus on removal vs continuation and we simply
continue through inertia.  I bet in that situation Rust penetration
will be fairly minimal and confined to enthusiastic subsystems with the
rest taking steps to isolate themselves from it.  What we'd need in
this case is some opinionated person running the tree and able to make
the call for us ... now who could that be?

James

[Ksummit-discuss] [TECH TOPIC] Rust

[James Bottomley]

On Sun, 2022-06-19 at 16:53 +0300, Laurent Pinchart wrote:
> On Sun, Jun 19, 2022 at 07:19:38AM -0600, Jens Axboe wrote:
> > On 6/19/22 6:56 AM, James Bottomley wrote:
[...]
> > > All that said, I expect once we start getting rust based patches,
> > > it will quickly become apparent where the rust receptive
> > > subsystems are and I would expect reviewers in those subsystems
> > > will quickly become rust fluent.  This doesn't answer the pain vs
> > > gain aspect, but I suspect we won't know that until we actually
> > > try.
> > 
> > FWIW, I agree with you here, James. And I have to say that the
> > original sentiment expressed here by Laurent is weirdly elitist,
> > and not something that seems positive if that's the case. I'd
> > suggest re-calibrating that mindset...
> 
> I'm not sure about the elitist part, but it's certainly not a
> positive statement. I'm concerned about the pain that rust in the
> kernel would inflict.

I get that, and I think it's a serious concern.  I just also think we
would waste more time second guessing this issue that we would consume
if we just tried it.  It's not like allowing Rust into the kernel is
some sort of Rubicon that once crossed we can never go back.  I think
we won't know the pain vs gain until at least a year into the
experiment of accepting Rust.  Even at that late stage, if everyone
agrees it caused more problems than it solves, I bet it could all be
quickly excised.

So the bottom line is I think while we have a community of volunteers
willing to try it we let them on the understanding the results will be
evaluated at some point.

>  Whether it's worth the gain or not is not for me to decide,
> but I'm certainly concerned that it could lead to a catastrophe if we
> don't carefully think about the issues, acknowledge them, and find
> ways to solve them. I don't think winging it is a real option here,
> but I'd be more than happy to be shown that my concerns are not
> founded :-)

Have more faith in the community and open source process. We've screwed
up many times (devfs anyone ...) learned from the mistake and fixed it.
I'm happy to bet that accepting Rust will be no different from all the
other screwups we allowed in and later fixed.  So I don't think there
will be a catastrophe.  Either the rust experiment works or it will
become fairly quickly apparent if it doesn't and it will get removed. 
The worst case, I suppose, is that the benefit is marginal in which
case there's no consensus on removal vs continuation and we simply
continue through inertia.  I bet in that situation Rust penetration
will be fairly minimal and confined to enthusiastic subsystems with the
rest taking steps to isolate themselves from it.  What we'd need in
this case is some opinionated person running the tree and able to make
the call for us ... now who could that be?

James

[MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[James Bottomley]

On Sun, 2022-06-19 at 10:27 -0400, James Bottomley wrote:
> On Sun, 2022-06-19 at 16:53 +0300, Laurent Pinchart wrote:
[...]
> >  Whether it's worth the gain or not is not for me to decide,
> > but I'm certainly concerned that it could lead to a catastrophe if
> > we don't carefully think about the issues, acknowledge them, and
> > find ways to solve them. I don't think winging it is a real option
> > here, but I'd be more than happy to be shown that my concerns are
> > not founded :-)
> 
> Have more faith in the community and open source process. We've
> screwed up many times (devfs anyone ...) learned from the mistake and
> fixed it. I'm happy to bet that accepting Rust will be no different
> from all the other screwups we allowed in and later fixed.  So I
> don't think there will be a catastrophe.  Either the rust experiment
> works or it will become fairly quickly apparent if it doesn't and it
> will get removed.  The worst case, I suppose, is that the benefit is
> marginal in which case there's no consensus on removal vs
> continuation and we simply continue through inertia.  I bet in that
> situation Rust penetration will be fairly minimal and confined to
> enthusiastic subsystems with the rest taking steps to isolate
> themselves from it.  What we'd need in this case is some opinionated
> person running the tree and able to make the call for us ... now who
> could that be?

I think there's a growing problem in Linux which is exemplified by this
Rust debate but which goes way beyond it: We're becoming too fearful of
making big decisions to sustain innovation in some areas.  This really
is a creeping cancer of inertia that has destroyed many projects before
us and if we're not careful, we'll go the same way.

The biggest area where we currently squelch innovation is around
anything that touches the user space ABI.  Allegations of having to get
everything right ab initio because we have to support it "forever" and
all the subsequent wittering and second guessing are really stifling
innovation in pretty much anything that could be exposed to user space.
I really think, to counter this, we need a crash course reminder of all
of our mistakes and how we climbed out of the hole they dug us into,
because without that we're becoming too fearful of making mistakes.

The object is not to avoid mistakes at any cost, it's to be confident
that if you make them, you're good enough to find a pathway out of them
again.

James

[Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[James Bottomley]

On Sun, 2022-06-19 at 10:27 -0400, James Bottomley wrote:
> On Sun, 2022-06-19 at 16:53 +0300, Laurent Pinchart wrote:
[...]
> >  Whether it's worth the gain or not is not for me to decide,
> > but I'm certainly concerned that it could lead to a catastrophe if
> > we don't carefully think about the issues, acknowledge them, and
> > find ways to solve them. I don't think winging it is a real option
> > here, but I'd be more than happy to be shown that my concerns are
> > not founded :-)
> 
> Have more faith in the community and open source process. We've
> screwed up many times (devfs anyone ...) learned from the mistake and
> fixed it. I'm happy to bet that accepting Rust will be no different
> from all the other screwups we allowed in and later fixed.  So I
> don't think there will be a catastrophe.  Either the rust experiment
> works or it will become fairly quickly apparent if it doesn't and it
> will get removed.  The worst case, I suppose, is that the benefit is
> marginal in which case there's no consensus on removal vs
> continuation and we simply continue through inertia.  I bet in that
> situation Rust penetration will be fairly minimal and confined to
> enthusiastic subsystems with the rest taking steps to isolate
> themselves from it.  What we'd need in this case is some opinionated
> person running the tree and able to make the call for us ... now who
> could that be?

I think there's a growing problem in Linux which is exemplified by this
Rust debate but which goes way beyond it: We're becoming too fearful of
making big decisions to sustain innovation in some areas.  This really
is a creeping cancer of inertia that has destroyed many projects before
us and if we're not careful, we'll go the same way.

The biggest area where we currently squelch innovation is around
anything that touches the user space ABI.  Allegations of having to get
everything right ab initio because we have to support it "forever" and
all the subsequent wittering and second guessing are really stifling
innovation in pretty much anything that could be exposed to user space.
I really think, to counter this, we need a crash course reminder of all
of our mistakes and how we climbed out of the hole they dug us into,
because without that we're becoming too fearful of making mistakes.

The object is not to avoid mistakes at any cost, it's to be confident
that if you make them, you're good enough to find a pathway out of them
again.

James

Re: [TECH TOPIC] Rust

[Geert Uytterhoeven]

Hi Laurent,

On Sun, Jun 19, 2022 at 3:49 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
> > On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> > > On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > > > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > > > All previous topics but this one are technical. To restore the
> > > > > balance a bit (and also because I believe it's important to the
> > > > > success of this project :-)), I'd like to also discuss the impact
> > > > > on the kernel maintenance at large, beyond just the maintainers
> > > > > who opt in to be early adopters of rust in their subsystem.
> > > >
> > > > Yes.  That is I think the most important point.
> > > >
> > > > I've played around with rust a bit for userspace project, and there
> > > > is things I like a lot like strict type safety and the ownership
> > > > model, and things I detest like verbose and unlogic syntax, the
> > > > cargo cult of vendoring libraries instead of a sane shared library
> > > > and versioning model (althought that should be largerly irrelevant
> > > > for the kernel), and compared to many other languages it seems
> > > > overall rather sane.
> >
> > This is more or less my assessment from playing with rust as well.
> >
> > > > But I'm really worried about the impact on the kernel, as interface
> > > > between languages are a real pain, and so far I'm not convinced at
> > > > all that this pain is worth the gain, even if that could change in
> > > > the future.
> > >
> > > One point I'd like to explicitly address, as it seems to be very
> > > relevant to me when it comes to whether or not the pain is worth the
> > > gain, is how we'll deal with the fact that the rust compiler will
> > > tell a non-negligible [*] part of the kernel developers that they
> > > don't have the skills to write kernel code. Will we have a credible
> > > option to offer there to help people improve their skills set, and
> > > how will we deal with the fact that some people will be left on the
> > > side of the road ? Or would we acknowledge what may be the elephant
> > > in the room that this would actually be a good thing for the kernel
> > > code quality ? What about the impact on a community that is already
> > > overworked and prone to burn-out ?
> > >
> > > [*] I have no way to quantify this at the moment, maybe I'm overly
> > > pessimistic, and the number will likely vary depending on areas,
> > > probably impacting BSPs differently than mainline.
> >
> > I don't think that's a huge concern.  After all rust isn't going to
> > penetrate every subsystem all at once ... and realistically it will
> > only penetrate some subsystems if there's the ability to review it.
> >
> > I really doubt anyone at the maintainer or reviewer level of the kernel
> > doesn't have the ability to learn rust (now whether they have the
> > desire to is quite another matter) and I'd be surprised if we can find
> > any kernel developer who only speaks C.  I think the biggest problem
> > with rust is that there's definitely an anti-C bias in the language.
> > What I mean by that is if you look at Java they chose to be as close to
> > C where possible to make the language easier to learn.  Rust seems to
> > have taken the opposite view and picked a lot of things which could
> > have been C like (typing, function calls, returns, mutability, etc.)
> > and done them differently just because they wanted to be different from
> > C.  That does cause issues for C people because you tend to trip over
> > the anti-C biases in the language and are constantly having to look the
> > basic syntax up.  This really increases the learning curve going from C
> > to rust, which gets annoying very fast, but it's not insurmountable.
>
> Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
> understand the syntax is different from C because it was inspired by
> other languages, not out of a desire to be anti-C. I could be wrong
> though, and it doesn't really matter anyway, I doubt there's anything we
> could do there.

When reading "things which could have been C like", I had the exact
same thought ;-)

In the past, many new paradigms were implemented in a variety of new
languages, usually inspired by other languages (e.g. some looked like
C, others like Pascal).

At the risk of sounding too procrastinating (perfect is the enemy
of good, and see James' thread about becoming too fearful...),
perhaps it is wise to question if Rust arrived too soon, and a more
C-like language implementing the same safety paradigms is just around
the corner, or already in its infancy?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[Ksummit-discuss] [TECH TOPIC] Rust

[Geert Uytterhoeven]

Hi Laurent,

On Sun, Jun 19, 2022 at 3:49 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
> > On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> > > On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > > > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > > > All previous topics but this one are technical. To restore the
> > > > > balance a bit (and also because I believe it's important to the
> > > > > success of this project :-)), I'd like to also discuss the impact
> > > > > on the kernel maintenance at large, beyond just the maintainers
> > > > > who opt in to be early adopters of rust in their subsystem.
> > > >
> > > > Yes.  That is I think the most important point.
> > > >
> > > > I've played around with rust a bit for userspace project, and there
> > > > is things I like a lot like strict type safety and the ownership
> > > > model, and things I detest like verbose and unlogic syntax, the
> > > > cargo cult of vendoring libraries instead of a sane shared library
> > > > and versioning model (althought that should be largerly irrelevant
> > > > for the kernel), and compared to many other languages it seems
> > > > overall rather sane.
> >
> > This is more or less my assessment from playing with rust as well.
> >
> > > > But I'm really worried about the impact on the kernel, as interface
> > > > between languages are a real pain, and so far I'm not convinced at
> > > > all that this pain is worth the gain, even if that could change in
> > > > the future.
> > >
> > > One point I'd like to explicitly address, as it seems to be very
> > > relevant to me when it comes to whether or not the pain is worth the
> > > gain, is how we'll deal with the fact that the rust compiler will
> > > tell a non-negligible [*] part of the kernel developers that they
> > > don't have the skills to write kernel code. Will we have a credible
> > > option to offer there to help people improve their skills set, and
> > > how will we deal with the fact that some people will be left on the
> > > side of the road ? Or would we acknowledge what may be the elephant
> > > in the room that this would actually be a good thing for the kernel
> > > code quality ? What about the impact on a community that is already
> > > overworked and prone to burn-out ?
> > >
> > > [*] I have no way to quantify this at the moment, maybe I'm overly
> > > pessimistic, and the number will likely vary depending on areas,
> > > probably impacting BSPs differently than mainline.
> >
> > I don't think that's a huge concern.  After all rust isn't going to
> > penetrate every subsystem all at once ... and realistically it will
> > only penetrate some subsystems if there's the ability to review it.
> >
> > I really doubt anyone at the maintainer or reviewer level of the kernel
> > doesn't have the ability to learn rust (now whether they have the
> > desire to is quite another matter) and I'd be surprised if we can find
> > any kernel developer who only speaks C.  I think the biggest problem
> > with rust is that there's definitely an anti-C bias in the language.
> > What I mean by that is if you look at Java they chose to be as close to
> > C where possible to make the language easier to learn.  Rust seems to
> > have taken the opposite view and picked a lot of things which could
> > have been C like (typing, function calls, returns, mutability, etc.)
> > and done them differently just because they wanted to be different from
> > C.  That does cause issues for C people because you tend to trip over
> > the anti-C biases in the language and are constantly having to look the
> > basic syntax up.  This really increases the learning curve going from C
> > to rust, which gets annoying very fast, but it's not insurmountable.
>
> Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
> understand the syntax is different from C because it was inspired by
> other languages, not out of a desire to be anti-C. I could be wrong
> though, and it doesn't really matter anyway, I doubt there's anything we
> could do there.

When reading "things which could have been C like", I had the exact
same thought ;-)

In the past, many new paradigms were implemented in a variety of new
languages, usually inspired by other languages (e.g. some looked like
C, others like Pascal).

At the risk of sounding too procrastinating (perfect is the enemy
of good, and see James' thread about becoming too fearful...),
perhaps it is wise to question if Rust arrived too soon, and a more
C-like language implementing the same safety paradigms is just around
the corner, or already in its infancy?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[Ksummit-discuss] [TECH TOPIC] Rust

[Matthew Wilcox]

I think there are a lot of languages which promise this. The question is
whether any of them will get the critical mass to ensure that we have a
steady supply of programmers who know the language. Imagine if we'd decided
to use Haskell, Erlang or Ada (languages I was taught in university which
no longer have significant programming communities).

Rust seems to have the momentum. More so than Go or any other new language
in recent years.

On Sun., Jun. 19, 2022, 15:08 Geert Uytterhoeven, <geert@linux-m68k.org>
wrote:

> Hi Laurent,
>
> On Sun, Jun 19, 2022 at 3:49 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> > On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
> > > On Sun, 2022-06-19 at 13:04 +0300, Laurent Pinchart wrote:
> > > > On Sat, Jun 18, 2022 at 11:13:59PM -0700, Christoph Hellwig wrote:
> > > > > On Sat, Jun 18, 2022 at 11:42:07PM +0300, Laurent Pinchart wrote:
> > > > > > All previous topics but this one are technical. To restore the
> > > > > > balance a bit (and also because I believe it's important to the
> > > > > > success of this project :-)), I'd like to also discuss the impact
> > > > > > on the kernel maintenance at large, beyond just the maintainers
> > > > > > who opt in to be early adopters of rust in their subsystem.
> > > > >
> > > > > Yes.  That is I think the most important point.
> > > > >
> > > > > I've played around with rust a bit for userspace project, and there
> > > > > is things I like a lot like strict type safety and the ownership
> > > > > model, and things I detest like verbose and unlogic syntax, the
> > > > > cargo cult of vendoring libraries instead of a sane shared library
> > > > > and versioning model (althought that should be largerly irrelevant
> > > > > for the kernel), and compared to many other languages it seems
> > > > > overall rather sane.
> > >
> > > This is more or less my assessment from playing with rust as well.
> > >
> > > > > But I'm really worried about the impact on the kernel, as interface
> > > > > between languages are a real pain, and so far I'm not convinced at
> > > > > all that this pain is worth the gain, even if that could change in
> > > > > the future.
> > > >
> > > > One point I'd like to explicitly address, as it seems to be very
> > > > relevant to me when it comes to whether or not the pain is worth the
> > > > gain, is how we'll deal with the fact that the rust compiler will
> > > > tell a non-negligible [*] part of the kernel developers that they
> > > > don't have the skills to write kernel code. Will we have a credible
> > > > option to offer there to help people improve their skills set, and
> > > > how will we deal with the fact that some people will be left on the
> > > > side of the road ? Or would we acknowledge what may be the elephant
> > > > in the room that this would actually be a good thing for the kernel
> > > > code quality ? What about the impact on a community that is already
> > > > overworked and prone to burn-out ?
> > > >
> > > > [*] I have no way to quantify this at the moment, maybe I'm overly
> > > > pessimistic, and the number will likely vary depending on areas,
> > > > probably impacting BSPs differently than mainline.
> > >
> > > I don't think that's a huge concern.  After all rust isn't going to
> > > penetrate every subsystem all at once ... and realistically it will
> > > only penetrate some subsystems if there's the ability to review it.
> > >
> > > I really doubt anyone at the maintainer or reviewer level of the kernel
> > > doesn't have the ability to learn rust (now whether they have the
> > > desire to is quite another matter) and I'd be surprised if we can find
> > > any kernel developer who only speaks C.  I think the biggest problem
> > > with rust is that there's definitely an anti-C bias in the language.
> > > What I mean by that is if you look at Java they chose to be as close to
> > > C where possible to make the language easier to learn.  Rust seems to
> > > have taken the opposite view and picked a lot of things which could
> > > have been C like (typing, function calls, returns, mutability, etc.)
> > > and done them differently just because they wanted to be different from
> > > C.  That does cause issues for C people because you tend to trip over
> > > the anti-C biases in the language and are constantly having to look the
> > > basic syntax up.  This really increases the learning curve going from C
> > > to rust, which gets annoying very fast, but it's not insurmountable.
> >
> > Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
> > understand the syntax is different from C because it was inspired by
> > other languages, not out of a desire to be anti-C. I could be wrong
> > though, and it doesn't really matter anyway, I doubt there's anything we
> > could do there.
>
> When reading "things which could have been C like", I had the exact
> same thought ;-)
>
> In the past, many new paradigms were implemented in a variety of new
> languages, usually inspired by other languages (e.g. some looked like
> C, others like Pascal).
>
> At the risk of sounding too procrastinating (perfect is the enemy
> of good, and see James' thread about becoming too fearful...),
> perhaps it is wise to question if Rust arrived too soon, and a more
> C-like language implementing the same safety paradigms is just around
> the corner, or already in its infancy?
>
> Gr{oetje,eeting}s,
>
>                         Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 --
> geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker.
> But
> when I'm talking to journalists I just say "programmer" or something like
> that.
>                                 -- Linus Torvalds
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/ksummit-discuss/attachments/20220619/82204f26/attachment-0001.html>

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Linus Walleij]

On Sun, Jun 19, 2022 at 9:08 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Sun, Jun 19, 2022 at 3:49 PM Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> > Time to invent a C-like version of rust ? :-) Jokes aside, as far as I
> > understand the syntax is different from C because it was inspired by
> > other languages, not out of a desire to be anti-C. I could be wrong
> > though, and it doesn't really matter anyway, I doubt there's anything we
> > could do there.
>
> When reading "things which could have been C like", I had the exact
> same thought ;-)
>
> In the past, many new paradigms were implemented in a variety of new
> languages, usually inspired by other languages (e.g. some looked like
> C, others like Pascal).

Some background as I understand it:

The first "real" operating system (using virtual memory etc)
for the Atlas Machine supervisor in 1962 was AFAICT implemented in
Atlas autocode which was a dialect of ALGOL, which was the
lingua franca at the time. In the meantime, ALGOL inspired
CPL -> BCPL -> B -> C.

Pascal was influenced directly from ALGOL as can be seen in its
keeping of BEGIN and END and also used in some operating systems.

The language family is referred to as "imperative languages", mostly
by those who do not like this language family.

Since this time, almost all operating systems were written in
languages derived from ALGOL one way or the other, like Linux.

There are exceptions, many of them failed visions, such as the Japanese
fifth generation computer that wrote the operating system in KL-1
(kernel language one) a dialect of Prolog. The partially successful
would be LISP as used in the Lisp machines.

Rust on the other hand had its first compiler written in OCaml, clearly
the author liked that language and it shows (e.g. the let keyword clearly
comes from OCaml).

OCaml has a different lineage: ISWIM -> ML -> Caml -> OCaml
based on the article in 1966 by Peter Landin on Lambda calculus in
programming languages. OCaml is loosely in the same language family as
Haskell or Miranda. This family is called functional languages.

My intuitive appraisal is that the ALGOL family has traditionally
been appreciated by engineers and the Lambda calculus-functional family
has been appreciated by mathematicians. (From here comes many jokes
about X = X + 1 being insane maths etc, people.)

Rust actually tries to join these two families taking some syntax
from both. For example adding the keyword mut is a clear compromise.
This article section 2 overview of Rust is very nice:
https://dl.acm.org/doi/fullHtml/10.1145/3443420
(The rest of the article is for the mathematically inclined, i.e. not me.)

Maybe this should be a blog @people.kernel.org rather than a mailing
list post.

Yours,
Linus Walleij

Re: [TECH TOPIC] Rust

[James Bottomley]

On Sun, 2022-06-19 at 21:08 +0200, Geert Uytterhoeven wrote:
> Hi Laurent,
> 
> On Sun, Jun 19, 2022 at 3:49 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> > On Sun, Jun 19, 2022 at 08:56:46AM -0400, James Bottomley wrote:
[...]
> > > I really doubt anyone at the maintainer or reviewer level of the 
> > > kernel doesn't have the ability to learn rust (now whether they
> > > have the desire to is quite another matter) and I'd be surprised
> > > if we can find any kernel developer who only speaks C.  I think
> > > the biggest problem with rust is that there's definitely an anti-
> > > C bias in the language. What I mean by that is if you look at
> > > Java they chose to be as close to C where possible to make the
> > > language easier to learn.  Rust seems to have taken the opposite
> > > view and picked a lot of things which could have been C like
> > > (typing, function calls, returns, mutability, etc.) and done them
> > > differently just because they wanted to be different from
> > > C.  That does cause issues for C people because you tend to trip
> > > over the anti-C biases in the language and are constantly having
> > > to look the basic syntax up.  This really increases the learning
> > > curve going from C to rust, which gets annoying very fast, but
> > > it's not insurmountable.
> > 
> > Time to invent a C-like version of rust ? :-) Jokes aside, as far
> > as I understand the syntax is different from C because it was
> > inspired by other languages, not out of a desire to be anti-C. I
> > could be wrong though, and it doesn't really matter anyway, I doubt
> > there's anything we could do there.
> 
> When reading "things which could have been C like", I had the exact
> same thought ;-)
> 
> In the past, many new paradigms were implemented in a variety of new
> languages, usually inspired by other languages (e.g. some looked like
> C, others like Pascal).
> 
> At the risk of sounding too procrastinating (perfect is the enemy
> of good, and see James' thread about becoming too fearful...),
> perhaps it is wise to question if Rust arrived too soon, and a more
> C-like language implementing the same safety paradigms is just around
> the corner, or already in its infancy?

I'm not really sure we should wait.  The problem caused by the anti-C
biases in Rust is merely an increase in the hump you have to climb to
get a handle on the language.  As I said above, it's annoying, but not
insurmountable.  I don't think we have to wait for an easier language
for C programmers to come along to begin the Rust experiment.  Assuming
the experiment has mixed results, and that's due to people not getting
over the learning hump, we can always adopt a more C like memory safe
language later, assuming one comes along.

James

Re: [TECH TOPIC] Rust

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

On Sat, 2022-06-18 at 22:33 +0200, Miguel Ojeda wrote:
> The effort to add Rust support to the kernel is ongoing. There has
> been progress in different areas during the last year, and there are
> several topics that could benefit from discussion:
> 
>   - Dividing the `kernel` crate into pieces, dependency management
> between internal crates, writing crates in the rest of the kernel
> tree, etc.
> 
>   - Whether to allow dependencies on external crates and vendoring of
> useful third-party crates.
> 
>   - Toolchain requirements in the future and status of Rust unstable features.
> 

This one is a constant pain point for me in another project which
combines C and Rust code. We are constantly being told "of *course*
that doesn't work; you haven't updated your Rust toolchain for a whole
week!"

It also makes bisection basically impossible at times, because during
bisection we find that older code doesn't build with the *new*
toolchain either.

I think we need to set *very* clear rules and expectations about how
many times a decade it's permitted to require a new rust toolchain.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]

Re: [TECH TOPIC] Rust

[Dan Carpenter]

On Sun, Jun 19, 2022 at 04:49:18PM +0300, Laurent Pinchart wrote:
> Writing rust code that compiles (without wrapping everything in unsafe)
> requires understanding of life time management of objects. This is also
> required to write safe C code, but as can be seen through the kernel
> it's more often that not completely ignored, especially in drivers (just
> look at how many drivers that expose a chardev to userspace are happy
> using devm_kzalloc() to allocate their private data structure that is
> accessible through userspace calls). It may be caused by a combination
> of both the compiler an the kernel allowing bad practices (the fact that
> devm_kzalloc() exists without a huge flashing red warning around it
> shows that we either don't understand the problem or don't care),

The answer is that we don't understand the problem.  I've been vaguely
aware that there is a problem related to this but I've never been clear
enough to the point where I could actually tell if code is buggy.

It sounds like something which could probably be detected with static
analysis.  Most of the static analysis devs hang out on kernel-janitors
so if you see a bug and you think it might happen more than once then
send an email to kernel-janitors@vger.kernel.org.

Sometimes if we look at a patch then maybe we can identify several
heuristics to spot the bug.

regards,
dan carpenter

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[Linus Walleij]

On Sun, Jun 19, 2022 at 4:45 PM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:

> I think there's a growing problem in Linux which is exemplified by this
> Rust debate but which goes way beyond it: We're becoming too fearful of
> making big decisions to sustain innovation in some areas.

I agree.

> This really
> is a creeping cancer of inertia that has destroyed many projects before
> us and if we're not careful, we'll go the same way.

Strong words. This phenomenon is known from organization theory.
Organization theory exist in dry and tasteless versions and some more
colorful and spicy versions. Let's revisit the most spicy and colorful of
them all, Cyril Norhcote Parkinson, a true representative of the
atomic space age.

In his famous book "Parkinson's Law" from 1957 (a book passed to
me from my great grandfather, and annoyingly relevant to this day),
in the last chapter titled "Pension point, or the age of retirement" he
touches on the subject. There might be an illicit copy here, page 42:
http://sas2.elte.hu/tg/ptorv/Parkinson-s-Law.pdf

Parkinson is writing satire and it is a clear hyperbole. But it isn't funny
if it isn't relevant. He saw the same thing as you see, and just
state (apparently based on nothing but his own experience) that a
persons "age of wisdom" is followed by the "age of obstruction".
Just constant risk avoidance. Blaeh. Boring.

I don't know about eBPF, but Rust is nice. Let's merge it and see what
happens. As one of my friends working in embedded systems said:
"this is the only new thing I have seen in my career, the rest is just
repetitions of the past", and I agree with that.

My biggest worry is that the kernel community can become irrelevant
for young people as a demographic. Young people like things like
Rust and web-based merge requests on github. They are also very
smart. So I see it as partly an inclusion problem.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[Laurent Pinchart]

Hi Linus,

On Wed, Jun 22, 2022 at 11:59:03AM +0200, Linus Walleij wrote:
> On Sun, Jun 19, 2022 at 4:45 PM James Bottomley wrote:
> 
> > I think there's a growing problem in Linux which is exemplified by this
> > Rust debate but which goes way beyond it: We're becoming too fearful of
> > making big decisions to sustain innovation in some areas.
> 
> I agree.
> 
> > This really
> > is a creeping cancer of inertia that has destroyed many projects before
> > us and if we're not careful, we'll go the same way.
> 
> Strong words. This phenomenon is known from organization theory.
> Organization theory exist in dry and tasteless versions and some more
> colorful and spicy versions. Let's revisit the most spicy and colorful of
> them all, Cyril Norhcote Parkinson, a true representative of the
> atomic space age.
> 
> In his famous book "Parkinson's Law" from 1957 (a book passed to
> me from my great grandfather, and annoyingly relevant to this day),
> in the last chapter titled "Pension point, or the age of retirement" he
> touches on the subject. There might be an illicit copy here, page 42:
> http://sas2.elte.hu/tg/ptorv/Parkinson-s-Law.pdf
> 
> Parkinson is writing satire and it is a clear hyperbole. But it isn't funny
> if it isn't relevant. He saw the same thing as you see, and just
> state (apparently based on nothing but his own experience) that a
> persons "age of wisdom" is followed by the "age of obstruction".
> Just constant risk avoidance. Blaeh. Boring.
> 
> I don't know about eBPF, but Rust is nice. Let's merge it and see what
> happens. As one of my friends working in embedded systems said:
> "this is the only new thing I have seen in my career, the rest is just
> repetitions of the past", and I agree with that.

The past two years may have increased the reluctance of following the
strategy of "let's ignore any possible warning, it's going to be fine".

There's a set of non-technical issues around kernel development, from
community management to hiring skilled developers. I understand lots of
the concerns that have been expressed (including some by myself) in that
context. Some may be classified as unfounded fears (it would be so much
easier if we could tell in advance which ones are founded and which ones
are not), and I think using rust in Linux could actually help in some of
the non-technical areas (by attracting developers from different
horizons to the kernel for instance).

This doesn't mean that all concerns should be ignored, and I read most
of them more as constructive feedback trying to find solutions than just
attempts to stall progress here. For instance, I see a widespread
concern that, as rust spreads in more places through the kernel, it will
generate extra work for people who are not able or ready yet to handle
this. This can be handled in two ways: we can ignore that concern, and
deal later with the potential conflicts it could generate, or we could
try to address it already.

Could the second option be extra work to handle an issue that may not
materialize ? Yes, of course. I see it more as an opportunity myself.
For instance, we could address that concern by creating and advertising
a support channel for developers and maintainers who need help dealing
with rust impacting their work (e.g. when doing tree-wide reworks) could
get help, and making sure we'll have a few rust-in-linux maintainers
there who can provide support (it could be a mailing list, an IRC
channel, ...). Not only would it send the message that the rust-in-linux
community listens to concerns and cares, but it would also result in
having a useful support resource from day one, as well as showing that
this project really has the means it requires to move forward.

> My biggest worry is that the kernel community can become irrelevant
> for young people as a demographic. Young people like things like
> Rust and web-based merge requests on github. They are also very
> smart. So I see it as partly an inclusion problem.

If you put rust and github in the same bag, you'll drastically increase
the size of the mob that will chase you with pitchforks :-)

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[Linus Walleij]

On Wed, Jun 22, 2022 at 12:58 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:

> This doesn't mean that all concerns should be ignored, and I read most
> of them more as constructive feedback trying to find solutions than just
> attempts to stall progress here. For instance, I see a widespread
> concern that, as rust spreads in more places through the kernel, it will
> generate extra work for people who are not able or ready yet to handle
> this. This can be handled in two ways: we can ignore that concern, and
> deal later with the potential conflicts it could generate, or we could
> try to address it already.

I agree. My first reactions to Miguel's and Wedsons Rust patches were
not the most enthusiastic either, more along the lines "you think you
can do better than 50 years of C?"

Then I looked closer and saw how Rust is designed to join imperative
and functional programming to get the best of all worlds, to use the
latter to solve some of the problems with the former. So now I am
theoretically convinced that it will add real value.

> Could the second option be extra work to handle an issue that may not
> materialize ? Yes, of course. I see it more as an opportunity myself.
> For instance, we could address that concern by creating and advertising
> a support channel for developers and maintainers who need help dealing
> with rust impacting their work (e.g. when doing tree-wide reworks) could
> get help, and making sure we'll have a few rust-in-linux maintainers
> there who can provide support (it could be a mailing list, an IRC
> channel, ...).

Why not both.
https://lore.kernel.org/rust-for-linux/ <- this exists!
I bet Miguel has an IRC channel as well.

> Not only would it send the message that the rust-in-linux
> community listens to concerns and cares, but it would also result in
> having a useful support resource from day one, as well as showing that
> this project really has the means it requires to move forward.

I agree. What I pointed out to Miguel & Wedson was that I expect commitment
of a personal type, what you once said (at a conference) required to
have kernel development in the open in your DNA. To feel personally
responsible. In clear: I expect the Rust-in-linux people to keep reading
the mailing list and respond to questions even if they no longer have
funding, even if it is not their current dayjob. The task will become too
important on a personal level to be left on the roadside.

The role model for that is Torvalds working at Transmeta a few years
back and not losing focus on Linux. That is a social responsibility that
goes beyond the dayjob. This request is probably more than a little
intimidating, but it is what we are looking for. If one does not find it as fun
and rewarding as to go to such lengths, then one should not go and
change core kernel infrastructure.

So it does not need to be a lot of people, but it needs to be dedicated
people. Such as the people who made Kasan, which we accepted
pretty quickly and we were not too conservative about Kasan despite
it being so intrusive.

> > My biggest worry is that the kernel community can become irrelevant
> > for young people as a demographic. Young people like things like
> > Rust and web-based merge requests on github. They are also very
> > smart. So I see it as partly an inclusion problem.
>
> If you put rust and github in the same bag, you'll drastically increase
> the size of the mob that will chase you with pitchforks :-)

Ha ha I know. But I have seen many beautiful projects on GitHub,
especially PostmarketOS, as a review tool it still lacks compared to email.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[Laurent Pinchart]

On Wed, Jun 22, 2022 at 02:01:02PM +0200, Linus Walleij wrote:
> On Wed, Jun 22, 2022 at 12:58 PM Laurent Pinchart wrote:
> 
> > This doesn't mean that all concerns should be ignored, and I read most
> > of them more as constructive feedback trying to find solutions than just
> > attempts to stall progress here. For instance, I see a widespread
> > concern that, as rust spreads in more places through the kernel, it will
> > generate extra work for people who are not able or ready yet to handle
> > this. This can be handled in two ways: we can ignore that concern, and
> > deal later with the potential conflicts it could generate, or we could
> > try to address it already.
> 
> I agree. My first reactions to Miguel's and Wedsons Rust patches were
> not the most enthusiastic either, more along the lines "you think you
> can do better than 50 years of C?"
> 
> Then I looked closer and saw how Rust is designed to join imperative
> and functional programming to get the best of all worlds, to use the
> latter to solve some of the problems with the former. So now I am
> theoretically convinced that it will add real value.

I think the theoretical technical value has been sold properly now. I
see less push back on that alone, more about whether the pain is worth
the gain.

> > Could the second option be extra work to handle an issue that may not
> > materialize ? Yes, of course. I see it more as an opportunity myself.
> > For instance, we could address that concern by creating and advertising
> > a support channel for developers and maintainers who need help dealing
> > with rust impacting their work (e.g. when doing tree-wide reworks) could
> > get help, and making sure we'll have a few rust-in-linux maintainers
> > there who can provide support (it could be a mailing list, an IRC
> > channel, ...).
> 
> Why not both.

I'd rather have both indeed. In some cases e-mail works best, but when
you're actively working on a task and hit a roadblocker, being able to
get help immediately is much better. This is especially true if we want
to take the psychological aspect into account, I think it would be
crucial here to offer very quick support for the first few problems a
developer or maintainer will encounter. Not that it should become sloppy
after that, but I think people will be much more likely to try and find
a solution by themselves if they have been helped before and know
support is available than when they hit the first issue.

> https://lore.kernel.org/rust-for-linux/ <- this exists!
> I bet Miguel has an IRC channel as well.

It's even better if no new process has to be put in place, then all that
remains to do is properly advertise this to ensure that nobody will miss
it. Is there a way we could get those resources advertised when hitting
a rust-related build failure for instance ?

> > Not only would it send the message that the rust-in-linux
> > community listens to concerns and cares, but it would also result in
> > having a useful support resource from day one, as well as showing that
> > this project really has the means it requires to move forward.
> 
> I agree. What I pointed out to Miguel & Wedson was that I expect commitment
> of a personal type, what you once said (at a conference) required to
> have kernel development in the open in your DNA. To feel personally
> responsible. In clear: I expect the Rust-in-linux people to keep reading
> the mailing list and respond to questions even if they no longer have
> funding, even if it is not their current dayjob. The task will become too
> important on a personal level to be left on the roadside.
> 
> The role model for that is Torvalds working at Transmeta a few years
> back and not losing focus on Linux. That is a social responsibility that
> goes beyond the dayjob. This request is probably more than a little
> intimidating, but it is what we are looking for. If one does not find it as fun
> and rewarding as to go to such lengths, then one should not go and
> change core kernel infrastructure.
> 
> So it does not need to be a lot of people, but it needs to be dedicated
> people. Such as the people who made Kasan, which we accepted
> pretty quickly and we were not too conservative about Kasan despite
> it being so intrusive.
> 
> > > My biggest worry is that the kernel community can become irrelevant
> > > for young people as a demographic. Young people like things like
> > > Rust and web-based merge requests on github. They are also very
> > > smart. So I see it as partly an inclusion problem.
> >
> > If you put rust and github in the same bag, you'll drastically increase
> > the size of the mob that will chase you with pitchforks :-)
> 
> Ha ha I know. But I have seen many beautiful projects on GitHub,
> especially PostmarketOS, as a review tool it still lacks compared to email.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[NeilBrown]

On Mon, 20 Jun 2022, James Bottomley wrote:
> 
> I think there's a growing problem in Linux which is exemplified by this
> Rust debate but which goes way beyond it: We're becoming too fearful of
> making big decisions to sustain innovation in some areas.  This really
> is a creeping cancer of inertia that has destroyed many projects before
> us and if we're not careful, we'll go the same way.

Is this because Linux is just too big?  Are we squeezing too much into
one project, and becoming afraid to push on one piece for fear of
breaking another?

Of course, breaking Linux into separate pieces would mean we would need
to create APIs that were at least a little bit stable.  But it might
also mean that individual sub-projects could take risks and either
flourish or die without an undue impact on the rest of the ecosystem.

NeilBrown

[TECH TOPIC] Why is devm_kzalloc() harmful and what can we do about it

[Laurent Pinchart]

On Tue, Jun 21, 2022 at 06:11:22PM +0300, Dan Carpenter wrote:
> On Sun, Jun 19, 2022 at 04:49:18PM +0300, Laurent Pinchart wrote:
> > Writing rust code that compiles (without wrapping everything in unsafe)
> > requires understanding of life time management of objects. This is also
> > required to write safe C code, but as can be seen through the kernel
> > it's more often that not completely ignored, especially in drivers (just
> > look at how many drivers that expose a chardev to userspace are happy
> > using devm_kzalloc() to allocate their private data structure that is
> > accessible through userspace calls). It may be caused by a combination
> > of both the compiler an the kernel allowing bad practices (the fact that
> > devm_kzalloc() exists without a huge flashing red warning around it
> > shows that we either don't understand the problem or don't care),
> 
> The answer is that we don't understand the problem.  I've been vaguely
> aware that there is a problem related to this but I've never been clear
> enough to the point where I could actually tell if code is buggy.
> 
> It sounds like something which could probably be detected with static
> analysis.  Most of the static analysis devs hang out on kernel-janitors
> so if you see a bug and you think it might happen more than once then
> send an email to kernel-janitors@vger.kernel.org.
> 
> Sometimes if we look at a patch then maybe we can identify several
> heuristics to spot the bug.

Sounds like this could be a worth a dedicated discussion at the kernel
summit ?


devm_kzalloc() has been introduced more than 15 years ago and has
steadily grown in usage through the kernel sources (more than 6000 calls
and counting). While it has helped lowering the number of memory leaks,
it is not the magic tool that many seem to think it is.

The devres family of functions tie the lifetime of the resources they
allocate to the lifetime of a struct device bind to a driver. This is
the right thing to do for many resources, for instance MMIO or
interrupts need to be released when the device is unbound from its
driver at the latest, and the corresponding devm_* helpers ensure this.
However, drivers that expose resources to userspace have, in many cases,
to ensure that those resources can be safely accessed after the device
is unbound from its driver. A particular example is character device
nodes, which userspace can keep open and close after the device has been
unbound from the driver. If the memory region that stores the struct
cdev instance is allocated by devm_kzalloc(), it will be freed before
the file release handler gets to run.

Most kernel developers are not aware of this issue that affects an ever
growing number of drivers. The problem has been discussed in the past
([1], [2]) - interestingly in the context of Kernel Summit proposals,
but never scheduled there - but never addressed.

This talk proposal aims at raising awareness of the problem, present a
possible solution that has been proposed as an RFC ([3]), and discuss
what we can do to solve the issue. Solutions at the technical, community
and process levels will be discussed, as addressing the devm_kzalloc()
hamr also requires a plan to teach the kernel community and catch new
offending code when it gets submitted.

[1] https://lore.kernel.org/all/2111196.TG1k3f53YQ@avalon/
[2] https://lore.kernel.org/all/YOagA4bgdGYos5aa@kroah.com/
[3] https://lore.kernel.org/linux-media/20171116003349.19235-1-laurent.pinchart+renesas@ideasonboard.com/

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Why is devm_kzalloc() harmful and what can we do about it

[Dan Carpenter]

On Thu, Jun 23, 2022 at 01:58:46PM +0300, Laurent Pinchart wrote:
> The devres family of functions tie the lifetime of the resources they
> allocate to the lifetime of a struct device bind to a driver. This is
> the right thing to do for many resources, for instance MMIO or
> interrupts need to be released when the device is unbound from its
> driver at the latest, and the corresponding devm_* helpers ensure this.
> However, drivers that expose resources to userspace have, in many cases,
> to ensure that those resources can be safely accessed after the device
> is unbound from its driver. A particular example is character device
> nodes, which userspace can keep open and close after the device has been
> unbound from the driver. If the memory region that stores the struct
> cdev instance is allocated by devm_kzalloc(), it will be freed before
> the file release handler gets to run.
> 

This is a good general description of the problem, but it's not specific
enough for me to write a checker rule.  What I basically need is a patch
I guess, and then I could try write a checker rule for that exact code.

If your RFC patch were merged then the problem would be solved?

regards,
dan carpenter

Re: [TECH TOPIC] Why is devm_kzalloc() harmful and what can we do about it

[Laurent Pinchart]

Hi Dan,

On Thu, Jun 23, 2022 at 02:24:22PM +0300, Dan Carpenter wrote:
> On Thu, Jun 23, 2022 at 01:58:46PM +0300, Laurent Pinchart wrote:
> > The devres family of functions tie the lifetime of the resources they
> > allocate to the lifetime of a struct device bind to a driver. This is
> > the right thing to do for many resources, for instance MMIO or
> > interrupts need to be released when the device is unbound from its
> > driver at the latest, and the corresponding devm_* helpers ensure this.
> > However, drivers that expose resources to userspace have, in many cases,
> > to ensure that those resources can be safely accessed after the device
> > is unbound from its driver. A particular example is character device
> > nodes, which userspace can keep open and close after the device has been
> > unbound from the driver. If the memory region that stores the struct
> > cdev instance is allocated by devm_kzalloc(), it will be freed before
> > the file release handler gets to run.
> 
> This is a good general description of the problem, but it's not specific
> enough for me to write a checker rule.  What I basically need is a patch
> I guess, and then I could try write a checker rule for that exact code.
> 
> If your RFC patch were merged then the problem would be solved?

A new version of the RFC is needed, I'll try to work on that "soon".
Merging it will then solve part of the problem (races between userspace
calls and device unbind), but not the full problem (lifetime of objects
that can be accessed through the file descriptor release), and in V4L2
only.

The race condition fix should really move to the cdev level in my
opinion. For the lifetime management issue, we will need something else.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[James Bottomley]

On Thu, 2022-06-23 at 09:13 +1000, NeilBrown wrote:
> On Mon, 20 Jun 2022, James Bottomley wrote:
> > I think there's a growing problem in Linux which is exemplified by
> > this Rust debate but which goes way beyond it: We're becoming too
> > fearful of making big decisions to sustain innovation in some
> > areas.  This really is a creeping cancer of inertia that has
> > destroyed many projects before us and if we're not careful, we'll
> > go the same way.
> 
> Is this because Linux is just too big?

I don't think so ... it is historically true that larger projects tend
to have bigger indecision problems than smaller ones, but breaking them
up is treating the symptom not the cause.

>   Are we squeezing too much into one project, and becoming afraid to
> push on one piece for fear of breaking another?

Well, that's often the reason yes.  However it's not all "too many
moving parts"; look at the use space ABI problem: that would exist as a
club however large or small the project was.  The usespace ABI isn't
totally immutable as people project, it's just that mutations have to
be managed in a way that preserves backwards compatibility (or with the
agreement of all the ABI users), so the fear of adding something
because it exposes a user ABI and you don't know if its right and the
ABI would have to be supported forever if its not is somewhat bogus. 
We need to get the balance right in terms of giving the ABI a pathway
to evolve rather than squelching the innovation.

James

> Of course, breaking Linux into separate pieces would mean we would
> need to create APIs that were at least a little bit stable.  But it
> might also mean that individual sub-projects could take risks and
> either flourish or die without an undue impact on the rest of the
> ecosystem.
> 
> NeilBrown
> 

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Mauro Carvalho Chehab]

Em Sun, 19 Jun 2022 10:27:08 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> escreveu:

> On Sun, 2022-06-19 at 16:53 +0300, Laurent Pinchart wrote:
> > On Sun, Jun 19, 2022 at 07:19:38AM -0600, Jens Axboe wrote:  
> > > On 6/19/22 6:56 AM, James Bottomley wrote:  
> [...]
> > > > All that said, I expect once we start getting rust based patches,
> > > > it will quickly become apparent where the rust receptive
> > > > subsystems are and I would expect reviewers in those subsystems
> > > > will quickly become rust fluent.  This doesn't answer the pain vs
> > > > gain aspect, but I suspect we won't know that until we actually
> > > > try.  
> > > 
> > > FWIW, I agree with you here, James. And I have to say that the
> > > original sentiment expressed here by Laurent is weirdly elitist,
> > > and not something that seems positive if that's the case. I'd
> > > suggest re-calibrating that mindset...  
> > 
> > I'm not sure about the elitist part, but it's certainly not a
> > positive statement. I'm concerned about the pain that rust in the
> > kernel would inflict.  
> 
> I get that, and I think it's a serious concern.  I just also think we
> would waste more time second guessing this issue that we would consume
> if we just tried it.  It's not like allowing Rust into the kernel is
> some sort of Rubicon that once crossed we can never go back.  I think
> we won't know the pain vs gain until at least a year into the
> experiment of accepting Rust.  Even at that late stage, if everyone
> agrees it caused more problems than it solves, I bet it could all be
> quickly excised.
> 
> So the bottom line is I think while we have a community of volunteers
> willing to try it we let them on the understanding the results will be
> evaluated at some point.

Agreed.

> >  Whether it's worth the gain or not is not for me to decide,
> > but I'm certainly concerned that it could lead to a catastrophe if we
> > don't carefully think about the issues, acknowledge them, and find
> > ways to solve them. I don't think winging it is a real option here,
> > but I'd be more than happy to be shown that my concerns are not
> > founded :-)  
> 
> Have more faith in the community and open source process. We've screwed
> up many times (devfs anyone ...) learned from the mistake and fixed it.
> I'm happy to bet that accepting Rust will be no different from all the
> other screwups we allowed in and later fixed.  So I don't think there
> will be a catastrophe.  Either the rust experiment works or it will
> become fairly quickly apparent if it doesn't and it will get removed. 
> The worst case, I suppose, is that the benefit is marginal in which
> case there's no consensus on removal vs continuation and we simply
> continue through inertia. 

> I bet in that situation Rust penetration
> will be fairly minimal and confined to enthusiastic subsystems with the
> rest taking steps to isolate themselves from it.  

I suspect that Rust will be confined to enthusiastic subsystems for
quite a while, as porting it to more complex subsystems would require
lots of work on other subsystemd.

The thing is that using rust unsafe code doesn't make much sense, as
it won't bring any real advantage over C. So, subsystems C code will 
need to first be changed to provide the same sort of warranties that
rust will provide in safe mode. Such task should consider the 
subsystem's dependencies.

For instance, before being able to have rust (safe) code on driver 
subsystems, several other subsystems will need to be made compatible
with rust (also in safe mode):

- core parts: mm, vfs, devm_*, wait queues, threads, irq handling, driver
  core, ...;
- bus subsystems: PCI, USB, I2C, ...;
- other core driver-related subsystems like: power management, clocks,...
- finally, subsystem-specific core functions.

Which probably means that they would need to suffer some rewrite due to
rust safe mode restrictions, before finally receive Rust (safe) bindings.

Once done, such subsystem may start accepting new drivers for it either
in C or in Rust. At this point, maintainers and reviewers should be
fluent on both C and Rust.

-

Assuming that we get into a point were all the above happens for
subsystem XXX, at the Rust experiment validation point you mentioned
above, there will be some possible outcomes:

1) Rust experiment fails. On such case, the efforts to make the
subsystem C code ready to run safe Rust code will be welcomed,
as Linux should be safer. The work on providing Rust bindings will 
be wasted on such case. 

I can't measure how much would be spent on making C code safer and 
how much would be on writing Rust bindings though. If such efforts 
would be 80%-90% for making subsystems code safer, it should 
worth the efforts even if Rust code ends being removed.

2) Rust experiment succeeds. In long term it would mean that subsystems
should, at some point, stop accepting C code and start using only Rust
for new code, and several drivers in C would require conversion to Rust
or moved to staging in order to be deprecated.

3) Some maintainers would consider it a success while others would
consider it a failure. That's the worse case scenario. If we reach
that, some Kernel/Maintainers Summit hard discussions would likely 
be needed, in order to avoid subsystem's fragmentation with regards 
to accepting or not C and/or Rust code subsystem-wide.

Regards,
Mauro

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Are we becoming too fearful? [was Re: [TECH TOPIC] Rust]

[Mauro Carvalho Chehab]

Em Sun, 19 Jun 2022 10:45:00 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> escreveu:

> I think there's a growing problem in Linux which is exemplified by this
> Rust debate but which goes way beyond it: We're becoming too fearful of
> making big decisions to sustain innovation in some areas.

I don't think so. Rust is special in the sense that, if it succeeds
and the Kernel ends being Rewritten in Rust, it will affect everyone.
On a huge community like the Kernel, that means that this topic will
surely attract lots of discussions, and that decisions will be tough,
as it is hard to reach consensus. 

Yet, we're experimenting with it, meaning that the community is willing
to try innovating, even when this trying something different from
everything else already tried in the past.

> The biggest area where we currently squelch innovation is around
> anything that touches the user space ABI.  Allegations of having to get
> everything right ab initio because we have to support it "forever" and
> all the subsequent wittering and second guessing are really stifling
> innovation in pretty much anything that could be exposed to user space.

I agree with that. In the past, we have the EXPERIMENTAL kconfig where
we did some experiments even with uAPI/uAPI. Nowadays, lots of such
experiments are actually happening happen outside upstream, as:

1) it is very hard to have maintainers accept APIs that are used only
   by a single driver;

2) nobody wants to carry the costs of adding backward-compatible
   workarounds for APIs that ends requiring changes after new hardware
   devices would require changes on it.

Not sure how to address it without adding an "API staging" support
somehow.

Regards,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[James Bottomley]

On Sat, 2022-06-25 at 12:45 +0100, Mauro Carvalho Chehab wrote:
[...]
> Assuming that we get into a point were all the above happens for
> subsystem XXX, at the Rust experiment validation point you mentioned
> above, there will be some possible outcomes:
> 
> 1) Rust experiment fails. On such case, the efforts to make the
> subsystem C code ready to run safe Rust code will be welcomed,
> as Linux should be safer. The work on providing Rust bindings will 
> be wasted on such case. 

Not entirely ... we'll still have gone through the learning exercise of
how do you do this.  If another language promising additional safety
features comes along we'll be better at it next time.

> I can't measure how much would be spent on making C code safer and 
> how much would be on writing Rust bindings though. If such efforts 
> would be 80%-90% for making subsystems code safer, it should 
> worth the efforts even if Rust code ends being removed.
> 
> 2) Rust experiment succeeds. In long term it would mean that
> subsystems should, at some point, stop accepting C code and start
> using only Rust for new code, and several drivers in C would require
> conversion to Rust or moved to staging in order to be deprecated.

I don't think that's what success looks like.  I think we'd continue in
dual C/Rust mode almost indefinitely.  Look how long it took the kernel
to add enough flexibility just to compile the existing C with clang.

There may be a point where we ask should we deprecate C, but that's
independent of and probably way beyond the point where Rust is a
success.

> 3) Some maintainers would consider it a success while others would
> consider it a failure. That's the worse case scenario. If we reach
> that, some Kernel/Maintainers Summit hard discussions would likely 
> be needed, in order to avoid subsystem's fragmentation with regards 
> to accepting or not C and/or Rust code subsystem-wide.

So this is where we have to ask hard questions about what does success
(or failure) actually look like. I think success is really at least one
subsystem demonstrates support and enthusiasm for the Rust conversion
and it doesn't cause a huge drain on those who don't embrace it.  That
leaves open the door for more converts but doesn't force them.  We can
continually evaluate the maintenance burden in this mode and debate
whether it's too great or is acceptable.  I think failure is definitely
no subsystem wants to embrace rust.  The problem case is where a
subsystem embraces rust has issues but doesn't want to admit failure
because it would lead to the failure of the rust project ... this one
we'll need to examine in detail.

James

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Mauro Carvalho Chehab]

Em Sat, 25 Jun 2022 10:15:06 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> escreveu:

> On Sat, 2022-06-25 at 12:45 +0100, Mauro Carvalho Chehab wrote:
> [...]
> > Assuming that we get into a point were all the above happens for
> > subsystem XXX, at the Rust experiment validation point you mentioned
> > above, there will be some possible outcomes:
> > 
> > 1) Rust experiment fails. On such case, the efforts to make the
> > subsystem C code ready to run safe Rust code will be welcomed,
> > as Linux should be safer. The work on providing Rust bindings will 
> > be wasted on such case.   
> 
> Not entirely ... we'll still have gone through the learning exercise of
> how do you do this.  If another language promising additional safety
> features comes along we'll be better at it next time.

Indeed.

> 
> > I can't measure how much would be spent on making C code safer and 
> > how much would be on writing Rust bindings though. If such efforts 
> > would be 80%-90% for making subsystems code safer, it should 
> > worth the efforts even if Rust code ends being removed.
> > 
> > 2) Rust experiment succeeds. In long term it would mean that
> > subsystems should, at some point, stop accepting C code and start
> > using only Rust for new code, and several drivers in C would require
> > conversion to Rust or moved to staging in order to be deprecated.  
> 
> I don't think that's what success looks like.  I think we'd continue in
> dual C/Rust mode almost indefinitely. 

I disagree. The burden of maintaining both C and Rust code on subsystems
will likely force the hand to either switch to Rust or to remove it, as
it will increase with time and with more code porting to Rust or written
in C.

I mean, when new Rust code gets added, specially at subsystem's core, it
may end that some features will only work properly (or would be
thoughtfully tested) in Rust while others would only work properly in C.

So, if enough subsystem's developers got convinced about the advantages
of it, the long road to deprecate C may start happening - at least
subsystem-wide.

Granted, a full conversion will certainly require years or even decades, 
but that's a separate history.

> Look how long it took the kernel to add enough flexibility just to
> compile the existing C with clang.

It looks a lot more, but it wasn't that much of time.

Clang self-hosting started in 2010[1]. So, we're talking of ~12 years of 
since when it started compiling itself. 

During such time frame, Clang got mature enough to become compatible with 
real-world open source code, which probably happened somewhere between
Clang 3.5 (2014) and Clang 4.0 (2017).

[1] See: https://en.wikipedia.org/wiki/Clang

    IMO, before self-hosting, it was more like a compiler experiment, as
    clang wouldn't survive on a gcc-free world.

> There may be a point where we ask should we deprecate C, but that's
> independent of and probably way beyond the point where Rust is a
> success.
> 
> > 3) Some maintainers would consider it a success while others would
> > consider it a failure. That's the worse case scenario. If we reach
> > that, some Kernel/Maintainers Summit hard discussions would likely 
> > be needed, in order to avoid subsystem's fragmentation with regards 
> > to accepting or not C and/or Rust code subsystem-wide.  
> 
> So this is where we have to ask hard questions about what does success
> (or failure) actually look like. I think success is really at least one
> subsystem demonstrates support and enthusiasm for the Rust conversion
> and it doesn't cause a huge drain on those who don't embrace it.

I don't think that having a single subsystem adopting Rust would be enough for
it to be considered a success or a failure. It will very much depend on what
subsystem is, and, most importantly, how the Rust code on it will interact 
with other subsystems that depend on it.

E. g. one thing would be to have a subsystem for a new class of devices
using Rust. People that won't need such devices will barely notice it.
A very different thing would be to have a widely used subsystem
enthusiastically supporting Rust in a proper way.

> That leaves open the door for more converts but doesn't force them.  We 
> can continually evaluate the maintenance burden in this mode and debate
> whether it's too great or is acceptable. 

> I think failure is definitely no subsystem wants to embrace rust.

Agreed: this would be a failure.

> The problem case is where a subsystem embraces rust has issues but
> doesn't want to admit failure because it would lead to the failure of
> the rust project ... this one we'll need to examine in detail.

True. 

Regards,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Linus Walleij]

On Sat, Jun 25, 2022 at 1:45 PM Mauro Carvalho Chehab
<mchehab@kernel.org> wrote:

> - core parts: mm, vfs, devm_*, wait queues, threads, irq handling, driver
>   core, ...;
> - bus subsystems: PCI, USB, I2C, ...;
> - other core driver-related subsystems like: power management, clocks,...
> - finally, subsystem-specific core functions.
>
> Which probably means that they would need to suffer some rewrite due to
> rust safe mode restrictions, before finally receive Rust (safe) bindings.

I think it is important to keep in mind that Rust may save us
from a few crashes when used in drivers, but that can't be the
main target for Rust.

Rust needs to be put facing the enemy, i.e. people who exploit the
kernel to run arbitrary code and gain local root.

This does not happen in drivers, it happens in abstract code, such
as protocols and the userspace/kernelspace boundary.

OK the USB protocol may be close to the driver but
I'm thinking the network stack and I think during initial review I
suggested things like ksmbd as Windows SMB is known to have been
exploited by intelligence agencies we know things like that is facing
the enemy. I don't know what the ksmbd people feel about that, or
nfsd for that matter, but this kind of code is what matters most
for Rust deployment to pay off.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Theodore Ts'o]

On Sun, Jun 26, 2022 at 12:29:27AM +0200, Linus Walleij wrote:
> 
> I think it is important to keep in mind that Rust may save us
> from a few crashes when used in drivers, but that can't be the
> main target for Rust.
> 
> Rust needs to be put facing the enemy, i.e. people who exploit the
> kernel to run arbitrary code and gain local root.

In order to solve the problem you've articulated, the entire kernel
would have to be rewritten in Rust.  And that's not going to happen
for... a while.

Rust has more opportunities to improve things in drivers because (a)
they aren't tested as well (for example, it's a lot harder for fuzzers
to exercise drivers if the hardware isnt available on the VM or server
where the fuzzers are running), and (b) very often drivers are written
by less experienced engineers --- in some cases the drivers are
authored by hardware engineers.

> OK the USB protocol may be close to the driver but
> I'm thinking the network stack and I think during initial review I
> suggested things like ksmbd as Windows SMB is known to have been
> exploited by intelligence agencies we know things like that is facing
> the enemy. I don't know what the ksmbd people feel about that, or
> nfsd for that matter, but this kind of code is what matters most
> for Rust deployment to pay off.

ksmbd needs to interface with a large number of subsystems that are
written in C.  So even if you rewrite ksmbd in Rust, it will need to
made a huge number of unsafe calls into C code.  Speaking personally
my solution for ksmbd is "Just Say No".  Obviously, that won't be a
solution for everyone, but if you want a short-term solution, and you
care more about security than performance, I'd suggest using a
userspace file server, such as Samba or Ganesha.

Cheers,

					- Ted

Re: [Ksummit-discuss] [TECH TOPIC] Rust

[Al Viro]

On Sat, Jun 25, 2022 at 10:52:27PM -0400, Theodore Ts'o wrote:
 
> ksmbd needs to interface with a large number of subsystems that are
> written in C.  So even if you rewrite ksmbd in Rust, it will need to
> made a huge number of unsafe calls into C code.

Not to mention the fact that ksmbd bugs I've seen so far had nothing
to do with "memory safety" and they would translate into Rust just
fine, TYVM, even if the stuff outside of ksmbd had also been all in
Rust...