Rust in our code base

Rust in our code base

[Karol Herbst]

Hey everybody,

so I think it's time to have this discussion for real.

I am working on Rusticl
(https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
which I would like to merge quite soon.

Others might also plan on starting kernel drivers written in Rust (and
if people feel comfortable to discuss this as well, they might reply
here)

The overall implication of that is: if we are doing this, people (that
is we) have to accept that touching Rust code will be part of our
development process. There is no other sane way of doing it.

I am not willing to wrap things in Rusticl so changing gallium APIs
won't involve touching Rust code, and we also can't expect people to
design their kernel drivers in weird ways "just because somebody
doesn't want to deal with Rust"

If we are going to do this, we have to do it for real, which means,
Rust code will call C APIs directly and a change in those APIs will
also require changes in Rust code making use of those APIs.

I am so explicit on this very point, because we had some discussion on
IRC where this was seen as a no-go at least from some people, which
makes me think we have to find a mutual agreement on how it should be
going forward.

And I want to be very explicit here about the future of Rusticl as
well: if the agreement is that people don't want to have to deal with
Rust changing e.g. gallium, Rusticl is a dead project. I am not
willing to come up with some trashy external-internal API just to
maintain Rusticl outside of the mesa git repo.
And doing it on a kernel level is even more of a no-go.

So what are we all thinking about Rust in our core repos?

CCing a bunch of people who think are the most impacted by it either
from a reviewer/maintainer or developer perspective. If you think some
others should be explicitly aware of this, please point them to this
discussion here.

Karol

Re: Rust in our code base

[Rob Clark]

On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst <kherbst@redhat.com> wrote:
>
> Hey everybody,
>
> so I think it's time to have this discussion for real.
>
> I am working on Rusticl
> (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
> which I would like to merge quite soon.
>
> Others might also plan on starting kernel drivers written in Rust (and
> if people feel comfortable to discuss this as well, they might reply
> here)
>
> The overall implication of that is: if we are doing this, people (that
> is we) have to accept that touching Rust code will be part of our
> development process. There is no other sane way of doing it.
>
> I am not willing to wrap things in Rusticl so changing gallium APIs
> won't involve touching Rust code, and we also can't expect people to
> design their kernel drivers in weird ways "just because somebody
> doesn't want to deal with Rust"
>
> If we are going to do this, we have to do it for real, which means,
> Rust code will call C APIs directly and a change in those APIs will
> also require changes in Rust code making use of those APIs.
>
> I am so explicit on this very point, because we had some discussion on
> IRC where this was seen as a no-go at least from some people, which
> makes me think we have to find a mutual agreement on how it should be
> going forward.
>
> And I want to be very explicit here about the future of Rusticl as
> well: if the agreement is that people don't want to have to deal with
> Rust changing e.g. gallium, Rusticl is a dead project. I am not
> willing to come up with some trashy external-internal API just to
> maintain Rusticl outside of the mesa git repo.
> And doing it on a kernel level is even more of a no-go.
>
> So what are we all thinking about Rust in our core repos?

I think there has to be willingness on the part of rust folks to help
others who aren't so familiar with rust with these sorts of API
changes.  You can't completely impose the burden on others who have
never touched rust before.  That said, I expect a lot of API changes
over time are simple enough that other devs could figure out the
related rust side changes.

As long as folks who want to start introducing rust in mesa and drm
realize they are also signing up to play the role of rust tutor and
technical assistance, I don't see a problem.  But if they aren't
around and willing to help, I could see this going badly.

I do also wonder a bit about code tooling (indexers, etc).. I'm not
sure what the state of things when it comes to cross c<->rust
integration.  Ie. it is usually straightforward enough to track down
all the spots in C code which would be affected by some change.  It
might be easier to overlook things on the rust side.  On the mesa
side, pre-merge CI jobs help to catch these issues.  Less sure about
how to handle that on the kernel side.

BR,
-R

> CCing a bunch of people who think are the most impacted by it either
> from a reviewer/maintainer or developer perspective. If you think some
> others should be explicitly aware of this, please point them to this
> discussion here.
>
> Karol
>

Re: Rust in our code base

[Karol Herbst]

On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com> wrote:
>
> On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst <kherbst@redhat.com> wrote:
> >
> > Hey everybody,
> >
> > so I think it's time to have this discussion for real.
> >
> > I am working on Rusticl
> > (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
> > which I would like to merge quite soon.
> >
> > Others might also plan on starting kernel drivers written in Rust (and
> > if people feel comfortable to discuss this as well, they might reply
> > here)
> >
> > The overall implication of that is: if we are doing this, people (that
> > is we) have to accept that touching Rust code will be part of our
> > development process. There is no other sane way of doing it.
> >
> > I am not willing to wrap things in Rusticl so changing gallium APIs
> > won't involve touching Rust code, and we also can't expect people to
> > design their kernel drivers in weird ways "just because somebody
> > doesn't want to deal with Rust"
> >
> > If we are going to do this, we have to do it for real, which means,
> > Rust code will call C APIs directly and a change in those APIs will
> > also require changes in Rust code making use of those APIs.
> >
> > I am so explicit on this very point, because we had some discussion on
> > IRC where this was seen as a no-go at least from some people, which
> > makes me think we have to find a mutual agreement on how it should be
> > going forward.
> >
> > And I want to be very explicit here about the future of Rusticl as
> > well: if the agreement is that people don't want to have to deal with
> > Rust changing e.g. gallium, Rusticl is a dead project. I am not
> > willing to come up with some trashy external-internal API just to
> > maintain Rusticl outside of the mesa git repo.
> > And doing it on a kernel level is even more of a no-go.
> >
> > So what are we all thinking about Rust in our core repos?
>
> I think there has to be willingness on the part of rust folks to help
> others who aren't so familiar with rust with these sorts of API
> changes.  You can't completely impose the burden on others who have
> never touched rust before.  That said, I expect a lot of API changes
> over time are simple enough that other devs could figure out the
> related rust side changes.
>

yeah, I agree here. I wouldn't say it's all the responsibility of
developers changing APIs to also know how to change the code. So e.g.
if an MR fails to compile and it's because of rusticl, I will help out
and do the changes myself if necessary. But long term we have to
accept that API changes also come with the implication of also having
to touch Rust code.

Short term it might be a learning opportunity for some/most, but long
term it has to be accepted as a part of development to deal with Rust.

> As long as folks who want to start introducing rust in mesa and drm
> realize they are also signing up to play the role of rust tutor and
> technical assistance, I don't see a problem.  But if they aren't
> around and willing to help, I could see this going badly.
>

Yep, I fully agree here. This is also the main reason I am bringing
this up. Nobody should be left alone with having to deal with changing
the code. On the other hand a "not having to touch Rust code when
changing APIs" guarantee is something which is simply impossible to
have in any sane architecture. So we should figure out under which
circumstances it will be okay for everybody.

At least I don't see a way how I can structure Rusticl so that
somebody working on gallium won't have to also deal with rusticl. One
possibility would be to have a libgallium.so file I can link to, but
then it's all about "stable gallium API" vs "not having to touch rust
code" and I hope everybody prefers the second :)

> I do also wonder a bit about code tooling (indexers, etc).. I'm not
> sure what the state of things when it comes to cross c<->rust
> integration.  Ie. it is usually straightforward enough to track down
> all the spots in C code which would be affected by some change.  It
> might be easier to overlook things on the rust side.  On the mesa
> side, pre-merge CI jobs help to catch these issues.  Less sure about
> how to handle that on the kernel side.
>

At least for Rusticl it's all within meson/ninja. We use bindgen to
generate the bindings automatically so you simply run into compilation
issues. And for the kernel side I think that Linus wanted Rust to be
non optional. If something uses it, you also make sure the Rust side
compiles. And the build system is dynamic enough that you can also
wire up bindgen and make it part of the normal development model.

In regards to code tooling, for rust you usually rely on
rust-analyzer. I've already figured out with dcbaker on how to
integrate our meson setup with that. Atm I am able to get the full
experience with VScode. Not sure if we also need to figure out how
that can work with e.g. vim and how to deal with C <=> Rust
boundaries.

> BR,
> -R
>
> > CCing a bunch of people who think are the most impacted by it either
> > from a reviewer/maintainer or developer perspective. If you think some
> > others should be explicitly aware of this, please point them to this
> > discussion here.
> >
> > Karol
> >
>

Re: Rust in our code base

[Karol Herbst]

On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com> wrote:
>
> On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com> wrote:
> >
> > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst <kherbst@redhat.com> wrote:
> > >
> > > Hey everybody,
> > >
> > > so I think it's time to have this discussion for real.
> > >
> > > I am working on Rusticl
> > > (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
> > > which I would like to merge quite soon.
> > >
> > > Others might also plan on starting kernel drivers written in Rust (and
> > > if people feel comfortable to discuss this as well, they might reply
> > > here)
> > >
> > > The overall implication of that is: if we are doing this, people (that
> > > is we) have to accept that touching Rust code will be part of our
> > > development process. There is no other sane way of doing it.
> > >
> > > I am not willing to wrap things in Rusticl so changing gallium APIs
> > > won't involve touching Rust code, and we also can't expect people to
> > > design their kernel drivers in weird ways "just because somebody
> > > doesn't want to deal with Rust"
> > >
> > > If we are going to do this, we have to do it for real, which means,
> > > Rust code will call C APIs directly and a change in those APIs will
> > > also require changes in Rust code making use of those APIs.
> > >
> > > I am so explicit on this very point, because we had some discussion on
> > > IRC where this was seen as a no-go at least from some people, which
> > > makes me think we have to find a mutual agreement on how it should be
> > > going forward.
> > >
> > > And I want to be very explicit here about the future of Rusticl as
> > > well: if the agreement is that people don't want to have to deal with
> > > Rust changing e.g. gallium, Rusticl is a dead project. I am not
> > > willing to come up with some trashy external-internal API just to
> > > maintain Rusticl outside of the mesa git repo.
> > > And doing it on a kernel level is even more of a no-go.
> > >
> > > So what are we all thinking about Rust in our core repos?
> >
> > I think there has to be willingness on the part of rust folks to help
> > others who aren't so familiar with rust with these sorts of API
> > changes.  You can't completely impose the burden on others who have
> > never touched rust before.  That said, I expect a lot of API changes
> > over time are simple enough that other devs could figure out the
> > related rust side changes.
> >
>
> yeah, I agree here. I wouldn't say it's all the responsibility of
> developers changing APIs to also know how to change the code. So e.g.
> if an MR fails to compile and it's because of rusticl, I will help out
> and do the changes myself if necessary. But long term we have to
> accept that API changes also come with the implication of also having
> to touch Rust code.
>
> Short term it might be a learning opportunity for some/most, but long
> term it has to be accepted as a part of development to deal with Rust.
>
> > As long as folks who want to start introducing rust in mesa and drm
> > realize they are also signing up to play the role of rust tutor and
> > technical assistance, I don't see a problem.  But if they aren't
> > around and willing to help, I could see this going badly.
> >
>
> Yep, I fully agree here. This is also the main reason I am bringing
> this up. Nobody should be left alone with having to deal with changing
> the code. On the other hand a "not having to touch Rust code when
> changing APIs" guarantee is something which is simply impossible to
> have in any sane architecture. So we should figure out under which
> circumstances it will be okay for everybody.
>
> At least I don't see a way how I can structure Rusticl so that
> somebody working on gallium won't have to also deal with rusticl. One
> possibility would be to have a libgallium.so file I can link to, but
> then it's all about "stable gallium API" vs "not having to touch rust
> code" and I hope everybody prefers the second :)
>

uhm... I meant "stable gallium API" vs "dealing with Rust code on API changes"

> > I do also wonder a bit about code tooling (indexers, etc).. I'm not
> > sure what the state of things when it comes to cross c<->rust
> > integration.  Ie. it is usually straightforward enough to track down
> > all the spots in C code which would be affected by some change.  It
> > might be easier to overlook things on the rust side.  On the mesa
> > side, pre-merge CI jobs help to catch these issues.  Less sure about
> > how to handle that on the kernel side.
> >
>
> At least for Rusticl it's all within meson/ninja. We use bindgen to
> generate the bindings automatically so you simply run into compilation
> issues. And for the kernel side I think that Linus wanted Rust to be
> non optional. If something uses it, you also make sure the Rust side
> compiles. And the build system is dynamic enough that you can also
> wire up bindgen and make it part of the normal development model.
>
> In regards to code tooling, for rust you usually rely on
> rust-analyzer. I've already figured out with dcbaker on how to
> integrate our meson setup with that. Atm I am able to get the full
> experience with VScode. Not sure if we also need to figure out how
> that can work with e.g. vim and how to deal with C <=> Rust
> boundaries.
>
> > BR,
> > -R
> >
> > > CCing a bunch of people who think are the most impacted by it either
> > > from a reviewer/maintainer or developer perspective. If you think some
> > > others should be explicitly aware of this, please point them to this
> > > discussion here.
> > >
> > > Karol
> > >
> >

Re: Rust in our code base

[Rob Clark]

On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst@redhat.com> wrote:
>
> On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com> wrote:
> >
> > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com> wrote:
> > >
> > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst <kherbst@redhat.com> wrote:
> > > >
> > > > Hey everybody,
> > > >
> > > > so I think it's time to have this discussion for real.
> > > >
> > > > I am working on Rusticl
> > > > (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
> > > > which I would like to merge quite soon.
> > > >
> > > > Others might also plan on starting kernel drivers written in Rust (and
> > > > if people feel comfortable to discuss this as well, they might reply
> > > > here)
> > > >
> > > > The overall implication of that is: if we are doing this, people (that
> > > > is we) have to accept that touching Rust code will be part of our
> > > > development process. There is no other sane way of doing it.
> > > >
> > > > I am not willing to wrap things in Rusticl so changing gallium APIs
> > > > won't involve touching Rust code, and we also can't expect people to
> > > > design their kernel drivers in weird ways "just because somebody
> > > > doesn't want to deal with Rust"
> > > >
> > > > If we are going to do this, we have to do it for real, which means,
> > > > Rust code will call C APIs directly and a change in those APIs will
> > > > also require changes in Rust code making use of those APIs.
> > > >
> > > > I am so explicit on this very point, because we had some discussion on
> > > > IRC where this was seen as a no-go at least from some people, which
> > > > makes me think we have to find a mutual agreement on how it should be
> > > > going forward.
> > > >
> > > > And I want to be very explicit here about the future of Rusticl as
> > > > well: if the agreement is that people don't want to have to deal with
> > > > Rust changing e.g. gallium, Rusticl is a dead project. I am not
> > > > willing to come up with some trashy external-internal API just to
> > > > maintain Rusticl outside of the mesa git repo.
> > > > And doing it on a kernel level is even more of a no-go.
> > > >
> > > > So what are we all thinking about Rust in our core repos?
> > >
> > > I think there has to be willingness on the part of rust folks to help
> > > others who aren't so familiar with rust with these sorts of API
> > > changes.  You can't completely impose the burden on others who have
> > > never touched rust before.  That said, I expect a lot of API changes
> > > over time are simple enough that other devs could figure out the
> > > related rust side changes.
> > >
> >
> > yeah, I agree here. I wouldn't say it's all the responsibility of
> > developers changing APIs to also know how to change the code. So e.g.
> > if an MR fails to compile and it's because of rusticl, I will help out
> > and do the changes myself if necessary. But long term we have to
> > accept that API changes also come with the implication of also having
> > to touch Rust code.
> >
> > Short term it might be a learning opportunity for some/most, but long
> > term it has to be accepted as a part of development to deal with Rust.
> >
> > > As long as folks who want to start introducing rust in mesa and drm
> > > realize they are also signing up to play the role of rust tutor and
> > > technical assistance, I don't see a problem.  But if they aren't
> > > around and willing to help, I could see this going badly.
> > >
> >
> > Yep, I fully agree here. This is also the main reason I am bringing
> > this up. Nobody should be left alone with having to deal with changing
> > the code. On the other hand a "not having to touch Rust code when
> > changing APIs" guarantee is something which is simply impossible to
> > have in any sane architecture. So we should figure out under which
> > circumstances it will be okay for everybody.

Yeah, this sounds fine to me.

> > At least I don't see a way how I can structure Rusticl so that
> > somebody working on gallium won't have to also deal with rusticl. One
> > possibility would be to have a libgallium.so file I can link to, but
> > then it's all about "stable gallium API" vs "not having to touch rust
> > code" and I hope everybody prefers the second :)
> >
>
> uhm... I meant "stable gallium API" vs "dealing with Rust code on API changes"

Yeah, stable gallium ABI makes as much sense as stable drm (in-kernel)
ABI (ie. it doesn't).  So even if we tried to make it so that folks
who haven't used rust much (or at all) yet on the gallium side, we'll
still have to deal with it on the kernel side.  So let's not design
our APIs to avoid a learning curve.. lets' just recognize that the
learning curve will be there.

> > > I do also wonder a bit about code tooling (indexers, etc).. I'm not
> > > sure what the state of things when it comes to cross c<->rust
> > > integration.  Ie. it is usually straightforward enough to track down
> > > all the spots in C code which would be affected by some change.  It
> > > might be easier to overlook things on the rust side.  On the mesa
> > > side, pre-merge CI jobs help to catch these issues.  Less sure about
> > > how to handle that on the kernel side.
> > >
> >
> > At least for Rusticl it's all within meson/ninja. We use bindgen to
> > generate the bindings automatically so you simply run into compilation
> > issues. And for the kernel side I think that Linus wanted Rust to be
> > non optional. If something uses it, you also make sure the Rust side
> > compiles. And the build system is dynamic enough that you can also
> > wire up bindgen and make it part of the normal development model.
> >
> > In regards to code tooling, for rust you usually rely on
> > rust-analyzer. I've already figured out with dcbaker on how to
> > integrate our meson setup with that. Atm I am able to get the full
> > experience with VScode. Not sure if we also need to figure out how
> > that can work with e.g. vim and how to deal with C <=> Rust
> > boundaries.

Yeah, I've used vscode with rust codebase (crosvm).. just not for a
project that includes the combination of a significant amount of both
.c and .rs.

If bindgen is integrated with the build system, that sounds fine as
far as helping to not overlook changes that are needed on the .rs side
of things.. crosvm seems to mainly make bindgen a manual step, but
then again it is mostly used for external dependencies like the
interface to virglrenderer

Side note, from my experience with crosvm, please be careful about any
external rust dependencies that are used.. rust still seems to be
young enough and evolving quickly enough that thing can quickly turn
into a big headache

BR,
-R

> > > BR,
> > > -R
> > >
> > > > CCing a bunch of people who think are the most impacted by it either
> > > > from a reviewer/maintainer or developer perspective. If you think some
> > > > others should be explicitly aware of this, please point them to this
> > > > discussion here.
> > > >
> > > > Karol
> > > >
> > >
>

Re: Rust in our code base

[Karol Herbst]

On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark@gmail.com> wrote:
>
> On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst@redhat.com> wrote:
> >
> > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com> wrote:
> > >
> > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com> wrote:
> > > >
> > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst <kherbst@redhat.com> wrote:
> > > > >
> > > > > Hey everybody,
> > > > >
> > > > > so I think it's time to have this discussion for real.
> > > > >
> > > > > I am working on Rusticl
> > > > > (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15439)
> > > > > which I would like to merge quite soon.
> > > > >
> > > > > Others might also plan on starting kernel drivers written in Rust (and
> > > > > if people feel comfortable to discuss this as well, they might reply
> > > > > here)
> > > > >
> > > > > The overall implication of that is: if we are doing this, people (that
> > > > > is we) have to accept that touching Rust code will be part of our
> > > > > development process. There is no other sane way of doing it.
> > > > >
> > > > > I am not willing to wrap things in Rusticl so changing gallium APIs
> > > > > won't involve touching Rust code, and we also can't expect people to
> > > > > design their kernel drivers in weird ways "just because somebody
> > > > > doesn't want to deal with Rust"
> > > > >
> > > > > If we are going to do this, we have to do it for real, which means,
> > > > > Rust code will call C APIs directly and a change in those APIs will
> > > > > also require changes in Rust code making use of those APIs.
> > > > >
> > > > > I am so explicit on this very point, because we had some discussion on
> > > > > IRC where this was seen as a no-go at least from some people, which
> > > > > makes me think we have to find a mutual agreement on how it should be
> > > > > going forward.
> > > > >
> > > > > And I want to be very explicit here about the future of Rusticl as
> > > > > well: if the agreement is that people don't want to have to deal with
> > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am not
> > > > > willing to come up with some trashy external-internal API just to
> > > > > maintain Rusticl outside of the mesa git repo.
> > > > > And doing it on a kernel level is even more of a no-go.
> > > > >
> > > > > So what are we all thinking about Rust in our core repos?
> > > >
> > > > I think there has to be willingness on the part of rust folks to help
> > > > others who aren't so familiar with rust with these sorts of API
> > > > changes.  You can't completely impose the burden on others who have
> > > > never touched rust before.  That said, I expect a lot of API changes
> > > > over time are simple enough that other devs could figure out the
> > > > related rust side changes.
> > > >
> > >
> > > yeah, I agree here. I wouldn't say it's all the responsibility of
> > > developers changing APIs to also know how to change the code. So e.g.
> > > if an MR fails to compile and it's because of rusticl, I will help out
> > > and do the changes myself if necessary. But long term we have to
> > > accept that API changes also come with the implication of also having
> > > to touch Rust code.
> > >
> > > Short term it might be a learning opportunity for some/most, but long
> > > term it has to be accepted as a part of development to deal with Rust.
> > >
> > > > As long as folks who want to start introducing rust in mesa and drm
> > > > realize they are also signing up to play the role of rust tutor and
> > > > technical assistance, I don't see a problem.  But if they aren't
> > > > around and willing to help, I could see this going badly.
> > > >
> > >
> > > Yep, I fully agree here. This is also the main reason I am bringing
> > > this up. Nobody should be left alone with having to deal with changing
> > > the code. On the other hand a "not having to touch Rust code when
> > > changing APIs" guarantee is something which is simply impossible to
> > > have in any sane architecture. So we should figure out under which
> > > circumstances it will be okay for everybody.
>
> Yeah, this sounds fine to me.
>
> > > At least I don't see a way how I can structure Rusticl so that
> > > somebody working on gallium won't have to also deal with rusticl. One
> > > possibility would be to have a libgallium.so file I can link to, but
> > > then it's all about "stable gallium API" vs "not having to touch rust
> > > code" and I hope everybody prefers the second :)
> > >
> >
> > uhm... I meant "stable gallium API" vs "dealing with Rust code on API changes"
>
> Yeah, stable gallium ABI makes as much sense as stable drm (in-kernel)
> ABI (ie. it doesn't).  So even if we tried to make it so that folks
> who haven't used rust much (or at all) yet on the gallium side, we'll
> still have to deal with it on the kernel side.  So let's not design
> our APIs to avoid a learning curve.. lets' just recognize that the
> learning curve will be there.
>
> > > > I do also wonder a bit about code tooling (indexers, etc).. I'm not
> > > > sure what the state of things when it comes to cross c<->rust
> > > > integration.  Ie. it is usually straightforward enough to track down
> > > > all the spots in C code which would be affected by some change.  It
> > > > might be easier to overlook things on the rust side.  On the mesa
> > > > side, pre-merge CI jobs help to catch these issues.  Less sure about
> > > > how to handle that on the kernel side.
> > > >
> > >
> > > At least for Rusticl it's all within meson/ninja. We use bindgen to
> > > generate the bindings automatically so you simply run into compilation
> > > issues. And for the kernel side I think that Linus wanted Rust to be
> > > non optional. If something uses it, you also make sure the Rust side
> > > compiles. And the build system is dynamic enough that you can also
> > > wire up bindgen and make it part of the normal development model.
> > >
> > > In regards to code tooling, for rust you usually rely on
> > > rust-analyzer. I've already figured out with dcbaker on how to
> > > integrate our meson setup with that. Atm I am able to get the full
> > > experience with VScode. Not sure if we also need to figure out how
> > > that can work with e.g. vim and how to deal with C <=> Rust
> > > boundaries.
>
> Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> project that includes the combination of a significant amount of both
> .c and .rs.
>

we have meson generate a rust-project.json file which you can point
rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682

Sadly there is no "real" integration of rust and C code, so all you
get are references to the bindgen generated files, but you can't jump
to the actual C headers from there.

> If bindgen is integrated with the build system, that sounds fine as
> far as helping to not overlook changes that are needed on the .rs side
> of things.. crosvm seems to mainly make bindgen a manual step, but
> then again it is mostly used for external dependencies like the
> interface to virglrenderer
>
> Side note, from my experience with crosvm, please be careful about any
> external rust dependencies that are used.. rust still seems to be
> young enough and evolving quickly enough that thing can quickly turn
> into a big headache
>

That's easy: meson doesn't support external crates, so no external
rust dependencies are used at the moment :)

But yeah, I was planning on using some where it really makes sense,
but we can have this discussion once we get there. So far it all works
without any external dependency besides bindgen.

> BR,
> -R
>
> > > > BR,
> > > > -R
> > > >
> > > > > CCing a bunch of people who think are the most impacted by it either
> > > > > from a reviewer/maintainer or developer perspective. If you think some
> > > > > others should be explicitly aware of this, please point them to this
> > > > > discussion here.
> > > > >
> > > > > Karol
> > > > >
> > > >
> >
>

Re: Rust in our code base

[Jason Ekstrand]

+mesa-dev and my jlekstrand.net e-mail

On Sun, 2022-08-21 at 20:44 +0200, Karol Herbst wrote:
> On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark@gmail.com>
> wrote:
> > 
> > On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst@redhat.com>
> > wrote:
> > > 
> > > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com>
> > > wrote:
> > > > 
> > > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com>
> > > > wrote:
> > > > > 
> > > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst
> > > > > <kherbst@redhat.com> wrote:
> > > > > > 
> > > > > > Hey everybody,
> > > > > > 
> > > > > > so I think it's time to have this discussion for real.
> > > > > > 
> > > > > > I am working on Rusticl
> > > > > > (
> > > > > > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15
> > > > > > 439)
> > > > > > which I would like to merge quite soon.
> > > > > > 
> > > > > > Others might also plan on starting kernel drivers written
> > > > > > in Rust (and
> > > > > > if people feel comfortable to discuss this as well, they
> > > > > > might reply
> > > > > > here)
> > > > > > 
> > > > > > The overall implication of that is: if we are doing this,
> > > > > > people (that
> > > > > > is we) have to accept that touching Rust code will be part
> > > > > > of our
> > > > > > development process. There is no other sane way of doing
> > > > > > it.
> > > > > > 
> > > > > > I am not willing to wrap things in Rusticl so changing
> > > > > > gallium APIs
> > > > > > won't involve touching Rust code, and we also can't expect
> > > > > > people to
> > > > > > design their kernel drivers in weird ways "just because
> > > > > > somebody
> > > > > > doesn't want to deal with Rust"
> > > > > > 
> > > > > > If we are going to do this, we have to do it for real,
> > > > > > which means,
> > > > > > Rust code will call C APIs directly and a change in those
> > > > > > APIs will
> > > > > > also require changes in Rust code making use of those APIs.
> > > > > > 
> > > > > > I am so explicit on this very point, because we had some
> > > > > > discussion on
> > > > > > IRC where this was seen as a no-go at least from some
> > > > > > people, which
> > > > > > makes me think we have to find a mutual agreement on how it
> > > > > > should be
> > > > > > going forward.
> > > > > > 
> > > > > > And I want to be very explicit here about the future of
> > > > > > Rusticl as
> > > > > > well: if the agreement is that people don't want to have to
> > > > > > deal with
> > > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am
> > > > > > not
> > > > > > willing to come up with some trashy external-internal API
> > > > > > just to
> > > > > > maintain Rusticl outside of the mesa git repo.
> > > > > > And doing it on a kernel level is even more of a no-go.
> > > > > > 
> > > > > > So what are we all thinking about Rust in our core repos?
> > > > > 
> > > > > I think there has to be willingness on the part of rust folks
> > > > > to help
> > > > > others who aren't so familiar with rust with these sorts of
> > > > > API
> > > > > changes.  You can't completely impose the burden on others
> > > > > who have
> > > > > never touched rust before.  That said, I expect a lot of API
> > > > > changes
> > > > > over time are simple enough that other devs could figure out
> > > > > the
> > > > > related rust side changes.
> > > > > 
> > > > 
> > > > yeah, I agree here. I wouldn't say it's all the responsibility
> > > > of
> > > > developers changing APIs to also know how to change the code.
> > > > So e.g.
> > > > if an MR fails to compile and it's because of rusticl, I will
> > > > help out
> > > > and do the changes myself if necessary. But long term we have
> > > > to
> > > > accept that API changes also come with the implication of also
> > > > having
> > > > to touch Rust code.
> > > > 
> > > > Short term it might be a learning opportunity for some/most,
> > > > but long
> > > > term it has to be accepted as a part of development to deal
> > > > with Rust.
> > > > 
> > > > > As long as folks who want to start introducing rust in mesa
> > > > > and drm
> > > > > realize they are also signing up to play the role of rust
> > > > > tutor and
> > > > > technical assistance, I don't see a problem.  But if they
> > > > > aren't
> > > > > around and willing to help, I could see this going badly.
> > > > > 
> > > > 
> > > > Yep, I fully agree here. This is also the main reason I am
> > > > bringing
> > > > this up. Nobody should be left alone with having to deal with
> > > > changing
> > > > the code. On the other hand a "not having to touch Rust code
> > > > when
> > > > changing APIs" guarantee is something which is simply
> > > > impossible to
> > > > have in any sane architecture. So we should figure out under
> > > > which
> > > > circumstances it will be okay for everybody.
> > 
> > Yeah, this sounds fine to me.
> > 
> > > > At least I don't see a way how I can structure Rusticl so that
> > > > somebody working on gallium won't have to also deal with
> > > > rusticl. One
> > > > possibility would be to have a libgallium.so file I can link
> > > > to, but
> > > > then it's all about "stable gallium API" vs "not having to
> > > > touch rust
> > > > code" and I hope everybody prefers the second :)
> > > > 
> > > 
> > > uhm... I meant "stable gallium API" vs "dealing with Rust code on
> > > API changes"
> > 
> > Yeah, stable gallium ABI makes as much sense as stable drm (in-
> > kernel)
> > ABI (ie. it doesn't).  So even if we tried to make it so that folks
> > who haven't used rust much (or at all) yet on the gallium side,
> > we'll
> > still have to deal with it on the kernel side.  So let's not design
> > our APIs to avoid a learning curve.. lets' just recognize that the
> > learning curve will be there.

I think this is mostly fine.  If you're making a gallium change that
touches all the drivers, you need cross-project review as well. 
Learning a bit of rust or asking one of the rust-knowledgable people to
update rusticl's wrappers seems reasonable at that point.

Where I would take issue is if various wrappers or inlines are
duplicated in rust such that a change which wouldn't break drivers
because they're all using the wrappers do break rust code.  I know
that, at least at one point, rusticl had some of this which is why I'm
pointing it out specifically.  I've not looked at the branch in a
while, though.

> > > > > I do also wonder a bit about code tooling (indexers, etc)..
> > > > > I'm not
> > > > > sure what the state of things when it comes to cross c<->rust
> > > > > integration.  Ie. it is usually straightforward enough to
> > > > > track down
> > > > > all the spots in C code which would be affected by some
> > > > > change.  It
> > > > > might be easier to overlook things on the rust side.  On the
> > > > > mesa
> > > > > side, pre-merge CI jobs help to catch these issues.  Less
> > > > > sure about
> > > > > how to handle that on the kernel side.
> > > > > 
> > > > 
> > > > At least for Rusticl it's all within meson/ninja. We use
> > > > bindgen to
> > > > generate the bindings automatically so you simply run into
> > > > compilation
> > > > issues. And for the kernel side I think that Linus wanted Rust
> > > > to be
> > > > non optional. If something uses it, you also make sure the Rust
> > > > side
> > > > compiles. And the build system is dynamic enough that you can
> > > > also
> > > > wire up bindgen and make it part of the normal development
> > > > model.
> > > > 
> > > > In regards to code tooling, for rust you usually rely on
> > > > rust-analyzer. I've already figured out with dcbaker on how to
> > > > integrate our meson setup with that. Atm I am able to get the
> > > > full
> > > > experience with VScode. Not sure if we also need to figure out
> > > > how
> > > > that can work with e.g. vim and how to deal with C <=> Rust
> > > > boundaries.
> > 
> > Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> > project that includes the combination of a significant amount of
> > both
> > .c and .rs.
> > 
> 
> we have meson generate a rust-project.json file which you can point
> rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682
> 
> Sadly there is no "real" integration of rust and C code, so all you
> get are references to the bindgen generated files, but you can't jump
> to the actual C headers from there.

I would be very surprised if any sort of C refactoring tool could see
through bindgen now, or in the future.  Seeing across language
boundaries is tractable in C because it's basically a superset of C++.
(Yes, I know it's not really.)  I wouldn't expect a static analyzer to
see across a C/Python or C/Java boundary, so I don't expect it to be
able to see across C/rust.  (Sadly, the boundary is sharp enough that
that's a reasonable comparison.)

> > If bindgen is integrated with the build system, that sounds fine as
> > far as helping to not overlook changes that are needed on the .rs
> > side
> > of things.. crosvm seems to mainly make bindgen a manual step, but
> > then again it is mostly used for external dependencies like the
> > interface to virglrenderer
> > 
> > Side note, from my experience with crosvm, please be careful about
> > any
> > external rust dependencies that are used.. rust still seems to be
> > young enough and evolving quickly enough that thing can quickly
> > turn
> > into a big headache
> > 
> 
> That's easy: meson doesn't support external crates, so no external
> rust dependencies are used at the moment :)
> 
> But yeah, I was planning on using some where it really makes sense,
> but we can have this discussion once we get there. So far it all
> works
> without any external dependency besides bindgen.

I think most of us are agreed here.  Mesa has pretty strict rules about
pulling in piles of external dependencies and we don't want that to
change just because we added a bit of rust.  There may be a crate or
two that we want to pull in in the future but, for now, everything is
core.

One point not mentioned above is Rust compiler versions.  Because Rust
is a rapidly evolving language, new features are being added to the
language and standard library every day.  If we impose the same
requirements on Rust that we're imposing on GCC where we still compile
with a version from 10-15 years ago, that's going to really hamper
development.  We probably don't want to require people to always pull
the very latest compiler from nightly but I do think we want to be able
to bump the compiler version once a year or something.  IDK how distros
and others will feel about that.  Hopefully, they feel about the same
way towards compiler requirement bumps as they feel towards rust
itself.

--Jason

> > BR,
> > -R
> > 
> > > > > BR,
> > > > > -R
> > > > > 
> > > > > > CCing a bunch of people who think are the most impacted by
> > > > > > it either
> > > > > > from a reviewer/maintainer or developer perspective. If you
> > > > > > think some
> > > > > > others should be explicitly aware of this, please point
> > > > > > them to this
> > > > > > discussion here.
> > > > > > 
> > > > > > Karol
> > > > > > 
> > > > > 
> > > 
> > 
> 

Re: Rust in our code base

[Karol Herbst]

On Wed, Aug 24, 2022 at 5:18 PM Jason Ekstrand
<jason.ekstrand@collabora.com> wrote:
>
> +mesa-dev and my jlekstrand.net e-mail
>
> On Sun, 2022-08-21 at 20:44 +0200, Karol Herbst wrote:
> > On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark@gmail.com>
> > wrote:
> > >
> > > On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst@redhat.com>
> > > wrote:
> > > >
> > > > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com>
> > > > wrote:
> > > > >
> > > > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst
> > > > > > <kherbst@redhat.com> wrote:
> > > > > > >
> > > > > > > Hey everybody,
> > > > > > >
> > > > > > > so I think it's time to have this discussion for real.
> > > > > > >
> > > > > > > I am working on Rusticl
> > > > > > > (
> > > > > > > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15
> > > > > > > 439)
> > > > > > > which I would like to merge quite soon.
> > > > > > >
> > > > > > > Others might also plan on starting kernel drivers written
> > > > > > > in Rust (and
> > > > > > > if people feel comfortable to discuss this as well, they
> > > > > > > might reply
> > > > > > > here)
> > > > > > >
> > > > > > > The overall implication of that is: if we are doing this,
> > > > > > > people (that
> > > > > > > is we) have to accept that touching Rust code will be part
> > > > > > > of our
> > > > > > > development process. There is no other sane way of doing
> > > > > > > it.
> > > > > > >
> > > > > > > I am not willing to wrap things in Rusticl so changing
> > > > > > > gallium APIs
> > > > > > > won't involve touching Rust code, and we also can't expect
> > > > > > > people to
> > > > > > > design their kernel drivers in weird ways "just because
> > > > > > > somebody
> > > > > > > doesn't want to deal with Rust"
> > > > > > >
> > > > > > > If we are going to do this, we have to do it for real,
> > > > > > > which means,
> > > > > > > Rust code will call C APIs directly and a change in those
> > > > > > > APIs will
> > > > > > > also require changes in Rust code making use of those APIs.
> > > > > > >
> > > > > > > I am so explicit on this very point, because we had some
> > > > > > > discussion on
> > > > > > > IRC where this was seen as a no-go at least from some
> > > > > > > people, which
> > > > > > > makes me think we have to find a mutual agreement on how it
> > > > > > > should be
> > > > > > > going forward.
> > > > > > >
> > > > > > > And I want to be very explicit here about the future of
> > > > > > > Rusticl as
> > > > > > > well: if the agreement is that people don't want to have to
> > > > > > > deal with
> > > > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am
> > > > > > > not
> > > > > > > willing to come up with some trashy external-internal API
> > > > > > > just to
> > > > > > > maintain Rusticl outside of the mesa git repo.
> > > > > > > And doing it on a kernel level is even more of a no-go.
> > > > > > >
> > > > > > > So what are we all thinking about Rust in our core repos?
> > > > > >
> > > > > > I think there has to be willingness on the part of rust folks
> > > > > > to help
> > > > > > others who aren't so familiar with rust with these sorts of
> > > > > > API
> > > > > > changes.  You can't completely impose the burden on others
> > > > > > who have
> > > > > > never touched rust before.  That said, I expect a lot of API
> > > > > > changes
> > > > > > over time are simple enough that other devs could figure out
> > > > > > the
> > > > > > related rust side changes.
> > > > > >
> > > > >
> > > > > yeah, I agree here. I wouldn't say it's all the responsibility
> > > > > of
> > > > > developers changing APIs to also know how to change the code.
> > > > > So e.g.
> > > > > if an MR fails to compile and it's because of rusticl, I will
> > > > > help out
> > > > > and do the changes myself if necessary. But long term we have
> > > > > to
> > > > > accept that API changes also come with the implication of also
> > > > > having
> > > > > to touch Rust code.
> > > > >
> > > > > Short term it might be a learning opportunity for some/most,
> > > > > but long
> > > > > term it has to be accepted as a part of development to deal
> > > > > with Rust.
> > > > >
> > > > > > As long as folks who want to start introducing rust in mesa
> > > > > > and drm
> > > > > > realize they are also signing up to play the role of rust
> > > > > > tutor and
> > > > > > technical assistance, I don't see a problem.  But if they
> > > > > > aren't
> > > > > > around and willing to help, I could see this going badly.
> > > > > >
> > > > >
> > > > > Yep, I fully agree here. This is also the main reason I am
> > > > > bringing
> > > > > this up. Nobody should be left alone with having to deal with
> > > > > changing
> > > > > the code. On the other hand a "not having to touch Rust code
> > > > > when
> > > > > changing APIs" guarantee is something which is simply
> > > > > impossible to
> > > > > have in any sane architecture. So we should figure out under
> > > > > which
> > > > > circumstances it will be okay for everybody.
> > >
> > > Yeah, this sounds fine to me.
> > >
> > > > > At least I don't see a way how I can structure Rusticl so that
> > > > > somebody working on gallium won't have to also deal with
> > > > > rusticl. One
> > > > > possibility would be to have a libgallium.so file I can link
> > > > > to, but
> > > > > then it's all about "stable gallium API" vs "not having to
> > > > > touch rust
> > > > > code" and I hope everybody prefers the second :)
> > > > >
> > > >
> > > > uhm... I meant "stable gallium API" vs "dealing with Rust code on
> > > > API changes"
> > >
> > > Yeah, stable gallium ABI makes as much sense as stable drm (in-
> > > kernel)
> > > ABI (ie. it doesn't).  So even if we tried to make it so that folks
> > > who haven't used rust much (or at all) yet on the gallium side,
> > > we'll
> > > still have to deal with it on the kernel side.  So let's not design
> > > our APIs to avoid a learning curve.. lets' just recognize that the
> > > learning curve will be there.
>
> I think this is mostly fine.  If you're making a gallium change that
> touches all the drivers, you need cross-project review as well.
> Learning a bit of rust or asking one of the rust-knowledgable people to
> update rusticl's wrappers seems reasonable at that point.
>
> Where I would take issue is if various wrappers or inlines are
> duplicated in rust such that a change which wouldn't break drivers
> because they're all using the wrappers do break rust code.  I know
> that, at least at one point, rusticl had some of this which is why I'm
> pointing it out specifically.  I've not looked at the branch in a
> while, though.
>

The only thing I am aware of is that I have an iterator written in
rust for glsl's list implementation in order to iterate over variables
inside nir shaders. But yeah, that has the potential to blow up if
somebody decides to change how the list works without changing the
definition of the list. But as I am actually running piglit on rusticl
with llvmpipe, this might not be a problem we wouldn't catch.

> > > > > > I do also wonder a bit about code tooling (indexers, etc)..
> > > > > > I'm not
> > > > > > sure what the state of things when it comes to cross c<->rust
> > > > > > integration.  Ie. it is usually straightforward enough to
> > > > > > track down
> > > > > > all the spots in C code which would be affected by some
> > > > > > change.  It
> > > > > > might be easier to overlook things on the rust side.  On the
> > > > > > mesa
> > > > > > side, pre-merge CI jobs help to catch these issues.  Less
> > > > > > sure about
> > > > > > how to handle that on the kernel side.
> > > > > >
> > > > >
> > > > > At least for Rusticl it's all within meson/ninja. We use
> > > > > bindgen to
> > > > > generate the bindings automatically so you simply run into
> > > > > compilation
> > > > > issues. And for the kernel side I think that Linus wanted Rust
> > > > > to be
> > > > > non optional. If something uses it, you also make sure the Rust
> > > > > side
> > > > > compiles. And the build system is dynamic enough that you can
> > > > > also
> > > > > wire up bindgen and make it part of the normal development
> > > > > model.
> > > > >
> > > > > In regards to code tooling, for rust you usually rely on
> > > > > rust-analyzer. I've already figured out with dcbaker on how to
> > > > > integrate our meson setup with that. Atm I am able to get the
> > > > > full
> > > > > experience with VScode. Not sure if we also need to figure out
> > > > > how
> > > > > that can work with e.g. vim and how to deal with C <=> Rust
> > > > > boundaries.
> > >
> > > Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> > > project that includes the combination of a significant amount of
> > > both
> > > .c and .rs.
> > >
> >
> > we have meson generate a rust-project.json file which you can point
> > rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682
> >
> > Sadly there is no "real" integration of rust and C code, so all you
> > get are references to the bindgen generated files, but you can't jump
> > to the actual C headers from there.
>
> I would be very surprised if any sort of C refactoring tool could see
> through bindgen now, or in the future.  Seeing across language
> boundaries is tractable in C because it's basically a superset of C++.
> (Yes, I know it's not really.)  I wouldn't expect a static analyzer to
> see across a C/Python or C/Java boundary, so I don't expect it to be
> able to see across C/rust.  (Sadly, the boundary is sharp enough that
> that's a reasonable comparison.)
>
> > > If bindgen is integrated with the build system, that sounds fine as
> > > far as helping to not overlook changes that are needed on the .rs
> > > side
> > > of things.. crosvm seems to mainly make bindgen a manual step, but
> > > then again it is mostly used for external dependencies like the
> > > interface to virglrenderer
> > >
> > > Side note, from my experience with crosvm, please be careful about
> > > any
> > > external rust dependencies that are used.. rust still seems to be
> > > young enough and evolving quickly enough that thing can quickly
> > > turn
> > > into a big headache
> > >
> >
> > That's easy: meson doesn't support external crates, so no external
> > rust dependencies are used at the moment :)
> >
> > But yeah, I was planning on using some where it really makes sense,
> > but we can have this discussion once we get there. So far it all
> > works
> > without any external dependency besides bindgen.
>
> I think most of us are agreed here.  Mesa has pretty strict rules about
> pulling in piles of external dependencies and we don't want that to
> change just because we added a bit of rust.  There may be a crate or
> two that we want to pull in in the future but, for now, everything is
> core.
>
> One point not mentioned above is Rust compiler versions.  Because Rust
> is a rapidly evolving language, new features are being added to the
> language and standard library every day.  If we impose the same
> requirements on Rust that we're imposing on GCC where we still compile
> with a version from 10-15 years ago, that's going to really hamper
> development.  We probably don't want to require people to always pull
> the very latest compiler from nightly but I do think we want to be able
> to bump the compiler version once a year or something.  IDK how distros
> and others will feel about that.  Hopefully, they feel about the same
> way towards compiler requirement bumps as they feel towards rust
> itself.
>

yeah.. atm I actually require 1.59 which is quite new (Feb 2022). I
think we might want to make sure that at least debians rust compiles
rusticl in the future, but they also seem to be willing to packaging
specific versions if they need it for other things like Firefox. They
have rustc (1.48) and rustc-mozilla (1.51), where the latter is even
available in stretch. So we might actually decide to simply depend on
the same rust version Firefox uses, because that's something
distributions will have to deal with anyway.

> --Jason
>
> > > BR,
> > > -R
> > >
> > > > > > BR,
> > > > > > -R
> > > > > >
> > > > > > > CCing a bunch of people who think are the most impacted by
> > > > > > > it either
> > > > > > > from a reviewer/maintainer or developer perspective. If you
> > > > > > > think some
> > > > > > > others should be explicitly aware of this, please point
> > > > > > > them to this
> > > > > > > discussion here.
> > > > > > >
> > > > > > > Karol
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: Rust in our code base

[Karol Herbst]

will merge Rusticl tomorrow or so unless somebody complains.

On Wed, Aug 24, 2022 at 5:34 PM Karol Herbst <kherbst@redhat.com> wrote:
>
> On Wed, Aug 24, 2022 at 5:18 PM Jason Ekstrand
> <jason.ekstrand@collabora.com> wrote:
> >
> > +mesa-dev and my jlekstrand.net e-mail
> >
> > On Sun, 2022-08-21 at 20:44 +0200, Karol Herbst wrote:
> > > On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark@gmail.com>
> > > wrote:
> > > >
> > > > On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst@redhat.com>
> > > > wrote:
> > > > >
> > > > > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst@redhat.com>
> > > > > wrote:
> > > > > >
> > > > > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst
> > > > > > > <kherbst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > Hey everybody,
> > > > > > > >
> > > > > > > > so I think it's time to have this discussion for real.
> > > > > > > >
> > > > > > > > I am working on Rusticl
> > > > > > > > (
> > > > > > > > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15
> > > > > > > > 439)
> > > > > > > > which I would like to merge quite soon.
> > > > > > > >
> > > > > > > > Others might also plan on starting kernel drivers written
> > > > > > > > in Rust (and
> > > > > > > > if people feel comfortable to discuss this as well, they
> > > > > > > > might reply
> > > > > > > > here)
> > > > > > > >
> > > > > > > > The overall implication of that is: if we are doing this,
> > > > > > > > people (that
> > > > > > > > is we) have to accept that touching Rust code will be part
> > > > > > > > of our
> > > > > > > > development process. There is no other sane way of doing
> > > > > > > > it.
> > > > > > > >
> > > > > > > > I am not willing to wrap things in Rusticl so changing
> > > > > > > > gallium APIs
> > > > > > > > won't involve touching Rust code, and we also can't expect
> > > > > > > > people to
> > > > > > > > design their kernel drivers in weird ways "just because
> > > > > > > > somebody
> > > > > > > > doesn't want to deal with Rust"
> > > > > > > >
> > > > > > > > If we are going to do this, we have to do it for real,
> > > > > > > > which means,
> > > > > > > > Rust code will call C APIs directly and a change in those
> > > > > > > > APIs will
> > > > > > > > also require changes in Rust code making use of those APIs.
> > > > > > > >
> > > > > > > > I am so explicit on this very point, because we had some
> > > > > > > > discussion on
> > > > > > > > IRC where this was seen as a no-go at least from some
> > > > > > > > people, which
> > > > > > > > makes me think we have to find a mutual agreement on how it
> > > > > > > > should be
> > > > > > > > going forward.
> > > > > > > >
> > > > > > > > And I want to be very explicit here about the future of
> > > > > > > > Rusticl as
> > > > > > > > well: if the agreement is that people don't want to have to
> > > > > > > > deal with
> > > > > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am
> > > > > > > > not
> > > > > > > > willing to come up with some trashy external-internal API
> > > > > > > > just to
> > > > > > > > maintain Rusticl outside of the mesa git repo.
> > > > > > > > And doing it on a kernel level is even more of a no-go.
> > > > > > > >
> > > > > > > > So what are we all thinking about Rust in our core repos?
> > > > > > >
> > > > > > > I think there has to be willingness on the part of rust folks
> > > > > > > to help
> > > > > > > others who aren't so familiar with rust with these sorts of
> > > > > > > API
> > > > > > > changes.  You can't completely impose the burden on others
> > > > > > > who have
> > > > > > > never touched rust before.  That said, I expect a lot of API
> > > > > > > changes
> > > > > > > over time are simple enough that other devs could figure out
> > > > > > > the
> > > > > > > related rust side changes.
> > > > > > >
> > > > > >
> > > > > > yeah, I agree here. I wouldn't say it's all the responsibility
> > > > > > of
> > > > > > developers changing APIs to also know how to change the code.
> > > > > > So e.g.
> > > > > > if an MR fails to compile and it's because of rusticl, I will
> > > > > > help out
> > > > > > and do the changes myself if necessary. But long term we have
> > > > > > to
> > > > > > accept that API changes also come with the implication of also
> > > > > > having
> > > > > > to touch Rust code.
> > > > > >
> > > > > > Short term it might be a learning opportunity for some/most,
> > > > > > but long
> > > > > > term it has to be accepted as a part of development to deal
> > > > > > with Rust.
> > > > > >
> > > > > > > As long as folks who want to start introducing rust in mesa
> > > > > > > and drm
> > > > > > > realize they are also signing up to play the role of rust
> > > > > > > tutor and
> > > > > > > technical assistance, I don't see a problem.  But if they
> > > > > > > aren't
> > > > > > > around and willing to help, I could see this going badly.
> > > > > > >
> > > > > >
> > > > > > Yep, I fully agree here. This is also the main reason I am
> > > > > > bringing
> > > > > > this up. Nobody should be left alone with having to deal with
> > > > > > changing
> > > > > > the code. On the other hand a "not having to touch Rust code
> > > > > > when
> > > > > > changing APIs" guarantee is something which is simply
> > > > > > impossible to
> > > > > > have in any sane architecture. So we should figure out under
> > > > > > which
> > > > > > circumstances it will be okay for everybody.
> > > >
> > > > Yeah, this sounds fine to me.
> > > >
> > > > > > At least I don't see a way how I can structure Rusticl so that
> > > > > > somebody working on gallium won't have to also deal with
> > > > > > rusticl. One
> > > > > > possibility would be to have a libgallium.so file I can link
> > > > > > to, but
> > > > > > then it's all about "stable gallium API" vs "not having to
> > > > > > touch rust
> > > > > > code" and I hope everybody prefers the second :)
> > > > > >
> > > > >
> > > > > uhm... I meant "stable gallium API" vs "dealing with Rust code on
> > > > > API changes"
> > > >
> > > > Yeah, stable gallium ABI makes as much sense as stable drm (in-
> > > > kernel)
> > > > ABI (ie. it doesn't).  So even if we tried to make it so that folks
> > > > who haven't used rust much (or at all) yet on the gallium side,
> > > > we'll
> > > > still have to deal with it on the kernel side.  So let's not design
> > > > our APIs to avoid a learning curve.. lets' just recognize that the
> > > > learning curve will be there.
> >
> > I think this is mostly fine.  If you're making a gallium change that
> > touches all the drivers, you need cross-project review as well.
> > Learning a bit of rust or asking one of the rust-knowledgable people to
> > update rusticl's wrappers seems reasonable at that point.
> >
> > Where I would take issue is if various wrappers or inlines are
> > duplicated in rust such that a change which wouldn't break drivers
> > because they're all using the wrappers do break rust code.  I know
> > that, at least at one point, rusticl had some of this which is why I'm
> > pointing it out specifically.  I've not looked at the branch in a
> > while, though.
> >
>
> The only thing I am aware of is that I have an iterator written in
> rust for glsl's list implementation in order to iterate over variables
> inside nir shaders. But yeah, that has the potential to blow up if
> somebody decides to change how the list works without changing the
> definition of the list. But as I am actually running piglit on rusticl
> with llvmpipe, this might not be a problem we wouldn't catch.
>
> > > > > > > I do also wonder a bit about code tooling (indexers, etc)..
> > > > > > > I'm not
> > > > > > > sure what the state of things when it comes to cross c<->rust
> > > > > > > integration.  Ie. it is usually straightforward enough to
> > > > > > > track down
> > > > > > > all the spots in C code which would be affected by some
> > > > > > > change.  It
> > > > > > > might be easier to overlook things on the rust side.  On the
> > > > > > > mesa
> > > > > > > side, pre-merge CI jobs help to catch these issues.  Less
> > > > > > > sure about
> > > > > > > how to handle that on the kernel side.
> > > > > > >
> > > > > >
> > > > > > At least for Rusticl it's all within meson/ninja. We use
> > > > > > bindgen to
> > > > > > generate the bindings automatically so you simply run into
> > > > > > compilation
> > > > > > issues. And for the kernel side I think that Linus wanted Rust
> > > > > > to be
> > > > > > non optional. If something uses it, you also make sure the Rust
> > > > > > side
> > > > > > compiles. And the build system is dynamic enough that you can
> > > > > > also
> > > > > > wire up bindgen and make it part of the normal development
> > > > > > model.
> > > > > >
> > > > > > In regards to code tooling, for rust you usually rely on
> > > > > > rust-analyzer. I've already figured out with dcbaker on how to
> > > > > > integrate our meson setup with that. Atm I am able to get the
> > > > > > full
> > > > > > experience with VScode. Not sure if we also need to figure out
> > > > > > how
> > > > > > that can work with e.g. vim and how to deal with C <=> Rust
> > > > > > boundaries.
> > > >
> > > > Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> > > > project that includes the combination of a significant amount of
> > > > both
> > > > .c and .rs.
> > > >
> > >
> > > we have meson generate a rust-project.json file which you can point
> > > rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682
> > >
> > > Sadly there is no "real" integration of rust and C code, so all you
> > > get are references to the bindgen generated files, but you can't jump
> > > to the actual C headers from there.
> >
> > I would be very surprised if any sort of C refactoring tool could see
> > through bindgen now, or in the future.  Seeing across language
> > boundaries is tractable in C because it's basically a superset of C++.
> > (Yes, I know it's not really.)  I wouldn't expect a static analyzer to
> > see across a C/Python or C/Java boundary, so I don't expect it to be
> > able to see across C/rust.  (Sadly, the boundary is sharp enough that
> > that's a reasonable comparison.)
> >
> > > > If bindgen is integrated with the build system, that sounds fine as
> > > > far as helping to not overlook changes that are needed on the .rs
> > > > side
> > > > of things.. crosvm seems to mainly make bindgen a manual step, but
> > > > then again it is mostly used for external dependencies like the
> > > > interface to virglrenderer
> > > >
> > > > Side note, from my experience with crosvm, please be careful about
> > > > any
> > > > external rust dependencies that are used.. rust still seems to be
> > > > young enough and evolving quickly enough that thing can quickly
> > > > turn
> > > > into a big headache
> > > >
> > >
> > > That's easy: meson doesn't support external crates, so no external
> > > rust dependencies are used at the moment :)
> > >
> > > But yeah, I was planning on using some where it really makes sense,
> > > but we can have this discussion once we get there. So far it all
> > > works
> > > without any external dependency besides bindgen.
> >
> > I think most of us are agreed here.  Mesa has pretty strict rules about
> > pulling in piles of external dependencies and we don't want that to
> > change just because we added a bit of rust.  There may be a crate or
> > two that we want to pull in in the future but, for now, everything is
> > core.
> >
> > One point not mentioned above is Rust compiler versions.  Because Rust
> > is a rapidly evolving language, new features are being added to the
> > language and standard library every day.  If we impose the same
> > requirements on Rust that we're imposing on GCC where we still compile
> > with a version from 10-15 years ago, that's going to really hamper
> > development.  We probably don't want to require people to always pull
> > the very latest compiler from nightly but I do think we want to be able
> > to bump the compiler version once a year or something.  IDK how distros
> > and others will feel about that.  Hopefully, they feel about the same
> > way towards compiler requirement bumps as they feel towards rust
> > itself.
> >
>
> yeah.. atm I actually require 1.59 which is quite new (Feb 2022). I
> think we might want to make sure that at least debians rust compiles
> rusticl in the future, but they also seem to be willing to packaging
> specific versions if they need it for other things like Firefox. They
> have rustc (1.48) and rustc-mozilla (1.51), where the latter is even
> available in stretch. So we might actually decide to simply depend on
> the same rust version Firefox uses, because that's something
> distributions will have to deal with anyway.
>
> > --Jason
> >
> > > > BR,
> > > > -R
> > > >
> > > > > > > BR,
> > > > > > > -R
> > > > > > >
> > > > > > > > CCing a bunch of people who think are the most impacted by
> > > > > > > > it either
> > > > > > > > from a reviewer/maintainer or developer perspective. If you
> > > > > > > > think some
> > > > > > > > others should be explicitly aware of this, please point
> > > > > > > > them to this
> > > > > > > > discussion here.
> > > > > > > >
> > > > > > > > Karol
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >