Writing the Apple AGX GPU driver in Rust?

Writing the Apple AGX GPU driver in Rust?

[Asahi Lina]

Hi!

I'm starting work on a new kernel GPU driver for the Apple AGX (found in
M1 and M2 family chips). These GPUs run firmware and have fairly complex
shared memory data structures that need to be managed by the host, so
I've been leaning towards Rust for its safety, better metaprogramming,
and general expressiveness. I have a prototype driver written in Python
(running in userspace from a remote host, long story), and having a
higher-level language has been very helpful in reverse engineering the
GPU and prototyping different ideas for how the driver should work.

I realize it's the early days of Rust on Linux and this is an ambitious
challenge, but I'm willing to learn and the driver will take some time
to stabilize to the point of being upstreamable either way (in
particular the UAPI), so writing it in Rust feels like less of a gamble
at this point than it used to be, given that it sounds like Rust will be
merged in the next few kernel cycles at the latest.

I'd like to hear your thoughts about how crazy an idea this is. ^_^

I think the portion of the driver that would most benefit from Rust
would be the firmware interaction parts (managing shared memory data
structures and interacting with them), so I see several possibilities:
the whole driver could be Rust (which would involve writing bindings for
the render portion of the DRM subsystem), or just the bulk of the
firmware interaction logic could be done in Rust, and then the top-level
driver would be written in C and call into the Rust abstraction layer
for the firmware, or something in between.

(For those not familiar with DRM: the display/KMS API is very extensive
and would be quite an ordeal to bind to Rust I imagine, but I think the
render/3D part of the API is significantly simpler, especially the
subset likely to be used by modern drivers. This is a render-only
device, no display, that's a different driver.)

More specifically, there are some challenges that I think would really
benefit from Rust's features. One is that we have to support multiple
firmware versions in one driver, across multiple GPUs (unlike macOS,
which ships a separate driver build for each GPU and only supports a
single, matched firmware in any given macOS version). The firmware ABI
is not stable, so every new firmware needs many scattered changes to the
data structures. Doing this in C without copying and pasting large
amounts of code would require some horrible multiple-compilation #ifdef
hacks, and really isn't pretty.

I was wondering if it could be done better in Rust, so a couple days ago
I tried my hand at writing a proc macro to automatically generate
versioned variants of structures within a single compile, along with
versioned implementations with conditionally-compiled portions. I
prototyped it in userspace, and this is the result (along with some
example structure definitions, so far only with 2 firmware variants and
some dummy code):

https://github.com/asahilina/gpu-rust-playground

Do you think this kind of approach would be fit for the kernel? (Please
be gentle, I have very little Rust experience and this is my first time
writing a proc macro... I did avoid adding any external cargo deps,
since I know that's an issue in the kernel. The actual struct defs were
automatically generated from the existing Python ones, so I know some of
the naming conventions aren't very idiomatic).

The other problem I thought Rust might be able to help with is the
actual management of GPU objects in memory. These objects have two
addresses (a CPU address and a GPU address), and it would be nice if the
type system could help ensure correctness here too. Structures can be
firmware-owned, CPU-owned, or shared, and their state may change at
certain points in time. Perhaps the type system can be coaxed into
extending the usual correctness guarantees of object ownership to this
model, including things like making sure there are no dangling GPU
pointers in GPU structures pointing to dropped data, etc? (using
PhantomData perhaps?). This also ties into the actual memory mapping
(there are different mapping modes depending on whether data is shared
or not). I haven't started exploring this yet, so I'd love to hear any
thoughts about it!

Beyond the GPU driver, the firmware version ABI instability issue also
affects other drivers for these platforms (display/DCP for one, but
quite possibly others too), so that macro scaffolding might be a good
incentive for other drivers for these chips to also partially or fully
move to Rust.

Please let me know your thoughts on this crazy idea! I haven't seen a
lot of ML activity lately, so if there is any better place to discuss
this (Zulip? I saw it's invite-only...) please let me know.

Thanks!!

~~ Lina

Re: Writing the Apple AGX GPU driver in Rust?

[Alex Gaynor]

Hi Lina,

This is very exciting! I think it'll be a big project: I don't know
anything about GPU drivers in particular, but this would be the first
one written in Rust, so in addition to the driver work there'd be the
work of implementing the Rust API for interacting with the kernel.

That said, if you're up for it, I think we'd be very excited to see
how to support it!

In terms of where we are, in addition to this malign list, there is
the Zulip (I can't remember why it's invite only, but we're happy to
invite anyone who is interested!) and the Github issue tracker.

Alex

On Thu, Aug 11, 2022 at 10:29 AM Asahi Lina <lina@asahilina.net> wrote:
>
> Hi!
>
> I'm starting work on a new kernel GPU driver for the Apple AGX (found in
> M1 and M2 family chips). These GPUs run firmware and have fairly complex
> shared memory data structures that need to be managed by the host, so
> I've been leaning towards Rust for its safety, better metaprogramming,
> and general expressiveness. I have a prototype driver written in Python
> (running in userspace from a remote host, long story), and having a
> higher-level language has been very helpful in reverse engineering the
> GPU and prototyping different ideas for how the driver should work.
>
> I realize it's the early days of Rust on Linux and this is an ambitious
> challenge, but I'm willing to learn and the driver will take some time
> to stabilize to the point of being upstreamable either way (in
> particular the UAPI), so writing it in Rust feels like less of a gamble
> at this point than it used to be, given that it sounds like Rust will be
> merged in the next few kernel cycles at the latest.
>
> I'd like to hear your thoughts about how crazy an idea this is. ^_^
>
> I think the portion of the driver that would most benefit from Rust
> would be the firmware interaction parts (managing shared memory data
> structures and interacting with them), so I see several possibilities:
> the whole driver could be Rust (which would involve writing bindings for
> the render portion of the DRM subsystem), or just the bulk of the
> firmware interaction logic could be done in Rust, and then the top-level
> driver would be written in C and call into the Rust abstraction layer
> for the firmware, or something in between.
>
> (For those not familiar with DRM: the display/KMS API is very extensive
> and would be quite an ordeal to bind to Rust I imagine, but I think the
> render/3D part of the API is significantly simpler, especially the
> subset likely to be used by modern drivers. This is a render-only
> device, no display, that's a different driver.)
>
> More specifically, there are some challenges that I think would really
> benefit from Rust's features. One is that we have to support multiple
> firmware versions in one driver, across multiple GPUs (unlike macOS,
> which ships a separate driver build for each GPU and only supports a
> single, matched firmware in any given macOS version). The firmware ABI
> is not stable, so every new firmware needs many scattered changes to the
> data structures. Doing this in C without copying and pasting large
> amounts of code would require some horrible multiple-compilation #ifdef
> hacks, and really isn't pretty.
>
> I was wondering if it could be done better in Rust, so a couple days ago
> I tried my hand at writing a proc macro to automatically generate
> versioned variants of structures within a single compile, along with
> versioned implementations with conditionally-compiled portions. I
> prototyped it in userspace, and this is the result (along with some
> example structure definitions, so far only with 2 firmware variants and
> some dummy code):
>
> https://github.com/asahilina/gpu-rust-playground
>
> Do you think this kind of approach would be fit for the kernel? (Please
> be gentle, I have very little Rust experience and this is my first time
> writing a proc macro... I did avoid adding any external cargo deps,
> since I know that's an issue in the kernel. The actual struct defs were
> automatically generated from the existing Python ones, so I know some of
> the naming conventions aren't very idiomatic).
>
> The other problem I thought Rust might be able to help with is the
> actual management of GPU objects in memory. These objects have two
> addresses (a CPU address and a GPU address), and it would be nice if the
> type system could help ensure correctness here too. Structures can be
> firmware-owned, CPU-owned, or shared, and their state may change at
> certain points in time. Perhaps the type system can be coaxed into
> extending the usual correctness guarantees of object ownership to this
> model, including things like making sure there are no dangling GPU
> pointers in GPU structures pointing to dropped data, etc? (using
> PhantomData perhaps?). This also ties into the actual memory mapping
> (there are different mapping modes depending on whether data is shared
> or not). I haven't started exploring this yet, so I'd love to hear any
> thoughts about it!
>
> Beyond the GPU driver, the firmware version ABI instability issue also
> affects other drivers for these platforms (display/DCP for one, but
> quite possibly others too), so that macro scaffolding might be a good
> incentive for other drivers for these chips to also partially or fully
> move to Rust.
>
> Please let me know your thoughts on this crazy idea! I haven't seen a
> lot of ML activity lately, so if there is any better place to discuss
> this (Zulip? I saw it's invite-only...) please let me know.
>
> Thanks!!
>
> ~~ Lina



-- 
All that is necessary for evil to succeed is for good people to do nothing.

Re: Writing the Apple AGX GPU driver in Rust?

[Miguel Ojeda]

On Thu, Aug 11, 2022 at 4:29 PM Asahi Lina <lina@asahilina.net> wrote:
>
> I'd like to hear your thoughts about how crazy an idea this is. ^_^

It is less crazy than writing it in Python, for sure ;)

But joking aside, using Rust for a particular subsystem is up to the
maintainers of the subsystem. From what I understand (please correct
me if I am wrong!), you are part of the Asahi team, so you are your
own maintainers, thus it is up to you to decide!

Now, of course, reality is more complex than that, and you will
probably want to discuss things with the DRM people to see if they
would be willing to accept and/or maintain their part of the Rust
bindings you may need etc.

In any case, one quick note: so far, we have been doing the "Rust
driver that talks to Rust safe abstractions that for the most part
wrap existing C facilities" approach. While it is of course
technically possible to mix C and Rust in other ways, and it may be
the case that we need to do it as some point, it would be best to
avoid having to expose C APIs from Rust code, to avoid losing the
advantages of the type system. So writing the entire driver in Rust
would be the way to go if I understood you correctly, and would make
also things much easier for others later too, since some safe
abstractions for DRM would be there already etc., which I am sure
people will appreciate a lot in the future!

As for the proc macros: so far we generally try to avoid them as much
as possible, but a lot of details on how we will expand on that are
still in the air (and we have not needed to generate a lot of
hardware-description-based code anyway). Some of that will likely be
discussed in Kangrejos and LPC in September. In any case, when talking
about proc macros, it can be useful to consider whether "raw" code
generation in an independent build step may be easier or not (i.e.
like your previous Python approach, if I understood correctly; but
maybe based on a Rust host program which we have support for) --
especially if that code does not change often.

(I sent you the Zulip invitation soon after seeing your message -- if
you have not received it, let me know!)

Cheers,
Miguel

Re: Writing the Apple AGX GPU driver in Rust?

[Asahi Lina]

On 12/08/2022 03.45, Miguel Ojeda wrote:
> On Thu, Aug 11, 2022 at 4:29 PM Asahi Lina <lina@asahilina.net> wrote:
>>
>> I'd like to hear your thoughts about how crazy an idea this is. ^_^
> 
> It is less crazy than writing it in Python, for sure ;)

Well, considering I'm crazy enough to sign up to reverse engineer
Apple's GPU in the first place... ^^;

> But joking aside, using Rust for a particular subsystem is up to the
> maintainers of the subsystem. From what I understand (please correct
> me if I am wrong!), you are part of the Asahi team, so you are your
> own maintainers, thus it is up to you to decide!

Kind of! marcan & Sven are the maintainers for the soc/apple subtree for
Apple Silicon hardware and that covers a couple drivers and the device
trees directly, but most individual drivers go in other subsystems and
therefore go through those maintainers (the team has a global
MAINTAINERS entry that tries to cover everything, but it doesn't
override individual subsystem maintainers). The GPU driver would fall
under the DRM subsystem (though it would have RTKit as a dependency,
which is in soc/apple).

> Now, of course, reality is more complex than that, and you will
> probably want to discuss things with the DRM people to see if they
> would be willing to accept and/or maintain their part of the Rust
> bindings you may need etc.

Yes, I'll also ask around the DRM folks to see what their take on this
is! I just wanted to get an idea from the Rust side first.

But this also ties into the question of where the Rust boundary is: even
if, for some reason, the DRM folks were against Rust bindings entirely,
I would still want to write parts of my driver in it, even if it means
calling into them from a smaller DRM C driver, since it would make my
life much easier as the author/maintainer of this particular driver!

> In any case, one quick note: so far, we have been doing the "Rust
> driver that talks to Rust safe abstractions that for the most part
> wrap existing C facilities" approach. While it is of course
> technically possible to mix C and Rust in other ways, and it may be
> the case that we need to do it as some point, it would be best to
> avoid having to expose C APIs from Rust code, to avoid losing the
> advantages of the type system. So writing the entire driver in Rust
> would be the way to go if I understood you correctly, and would make
> also things much easier for others later too, since some safe
> abstractions for DRM would be there already etc., which I am sure
> people will appreciate a lot in the future!

Keep in mind that GPU drivers are complex and themselves consist of
multiple semi-independent components (this isn't a GPIO driver!). Safety
isn't all-or-nothing, right? So it might make sense to focus on doing
the subset that benefits most from Rust's features and safety (which in
my mind is all the GPU firmware interaction, including render job
submission/management, high level memory management, etc.) in Rust first
and then consider moving more code to Rust over time, instead of jumping
straight into "let's bind all of DRM into Rust".

In particular, trying to make a comprehensive DRM binding as a
prerequisite is a huge task, especially if you want it to cover KMS too.
DRM isn't a self-contained API surface like most smaller subsystems, but
rather a large collection of modular components that are intended to be
reused by drivers. On top of that, KMS is quite complex from a
programming model standpoint. There's a lot that goes into display drivers!

But when you focus on GPU render drivers, a lot of it can be simplified
down to a small interface that mostly maps to the userspace interface:
alloc/dealloc memory, submit job, wait for job (there's more to it than
that, but that's really the bulk of it).

Later today I'm going to start looking at other DRM drivers to try to
come up with a more concrete plan, but based on what I know so far, the
driver is going to have at least three major parts (which already exist
as such in the Python prototype):

1. GPU MMU (UAT) page table management
2. Object alloc/dealloc, ownership and lifetime management
3. Firmware interaction, including:
  - Initialization
  - Logging
  - Render job submission
  - Fault handling
  - Stats

And I'm thinking (1) will likely reuse a lot of common MMU code and is
fairly self contained, so that could be written in C instead of trying
to bind those APIs; DRM already provides some facilities for (2) (GEM)
which I'd then want to bind to Rust and extend, and build a richer,
safer layer on top of; and (3) I definitely want to do in Rust because
that's where all the real complexity is, but it's almost all bespoke
code (though it'd talk to RTKit, that needs a Rust binding), so the
question of whether the underlying interface to the DRM render subsystem
is a full Rust binding or some calls from a small C driver is
comparatively academic, I think (and could be refactored later).

As another example, we already have a display driver written in C which
uses the DRM KMS interface, and has a similar firmware ABI issue (plus a
bunch of crazy complexity in the ABI that we haven't even gotten to
yet). That whole "talking to firmware" portion would benefit a lot from
being rewritten in Rust using the same proc macro approach I tried (and
significantly improve safety on its own, since we have to assume
firmware is untrusted...), effectively using Rust as a binding language
for the firmware call ABIs, but I'm not sure that rewriting that entire
driver in 100% Rust is practical in the short term, with the complexity
of the DRM KMS API that are involved.

Of course, it would be great to move towards a world with more Rust! I'm
just trying to be practical and not bite off more than I can chew right
off the bat. I do hope that if I can do this in Rust, that might help
the Rust-for-Linux project along the way and maybe encourage other folks
to try writing new drivers to Rust (or rewrite existing ones).

> As for the proc macros: so far we generally try to avoid them as much
> as possible, but a lot of details on how we will expand on that are
> still in the air (and we have not needed to generate a lot of
> hardware-description-based code anyway). Some of that will likely be
> discussed in Kangrejos and LPC in September. In any case, when talking
> about proc macros, it can be useful to consider whether "raw" code
> generation in an independent build step may be easier or not (i.e.
> like your previous Python approach, if I understood correctly; but
> maybe based on a Rust host program which we have support for) --
> especially if that code does not change often.

Sorry, I think my mention of auto-generated structs was a bit confusing.
The goal here is to implement support for a (potentially increasingly
large) set of firmware/hardware combinations without outright copying
and pasting the entire driver for each one. That's done, in both my
Python prototype and the Rust test I linked, by having a facility for
struct variants with different subsets or types of fields, based on the
version. In the Python driver, that's currently determined at import
time (not runtime due to a technicality, but it's fixable), while in the
Rust example I wrote, it gets monomorphized at compile time into all
possible struct variants (and impls of code that manage them) by the
proc macro, so in the end there is ~zero runtime cost other than the
top-level dynamic dispatch to the right version of the code (at the cost
of binary size, but this seems like the right tradeoff for a GPU driver).

The auto-generation is just that I wrote a feature into the Python
prototype to print out its structures (the generic ones, complete with
version conditionals) as Rust syntax, to make it easier to keep them synced.

This could all be achieved with offline code generation, but I don't
think the Linux kernel is very keen on having in-tree
generators/preprocessors for driver code, and I also don't want to do
the insane thing AMD did with their driver and submit 100+ megabytes of
auto-generated headers; that is clearly *not* the way to go, especially
for a reverse engineering project where all of these definitions are
going to keep changing as we find out more about the hardware. It would
make diffs unreviewable, since every time we rename or fix a field we'd
be touching potentially a dozen variants for different
versions/hardware, and we'd still have the problem of how to handle
multiple code variants for multiple structure versions.

That's why I went with the proc macro approach, which lets me write the
code once and keep it looking like normal Rust code with something that
looks similar to #[cfg()] conditionals, except then it magically becomes
many code variants in a single compilation step, without involving the
build system (more than it already is for proc macros). If I were doing
this in C, the only reasonable way I can think of doing it would be
#ifdefs and multiple compilation of the same files with different
defines (or multiple #includes into one compilation unit...).

> (I sent you the Zulip invitation soon after seeing your message -- if
> you have not received it, let me know!)

Thanks, I just joined!

By the way, just a minor thing: I noticed that all the rust-for-linux
code seems to be GPL-2.0. Was this an explicit decision, or just a
default? The DRM core is dual-licensed GPL/MIT, as are some of the major
drivers (i915/radeon/amdgpu at least), and this allows OpenBSD to also
use this code directly. I'd definitely want to use the same approach for
this driver, since I want it to be useful for other kernels too. So I
wonder if it might make sense to license some of the Rust core (in
particular, proc macros and common code written in Rust) more
permissively, so it can be directly pulled in if other projects want to
reuse some of the drivers that depend on it.

I think the subsystem bindings matter less here, since those are
naturally very Linux-specific, except for DRM of course, so this is
mostly about common scaffolding code.

(It's probably a good time to think about this if it hasn't come up in
the past, since changing the license after it's upstream would be a lot
harder!)

~~ Lina

Re: Writing the Apple AGX GPU driver in Rust?

[Miguel Ojeda]

On Fri, Aug 12, 2022 at 6:01 AM Asahi Lina <lina@asahilina.net> wrote:
>
> But this also ties into the question of where the Rust boundary is: even
> if, for some reason, the DRM folks were against Rust bindings entirely,
> I would still want to write parts of my driver in it, even if it means
> calling into them from a smaller DRM C driver, since it would make my
> life much easier as the author/maintainer of this particular driver!

I guess it depends -- for instance, if they would need to understand
what is going on in your other Rust code for some reason (I don't know
if that would be the case here), and they cannot easily follow it
because it is in Rust, then it could still affect them indirectly at
some point, so they could have a reason to oppose it I guess
(personally, I think if everybody is reasonable and willing to learn a
bit, that will not be a problem).

In other words, we would love to see Rust used everywhere (of course)
but we have to be mindful about where/how we introduce it (we are not
even in mainline yet), and be as much in agreement as possible with
maintainers where it may affect them.

> In particular, trying to make a comprehensive DRM binding as a
> prerequisite is a huge task, especially if you want it to cover KMS too.
> DRM isn't a self-contained API surface like most smaller subsystems, but
> rather a large collection of modular components that are intended to be
> reused by drivers. On top of that, KMS is quite complex from a
> programming model standpoint. There's a lot that goes into display drivers!

There is no requirement to create abstractions for everything of a
particular subsystem (sorry if that was unclear), i.e. what I
understood from your previous message is that the subset you need
(e.g. the render portion of DRM) would be feasible to tackle. Is that
correct? (i.e. I am referring to the first approach you mentioned)

> Of course, it would be great to move towards a world with more Rust! I'm
> just trying to be practical and not bite off more than I can chew right
> off the bat. I do hope that if I can do this in Rust, that might help
> the Rust-for-Linux project along the way and maybe encourage other folks
> to try writing new drivers to Rust (or rewrite existing ones).

Yes, definitely! In fact, I am confident a lot of people would love to
see Rust used in such a "big driver", and it would be a key driver to
showcase what is possible (this is what I meant when I said if you
happened to write a subset of the DRM abstractions, then others could
follow your lead later on).

> That's why I went with the proc macro approach, which lets me write the
> code once and keep it looking like normal Rust code with something that
> looks similar to #[cfg()] conditionals, except then it magically becomes
> many code variants in a single compilation step, without involving the
> build system (more than it already is for proc macros). If I were doing
> this in C, the only reasonable way I can think of doing it would be
> #ifdefs and multiple compilation of the same files with different
> defines (or multiple #includes into one compilation unit...).

It is probably a good idea to check with those sending the MiBs of
generated code, in case it is a policy.

> Thanks, I just joined!

My pleasure!

> By the way, just a minor thing: I noticed that all the rust-for-linux
> code seems to be GPL-2.0. Was this an explicit decision, or just a
> default? The DRM core is dual-licensed GPL/MIT, as are some of the major
> drivers (i915/radeon/amdgpu at least), and this allows OpenBSD to also
> use this code directly. I'd definitely want to use the same approach for
> this driver, since I want it to be useful for other kernels too. So I
> wonder if it might make sense to license some of the Rust core (in
> particular, proc macros and common code written in Rust) more
> permissively, so it can be directly pulled in if other projects want to
> reuse some of the drivers that depend on it.
>
> I think the subsystem bindings matter less here, since those are
> naturally very Linux-specific, except for DRM of course, so this is
> mostly about common scaffolding code.
>
> (It's probably a good time to think about this if it hasn't come up in
> the past, since changing the license after it's upstream would be a lot
> harder!)

We discussed it soon after starting and everyone involved agreed to
use GPL-2.0 everywhere. There was also some discussion about a year
ago about reusing a file for GRUB modules written in Rust.

I can raise this point in Kangrejos/LPC to see what is everyone's
position on it.

Cheers,
Miguel

Re: Writing the Apple AGX GPU driver in Rust?

[Asahi Lina]

(Resend, sorry, forgot to Cc: the list!)

On 13/08/2022 02.48, Miguel Ojeda wrote:
> I guess it depends -- for instance, if they would need to understand
> what is going on in your other Rust code for some reason (I don't know
> if that would be the case here), and they cannot easily follow it
> because it is in Rust, then it could still affect them indirectly at
> some point, so they could have a reason to oppose it I guess
> (personally, I think if everybody is reasonable and willing to learn a
> bit, that will not be a problem).
> 
> In other words, we would love to see Rust used everywhere (of course)
> but we have to be mindful about where/how we introduce it (we are not
> even in mainline yet), and be as much in agreement as possible with
> maintainers where it may affect them.

Of course! What I meant is that if, for example, a subsystem maintainer
is concerned about having to maintain the Rust bindings when the C API
changes, having only effectively internal parts of the driver be written
in Rust would insulate them from that issue (for the most part). I
didn't mean I'd want to push it through without their agreement!

I've heard good things about the DRM folks either way, so fingers crossed ^^

How are Rust bindings usually maintained? For something as complex as
DRM, my instinct would be to just let it be maintained by the driver
authors who use that binding (at least initially), adding/changing
things as necessary (and the DRM folks for changes affecting the C core
that need binding updates). But I guess that's also something to discuss
with the DRM team.

>> In particular, trying to make a comprehensive DRM binding as a
>> prerequisite is a huge task, especially if you want it to cover KMS too.
>> DRM isn't a self-contained API surface like most smaller subsystems, but
>> rather a large collection of modular components that are intended to be
>> reused by drivers. On top of that, KMS is quite complex from a
>> programming model standpoint. There's a lot that goes into display drivers!
> 
> There is no requirement to create abstractions for everything of a
> particular subsystem (sorry if that was unclear), i.e. what I
> understood from your previous message is that the subset you need
> (e.g. the render portion of DRM) would be feasible to tackle. Is that
> correct? (i.e. I am referring to the first approach you mentioned)

That's right, though even that portion is really a significant number of
tools/modules, many of them optional. I started looking at that
yesterday, using the Panfrost driver as an example. I also would want to
reuse some Panfrost code where it makes sense, so that also ties into
the question of whether the driver would be 100% rust or not... Here's
what I found:

The MMU code (~750 lines) is largely straightfoward and uses the
io_pgtable common pagetable code. I think I'd end up with mostly the
same approach for AGX, to the point where I can probably copy and paste
the Panfrost code as an example and just adapt it. So, it might make
sense to just leave this bit in C (at least initially), and that saves
us from having to bind io_pgtable and drm_mm. There is some very minor
firmware interaction here, but it is very self contained and doesn't
seem to change from version to version.

The GEM userspace object management glue (~290 lines) is also largely
identical to Panfrost (this doesn't really change much among unified
memory GPUs). That could also be largely copied and adapted, and keeps
most of the drm_gem usage in C land too, so that binding could be avoided.

The driver entrypoint (platform_device, module, etc.) bit is largely
trivial, and it's fairly academic whether that is written in Rust or C.
AGX does its own power management internally, so we don't need to
interact with anything other than basic runtime_pm to suspend the
firmware coprocessor (if desired; it might not actually save much
power). No clocks/resets/anything like that.

The core drm_driver (~690 lines) that includes the callbacks for
userspace requests could be done in Rust by binding that API, but it
could also make sense to leave in C if enough of it calls into other C
bits (the MMU/GEM stuff) that it makes sense to avoid the language
crossing there. If this is C, then possibly only the job submission call
(which is where ~all the driver complexity is) and other auxiliary calls
we define for our driver (tile buffer management, performance counters,
etc.) would need to call into Rust.

The job management is done using drm_sched. That part definitely needs
to be written from scratch for AGX, probably in Rust. I need to talk to
the DRM folks to decide if we need to use drm_sched or we can get away
without it, since the AGX firmware does its own scheduling (which may or
may not be enough for what Linux needs). So if that gets used, drm_sched
probably needs a Rust binding.

Fences and buffer locking use the dma_fence and dma_resv APIs, which
would also need Rust bindings unless that logic lives in a C shim
handling the drm_driver.

The coprocessor interface uses the shared RTKit code that the Asahi team
already wrote. If the driver core is C, that could just be initialized
from C and then Rust would only need the send/receive message functions.
If the driver core is in Rust, that would need a full Rust binding.
Honestly, since I'm very familiar with this bit and it's a simple API,
I'd probably end up binding it fully to Rust regardless, for practice if
nothing else...

The firmware structures need to be allocated from shared memory pools
and mapped using the MMU code I mentioned earlier. I think this depends
partially on how we end up handling this in the Rust side, but if this
were a C driver, the dma_pool API would be the obvious answer. Is any of
that bound to Rust yet? If not, that would be something worth doing,
since it's something a lot of drivers are going to need, of all kinds...

And finally, the real bulk of the driver is going to be interacting with
firmware, which mostly means allocating and mapping structures in shared
GPU memory and interacting with them. Where exactly the boundary is for
all of this depends on how much of what I mentioned earlier is C vs
Rust, but either way, there's going to be a lot of code here, and this
is the part I really want to do in safe Rust since it's where all the
complexity is.

>> Of course, it would be great to move towards a world with more Rust! I'm
>> just trying to be practical and not bite off more than I can chew right
>> off the bat. I do hope that if I can do this in Rust, that might help
>> the Rust-for-Linux project along the way and maybe encourage other folks
>> to try writing new drivers to Rust (or rewrite existing ones).
> 
> Yes, definitely! In fact, I am confident a lot of people would love to
> see Rust used in such a "big driver", and it would be a key driver to
> showcase what is possible (this is what I meant when I said if you
> happened to write a subset of the DRM abstractions, then others could
> follow your lead later on).

I have a question here: what's your view on directly using bound C APIs
from (unsafe) Rust, instead of trying to build a fully safe abstraction?

Of course safe abstractions are better, but trying to design safe
abstractions for everything from the get go might be a pretty big hill
to climb, especially since I'm actually quite new to Rust... so I'm
wondering whether you think it would be okay to start out by mixing safe
abstractions with direct C API usage, and then over time move towards
making more of it safe. I will probably need help from more experienced
Rust folks for that... but in the meantime I could get ahead in writing
all the firmware management code (which should be almost entirely safe
Rust - at least I want that layer to be abstracted out properly) and the
bits that hit unsafe C APIs could be refactored later.

More specifically, I'd like to ask for some help coming up with a Rust
GPU object management approach / API that is safe and works well. I
think if I can do that "right", that will be the most important safety
backbone for the whole driver, and it doesn't have much of a C API
surface behind it (just alloc/dealloc/map/unmap). If it works well,
whether the rest of the DRM/etc APIs are safely wrapped or not initially
is much less of an issue, and that could be refactored later on quite
easily.

> 
>> That's why I went with the proc macro approach, which lets me write the
>> code once and keep it looking like normal Rust code with something that
>> looks similar to #[cfg()] conditionals, except then it magically becomes
>> many code variants in a single compilation step, without involving the
>> build system (more than it already is for proc macros). If I were doing
>> this in C, the only reasonable way I can think of doing it would be
>> #ifdefs and multiple compilation of the same files with different
>> defines (or multiple #includes into one compilation unit...).
> 
> It is probably a good idea to check with those sending the MiBs of
> generated code, in case it is a policy.

I'm pretty sure that's not policy, more like a compromise due to the way
AMD internally handles this... that entire driver was originally quite
foreign code (it was written as OS-agnostic) and I think it was quite
painful to review, if I remember correctly...

Ultimately, the way different drivers approach these problems is up to
driver developers within reason, I think, as long as it doesn't break
with kernel build tooling/code standards. That's why I'm asking here
about the proc macro, because that falls squarely in the Rust-for-linux
camp!

> We discussed it soon after starting and everyone involved agreed to
> use GPL-2.0 everywhere. There was also some discussion about a year
> ago about reusing a file for GRUB modules written in Rust.
> 
> I can raise this point in Kangrejos/LPC to see what is everyone's
> position on it.

It would be nice to see what others think, especially for the DRM case
since that actively shares code with OpenBSD/FreeBSD. I assume at least
making the DRM bindings (and the driver, and my proc macro)
dual-licensed wouldn't be an issue? Of course it's an open question
whether the BSDs would want to add Rust support anyway, it just feels a
bit silly for them to have to rewrite the other macros/helper code as a
prerequisite, because it's a GPLed dependency...

Keep in mind that this is different from supporting proprietary Rust
Linux modules! I don't know what the Rust-for-Linux team's position
there is, but I personally think proprietary modules are not great for
the ecosystem, so I don't really care about that.

On the other hand, if the people trying to get Windows to run on these
machines want to try to port my DRM driver to Windows one day, that
would be pretty funny and I'd encourage it! I hear there's Rust for the
Windows kernel these days...

Cheers,
~~ Lina

Re: Writing the Apple AGX GPU driver in Rust?

[Miguel Ojeda]

On Sat, Aug 13, 2022 at 8:17 AM Asahi Lina <lina@asahilina.net> wrote:
>
> Of course! What I meant is that if, for example, a subsystem maintainer
> is concerned about having to maintain the Rust bindings when the C API
> changes, having only effectively internal parts of the driver be written
> in Rust would insulate them from that issue (for the most part). I

Yeah, it would insulate them, and it could be a last-resort solution,
but we want to encourage the safe abstractions to get written,
supported and reused for many drivers, as well as getting maintainers
on board.

For "core" subsystems (in the sense of APIs that are used
independently by drivers outside that subsystem), I think abstractions
should be allowed to be written even if the subsystem does not want to
maintain them today, as long as somebody is responsible for them.

In the end I think it will be up to high-level maintainers to decide
how they would like the kernel to approach this.

> didn't mean I'd want to push it through without their agreement!

Sorry, I didn't mean you would want to!

> I've heard good things about the DRM folks either way, so fingers crossed ^^
>
> How are Rust bindings usually maintained? For something as complex as
> DRM, my instinct would be to just let it be maintained by the driver
> authors who use that binding (at least initially), adding/changing
> things as necessary (and the DRM folks for changes affecting the C core
> that need binding updates). But I guess that's also something to discuss
> with the DRM team.

So far, for the example drivers etc., we have written the abstractions
that were needed ourselves, but they are not in mainline, so "nobody"
officially maintains them yet.

The goal is to get the respective maintainers to maintain their own
abstractions as soon as possible, since they are the experts and the
ones that know what is best for their subsystem, future changes etc.,
and it is probably the only way that actually scales long-term; but in
the beginning and until things settle, it is indeed a bit of a
chicken-and-egg problem (see also the point above about the "core"
subsystems).

Starting with v9 we sent a very small subset of all of the
abstractions we have, so it will be easier to submit the rest piece by
piece to the respective owners.

(By the way, in case it is confusing, I avoided the term "bindings"
here because we typically refer with it to the raw, automatically
generated ones).

> That's right, though even that portion is really a significant number of
> tools/modules, many of them optional. I started looking at that

If some tools/modules are optional, and you don't need them for your
driver, I think it is fair to only cover what you need (even down to
the function), especially nowadays that we are starting.

> were a C driver, the dma_pool API would be the obvious answer. Is any of
> that bound to Rust yet? If not, that would be something worth doing,
> since it's something a lot of drivers are going to need, of all kinds...

Wedson wrote a NVMe driver prototype which included a draft DMA module
(e.g. with a `Pool` type to wrap `dma_pool`). Andreas took that code
and is working on the NVMe driver now. FUJITA Tomonori has also been
working on another NVMe driver, too, and wants to submit DMA
abstractions too from what I understand.

> I have a question here: what's your view on directly using bound C APIs
> from (unsafe) Rust, instead of trying to build a fully safe abstraction?
>
> Of course safe abstractions are better, but trying to design safe
> abstractions for everything from the get go might be a pretty big hill
> to climb, especially since I'm actually quite new to Rust... so I'm
> wondering whether you think it would be okay to start out by mixing safe
> abstractions with direct C API usage, and then over time move towards
> making more of it safe. I will probably need help from more experienced
> Rust folks for that... but in the meantime I could get ahead in writing
> all the firmware management code (which should be almost entirely safe
> Rust - at least I want that layer to be abstracted out properly) and the
> bits that hit unsafe C APIs could be refactored later.

We are avoiding direct C calls from Rust drivers (and our goal is to
forbid them) -- but it is possible to create unsafe abstractions.

That way, you don't need to figure everything out in the beginning and
how safe the abstraction could be, but your driver gets already a more
Rust-like API, the advantages of the Rust type system, preconditions
get documented, etc.

Then later you can refine the abstractions to reduce the amount of
unsafe APIs as much as possible.

> More specifically, I'd like to ask for some help coming up with a Rust
> GPU object management approach / API that is safe and works well. I
> think if I can do that "right", that will be the most important safety
> backbone for the whole driver, and it doesn't have much of a C API
> surface behind it (just alloc/dealloc/map/unmap). If it works well,
> whether the rest of the DRM/etc APIs are safely wrapped or not initially
> is much less of an issue, and that could be refactored later on quite
> easily.

There are some people with quite some experience already coming up
with non-trivial abstractions (Wedson, Boqun, Andreas...), so I think
you will not be alone. Björn and Gary are our main in-house Rust
experts, and can help with out-of-the-box approaches too :)

> Ultimately, the way different drivers approach these problems is up to
> driver developers within reason, I think, as long as it doesn't break
> with kernel build tooling/code standards. That's why I'm asking here
> about the proc macro, because that falls squarely in the Rust-for-linux
> camp!

We try to avoid proc macros where possible, in order not to make
things too implicit or "magical". But sometimes it is the right
solution/balance -- this may be one of those cases. We just keep the
bar high for new ones (e.g. Gary led the `#[vtable]` push).

> It would be nice to see what others think, especially for the DRM case
> since that actively shares code with OpenBSD/FreeBSD. I assume at least
> making the DRM bindings (and the driver, and my proc macro)
> dual-licensed wouldn't be an issue? Of course it's an open question

From what I have been told, dual-licensing in the kernel quickly makes
things trickier. Whether your case would be an issue or not depends on
the tree that will pick up your patches, whether they accept
dual-licensed stuff, whether the thing being abstracted is GPL-only or
not, etc.

Cheers,
Miguel

Re: Writing the Apple AGX GPU driver in Rust?

[Asahi Lina]

On 13/08/2022 22.05, Miguel Ojeda wrote:
> For "core" subsystems (in the sense of APIs that are used
> independently by drivers outside that subsystem), I think abstractions
> should be allowed to be written even if the subsystem does not want to
> maintain them today, as long as somebody is responsible for them.

I think the main issue there is that refactoring that changes the C API
also has to change the abstractions, so if the people making those
changes don't want to touch Rust... how would that work out? Refactoring
patches couldn't be submitted if they break the Rust side.

(Of course, ideally people aren't scared of Rust and simply make the
changes themselves!)

> The goal is to get the respective maintainers to maintain their own
> abstractions as soon as possible, since they are the experts and the
> ones that know what is best for their subsystem, future changes etc.,
> and it is probably the only way that actually scales long-term; but in
> the beginning and until things settle, it is indeed a bit of a
> chicken-and-egg problem (see also the point above about the "core"
> subsystems).

Right, that makes sense.

> If some tools/modules are optional, and you don't need them for your
> driver, I think it is fair to only cover what you need (even down to
> the function), especially nowadays that we are starting.

That's good to know! A lot of the APIs have pretty long function lists,
of which you only end up using a subset for any given driver.

> Wedson wrote a NVMe driver prototype which included a draft DMA module
> (e.g. with a `Pool` type to wrap `dma_pool`). Andreas took that code
> and is working on the NVMe driver now. FUJITA Tomonori has also been
> working on another NVMe driver, too, and wants to submit DMA
> abstractions too from what I understand.

Cool! Do you know where these branches are being developed? I didn't see
them in the main Rust-for-Linux tree yet.

> We are avoiding direct C calls from Rust drivers (and our goal is to
> forbid them) -- but it is possible to create unsafe abstractions.
> 
> That way, you don't need to figure everything out in the beginning and
> how safe the abstraction could be, but your driver gets already a more
> Rust-like API, the advantages of the Rust type system, preconditions
> get documented, etc.
> 
> Then later you can refine the abstractions to reduce the amount of
> unsafe APIs as much as possible.

By unsafe APIs you mean abstractions to make the API more idiomatic
Rust, without necessarily enforcing all invariants (and therefore marked
unsafe), right? I think I can work with that, if so! It would definitely
be very useful to start to document the preconditions as part of it (the
C kernel documentation is often quite terrible at this...).

>> More specifically, I'd like to ask for some help coming up with a Rust
>> GPU object management approach / API that is safe and works well. I
>> think if I can do that "right", that will be the most important safety
>> backbone for the whole driver, and it doesn't have much of a C API
>> surface behind it (just alloc/dealloc/map/unmap). If it works well,
>> whether the rest of the DRM/etc APIs are safely wrapped or not initially
>> is much less of an issue, and that could be refactored later on quite
>> easily.
> 
> There are some people with quite some experience already coming up
> with non-trivial abstractions (Wedson, Boqun, Andreas...), so I think
> you will not be alone. Björn and Gary are our main in-house Rust
> experts, and can help with out-of-the-box approaches too :)

That's glad to hear! I see Björn already replied on Zulip, so hopefully
they can help figure it out ^^

> We try to avoid proc macros where possible, in order not to make
> things too implicit or "magical". But sometimes it is the right
> solution/balance -- this may be one of those cases. We just keep the
> bar high for new ones (e.g. Gary led the `#[vtable]` push).

Fair enough! I couldn't come up with any reasonable way of doing this
without proc macros (even with a lot of brainstorming with folks more
experienced with Rust the other day)...

>> It would be nice to see what others think, especially for the DRM case
>> since that actively shares code with OpenBSD/FreeBSD. I assume at least
>> making the DRM bindings (and the driver, and my proc macro)
>> dual-licensed wouldn't be an issue? Of course it's an open question
> 
> From what I have been told, dual-licensing in the kernel quickly makes
> things trickier. Whether your case would be an issue or not depends on
> the tree that will pick up your patches, whether they accept
> dual-licensed stuff, whether the thing being abstracted is GPL-only or
> not, etc.

That's odd, I've never heard of dual-licensing being an issue in the
kernel. Lots of drivers are dual licensed, as is the DRM core as I
mentioned... I did a quick search for SPDX lines and about 10% of them
are some kind of dual license in the entire kernel.

The Asahi Linux project encourages dual-licensing so other OSes and
projects can also use the code if they want, and I don't think anyone
has mentioned that being an issue for the things that have been
submitted so far... As far as I know, you can license drivers however
you want (as long as one license is GPL or GPL-compatible), and then if
someone wants to use it under the non-GPL license in another context,
it's up to them to replace any usage of GPL-only APIs with
reimplementations (though usually you'd have to change all that code
interfacing with the rest of the OS anyway, so it's often a moot issue).

Of all drivers, this one is probably the most important one that other
OSes would want to port directly... and of course, DRM accepts
dual-license drivers, and the largest ones already are. That doesn't
mean all the abstractions have to be dual-license, but I think the
abstractions for code that itself is dual-license should be, like much
of DRM... otherwise merely using Rust ends up having more restricting
licensing than not, which doesn't feel right?

For example, this fairly large subset of Linux DRM is used directly by
FreeBSD under a permissive license:

https://github.com/freebsd/drm-kmod/tree/master/drivers/gpu/drm

Cheers,
~~ Lina

Re: Writing the Apple AGX GPU driver in Rust?

[Greg KH]

On Sun, Aug 14, 2022 at 12:06:08AM +0900, Asahi Lina wrote:
> > From what I have been told, dual-licensing in the kernel quickly makes
> > things trickier. Whether your case would be an issue or not depends on
> > the tree that will pick up your patches, whether they accept
> > dual-licensed stuff, whether the thing being abstracted is GPL-only or
> > not, etc.
> 
> That's odd, I've never heard of dual-licensing being an issue in the
> kernel. Lots of drivers are dual licensed, as is the DRM core as I
> mentioned... I did a quick search for SPDX lines and about 10% of them
> are some kind of dual license in the entire kernel.

Most of those are legacy drivers and/or for the few Linux subsystems
that do allow this, like DRM.

For some subsystems that did dual-licensing, like IB, they regret doing
that as it has caused nothing but pain and suffering over time and there
was no benefit in the end at all..  Other subsystems can handle this
properly but it is a lot of extra work that usually isn't worth it
(again, DRM is the exception, they know what they are doing here.)

For new stuff, in subsystems that do not do this, it gets really really
tricky very quickly.  Talk to licensing lawyers about all of the issues
involved if you want the gory details.  Because of this, I require new
files that are added to subsystems that I maintain to have a lawyer from
the company involved to sign-off on the patch to verify that they
understand the issues and are ok with this.

One of the biggest issue is having dual-licensed code use code from
gpl-only subsystems.  What happens then is ripe for lots and lots of
arguing by lots of lawyers.  For core interface code, like the Rust core
is, avoiding all of that nightmare is a good idea by just keeping it
simple as a single license (GPL) to remove any arguing at all.

So be careful here.  If you are adding a new driver, sticking with what
the existing subsystem does is good (and probably required as you did
copy code from those existing drivers when writing your driver).  But
for new stuff and new subsystems, be very careful and consult a lawyer
if you have problems.

And as many people have stated many times in the past, if you can't
consult a lawyer, then just stick with the single license as it's not
worth the trouble or problems of trying to do something out-of-the
ordinary.

But again, I'm not a lawyer, talk to one if you want the full gory
details.

thanks,

greg k-h

Re: Writing the Apple AGX GPU driver in Rust?

[Asahi Lina]

On 14/08/2022 00.30, Greg KH wrote:
> On Sun, Aug 14, 2022 at 12:06:08AM +0900, Asahi Lina wrote:
>>> From what I have been told, dual-licensing in the kernel quickly makes
>>> things trickier. Whether your case would be an issue or not depends on
>>> the tree that will pick up your patches, whether they accept
>>> dual-licensed stuff, whether the thing being abstracted is GPL-only or
>>> not, etc.
>>
>> That's odd, I've never heard of dual-licensing being an issue in the
>> kernel. Lots of drivers are dual licensed, as is the DRM core as I
>> mentioned... I did a quick search for SPDX lines and about 10% of them
>> are some kind of dual license in the entire kernel.
> 
> Most of those are legacy drivers and/or for the few Linux subsystems
> that do allow this, like DRM.

I see a lot of dual-licensed drivers in drivers/net/ and sound/soc/...
and nobody mentioned the license being an issue with drivers/soc/apple/
either (the Asahi drivers there are dual-licensed).

Of course, if a driver is based on another driver, then it should follow
the same license (or at least those portions should - if the driver has
several large and independent parts in different files, they could have
different licenses).

> For some subsystems that did dual-licensing, like IB, they regret doing
> that as it has caused nothing but pain and suffering over time and there
> was no benefit in the end at all..  Other subsystems can handle this
> properly but it is a lot of extra work that usually isn't worth it
> (again, DRM is the exception, they know what they are doing here.)

Right, but then the DRM bindings should also be dual-licensed, right?
I'm writing a DRM driver here, that's why I'm bringing this up...

> One of the biggest issue is having dual-licensed code use code from
> gpl-only subsystems.  What happens then is ripe for lots and lots of
> arguing by lots of lawyers.  For core interface code, like the Rust core
> is, avoiding all of that nightmare is a good idea by just keeping it
> simple as a single license (GPL) to remove any arguing at all.

I'm talking about bespoke scaffolding, especially proc macros. Those by
definition do not link with the rest of the kernel, since they run
inside the compiler. I'm not talking about kernel core abstractions...
those should be GPL-only if what they abstract over is GPL-only.

Really, I'm just saying I think it makes sense for Rust abstractions to
follow the license of whatever they abstract, and it might make sense
for shared, scaffolding-type stuff like proc macros to be more
permissive to encourage other OS projects to also adopt Rust without
having to reinvent those wheels.

~~ Lina

Re: Writing the Apple AGX GPU driver in Rust?

[Miguel Ojeda]

On Sat, Aug 13, 2022 at 5:06 PM Asahi Lina <lina@asahilina.net> wrote:
>
> I think the main issue there is that refactoring that changes the C API
> also has to change the abstractions, so if the people making those
> changes don't want to touch Rust... how would that work out? Refactoring
> patches couldn't be submitted if they break the Rust side.
>
> (Of course, ideally people aren't scared of Rust and simply make the
> changes themselves!)

That is the conundrum, indeed. And yeah, I would think if people is
reasonable, it will eventually work out.

Given Rust is supposed to be introduced in an experimental capacity,
changing the Rust side shouldn't be seen as "scary" (initially at
least), in that if somebody breaks it by mistake, it can always be
fixed, even if takes a bit longer. I think that would encourage others
to do the changes by themselves.

An orthogonal idea could be relaxing the "do not break callers" rule a
bit for Rust, so that it is possible to temporarily break the Rust
side on purpose if there is a good reason to do so (e.g. if a change
really cannot wait for the Rust side to be fixed), especially early in
a cycle.

But yeah, this topic is still ongoing.

> That's good to know! A lot of the APIs have pretty long function lists,
> of which you only end up using a subset for any given driver.

Yeah, precisely -- I hope that makes it way more enticing for you!
(see also below on the unsafe abstractions).

> Cool! Do you know where these branches are being developed? I didn't see
> them in the main Rust-for-Linux tree yet.

I am aware where some of it is, but please contact them directly if
possible (I can help with that) -- they can give you their latest
updates.

> By unsafe APIs you mean abstractions to make the API more idiomatic
> Rust, without necessarily enforcing all invariants (and therefore marked
> unsafe), right? I think I can work with that, if so! It would definitely
> be very useful to start to document the preconditions as part of it (the
> C kernel documentation is often quite terrible at this...).

Exactly, I am referring to having `unsafe fn` functions and methods
which may just require the caller to "do the right thing" (also unsafe
traits etc.). The invariants that you require would still be enforced
-- by the caller.

And, precisely, documenting the preconditions for those helps a lot to
come up later with safe abstractions. In fact, at times you will
likely realize how you will want the safe abstraction to be while
doing it.

After all, making safe abstractions is all about reducing as much as
possible the amount of times we require something from users. And to
do that, one needs to know what we are actually requiring from clients
to begin with.

> That's odd, I've never heard of dual-licensing being an issue in the
> kernel. Lots of drivers are dual licensed, as is the DRM core as I

I see Greg has got involved with details about this (thanks!), so I
defer to him...

Cheers,
Miguel