distribute linux-headers to build out-of-tree Rust modules

distribute linux-headers to build out-of-tree Rust modules

[Andrea Righi]

Hello,

I'm trying to distribute a linux-headers package (deb) that would allow
to build out-of-tree Rust modules.

Everything looks fine, except for the following error:

error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995
  |
  = note: the following crate versions were found:
          crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta

All I need to do is to "refresh" the headers by running a
"make modules_prepare" after installing my linux-headers package on my
system and then everything works perfectly.

Is this because my system, where I try to build the out-of-tree module,
has different toolchain versions (e.g., binutils) than the build host?

Any pointer/documentation to better understand how the target triple
version is determined?

Thanks,
-Andrea

Re: distribute linux-headers to build out-of-tree Rust modules

[Geert Stappers]

On Sat, Mar 11, 2023 at 03:01:00PM +0100, Andrea Righi wrote:
> Hello,
> 
> I'm trying to distribute a linux-headers package (deb) that would allow
> to build out-of-tree Rust modules.
> 
> Everything looks fine, except for the following error:
> 
> error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995

Number ends with 4995

>   |
>   = note: the following crate versions were found:
>           crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta

Number ends with 2825



> All I need to do is to "refresh" the headers by running a
> "make modules_prepare" after installing my linux-headers package on my
> system and then everything works perfectly.

Acknowledge

 
> Is this because my system, where I try to build the out-of-tree module,
> has different toolchain versions (e.g., binutils) than the build host?
> 
> Any pointer/documentation to better understand how the target triple
> version is determined?


I don't know and I could not resist to ask:


  Would it make sense that

     make

  would include

     make modules_prepare


  ?



Groeten
Geert Stappers
-- 
Silence is hard to parse

Re: distribute linux-headers to build out-of-tree Rust modules

[Björn Roy Baron]

On Saturday, March 11th, 2023 at 15:01, Andrea Righi <andrea.righi@canonical.com> wrote:

> Hello,
> 
> I'm trying to distribute a linux-headers package (deb) that would allow
> to build out-of-tree Rust modules.
> 
> Everything looks fine, except for the following error:
> 
> error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995
> |
> = note: the following crate versions were found:
> crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta
> 
> All I need to do is to "refresh" the headers by running a
> "make modules_prepare" after installing my linux-headers package on my
> system and then everything works perfectly.
> 
> Is this because my system, where I try to build the out-of-tree module,
> has different toolchain versions (e.g., binutils) than the build host?
> 
> Any pointer/documentation to better understand how the target triple
> version is determined?

The target id for external targets should be determined based on the filename and the hash of the target specification json file since https://github.com/rust-lang/rust/pull/98225. That it happens to work with "make modules_prepare" may be that it recompiles all rust code for the new target triple and through luck it just so happens to have the exact same abi as the original. There is no guarantee that two compiled rust crates have the same abi unless they have the exact same contents. To debug why the target id changed could you try running the following rust program in both the linux source dir and the /usr/src/linux-headers-6.2.0-16-generic dir?

    use std::hash::{Hash, Hasher};

    fn main() {
        let content = std::fs::read_to_string("rust/target.json").unwrap();
        let mut hasher = std::collections::hash_map::DefaultHasher::new();
        content.hash(&mut hasher);
        println!("{}", hasher.finish());
    }

Cheers,
Bjorn

Re: distribute linux-headers to build out-of-tree Rust modules

[Andrea Righi]

On Sat, Mar 11, 2023 at 04:27:52PM +0000, Björn Roy Baron wrote:
> On Saturday, March 11th, 2023 at 15:01, Andrea Righi <andrea.righi@canonical.com> wrote:
> 
> > Hello,
> > 
> > I'm trying to distribute a linux-headers package (deb) that would allow
> > to build out-of-tree Rust modules.
> > 
> > Everything looks fine, except for the following error:
> > 
> > error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995
> > |
> > = note: the following crate versions were found:
> > crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta
> > 
> > All I need to do is to "refresh" the headers by running a
> > "make modules_prepare" after installing my linux-headers package on my
> > system and then everything works perfectly.
> > 
> > Is this because my system, where I try to build the out-of-tree module,
> > has different toolchain versions (e.g., binutils) than the build host?
> > 
> > Any pointer/documentation to better understand how the target triple
> > version is determined?
> 
> The target id for external targets should be determined based on the filename and the hash of the target specification json file since https://github.com/rust-lang/rust/pull/98225. That it happens to work with "make modules_prepare" may be that it recompiles all rust code for the new target triple and through luck it just so happens to have the exact same abi as the original. There is no guarantee that two compiled rust crates have the same abi unless they have the exact same contents. To debug why the target id changed could you try running the following rust program in both the linux source dir and the /usr/src/linux-headers-6.2.0-16-generic dir?

Ah! This is exactly the problem I'm facing! I don't have
https://github.com/rust-lang/rust/pull/98225, because I'm using
rustc-1.62 and I see that this change has been applied to rustc-1.63.

So, theoretically if I apply this to my rustc everything should work
fine.

> 
>     use std::hash::{Hash, Hasher};
> 
>     fn main() {
>         let content = std::fs::read_to_string("rust/target.json").unwrap();
>         let mut hasher = std::collections::hash_map::DefaultHasher::new();
>         content.hash(&mut hasher);
>         println!("{}", hasher.finish());
>     }

$ pwd
/usr/src/linux-headers-6.2.0-16-generic
$ ./test
5559158138856098584

$ pwd
/home/ubuntu/src/linux
$ ./test
5559158138856098584

^ Same hash, but at this point I think we already know that the problem
is the patch above that I'm missing in rustc.

Thank you so much for your help!

-Andrea

Re: distribute linux-headers to build out-of-tree Rust modules

[Vincenzo Palazzo]

On Sat Mar 11, 2023 at 9:03 PM CET, Andrea Righi wrote:
> On Sat, Mar 11, 2023 at 04:27:52PM +0000, Björn Roy Baron wrote:
> > On Saturday, March 11th, 2023 at 15:01, Andrea Righi <andrea.righi@canonical.com> wrote:
> > 
> > > Hello,
> > > 
> > > I'm trying to distribute a linux-headers package (deb) that would allow
> > > to build out-of-tree Rust modules.
> > > 
> > > Everything looks fine, except for the following error:
> > > 
> > > error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995
> > > |
> > > = note: the following crate versions were found:
> > > crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta
> > > 
> > > All I need to do is to "refresh" the headers by running a
> > > "make modules_prepare" after installing my linux-headers package on my
> > > system and then everything works perfectly.
> > > 
> > > Is this because my system, where I try to build the out-of-tree module,
> > > has different toolchain versions (e.g., binutils) than the build host?
> > > 
> > > Any pointer/documentation to better understand how the target triple
> > > version is determined?
> > 
> > The target id for external targets should be determined based on the filename and the hash of the target specification json file since https://github.com/rust-lang/rust/pull/98225. That it happens to work with "make modules_prepare" may be that it recompiles all rust code for the new target triple and through luck it just so happens to have the exact same abi as the original. There is no guarantee that two compiled rust crates have the same abi unless they have the exact same contents. To debug why the target id changed could you try running the following rust program in both the linux source dir and the /usr/src/linux-headers-6.2.0-16-generic dir?
>

> $ pwd
> /usr/src/linux-headers-6.2.0-16-generic
> $ ./test
> 5559158138856098584
>
> $ pwd
> /home/ubuntu/src/linux
> $ ./test
> 5559158138856098584
>
> ^ Same hash, but at this point I think we already know that the problem
> is the patch above that I'm missing in rustc.

Maybe we should increase the compiler version required by rust in the
kernel?

Cheers!

Vincent.

Re: distribute linux-headers to build out-of-tree Rust modules

[Andrea Righi]

On Sun, Mar 12, 2023 at 02:39:20PM +0100, Vincenzo Palazzo wrote:
> On Sat Mar 11, 2023 at 9:03 PM CET, Andrea Righi wrote:
> > On Sat, Mar 11, 2023 at 04:27:52PM +0000, Björn Roy Baron wrote:
> > > On Saturday, March 11th, 2023 at 15:01, Andrea Righi <andrea.righi@canonical.com> wrote:
> > > 
> > > > Hello,
> > > > 
> > > > I'm trying to distribute a linux-headers package (deb) that would allow
> > > > to build out-of-tree Rust modules.
> > > > 
> > > > Everything looks fine, except for the following error:
> > > > 
> > > > error[E0461]: couldn't find crate `core` with expected target triple target-10951614936105594995
> > > > |
> > > > = note: the following crate versions were found:
> > > > crate `core`, target triple target-3759196539525232825: /usr/src/linux-headers-6.2.0-16-generic/rust/libcore.rmeta
> > > > 
> > > > All I need to do is to "refresh" the headers by running a
> > > > "make modules_prepare" after installing my linux-headers package on my
> > > > system and then everything works perfectly.
> > > > 
> > > > Is this because my system, where I try to build the out-of-tree module,
> > > > has different toolchain versions (e.g., binutils) than the build host?
> > > > 
> > > > Any pointer/documentation to better understand how the target triple
> > > > version is determined?
> > > 
> > > The target id for external targets should be determined based on the filename and the hash of the target specification json file since https://github.com/rust-lang/rust/pull/98225. That it happens to work with "make modules_prepare" may be that it recompiles all rust code for the new target triple and through luck it just so happens to have the exact same abi as the original. There is no guarantee that two compiled rust crates have the same abi unless they have the exact same contents. To debug why the target id changed could you try running the following rust program in both the linux source dir and the /usr/src/linux-headers-6.2.0-16-generic dir?
> >
> 
> > $ pwd
> > /usr/src/linux-headers-6.2.0-16-generic
> > $ ./test
> > 5559158138856098584
> >
> > $ pwd
> > /home/ubuntu/src/linux
> > $ ./test
> > 5559158138856098584
> >
> > ^ Same hash, but at this point I think we already know that the problem
> > is the patch above that I'm missing in rustc.
> 
> Maybe we should increase the compiler version required by rust in the
> kernel?
> 
> Cheers!
> 
> Vincent.

I haven't tried with rustc 1.63 yet, but if anybody else is interested I
just applied these two additional commits on top of rustc 1.62 and now I
can properly build out-of-tree kernel modules:

 https://github.com/rust-lang/rust/commit/072b7db56161ee4c7a4411d8398c90512c153e16
 https://github.com/rust-lang/rust/commit/fc1df4ff170ec137ecab7d7aa7f4dc894bb17449

Tested here:
 https://launchpad.net/~arighi/+archive/ubuntu/linux-test

See also:
 https://bugs.launchpad.net/bugs/2011355

-Andrea

Re: distribute linux-headers to build out-of-tree Rust modules

[Miguel Ojeda]

On Sun, Mar 12, 2023 at 2:45 PM Vincenzo Palazzo
<vincenzopalazzodev@gmail.com> wrote:
>
> Maybe we should increase the compiler version required by rust in the
> kernel?

There will be an upgrade soon. :)

Cheers,
Miguel