Re: [RFC v3 13/32] rust: use vendored-sources

["Marc-André Lureau" <marcandre.lureau@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3923 bytes --]

Hi

On Thu, Sep 9, 2021 at 8:04 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
> >
> > "cargo vendor" helps gathering and updating the dependencies.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  configure                 | 8 ++++++++
> >  meson.build               | 7 ++++++-
> >  .cargo/config.toml.in     | 5 +++++
> >  .cargo/meson.build        | 5 +++++
> >  .gitmodules               | 4 ++++
> >  rust/vendored             | 1 +
> >  scripts/archive-source.sh | 2 +-
> >  scripts/cargo_wrapper.py  | 1 +
> >  8 files changed, 31 insertions(+), 2 deletions(-)
> >  create mode 100644 .cargo/config.toml.in
> >  create mode 100644 .cargo/meson.build
> >  create mode 160000 rust/vendored
>
> So, this is a lot of extra code in a submodule. Historically we've
> found that submodules are a colossal pain, and so I think we should
> think about whether we really want to have all our rust dependencies
> in a submodule forever.
>
> I am definitely only at the beginner stage with Rust, but I think
> we should have a discussion about what the different alternative
> options are here, and what we want to achieve, so that we know
> why we're doing this and what we're gaining from the pain...
>
> For instance, could we instead commit Cargo.lock in git and
> use that to nail down specific versions of the dependencies ?
>
>
Yes, that's the main reason this file exists I think.

FWIW, the "why submodules" for the C dependencies we ship
> like that is basically
>  * C doesn't have a package manager, so if we need a dependency that
>    distros don't ship then we need to wrap it up and provide it ourselves
>

Have we considered meson wrap? I never really looked at it in detail, not
sure if that would work for us.
https://mesonbuild.com/Wrap-dependency-system-manual.html

 * where we ship binary blobs (guest BIOS etc) we want to also ship
>    the source code for those blobs
> I think for Rust dependencies those don't really apply.
>

And we mirror all those dependencies at the same location.


> Overall, I think that to the extent that we can look like a "normal"
> user of Rust, that's a good plan. Distros may well want to be able
> to do "build against our packaged rust stuff rather than downloading
> from crates.io" but I imagine they have machinery for that already;
> if we act like most other Rust programs we have better chances of
> not breaking that machinery.
>

True, at least on Fedora, there is machinery to package "regular" Rust
programs/crates in an automated way.  Vendoring dependencies should work
equally, but may not conform with distro policies, so they have extra work
eventually (it seems vendoring is more and more common though, with go
projects for example)


> We do already effectively do "download code when QEMU is built" --
> the makefile invokes scripts/git-submodule-update which pulls
> down submodule code. (Thanks to Ian for pointing out this framing
> of the question.)
>
> (I'm not personally a fan of the "download everything from crates.io"
> Rust ecosystem, but it is what it is, and wishing the Rust world
> worked more like a trad Linux-distro-provides-all-your-dependencies
> isn't, alas, going to make it so :-))
>
>
A nice alternative to vendoring that could work well for QEMU is to mirror
the Rust crate we use, so we have similar control and guarantee over them
as submodules, and use `[patch.crates-io]` to point at qemu-project
locations.


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 5979 bytes --]