Moving QEMU downloads to GitLab Releases?

Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Hi Mike,
QEMU downloads are currently hosted on qemu.org's Apache web server.
Paolo and I were discussing ways to reduce qemu.org network traffic to
save money and eventually turn off the qemu.org server since there is no
full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
GitLab Releases.

Since you create and sign QEMU releases I wanted to see what you think
about the idea. GitLab Releases has two ways of creating release assets:
archiving a git tree and attaching arbitrary binaries. The
scripts/make-release script fetches submodules and generates version
files, so it may be necessary to treat QEMU tarballs as arbitrary
binaries instead of simply letting GitLab create git tree archives:
https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries

Releases can be uploaded via the GitLab API from your local machine or
deployed as a GitLab CI job. Uploading from your local machine would be
the closest to the current workflow.

In the long term we could have a CI job that automatically publishes
QEMU releases when a new qemu.git tag is pushed. The release process
could be fully automated so that manual steps are no longer necessary,
although we'd have to trust GitLab with QEMU GPG signing keys.

What do you think?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> Hi Mike,
> QEMU downloads are currently hosted on qemu.org's Apache web server.
> Paolo and I were discussing ways to reduce qemu.org network traffic to
> save money and eventually turn off the qemu.org server since there is no
> full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> GitLab Releases.

Daniel Berrange posted this in another discussion:

"gitlab releases have a per-file size limit that is somewhere on the
order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't
believe that's going to be viable.

The gitlab package registry doesn't directly support plain file
downloads afaik, and is also size limited to 50 MB per package

I'd love to find a good solution for large release artifact hosting,
since I need a better solution for virt-viewer windows MSI releases
wich are similarly large to QEMUs. For that I'm using pagure.io
provided by Fedora, but I don't have confidence in that service
surviving long term."

So it looks like GitLab Releases won't work for QEMU :(.

Stefan

Re: Moving QEMU downloads to GitLab Releases?

[Eldon Stegall]

Hello!
I'd be happy to help with this. I'm mostly a consumer of QEMU, but
greatly appreciate all the work this community has done, and was able
to contribute a little by helping with QEMU advent this past year. I
would be happy to help streamline some of this activities if that would
be welcome, and would gratefully contribute time and resources. Hosting
and serving data like this has been core to my recent experience.

I would be happy to suggest and build out a distribution strategy for
these packages, and believe I could cut some costs, and even convince a
small consultancy I am a part of here that uses QEMU to foot a
reasonable bill.

A brief introduction, since I haven't had the pleasure of attending
FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security
Architect, based out of Atlanta, previously with companies like
DataStax, but now working on AWS video pipelines for a startup here.

Thanks, and hopefully I can be of service!
Eldon

On Thu, Sep 30, 2021 at 03:28:53PM +0100, Stefan Hajnoczi wrote:
> On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > Hi Mike,
> > QEMU downloads are currently hosted on qemu.org's Apache web server.
> > Paolo and I were discussing ways to reduce qemu.org network traffic to
> > save money and eventually turn off the qemu.org server since there is no
> > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> > GitLab Releases.
> 
> Daniel Berrange posted this in another discussion:
> 
> "gitlab releases have a per-file size limit that is somewhere on the
> order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't
> believe that's going to be viable.
> 
> The gitlab package registry doesn't directly support plain file
> downloads afaik, and is also size limited to 50 MB per package
> 
> I'd love to find a good solution for large release artifact hosting,
> since I need a better solution for virt-viewer windows MSI releases
> wich are similarly large to QEMUs. For that I'm using pagure.io
> provided by Fedora, but I don't have confidence in that service
> surviving long term."
> 
> So it looks like GitLab Releases won't work for QEMU :(.
> 
> Stefan
> 

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]

On Thu, Sep 30, 2021 at 03:57:49PM +0000, Eldon Stegall wrote:
> Hello!
> I'd be happy to help with this. I'm mostly a consumer of QEMU, but
> greatly appreciate all the work this community has done, and was able
> to contribute a little by helping with QEMU advent this past year. I
> would be happy to help streamline some of this activities if that would
> be welcome, and would gratefully contribute time and resources. Hosting
> and serving data like this has been core to my recent experience.
> 
> I would be happy to suggest and build out a distribution strategy for
> these packages, and believe I could cut some costs, and even convince a
> small consultancy I am a part of here that uses QEMU to foot a
> reasonable bill.
> 
> A brief introduction, since I haven't had the pleasure of attending
> FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security
> Architect, based out of Atlanta, previously with companies like
> DataStax, but now working on AWS video pipelines for a startup here.

Thanks for joining the discussion and for running last year's QEMU
Advent Calendar, Eldon.

Any ideas for moving download.qemu.org to a hosted service would be
appreciated! We haven't compared CDN and cloud providers closely yet. If
you have experience in this area or time to check them out, then that
would be valuable.

QEMU has funds if there is a cost for file hosting (probably less than
$100/month). Some providers may be willing to support an open source
project for free. Possible providers include CloudFlare, Akamai, Fastly,
Microsoft Azure, Google Cloud Storage, etc.

We need to keep the security of QEMU releases in mind. Mike Roth
signs and publishes releases. Whoever facilitates or hosts the files
should not be able to modify the files after Mike has blessed them. One
way to do this is to keep hosting the .sig files on download.qemu.org
and to redirect the actual tarballs to a file hosting provider. A way to
securely publish files without hosting anything on qemu.org would be
even better though (maybe it's enough to publish signatures on the
static GitLab Pages website).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Daniel P. Berrangé]

On Fri, Oct 01, 2021 at 08:11:35AM +0100, Stefan Hajnoczi wrote:
> We need to keep the security of QEMU releases in mind. Mike Roth
> signs and publishes releases. Whoever facilitates or hosts the files
> should not be able to modify the files after Mike has blessed them. One
> way to do this is to keep hosting the .sig files on download.qemu.org
> and to redirect the actual tarballs to a file hosting provider. A way to
> securely publish files without hosting anything on qemu.org would be
> even better though (maybe it's enough to publish signatures on the
> static GitLab Pages website).

If someone modifies the download files, then when you verify the sig
it will be detected. It doesn't matter whether the sig is on the same
host or not, because if someone modifies the sig too, then it will
still fail validation. The important thing is that the user has got
the right public key to verify with.

IOW, hosting the .sig separately is not required. We need to ensure
that our public key, however, is published & discoverable in a
trustworthy place that is separate from the download server. We fail
at that today because www.qemu.org and download.qemu.org are the
same server.

So it will be beneficial if the download site is split off from
the public website, compared to our current setup.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]

On Fri, Oct 01, 2021 at 09:52:20AM +0100, Daniel P. Berrangé wrote:
> On Fri, Oct 01, 2021 at 08:11:35AM +0100, Stefan Hajnoczi wrote:
> > We need to keep the security of QEMU releases in mind. Mike Roth
> > signs and publishes releases. Whoever facilitates or hosts the files
> > should not be able to modify the files after Mike has blessed them. One
> > way to do this is to keep hosting the .sig files on download.qemu.org
> > and to redirect the actual tarballs to a file hosting provider. A way to
> > securely publish files without hosting anything on qemu.org would be
> > even better though (maybe it's enough to publish signatures on the
> > static GitLab Pages website).
> 
> If someone modifies the download files, then when you verify the sig
> it will be detected. It doesn't matter whether the sig is on the same
> host or not, because if someone modifies the sig too, then it will
> still fail validation. The important thing is that the user has got
> the right public key to verify with.
> 
> IOW, hosting the .sig separately is not required. We need to ensure
> that our public key, however, is published & discoverable in a
> trustworthy place that is separate from the download server. We fail
> at that today because www.qemu.org and download.qemu.org are the
> same server.
> 
> So it will be beneficial if the download site is split off from
> the public website, compared to our current setup.

You're right. Thanks for pointing this out. I was thinking of the .sig
as a checksum but it's a signature :-).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 1912 bytes --]

On Fri, Oct 1, 2021 at 12:20 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Thu, Sep 30, 2021 at 03:57:49PM +0000, Eldon Stegall wrote:
> > Hello!
> > I'd be happy to help with this. I'm mostly a consumer of QEMU, but
> > greatly appreciate all the work this community has done, and was able
> > to contribute a little by helping with QEMU advent this past year. I
> > would be happy to help streamline some of this activities if that would
> > be welcome, and would gratefully contribute time and resources. Hosting
> > and serving data like this has been core to my recent experience.
> >
> > I would be happy to suggest and build out a distribution strategy for
> > these packages, and believe I could cut some costs, and even convince a
> > small consultancy I am a part of here that uses QEMU to foot a
> > reasonable bill.
> >
> > A brief introduction, since I haven't had the pleasure of attending
> > FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security
> > Architect, based out of Atlanta, previously with companies like
> > DataStax, but now working on AWS video pipelines for a startup here.
>
> Thanks for joining the discussion and for running last year's QEMU
> Advent Calendar, Eldon.
>
> Any ideas for moving download.qemu.org to a hosted service would be
> appreciated! We haven't compared CDN and cloud providers closely yet. If
> you have experience in this area or time to check them out, then that
> would be valuable.
>
> QEMU has funds if there is a cost for file hosting (probably less than
> $100/month). Some providers may be willing to support an open source
> project for free. Possible providers include CloudFlare, Akamai, Fastly,
> Microsoft Azure, Google Cloud Storage, etc.
>

https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/

Let me know if ya'll apply and I'm happy to push it through.

Regards,

Anthony Liguori

[-- Attachment #2: Type: text/html, Size: 2617 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Thomas Huth]

On 30/09/2021 16.28, Stefan Hajnoczi wrote:
> On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> Hi Mike,
>> QEMU downloads are currently hosted on qemu.org's Apache web server.
>> Paolo and I were discussing ways to reduce qemu.org network traffic to
>> save money and eventually turn off the qemu.org server since there is no
>> full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
>> GitLab Releases.
> 
> Daniel Berrange posted this in another discussion:
> 
> "gitlab releases have a per-file size limit that is somewhere on the
> order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't
> believe that's going to be viable.
> 
> The gitlab package registry doesn't directly support plain file
> downloads afaik, and is also size limited to 50 MB per package

Maybe we should also discuss again whether we could decrease the size of the 
release tarballs. Someone recently mentioned that we could e.g. remove the 
edk2 sources from the tarballs - edk2 is licensed under a BSD-style license, 
so we are not forced to ship its sources in our tarballs (unlike the other 
firmwares which are licensed under GPL).

Or maybe we could also finally move the firmware images completely into a 
separate tarball instead?

  Thomas

Re: Moving QEMU downloads to GitLab Releases?

[Gerd Hoffmann]

  Hi,

> Maybe we should also discuss again whether we could decrease the size of the
> release tarballs. Someone recently mentioned that we could e.g. remove the
> edk2 sources from the tarballs - edk2 is licensed under a BSD-style license,
> so we are not forced to ship its sources in our tarballs (unlike the other
> firmwares which are licensed under GPL).
> 
> Or maybe we could also finally move the firmware images completely into a
> separate tarball instead?

The idea of a separate qemu-firmware repo is floating around for a
while already but never really took off.

Now with hosting moved to gitlab we should be able to run automated
firmware binary builds in CI and purge all firmware from the qemu git
repo (both submodules and binaries) and add a script fetching those
from gitlab instead.

take care,
  Gerd

Re: Moving QEMU downloads to GitLab Releases?

[Philippe Mathieu-Daudé]

On 10/1/21 12:34, Gerd Hoffmann wrote:>> Maybe we should also discuss
again whether we could decrease the size of the
>> release tarballs. Someone recently mentioned that we could e.g. remove the
>> edk2 sources from the tarballs - edk2 is licensed under a BSD-style license,
>> so we are not forced to ship its sources in our tarballs (unlike the other
>> firmwares which are licensed under GPL).
>>
>> Or maybe we could also finally move the firmware images completely into a
>> separate tarball instead?
> 
> The idea of a separate qemu-firmware repo is floating around for a
> while already but never really took off.
> 
> Now with hosting moved to gitlab we should be able to run automated
> firmware binary builds in CI and purge all firmware from the qemu git
> repo (both submodules and binaries) and add a script fetching those
> from gitlab instead.

We already have some firmware jobs which might be used as template:

.gitlab-ci.d/edk2.yml
.gitlab-ci.d/opensbi.yml

Re: Moving QEMU downloads to GitLab Releases?

[Philippe Mathieu-Daudé]

On 9/30/21 15:40, Stefan Hajnoczi wrote:
> Hi Mike,
> QEMU downloads are currently hosted on qemu.org's Apache web server.
> Paolo and I were discussing ways to reduce qemu.org network traffic to
> save money and eventually turn off the qemu.org server since there is no
> full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> GitLab Releases.
> 
> Since you create and sign QEMU releases I wanted to see what you think
> about the idea. GitLab Releases has two ways of creating release assets:
> archiving a git tree and attaching arbitrary binaries. The
> scripts/make-release script fetches submodules and generates version
> files, so it may be necessary to treat QEMU tarballs as arbitrary
> binaries instead of simply letting GitLab create git tree archives:
> https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries
> 
> Releases can be uploaded via the GitLab API from your local machine or
> deployed as a GitLab CI job. Uploading from your local machine would be
> the closest to the current workflow.
> 
> In the long term we could have a CI job that automatically publishes
> QEMU releases when a new qemu.git tag is pushed. The release process
> could be fully automated so that manual steps are no longer necessary,
> although we'd have to trust GitLab with QEMU GPG signing keys.

Before having to trust a SaaS for GPG signing, could this work?

- make-release script should produce a reproducible tarball in a
  gitlab job, along with a file containing the tarball hash.

- Mike (or whoever is responsible of releases) keeps doing a local
  manual build

- Mike checks the local hash matches the Gitlab artifact hash

- Mike signs its local build/hash and uses the GitLab API to upload
  that .sig to job artifacts

- we can have an extra manual job that checks the tarball.sig
  (hash and pubkey) and on success deploys updating the download
  page, adding the new release

Regards,

Phil.

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]

On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote:
> On 9/30/21 15:40, Stefan Hajnoczi wrote:
> > Hi Mike,
> > QEMU downloads are currently hosted on qemu.org's Apache web server.
> > Paolo and I were discussing ways to reduce qemu.org network traffic to
> > save money and eventually turn off the qemu.org server since there is no
> > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> > GitLab Releases.
> > 
> > Since you create and sign QEMU releases I wanted to see what you think
> > about the idea. GitLab Releases has two ways of creating release assets:
> > archiving a git tree and attaching arbitrary binaries. The
> > scripts/make-release script fetches submodules and generates version
> > files, so it may be necessary to treat QEMU tarballs as arbitrary
> > binaries instead of simply letting GitLab create git tree archives:
> > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries
> > 
> > Releases can be uploaded via the GitLab API from your local machine or
> > deployed as a GitLab CI job. Uploading from your local machine would be
> > the closest to the current workflow.
> > 
> > In the long term we could have a CI job that automatically publishes
> > QEMU releases when a new qemu.git tag is pushed. The release process
> > could be fully automated so that manual steps are no longer necessary,
> > although we'd have to trust GitLab with QEMU GPG signing keys.
> 
> Before having to trust a SaaS for GPG signing, could this work?
> 
> - make-release script should produce a reproducible tarball in a
>   gitlab job, along with a file containing the tarball hash.
> 
> - Mike (or whoever is responsible of releases) keeps doing a local
>   manual build
> 
> - Mike checks the local hash matches the Gitlab artifact hash
> 
> - Mike signs its local build/hash and uses the GitLab API to upload
>   that .sig to job artifacts
> 
> - we can have an extra manual job that checks the tarball.sig
>   (hash and pubkey) and on success deploys updating the download
>   page, adding the new release

I wonder what Mike sees as the way forward.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Michael Roth]

Quoting Stefan Hajnoczi (2021-10-04 04:01:22)
> On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote:
> > On 9/30/21 15:40, Stefan Hajnoczi wrote:
> > > Hi Mike,
> > > QEMU downloads are currently hosted on qemu.org's Apache web server.
> > > Paolo and I were discussing ways to reduce qemu.org network traffic to
> > > save money and eventually turn off the qemu.org server since there is no
> > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> > > GitLab Releases.
> > > 
> > > Since you create and sign QEMU releases I wanted to see what you think
> > > about the idea. GitLab Releases has two ways of creating release assets:
> > > archiving a git tree and attaching arbitrary binaries. The
> > > scripts/make-release script fetches submodules and generates version
> > > files, so it may be necessary to treat QEMU tarballs as arbitrary
> > > binaries instead of simply letting GitLab create git tree archives:
> > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries
> > > 
> > > Releases can be uploaded via the GitLab API from your local machine or
> > > deployed as a GitLab CI job. Uploading from your local machine would be
> > > the closest to the current workflow.
> > > 
> > > In the long term we could have a CI job that automatically publishes
> > > QEMU releases when a new qemu.git tag is pushed. The release process
> > > could be fully automated so that manual steps are no longer necessary,
> > > although we'd have to trust GitLab with QEMU GPG signing keys.
> > 
> > Before having to trust a SaaS for GPG signing, could this work?
> > 
> > - make-release script should produce a reproducible tarball in a
> >   gitlab job, along with a file containing the tarball hash.
> > 
> > - Mike (or whoever is responsible of releases) keeps doing a local
> >   manual build
> > 
> > - Mike checks the local hash matches the Gitlab artifact hash
> > 
> > - Mike signs its local build/hash and uses the GitLab API to upload
> >   that .sig to job artifacts
> > 
> > - we can have an extra manual job that checks the tarball.sig
> >   (hash and pubkey) and on success deploys updating the download
> >   page, adding the new release
> 
> I wonder what Mike sees as the way forward.

Hi Stefan, Philippe,

In general I like the idea, since we could also have the CI do the full
gamut of testing against the binaries built from said tarball, so the
Release Person can just regenerate the tarball and provide a sig to
attest that it came from the proper sources. Currently I do make check
and make check-acceptance and a few other sanity checks, which I guess
would no longer be needed then.

But I think the more immediate issue is where/how to host those
tarballs. Even moving all the ROMs/capstone out of the source tree still
results in an xz-compressed tarball size ~25MB, which is well above the
10MB limit mentioned earlier. We could break it out into target-specific
tarballs, maybe further into softmmu/user variants, but that sounds
painful for both users and maintainers who need to deal with the
resulting source tree differences.

What I'm wondering is whether we could just use the archive files
generated by gitlab when we tag our releases? E.g.:

  https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2

If we paired that with an in-tree script similar to make-release for
users to download individual ROM sources/subprojects used for a particular
tagged release, would that be sufficient for GPL compliance and verifying
what sources the binaries were built from? Are there any other
considerations WRT ROMs/etc.?

With something like that in place, Release Person could just do a git
checkout, verify the Maintainer's sig/tag (in case we don't necessarily
trust the git host), generate the tarball, verify the hash matches what
gitlab published (or verify/diff individual files if the bz/gz hashes
require a specific environment), then sign the gitlab tarball and add
the sig to qemu.org download page along with a link the gitlab-generated
tarball.

We could also publish the Maintainer and Release Person public keys on
qemu.org download page so users can verify this as well using the same
process.

Users that want the additional sources can then do it locally via
above-mentioned script, which would be part of the now-signed tarball
and so could be 'trusted' assuming the individual project hosts weren't
compromised (which is still an assumption that's needed with the current
process anyway).

I guess the main question is who is using the ROM/BIOS sources in the
tarballs, and would this 2-step process work for those users? If there
are distros relying on them then maybe there are some more logistics to
consider since the make-release downloads are both time-consuming and
prone to network errors/stalls since it relies on the availability of a
good number of different hosts.

-Mike

> 
> Stefan

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5830 bytes --]

On Mon, Oct 04, 2021 at 02:34:49PM -0500, Michael Roth wrote:
> Quoting Stefan Hajnoczi (2021-10-04 04:01:22)
> > On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote:
> > > On 9/30/21 15:40, Stefan Hajnoczi wrote:
> > > > Hi Mike,
> > > > QEMU downloads are currently hosted on qemu.org's Apache web server.
> > > > Paolo and I were discussing ways to reduce qemu.org network traffic to
> > > > save money and eventually turn off the qemu.org server since there is no
> > > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> > > > GitLab Releases.
> > > > 
> > > > Since you create and sign QEMU releases I wanted to see what you think
> > > > about the idea. GitLab Releases has two ways of creating release assets:
> > > > archiving a git tree and attaching arbitrary binaries. The
> > > > scripts/make-release script fetches submodules and generates version
> > > > files, so it may be necessary to treat QEMU tarballs as arbitrary
> > > > binaries instead of simply letting GitLab create git tree archives:
> > > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries
> > > > 
> > > > Releases can be uploaded via the GitLab API from your local machine or
> > > > deployed as a GitLab CI job. Uploading from your local machine would be
> > > > the closest to the current workflow.
> > > > 
> > > > In the long term we could have a CI job that automatically publishes
> > > > QEMU releases when a new qemu.git tag is pushed. The release process
> > > > could be fully automated so that manual steps are no longer necessary,
> > > > although we'd have to trust GitLab with QEMU GPG signing keys.
> > > 
> > > Before having to trust a SaaS for GPG signing, could this work?
> > > 
> > > - make-release script should produce a reproducible tarball in a
> > >   gitlab job, along with a file containing the tarball hash.
> > > 
> > > - Mike (or whoever is responsible of releases) keeps doing a local
> > >   manual build
> > > 
> > > - Mike checks the local hash matches the Gitlab artifact hash
> > > 
> > > - Mike signs its local build/hash and uses the GitLab API to upload
> > >   that .sig to job artifacts
> > > 
> > > - we can have an extra manual job that checks the tarball.sig
> > >   (hash and pubkey) and on success deploys updating the download
> > >   page, adding the new release
> > 
> > I wonder what Mike sees as the way forward.
> 
> Hi Stefan, Philippe,
> 
> In general I like the idea, since we could also have the CI do the full
> gamut of testing against the binaries built from said tarball, so the
> Release Person can just regenerate the tarball and provide a sig to
> attest that it came from the proper sources. Currently I do make check
> and make check-acceptance and a few other sanity checks, which I guess
> would no longer be needed then.
> 
> But I think the more immediate issue is where/how to host those
> tarballs. Even moving all the ROMs/capstone out of the source tree still
> results in an xz-compressed tarball size ~25MB, which is well above the
> 10MB limit mentioned earlier. We could break it out into target-specific
> tarballs, maybe further into softmmu/user variants, but that sounds
> painful for both users and maintainers who need to deal with the
> resulting source tree differences.
> 
> What I'm wondering is whether we could just use the archive files
> generated by gitlab when we tag our releases? E.g.:
> 
>   https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2
> 
> If we paired that with an in-tree script similar to make-release for
> users to download individual ROM sources/subprojects used for a particular
> tagged release, would that be sufficient for GPL compliance and verifying
> what sources the binaries were built from? Are there any other
> considerations WRT ROMs/etc.?
> 
> With something like that in place, Release Person could just do a git
> checkout, verify the Maintainer's sig/tag (in case we don't necessarily
> trust the git host), generate the tarball, verify the hash matches what
> gitlab published (or verify/diff individual files if the bz/gz hashes
> require a specific environment), then sign the gitlab tarball and add
> the sig to qemu.org download page along with a link the gitlab-generated
> tarball.
> 
> We could also publish the Maintainer and Release Person public keys on
> qemu.org download page so users can verify this as well using the same
> process.
> 
> Users that want the additional sources can then do it locally via
> above-mentioned script, which would be part of the now-signed tarball
> and so could be 'trusted' assuming the individual project hosts weren't
> compromised (which is still an assumption that's needed with the current
> process anyway).
> 
> I guess the main question is who is using the ROM/BIOS sources in the
> tarballs, and would this 2-step process work for those users? If there
> are distros relying on them then maybe there are some more logistics to
> consider since the make-release downloads are both time-consuming and
> prone to network errors/stalls since it relies on the availability of a
> good number of different hosts.

Great, maybe Gerd can comment on splitting out firmware.

QEMU mirrors firmware sources on GitLab too. We're able to provide the
same level of download availability on our mirror firmware repos as for
the main qemu.git repo.

I think qemu.git should include information about where to get the
firmware sources in order to comply with the GPL. I'm not sure if the
git submodules metadata counts from a GPL perspective, but that seems
enough since it has both a mirror repo URL and the exact hash used to
build the binary blob.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Gerd Hoffmann]

  Hi,

> > I guess the main question is who is using the ROM/BIOS sources in the
> > tarballs, and would this 2-step process work for those users? If there
> > are distros relying on them then maybe there are some more logistics to
> > consider since the make-release downloads are both time-consuming and
> > prone to network errors/stalls since it relies on the availability of a
> > good number of different hosts.
> 
> Great, maybe Gerd can comment on splitting out firmware.

I think the binaries are mostly a convenience feature for developers.
Distros typically build from source anyway, and typically they build
from upstream source instead of qemu submodule.

The idea to split them out to a separate repo is exists for quite a
while.  I have an old qemu-firmware repo laying around on my disk, which
basically moves roms/ + submodules and the binaries built from it over.

I think with the switch to gitlab it doesn't make sense any more to
commit pre-built firmware binaries to a git repo.  Instead we should
build the firmware in gitlab ci, i.e. effectively move the build rules
we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
firmware binaries as build artifacts or gitlab pages.

When done just remove the pre-build binaries from git and add a helper
script to fetch firmware binaries from gitlab instead.

> QEMU mirrors firmware sources on GitLab too. We're able to provide the
> same level of download availability on our mirror firmware repos as for
> the main qemu.git repo.

I think enabling CI for the firmware mirrors should work given that it
is possible to specify a custom CI/CD configuration file, i.e. use
something like

    /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml

or

    /qemu-project/qemu-firmware/seabios.yml

as config file for the 

    /qemu-project/seabios/

mirror.  Then we can publish binaries using gitlab pages at

    https://qemu-project.gitlab.io/seabios/

That way we also don't need the roms/ submodules any more, i.e. we
can remove both sources and binaries for the firmware from the
release tarballs.

take care,
  Gerd

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > I guess the main question is who is using the ROM/BIOS sources in the
> > > tarballs, and would this 2-step process work for those users? If there
> > > are distros relying on them then maybe there are some more logistics to
> > > consider since the make-release downloads are both time-consuming and
> > > prone to network errors/stalls since it relies on the availability of a
> > > good number of different hosts.
> > 
> > Great, maybe Gerd can comment on splitting out firmware.
> 
> I think the binaries are mostly a convenience feature for developers.
> Distros typically build from source anyway, and typically they build
> from upstream source instead of qemu submodule.
> 
> The idea to split them out to a separate repo is exists for quite a
> while.  I have an old qemu-firmware repo laying around on my disk, which
> basically moves roms/ + submodules and the binaries built from it over.
> 
> I think with the switch to gitlab it doesn't make sense any more to
> commit pre-built firmware binaries to a git repo.  Instead we should
> build the firmware in gitlab ci, i.e. effectively move the build rules
> we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
> firmware binaries as build artifacts or gitlab pages.
> 
> When done just remove the pre-build binaries from git and add a helper
> script to fetch firmware binaries from gitlab instead.
> 
> > QEMU mirrors firmware sources on GitLab too. We're able to provide the
> > same level of download availability on our mirror firmware repos as for
> > the main qemu.git repo.
> 
> I think enabling CI for the firmware mirrors should work given that it
> is possible to specify a custom CI/CD configuration file, i.e. use
> something like
> 
>     /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml
> 
> or
> 
>     /qemu-project/qemu-firmware/seabios.yml
> 
> as config file for the 
> 
>     /qemu-project/seabios/
> 
> mirror.  Then we can publish binaries using gitlab pages at
> 
>     https://qemu-project.gitlab.io/seabios/
> 
> That way we also don't need the roms/ submodules any more, i.e. we
> can remove both sources and binaries for the firmware from the
> release tarballs.

Thanks!

For developer convenience it would be nice to avoid manual steps after
git clone qemu.git. Maybe ./configure should check for firmware blobs
and automatically download them. There could be a ./configure
--disable-firmware-download option that distros can use to skip the
download when building everything from source.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3739 bytes --]

On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote:
> >   Hi,
> >
> > > > I guess the main question is who is using the ROM/BIOS sources in the
> > > > tarballs, and would this 2-step process work for those users? If
> there
> > > > are distros relying on them then maybe there are some more logistics
> to
> > > > consider since the make-release downloads are both time-consuming and
> > > > prone to network errors/stalls since it relies on the availability
> of a
> > > > good number of different hosts.
> > >
> > > Great, maybe Gerd can comment on splitting out firmware.
> >
> > I think the binaries are mostly a convenience feature for developers.
> > Distros typically build from source anyway, and typically they build
> > from upstream source instead of qemu submodule.
> >
> > The idea to split them out to a separate repo is exists for quite a
> > while.  I have an old qemu-firmware repo laying around on my disk, which
> > basically moves roms/ + submodules and the binaries built from it over.
> >
> > I think with the switch to gitlab it doesn't make sense any more to
> > commit pre-built firmware binaries to a git repo.  Instead we should
> > build the firmware in gitlab ci, i.e. effectively move the build rules
> > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
> > firmware binaries as build artifacts or gitlab pages.
> >
> > When done just remove the pre-build binaries from git and add a helper
> > script to fetch firmware binaries from gitlab instead.
> >
> > > QEMU mirrors firmware sources on GitLab too. We're able to provide the
> > > same level of download availability on our mirror firmware repos as for
> > > the main qemu.git repo.
> >
> > I think enabling CI for the firmware mirrors should work given that it
> > is possible to specify a custom CI/CD configuration file, i.e. use
> > something like
> >
> >     /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml
> >
> > or
> >
> >     /qemu-project/qemu-firmware/seabios.yml
> >
> > as config file for the
> >
> >     /qemu-project/seabios/
> >
> > mirror.  Then we can publish binaries using gitlab pages at
> >
> >     https://qemu-project.gitlab.io/seabios/
> >
> > That way we also don't need the roms/ submodules any more, i.e. we
> > can remove both sources and binaries for the firmware from the
> > release tarballs.
>
> Thanks!
>
> For developer convenience it would be nice to avoid manual steps after
> git clone qemu.git. Maybe ./configure should check for firmware blobs
> and automatically download them. There could be a ./configure
> --disable-firmware-download option that distros can use to skip the
> download when building everything from source.
>

One thing to keep in mind about the downstream consumers: Many
of them have two phases to their build process that run asynchronously
to each other. There is a fetch phase that grabs everything, and a build
phase that builds the binaries. The second phase may not have access
to the internet for a variety of reasons (these days being a security
measure, for good or ill). Please make sure that any such plans
allow for this model.

Today, that's all dealt with by grabbing tarballs from github which
is how the submodules are dealt with. So long as the images
had well known names, and could be fetched with the normal
wget/cgit/fetch programs, that would suffice. Requiring use of
some weird bespoke script would cause friction for the downstreams
using qemu.

So while I'm all for making things a little more independent,
let's not do it in a way that makes life difficult for downstreams.
There are more there than just a couple of big distro builders.

Warner

[-- Attachment #2: Type: text/html, Size: 4819 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4167 bytes --]

On Mon, Oct 11, 2021 at 08:28:34AM -0600, Warner Losh wrote:
> On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> > On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote:
> > >   Hi,
> > >
> > > > > I guess the main question is who is using the ROM/BIOS sources in the
> > > > > tarballs, and would this 2-step process work for those users? If
> > there
> > > > > are distros relying on them then maybe there are some more logistics
> > to
> > > > > consider since the make-release downloads are both time-consuming and
> > > > > prone to network errors/stalls since it relies on the availability
> > of a
> > > > > good number of different hosts.
> > > >
> > > > Great, maybe Gerd can comment on splitting out firmware.
> > >
> > > I think the binaries are mostly a convenience feature for developers.
> > > Distros typically build from source anyway, and typically they build
> > > from upstream source instead of qemu submodule.
> > >
> > > The idea to split them out to a separate repo is exists for quite a
> > > while.  I have an old qemu-firmware repo laying around on my disk, which
> > > basically moves roms/ + submodules and the binaries built from it over.
> > >
> > > I think with the switch to gitlab it doesn't make sense any more to
> > > commit pre-built firmware binaries to a git repo.  Instead we should
> > > build the firmware in gitlab ci, i.e. effectively move the build rules
> > > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the
> > > firmware binaries as build artifacts or gitlab pages.
> > >
> > > When done just remove the pre-build binaries from git and add a helper
> > > script to fetch firmware binaries from gitlab instead.
> > >
> > > > QEMU mirrors firmware sources on GitLab too. We're able to provide the
> > > > same level of download availability on our mirror firmware repos as for
> > > > the main qemu.git repo.
> > >
> > > I think enabling CI for the firmware mirrors should work given that it
> > > is possible to specify a custom CI/CD configuration file, i.e. use
> > > something like
> > >
> > >     /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml
> > >
> > > or
> > >
> > >     /qemu-project/qemu-firmware/seabios.yml
> > >
> > > as config file for the
> > >
> > >     /qemu-project/seabios/
> > >
> > > mirror.  Then we can publish binaries using gitlab pages at
> > >
> > >     https://qemu-project.gitlab.io/seabios/
> > >
> > > That way we also don't need the roms/ submodules any more, i.e. we
> > > can remove both sources and binaries for the firmware from the
> > > release tarballs.
> >
> > Thanks!
> >
> > For developer convenience it would be nice to avoid manual steps after
> > git clone qemu.git. Maybe ./configure should check for firmware blobs
> > and automatically download them. There could be a ./configure
> > --disable-firmware-download option that distros can use to skip the
> > download when building everything from source.
> >
> 
> One thing to keep in mind about the downstream consumers: Many
> of them have two phases to their build process that run asynchronously
> to each other. There is a fetch phase that grabs everything, and a build
> phase that builds the binaries. The second phase may not have access
> to the internet for a variety of reasons (these days being a security
> measure, for good or ill). Please make sure that any such plans
> allow for this model.
> 
> Today, that's all dealt with by grabbing tarballs from github which
> is how the submodules are dealt with. So long as the images
> had well known names, and could be fetched with the normal
> wget/cgit/fetch programs, that would suffice. Requiring use of
> some weird bespoke script would cause friction for the downstreams
> using qemu.
> 
> So while I'm all for making things a little more independent,
> let's not do it in a way that makes life difficult for downstreams.
> There are more there than just a couple of big distro builders.

I think this is fine. Firmware blobs aren't needed to build QEMU, only
to run the system emulator.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Moving QEMU downloads to GitLab Releases?

[Gerd Hoffmann]

  Hi,

> > So while I'm all for making things a little more independent,
> > let's not do it in a way that makes life difficult for downstreams.
> > There are more there than just a couple of big distro builders.
> 
> I think this is fine. Firmware blobs aren't needed to build QEMU, only
> to run the system emulator.

Yep.  But doing firmware builds outside gitlab should be easy, so we
should probably keep the core logic in a script / Makefile / whatever
(simliar to todays roms/Makefile), then simply invoke that from gitlab
ci yaml files.

take care,
  Gerd