* Moving QEMU downloads to GitLab Releases? @ 2021-09-30 13:40 Stefan Hajnoczi 2021-09-30 14:28 ` Stefan Hajnoczi 2021-10-01 7:39 ` Philippe Mathieu-Daudé 0 siblings, 2 replies; 19+ messages in thread From: Stefan Hajnoczi @ 2021-09-30 13:40 UTC (permalink / raw) To: michael.roth Cc: Paolo Bonzini, Thomas Huth, Daniel Berrange, Philippe Mathieu-Daudé, qemu-devel [-- Attachment #1: Type: text/plain, Size: 1295 bytes --] Hi Mike, QEMU downloads are currently hosted on qemu.org's Apache web server. Paolo and I were discussing ways to reduce qemu.org network traffic to save money and eventually turn off the qemu.org server since there is no full-time sysadmin for it. I'd like to discuss moving QEMU downloads to GitLab Releases. Since you create and sign QEMU releases I wanted to see what you think about the idea. GitLab Releases has two ways of creating release assets: archiving a git tree and attaching arbitrary binaries. The scripts/make-release script fetches submodules and generates version files, so it may be necessary to treat QEMU tarballs as arbitrary binaries instead of simply letting GitLab create git tree archives: https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries Releases can be uploaded via the GitLab API from your local machine or deployed as a GitLab CI job. Uploading from your local machine would be the closest to the current workflow. In the long term we could have a CI job that automatically publishes QEMU releases when a new qemu.git tag is pushed. The release process could be fully automated so that manual steps are no longer necessary, although we'd have to trust GitLab with QEMU GPG signing keys. What do you think? Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-09-30 13:40 Moving QEMU downloads to GitLab Releases? Stefan Hajnoczi @ 2021-09-30 14:28 ` Stefan Hajnoczi 2021-09-30 15:57 ` Eldon Stegall 2021-10-01 7:24 ` Thomas Huth 2021-10-01 7:39 ` Philippe Mathieu-Daudé 1 sibling, 2 replies; 19+ messages in thread From: Stefan Hajnoczi @ 2021-09-30 14:28 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, michael.roth, qemu-devel, Philippe Mathieu-Daudé, Paolo Bonzini On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote: > > Hi Mike, > QEMU downloads are currently hosted on qemu.org's Apache web server. > Paolo and I were discussing ways to reduce qemu.org network traffic to > save money and eventually turn off the qemu.org server since there is no > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > GitLab Releases. Daniel Berrange posted this in another discussion: "gitlab releases have a per-file size limit that is somewhere on the order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't believe that's going to be viable. The gitlab package registry doesn't directly support plain file downloads afaik, and is also size limited to 50 MB per package I'd love to find a good solution for large release artifact hosting, since I need a better solution for virt-viewer windows MSI releases wich are similarly large to QEMUs. For that I'm using pagure.io provided by Fedora, but I don't have confidence in that service surviving long term." So it looks like GitLab Releases won't work for QEMU :(. Stefan ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-09-30 14:28 ` Stefan Hajnoczi @ 2021-09-30 15:57 ` Eldon Stegall 2021-10-01 7:11 ` Stefan Hajnoczi 2021-10-01 7:24 ` Thomas Huth 1 sibling, 1 reply; 19+ messages in thread From: Eldon Stegall @ 2021-09-30 15:57 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, michael.roth, qemu-devel, Philippe Mathieu-Daudé, Stefan Hajnoczi, Paolo Bonzini Hello! I'd be happy to help with this. I'm mostly a consumer of QEMU, but greatly appreciate all the work this community has done, and was able to contribute a little by helping with QEMU advent this past year. I would be happy to help streamline some of this activities if that would be welcome, and would gratefully contribute time and resources. Hosting and serving data like this has been core to my recent experience. I would be happy to suggest and build out a distribution strategy for these packages, and believe I could cut some costs, and even convince a small consultancy I am a part of here that uses QEMU to foot a reasonable bill. A brief introduction, since I haven't had the pleasure of attending FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security Architect, based out of Atlanta, previously with companies like DataStax, but now working on AWS video pipelines for a startup here. Thanks, and hopefully I can be of service! Eldon On Thu, Sep 30, 2021 at 03:28:53PM +0100, Stefan Hajnoczi wrote: > On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > > Hi Mike, > > QEMU downloads are currently hosted on qemu.org's Apache web server. > > Paolo and I were discussing ways to reduce qemu.org network traffic to > > save money and eventually turn off the qemu.org server since there is no > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > > GitLab Releases. > > Daniel Berrange posted this in another discussion: > > "gitlab releases have a per-file size limit that is somewhere on the > order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't > believe that's going to be viable. > > The gitlab package registry doesn't directly support plain file > downloads afaik, and is also size limited to 50 MB per package > > I'd love to find a good solution for large release artifact hosting, > since I need a better solution for virt-viewer windows MSI releases > wich are similarly large to QEMUs. For that I'm using pagure.io > provided by Fedora, but I don't have confidence in that service > surviving long term." > > So it looks like GitLab Releases won't work for QEMU :(. > > Stefan > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-09-30 15:57 ` Eldon Stegall @ 2021-10-01 7:11 ` Stefan Hajnoczi 2021-10-01 8:52 ` Daniel P. Berrangé 2021-10-11 15:00 ` Anthony Liguori 0 siblings, 2 replies; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-01 7:11 UTC (permalink / raw) To: Eldon Stegall Cc: Thomas Huth, Daniel Berrange, Stefan Hajnoczi, Philippe Mathieu-Daudé, qemu-devel, michael.roth, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 2165 bytes --] On Thu, Sep 30, 2021 at 03:57:49PM +0000, Eldon Stegall wrote: > Hello! > I'd be happy to help with this. I'm mostly a consumer of QEMU, but > greatly appreciate all the work this community has done, and was able > to contribute a little by helping with QEMU advent this past year. I > would be happy to help streamline some of this activities if that would > be welcome, and would gratefully contribute time and resources. Hosting > and serving data like this has been core to my recent experience. > > I would be happy to suggest and build out a distribution strategy for > these packages, and believe I could cut some costs, and even convince a > small consultancy I am a part of here that uses QEMU to foot a > reasonable bill. > > A brief introduction, since I haven't had the pleasure of attending > FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security > Architect, based out of Atlanta, previously with companies like > DataStax, but now working on AWS video pipelines for a startup here. Thanks for joining the discussion and for running last year's QEMU Advent Calendar, Eldon. Any ideas for moving download.qemu.org to a hosted service would be appreciated! We haven't compared CDN and cloud providers closely yet. If you have experience in this area or time to check them out, then that would be valuable. QEMU has funds if there is a cost for file hosting (probably less than $100/month). Some providers may be willing to support an open source project for free. Possible providers include CloudFlare, Akamai, Fastly, Microsoft Azure, Google Cloud Storage, etc. We need to keep the security of QEMU releases in mind. Mike Roth signs and publishes releases. Whoever facilitates or hosts the files should not be able to modify the files after Mike has blessed them. One way to do this is to keep hosting the .sig files on download.qemu.org and to redirect the actual tarballs to a file hosting provider. A way to securely publish files without hosting anything on qemu.org would be even better though (maybe it's enough to publish signatures on the static GitLab Pages website). Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 7:11 ` Stefan Hajnoczi @ 2021-10-01 8:52 ` Daniel P. Berrangé 2021-10-04 8:53 ` Stefan Hajnoczi 2021-10-11 15:00 ` Anthony Liguori 1 sibling, 1 reply; 19+ messages in thread From: Daniel P. Berrangé @ 2021-10-01 8:52 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Stefan Hajnoczi, Philippe Mathieu-Daudé, qemu-devel, Eldon Stegall, michael.roth, Paolo Bonzini On Fri, Oct 01, 2021 at 08:11:35AM +0100, Stefan Hajnoczi wrote: > We need to keep the security of QEMU releases in mind. Mike Roth > signs and publishes releases. Whoever facilitates or hosts the files > should not be able to modify the files after Mike has blessed them. One > way to do this is to keep hosting the .sig files on download.qemu.org > and to redirect the actual tarballs to a file hosting provider. A way to > securely publish files without hosting anything on qemu.org would be > even better though (maybe it's enough to publish signatures on the > static GitLab Pages website). If someone modifies the download files, then when you verify the sig it will be detected. It doesn't matter whether the sig is on the same host or not, because if someone modifies the sig too, then it will still fail validation. The important thing is that the user has got the right public key to verify with. IOW, hosting the .sig separately is not required. We need to ensure that our public key, however, is published & discoverable in a trustworthy place that is separate from the download server. We fail at that today because www.qemu.org and download.qemu.org are the same server. So it will be beneficial if the download site is split off from the public website, compared to our current setup. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 8:52 ` Daniel P. Berrangé @ 2021-10-04 8:53 ` Stefan Hajnoczi 0 siblings, 0 replies; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-04 8:53 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Thomas Huth, Stefan Hajnoczi, Philippe Mathieu-Daudé, qemu-devel, Eldon Stegall, michael.roth, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 1569 bytes --] On Fri, Oct 01, 2021 at 09:52:20AM +0100, Daniel P. Berrangé wrote: > On Fri, Oct 01, 2021 at 08:11:35AM +0100, Stefan Hajnoczi wrote: > > We need to keep the security of QEMU releases in mind. Mike Roth > > signs and publishes releases. Whoever facilitates or hosts the files > > should not be able to modify the files after Mike has blessed them. One > > way to do this is to keep hosting the .sig files on download.qemu.org > > and to redirect the actual tarballs to a file hosting provider. A way to > > securely publish files without hosting anything on qemu.org would be > > even better though (maybe it's enough to publish signatures on the > > static GitLab Pages website). > > If someone modifies the download files, then when you verify the sig > it will be detected. It doesn't matter whether the sig is on the same > host or not, because if someone modifies the sig too, then it will > still fail validation. The important thing is that the user has got > the right public key to verify with. > > IOW, hosting the .sig separately is not required. We need to ensure > that our public key, however, is published & discoverable in a > trustworthy place that is separate from the download server. We fail > at that today because www.qemu.org and download.qemu.org are the > same server. > > So it will be beneficial if the download site is split off from > the public website, compared to our current setup. You're right. Thanks for pointing this out. I was thinking of the .sig as a checksum but it's a signature :-). Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 7:11 ` Stefan Hajnoczi 2021-10-01 8:52 ` Daniel P. Berrangé @ 2021-10-11 15:00 ` Anthony Liguori 1 sibling, 0 replies; 19+ messages in thread From: Anthony Liguori @ 2021-10-11 15:00 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, Stefan Hajnoczi, Philippe Mathieu-Daudé, qemu-devel, Eldon Stegall, michael.roth, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 1912 bytes --] On Fri, Oct 1, 2021 at 12:20 AM Stefan Hajnoczi <stefanha@redhat.com> wrote: > On Thu, Sep 30, 2021 at 03:57:49PM +0000, Eldon Stegall wrote: > > Hello! > > I'd be happy to help with this. I'm mostly a consumer of QEMU, but > > greatly appreciate all the work this community has done, and was able > > to contribute a little by helping with QEMU advent this past year. I > > would be happy to help streamline some of this activities if that would > > be welcome, and would gratefully contribute time and resources. Hosting > > and serving data like this has been core to my recent experience. > > > > I would be happy to suggest and build out a distribution strategy for > > these packages, and believe I could cut some costs, and even convince a > > small consultancy I am a part of here that uses QEMU to foot a > > reasonable bill. > > > > A brief introduction, since I haven't had the pleasure of attending > > FOSDEM or any other QEMU meetups: I am a startup-oriented Cloud Security > > Architect, based out of Atlanta, previously with companies like > > DataStax, but now working on AWS video pipelines for a startup here. > > Thanks for joining the discussion and for running last year's QEMU > Advent Calendar, Eldon. > > Any ideas for moving download.qemu.org to a hosted service would be > appreciated! We haven't compared CDN and cloud providers closely yet. If > you have experience in this area or time to check them out, then that > would be valuable. > > QEMU has funds if there is a cost for file hosting (probably less than > $100/month). Some providers may be willing to support an open source > project for free. Possible providers include CloudFlare, Akamai, Fastly, > Microsoft Azure, Google Cloud Storage, etc. > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ Let me know if ya'll apply and I'm happy to push it through. Regards, Anthony Liguori [-- Attachment #2: Type: text/html, Size: 2617 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-09-30 14:28 ` Stefan Hajnoczi 2021-09-30 15:57 ` Eldon Stegall @ 2021-10-01 7:24 ` Thomas Huth 2021-10-01 10:34 ` Gerd Hoffmann 1 sibling, 1 reply; 19+ messages in thread From: Thomas Huth @ 2021-10-01 7:24 UTC (permalink / raw) To: Stefan Hajnoczi, Stefan Hajnoczi Cc: qemu-devel, michael.roth, Daniel Berrange, Philippe Mathieu-Daudé, Paolo Bonzini On 30/09/2021 16.28, Stefan Hajnoczi wrote: > On Thu, Sep 30, 2021 at 3:08 PM Stefan Hajnoczi <stefanha@redhat.com> wrote: >> >> Hi Mike, >> QEMU downloads are currently hosted on qemu.org's Apache web server. >> Paolo and I were discussing ways to reduce qemu.org network traffic to >> save money and eventually turn off the qemu.org server since there is no >> full-time sysadmin for it. I'd like to discuss moving QEMU downloads to >> GitLab Releases. > > Daniel Berrange posted this in another discussion: > > "gitlab releases have a per-file size limit that is somewhere on the > order of 10 MB IIRC. Our release tarballs are 100+ MB, so I don't > believe that's going to be viable. > > The gitlab package registry doesn't directly support plain file > downloads afaik, and is also size limited to 50 MB per package Maybe we should also discuss again whether we could decrease the size of the release tarballs. Someone recently mentioned that we could e.g. remove the edk2 sources from the tarballs - edk2 is licensed under a BSD-style license, so we are not forced to ship its sources in our tarballs (unlike the other firmwares which are licensed under GPL). Or maybe we could also finally move the firmware images completely into a separate tarball instead? Thomas ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 7:24 ` Thomas Huth @ 2021-10-01 10:34 ` Gerd Hoffmann 2021-10-01 12:50 ` Philippe Mathieu-Daudé 0 siblings, 1 reply; 19+ messages in thread From: Gerd Hoffmann @ 2021-10-01 10:34 UTC (permalink / raw) To: Thomas Huth Cc: Daniel Berrange, Stefan Hajnoczi, Philippe Mathieu-Daudé, qemu-devel, Stefan Hajnoczi, michael.roth, Paolo Bonzini Hi, > Maybe we should also discuss again whether we could decrease the size of the > release tarballs. Someone recently mentioned that we could e.g. remove the > edk2 sources from the tarballs - edk2 is licensed under a BSD-style license, > so we are not forced to ship its sources in our tarballs (unlike the other > firmwares which are licensed under GPL). > > Or maybe we could also finally move the firmware images completely into a > separate tarball instead? The idea of a separate qemu-firmware repo is floating around for a while already but never really took off. Now with hosting moved to gitlab we should be able to run automated firmware binary builds in CI and purge all firmware from the qemu git repo (both submodules and binaries) and add a script fetching those from gitlab instead. take care, Gerd ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 10:34 ` Gerd Hoffmann @ 2021-10-01 12:50 ` Philippe Mathieu-Daudé 0 siblings, 0 replies; 19+ messages in thread From: Philippe Mathieu-Daudé @ 2021-10-01 12:50 UTC (permalink / raw) To: Gerd Hoffmann, Thomas Huth Cc: Daniel Berrange, Stefan Hajnoczi, qemu-devel, Stefan Hajnoczi, michael.roth, Paolo Bonzini On 10/1/21 12:34, Gerd Hoffmann wrote:>> Maybe we should also discuss again whether we could decrease the size of the >> release tarballs. Someone recently mentioned that we could e.g. remove the >> edk2 sources from the tarballs - edk2 is licensed under a BSD-style license, >> so we are not forced to ship its sources in our tarballs (unlike the other >> firmwares which are licensed under GPL). >> >> Or maybe we could also finally move the firmware images completely into a >> separate tarball instead? > > The idea of a separate qemu-firmware repo is floating around for a > while already but never really took off. > > Now with hosting moved to gitlab we should be able to run automated > firmware binary builds in CI and purge all firmware from the qemu git > repo (both submodules and binaries) and add a script fetching those > from gitlab instead. We already have some firmware jobs which might be used as template: .gitlab-ci.d/edk2.yml .gitlab-ci.d/opensbi.yml ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-09-30 13:40 Moving QEMU downloads to GitLab Releases? Stefan Hajnoczi 2021-09-30 14:28 ` Stefan Hajnoczi @ 2021-10-01 7:39 ` Philippe Mathieu-Daudé 2021-10-04 9:01 ` Stefan Hajnoczi 1 sibling, 1 reply; 19+ messages in thread From: Philippe Mathieu-Daudé @ 2021-10-01 7:39 UTC (permalink / raw) To: Stefan Hajnoczi, michael.roth Cc: Paolo Bonzini, Thomas Huth, Daniel Berrange, qemu-devel On 9/30/21 15:40, Stefan Hajnoczi wrote: > Hi Mike, > QEMU downloads are currently hosted on qemu.org's Apache web server. > Paolo and I were discussing ways to reduce qemu.org network traffic to > save money and eventually turn off the qemu.org server since there is no > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > GitLab Releases. > > Since you create and sign QEMU releases I wanted to see what you think > about the idea. GitLab Releases has two ways of creating release assets: > archiving a git tree and attaching arbitrary binaries. The > scripts/make-release script fetches submodules and generates version > files, so it may be necessary to treat QEMU tarballs as arbitrary > binaries instead of simply letting GitLab create git tree archives: > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries > > Releases can be uploaded via the GitLab API from your local machine or > deployed as a GitLab CI job. Uploading from your local machine would be > the closest to the current workflow. > > In the long term we could have a CI job that automatically publishes > QEMU releases when a new qemu.git tag is pushed. The release process > could be fully automated so that manual steps are no longer necessary, > although we'd have to trust GitLab with QEMU GPG signing keys. Before having to trust a SaaS for GPG signing, could this work? - make-release script should produce a reproducible tarball in a gitlab job, along with a file containing the tarball hash. - Mike (or whoever is responsible of releases) keeps doing a local manual build - Mike checks the local hash matches the Gitlab artifact hash - Mike signs its local build/hash and uses the GitLab API to upload that .sig to job artifacts - we can have an extra manual job that checks the tarball.sig (hash and pubkey) and on success deploys updating the download page, adding the new release Regards, Phil. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-01 7:39 ` Philippe Mathieu-Daudé @ 2021-10-04 9:01 ` Stefan Hajnoczi 2021-10-04 19:34 ` Michael Roth 0 siblings, 1 reply; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-04 9:01 UTC (permalink / raw) To: Philippe Mathieu-Daudé Cc: michael.roth, Thomas Huth, Daniel Berrange, qemu-devel, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 2202 bytes --] On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote: > On 9/30/21 15:40, Stefan Hajnoczi wrote: > > Hi Mike, > > QEMU downloads are currently hosted on qemu.org's Apache web server. > > Paolo and I were discussing ways to reduce qemu.org network traffic to > > save money and eventually turn off the qemu.org server since there is no > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > > GitLab Releases. > > > > Since you create and sign QEMU releases I wanted to see what you think > > about the idea. GitLab Releases has two ways of creating release assets: > > archiving a git tree and attaching arbitrary binaries. The > > scripts/make-release script fetches submodules and generates version > > files, so it may be necessary to treat QEMU tarballs as arbitrary > > binaries instead of simply letting GitLab create git tree archives: > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries > > > > Releases can be uploaded via the GitLab API from your local machine or > > deployed as a GitLab CI job. Uploading from your local machine would be > > the closest to the current workflow. > > > > In the long term we could have a CI job that automatically publishes > > QEMU releases when a new qemu.git tag is pushed. The release process > > could be fully automated so that manual steps are no longer necessary, > > although we'd have to trust GitLab with QEMU GPG signing keys. > > Before having to trust a SaaS for GPG signing, could this work? > > - make-release script should produce a reproducible tarball in a > gitlab job, along with a file containing the tarball hash. > > - Mike (or whoever is responsible of releases) keeps doing a local > manual build > > - Mike checks the local hash matches the Gitlab artifact hash > > - Mike signs its local build/hash and uses the GitLab API to upload > that .sig to job artifacts > > - we can have an extra manual job that checks the tarball.sig > (hash and pubkey) and on success deploys updating the download > page, adding the new release I wonder what Mike sees as the way forward. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-04 9:01 ` Stefan Hajnoczi @ 2021-10-04 19:34 ` Michael Roth 2021-10-05 13:29 ` Stefan Hajnoczi 0 siblings, 1 reply; 19+ messages in thread From: Michael Roth @ 2021-10-04 19:34 UTC (permalink / raw) To: Philippe Mathieu-Daudé, Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, qemu-devel, Paolo Bonzini Quoting Stefan Hajnoczi (2021-10-04 04:01:22) > On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote: > > On 9/30/21 15:40, Stefan Hajnoczi wrote: > > > Hi Mike, > > > QEMU downloads are currently hosted on qemu.org's Apache web server. > > > Paolo and I were discussing ways to reduce qemu.org network traffic to > > > save money and eventually turn off the qemu.org server since there is no > > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > > > GitLab Releases. > > > > > > Since you create and sign QEMU releases I wanted to see what you think > > > about the idea. GitLab Releases has two ways of creating release assets: > > > archiving a git tree and attaching arbitrary binaries. The > > > scripts/make-release script fetches submodules and generates version > > > files, so it may be necessary to treat QEMU tarballs as arbitrary > > > binaries instead of simply letting GitLab create git tree archives: > > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries > > > > > > Releases can be uploaded via the GitLab API from your local machine or > > > deployed as a GitLab CI job. Uploading from your local machine would be > > > the closest to the current workflow. > > > > > > In the long term we could have a CI job that automatically publishes > > > QEMU releases when a new qemu.git tag is pushed. The release process > > > could be fully automated so that manual steps are no longer necessary, > > > although we'd have to trust GitLab with QEMU GPG signing keys. > > > > Before having to trust a SaaS for GPG signing, could this work? > > > > - make-release script should produce a reproducible tarball in a > > gitlab job, along with a file containing the tarball hash. > > > > - Mike (or whoever is responsible of releases) keeps doing a local > > manual build > > > > - Mike checks the local hash matches the Gitlab artifact hash > > > > - Mike signs its local build/hash and uses the GitLab API to upload > > that .sig to job artifacts > > > > - we can have an extra manual job that checks the tarball.sig > > (hash and pubkey) and on success deploys updating the download > > page, adding the new release > > I wonder what Mike sees as the way forward. Hi Stefan, Philippe, In general I like the idea, since we could also have the CI do the full gamut of testing against the binaries built from said tarball, so the Release Person can just regenerate the tarball and provide a sig to attest that it came from the proper sources. Currently I do make check and make check-acceptance and a few other sanity checks, which I guess would no longer be needed then. But I think the more immediate issue is where/how to host those tarballs. Even moving all the ROMs/capstone out of the source tree still results in an xz-compressed tarball size ~25MB, which is well above the 10MB limit mentioned earlier. We could break it out into target-specific tarballs, maybe further into softmmu/user variants, but that sounds painful for both users and maintainers who need to deal with the resulting source tree differences. What I'm wondering is whether we could just use the archive files generated by gitlab when we tag our releases? E.g.: https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2 If we paired that with an in-tree script similar to make-release for users to download individual ROM sources/subprojects used for a particular tagged release, would that be sufficient for GPL compliance and verifying what sources the binaries were built from? Are there any other considerations WRT ROMs/etc.? With something like that in place, Release Person could just do a git checkout, verify the Maintainer's sig/tag (in case we don't necessarily trust the git host), generate the tarball, verify the hash matches what gitlab published (or verify/diff individual files if the bz/gz hashes require a specific environment), then sign the gitlab tarball and add the sig to qemu.org download page along with a link the gitlab-generated tarball. We could also publish the Maintainer and Release Person public keys on qemu.org download page so users can verify this as well using the same process. Users that want the additional sources can then do it locally via above-mentioned script, which would be part of the now-signed tarball and so could be 'trusted' assuming the individual project hosts weren't compromised (which is still an assumption that's needed with the current process anyway). I guess the main question is who is using the ROM/BIOS sources in the tarballs, and would this 2-step process work for those users? If there are distros relying on them then maybe there are some more logistics to consider since the make-release downloads are both time-consuming and prone to network errors/stalls since it relies on the availability of a good number of different hosts. -Mike > > Stefan ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-04 19:34 ` Michael Roth @ 2021-10-05 13:29 ` Stefan Hajnoczi 2021-10-11 7:21 ` Gerd Hoffmann 0 siblings, 1 reply; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-05 13:29 UTC (permalink / raw) To: Michael Roth, Gerd Hoffmann Cc: Paolo Bonzini, Thomas Huth, Daniel Berrange, Philippe Mathieu-Daudé, qemu-devel [-- Attachment #1: Type: text/plain, Size: 5830 bytes --] On Mon, Oct 04, 2021 at 02:34:49PM -0500, Michael Roth wrote: > Quoting Stefan Hajnoczi (2021-10-04 04:01:22) > > On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote: > > > On 9/30/21 15:40, Stefan Hajnoczi wrote: > > > > Hi Mike, > > > > QEMU downloads are currently hosted on qemu.org's Apache web server. > > > > Paolo and I were discussing ways to reduce qemu.org network traffic to > > > > save money and eventually turn off the qemu.org server since there is no > > > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > > > > GitLab Releases. > > > > > > > > Since you create and sign QEMU releases I wanted to see what you think > > > > about the idea. GitLab Releases has two ways of creating release assets: > > > > archiving a git tree and attaching arbitrary binaries. The > > > > scripts/make-release script fetches submodules and generates version > > > > files, so it may be necessary to treat QEMU tarballs as arbitrary > > > > binaries instead of simply letting GitLab create git tree archives: > > > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries > > > > > > > > Releases can be uploaded via the GitLab API from your local machine or > > > > deployed as a GitLab CI job. Uploading from your local machine would be > > > > the closest to the current workflow. > > > > > > > > In the long term we could have a CI job that automatically publishes > > > > QEMU releases when a new qemu.git tag is pushed. The release process > > > > could be fully automated so that manual steps are no longer necessary, > > > > although we'd have to trust GitLab with QEMU GPG signing keys. > > > > > > Before having to trust a SaaS for GPG signing, could this work? > > > > > > - make-release script should produce a reproducible tarball in a > > > gitlab job, along with a file containing the tarball hash. > > > > > > - Mike (or whoever is responsible of releases) keeps doing a local > > > manual build > > > > > > - Mike checks the local hash matches the Gitlab artifact hash > > > > > > - Mike signs its local build/hash and uses the GitLab API to upload > > > that .sig to job artifacts > > > > > > - we can have an extra manual job that checks the tarball.sig > > > (hash and pubkey) and on success deploys updating the download > > > page, adding the new release > > > > I wonder what Mike sees as the way forward. > > Hi Stefan, Philippe, > > In general I like the idea, since we could also have the CI do the full > gamut of testing against the binaries built from said tarball, so the > Release Person can just regenerate the tarball and provide a sig to > attest that it came from the proper sources. Currently I do make check > and make check-acceptance and a few other sanity checks, which I guess > would no longer be needed then. > > But I think the more immediate issue is where/how to host those > tarballs. Even moving all the ROMs/capstone out of the source tree still > results in an xz-compressed tarball size ~25MB, which is well above the > 10MB limit mentioned earlier. We could break it out into target-specific > tarballs, maybe further into softmmu/user variants, but that sounds > painful for both users and maintainers who need to deal with the > resulting source tree differences. > > What I'm wondering is whether we could just use the archive files > generated by gitlab when we tag our releases? E.g.: > > https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2 > > If we paired that with an in-tree script similar to make-release for > users to download individual ROM sources/subprojects used for a particular > tagged release, would that be sufficient for GPL compliance and verifying > what sources the binaries were built from? Are there any other > considerations WRT ROMs/etc.? > > With something like that in place, Release Person could just do a git > checkout, verify the Maintainer's sig/tag (in case we don't necessarily > trust the git host), generate the tarball, verify the hash matches what > gitlab published (or verify/diff individual files if the bz/gz hashes > require a specific environment), then sign the gitlab tarball and add > the sig to qemu.org download page along with a link the gitlab-generated > tarball. > > We could also publish the Maintainer and Release Person public keys on > qemu.org download page so users can verify this as well using the same > process. > > Users that want the additional sources can then do it locally via > above-mentioned script, which would be part of the now-signed tarball > and so could be 'trusted' assuming the individual project hosts weren't > compromised (which is still an assumption that's needed with the current > process anyway). > > I guess the main question is who is using the ROM/BIOS sources in the > tarballs, and would this 2-step process work for those users? If there > are distros relying on them then maybe there are some more logistics to > consider since the make-release downloads are both time-consuming and > prone to network errors/stalls since it relies on the availability of a > good number of different hosts. Great, maybe Gerd can comment on splitting out firmware. QEMU mirrors firmware sources on GitLab too. We're able to provide the same level of download availability on our mirror firmware repos as for the main qemu.git repo. I think qemu.git should include information about where to get the firmware sources in order to comply with the GPL. I'm not sure if the git submodules metadata counts from a GPL perspective, but that seems enough since it has both a mirror repo URL and the exact hash used to build the binary blob. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-05 13:29 ` Stefan Hajnoczi @ 2021-10-11 7:21 ` Gerd Hoffmann 2021-10-11 10:58 ` Stefan Hajnoczi 0 siblings, 1 reply; 19+ messages in thread From: Gerd Hoffmann @ 2021-10-11 7:21 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, Michael Roth, Philippe Mathieu-Daudé, qemu-devel, Paolo Bonzini Hi, > > I guess the main question is who is using the ROM/BIOS sources in the > > tarballs, and would this 2-step process work for those users? If there > > are distros relying on them then maybe there are some more logistics to > > consider since the make-release downloads are both time-consuming and > > prone to network errors/stalls since it relies on the availability of a > > good number of different hosts. > > Great, maybe Gerd can comment on splitting out firmware. I think the binaries are mostly a convenience feature for developers. Distros typically build from source anyway, and typically they build from upstream source instead of qemu submodule. The idea to split them out to a separate repo is exists for quite a while. I have an old qemu-firmware repo laying around on my disk, which basically moves roms/ + submodules and the binaries built from it over. I think with the switch to gitlab it doesn't make sense any more to commit pre-built firmware binaries to a git repo. Instead we should build the firmware in gitlab ci, i.e. effectively move the build rules we have right now in roms/Makefile to .gitlab-ci.d/, then publish the firmware binaries as build artifacts or gitlab pages. When done just remove the pre-build binaries from git and add a helper script to fetch firmware binaries from gitlab instead. > QEMU mirrors firmware sources on GitLab too. We're able to provide the > same level of download availability on our mirror firmware repos as for > the main qemu.git repo. I think enabling CI for the firmware mirrors should work given that it is possible to specify a custom CI/CD configuration file, i.e. use something like /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml or /qemu-project/qemu-firmware/seabios.yml as config file for the /qemu-project/seabios/ mirror. Then we can publish binaries using gitlab pages at https://qemu-project.gitlab.io/seabios/ That way we also don't need the roms/ submodules any more, i.e. we can remove both sources and binaries for the firmware from the release tarballs. take care, Gerd ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-11 7:21 ` Gerd Hoffmann @ 2021-10-11 10:58 ` Stefan Hajnoczi 2021-10-11 14:28 ` Warner Losh 0 siblings, 1 reply; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-11 10:58 UTC (permalink / raw) To: Gerd Hoffmann Cc: Thomas Huth, Daniel Berrange, Michael Roth, Philippe Mathieu-Daudé, qemu-devel, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 2654 bytes --] On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote: > Hi, > > > > I guess the main question is who is using the ROM/BIOS sources in the > > > tarballs, and would this 2-step process work for those users? If there > > > are distros relying on them then maybe there are some more logistics to > > > consider since the make-release downloads are both time-consuming and > > > prone to network errors/stalls since it relies on the availability of a > > > good number of different hosts. > > > > Great, maybe Gerd can comment on splitting out firmware. > > I think the binaries are mostly a convenience feature for developers. > Distros typically build from source anyway, and typically they build > from upstream source instead of qemu submodule. > > The idea to split them out to a separate repo is exists for quite a > while. I have an old qemu-firmware repo laying around on my disk, which > basically moves roms/ + submodules and the binaries built from it over. > > I think with the switch to gitlab it doesn't make sense any more to > commit pre-built firmware binaries to a git repo. Instead we should > build the firmware in gitlab ci, i.e. effectively move the build rules > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the > firmware binaries as build artifacts or gitlab pages. > > When done just remove the pre-build binaries from git and add a helper > script to fetch firmware binaries from gitlab instead. > > > QEMU mirrors firmware sources on GitLab too. We're able to provide the > > same level of download availability on our mirror firmware repos as for > > the main qemu.git repo. > > I think enabling CI for the firmware mirrors should work given that it > is possible to specify a custom CI/CD configuration file, i.e. use > something like > > /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml > > or > > /qemu-project/qemu-firmware/seabios.yml > > as config file for the > > /qemu-project/seabios/ > > mirror. Then we can publish binaries using gitlab pages at > > https://qemu-project.gitlab.io/seabios/ > > That way we also don't need the roms/ submodules any more, i.e. we > can remove both sources and binaries for the firmware from the > release tarballs. Thanks! For developer convenience it would be nice to avoid manual steps after git clone qemu.git. Maybe ./configure should check for firmware blobs and automatically download them. There could be a ./configure --disable-firmware-download option that distros can use to skip the download when building everything from source. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-11 10:58 ` Stefan Hajnoczi @ 2021-10-11 14:28 ` Warner Losh 2021-10-11 15:46 ` Stefan Hajnoczi 0 siblings, 1 reply; 19+ messages in thread From: Warner Losh @ 2021-10-11 14:28 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, Michael Roth, QEMU Developers, Philippe Mathieu-Daudé, Gerd Hoffmann, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 3739 bytes --] On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote: > On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote: > > Hi, > > > > > > I guess the main question is who is using the ROM/BIOS sources in the > > > > tarballs, and would this 2-step process work for those users? If > there > > > > are distros relying on them then maybe there are some more logistics > to > > > > consider since the make-release downloads are both time-consuming and > > > > prone to network errors/stalls since it relies on the availability > of a > > > > good number of different hosts. > > > > > > Great, maybe Gerd can comment on splitting out firmware. > > > > I think the binaries are mostly a convenience feature for developers. > > Distros typically build from source anyway, and typically they build > > from upstream source instead of qemu submodule. > > > > The idea to split them out to a separate repo is exists for quite a > > while. I have an old qemu-firmware repo laying around on my disk, which > > basically moves roms/ + submodules and the binaries built from it over. > > > > I think with the switch to gitlab it doesn't make sense any more to > > commit pre-built firmware binaries to a git repo. Instead we should > > build the firmware in gitlab ci, i.e. effectively move the build rules > > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the > > firmware binaries as build artifacts or gitlab pages. > > > > When done just remove the pre-build binaries from git and add a helper > > script to fetch firmware binaries from gitlab instead. > > > > > QEMU mirrors firmware sources on GitLab too. We're able to provide the > > > same level of download availability on our mirror firmware repos as for > > > the main qemu.git repo. > > > > I think enabling CI for the firmware mirrors should work given that it > > is possible to specify a custom CI/CD configuration file, i.e. use > > something like > > > > /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml > > > > or > > > > /qemu-project/qemu-firmware/seabios.yml > > > > as config file for the > > > > /qemu-project/seabios/ > > > > mirror. Then we can publish binaries using gitlab pages at > > > > https://qemu-project.gitlab.io/seabios/ > > > > That way we also don't need the roms/ submodules any more, i.e. we > > can remove both sources and binaries for the firmware from the > > release tarballs. > > Thanks! > > For developer convenience it would be nice to avoid manual steps after > git clone qemu.git. Maybe ./configure should check for firmware blobs > and automatically download them. There could be a ./configure > --disable-firmware-download option that distros can use to skip the > download when building everything from source. > One thing to keep in mind about the downstream consumers: Many of them have two phases to their build process that run asynchronously to each other. There is a fetch phase that grabs everything, and a build phase that builds the binaries. The second phase may not have access to the internet for a variety of reasons (these days being a security measure, for good or ill). Please make sure that any such plans allow for this model. Today, that's all dealt with by grabbing tarballs from github which is how the submodules are dealt with. So long as the images had well known names, and could be fetched with the normal wget/cgit/fetch programs, that would suffice. Requiring use of some weird bespoke script would cause friction for the downstreams using qemu. So while I'm all for making things a little more independent, let's not do it in a way that makes life difficult for downstreams. There are more there than just a couple of big distro builders. Warner [-- Attachment #2: Type: text/html, Size: 4819 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-11 14:28 ` Warner Losh @ 2021-10-11 15:46 ` Stefan Hajnoczi 2021-10-11 16:27 ` Gerd Hoffmann 0 siblings, 1 reply; 19+ messages in thread From: Stefan Hajnoczi @ 2021-10-11 15:46 UTC (permalink / raw) To: Warner Losh Cc: Thomas Huth, Daniel Berrange, Michael Roth, QEMU Developers, Philippe Mathieu-Daudé, Gerd Hoffmann, Paolo Bonzini [-- Attachment #1: Type: text/plain, Size: 4167 bytes --] On Mon, Oct 11, 2021 at 08:28:34AM -0600, Warner Losh wrote: > On Mon, Oct 11, 2021 at 4:59 AM Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > On Mon, Oct 11, 2021 at 09:21:30AM +0200, Gerd Hoffmann wrote: > > > Hi, > > > > > > > > I guess the main question is who is using the ROM/BIOS sources in the > > > > > tarballs, and would this 2-step process work for those users? If > > there > > > > > are distros relying on them then maybe there are some more logistics > > to > > > > > consider since the make-release downloads are both time-consuming and > > > > > prone to network errors/stalls since it relies on the availability > > of a > > > > > good number of different hosts. > > > > > > > > Great, maybe Gerd can comment on splitting out firmware. > > > > > > I think the binaries are mostly a convenience feature for developers. > > > Distros typically build from source anyway, and typically they build > > > from upstream source instead of qemu submodule. > > > > > > The idea to split them out to a separate repo is exists for quite a > > > while. I have an old qemu-firmware repo laying around on my disk, which > > > basically moves roms/ + submodules and the binaries built from it over. > > > > > > I think with the switch to gitlab it doesn't make sense any more to > > > commit pre-built firmware binaries to a git repo. Instead we should > > > build the firmware in gitlab ci, i.e. effectively move the build rules > > > we have right now in roms/Makefile to .gitlab-ci.d/, then publish the > > > firmware binaries as build artifacts or gitlab pages. > > > > > > When done just remove the pre-build binaries from git and add a helper > > > script to fetch firmware binaries from gitlab instead. > > > > > > > QEMU mirrors firmware sources on GitLab too. We're able to provide the > > > > same level of download availability on our mirror firmware repos as for > > > > the main qemu.git repo. > > > > > > I think enabling CI for the firmware mirrors should work given that it > > > is possible to specify a custom CI/CD configuration file, i.e. use > > > something like > > > > > > /qemu-project/qemu/.gitlab-ci.d/firmware/seabios.yml > > > > > > or > > > > > > /qemu-project/qemu-firmware/seabios.yml > > > > > > as config file for the > > > > > > /qemu-project/seabios/ > > > > > > mirror. Then we can publish binaries using gitlab pages at > > > > > > https://qemu-project.gitlab.io/seabios/ > > > > > > That way we also don't need the roms/ submodules any more, i.e. we > > > can remove both sources and binaries for the firmware from the > > > release tarballs. > > > > Thanks! > > > > For developer convenience it would be nice to avoid manual steps after > > git clone qemu.git. Maybe ./configure should check for firmware blobs > > and automatically download them. There could be a ./configure > > --disable-firmware-download option that distros can use to skip the > > download when building everything from source. > > > > One thing to keep in mind about the downstream consumers: Many > of them have two phases to their build process that run asynchronously > to each other. There is a fetch phase that grabs everything, and a build > phase that builds the binaries. The second phase may not have access > to the internet for a variety of reasons (these days being a security > measure, for good or ill). Please make sure that any such plans > allow for this model. > > Today, that's all dealt with by grabbing tarballs from github which > is how the submodules are dealt with. So long as the images > had well known names, and could be fetched with the normal > wget/cgit/fetch programs, that would suffice. Requiring use of > some weird bespoke script would cause friction for the downstreams > using qemu. > > So while I'm all for making things a little more independent, > let's not do it in a way that makes life difficult for downstreams. > There are more there than just a couple of big distro builders. I think this is fine. Firmware blobs aren't needed to build QEMU, only to run the system emulator. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Moving QEMU downloads to GitLab Releases? 2021-10-11 15:46 ` Stefan Hajnoczi @ 2021-10-11 16:27 ` Gerd Hoffmann 0 siblings, 0 replies; 19+ messages in thread From: Gerd Hoffmann @ 2021-10-11 16:27 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Thomas Huth, Daniel Berrange, Michael Roth, Philippe Mathieu-Daudé, QEMU Developers, Paolo Bonzini, Warner Losh Hi, > > So while I'm all for making things a little more independent, > > let's not do it in a way that makes life difficult for downstreams. > > There are more there than just a couple of big distro builders. > > I think this is fine. Firmware blobs aren't needed to build QEMU, only > to run the system emulator. Yep. But doing firmware builds outside gitlab should be easy, so we should probably keep the core logic in a script / Makefile / whatever (simliar to todays roms/Makefile), then simply invoke that from gitlab ci yaml files. take care, Gerd ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2021-10-11 16:30 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-30 13:40 Moving QEMU downloads to GitLab Releases? Stefan Hajnoczi 2021-09-30 14:28 ` Stefan Hajnoczi 2021-09-30 15:57 ` Eldon Stegall 2021-10-01 7:11 ` Stefan Hajnoczi 2021-10-01 8:52 ` Daniel P. Berrangé 2021-10-04 8:53 ` Stefan Hajnoczi 2021-10-11 15:00 ` Anthony Liguori 2021-10-01 7:24 ` Thomas Huth 2021-10-01 10:34 ` Gerd Hoffmann 2021-10-01 12:50 ` Philippe Mathieu-Daudé 2021-10-01 7:39 ` Philippe Mathieu-Daudé 2021-10-04 9:01 ` Stefan Hajnoczi 2021-10-04 19:34 ` Michael Roth 2021-10-05 13:29 ` Stefan Hajnoczi 2021-10-11 7:21 ` Gerd Hoffmann 2021-10-11 10:58 ` Stefan Hajnoczi 2021-10-11 14:28 ` Warner Losh 2021-10-11 15:46 ` Stefan Hajnoczi 2021-10-11 16:27 ` Gerd Hoffmann
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.