[Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Daniel P. Berrange]

A while back now I posted an RFC patch that changes qemu-X.Y.Z.tar.bz2
archive to *not* have any ROMs or 3rd party libs bundled, and create a
new dist qemu-bundled-X.Y.Z.tar.bz2 for the fully bundled dist:

  http://lists.gnu.org/archive/html/qemu-devel/2017-04/msg03335.html

With 2.10 out of the way, and KVM Forum approaching, I figure now is
a reasonable time to revive the idea to see if it has support

The core issues I'm aiming to solve are:

 - Distro vendors don't want the bundled ROMs / libs. They want to
   fully build everything from source to ensure they are distributing
   clean soure & builds in compliance with the licenses. Currently they
   strip bundled bits from the build tree, but would prefer if the source
   dist did not have them either.

 - The qemu release dists get ever larger as we add more ROMS. Adding
   EFI ROM builds for i386, x86_64, and aarch64 will make the dists
   larger still.

There are the following options I see

  1. Keep existing dist, and add a new minimal one

       qemu-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs
       qemu-minimal-X.Y.Z.tar.bz2 - qemu source only

     Least impact for current non-distro users, distros just switch.

  2. Change existing dist, and add a new one with everything

       qemu-X.Y.Z.tar.bz2 - qemu source only
       qemu-full-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs

     Non-distro users need to download a different dist from what they
     have known previously, but otherwise unchanged build process.

  3. Change existing dist, and add a new one with bundled bits

       qemu-X.Y.Z.tar.bz2 - qemu source only
       qemu-addons-X.Y.Z.tar.bz2 - bundled ROMS + libs

     Non-distro users need to manually download & install an
     extra piece compared to now.

  4. Change existing dist, never distribute bundled ROMs + libs at all

       qemu-X.Y.Z.tar.bz2 - qemu source only

     Only here for completeness, not a serious suggestion

  5. Change nothing

     Continue ignoring the problem I aim to solve.


My patch did option 2, but I really open to any of 1/2/3 without a
strong preference, as long as we can get some agreement on one. I
guess option 1 might be easiest to get agreement on given its minimal
impact.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Peter Maydell]

On 31 August 2017 at 13:29, Daniel P. Berrange <berrange@redhat.com> wrote:
> A while back now I posted an RFC patch that changes qemu-X.Y.Z.tar.bz2
> archive to *not* have any ROMs or 3rd party libs bundled, and create a
> new dist qemu-bundled-X.Y.Z.tar.bz2 for the fully bundled dist:
>
>   http://lists.gnu.org/archive/html/qemu-devel/2017-04/msg03335.html
>
> With 2.10 out of the way, and KVM Forum approaching, I figure now is
> a reasonable time to revive the idea to see if it has support
>
> The core issues I'm aiming to solve are:
>
>  - Distro vendors don't want the bundled ROMs / libs. They want to
>    fully build everything from source to ensure they are distributing
>    clean soure & builds in compliance with the licenses. Currently they
>    strip bundled bits from the build tree, but would prefer if the source
>    dist did not have them either.
>
>  - The qemu release dists get ever larger as we add more ROMS. Adding
>    EFI ROM builds for i386, x86_64, and aarch64 will make the dists
>    larger still.

I think these make sense. Should we tie this into the
recent suggestion (by Gerd?) that we should put all the
rom blobs into git submodules, and otherwise generally
try to regularise our handling of blobs?

> There are the following options I see
>
>   1. Keep existing dist, and add a new minimal one
>
>        qemu-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs
>        qemu-minimal-X.Y.Z.tar.bz2 - qemu source only
>
>      Least impact for current non-distro users, distros just switch.
>
>   2. Change existing dist, and add a new one with everything
>
>        qemu-X.Y.Z.tar.bz2 - qemu source only
>        qemu-full-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs
>
>      Non-distro users need to download a different dist from what they
>      have known previously, but otherwise unchanged build process.
>
>   3. Change existing dist, and add a new one with bundled bits
>
>        qemu-X.Y.Z.tar.bz2 - qemu source only
>        qemu-addons-X.Y.Z.tar.bz2 - bundled ROMS + libs
>
>      Non-distro users need to manually download & install an
>      extra piece compared to now.

I don't use the tarballs, so I don't really have
a strong preference.

thanks
-- PMM

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Philippe Mathieu-Daudé]

On 08/31/2017 09:29 AM, Daniel P. Berrange wrote:
[...]> There are the following options I see
> 
>    1. Keep existing dist, and add a new minimal one
> 
>         qemu-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs
>         qemu-minimal-X.Y.Z.tar.bz2 - qemu source only
> 
>       Least impact for current non-distro users, distros just switch.
> 
>    2. Change existing dist, and add a new one with everything
> 
>         qemu-X.Y.Z.tar.bz2 - qemu source only
>         qemu-full-X.Y.Z.tar.bz2 - qemu source + bundled ROMS + libs
> 
>       Non-distro users need to download a different dist from what they
>       have known previously, but otherwise unchanged build process.
> 
>    3. Change existing dist, and add a new one with bundled bits
> 
>         qemu-X.Y.Z.tar.bz2 - qemu source only
>         qemu-addons-X.Y.Z.tar.bz2 - bundled ROMS + libs
> 
>       Non-distro users need to manually download & install an
>       extra piece compared to now.

Debian style:

linux-kbuild-4.9 - Kbuild infrastructure for Linux 4.9
linux-kbuild-4.12 - Kbuild infrastructure for Linux 4.12
linux-source - Linux kernel source (meta-package)
linux-source-4.12 - Linux kernel source for version 4.12 with Debian patches
firmware-linux - Binary firmware for various drivers in the Linux kernel 
(meta-package)
firmware-linux-nonfree - Binary firmware for various drivers in the 
Linux kernel (meta-package)
firmware-misc-nonfree - Binary firmware for various drivers in the Linux 
kernel
firmware-zd1211 - binary firmware for the zd1211rw wireless driver
firmware-bnx2 - Binary firmware for Broadcom NetXtremeII
...
qemu-efi - UEFI firmware for 64-bit ARM virtual machines
grub-firmware-qemu - GRUB firmware image for QEMU
ipxe-qemu - PXE boot firmware - ROM images for qemu
ovmf - UEFI firmware for 64-bit x86 virtual machines
vgabios - VGA BIOS software for the Bochs and Qemu emulated VGA card
seabios - Legacy BIOS implementation
sgabios - bios option rom to provide legacy serial console for x86

$ apt-cache show qemu-system-x86
Package: qemu-system-x86
Provides: qemu-system-i386, qemu-system-x86-64
Suggests: samba, vde2, qemu-block-extra, kmod, sgabios, ovmf

$ apt-cache show qemu-system-arm
Recommends: qemu-utils, ipxe-qemu, qemu-efi

This seems to me the cleaner one, the distrib choose to provide 
dependencies in the package description, a package to build, and keep 
exclusive licenses in different packages.

> 
>    4. Change existing dist, never distribute bundled ROMs + libs at all
> 
>         qemu-X.Y.Z.tar.bz2 - qemu source only
> 
>       Only here for completeness, not a serious suggestion

Linux style :)

> 
>    5. Change nothing
> 
>       Continue ignoring the problem I aim to solve.
> 
> 
> My patch did option 2, but I really open to any of 1/2/3 without a
> strong preference, as long as we can get some agreement on one. I
> guess option 1 might be easiest to get agreement on given its minimal
> impact.

It would be easier to have people building from sources to answer but 
they usually don't read this list, you might want to ask on qemu-stable.

Regards,

Phil.

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Gerd Hoffmann]

  Hi,

> > - The qemu release dists get ever larger as we add more ROMS.
> > Adding
> >    EFI ROM builds for i386, x86_64, and aarch64 will make the dists
> >    larger still.
> 
> I think these make sense. Should we tie this into the
> recent suggestion (by Gerd?) that we should put all the
> rom blobs into git submodules, and otherwise generally
> try to regularise our handling of blobs?

It surely makes sense to consider both together, so we can work out a
reasonable workflow for firmware updates and distribution.

I think it makes sense to create a separate project for the firmware
blobs.  Move over the firmware binaries and source submodules to the
new project.  This way updating both firmware sources and binaries can
be done with a single commit, like we handle this today, just in the
new firmware repo instead of the main qemu repo.

When moving only the binaries to a separate git submodule and
continuing to have the sources as main qemu repo submodules firmware
updates become more complicated.

> >   3. Change existing dist, and add a new one with bundled bits
> > 
> >        qemu-X.Y.Z.tar.bz2 - qemu source only
> >        qemu-addons-X.Y.Z.tar.bz2 - bundled ROMS + libs

With the scheme above it makes sense to have a qemu-firmware-
${version}.tar.gz addon tarball.  Maybe even two (one with the prebuilt
binaries and one with the sources).

Question is what to do with the non-firmware submodules (pixman, dtc,
more?) then.  I think they are not that big, so I doubt it is worth the
hassle to create two tarball versions.  And license-wise it isn't a
issue too.

cheers,
  Gerd

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Daniel P. Berrange]

On Fri, Sep 01, 2017 at 10:31:17AM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > - The qemu release dists get ever larger as we add more ROMS.
> > > Adding
> > >    EFI ROM builds for i386, x86_64, and aarch64 will make the dists
> > >    larger still.
> > 
> > I think these make sense. Should we tie this into the
> > recent suggestion (by Gerd?) that we should put all the
> > rom blobs into git submodules, and otherwise generally
> > try to regularise our handling of blobs?
> 
> It surely makes sense to consider both together, so we can work out a
> reasonable workflow for firmware updates and distribution.
> 
> I think it makes sense to create a separate project for the firmware
> blobs.  Move over the firmware binaries and source submodules to the
> new project.  This way updating both firmware sources and binaries can
> be done with a single commit, like we handle this today, just in the
> new firmware repo instead of the main qemu repo.
> 
> When moving only the binaries to a separate git submodule and
> continuing to have the sources as main qemu repo submodules firmware
> updates become more complicated.
> 
> > >   3. Change existing dist, and add a new one with bundled bits
> > > 
> > >        qemu-X.Y.Z.tar.bz2 - qemu source only
> > >        qemu-addons-X.Y.Z.tar.bz2 - bundled ROMS + libs
> 
> With the scheme above it makes sense to have a qemu-firmware-
> ${version}.tar.gz addon tarball.  Maybe even two (one with the prebuilt
> binaries and one with the sources).
> 
> Question is what to do with the non-firmware submodules (pixman, dtc,
> more?) then.  I think they are not that big, so I doubt it is worth the
> hassle to create two tarball versions.  And license-wise it isn't a
> issue too.

I've always wondered why we need to specialcase those modules at all.
Personally I'd just remove them entirely and let people install the
-dev packages on their distro just like they do for any other build
prerequisite of QEMU.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Gerd Hoffmann]

  Hi,

> > Question is what to do with the non-firmware submodules (pixman,
> > dtc,
> > more?) then.  I think they are not that big, so I doubt it is worth
> > the
> > hassle to create two tarball versions.  And license-wise it isn't a
> > issue too.
> 
> I've always wondered why we need to specialcase those modules at all.
> Personally I'd just remove them entirely and let people install the
> -dev packages on their distro just like they do for any other build
> prerequisite of QEMU.

Yes, we can probably do that now with pixman.  A few years back when
the submodule was added there was the problem that some distros shipped
too old versions of pixman.  Known offender: RHEL-5.  But RHEL-5 is EOL
meanwhile and I think qemu raised the minimum glib version to something
newer than what RHEL-5 ships, so it qemu doesn't build on RHEL-5
anyway.

Dunno about dtc.

cheers,
  Gerd

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Peter Maydell]

On 1 September 2017 at 10:24, Daniel P. Berrange <berrange@redhat.com> wrote:
> I've always wondered why we need to specialcase those modules at all.
> Personally I'd just remove them entirely and let people install the
> -dev packages on their distro just like they do for any other build
> prerequisite of QEMU.

Two uses: (1) we've tended to want newer versions of dtc than
distros actually had released (in fact right now we need a
pretty new dtc that not many distros have) and (2) for
hosts like OSX and Windows dependencies are not
a simple matter of "just install the distro's dev package".

thanks
-- PMM

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Gerd Hoffmann]

  Hi,

> I think it makes sense to create a separate project for the firmware
> blobs.  Move over the firmware binaries and source submodules to the
> new project.  This way updating both firmware sources and binaries
> can
> be done with a single commit, like we handle this today, just in the
> new firmware repo instead of the main qemu repo.

i.e. something like this:

https://www.kraxel.org/cgit/qemu-firmware/

(only seabios is there for now)

cheers,
  Gerd

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Daniel P. Berrange]

On Fri, Sep 01, 2017 at 04:11:44PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > I think it makes sense to create a separate project for the firmware
> > blobs.  Move over the firmware binaries and source submodules to the
> > new project.  This way updating both firmware sources and binaries
> > can
> > be done with a single commit, like we handle this today, just in the
> > new firmware repo instead of the main qemu repo.
> 
> i.e. something like this:
> 
> https://www.kraxel.org/cgit/qemu-firmware/
> 
> (only seabios is there for now)

So presumably we'd want a 'make install' target there too, to place the
various ROMS in the location QEMU expects ?

So non-distro users would need to download & install the ROMs,
and then download, build & install QEMU ?

How would you view ROM install working for devs working against GIT ?

eg when we update a ROM in the firmware directory, how do devs know to
update their firmware. Would the qemu-firmware.git be a submodule of
qemu.git, so that a 'make install' would deploy roms for devs ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] RFC: changing ROM bundling in tar dists for releases

[Gerd Hoffmann]

  Hi,

> > i.e. something like this:
> > 
> > https://www.kraxel.org/cgit/qemu-firmware/
> > 
> > (only seabios is there for now)
> 
> So presumably we'd want a 'make install' target there too, to place
> the
> various ROMS in the location QEMU expects ?

Yes.

Or place them in $prefix/share/qemu-firmware, and add that directory to
the qemu default search path.

> So non-distro users would need to download & install the ROMs,
> and then download, build & install QEMU ?

Yes.

> How would you view ROM install working for devs working against GIT ?
> 
> eg when we update a ROM in the firmware directory, how do devs know
> to
> update their firmware.  Would the qemu-firmware.git be a submodule of
> qemu.git, so that a 'make install' would deploy roms for devs ?

Not sure yet how to handle that best.  I'd tend to not make it a
submodule.  Typically there are no lockstep updates for qemu and
firmware, in most cases new firmware is only required if you want use a
new feature which needs firmware support.  So I think we don't need a
tight coupling between qemu and firmware.  Just updating the firmware
checkout now and then (or even simply using the distro builds) should
work in the vast majority of cases.

cheers,
  Gerd