[Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Igor Mammedov]

As I recall there were issues with FAT driver licensing in edk2,
but I've heard there were some changes in that regard.

Is there any other reasons why we are not putting subj.
in QEMU tree like we do with SeaBIOS and other roms?

PS:
Mostly I'm interested in having AVMF ROM in QEMU,
to make testing of ACPI tables (tests/bios-tables-test)
for virt-arm work (it has 0 coverage right now).
It would be useful to have OVMF blob as well so we could
notice if some OVMF related stuff is broken or changed
unexpectedly.

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Peter Maydell]

On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com> wrote:
> As I recall there were issues with FAT driver licensing in edk2,
> but I've heard there were some changes in that regard.
>
> Is there any other reasons why we are not putting subj.
> in QEMU tree like we do with SeaBIOS and other roms?

I suspect the primary answer is "nobody who's willing to
maintain, test and update the resulting binary blobs has
stepped forward to say they want to do so" :-)

(I think that shipping them in the QEMU tree would be
nice but is principally a convenience for our direct
users, since distros are going to want to build their
own ROM blobs from source anyway.)

thanks
-- PMM

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Laszlo Ersek]

On 08/07/17 16:40, Peter Maydell wrote:
> On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com> wrote:
>> As I recall there were issues with FAT driver licensing in edk2,
>> but I've heard there were some changes in that regard.
>>
>> Is there any other reasons why we are not putting subj.
>> in QEMU tree like we do with SeaBIOS and other roms?
> 
> I suspect the primary answer is "nobody who's willing to
> maintain, test and update the resulting binary blobs has
> stepped forward to say they want to do so" :-)
> 
> (I think that shipping them in the QEMU tree would be
> nice but is principally a convenience for our direct
> users, since distros are going to want to build their
> own ROM blobs from source anyway.)

I agree that OVMF and ArmVirtQemu firmware binaries (and matching
varstore templates, likely compressed) should be bundled with QEMU.
There are no license-related reasons left that would prevent this.

Please let us discuss this when Gerd returns from vacation. (CC'ing Gerd.)

NB, upstream edk2 does not do releases. Similarly to iPXE -- edk2 is
production quality at all times! /me ducks ;)

Thanks
Laszlo

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Gerd Hoffmann]

On Mon, 2017-08-07 at 18:51 +0200, Laszlo Ersek wrote:
> On 08/07/17 16:40, Peter Maydell wrote:
> > On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com>
> > wrote:
> > > As I recall there were issues with FAT driver licensing in edk2,
> > > but I've heard there were some changes in that regard.
> > > 
> > > Is there any other reasons why we are not putting subj.
> > > in QEMU tree like we do with SeaBIOS and other roms?
> > 
> > I suspect the primary answer is "nobody who's willing to
> > maintain, test and update the resulting binary blobs has
> > stepped forward to say they want to do so" :-)
> > 
> > (I think that shipping them in the QEMU tree would be
> > nice but is principally a convenience for our direct
> > users, since distros are going to want to build their
> > own ROM blobs from source anyway.)
> 
> I agree that OVMF and ArmVirtQemu firmware binaries (and matching
> varstore templates, likely compressed) should be bundled with QEMU.
> There are no license-related reasons left that would prevent this.
> 
> Please let us discuss this when Gerd returns from vacation. (CC'ing
> Gerd.)

slighly oldish wip branch:
https://www.kraxel.org/cgit/qemu/log/?h=work/edk2

Related question (as the edk2 blobs are pretty big):  Do we want commit
this to the qemu repo directly?   Or should we create a qemu-firmware
repo for the precompiled blobs and hook it up as submodule?

cheers,
  Gerd

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Igor Mammedov]

On Mon, 21 Aug 2017 13:58:26 +0200
Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Mon, 2017-08-07 at 18:51 +0200, Laszlo Ersek wrote:
> > On 08/07/17 16:40, Peter Maydell wrote:  
> > > On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com>
> > > wrote:  
> > > > As I recall there were issues with FAT driver licensing in edk2,
> > > > but I've heard there were some changes in that regard.
> > > > 
> > > > Is there any other reasons why we are not putting subj.
> > > > in QEMU tree like we do with SeaBIOS and other roms?  
> > > 
> > > I suspect the primary answer is "nobody who's willing to
> > > maintain, test and update the resulting binary blobs has
> > > stepped forward to say they want to do so" :-)
> > > 
> > > (I think that shipping them in the QEMU tree would be
> > > nice but is principally a convenience for our direct
> > > users, since distros are going to want to build their
> > > own ROM blobs from source anyway.)  
> > 
> > I agree that OVMF and ArmVirtQemu firmware binaries (and matching
> > varstore templates, likely compressed) should be bundled with QEMU.
> > There are no license-related reasons left that would prevent this.
> > 
> > Please let us discuss this when Gerd returns from vacation. (CC'ing
> > Gerd.)  
> 
> slighly oldish wip branch:
> https://www.kraxel.org/cgit/qemu/log/?h=work/edk2
> 
> Related question (as the edk2 blobs are pretty big):  Do we want commit
> this to the qemu repo directly?   Or should we create a qemu-firmware
> repo for the precompiled blobs and hook it up as submodule?
I suppose both would work for me (making make check work),
though I'd prefer in tree blob (as I didn't have much experience with submodules).
So it's up to you to pick way that makes it simpler to maintain.

> 
> cheers,
>   Gerd
> 

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Peter Maydell]

On 21 August 2017 at 12:58, Gerd Hoffmann <kraxel@redhat.com> wrote:
> Related question (as the edk2 blobs are pretty big):  Do we want commit
> this to the qemu repo directly?   Or should we create a qemu-firmware
> repo for the precompiled blobs and hook it up as submodule?

Having all our precompiled blobs be in a submodule would
maybe be handy for properly keeping them separate from the
QEMU code -- would that be useful for downstream distro
packagers?

In any case this seems like a good opportunity to write down
exactly what our policy about ROM blobs is (eg where they
are, how we update them, requirement for source, whether
we allow source to be in a submodule not on qemu.org...)

thanks
-- PMM

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Alex Bennée]

Gerd Hoffmann <kraxel@redhat.com> writes:

> On Mon, 2017-08-07 at 18:51 +0200, Laszlo Ersek wrote:
>> On 08/07/17 16:40, Peter Maydell wrote:
>> > On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com>
>> > wrote:
>> > > As I recall there were issues with FAT driver licensing in edk2,
>> > > but I've heard there were some changes in that regard.
>> > >
>> > > Is there any other reasons why we are not putting subj.
>> > > in QEMU tree like we do with SeaBIOS and other roms?
>> >
>> > I suspect the primary answer is "nobody who's willing to
>> > maintain, test and update the resulting binary blobs has
>> > stepped forward to say they want to do so" :-)
>> >
>> > (I think that shipping them in the QEMU tree would be
>> > nice but is principally a convenience for our direct
>> > users, since distros are going to want to build their
>> > own ROM blobs from source anyway.)
>>
>> I agree that OVMF and ArmVirtQemu firmware binaries (and matching
>> varstore templates, likely compressed) should be bundled with QEMU.
>> There are no license-related reasons left that would prevent this.
>>
>> Please let us discuss this when Gerd returns from vacation. (CC'ing
>> Gerd.)
>
> slighly oldish wip branch:
> https://www.kraxel.org/cgit/qemu/log/?h=work/edk2
>
> Related question (as the edk2 blobs are pretty big):  Do we want commit
> this to the qemu repo directly?   Or should we create a qemu-firmware
> repo for the precompiled blobs and hook it up as submodule?

How big? Things like github tend to get picky with blobs over 50Mb and
git isn't really set-up for large binary bits hence things like git-lfs.
Unfortunately there seems to be multiple solutions to the blobs in git
problem.

We should also take some care to reproducibility so they can be updated
by someone other than the original author.

>
> cheers,
>   Gerd


--
Alex Bennée

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Laszlo Ersek]

On 08/21/17 16:03, Alex Bennée wrote:
> 
> Gerd Hoffmann <kraxel@redhat.com> writes:
> 
>> On Mon, 2017-08-07 at 18:51 +0200, Laszlo Ersek wrote:
>>> On 08/07/17 16:40, Peter Maydell wrote:
>>>> On 7 August 2017 at 15:31, Igor Mammedov <imammedo@redhat.com>
>>>> wrote:
>>>>> As I recall there were issues with FAT driver licensing in edk2,
>>>>> but I've heard there were some changes in that regard.
>>>>>
>>>>> Is there any other reasons why we are not putting subj.
>>>>> in QEMU tree like we do with SeaBIOS and other roms?
>>>>
>>>> I suspect the primary answer is "nobody who's willing to
>>>> maintain, test and update the resulting binary blobs has
>>>> stepped forward to say they want to do so" :-)
>>>>
>>>> (I think that shipping them in the QEMU tree would be
>>>> nice but is principally a convenience for our direct
>>>> users, since distros are going to want to build their
>>>> own ROM blobs from source anyway.)
>>>
>>> I agree that OVMF and ArmVirtQemu firmware binaries (and matching
>>> varstore templates, likely compressed) should be bundled with QEMU.
>>> There are no license-related reasons left that would prevent this.
>>>
>>> Please let us discuss this when Gerd returns from vacation. (CC'ing
>>> Gerd.)
>>
>> slighly oldish wip branch:
>> https://www.kraxel.org/cgit/qemu/log/?h=work/edk2
>>
>> Related question (as the edk2 blobs are pretty big):  Do we want commit
>> this to the qemu repo directly?   Or should we create a qemu-firmware
>> repo for the precompiled blobs and hook it up as submodule?
> 
> How big?

The ArmVirtQemu firmware binary, and its matching varstore template,
need to be padded to 64MB each, after the edk2 build process completes,
before the fw binary, and a variable store created from the varstore
template, can be passed to QEMU.

So, "pretty big".

Thanks
Laszlo

> Things like github tend to get picky with blobs over 50Mb and
> git isn't really set-up for large binary bits hence things like git-lfs.
> Unfortunately there seems to be multiple solutions to the blobs in git
> problem.
> 
> We should also take some care to reproducibility so they can be updated
> by someone other than the original author.
> 
>>
>> cheers,
>>   Gerd
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Gerd Hoffmann]

  Hi,

> > How big?
> 
> The ArmVirtQemu firmware binary, and its matching varstore template,
> need to be padded to 64MB each, after the edk2 build process
> completes,
> before the fw binary, and a variable store created from the varstore
> template, can be passed to QEMU.
> 
> So, "pretty big".

Most of it is zeros though, so it wouldn't change on updates.
Actual content is almost 3MB (code + vars template combined).

cheers,
  Gerd

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Gerd Hoffmann]

  Hi,

> Having all our precompiled blobs be in a submodule would
> maybe be handy for properly keeping them separate from the
> QEMU code -- would that be useful for downstream distro
> packagers?

I like the clear separation.

Also you don't have to fetch the firmware submodule if you don't need
the prebuilds.

It would be a good opportunity to cleanup the mess we have in pc-bios/
right now.

Clear disadvantage is that firmware updates will become more
complicated then as two repos must be updated.  So either two pull
requests per firmware update, or write access to the firmware binary
repo for all firmware maintainers.

Or maybe make the firmware sources submodules of the firmware binary
repo instead of the qemu repo.

> In any case this seems like a good opportunity to write down
> exactly what our policy about ROM blobs is (eg where they
> are, how we update them, requirement for source, whether
> we allow source to be in a submodule not on qemu.org...)

Tried to write down the current state.

cheers,
  Gerd

---------------------- docs/firmware.txt ------------------------

prebuilt firmware in qemu
=========================

The process of building firmware isn't as easy as building qemu
itself, for example because you must have cross compilers installed
when building firmware for other architectures.  So qemu ships
prebuilt firmware binaries for convenience reasons.


directories
-----------

pc-bios/
    Firmware binaries.
    Also other files like keymaps and logos which are copied to
    $prefix/share/qemu by "make install".

pc-bios/optionrom/
    Firmware sources, part of the qemu project.

roms/
    Firmware sources, third-party projects, as git submodules.


third-party git repos
---------------------

For third party firmware sources it is recommended to have a git
mirror on git.qemu.org.  In case the firmware license is GPL the git
mirror is mandatory for GPL compilance reasons:  We ship binaries, so
we must provide sources too.


building the firmware
---------------------

Firmware builds should be done using rules in roms/Makefile.  That
serves as documentation how the firmware is built, and it also makes
firmware updates easier for the maintainer.

"make -C roms" prints a list of firmware build targets.
"make -C roms $target" kicks a build.


notes for specific firmwares
----------------------------

ipxe: see https://wiki.qemu.org/IpxeDownstreamForQemu

Re: [Qemu-devel] AVMF & OVMF blobs in QEMU tree???

[Laszlo Ersek]

On 08/24/17 12:55, Gerd Hoffmann wrote:
>   Hi,
> 
>> Having all our precompiled blobs be in a submodule would
>> maybe be handy for properly keeping them separate from the
>> QEMU code -- would that be useful for downstream distro
>> packagers?
> 
> I like the clear separation.
> 
> Also you don't have to fetch the firmware submodule if you don't need
> the prebuilds.
> 
> It would be a good opportunity to cleanup the mess we have in pc-bios/
> right now.
> 
> Clear disadvantage is that firmware updates will become more
> complicated then as two repos must be updated.  So either two pull
> requests per firmware update, or write access to the firmware binary
> repo for all firmware maintainers.
> 
> Or maybe make the firmware sources submodules of the firmware binary
> repo instead of the qemu repo.
> 
>> In any case this seems like a good opportunity to write down
>> exactly what our policy about ROM blobs is (eg where they
>> are, how we update them, requirement for source, whether
>> we allow source to be in a submodule not on qemu.org...)
> 
> Tried to write down the current state.
> 
> cheers,
>   Gerd
> 
> ---------------------- docs/firmware.txt ------------------------
> 
> prebuilt firmware in qemu
> =========================
> 
> The process of building firmware isn't as easy as building qemu
> itself, for example because you must have cross compilers installed
> when building firmware for other architectures.  So qemu ships
> prebuilt firmware binaries for convenience reasons.
> 
> 
> directories
> -----------
> 
> pc-bios/
>     Firmware binaries.
>     Also other files like keymaps and logos which are copied to
>     $prefix/share/qemu by "make install".
> 
> pc-bios/optionrom/
>     Firmware sources, part of the qemu project.
> 
> roms/
>     Firmware sources, third-party projects, as git submodules.
> 
> 
> third-party git repos
> ---------------------
> 
> For third party firmware sources it is recommended to have a git
> mirror on git.qemu.org.  In case the firmware license is GPL the git
> mirror is mandatory for GPL compilance reasons:  We ship binaries, so
> we must provide sources too.
> 
> 
> building the firmware
> ---------------------
> 
> Firmware builds should be done using rules in roms/Makefile.  That
> serves as documentation how the firmware is built, and it also makes
> firmware updates easier for the maintainer.
> 
> "make -C roms" prints a list of firmware build targets.
> "make -C roms $target" kicks a build.
> 
> 
> notes for specific firmwares
> ----------------------------
> 
> ipxe: see https://wiki.qemu.org/IpxeDownstreamForQemu
> 

Looks good to me!

Reviewed-by: Laszlo Ersek <lersek@redhat.com>

Thanks
Laszlo