[Qemu-devel] bundling edk2 platform firmware images with QEMU

[Qemu-devel] bundling edk2 platform firmware images with QEMU

[Laszlo Ersek]

Hi,

I have a mostly-ready-for-posting patch set for $SUBJECT. My question is
what QEMU release I should be targeting with it.

The Soft Feature Freeze for 4.0 is on 2019-03-12. Here's why that's a
bit inconvenient for me.

The upcoming EDK2 stable release is edk2-stable201903, and it is planned
for... today.

  https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201903-tag-planning

But, it's being blocked (at least one CVE fix still needs merging, but
there could be something else too). I don't know what that will mean for
the actual tag date. Maybe next Monday (the 11th)?

In my series, I'd like to advance QEMU's roms/edk2 submodule to this new
release. But that might leave us with 1 day before the QEMU 4.0 Soft
Feature Freeze (see above), i.e., for me to post the series, and for a
submaintainer to send a pullreq with it. That's a bit too tight.

I'm not in a mortal rush to get this into 4.0, but the next release
cycles (in three months, approximately?) might align similarly, between
edk2 and QEMU. It would be best to avoid QEMU carrying edk2 platform
firmware that is at all times at least three months old. The main reason
is that CVEs tend to exist, for both edk2 proper, and for the specific
OpenSSL release that is bundled with the given edk2 stable tag. And edk2
doesn't yet have stable *branches*.

Should we try to squeeze my set into 4.0 (possibly moving the Soft
Feature Freeze), or just aim for 4.1?

Also, who'd be the maintainer to queue my set? I mostly thought of Gerd,
due to his work on iPXE and SeaBIOS. Here's the current diffstat:

 Makefile                                       |  17 +-
 pc-bios/README                                 |  11 +
 pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
 pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
 pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
 pc-bios/descriptors/60-edk2-arm.json           |  31 +++
 pc-bios/descriptors/60-edk2-i386.json          |  33 +++
 pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
 pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
 pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
 pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
 pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
 pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
 pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
 pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
 pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
 pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
 roms/Makefile                                  |   9 +-
 roms/Makefile.edk2                             | 134 ++++++++++
 roms/edk2-build.sh                             |  52 ++++
 roms/edk2-funcs.sh                             | 265 ++++++++++++++++++++
 tests/uefi-test-tools/build.sh                 |  97 +------
 22 files changed, 898 insertions(+), 94 deletions(-)

Thanks,
Laszlo

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Daniel P. Berrangé]

On Fri, Mar 08, 2019 at 02:09:53PM +0100, Laszlo Ersek wrote:
> Hi,
> 
> I have a mostly-ready-for-posting patch set for $SUBJECT. My question is
> what QEMU release I should be targeting with it.
> 
> The Soft Feature Freeze for 4.0 is on 2019-03-12. Here's why that's a
> bit inconvenient for me.
> 
> The upcoming EDK2 stable release is edk2-stable201903, and it is planned
> for... today.
> 
>   https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201903-tag-planning
> 
> But, it's being blocked (at least one CVE fix still needs merging, but
> there could be something else too). I don't know what that will mean for
> the actual tag date. Maybe next Monday (the 11th)?
> 
> In my series, I'd like to advance QEMU's roms/edk2 submodule to this new
> release. But that might leave us with 1 day before the QEMU 4.0 Soft
> Feature Freeze (see above), i.e., for me to post the series, and for a
> submaintainer to send a pullreq with it. That's a bit too tight.

IMHO if you advanced the submodule hash to a nearly-released version
before freeze, it would be fine to then advance it again to the actually
released commit hash during QEMU freeze, because presumably the EDK
changes are similarly bugfix only at this point in its release process.

> 
> I'm not in a mortal rush to get this into 4.0, but the next release
> cycles (in three months, approximately?) might align similarly, between
> edk2 and QEMU. It would be best to avoid QEMU carrying edk2 platform
> firmware that is at all times at least three months old. The main reason
> is that CVEs tend to exist, for both edk2 proper, and for the specific
> OpenSSL release that is bundled with the given edk2 stable tag. And edk2
> doesn't yet have stable *branches*.
> 
> Should we try to squeeze my set into 4.0 (possibly moving the Soft
> Feature Freeze), or just aim for 4.1?
> 
> Also, who'd be the maintainer to queue my set? I mostly thought of Gerd,
> due to his work on iPXE and SeaBIOS. Here's the current diffstat:
> 
>  Makefile                                       |  17 +-
>  pc-bios/README                                 |  11 +
>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes


Yikes, am I really reading those sizes right ? The biggest ROMs there
are 64 MB, so this is proposing to add ~206 MB of binaries to the
pc-bios directory ?

I think this is a very undesirable thing to do.

Consider that we'll need to refresh those ROMs multiple times a year to
track updates or security fixes. It will result in our .git repo size
growing massively over time. I don't think people will like cloning
multi-GB  sized repos after a few years of ROM updates.

As I've mentioned before, I think QEMU should get out of the business
of distributing ROMs in its primary released qemu-x.x.x.tar.gz archives,
and provide them as a separate tar.gz bundle. Even better if we can
move the existing ROMS out of git too, though we have to consider how
developers biulding from git would access the ROMs & know when they
need to acquire new copies.

The main important things to version control are the build config and
the git submodule version information.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Daniel P. Berrangé]

On Fri, Mar 08, 2019 at 01:42:11PM +0000, Daniel P. Berrangé wrote:
> On Fri, Mar 08, 2019 at 02:09:53PM +0100, Laszlo Ersek wrote:
> > Hi,
> > 
> > I have a mostly-ready-for-posting patch set for $SUBJECT. My question is
> > what QEMU release I should be targeting with it.
> > 
> > The Soft Feature Freeze for 4.0 is on 2019-03-12. Here's why that's a
> > bit inconvenient for me.
> > 
> > The upcoming EDK2 stable release is edk2-stable201903, and it is planned
> > for... today.
> > 
> >   https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201903-tag-planning
> > 
> > But, it's being blocked (at least one CVE fix still needs merging, but
> > there could be something else too). I don't know what that will mean for
> > the actual tag date. Maybe next Monday (the 11th)?
> > 
> > In my series, I'd like to advance QEMU's roms/edk2 submodule to this new
> > release. But that might leave us with 1 day before the QEMU 4.0 Soft
> > Feature Freeze (see above), i.e., for me to post the series, and for a
> > submaintainer to send a pullreq with it. That's a bit too tight.
> 
> IMHO if you advanced the submodule hash to a nearly-released version
> before freeze, it would be fine to then advance it again to the actually
> released commit hash during QEMU freeze, because presumably the EDK
> changes are similarly bugfix only at this point in its release process.
> 
> > 
> > I'm not in a mortal rush to get this into 4.0, but the next release
> > cycles (in three months, approximately?) might align similarly, between
> > edk2 and QEMU. It would be best to avoid QEMU carrying edk2 platform
> > firmware that is at all times at least three months old. The main reason
> > is that CVEs tend to exist, for both edk2 proper, and for the specific
> > OpenSSL release that is bundled with the given edk2 stable tag. And edk2
> > doesn't yet have stable *branches*.
> > 
> > Should we try to squeeze my set into 4.0 (possibly moving the Soft
> > Feature Freeze), or just aim for 4.1?
> > 
> > Also, who'd be the maintainer to queue my set? I mostly thought of Gerd,
> > due to his work on iPXE and SeaBIOS. Here's the current diffstat:
> > 
> >  Makefile                                       |  17 +-
> >  pc-bios/README                                 |  11 +
> >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
> >  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
> >  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
> 
> 
> Yikes, am I really reading those sizes right ? The biggest ROMs there
> are 64 MB, so this is proposing to add ~206 MB of binaries to the
> pc-bios directory ?
> 
> I think this is a very undesirable thing to do.
> 
> Consider that we'll need to refresh those ROMs multiple times a year to
> track updates or security fixes. It will result in our .git repo size
> growing massively over time. I don't think people will like cloning
> multi-GB  sized repos after a few years of ROM updates.
> 
> As I've mentioned before, I think QEMU should get out of the business
> of distributing ROMs in its primary released qemu-x.x.x.tar.gz archives,
> and provide them as a separate tar.gz bundle. Even better if we can
> move the existing ROMS out of git too, though we have to consider how
> developers biulding from git would access the ROMs & know when they
> need to acquire new copies.
> 
> The main important things to version control are the build config and
> the git submodule version information.

Oh, and the json firmware description files make sense. Even without
the blobs, those metadata files set guidance on what variants of EDK
a vendor should aim to ship and can likely be used "as-is" by the
vendor's own downstream firmware biulds.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Laszlo Ersek]

On 03/08/19 14:42, Daniel P. Berrangé wrote:
> On Fri, Mar 08, 2019 at 02:09:53PM +0100, Laszlo Ersek wrote:
>> Hi,
>>
>> I have a mostly-ready-for-posting patch set for $SUBJECT. My question is
>> what QEMU release I should be targeting with it.
>>
>> The Soft Feature Freeze for 4.0 is on 2019-03-12. Here's why that's a
>> bit inconvenient for me.
>>
>> The upcoming EDK2 stable release is edk2-stable201903, and it is planned
>> for... today.
>>
>>   https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201903-tag-planning
>>
>> But, it's being blocked (at least one CVE fix still needs merging, but
>> there could be something else too). I don't know what that will mean for
>> the actual tag date. Maybe next Monday (the 11th)?
>>
>> In my series, I'd like to advance QEMU's roms/edk2 submodule to this new
>> release. But that might leave us with 1 day before the QEMU 4.0 Soft
>> Feature Freeze (see above), i.e., for me to post the series, and for a
>> submaintainer to send a pullreq with it. That's a bit too tight.
> 
> IMHO if you advanced the submodule hash to a nearly-released version
> before freeze, it would be fine to then advance it again to the actually
> released commit hash during QEMU freeze, because presumably the EDK
> changes are similarly bugfix only at this point in its release process.

Great idea! Thanks!

And, indeed, I had shamelessly stolen the SFF and HFF concepts from
QEMU, when I proposed them for edk2.

Edk2 doesn't have maintainer trees and pull requests, so I tried to
"adopt" the SFF and HFF definitions. We should likely improve those over
time, as the edk2 workflow too (hopefully!) matures.

https://github.com/tianocore/tianocore.github.io/wiki/SoftFeatureFreeze
https://github.com/tianocore/tianocore.github.io/wiki/HardFeatureFreeze

>> I'm not in a mortal rush to get this into 4.0, but the next release
>> cycles (in three months, approximately?) might align similarly, between
>> edk2 and QEMU. It would be best to avoid QEMU carrying edk2 platform
>> firmware that is at all times at least three months old. The main reason
>> is that CVEs tend to exist, for both edk2 proper, and for the specific
>> OpenSSL release that is bundled with the given edk2 stable tag. And edk2
>> doesn't yet have stable *branches*.
>>
>> Should we try to squeeze my set into 4.0 (possibly moving the Soft
>> Feature Freeze), or just aim for 4.1?
>>
>> Also, who'd be the maintainer to queue my set? I mostly thought of Gerd,
>> due to his work on iPXE and SeaBIOS. Here's the current diffstat:
>>
>>  Makefile                                       |  17 +-
>>  pc-bios/README                                 |  11 +
>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
> 
> 
> Yikes, am I really reading those sizes right ? The biggest ROMs there
> are 64 MB, so this is proposing to add ~206 MB of binaries to the
> pc-bios directory ?

Sort of.

This is because of how the "virt" machine type's pflash chips are sized.
The edk2 build produces a 2MB firmware executable and a 768KB varstore
template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad
either [*], and (b) if we used qcow2 with compression, then libvirt
couldn't deal with the images [**]. Hence we have to pad the files
manually, after the edk2 build completes.

[*] We've just been discussing auto-padding in a parallel patch from
Alex and Markus (the latest version is "[PATCH v7] pflash: Require
backend size to match device, improve errors", in which the padding has
already been dropped). But such padding would only work for read-only
pflash mappings anyway, not for variable store files.

[**] The root cause for not using qcow2 with pflash images is that qcow2
would make them candidates for "savevm" to dump live memory contents
into them, which is Very Bad (TM).

> I think this is a very undesirable thing to do.

Yup, not ideal.

> Consider that we'll need to refresh those ROMs multiple times a year to
> track updates or security fixes. It will result in our .git repo size
> growing massively over time.

Actually, it's not *that* bad. Earlier I investigated how the formatted
variant of the patch (adding these blobs) would look like.

First, I compressed all the "edk2-*fd" files (individually) with "gzip
--best", just to get a byte count. That yielded almost precisely 9MB, in
total.

Second, the formatted patch, adding these blobs, is 12.6MB. Taking the
4/3 blowup factor from base64 encoding, the "unexplained overhead" is
not that huge (0.6 MB).

Obviously such a patch is still impossible to post to the list, hence
I'd format the series with "--no-binary" (which produces the view that
you get in "gitk" too). My point is that the initial internal storage
requirement, especially after git-gc / packing, would not exceed 9MB by
much.

There are two other mitigating factors:

- "git checkout" is smart enough to punch holes into the files in the
working directory.

$ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd
2.1M    edk2-arm-code.fd
2.1M    edk2-aarch64-code.fd
772K    edk2-arm-vars.fd
4.8M    total

This happens despite the fact that the Makefile performs the padding
with "cat /dev/zero" and "head".

- whenever we update binary files and format those changes to patch
emails, git only formats the actual changes. Given that the padding
(which covers most of these files) is virtually never expected to
change, the incremental updates (which are compressed as well) shouldn't
be huge.

> I don't think people will like cloning
> multi-GB  sized repos after a few years of ROM updates.

It shouldn't be more than a few MB with every update, in effect.

> As I've mentioned before, I think QEMU should get out of the business
> of distributing ROMs in its primary released qemu-x.x.x.tar.gz archives,
> and provide them as a separate tar.gz bundle. Even better if we can
> move the existing ROMS out of git too, though we have to consider how
> developers biulding from git would access the ROMs & know when they
> need to acquire new copies.

I'm 100% fine with not bundling any firmware binaries for end-users;
however I've been doing this work because Igor needs full platform
firmware binaries for ACPI unit tests. In particular, in aarch64 guests,
there is no ACPI without UEFI, and in UEFI guests, finding the ACPI
entry point (RSD PTR) is convoluted enough that it needs dedicated guest
support.

The UEFI test helper application for that is already upstream (and it is
really small, see 4d3f50eb489e..503bb0b975ab). This application "preps"
the guest RAM so that the qtest program can find the ACPI entry point
via raw memory reads.

But, the UEFI helper app (which is provided as a UEFI-bootable ISO
image) must be launched against a full platform firmware, from the qtest
case. Originally we thought the qtest case could use firmware images
right from the build host, but locating those on any particular host
proved a mess, so Igor asked me to add these images.

> The main important things to version control are the build config and
> the git submodule version information.

Gerd has been offering an external firmware repo at
<https://www.kraxel.org/repos/> for years now, and the build scripts (in
the RPM spec file) are tracked in a publicly cloneable git repo. I'm
entirely okay with your proposal, but personally I wouldn't like to
invest time into (further) developing such a service.

(I've done the current work because it looked necessary after quite a
bit of navel gazing for alternative approaches -- nobody else appeared
enthusiastic about implementing the one approach that seemed workable.)

Thanks,
Laszlo

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Daniel P. Berrangé]

On Fri, Mar 08, 2019 at 03:48:07PM +0100, Laszlo Ersek wrote:
> On 03/08/19 14:42, Daniel P. Berrangé wrote:
> >>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> >>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> >>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> >>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
> >>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
> >>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
> >>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
> >>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
> >>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
> > 
> > 
> > Yikes, am I really reading those sizes right ? The biggest ROMs there
> > are 64 MB, so this is proposing to add ~206 MB of binaries to the
> > pc-bios directory ?
> 
> Sort of.
> 
> This is because of how the "virt" machine type's pflash chips are sized.
> The edk2 build produces a 2MB firmware executable and a 768KB varstore
> template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad
> either [*], and (b) if we used qcow2 with compression, then libvirt
> couldn't deal with the images [**]. Hence we have to pad the files
> manually, after the edk2 build completes.
> 
> [*] We've just been discussing auto-padding in a parallel patch from
> Alex and Markus (the latest version is "[PATCH v7] pflash: Require
> backend size to match device, improve errors", in which the padding has
> already been dropped). But such padding would only work for read-only
> pflash mappings anyway, not for variable store files.
> 
> [**] The root cause for not using qcow2 with pflash images is that qcow2
> would make them candidates for "savevm" to dump live memory contents
> into them, which is Very Bad (TM).

If using  qcow2 compressed images for ROMs is the desired best answer,
then surely we can figure out a way to fix this savevm problem ? Even
if it is just an internal hack there must be a way we can mark a qcow2
file as being used by the ROM so savevm ignores it.

We've not been very motivated to solve this before since "savevm" command
is supposedly deprecated to be replaced by a sane QMP variant, but we've
been waiting 5 years for that to happen, so in the mean time a hack to
resolve the problem with firmware images feels prudent.

> > Consider that we'll need to refresh those ROMs multiple times a year to
> > track updates or security fixes. It will result in our .git repo size
> > growing massively over time.
> 
> Actually, it's not *that* bad. Earlier I investigated how the formatted
> variant of the patch (adding these blobs) would look like.
> 
> First, I compressed all the "edk2-*fd" files (individually) with "gzip
> --best", just to get a byte count. That yielded almost precisely 9MB, in
> total.
> 
> Second, the formatted patch, adding these blobs, is 12.6MB. Taking the
> 4/3 blowup factor from base64 encoding, the "unexplained overhead" is
> not that huge (0.6 MB).

Ok, that's less alarming.

Though we're still basically doubling the size of the pc-bios dir
which is already significant.

I wonder how much of out .git/objects 230 MB size is due to the current
ROMs.

> - "git checkout" is smart enough to punch holes into the files in the
> working directory.
> 
> $ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd
> 2.1M    edk2-arm-code.fd
> 2.1M    edk2-aarch64-code.fd
> 772K    edk2-arm-vars.fd
> 4.8M    total

That's nice.

> This happens despite the fact that the Makefile performs the padding
> with "cat /dev/zero" and "head".

You can resize a file using holes with dd if you set count=0 and
tell it to seek in output eg something like this probably works:

  dd if=/dev/zero of=firmware.img seek=64 bs=1M count=0



> I'm 100% fine with not bundling any firmware binaries for end-users;
> however I've been doing this work because Igor needs full platform
> firmware binaries for ACPI unit tests. In particular, in aarch64 guests,
> there is no ACPI without UEFI, and in UEFI guests, finding the ACPI
> entry point (RSD PTR) is convoluted enough that it needs dedicated guest
> support.

Yes tricky. I don't have a particularly good answer for that, especially
if it is a test that you'd expect all developers to run every time. If
it is a rarely run test then it could be justified in having it download
large blobs from a lookaside cache similar to how we download disk images
for *BSD tests.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Igor Mammedov]

On Fri, 8 Mar 2019 15:06:12 +0000
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Fri, Mar 08, 2019 at 03:48:07PM +0100, Laszlo Ersek wrote:
> > On 03/08/19 14:42, Daniel P. Berrangé wrote:  
> > >>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> > >>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> > >>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> > >>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
> > >>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
> > >>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
> > >>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
> > >>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
> > >>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes  
> > > 
> > > 
> > > Yikes, am I really reading those sizes right ? The biggest ROMs there
> > > are 64 MB, so this is proposing to add ~206 MB of binaries to the
> > > pc-bios directory ?  
> > 
> > Sort of.
> > 
> > This is because of how the "virt" machine type's pflash chips are sized.
> > The edk2 build produces a 2MB firmware executable and a 768KB varstore
> > template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad
> > either [*], and (b) if we used qcow2 with compression, then libvirt
> > couldn't deal with the images [**]. Hence we have to pad the files
> > manually, after the edk2 build completes.
> > 
> > [*] We've just been discussing auto-padding in a parallel patch from
> > Alex and Markus (the latest version is "[PATCH v7] pflash: Require
> > backend size to match device, improve errors", in which the padding has
> > already been dropped). But such padding would only work for read-only
> > pflash mappings anyway, not for variable store files.
> > 
> > [**] The root cause for not using qcow2 with pflash images is that qcow2
> > would make them candidates for "savevm" to dump live memory contents
> > into them, which is Very Bad (TM).  
> 
> If using  qcow2 compressed images for ROMs is the desired best answer,
> then surely we can figure out a way to fix this savevm problem ? Even
> if it is just an internal hack there must be a way we can mark a qcow2
> file as being used by the ROM so savevm ignores it.
> 
> We've not been very motivated to solve this before since "savevm" command
> is supposedly deprecated to be replaced by a sane QMP variant, but we've
> been waiting 5 years for that to happen, so in the mean time a hack to
> resolve the problem with firmware images feels prudent.
> 
> > > Consider that we'll need to refresh those ROMs multiple times a year to
> > > track updates or security fixes. It will result in our .git repo size
> > > growing massively over time.  
> > 
> > Actually, it's not *that* bad. Earlier I investigated how the formatted
> > variant of the patch (adding these blobs) would look like.
> > 
> > First, I compressed all the "edk2-*fd" files (individually) with "gzip
> > --best", just to get a byte count. That yielded almost precisely 9MB, in
> > total.
> > 
> > Second, the formatted patch, adding these blobs, is 12.6MB. Taking the
> > 4/3 blowup factor from base64 encoding, the "unexplained overhead" is
> > not that huge (0.6 MB).  
> 
> Ok, that's less alarming.
> 
> Though we're still basically doubling the size of the pc-bios dir
> which is already significant.
> 
> I wonder how much of out .git/objects 230 MB size is due to the current
> ROMs.
> 
> > - "git checkout" is smart enough to punch holes into the files in the
> > working directory.
> > 
> > $ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd
> > 2.1M    edk2-arm-code.fd
> > 2.1M    edk2-aarch64-code.fd
> > 772K    edk2-arm-vars.fd
> > 4.8M    total  
> 
> That's nice.
> 
> > This happens despite the fact that the Makefile performs the padding
> > with "cat /dev/zero" and "head".  
> 
> You can resize a file using holes with dd if you set count=0 and
> tell it to seek in output eg something like this probably works:
> 
>   dd if=/dev/zero of=firmware.img seek=64 bs=1M count=0
> 
> 
> 
> > I'm 100% fine with not bundling any firmware binaries for end-users;
> > however I've been doing this work because Igor needs full platform
> > firmware binaries for ACPI unit tests. In particular, in aarch64 guests,
> > there is no ACPI without UEFI, and in UEFI guests, finding the ACPI
> > entry point (RSD PTR) is convoluted enough that it needs dedicated guest
> > support.  
> 
> Yes tricky. I don't have a particularly good answer for that, especially
> if it is a test that you'd expect all developers to run every time. If
> it is a rarely run test then it could be justified in having it download
> large blobs from a lookaside cache similar to how we download disk images
> for *BSD tests.

I'd say  it's intended to run on every 'make check' for bios-tables-test
(arm/x86 variants) like we do now with Seabios. The goal is to catch
regressions in ACPI table (as we have zero coverage there).
Secondary motivation is to help with merging NEMU fork. Intel generalizes
a bunch ACPI code across x86 and ARM boards and I'm not ready to manually
test every iteration posted on list.
 
The last time we talked where to put firmware, we've got to consensus
to bundle UEFI within qemu like we do with other firmware.
Question of moving firmwares somewhere else is still open but it's
a separate topic and shouldn't block us from properly testing arm board
now (and other related projects that depend on it).

As for soft-freeze, maybe it's acceptable to merge this series during it
as its primary(the only) user would be 'make check'. I'll respin tests
after rebasing previous iteration on top of this series.

> Regards,
> Daniel

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Laszlo Ersek]

On 03/08/19 16:06, Daniel P. Berrangé wrote:
> On Fri, Mar 08, 2019 at 03:48:07PM +0100, Laszlo Ersek wrote:
>> On 03/08/19 14:42, Daniel P. Berrangé wrote:
>>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>>>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
>>>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>>>
>>>
>>> Yikes, am I really reading those sizes right ? The biggest ROMs there
>>> are 64 MB, so this is proposing to add ~206 MB of binaries to the
>>> pc-bios directory ?
>>
>> Sort of.
>>
>> This is because of how the "virt" machine type's pflash chips are sized.
>> The edk2 build produces a 2MB firmware executable and a 768KB varstore
>> template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad
>> either [*], and (b) if we used qcow2 with compression, then libvirt
>> couldn't deal with the images [**]. Hence we have to pad the files
>> manually, after the edk2 build completes.
>>
>> [*] We've just been discussing auto-padding in a parallel patch from
>> Alex and Markus (the latest version is "[PATCH v7] pflash: Require
>> backend size to match device, improve errors", in which the padding has
>> already been dropped). But such padding would only work for read-only
>> pflash mappings anyway, not for variable store files.
>>
>> [**] The root cause for not using qcow2 with pflash images is that qcow2
>> would make them candidates for "savevm" to dump live memory contents
>> into them, which is Very Bad (TM).
> 
> If using  qcow2 compressed images for ROMs is the desired best answer,
> then surely we can figure out a way to fix this savevm problem ? Even
> if it is just an internal hack there must be a way we can mark a qcow2
> file as being used by the ROM so savevm ignores it.

This was one of the first ideas (if not "the" first one) that came up years ago (in January 2016), and it was rejected. I'm not saying it's wrong, but it was rejected.

* [Qemu-devel] [PATCH 1/1] blk: do not select PFLASH device for internal snapshot
  https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01824.html
  http://mid.mail-archive.com/1452578622-4492-1-git-send-email-den@openvz.org

* https://bugzilla.redhat.com/show_bug.cgi?id=1180955

* https://bugzilla.redhat.com/show_bug.cgi?id=1214187

> We've not been very motivated to solve this before

We gave up in frustration, not because we didn't care.

The kludge that we put in place (in libvirtd) simply prevents us from "going there" now, through libvirt anyway:

* https://bugzilla.redhat.com/show_bug.cgi?id=1431852
* commit 9e2465834f4b ("qemu: snapshot: Forbid internal snapshots with pflash firmware", 2017-03-01)

> since "savevm" command
> is supposedly deprecated to be replaced by a sane QMP variant, but we've
> been waiting 5 years for that to happen, so in the mean time a hack to
> resolve the problem with firmware images feels prudent.

I've attempted this route multiple times in the past, speaking in general. Folks generally keep rejecting patches they have rejected  before.

(And if they do change their minds, just because a number of years have passed, then I'm left wondering how that is reasonable.

That's not to say that I wouldn't be happy about it, of course -- I just don't have capacity for likely futile tries.)

>>> Consider that we'll need to refresh those ROMs multiple times a year to
>>> track updates or security fixes. It will result in our .git repo size
>>> growing massively over time.
>>
>> Actually, it's not *that* bad. Earlier I investigated how the formatted
>> variant of the patch (adding these blobs) would look like.
>>
>> First, I compressed all the "edk2-*fd" files (individually) with "gzip
>> --best", just to get a byte count. That yielded almost precisely 9MB, in
>> total.
>>
>> Second, the formatted patch, adding these blobs, is 12.6MB. Taking the
>> 4/3 blowup factor from base64 encoding, the "unexplained overhead" is
>> not that huge (0.6 MB).
> 
> Ok, that's less alarming.
> 
> Though we're still basically doubling the size of the pc-bios dir
> which is already significant.
> 
> I wonder how much of out .git/objects 230 MB size is due to the current
> ROMs.

I think it would be awesome if git provided some statistics like this! "What is the footprint of the full history of this set of files?"

> 
>> - "git checkout" is smart enough to punch holes into the files in the
>> working directory.
>>
>> $ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd
>> 2.1M    edk2-arm-code.fd
>> 2.1M    edk2-aarch64-code.fd
>> 772K    edk2-arm-vars.fd
>> 4.8M    total
> 
> That's nice.
> 
>> This happens despite the fact that the Makefile performs the padding
>> with "cat /dev/zero" and "head".
> 
> You can resize a file using holes with dd if you set count=0 and
> tell it to seek in output eg something like this probably works:
> 
>   dd if=/dev/zero of=firmware.img seek=64 bs=1M count=0

Well the actual command I use is

  cat FILE-TO-PAD /dev/zero | head -c 64M > PADDED-FILE

Maybe this can be reworked to

  cp FILE-TO-PAD PADDED-FILE
  [dd command with conv=notrunc?]

It would be even easier with the "truncate" utility:

  cp FILE-TO-PAD PADDED-FILE
  truncate --size=64M PADDED-FILE

"truncate" is part of GNU coreutils, so maybe I should use "truncate" after all.

I'll work this into the series.

>> I'm 100% fine with not bundling any firmware binaries for end-users;
>> however I've been doing this work because Igor needs full platform
>> firmware binaries for ACPI unit tests. In particular, in aarch64 guests,
>> there is no ACPI without UEFI, and in UEFI guests, finding the ACPI
>> entry point (RSD PTR) is convoluted enough that it needs dedicated guest
>> support.
> 
> Yes tricky. I don't have a particularly good answer for that, especially
> if it is a test that you'd expect all developers to run every time. If
> it is a rarely run test then it could be justified in having it download
> large blobs from a lookaside cache similar to how we download disk images
> for *BSD tests.

I think it would be part of "make check".

Thanks
Laszlo

Re: [Qemu-devel] bundling edk2 platform firmware images with QEMU

[Laszlo Ersek]

On 03/08/19 16:51, Igor Mammedov wrote:
> On Fri, 8 Mar 2019 15:06:12 +0000
> Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
>> On Fri, Mar 08, 2019 at 03:48:07PM +0100, Laszlo Ersek wrote:
>>> On 03/08/19 14:42, Daniel P. Berrangé wrote:  
>>>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>>>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>>>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>>>>  pc-bios/edk2-licenses.txt                      | 209 +++++++++++++++
>>>>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>>>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes  
>>>>
>>>>
>>>> Yikes, am I really reading those sizes right ? The biggest ROMs there
>>>> are 64 MB, so this is proposing to add ~206 MB of binaries to the
>>>> pc-bios directory ?  
>>>
>>> Sort of.
>>>
>>> This is because of how the "virt" machine type's pflash chips are sized.
>>> The edk2 build produces a 2MB firmware executable and a 768KB varstore
>>> template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad
>>> either [*], and (b) if we used qcow2 with compression, then libvirt
>>> couldn't deal with the images [**]. Hence we have to pad the files
>>> manually, after the edk2 build completes.
>>>
>>> [*] We've just been discussing auto-padding in a parallel patch from
>>> Alex and Markus (the latest version is "[PATCH v7] pflash: Require
>>> backend size to match device, improve errors", in which the padding has
>>> already been dropped). But such padding would only work for read-only
>>> pflash mappings anyway, not for variable store files.
>>>
>>> [**] The root cause for not using qcow2 with pflash images is that qcow2
>>> would make them candidates for "savevm" to dump live memory contents
>>> into them, which is Very Bad (TM).  
>>
>> If using  qcow2 compressed images for ROMs is the desired best answer,
>> then surely we can figure out a way to fix this savevm problem ? Even
>> if it is just an internal hack there must be a way we can mark a qcow2
>> file as being used by the ROM so savevm ignores it.
>>
>> We've not been very motivated to solve this before since "savevm" command
>> is supposedly deprecated to be replaced by a sane QMP variant, but we've
>> been waiting 5 years for that to happen, so in the mean time a hack to
>> resolve the problem with firmware images feels prudent.
>>
>>>> Consider that we'll need to refresh those ROMs multiple times a year to
>>>> track updates or security fixes. It will result in our .git repo size
>>>> growing massively over time.  
>>>
>>> Actually, it's not *that* bad. Earlier I investigated how the formatted
>>> variant of the patch (adding these blobs) would look like.
>>>
>>> First, I compressed all the "edk2-*fd" files (individually) with "gzip
>>> --best", just to get a byte count. That yielded almost precisely 9MB, in
>>> total.
>>>
>>> Second, the formatted patch, adding these blobs, is 12.6MB. Taking the
>>> 4/3 blowup factor from base64 encoding, the "unexplained overhead" is
>>> not that huge (0.6 MB).  
>>
>> Ok, that's less alarming.
>>
>> Though we're still basically doubling the size of the pc-bios dir
>> which is already significant.
>>
>> I wonder how much of out .git/objects 230 MB size is due to the current
>> ROMs.
>>
>>> - "git checkout" is smart enough to punch holes into the files in the
>>> working directory.
>>>
>>> $ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd
>>> 2.1M    edk2-arm-code.fd
>>> 2.1M    edk2-aarch64-code.fd
>>> 772K    edk2-arm-vars.fd
>>> 4.8M    total  
>>
>> That's nice.
>>
>>> This happens despite the fact that the Makefile performs the padding
>>> with "cat /dev/zero" and "head".  
>>
>> You can resize a file using holes with dd if you set count=0 and
>> tell it to seek in output eg something like this probably works:
>>
>>   dd if=/dev/zero of=firmware.img seek=64 bs=1M count=0
>>
>>
>>
>>> I'm 100% fine with not bundling any firmware binaries for end-users;
>>> however I've been doing this work because Igor needs full platform
>>> firmware binaries for ACPI unit tests. In particular, in aarch64 guests,
>>> there is no ACPI without UEFI, and in UEFI guests, finding the ACPI
>>> entry point (RSD PTR) is convoluted enough that it needs dedicated guest
>>> support.  
>>
>> Yes tricky. I don't have a particularly good answer for that, especially
>> if it is a test that you'd expect all developers to run every time. If
>> it is a rarely run test then it could be justified in having it download
>> large blobs from a lookaside cache similar to how we download disk images
>> for *BSD tests.
> 
> I'd say  it's intended to run on every 'make check' for bios-tables-test
> (arm/x86 variants) like we do now with Seabios. The goal is to catch
> regressions in ACPI table (as we have zero coverage there).
> Secondary motivation is to help with merging NEMU fork. Intel generalizes
> a bunch ACPI code across x86 and ARM boards and I'm not ready to manually
> test every iteration posted on list.
>  
> The last time we talked where to put firmware, we've got to consensus
> to bundle UEFI within qemu like we do with other firmware.
> Question of moving firmwares somewhere else is still open but it's
> a separate topic and shouldn't block us from properly testing arm board
> now (and other related projects that depend on it).
> 
> As for soft-freeze, maybe it's acceptable to merge this series during it
> as its primary(the only) user would be 'make check'. I'll respin tests
> after rebasing previous iteration on top of this series.

OK, let me try this tonight (hopefully):

(1) I'll employ "truncate" for padding the arm images,

(2) I'll follow Daniel's advice about moving forward to the current edk2
master HEAD (which is pretty close to the upcoming edk2-stable201903
release)

(3) I'll push the series to github or wherever for easy fetching, and
generate the patches (for posting) with --no-binary.

We can discuss better while seeing specifics.

Thanks!
Laszlo