[Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Kashyap Chamarthy]

Problem background
------------------

The various OVMF binary file names and paths are slightly different[+]
for each Linux distribution.  And each high-level management tool
(libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its
own approach to detect and configure the said OVMF files.  This email
thread is about arriving at some common understanding to make this a bit
more "unified" by addressing these needs in QEMU and libvirt.


Suggested approach
------------------

Based on an upstream discussion on 'virt-tools'[1] mailing list and some
Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion
to define a firmware metadata format and file (example in [1]):

  - For each firmware file we need a metadata file in a well defined
    location, e.g. /usr/share/qemu/bios/ that lists stuff like:

      - Path to the firmware binary
      - Path to the pre-built OVMF 'vars' file (if any)
      - Support architectures - associated QEMU feature flags (Secure
        Boot)
      - If the binary provides / requires SMM (System Management Mode)

     Essentially, QEMU would define[*] the file format, and provide
     metadata files from any ROMs it ships directly.  If Linux
     distributions / vendors ship extra ROMs like OVMF, etc then they
     should provide suitable metadata files.

  - Libvirt can read these metadata files and then pick the correct
    firmware binary based on the settings for the guest.

  - Management tools can then wire up the libvirt-based OVMF SB (Secure
    Boot) configuration.

[*] Open question: Who, between QEMU and libvirt, should define the said
firmware metadata format and file?


References
---------- 

[1] A past proposal from Gerd to create a sort of a "firmware registry"

    https://www.redhat.com/archives/virt-tools-list/2014-September/msg00145.html

[2] A libvirt upstream RFE bug requesting to simplify handling UEFI
    https://bugzilla.redhat.com/show_bug.cgi?id=1295146 -- RFE: provide
    a bios=uefi XML convenience option

[3] DanPB wrote a PoC in libvirt:
    https://www.redhat.com/archives/libvir-list/2016-October/msg00045.html
    -- [libvirt] [PATCH 0/3] Make UEFI firmware config simpler

    But later came to the conclusion that it is flawed, and said we go
    the route of "Suggested approach" mentioned earlier).


[*] OVMF package path names for different distributions
-------------------------------------------------------

  - Debian (https://packages.debian.org/stretch/all/ovmf/filelist)
     - /usr/share/OVMF/OVMF_CODE.fd
     - /usr/share/OVMF/OVMF_VARS.fd

  - OpenSUSE (package: "qemu-ovmf-x86_64" -- and it has *35*
    files in that package):
     - /usr/share/qemu/ovmf-x86_64-opensuse-code.bin
     - /usr/share/qemu/ovmf-x86_64-opensuse-vars.bin [...]

     Their RPM spec file gives some hints (where all the files with
     'ms' means signed with MS keys; the files with 'opensuse-4096'
     means signed with OpenSUSE 4096 bit CA keys -- I think that's one
     of the points of Secure Boot, to let people have control over
     system keys):
     https://build.opensuse.org/package/view_file/openSUSE:Factory/ovmf/ovmf.spec?expand=1

  - Fedora 27 (package: "edk2-ovmf", x86_64):
     - /usr/share/edk2/ovmf/OVMF_CODE.fd
     - /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
     - /usr/share/edk2/ovmf/OVMF_VARS.fd

  - RHEL-7.4 (package: "OVMF", x86_64):
     - /usr/share/OVMF/OVMF_CODE.secboot.fd
     - /usr/share/OVMF/OVMF_VARS.fd

  - Gerd's firmware repo from Git:
     - /usr/share/edk2.git/ovmf-x64/OVMF_VARS-need-smm.fd
     - /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd

--
/kashyap

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Wed, Mar 07, 2018 at 03:49:51PM +0100, Kashyap Chamarthy wrote:
> Problem background
> ------------------
> 
> The various OVMF binary file names and paths are slightly different[+]
> for each Linux distribution.  And each high-level management tool
> (libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its
> own approach to detect and configure the said OVMF files.  This email
> thread is about arriving at some common understanding to make this a bit
> more "unified" by addressing these needs in QEMU and libvirt.

Let me expand on the problem & desired solution from libvirt's POV

Currently, if you are happy with the default firmware for a particular
machine type, life is easy. QEMU knows the right path to use automatically.

This was fine until people started wanting to use UEFI firemware with
x86_64 Q35 and/or aarch64  virt machine types, because Q35 defaults to
SeaBIOS and aarch64 defaults to no firmware.

So mgmt applications wanting to use UEFI have to provide a full path to
the firmware to libvirt.

>From a libvirt POV, we want to make it possible to just ask for a
firmware type eg for x86_64 we want to allow apps to just do either

  <os firmware="bios"/>

or

  <os firmware="efi"/>

and then have libvirt automatically provide the best firmware image
path to QEMU. We'll still allow apps to provide an explicit path
themselves if they really want to, because if nothing else that's
useful for people who need to test out ad-hoc firmware builds.

IOW, in general if you (sysadmin/mgmt app) want to enable EFI for a
guest, you should never need to care about firmware paths. This will
give libvirt QEMU driver parity with what's possible for vmware
guests in this area.

With this in mind, when we talk about providing metadata files for
firmware below, we should bear in mind that we likely want this
metadata to be general purpose, not something specific to OVMF.
IOW all existing QEMU firmware images, for all architetures should
be covered & whatever custom firmware users might have.

> Suggested approach
> ------------------
> 
> Based on an upstream discussion on 'virt-tools'[1] mailing list and some
> Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion
> to define a firmware metadata format and file (example in [1]):
> 
>   - For each firmware file we need a metadata file in a well defined
>     location, e.g. /usr/share/qemu/bios/ that lists stuff like:
> 
>       - Path to the firmware binary
>       - Path to the pre-built OVMF 'vars' file (if any)

Not neccessarily just OVMF - anything that uses the same pflash
config mechanism.

>       - Support architectures - associated QEMU feature flags (Secure
>         Boot)

We need to have ability to list machine types too, since historically
there have been cases where we screwed up and needed to provide differnt
builds of the same firmware to different machine types.

>       - If the binary provides / requires SMM (System Management Mode)

I think we just need a well defined list of properties / features
here as it might need to track morethan just SMM.

>      Essentially, QEMU would define[*] the file format, and provide
>      metadata files from any ROMs it ships directly.  If Linux
>      distributions / vendors ship extra ROMs like OVMF, etc then they
>      should provide suitable metadata files.

We need to define what order the metadata files are processed in,
so that if there are multuple firemware images that satisfy a given
guest config, we have a predictable / determinstic choice. This could
be as simple as recommending each filename start with a 2 digit number.

eg

     50-ovmf.json

That lets end user drop in a 20-ovmf.json if they want to provide a
different ovmf by default.

>   - Libvirt can read these metadata files and then pick the correct
>     firmware binary based on the settings for the guest.

IOW, we'll look at various things like firmware=efi|bios, and whether
secure boot is requested for the guest. I don't think we actually
have an explicit attribute for requesting secureboot=on|off, so we'd
probably want to add one. THis is then used to pick the default
firmware.

As a specific non-goal:

If multiple metadata files are all satisfactory for a given guest
config, libvirt is just going to pick the first acceptable match.
If users/apps want to have finer grained control, they can still
pass an explicit path themselves as done today to override the
default choice.  

>   - Management tools can then wire up the libvirt-based OVMF SB (Secure
>     Boot) configuration.

NB, mgmt tools should never need to read the metadata files - they'll
just request libvirt to use firmware=efi|bios, etc

> [*] Open question: Who, between QEMU and libvirt, should define the said
> firmware metadata format and file?

IMHO QEMU should be defining the format, because the file will contain
info about certain QEMU features associated with the firmware (eg smm).
Also there are potentially other non-libvirt mgmt apps that spawn QEMU
which would like this info (eg libguestfs), so having libvirt define the
format is inappropriate.

I'd suggest we just need something in docs/specs/firmware-metadata.rst
for QEMU source tree.

Potentially QEMU could even use the metadata files itself for finding
the default firmeware images, instead of compiling this info into its
binaries. I wouldn't suggest we need todo that right away, but bear it
in mind as a potential use case.

> 
> 
> References
> ---------- 
> 
> [1] A past proposal from Gerd to create a sort of a "firmware registry"
> 
>     https://www.redhat.com/archives/virt-tools-list/2014-September/msg00145.html
> 
> [2] A libvirt upstream RFE bug requesting to simplify handling UEFI
>     https://bugzilla.redhat.com/show_bug.cgi?id=1295146 -- RFE: provide
>     a bios=uefi XML convenience option
> 
> [3] DanPB wrote a PoC in libvirt:
>     https://www.redhat.com/archives/libvir-list/2016-October/msg00045.html
>     -- [libvirt] [PATCH 0/3] Make UEFI firmware config simpler
> 
>     But later came to the conclusion that it is flawed, and said we go
>     the route of "Suggested approach" mentioned earlier).
> 
> 
> [*] OVMF package path names for different distributions
> -------------------------------------------------------
> 
>   - Debian (https://packages.debian.org/stretch/all/ovmf/filelist)
>      - /usr/share/OVMF/OVMF_CODE.fd
>      - /usr/share/OVMF/OVMF_VARS.fd
> 
>   - OpenSUSE (package: "qemu-ovmf-x86_64" -- and it has *35*
>     files in that package):
>      - /usr/share/qemu/ovmf-x86_64-opensuse-code.bin
>      - /usr/share/qemu/ovmf-x86_64-opensuse-vars.bin [...]
> 
>      Their RPM spec file gives some hints (where all the files with
>      'ms' means signed with MS keys; the files with 'opensuse-4096'
>      means signed with OpenSUSE 4096 bit CA keys -- I think that's one
>      of the points of Secure Boot, to let people have control over
>      system keys):
>      https://build.opensuse.org/package/view_file/openSUSE:Factory/ovmf/ovmf.spec?expand=1
> 
>   - Fedora 27 (package: "edk2-ovmf", x86_64):
>      - /usr/share/edk2/ovmf/OVMF_CODE.fd
>      - /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
>      - /usr/share/edk2/ovmf/OVMF_VARS.fd
> 
>   - RHEL-7.4 (package: "OVMF", x86_64):
>      - /usr/share/OVMF/OVMF_CODE.secboot.fd
>      - /usr/share/OVMF/OVMF_VARS.fd
> 
>   - Gerd's firmware repo from Git:
>      - /usr/share/edk2.git/ovmf-x64/OVMF_VARS-need-smm.fd
>      - /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd
> 
> --
> /kashyap

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Gerd Hoffmann]

> Suggested approach
> ------------------
> 
> Based on an upstream discussion on 'virt-tools'[1] mailing list and some
> Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion
> to define a firmware metadata format and file (example in [1]):
> 
>   - For each firmware file we need a metadata file in a well defined
>     location, e.g. /usr/share/qemu/bios/ that lists stuff like:
> 
>       - Path to the firmware binary
>       - Path to the pre-built OVMF 'vars' file (if any)

How to load the binary (using -bios, -pflash, possibly also -kernel, for
uboot @ arm).

>       - Support architectures - associated QEMU feature flags (Secure
>         Boot)

Also machine types.  ovmf builds with smm don't boot on pc.  coreboot
has hardware-specific roms too, so the pc build wouldn't boot on q35 and
visa versa.  Same on arm, where the firmware typically is board-specific.

>       - If the binary provides / requires SMM (System Management Mode)

Possibly a more generic "flags" or "properties" thing, I can easily
imagine that simliar requirements show up on other platforms too.

Also a "name" and a "description" field would be useful.

cheers,
  Gerd

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Gerd Hoffmann]

  Hi,

> > [*] Open question: Who, between QEMU and libvirt, should define the said
> > firmware metadata format and file?
> 
> IMHO QEMU should be defining the format, because the file will contain
> info about certain QEMU features associated with the firmware (eg smm).
> Also there are potentially other non-libvirt mgmt apps that spawn QEMU
> which would like this info (eg libguestfs), so having libvirt define the
> format is inappropriate.
> 
> I'd suggest we just need something in docs/specs/firmware-metadata.rst
> for QEMU source tree.
> 
> Potentially QEMU could even use the metadata files itself for finding
> the default firmeware images, instead of compiling this info into its
> binaries. I wouldn't suggest we need todo that right away, but bear it
> in mind as a potential use case.

With qemu using this itself in mind it probably makes sense to specify
this as qapi schema.  That'll simplify parsing and using these files in
qemu, and possibly simplifies things on the libvirt side too.

cheers,
  Gerd

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Thu, Mar 08, 2018 at 08:45:07AM +0100, Gerd Hoffmann wrote:
> > Suggested approach
> > ------------------
> > 
> > Based on an upstream discussion on 'virt-tools'[1] mailing list and some
> > Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion
> > to define a firmware metadata format and file (example in [1]):
> > 
> >   - For each firmware file we need a metadata file in a well defined
> >     location, e.g. /usr/share/qemu/bios/ that lists stuff like:
> > 
> >       - Path to the firmware binary
> >       - Path to the pre-built OVMF 'vars' file (if any)
> 
> How to load the binary (using -bios, -pflash, possibly also -kernel, for
> uboot @ arm).

I wonder if there's value in also using this for describing secondary
device specific ROMs like iPXE and friends.

> 
> >       - Support architectures - associated QEMU feature flags (Secure
> >         Boot)
> 
> Also machine types.  ovmf builds with smm don't boot on pc.  coreboot
> has hardware-specific roms too, so the pc build wouldn't boot on q35 and
> visa versa.  Same on arm, where the firmware typically is board-specific.
> 
> >       - If the binary provides / requires SMM (System Management Mode)
> 
> Possibly a more generic "flags" or "properties" thing, I can easily
> imagine that simliar requirements show up on other platforms too.
> 
> Also a "name" and a "description" field would be useful.
> 
> cheers,
>   Gerd
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > [*] Open question: Who, between QEMU and libvirt, should define the said
> > > firmware metadata format and file?
> > 
> > IMHO QEMU should be defining the format, because the file will contain
> > info about certain QEMU features associated with the firmware (eg smm).
> > Also there are potentially other non-libvirt mgmt apps that spawn QEMU
> > which would like this info (eg libguestfs), so having libvirt define the
> > format is inappropriate.
> > 
> > I'd suggest we just need something in docs/specs/firmware-metadata.rst
> > for QEMU source tree.
> > 
> > Potentially QEMU could even use the metadata files itself for finding
> > the default firmeware images, instead of compiling this info into its
> > binaries. I wouldn't suggest we need todo that right away, but bear it
> > in mind as a potential use case.
> 
> With qemu using this itself in mind it probably makes sense to specify
> this as qapi schema.  That'll simplify parsing and using these files in
> qemu, and possibly simplifies things on the libvirt side too.

I was thinking of an 'ini' style format, similar to that used by systemd
unit files, but a JSON format file is a nicer fit with QEMU & Libvirt if
we describe it with qapi.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

(
Ard, the thread starts here:

http://mid.mail-archive.com/20180307144951.d75lo5rgzi2vf27z@eukaryote
)

On 03/07/18 15:49, Kashyap Chamarthy wrote:
> Problem background
> ------------------
> 
> The various OVMF binary file names and paths are slightly different[+]
> for each Linux distribution.  And each high-level management tool
> (libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its
> own approach to detect and configure the said OVMF files.  This email
> thread is about arriving at some common understanding to make this a bit
> more "unified" by addressing these needs in QEMU and libvirt.

I've read Dan's message <20180307151836.GK20201@redhat.com> and Gerd's
messages <20180308075245.lgzredyhn2paawg4@sirius.home.kraxel.org>,
<20180308074507.nwho4tddsoxb3b7v@sirius.home.kraxel.org>.

Those seem to cover everything, I don't have anything to add wrt.
purpose, use case, flexibility etc.

I suggest (or agree) that the property list be composed of free-form
name=value pairs (at least conceptually). I understand Gerd is proposing
a QAPI schema for this, so maybe do { property_name : "foo",
property_value : "bar" }, or similar. The registry of properties (names,
possible values, meanings) should be kept separate (although possibly
still under QEMU).

For OVMF (x86), I guess the initial set of properties should come from
the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
might grow or change incompatibly over time, so this is just a raw
starter idea.)

We have:

(0) ARCH (one of IA32, IA32X64, X64) -- the bitnesses of the PEI and DXE
phases of the firmware.

IA32 stands for "32-bit PEI and DXE". Such firmware is usable for
booting 32-bit OSes only, and runs on both qemu-system-i386 and
qemu-system-x86_64.

IA32X64 stands for "32-bit PEI, 64-bit DXE". Needs qemu-system-x86_64
and runs 64-bit OSes only.

X64 stands for "64-bit PEI and DXE". Needs qemu-system-x86_64 and runs
64-bit OSes only.


(1) SECURE_BOOT_ENABLE (boolean) -- whether the Secure Boot UEFI feature
is built into the firmware image.

This decides whether the Secure Boot software interfaces will be
available to the guest OS. Turning on just this flag does not imply that
the guest OS cannot circumvent the SB software interfaces by direct
(guest kernel) access to the pflash chip.


(2) SMM_REQUIRE (boolean) -- whether the SMM driver stack is included in
the firmware.

If this flag is enabled, then SMM emulation is required from the board
(implying Q35), otherwise the firmware will not boot.

If enabled in combination with SECURE_BOOT_ENABLE, *and* the "secure"
property of the "cfi.pflash01" driver is set to "on", then the
circumvention under (1) is prevented by QEMU, and Secure Boot becomes
actually secure.

Regarding 32-bit x86 (qemu-system-i386), the compatible CPU models are
strongly limited; one model that works is "coreduo", but the NX flag has
to be disabled even on that.

If ARCH (from under (0)) is X64, and SMM_REQUIRE is enabled, then ACPI
S3 suspend/resume has to be disabled ("disable_s3" property of the
"ICH9-LPC" driver), otherwise the firmware will not boot. IA32 and
IA32X64 are not limited like this.


(3) HTTP_BOOT_ENABLE (boolean) -- whether UEFI HTTP boot is available in
the firmware image, in addition to the default PXE boot.


(4) TLS_ENABLE (boolean) -- configurable independently of
HTTP_BOOT_ENABLE, but only really makes sense in combination.

Determines whether HTTPS boot is available in the firmware image.


(5) NETWORK_IP6_ENABLE (boolean) -- determines whether IPv6 support is
available.

Orthogonal to all of: PXE, HTTP, HTTPS boot.


(6) FD_SIZE_IN_KB (one of: 1024, 2048, 4096) -- the size of the combined
firmware image (executable portion and variable store together), in KB.

(Boolean shorthands are FD_SIZE_1MB, FD_SIZE_2MB, FD_SIZE_4MB.)

Firmware images where this value is 1024 and 2048 are "compatible" with
each other in the sense that the variable store files they use are
identically structured. The variable store files are 128KB in size, and
the actual variable space they offer is 56KB. This is generally
sufficient for a low number of UEFI variables, and very basic key
enrollment for Secure Boot.

A firmware image where the value is 4096 is incompatible. The variable
store file is 528KB in size, and the actual variable space it offers is
256KB. This is considered "generous" by recent-ish industry practice,
and it is large enough for passing the Microsoft SVVP test cases related
to Secure Boot. (Obviously there are other requirements presented by
those test cases; I'm just saying that the size requirement is satisfied.)

The variable store template files are pre-formatted in all cases
(meaning "wire format" only -- on a logical level, they are "empty").
Under some circumstances, it could be desirable to provide varstore
template files that are pre-populated with various certificates
enrolled. It might make sense to describe such facts with another property.


(7) DEBUG_ON_SERIAL_PORT (boolean) -- whether OVMF sends its debug
messages to the QEMU debug IO port, or to the serial port.


(8) SOURCE_DEBUG_ENABLE (boolean) -- whether OVMF includes the edk2
debug agent that allows it to be debugged from a proprietary debugger
program, likely connected via the emulated serial port.


(9) CSM_ENABLE (boolean) -- whether OVMF includes traditional BIOS
support, by including the SeaBIOS Compatibility Support Module.

Including the SeaBIOS CSM makes OVMF capable of booting traditional (not
UEFI) OS-es. Such boots are never covered by Secure Boot.

Given that the SeaBIOS CSM itself can be built with various
configurations, it might make sense to list further properties when this
property is enabled.


(10) E1000_ENABLE (boolean) -- whether OVMF includes the
non-redistributable, binary only E1000(E) UEFI driver module from Intel
(previously known as "PROEFI", more recently known as "BootUtil").

This driver can drive QEMU's e1000 and e1000e cards for network booting,
which is why the option exists at all.


(11) USE_OLD_SHELL (boolean) -- whether the UEFI shell implementation
built into OVMF is the old (EDK1 / EFI-1 style) shell which lives in a
separate repository, or the new (EDK2 / UEFI-2 style) shell which lives
within the edk2 project.


(12) TOOLCHAIN (string): the edk2 toolchain identifier with which the
firmware was built.

(13) TARGET (one of NOOPT, DEBUG, RELEASE) -- the target for which the
firmware image was built.

NOOPT means "no optimization, DEBUG msgs/ASSERTs enabled".
DEBUG means "optimization enabled, DEBUG msgs/ASSERTs enabled".
RELEASE means "optimization enabled, DEBUG msgs/ASSERTs disabled".



For ArmVirt, we have:

(12) ARCH (one of ARM and AARCH64) -- 32-bit / 64-bit distinction.

Mixed PEI and DXE bitness is not used, unlike on x86.


(13, 14) TOOLCHAIN and TARGET -- see (12) and (13) above

(15) SECURE_BOOT_ENABLE (boolean) -- see (1).

Note that SMM_REQUIRE (or its architectural match on ARM/AARCH64) is
currently not supported in the ArmVirt firmwares, hence the Secure Boot
feature should only be included in the firmware for development /
testing purposes.

(16) HTTP_BOOT_ENABLE (boolean) -- see (3).

(17) Well, what do I call this, let's call it ENTRY_POINT (one of Qemu,
QemuKernel, Xen).

This distinguishes what VMM the ArmVirt firmware was built for, and for
QEMU, it also distinguishes "boot from flash" (Qemu) or "boot as payload
for another, earlier boot firmware" (QemuKernel).


In the above, I tried to be comprehensive; pick whatever you think makes
sense to track in the metadata files.

Since Dan mentions we should have a "just gimme EFI" shorthand, I can
try suggesting specific combinations:

- For x86:

property_name         property_value
--------------------  --------------
ARCH                  IA32X64
SECURE_BOOT_ENABLE    TRUE
SMM_REQUIRE           TRUE
HTTP_BOOT_ENABLE      TRUE
TLS_ENABLE            TRUE
NETWORK_IP6_ENABLE    TRUE
FD_SIZE_IN_KB         4096
DEBUG_ON_SERIAL_PORT  FALSE
SOURCE_DEBUG_ENABLE   FALSE
CSM_ENABLE            FALSE
E1000_ENABLE          FALSE
USE_OLD_SHELL         FALSE
TOOLCHAIN             GCC5
TARGET                DEBUG

This gives you a firmware image that requires Q35, with SMM emulation
enabled. Secure Boot, when configured in the guest, is actually secure.
Microsoft SVVP SB testing can be passed (with certificates properly
enrolled, in addition), if you need that. ACPI S3 is supported. All
kinds of netbooting are enabled (PXE, HTTP, HTTPS, over both IPv4 and
IPv6). Debug messages are sent to the QEMU debug port (not messing up
your serial port / terminal traffic). No proprietary or
proprietary-interfacing components included. A pure UEFI build; if you
need to boot a legacy OS, just use standalone SeaBIOS. Built with at
least gcc-5, hence LTO (link time optimization) was utilized, allowing
for better performance and better firmware space utilization. Debug
messages are logged and assertions are enforced. Only 64-bit OSes are
supported, qemu-system-x86_64 is required.

- For arm:

property_name         property_value
--------------------  --------------
ARCH                  AARCH64
TOOLCHAIN             GCC5
TARGET                DEBUG
SECURE_BOOT_ENABLE    FALSE
HTTP_BOOT_ENABLE      TRUE
ENTRY_POINT           QEMU

qemu-system-aarch64 is required, only 64-bit OSes are supported.
Toolchain/target: see above. Debug messages can only be sent to the
PL011 UART (no separate debug port on qemu-system-aarch64 / "virt").
Secure Boot is not included, because at this point it cannot be secured
in virtual hardware (no Management Mode equivalent just yet). Allow HTTP
booting, and build to be booted on QEMU from pflash.

Final note: variable store formats are entirely incompatible between
OVMF and ArmVirt.

Thanks,
Laszlo

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Thu, Mar 08, 2018 at 12:10:30PM +0100, Laszlo Ersek wrote:
> (
> Ard, the thread starts here:
> 
> http://mid.mail-archive.com/20180307144951.d75lo5rgzi2vf27z@eukaryote
> )
> 
> On 03/07/18 15:49, Kashyap Chamarthy wrote:
> > Problem background
> > ------------------
> > 
> > The various OVMF binary file names and paths are slightly different[+]
> > for each Linux distribution.  And each high-level management tool
> > (libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its
> > own approach to detect and configure the said OVMF files.  This email
> > thread is about arriving at some common understanding to make this a bit
> > more "unified" by addressing these needs in QEMU and libvirt.
> 
> I've read Dan's message <20180307151836.GK20201@redhat.com> and Gerd's
> messages <20180308075245.lgzredyhn2paawg4@sirius.home.kraxel.org>,
> <20180308074507.nwho4tddsoxb3b7v@sirius.home.kraxel.org>.
> 
> Those seem to cover everything, I don't have anything to add wrt.
> purpose, use case, flexibility etc.
> 
> I suggest (or agree) that the property list be composed of free-form
> name=value pairs (at least conceptually). I understand Gerd is proposing
> a QAPI schema for this, so maybe do { property_name : "foo",
> property_value : "bar" }, or similar. The registry of properties (names,
> possible values, meanings) should be kept separate (although possibly
> still under QEMU).
> 
> For OVMF (x86), I guess the initial set of properties should come from
> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
> might grow or change incompatibly over time, so this is just a raw
> starter idea.)

I really don't want to see us using firmware implementation specific
property names in these files. It means libvirt will require knowledge
of what each different firmware's property names mean.

We need to have some core standardized set of property names that can
be provided by any firmware implementation using the same terminology.

If we want to /also/ provide some extra firmeware-specific property
names that would be ok for informative purposes, but when lbivirt is
picking which firmware file to use, it would only ever look at the
standardized property names/values.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 03/08/18 16:47, Daniel P. Berrangé wrote:
> On Thu, Mar 08, 2018 at 12:10:30PM +0100, Laszlo Ersek wrote:

>> I suggest (or agree) that the property list be composed of free-form
>> name=value pairs (at least conceptually). I understand Gerd is proposing
>> a QAPI schema for this, so maybe do { property_name : "foo",
>> property_value : "bar" }, or similar. The registry of properties (names,
>> possible values, meanings) should be kept separate (although possibly
>> still under QEMU).
>>
>> For OVMF (x86), I guess the initial set of properties should come from
>> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
>> might grow or change incompatibly over time, so this is just a raw
>> starter idea.)
> 
> I really don't want to see us using firmware implementation specific
> property names in these files. It means libvirt will require knowledge
> of what each different firmware's property names mean.
> 
> We need to have some core standardized set of property names that can
> be provided by any firmware implementation using the same terminology.
> 
> If we want to /also/ provide some extra firmeware-specific property
> names that would be ok for informative purposes, but when lbivirt is
> picking which firmware file to use, it would only ever look at the
> standardized property names/values.

This is a reasonable requirement from the libvirt side.

Unfortunately (or not), it requires someone (or a tight group of people)
to collect the features of all virtual firmwares in existence, and
extract a common set of properties that maps back to each firmware one
way or another. This is not unusual (basically this is how all standards
bodies work that intend to codify existing practice), it just needs a
bunch of work and coordination. We'll have to maintain a registry.

Personally I can't comment on anything else than OVMF and the ArmVirt
firmwares.

Thanks,
Laszlo

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Kashyap Chamarthy]

On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > [*] Open question: Who, between QEMU and libvirt, should define the said
> > > firmware metadata format and file?
> > 
> > IMHO QEMU should be defining the format, because the file will contain
> > info about certain QEMU features associated with the firmware (eg smm).
> > Also there are potentially other non-libvirt mgmt apps that spawn QEMU
> > which would like this info (eg libguestfs), so having libvirt define the
> > format is inappropriate.
> > 
> > I'd suggest we just need something in docs/specs/firmware-metadata.rst
> > for QEMU source tree.
> > 
> > Potentially QEMU could even use the metadata files itself for finding
> > the default firmeware images, instead of compiling this info into its
> > binaries. I wouldn't suggest we need todo that right away, but bear it
> > in mind as a potential use case.
> 
> With qemu using this itself in mind it probably makes sense to specify
> this as qapi schema.  That'll simplify parsing and using these files in
> qemu, and possibly simplifies things on the libvirt side too.

Yeah, FWIW, I too find using the QAPI schema for the metadata file
format more appealing (rather than a separate file format).  And
libvirt already has infrastructure to handle QAPI.

-- 
/kashyap

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Kashyap Chamarthy]

On Thu, Mar 08, 2018 at 09:47:27PM +0100, Laszlo Ersek wrote:
> On 03/08/18 16:47, Daniel P. Berrangé wrote:
> > On Thu, Mar 08, 2018 at 12:10:30PM +0100, Laszlo Ersek wrote:

[...]

> >> For OVMF (x86), I guess the initial set of properties should come from
> >> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
> >> might grow or change incompatibly over time, so this is just a raw
> >> starter idea.)
> > 
> > I really don't want to see us using firmware implementation specific
> > property names in these files. It means libvirt will require knowledge
> > of what each different firmware's property names mean.
> > 
> > We need to have some core standardized set of property names that can
> > be provided by any firmware implementation using the same terminology.
> > 
> > If we want to /also/ provide some extra firmeware-specific property
> > names that would be ok for informative purposes, but when lbivirt is
> > picking which firmware file to use, it would only ever look at the
> > standardized property names/values.
> 
> This is a reasonable requirement from the libvirt side.
> 
> Unfortunately (or not), it requires someone (or a tight group of people)
> to collect the features of all virtual firmwares in existence, and
> extract a common set of properties that maps back to each firmware one
> way or another. 

Hmm, if people consider the above worthwhile (no clue how much time &
investigation it takes to arrive at a common set of properties) maybe
slowly we should start collecting such a page?  From a quick look up,
list of open source firmware implementations I found so (besides OVMF &
ArmVirt):

  - OpenBIOS
  - SmartFirmware
  - OpenBoot
  - CoreBoot
  - U-Boot
  - SLOF
  - ...

Source: https://en.wikipedia.org/wiki/OpenBIOS

I notice you said "virtual firmwares".  I couldn't find such a list from
my look up.

Hmm, I also wonder if the "arriving at a common set of properties across
existing virtual firmwares" is an absolute blocker.

> This is not unusual (basically this is how all standards
> bodies work that intend to codify existing practice), it just needs a
> bunch of work and coordination. We'll have to maintain a registry.
> 
> Personally I can't comment on anything else than OVMF and the ArmVirt
> firmwares.
> 
> Thanks,
> Laszlo

-- 
/kashyap

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Gerd Hoffmann]

  Hi,

> For OVMF (x86), I guess the initial set of properties should come from
> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
> might grow or change incompatibly over time, so this is just a raw
> starter idea.)

> (0) ARCH (one of IA32, IA32X64, X64) -- the bitnesses of the PEI and DXE
> phases of the firmware.
> 
> IA32 stands for "32-bit PEI and DXE". Such firmware is usable for
> booting 32-bit OSes only, and runs on both qemu-system-i386 and
> qemu-system-x86_64.
> 
> IA32X64 stands for "32-bit PEI, 64-bit DXE". Needs qemu-system-x86_64
> and runs 64-bit OSes only.
> 
> X64 stands for "64-bit PEI and DXE". Needs qemu-system-x86_64 and runs
> 64-bit OSes only.

I'd simply put that info the 'arch' bucket.  IA32 gets 'i386' (assuming
we simply use qemu target names), IA32X64 + X64 get 'x86_64'.

> (1) SECURE_BOOT_ENABLE (boolean) -- whether the Secure Boot UEFI feature
> is built into the firmware image.

Probably useful for upper management (above libvirt), for enabling
secure boot key enrollment etc. in the UI.

> (2) SMM_REQUIRE (boolean) -- whether the SMM driver stack is included in
> the firmware.

Needed (libvirt must enable smm).

> (3) HTTP_BOOT_ENABLE (boolean) -- whether UEFI HTTP boot is available in
> the firmware image, in addition to the default PXE boot.

> (4) TLS_ENABLE (boolean) -- configurable independently of
> HTTP_BOOT_ENABLE, but only really makes sense in combination.

> (5) NETWORK_IP6_ENABLE (boolean) -- determines whether IPv6 support is
> available.

Do we actually need formal flags for these?  Which application would
use them?

For humans this can go into the freeform "description" text field.

> (6) FD_SIZE_IN_KB (one of: 1024, 2048, 4096) -- the size of the combined
> firmware image (executable portion and variable store together), in KB.

Hmm, dunno.  Might be useful as varstores are not portable between
builds of different sizes, so libvirt could use this for sanity checks.

> (7) DEBUG_ON_SERIAL_PORT (boolean) -- whether OVMF sends its debug
> messages to the QEMU debug IO port, or to the serial port.

> (8) SOURCE_DEBUG_ENABLE (boolean) -- whether OVMF includes the edk2
> debug agent that allows it to be debugged from a proprietary debugger
> program, likely connected via the emulated serial port.

> (9) CSM_ENABLE (boolean) -- whether OVMF includes traditional BIOS
> support, by including the SeaBIOS Compatibility Support Module.

Same question as for 3+4+5: Do we really need those?

> Given that the SeaBIOS CSM itself can be built with various
> configurations, it might make sense to list further properties when this
> property is enabled.

I doubt this is useful, established practice is that people just use
seabios directly instead if using the ovmf+csm combo.

> (10) E1000_ENABLE (boolean) -- whether OVMF includes the
> non-redistributable, binary only E1000(E) UEFI driver module from Intel
> (previously known as "PROEFI", more recently known as "BootUtil").

> (11) USE_OLD_SHELL (boolean) -- whether the UEFI shell implementation
> built into OVMF is the old (EDK1 / EFI-1 style) shell which lives in a
> separate repository, or the new (EDK2 / UEFI-2 style) shell which lives
> within the edk2 project.

> (12) TOOLCHAIN (string): the edk2 toolchain identifier with which the
> firmware was built.

> (13) TARGET (one of NOOPT, DEBUG, RELEASE) -- the target for which the
> firmware image was built.

Same question as for 3+4+5+7+8+9: Do we really need those?

> For ArmVirt, we have:
> 
> (12) ARCH (one of ARM and AARCH64) -- 32-bit / 64-bit distinction.

Likewise just use arch.

> (17) Well, what do I call this, let's call it ENTRY_POINT (one of Qemu,
> QemuKernel, Xen).
> 
> This distinguishes what VMM the ArmVirt firmware was built for, and for
> QEMU, it also distinguishes "boot from flash" (Qemu) or "boot as payload
> for another, earlier boot firmware" (QemuKernel).

Hmm, that looks simliar to u-boot, which I think can also be build as
both flash image and kernel image (loadable via -kernel).  So I guess we
want that one, as how-to-load-this-blob option (-bios vs -pflash vs -kernel).

cheers,
  Gerd

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 03/09/18 12:27, Kashyap Chamarthy wrote:
> On Thu, Mar 08, 2018 at 09:47:27PM +0100, Laszlo Ersek wrote:
>> On 03/08/18 16:47, Daniel P. Berrangé wrote:
>>> On Thu, Mar 08, 2018 at 12:10:30PM +0100, Laszlo Ersek wrote:
> 
> [...]
> 
>>>> For OVMF (x86), I guess the initial set of properties should come from
>>>> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
>>>> might grow or change incompatibly over time, so this is just a raw
>>>> starter idea.)
>>>
>>> I really don't want to see us using firmware implementation specific
>>> property names in these files. It means libvirt will require knowledge
>>> of what each different firmware's property names mean.
>>>
>>> We need to have some core standardized set of property names that can
>>> be provided by any firmware implementation using the same terminology.
>>>
>>> If we want to /also/ provide some extra firmeware-specific property
>>> names that would be ok for informative purposes, but when lbivirt is
>>> picking which firmware file to use, it would only ever look at the
>>> standardized property names/values.
>>
>> This is a reasonable requirement from the libvirt side.
>>
>> Unfortunately (or not), it requires someone (or a tight group of people)
>> to collect the features of all virtual firmwares in existence, and
>> extract a common set of properties that maps back to each firmware one
>> way or another. 
> 
> Hmm, if people consider the above worthwhile (no clue how much time &
> investigation it takes to arrive at a common set of properties) maybe
> slowly we should start collecting such a page?  From a quick look up,
> list of open source firmware implementations I found so (besides OVMF &
> ArmVirt):
> 
>   - OpenBIOS
>   - SmartFirmware
>   - OpenBoot
>   - CoreBoot
>   - U-Boot
>   - SLOF
>   - ...
> 
> Source: https://en.wikipedia.org/wiki/OpenBIOS
> 
> I notice you said "virtual firmwares".  I couldn't find such a list from
> my look up.
> 
> Hmm, I also wonder if the "arriving at a common set of properties across
> existing virtual firmwares" is an absolute blocker.

That's for Daniel to decide. I can't sensibly generalize from OVMF &
ArmVirt to other firmwares, without knowing them.

Thanks
Laszlo

> 
>> This is not unusual (basically this is how all standards
>> bodies work that intend to codify existing practice), it just needs a
>> bunch of work and coordination. We'll have to maintain a registry.
>>
>> Personally I can't comment on anything else than OVMF and the ArmVirt
>> firmwares.
>>
>> Thanks,
>> Laszlo
> 

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 03/09/18 15:27, Gerd Hoffmann wrote:
>   Hi,
> 
>> For OVMF (x86), I guess the initial set of properties should come from
>> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
>> might grow or change incompatibly over time, so this is just a raw
>> starter idea.)
> 
>> (0) ARCH (one of IA32, IA32X64, X64) -- the bitnesses of the PEI and DXE
>> phases of the firmware.
>>
>> IA32 stands for "32-bit PEI and DXE". Such firmware is usable for
>> booting 32-bit OSes only, and runs on both qemu-system-i386 and
>> qemu-system-x86_64.
>>
>> IA32X64 stands for "32-bit PEI, 64-bit DXE". Needs qemu-system-x86_64
>> and runs 64-bit OSes only.
>>
>> X64 stands for "64-bit PEI and DXE". Needs qemu-system-x86_64 and runs
>> 64-bit OSes only.
> 
> I'd simply put that info the 'arch' bucket.  IA32 gets 'i386' (assuming
> we simply use qemu target names), IA32X64 + X64 get 'x86_64'.
> 
>> (1) SECURE_BOOT_ENABLE (boolean) -- whether the Secure Boot UEFI feature
>> is built into the firmware image.
> 
> Probably useful for upper management (above libvirt), for enabling
> secure boot key enrollment etc. in the UI.
> 
>> (2) SMM_REQUIRE (boolean) -- whether the SMM driver stack is included in
>> the firmware.
> 
> Needed (libvirt must enable smm).
> 
>> (3) HTTP_BOOT_ENABLE (boolean) -- whether UEFI HTTP boot is available in
>> the firmware image, in addition to the default PXE boot.
> 
>> (4) TLS_ENABLE (boolean) -- configurable independently of
>> HTTP_BOOT_ENABLE, but only really makes sense in combination.
> 
>> (5) NETWORK_IP6_ENABLE (boolean) -- determines whether IPv6 support is
>> available.
> 
> Do we actually need formal flags for these?  Which application would
> use them?
> 
> For humans this can go into the freeform "description" text field.
> 
>> (6) FD_SIZE_IN_KB (one of: 1024, 2048, 4096) -- the size of the combined
>> firmware image (executable portion and variable store together), in KB.
> 
> Hmm, dunno.  Might be useful as varstores are not portable between
> builds of different sizes, so libvirt could use this for sanity checks.
> 
>> (7) DEBUG_ON_SERIAL_PORT (boolean) -- whether OVMF sends its debug
>> messages to the QEMU debug IO port, or to the serial port.
> 
>> (8) SOURCE_DEBUG_ENABLE (boolean) -- whether OVMF includes the edk2
>> debug agent that allows it to be debugged from a proprietary debugger
>> program, likely connected via the emulated serial port.
> 
>> (9) CSM_ENABLE (boolean) -- whether OVMF includes traditional BIOS
>> support, by including the SeaBIOS Compatibility Support Module.
> 
> Same question as for 3+4+5: Do we really need those?
> 
>> Given that the SeaBIOS CSM itself can be built with various
>> configurations, it might make sense to list further properties when this
>> property is enabled.
> 
> I doubt this is useful, established practice is that people just use
> seabios directly instead if using the ovmf+csm combo.
> 
>> (10) E1000_ENABLE (boolean) -- whether OVMF includes the
>> non-redistributable, binary only E1000(E) UEFI driver module from Intel
>> (previously known as "PROEFI", more recently known as "BootUtil").
> 
>> (11) USE_OLD_SHELL (boolean) -- whether the UEFI shell implementation
>> built into OVMF is the old (EDK1 / EFI-1 style) shell which lives in a
>> separate repository, or the new (EDK2 / UEFI-2 style) shell which lives
>> within the edk2 project.
> 
>> (12) TOOLCHAIN (string): the edk2 toolchain identifier with which the
>> firmware was built.
> 
>> (13) TARGET (one of NOOPT, DEBUG, RELEASE) -- the target for which the
>> firmware image was built.
> 
> Same question as for 3+4+5+7+8+9: Do we really need those?

I can't judge that. To me all these flags are meaningful. It doesn't
hurt to keep them spelled out in the metadata; they don't take much
space, they are easy to ignore, and if someone searches for them, they
are there.

I'll offer an analogy. Sometimes it would be useful to know the full
kernel config used on a distro's Live or Installer ISO. Just today I got
a question off-list as to whether the aarch64 Fedora 28 Live iso has
CONFIG_ARM64_PMEM enabled or not. I have no clue, but this type of
question is not far from the domain of "libosinfo" (which basically
tells you what OS release has drivers for what devices). Is "libfwinfo"
such a stretch? Maybe. :)

So, whether this level of detail is useful to end-users and/or
management applications, I cannot tell; I just wanted to expose the
"current spectrum" (for OVMF and ArmVirtQemu) in reasonable detail.

Thanks
Laszlo

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Fri, Mar 09, 2018 at 04:18:45PM +0100, Laszlo Ersek wrote:
> On 03/09/18 15:27, Gerd Hoffmann wrote:
> >   Hi,
> > 
> >> For OVMF (x86), I guess the initial set of properties should come from
> >> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
> >> might grow or change incompatibly over time, so this is just a raw
> >> starter idea.)
> > 
> >> (0) ARCH (one of IA32, IA32X64, X64) -- the bitnesses of the PEI and DXE
> >> phases of the firmware.
> >>
> >> IA32 stands for "32-bit PEI and DXE". Such firmware is usable for
> >> booting 32-bit OSes only, and runs on both qemu-system-i386 and
> >> qemu-system-x86_64.
> >>
> >> IA32X64 stands for "32-bit PEI, 64-bit DXE". Needs qemu-system-x86_64
> >> and runs 64-bit OSes only.
> >>
> >> X64 stands for "64-bit PEI and DXE". Needs qemu-system-x86_64 and runs
> >> 64-bit OSes only.
> > 
> > I'd simply put that info the 'arch' bucket.  IA32 gets 'i386' (assuming
> > we simply use qemu target names), IA32X64 + X64 get 'x86_64'.
> > 
> >> (1) SECURE_BOOT_ENABLE (boolean) -- whether the Secure Boot UEFI feature
> >> is built into the firmware image.
> > 
> > Probably useful for upper management (above libvirt), for enabling
> > secure boot key enrollment etc. in the UI.
> > 
> >> (2) SMM_REQUIRE (boolean) -- whether the SMM driver stack is included in
> >> the firmware.
> > 
> > Needed (libvirt must enable smm).
> > 
> >> (3) HTTP_BOOT_ENABLE (boolean) -- whether UEFI HTTP boot is available in
> >> the firmware image, in addition to the default PXE boot.
> > 
> >> (4) TLS_ENABLE (boolean) -- configurable independently of
> >> HTTP_BOOT_ENABLE, but only really makes sense in combination.
> > 
> >> (5) NETWORK_IP6_ENABLE (boolean) -- determines whether IPv6 support is
> >> available.
> > 
> > Do we actually need formal flags for these?  Which application would
> > use them?
> > 
> > For humans this can go into the freeform "description" text field.
> > 
> >> (6) FD_SIZE_IN_KB (one of: 1024, 2048, 4096) -- the size of the combined
> >> firmware image (executable portion and variable store together), in KB.
> > 
> > Hmm, dunno.  Might be useful as varstores are not portable between
> > builds of different sizes, so libvirt could use this for sanity checks.
> > 
> >> (7) DEBUG_ON_SERIAL_PORT (boolean) -- whether OVMF sends its debug
> >> messages to the QEMU debug IO port, or to the serial port.
> > 
> >> (8) SOURCE_DEBUG_ENABLE (boolean) -- whether OVMF includes the edk2
> >> debug agent that allows it to be debugged from a proprietary debugger
> >> program, likely connected via the emulated serial port.
> > 
> >> (9) CSM_ENABLE (boolean) -- whether OVMF includes traditional BIOS
> >> support, by including the SeaBIOS Compatibility Support Module.
> > 
> > Same question as for 3+4+5: Do we really need those?
> > 
> >> Given that the SeaBIOS CSM itself can be built with various
> >> configurations, it might make sense to list further properties when this
> >> property is enabled.
> > 
> > I doubt this is useful, established practice is that people just use
> > seabios directly instead if using the ovmf+csm combo.
> > 
> >> (10) E1000_ENABLE (boolean) -- whether OVMF includes the
> >> non-redistributable, binary only E1000(E) UEFI driver module from Intel
> >> (previously known as "PROEFI", more recently known as "BootUtil").
> > 
> >> (11) USE_OLD_SHELL (boolean) -- whether the UEFI shell implementation
> >> built into OVMF is the old (EDK1 / EFI-1 style) shell which lives in a
> >> separate repository, or the new (EDK2 / UEFI-2 style) shell which lives
> >> within the edk2 project.
> > 
> >> (12) TOOLCHAIN (string): the edk2 toolchain identifier with which the
> >> firmware was built.
> > 
> >> (13) TARGET (one of NOOPT, DEBUG, RELEASE) -- the target for which the
> >> firmware image was built.
> > 
> > Same question as for 3+4+5+7+8+9: Do we really need those?
> 
> I can't judge that. To me all these flags are meaningful. It doesn't
> hurt to keep them spelled out in the metadata; they don't take much
> space, they are easy to ignore, and if someone searches for them, they
> are there.
> 
> I'll offer an analogy. Sometimes it would be useful to know the full
> kernel config used on a distro's Live or Installer ISO. Just today I got
> a question off-list as to whether the aarch64 Fedora 28 Live iso has
> CONFIG_ARM64_PMEM enabled or not. I have no clue, but this type of
> question is not far from the domain of "libosinfo" (which basically
> tells you what OS release has drivers for what devices). Is "libfwinfo"
> such a stretch? Maybe. :)
> 
> So, whether this level of detail is useful to end-users and/or
> management applications, I cannot tell; I just wanted to expose the
> "current spectrum" (for OVMF and ArmVirtQemu) in reasonable detail.

The kernel  config is typically shipped by kernel RPMs in Fedora, but it is
purely there as informative text. No application is expected to read the
/boot/config-XXXX files to apply functional logic.  So with that analogy
its fine if you want to ship a record of build config along with the
firmware files, I just don't think it is something libvirt would interpret.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Daniel P. Berrangé]

On Thu, Mar 08, 2018 at 09:47:27PM +0100, Laszlo Ersek wrote:
> On 03/08/18 16:47, Daniel P. Berrangé wrote:
> > On Thu, Mar 08, 2018 at 12:10:30PM +0100, Laszlo Ersek wrote:
> 
> >> I suggest (or agree) that the property list be composed of free-form
> >> name=value pairs (at least conceptually). I understand Gerd is proposing
> >> a QAPI schema for this, so maybe do { property_name : "foo",
> >> property_value : "bar" }, or similar. The registry of properties (names,
> >> possible values, meanings) should be kept separate (although possibly
> >> still under QEMU).
> >>
> >> For OVMF (x86), I guess the initial set of properties should come from
> >> the "-D FOO[=BAR]" build flags that OVMF currently supports. (The list
> >> might grow or change incompatibly over time, so this is just a raw
> >> starter idea.)
> > 
> > I really don't want to see us using firmware implementation specific
> > property names in these files. It means libvirt will require knowledge
> > of what each different firmware's property names mean.
> > 
> > We need to have some core standardized set of property names that can
> > be provided by any firmware implementation using the same terminology.
> > 
> > If we want to /also/ provide some extra firmeware-specific property
> > names that would be ok for informative purposes, but when lbivirt is
> > picking which firmware file to use, it would only ever look at the
> > standardized property names/values.
> 
> This is a reasonable requirement from the libvirt side.
> 
> Unfortunately (or not), it requires someone (or a tight group of people)
> to collect the features of all virtual firmwares in existence, and
> extract a common set of properties that maps back to each firmware one
> way or another. This is not unusual (basically this is how all standards
> bodies work that intend to codify existing practice), it just needs a
> bunch of work and coordination. We'll have to maintain a registry.
> 
> Personally I can't comment on anything else than OVMF and the ArmVirt
> firmwares.

I don't think it is actually a big problem. Today there is a very small
set of features we'll care about when selecting between firmware files.
For most architectures/firmwares, I expect there will just be a single
firmware image, tagged with architecture name, and possibly machine
type.

I think EFI will be the only case we have to start off with, where we
need to define a few extra standard features (for the SMM + secureboot
essentially).  We can just iterate on this as more use cases / features
come to light.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 03/08/18 11:17, Daniel P. Berrangé wrote:
> On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote:
>>   Hi,
>>
>>>> [*] Open question: Who, between QEMU and libvirt, should define the said
>>>> firmware metadata format and file?
>>>
>>> IMHO QEMU should be defining the format, because the file will contain
>>> info about certain QEMU features associated with the firmware (eg smm).
>>> Also there are potentially other non-libvirt mgmt apps that spawn QEMU
>>> which would like this info (eg libguestfs), so having libvirt define the
>>> format is inappropriate.
>>>
>>> I'd suggest we just need something in docs/specs/firmware-metadata.rst
>>> for QEMU source tree.
>>>
>>> Potentially QEMU could even use the metadata files itself for finding
>>> the default firmeware images, instead of compiling this info into its
>>> binaries. I wouldn't suggest we need todo that right away, but bear it
>>> in mind as a potential use case.
>>
>> With qemu using this itself in mind it probably makes sense to specify
>> this as qapi schema.  That'll simplify parsing and using these files in
>> qemu, and possibly simplifies things on the libvirt side too.
> 
> I was thinking of an 'ini' style format, similar to that used by systemd
> unit files, but a JSON format file is a nicer fit with QEMU & Libvirt if
> we describe it with qapi.

I've created an RFC-level "qapi/firmware.json" schema file, based on
this discussion. It "builds", and the generated documentation looks
acceptable, superficially speaking.

Before I post "qapi/firmware.json" for getting comments, I'd like to
write JSON text that (a) describes firmware that I use, and (b) conforms
to the schema. IOW, I'd like to validate whether the schema is good
enough for describing at least such firmware that I know.

Is there a tool that generates example JSON objects from a given schema?
I vaguely recall there used to be one. Otherwise, writing the examples
manually looks arduous (and I wouldn't know how to verify them against
the schema).

Thanks!
Laszlo

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

On 04/06/2018 12:28 PM, Laszlo Ersek wrote:

> I've created an RFC-level "qapi/firmware.json" schema file, based on
> this discussion. It "builds", and the generated documentation looks
> acceptable, superficially speaking.
> 
> Before I post "qapi/firmware.json" for getting comments, I'd like to
> write JSON text that (a) describes firmware that I use, and (b) conforms
> to the schema. IOW, I'd like to validate whether the schema is good
> enough for describing at least such firmware that I know.
> 
> Is there a tool that generates example JSON objects from a given schema?

I know the QMP shell (scripts/qmp/qmp-shell) lets you enter commands
with a lot less typing than full JSON, and has a mode where it will then
echo the full JSON command it constructed from what you typed.  To be
able to quickly validate examples, it may be sufficient to temporarily
add a new QMP command 'check-firmware':

{ 'command': 'check-firmware', 'boxed': true, 'data': 'Firmware' }

assuming 'Firmware' is your top-level 'struct' in the QAPI file, then
implement a trivial:

qmp_check_firmware(Firmware *obj, Error **errp) {
    return 0;
}

so that you can then run QMP shell, and type:

check-firmware arg1=foo arg2=bar ...

which will then generate the corresponding JSON, then either
successfully do nothing (what you typed validated, AND you have the JSON
output printed), or print an error (what you typed failed QAPI
validation, perhaps because it had an unrecognized key, was missing a
required key, used a wrong type, etc).

> I vaguely recall there used to be one. Otherwise, writing the examples
> manually looks arduous (and I wouldn't know how to verify them against
> the schema).

Similarly, if you generate a command the produces a 'Firmware' as the
return value, then you can populate the generated C struct (since you
did manage to run the QAPI generator over your new file, you should be
able to look at the C struct it generated), then output that over QMP to
show the counterpart JSON that matches the struct as populated.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 04/06/18 20:10, Eric Blake wrote:
> On 04/06/2018 12:28 PM, Laszlo Ersek wrote:
>
>> I've created an RFC-level "qapi/firmware.json" schema file, based on
>> this discussion. It "builds", and the generated documentation looks
>> acceptable, superficially speaking.
>>
>> Before I post "qapi/firmware.json" for getting comments, I'd like to
>> write JSON text that (a) describes firmware that I use, and (b)
>> conforms to the schema. IOW, I'd like to validate whether the schema
>> is good enough for describing at least such firmware that I know.
>>
>> Is there a tool that generates example JSON objects from a given
>> schema?
>
> I know the QMP shell (scripts/qmp/qmp-shell) lets you enter commands
> with a lot less typing than full JSON, and has a mode where it will
> then echo the full JSON command it constructed from what you typed. To
> be able to quickly validate examples, it may be sufficient to
> temporarily add a new QMP command 'check-firmware':
>
> { 'command': 'check-firmware', 'boxed': true, 'data': 'Firmware' }
>
> assuming 'Firmware' is your top-level 'struct' in the QAPI file, then
> implement a trivial:
>
> qmp_check_firmware(Firmware *obj, Error **errp) {
>     return 0;
> }
>
> so that you can then run QMP shell, and type:
>
> check-firmware arg1=foo arg2=bar ...
>
> which will then generate the corresponding JSON, then either
> successfully do nothing (what you typed validated, AND you have the
> JSON output printed), or print an error (what you typed failed QAPI
> validation, perhaps because it had an unrecognized key, was missing a
> required key, used a wrong type, etc).
>
>> I vaguely recall there used to be one. Otherwise, writing the
>> examples manually looks arduous (and I wouldn't know how to verify
>> them against the schema).
>
> Similarly, if you generate a command the produces a 'Firmware' as the
> return value, then you can populate the generated C struct (since you
> did manage to run the QAPI generator over your new file, you should be
> able to look at the C struct it generated), then output that over QMP
> to show the counterpart JSON that matches the struct as populated.
>

The top level structure is complex / nested, but that doesn't appear to
be an issue. According to the script,

# key=value pairs also support Python or JSON object literal subset notations,
# without spaces. Dictionaries/objects {} are supported as are arrays [].
#
#    example-command arg-name1={'key':'value','obj'={'prop':"value"}}
#
# Both JSON and Python formatting should work, including both styles of
# string literal quotes. Both paradigms of literal values should work,
# including null/true/false for JSON and None/True/False for Python.

This looks awesome, because it should let me provide messy nested input
(which I'll obviously compose in my $EDITOR and then paste it), and then
the QMP shell will both validate and pretty print that. I'm going to try
this.

Thank you, Eric!
Laszlo

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Kashyap Chamarthy]

On Fri, Apr 06, 2018 at 08:21:00PM +0200, Laszlo Ersek wrote:
> On 04/06/18 20:10, Eric Blake wrote:

[...]

> # key=value pairs also support Python or JSON object literal subset notations,
> # without spaces. Dictionaries/objects {} are supported as are arrays [].
> #
> #    example-command arg-name1={'key':'value','obj'={'prop':"value"}}
> #
> # Both JSON and Python formatting should work, including both styles of
> # string literal quotes. Both paradigms of literal values should work,
> # including null/true/false for JSON and None/True/False for Python.

A couple of examples of the the key-value pairs and using JSON dicts
with 'qmp-shell'.

Key-value pairs:

  (QEMU) blockdev-snapshot node=node-Base snapshot-file=./overlay1.qcow2 overlay=node-Overlay1
  [...]

A combination key-value pairs and JSON dicts:

   (QEMU) blockdev-add driver=qcow2 node-name=node-overlay1 file={"driver":"file","filename":"overlay1.qcow2"}
   [...]

> This looks awesome, because it should let me provide messy nested input
> (which I'll obviously compose in my $EDITOR and then paste it), and then
> the QMP shell will both validate and pretty print that. I'm going to try
> this.

A couple of notes when using the 'qmp-shell':

  - Assuming you started QEMU with: `[...] -qmp
    unix:/tmp/qmp-sock,server,nowait`, you might want to use the '-p'
    command-line option to prett-print the JSON:
    
        $ qmp-shell -v -p /tmp/qmp-sock

  - It also provides a persistent command history in a convenient file:
    '~/.qmp-shell_history'

-- 
/kashyap

Re: [Qemu-devel] [RFC] Defining firmware (OVMF, et al) metadata format & file

[Laszlo Ersek]

On 04/09/18 11:02, Kashyap Chamarthy wrote:

>   - It also provides a persistent command history in a convenient file:
>     '~/.qmp-shell_history'

I noticed it used readline, but I didn't know about the dedicated
history file. Nice! Thanks!

Laszlo