[PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Daniel P. Berrangé]

The current firmware descriptor schema for flash requires that both the
executable to NVRAM template paths be provided. This is fine for the
most common usage of EDK2 builds in virtualization where the separate
_CODE and _VARS files are provided.

With confidential computing technology like AMD SEV, persistent storage
of variables may be completely disabled because the firmware requires a
known clean state on every cold boot. There is no way to express this
in the firmware descriptor today.

Even with regular EDK2 builds it is possible to create a firmware that
has both executable code and variable persistence in a single file. This
hasn't been commonly used, since it would mean every guest bootup would
need to clone the full firmware file, leading to redundant duplicate
storage of the code portion. In some scenarios this may not matter and
might even be beneficial. For example if a public cloud allows users to
bring their own firmware, such that the user can pre-enroll their own
secure boot keys, you're going to have this copied on disk for each
tenant already. At this point the it can be simpler to just deal with
a single file rather than split builds. The firmware descriptor ought
to be able to express this combined firmware model too.

This all points towards expanding the schema for flash with a 'mode'
concept:

 - "split" - the current implicit behaviour with separate files
   for code and variables.

 - "combined" - the alternate behaviour where a single file contains
   both code and variables.

 - "stateless" - the confidential computing use case where storage
   of variables is completely disable, leaving only the code.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/interop/firmware.json | 54 ++++++++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 8 deletions(-)

In v2:

 - Mark 'mode' as optional field
 - Misc typos in docs

diff --git a/docs/interop/firmware.json b/docs/interop/firmware.json
index 8d8b0be030..f5d1d0b6e7 100644
--- a/docs/interop/firmware.json
+++ b/docs/interop/firmware.json
@@ -210,24 +210,61 @@
   'data'   : { 'filename' : 'str',
                'format'   : 'BlockdevDriver' } }
 
+
+##
+# @FirmwareFlashType:
+#
+# Describes how the firmware build handles code versus variable
+# persistence.
+#
+# @split: the executable file contains code while the NVRAM
+#         template provides variable storage. The executable
+#         must be configured read-only and can be shared between
+#         multiple guests. The NVRAM template must be cloned
+#         for each new guest and configured read-write.
+#
+# @combined: the executable file contains both code and
+#            variable storage. The executable must be cloned
+#            for each new guest and configured read-write.
+#            No NVRAM template will be specified.
+#
+# @stateless: the executable file contains code and variable
+#             storage is not persisted. The executed must
+#             be configured read-only and can be shared
+#             between multiple guests. No NVRAM template
+#             will be specified.
+#
+# Since: 7.0.0
+##
+{ 'enum': 'FirmwareFlashMode',
+  'data': [ 'split', 'combined', 'stateless' ] }
+
 ##
 # @FirmwareMappingFlash:
 #
 # Describes loading and mapping properties for the firmware executable
 # and its accompanying NVRAM file, when @FirmwareDevice is @flash.
 #
-# @executable: Identifies the firmware executable. The firmware
-#              executable may be shared by multiple virtual machine
-#              definitions. The preferred corresponding QEMU command
-#              line options are
+# @mode: describes how the firmware build handles code versus variable
+#        storage. If not present, it must be treated as if it was
+#        configured with value ``split``. Since: 7.0.0
+#
+# @executable: Identifies the firmware executable. The @mode
+#              indicates whether there will be an associated
+#              NVRAM template present. The preferred
+#              corresponding QEMU command line options are
 #                  -drive if=none,id=pflash0,readonly=on,file=@executable.@filename,format=@executable.@format
 #                  -machine pflash0=pflash0
-#              or equivalent -blockdev instead of -drive.
+#              or equivalent -blockdev instead of -drive. When
+#              @mode is ``combined`` the executable must be
+#              cloned before use and configured with readonly=off.
 #              With QEMU versions older than 4.0, you have to use
 #                  -drive if=pflash,unit=0,readonly=on,file=@executable.@filename,format=@executable.@format
 #
 # @nvram-template: Identifies the NVRAM template compatible with
-#                  @executable. Management software instantiates an
+#                  @executable, when @mode is set to ``split``,
+#                  otherwise it should not be present.
+#                  Management software instantiates an
 #                  individual copy -- a specific NVRAM file -- from
 #                  @nvram-template.@filename for each new virtual
 #                  machine definition created. @nvram-template.@filename
@@ -246,8 +283,9 @@
 # Since: 3.0
 ##
 { 'struct' : 'FirmwareMappingFlash',
-  'data'   : { 'executable'     : 'FirmwareFlashFile',
-               'nvram-template' : 'FirmwareFlashFile' } }
+  'data'   : { '*mode': 'FirmwareFlashMode',
+               'executable'     : 'FirmwareFlashFile',
+               '*nvram-template' : 'FirmwareFlashFile' } }
 
 ##
 # @FirmwareMappingKernel:
-- 
2.34.1

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Kashyap Chamarthy]

On Mon, Jan 31, 2022 at 12:55:09PM +0000, Daniel P. Berrangé wrote:
> The current firmware descriptor schema for flash requires that both the
> executable to NVRAM template paths be provided. This is fine for the
> most common usage of EDK2 builds in virtualization where the separate
> _CODE and _VARS files are provided.
> 
> With confidential computing technology like AMD SEV, persistent storage
> of variables may be completely disabled because the firmware requires a
> known clean state on every cold boot. There is no way to express this
> in the firmware descriptor today.
> 
> Even with regular EDK2 builds it is possible to create a firmware that
> has both executable code and variable persistence in a single file. This
> hasn't been commonly used, since it would mean every guest bootup would
> need to clone the full firmware file, leading to redundant duplicate
> storage of the code portion. In some scenarios this may not matter and
> might even be beneficial. For example if a public cloud allows users to
> bring their own firmware, such that the user can pre-enroll their own
> secure boot keys, you're going to have this copied on disk for each
> tenant already. At this point the it can be simpler to just deal with
> a single file rather than split builds. The firmware descriptor ought
> to be able to express this combined firmware model too.

Cool, TIL that it's possible to include both the executable and the
variables file into a single file.

I briefly wondered if in this "combined" mode whether the no. of
duplicate copies can ever fill up the storage.  I doubt that, as the
combined size of _VARS + _CODE is just about 2MB.  So it only starts
mattering if you're running tens of thousands of guests.

> This all points towards expanding the schema for flash with a 'mode'
> concept:
> 
>  - "split" - the current implicit behaviour with separate files
>    for code and variables.
> 
>  - "combined" - the alternate behaviour where a single file contains
>    both code and variables.
> 
>  - "stateless" - the confidential computing use case where storage
>    of variables is completely disable, leaving only the code.
>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/interop/firmware.json | 54 ++++++++++++++++++++++++++++++++------
>  1 file changed, 46 insertions(+), 8 deletions(-)
> 
> In v2:
> 
>  - Mark 'mode' as optional field
>  - Misc typos in docs
> 
> diff --git a/docs/interop/firmware.json b/docs/interop/firmware.json
> index 8d8b0be030..f5d1d0b6e7 100644
> --- a/docs/interop/firmware.json
> +++ b/docs/interop/firmware.json
> @@ -210,24 +210,61 @@
>    'data'   : { 'filename' : 'str',
>                 'format'   : 'BlockdevDriver' } }
>  
> +
> +##
> +# @FirmwareFlashType:
> +#
> +# Describes how the firmware build handles code versus variable
> +# persistence.
> +#
> +# @split: the executable file contains code while the NVRAM
> +#         template provides variable storage. The executable
> +#         must be configured read-only and can be shared between
> +#         multiple guests. The NVRAM template must be cloned
> +#         for each new guest and configured read-write.
> +#
> +# @combined: the executable file contains both code and
> +#            variable storage. The executable must be cloned
> +#            for each new guest and configured read-write.
> +#            No NVRAM template will be specified.

Given my above wondering, is it worth adding a note here about storage
considerations when running large number of guests in the "combined"
mode?  If not, ignore my comment.

> +# @stateless: the executable file contains code and variable
> +#             storage is not persisted. The executed must

I guess you meant: s/executed/executable/

Whoever is applying the patch can touch it up.

> +#             be configured read-only and can be shared
> +#             between multiple guests. No NVRAM template
> +#             will be specified.
> +#
> +# Since: 7.0.0
> +##
> +{ 'enum': 'FirmwareFlashMode',
> +  'data': [ 'split', 'combined', 'stateless' ] }
> +
>  ##
>  # @FirmwareMappingFlash:
>  #
>  # Describes loading and mapping properties for the firmware executable
>  # and its accompanying NVRAM file, when @FirmwareDevice is @flash.
>  #
> -# @executable: Identifies the firmware executable. The firmware
> -#              executable may be shared by multiple virtual machine
> -#              definitions. The preferred corresponding QEMU command
> -#              line options are
> +# @mode: describes how the firmware build handles code versus variable
> +#        storage. If not present, it must be treated as if it was
> +#        configured with value ``split``. Since: 7.0.0

For consistency, might want to capitalize the first word:
s/describes/Describes/  

(Here too, maintainer can touch it up.)

[...]

The concept looks very clear, and obviously useful.  FWIW:

    Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com>    

> -- 
> 2.34.1
> 

-- 
/kashyap

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Daniel P. Berrangé]

On Mon, Jan 31, 2022 at 03:00:33PM +0100, Kashyap Chamarthy wrote:
> On Mon, Jan 31, 2022 at 12:55:09PM +0000, Daniel P. Berrangé wrote:
> > The current firmware descriptor schema for flash requires that both the
> > executable to NVRAM template paths be provided. This is fine for the
> > most common usage of EDK2 builds in virtualization where the separate
> > _CODE and _VARS files are provided.
> > 
> > With confidential computing technology like AMD SEV, persistent storage
> > of variables may be completely disabled because the firmware requires a
> > known clean state on every cold boot. There is no way to express this
> > in the firmware descriptor today.
> > 
> > Even with regular EDK2 builds it is possible to create a firmware that
> > has both executable code and variable persistence in a single file. This
> > hasn't been commonly used, since it would mean every guest bootup would
> > need to clone the full firmware file, leading to redundant duplicate
> > storage of the code portion. In some scenarios this may not matter and
> > might even be beneficial. For example if a public cloud allows users to
> > bring their own firmware, such that the user can pre-enroll their own
> > secure boot keys, you're going to have this copied on disk for each
> > tenant already. At this point the it can be simpler to just deal with
> > a single file rather than split builds. The firmware descriptor ought
> > to be able to express this combined firmware model too.
> 
> Cool, TIL that it's possible to include both the executable and the
> variables file into a single file.
> 
> I briefly wondered if in this "combined" mode whether the no. of
> duplicate copies can ever fill up the storage.  I doubt that, as the
> combined size of _VARS + _CODE is just about 2MB.  So it only starts
> mattering if you're running tens of thousands of guests.

When guest root / data disk sizes are measured in 100's of MB, or
GBs, I struggle to get worried about even a 16 MB OVMF  blob being
copied per guest.

The firmware can be provided in qcow2 format too, so if really
concerned, just create a qcow2 file with a backing store pointing
to the readonly master, so you're only paying the price of the
delta for any guest VARs writes. That's more efficient than what
we do today with copying the separate raw format VARS.fd file.

> > This all points towards expanding the schema for flash with a 'mode'
> > concept:
> > 
> >  - "split" - the current implicit behaviour with separate files
> >    for code and variables.
> > 
> >  - "combined" - the alternate behaviour where a single file contains
> >    both code and variables.
> > 
> >  - "stateless" - the confidential computing use case where storage
> >    of variables is completely disable, leaving only the code.
> >
> > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  docs/interop/firmware.json | 54 ++++++++++++++++++++++++++++++++------
> >  1 file changed, 46 insertions(+), 8 deletions(-)
> > 
> > In v2:
> > 
> >  - Mark 'mode' as optional field
> >  - Misc typos in docs
> > 
> > diff --git a/docs/interop/firmware.json b/docs/interop/firmware.json
> > index 8d8b0be030..f5d1d0b6e7 100644
> > --- a/docs/interop/firmware.json
> > +++ b/docs/interop/firmware.json
> > @@ -210,24 +210,61 @@
> >    'data'   : { 'filename' : 'str',
> >                 'format'   : 'BlockdevDriver' } }
> >  
> > +
> > +##
> > +# @FirmwareFlashType:
> > +#
> > +# Describes how the firmware build handles code versus variable
> > +# persistence.
> > +#
> > +# @split: the executable file contains code while the NVRAM
> > +#         template provides variable storage. The executable
> > +#         must be configured read-only and can be shared between
> > +#         multiple guests. The NVRAM template must be cloned
> > +#         for each new guest and configured read-write.
> > +#
> > +# @combined: the executable file contains both code and
> > +#            variable storage. The executable must be cloned
> > +#            for each new guest and configured read-write.
> > +#            No NVRAM template will be specified.
> 
> Given my above wondering, is it worth adding a note here about storage
> considerations when running large number of guests in the "combined"
> mode?  If not, ignore my comment.

I don't think its worth worrying about.

> 
> > +# @stateless: the executable file contains code and variable
> > +#             storage is not persisted. The executed must
> 
> I guess you meant: s/executed/executable/

Opp yes.

> Whoever is applying the patch can touch it up.
> 
> > +#             be configured read-only and can be shared
> > +#             between multiple guests. No NVRAM template
> > +#             will be specified.
> > +#
> > +# Since: 7.0.0
> > +##
> > +{ 'enum': 'FirmwareFlashMode',
> > +  'data': [ 'split', 'combined', 'stateless' ] }
> > +
> >  ##
> >  # @FirmwareMappingFlash:
> >  #
> >  # Describes loading and mapping properties for the firmware executable
> >  # and its accompanying NVRAM file, when @FirmwareDevice is @flash.
> >  #
> > -# @executable: Identifies the firmware executable. The firmware
> > -#              executable may be shared by multiple virtual machine
> > -#              definitions. The preferred corresponding QEMU command
> > -#              line options are
> > +# @mode: describes how the firmware build handles code versus variable
> > +#        storage. If not present, it must be treated as if it was
> > +#        configured with value ``split``. Since: 7.0.0
> 
> For consistency, might want to capitalize the first word:
> s/describes/Describes/

Yep

> 
> (Here too, maintainer can touch it up.)
> 
> [...]
> 
> The concept looks very clear, and obviously useful.  FWIW:
> 
>     Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com>    

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Kashyap Chamarthy]

On Mon, Jan 31, 2022 at 02:36:46PM +0000, Daniel P. Berrangé wrote:
> On Mon, Jan 31, 2022 at 03:00:33PM +0100, Kashyap Chamarthy wrote:
> > On Mon, Jan 31, 2022 at 12:55:09PM +0000, Daniel P. Berrangé wrote:

[...]

> > I briefly wondered if in this "combined" mode whether the no. of
> > duplicate copies can ever fill up the storage.  I doubt that, as the
> > combined size of _VARS + _CODE is just about 2MB.  So it only starts
> > mattering if you're running tens of thousands of guests.
> 
> When guest root / data disk sizes are measured in 100's of MB, or
> GBs, I struggle to get worried about even a 16 MB OVMF  blob being
> copied per guest.

Heh, fair enough.

> The firmware can be provided in qcow2 format too, so if really
> concerned, just create a qcow2 file with a backing store pointing
> to the readonly master, so you're only paying the price of the
> delta for any guest VARs writes. That's more efficient than what
> we do today with copying the separate raw format VARS.fd file.

That's nice, I didn't know the qcow2 possibility in this context.  For
some reason I assumed the file format always has to be raw here.  Your
qcow2 point above should  be documented, if it isn't already.  Although
I don't know the right place for it.

[...]

-- 
/kashyap

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Daniel P. Berrangé]

On Mon, Jan 31, 2022 at 04:21:36PM +0100, Kashyap Chamarthy wrote:
> On Mon, Jan 31, 2022 at 02:36:46PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Jan 31, 2022 at 03:00:33PM +0100, Kashyap Chamarthy wrote:
> > > On Mon, Jan 31, 2022 at 12:55:09PM +0000, Daniel P. Berrangé wrote:
> 
> [...]
> 
> > > I briefly wondered if in this "combined" mode whether the no. of
> > > duplicate copies can ever fill up the storage.  I doubt that, as the
> > > combined size of _VARS + _CODE is just about 2MB.  So it only starts
> > > mattering if you're running tens of thousands of guests.
> > 
> > When guest root / data disk sizes are measured in 100's of MB, or
> > GBs, I struggle to get worried about even a 16 MB OVMF  blob being
> > copied per guest.
> 
> Heh, fair enough.
> 
> > The firmware can be provided in qcow2 format too, so if really
> > concerned, just create a qcow2 file with a backing store pointing
> > to the readonly master, so you're only paying the price of the
> > delta for any guest VARs writes. That's more efficient than what
> > we do today with copying the separate raw format VARS.fd file.
> 
> That's nice, I didn't know the qcow2 possibility in this context.  For
> some reason I assumed the file format always has to be raw here.  Your
> qcow2 point above should  be documented, if it isn't already.  Although
> I don't know the right place for it.

There's already a format field in the descriptor, but even if the
firmware is distributed as raw, libvirt can choose to put qcow2
overlay on it, as its all configured with -blockdev

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Kashyap Chamarthy]

On Mon, Jan 31, 2022 at 03:35:02PM +0000, Daniel P. Berrangé wrote:
> On Mon, Jan 31, 2022 at 04:21:36PM +0100, Kashyap Chamarthy wrote:
> > On Mon, Jan 31, 2022 at 02:36:46PM +0000, Daniel P. Berrangé wrote:

[...]

> > > The firmware can be provided in qcow2 format too, so if really
> > > concerned, just create a qcow2 file with a backing store pointing
> > > to the readonly master, so you're only paying the price of the
> > > delta for any guest VARs writes. That's more efficient than what
> > > we do today with copying the separate raw format VARS.fd file.
> > 
> > That's nice, I didn't know the qcow2 possibility in this context.  For
> > some reason I assumed the file format always has to be raw here.  Your
> > qcow2 point above should  be documented, if it isn't already.  Although
> > I don't know the right place for it.
> 
> There's already a format field in the descriptor, but even if the
> firmware is distributed as raw, libvirt can choose to put qcow2
> overlay on it, as its all configured with -blockdev

Ah, understood.  

I should've first checked the spec to look for the @format field.  For
others reading the thread, the @format bit is located here
infirmware.json:

  [...]
  # @FirmwareFlashFile:
  #
  # Defines common properties that are necessary for loading a firmware
  # file into a pflash chip. The corresponding QEMU command line option is
  # "-drive file=@filename,format=@format". Note however that the
  # option-argument shown here is incomplete; it is completed under
  # @FirmwareMappingFlash.
  #
  # @filename: Specifies the filename on the host filesystem where the
  #            firmware file can be found.
  #
  # @format: Specifies the block format of the file pointed-to by
  #          @filename, such as @raw or @qcow2.
  [...]

-- 
/kashyap

Re: [PATCH v2] docs: expand firmware descriptor to allow flash without NVRAM

[Gerd Hoffmann]

On Mon, Jan 31, 2022 at 04:21:36PM +0100, Kashyap Chamarthy wrote:
> On Mon, Jan 31, 2022 at 02:36:46PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Jan 31, 2022 at 03:00:33PM +0100, Kashyap Chamarthy wrote:
> > > On Mon, Jan 31, 2022 at 12:55:09PM +0000, Daniel P. Berrangé wrote:
> 
> [...]
> 
> > > I briefly wondered if in this "combined" mode whether the no. of
> > > duplicate copies can ever fill up the storage.  I doubt that, as the
> > > combined size of _VARS + _CODE is just about 2MB.  So it only starts
> > > mattering if you're running tens of thousands of guests.
> > 
> > When guest root / data disk sizes are measured in 100's of MB, or
> > GBs, I struggle to get worried about even a 16 MB OVMF  blob being
> > copied per guest.
> 
> Heh, fair enough.

Main advantage of the split is that it is much easier to update the
firmware code without smashing the guest vars, not so much the disk
space requirements.

take care,
  Gerd