[RFC] OVMF NVRAM Variable Retention

[RFC] OVMF NVRAM Variable Retention

[aaron.young]

   Hello all,

   I have been tasked with the issue of UEFI NVRAM variable retention 
with Xen HVM/OVMF based guests and wanted to consult the experts before 
I undertake any significant coding effort. I am new to Xen so any 
help/direction would be appreciated.

   What I hope to gain from this RFC inquiry is the following:

   1. Has there been any significant past discussion regarding this 
issue? If so, can somebody kindly point me to these discussion(s) and/or 
give me an overview of the findings?

   2. Is there any current coding effort going on in this area?

   3. Any suggestions or proposals on how to fix the issue including any 
issues or caveats that I should be aware of?


Issue details:

  In a nutshell, if a Xen guest which has been configured to use OVMF is 
destroyed and re-created, the UEFI NVRAM variables are not retained. 
This seems to be a well-known issue to which there is no resolution. The 
issue seems to center around the fact that these Xen guests are not 
configured to use a file-backed backing store for the NVRAM variables 
(such as is commonly used with QEMU/KVM guests).

  This issue can cause OVMF OS boot option(s) to be lost preventing the 
ability to boot the OS after a guest is destroyed/re-created. It also 
has implications on secure boot (which require NVRAM variables to be 
retained).


Some key observations:

  1. OVMF currently supports memory-mapped, file-backed 'pflash' devices 
from Qemu. i.e. Qemu can map varstore files into memory for use by OVMF 
via -drive parameters such as:

       -drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd   \
       -drive if=pflash,format=raw,file=OVMF_VARS.fd

     OVMF scans the flash device (i.e. the Non-volatile data storage FV) 
for a mapped varstore and if found will use it store NVRAM variables 
(via its QemuFlashFvbServicesRuntimeDxe driver)

     This is outlined well in the OVMF whitepaper 
(http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt)

  2. When Xen HVM/OVMF guests are created, the qemu process that is 
executed (from xl) does not contain these -drive arguments. Thus qemu 
does not map a varstore for use by OVMF causing OVMF to fallback and use 
a simple memory buffer for NVRAM variables (if I'm not mistaken here). 
This causes the variables to not be retained across destroy/create 
operations on the VM.

  3. Unlike qemu/kvm guests where both OVMF and the varstore file are 
loaded/mapped into memory and executed directly by qemu, Xen appears to 
only use qemu as a "device model" and OVMF is not mapped/loaded directly 
by qemu. Instead, under Xen, OVMF is loaded indirectly by hvmloader 
(which is loaded by qemu). This could complicate the fix idea below.


Possible idea for fix:

    Since OVMF currently has support for QEMU mapped varstore files, it 
seems the most straightforward way to fix this issue would be for Xen 
(i.e. xl) to pass a -drive argument to qemu to specify a varstore file 
and to enhance the Xen memory init code in qemu (i.e. 
xen_hvm.c:xen_[hvm|ram]_init()) to map this file into memory for use by 
OVMF (possibly taking advantage of the already present qemu code to do 
such a mapping (i.e. code out of pc_memory_init()). As mentioned above, 
this would have to (somehow) be compatible with how OVMF is loaded 
indirectly by hvmloader.

   Any comments/suggestions/opinions/caveats on this approach?

   Or any other suggested approaches on how to fix this issue?


Thank you very much!

    -Aaron Young




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF NVRAM Variable Retention

[Ross Lagerwall]

On 06/06/2018 04:01 PM, aaron.young@oracle.com wrote:
> 
>    Hello all,
> 
>    I have been tasked with the issue of UEFI NVRAM variable retention 
> with Xen HVM/OVMF based guests and wanted to consult the experts before 
> I undertake any significant coding effort. I am new to Xen so any 
> help/direction would be appreciated.
> 
>    What I hope to gain from this RFC inquiry is the following:
> 
>    1. Has there been any significant past discussion regarding this 
> issue? If so, can somebody kindly point me to these discussion(s) and/or 
> give me an overview of the findings?
> 
>    2. Is there any current coding effort going on in this area?

Yes! I'm working on this at the moment.

> 
>    3. Any suggestions or proposals on how to fix the issue including any 
> issues or caveats that I should be aware of?
> 
> 
> Issue details:
> 
>   In a nutshell, if a Xen guest which has been configured to use OVMF is 
> destroyed and re-created, the UEFI NVRAM variables are not retained. 
> This seems to be a well-known issue to which there is no resolution. The 
> issue seems to center around the fact that these Xen guests are not 
> configured to use a file-backed backing store for the NVRAM variables 
> (such as is commonly used with QEMU/KVM guests).
> 
>   This issue can cause OVMF OS boot option(s) to be lost preventing the 
> ability to boot the OS after a guest is destroyed/re-created. It also 
> has implications on secure boot (which require NVRAM variables to be 
> retained).
> 
> 
> Some key observations:
> 
>   1. OVMF currently supports memory-mapped, file-backed 'pflash' devices 
> from Qemu. i.e. Qemu can map varstore files into memory for use by OVMF 
> via -drive parameters such as:
> 
>        -drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd   \
>        -drive if=pflash,format=raw,file=OVMF_VARS.fd
> 
>      OVMF scans the flash device (i.e. the Non-volatile data storage FV) 
> for a mapped varstore and if found will use it store NVRAM variables 
> (via its QemuFlashFvbServicesRuntimeDxe driver)
> 
>      This is outlined well in the OVMF whitepaper 
> (http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt)
> 
>   2. When Xen HVM/OVMF guests are created, the qemu process that is 
> executed (from xl) does not contain these -drive arguments. Thus qemu 
> does not map a varstore for use by OVMF causing OVMF to fallback and use 
> a simple memory buffer for NVRAM variables (if I'm not mistaken here). 
> This causes the variables to not be retained across destroy/create 
> operations on the VM.
> 
>   3. Unlike qemu/kvm guests where both OVMF and the varstore file are 
> loaded/mapped into memory and executed directly by qemu, Xen appears to 
> only use qemu as a "device model" and OVMF is not mapped/loaded directly 
> by qemu. Instead, under Xen, OVMF is loaded indirectly by hvmloader 
> (which is loaded by qemu). This could complicate the fix idea below.
> 
> 
> Possible idea for fix:
> 
>     Since OVMF currently has support for QEMU mapped varstore files, it 
> seems the most straightforward way to fix this issue would be for Xen 
> (i.e. xl) to pass a -drive argument to qemu to specify a varstore file 
> and to enhance the Xen memory init code in qemu (i.e. 
> xen_hvm.c:xen_[hvm|ram]_init()) to map this file into memory for use by 
> OVMF (possibly taking advantage of the already present qemu code to do 
> such a mapping (i.e. code out of pc_memory_init()). As mentioned above, 
> this would have to (somehow) be compatible with how OVMF is loaded 
> indirectly by hvmloader.
> 
>    Any comments/suggestions/opinions/caveats on this approach?

I did this a while back. It is easy enough to do:

1) Have Xen load OVMF_code.fd rather than the combined blob.

2) Tweak the location in guest memory where hvmloader loads the blob.

3) Tweak QEMU to load the OVMF_vars.fd blob at the correct location.

4) Start QEMU with emulated flash for OVMF_vars.fd only.

Tweaking the locations ensures that the various parts of OVMF are at the 
location it expects. If you want, I could probably dig up some patches 
(hacks) which do this.

> 
>    Or any other suggested approaches on how to fix this issue?

However... I did not like this approach for two reasons:

1) Having an emulated flash blob is difficult to manage outside of the 
guest (i.e. populating initial state, updating variables if needed). For 
XenServer, we want more flexibility.

2) While Secure Boot can be enabled with this implementation, it is not 
sufficiently secure because the guest is able to write anything it wants 
to the emulated flash. KVM solved this problem with SMM mode but I don't 
like that solution either.

So I am busy implementing:

A UEFI driver frontend which implements variable services by proxying 
requests to a backend running in dom0 (could be part of QEMU). This 
ensures security because the guest cannot directly write to the variable 
storage and the authentication checks are done outside of guest context. 
It allows flexibility because the backend can store the variables in any 
form (e.g. an sqlite database) rather than being restricted to an 
emulated flash blob.

So far I've got the normal variable services implemented and am busy 
with the authenticated variable services.

Let me know if you have any questions. I hope to present this approach 
at the QEMU/KVM forum later this year.

Hope that helps,
-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF NVRAM Variable Retention

[Roger Pau Monné]

On Thu, Jun 07, 2018 at 09:59:10AM +0100, Ross Lagerwall wrote:
> On 06/06/2018 04:01 PM, aaron.young@oracle.com wrote:
> >    Any comments/suggestions/opinions/caveats on this approach?
> 
> I did this a while back. It is easy enough to do:
> 
> 1) Have Xen load OVMF_code.fd rather than the combined blob.
> 
> 2) Tweak the location in guest memory where hvmloader loads the blob.
> 
> 3) Tweak QEMU to load the OVMF_vars.fd blob at the correct location.
> 
> 4) Start QEMU with emulated flash for OVMF_vars.fd only.
> 
> Tweaking the locations ensures that the various parts of OVMF are at the
> location it expects. If you want, I could probably dig up some patches
> (hacks) which do this.

I don't like to tie UEFI (OVMF) to QEMU. We want to also run guests
with UEFI firmware without QEMU, see below.

> > 
> >    Or any other suggested approaches on how to fix this issue?
> 
> However... I did not like this approach for two reasons:
> 
> 1) Having an emulated flash blob is difficult to manage outside of the guest
> (i.e. populating initial state, updating variables if needed). For
> XenServer, we want more flexibility.
> 
> 2) While Secure Boot can be enabled with this implementation, it is not
> sufficiently secure because the guest is able to write anything it wants to
> the emulated flash. KVM solved this problem with SMM mode but I don't like
> that solution either.
> 
> So I am busy implementing:
> 
> A UEFI driver frontend which implements variable services by proxying
> requests to a backend running in dom0 (could be part of QEMU). This ensures
> security because the guest cannot directly write to the variable storage and
> the authentication checks are done outside of guest context. It allows
> flexibility because the backend can store the variables in any form (e.g. an
> sqlite database) rather than being restricted to an emulated flash blob.

This seems like a good idea, specially because that means we don't
need QEMU to load anything into the guest, and this approach could be
likely used by PVH guests running with UEFI also.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF NVRAM Variable Retention

[aaron.young]

On 06/07/18 01:59, Ross Lagerwall wrote:
> On 06/06/2018 04:01 PM, aaron.young@oracle.com wrote:
>>
>>    Hello all,
>>
>>    I have been tasked with the issue of UEFI NVRAM variable retention 
>> with Xen HVM/OVMF based guests and wanted to consult the experts 
>> before I undertake any significant coding effort. I am new to Xen so 
>> any help/direction would be appreciated.
>>
>>    What I hope to gain from this RFC inquiry is the following:
>>
>>    1. Has there been any significant past discussion regarding this 
>> issue? If so, can somebody kindly point me to these discussion(s) 
>> and/or give me an overview of the findings?
>>
>>    2. Is there any current coding effort going on in this area?
>
> Yes! I'm working on this at the moment.

     Great! I'm glad I asked before starting implementation.

>
>>
>>    3. Any suggestions or proposals on how to fix the issue including 
>> any issues or caveats that I should be aware of?
>>
>>
>> Issue details:
>>
>>   In a nutshell, if a Xen guest which has been configured to use OVMF 
>> is destroyed and re-created, the UEFI NVRAM variables are not 
>> retained. This seems to be a well-known issue to which there is no 
>> resolution. The issue seems to center around the fact that these Xen 
>> guests are not configured to use a file-backed backing store for the 
>> NVRAM variables (such as is commonly used with QEMU/KVM guests).
>>
>>   This issue can cause OVMF OS boot option(s) to be lost preventing 
>> the ability to boot the OS after a guest is destroyed/re-created. It 
>> also has implications on secure boot (which require NVRAM variables 
>> to be retained).
>>
>>
>> Some key observations:
>>
>>   1. OVMF currently supports memory-mapped, file-backed 'pflash' 
>> devices from Qemu. i.e. Qemu can map varstore files into memory for 
>> use by OVMF via -drive parameters such as:
>>
>>        -drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd \
>>        -drive if=pflash,format=raw,file=OVMF_VARS.fd
>>
>>      OVMF scans the flash device (i.e. the Non-volatile data storage 
>> FV) for a mapped varstore and if found will use it store NVRAM 
>> variables (via its QemuFlashFvbServicesRuntimeDxe driver)
>>
>>      This is outlined well in the OVMF whitepaper 
>> (http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt)
>>
>>   2. When Xen HVM/OVMF guests are created, the qemu process that is 
>> executed (from xl) does not contain these -drive arguments. Thus qemu 
>> does not map a varstore for use by OVMF causing OVMF to fallback and 
>> use a simple memory buffer for NVRAM variables (if I'm not mistaken 
>> here). This causes the variables to not be retained across 
>> destroy/create operations on the VM.
>>
>>   3. Unlike qemu/kvm guests where both OVMF and the varstore file are 
>> loaded/mapped into memory and executed directly by qemu, Xen appears 
>> to only use qemu as a "device model" and OVMF is not mapped/loaded 
>> directly by qemu. Instead, under Xen, OVMF is loaded indirectly by 
>> hvmloader (which is loaded by qemu). This could complicate the fix 
>> idea below.
>>
>>
>> Possible idea for fix:
>>
>>     Since OVMF currently has support for QEMU mapped varstore files, 
>> it seems the most straightforward way to fix this issue would be for 
>> Xen (i.e. xl) to pass a -drive argument to qemu to specify a varstore 
>> file and to enhance the Xen memory init code in qemu (i.e. 
>> xen_hvm.c:xen_[hvm|ram]_init()) to map this file into memory for use 
>> by OVMF (possibly taking advantage of the already present qemu code 
>> to do such a mapping (i.e. code out of pc_memory_init()). As 
>> mentioned above, this would have to (somehow) be compatible with how 
>> OVMF is loaded indirectly by hvmloader.
>>
>>    Any comments/suggestions/opinions/caveats on this approach?
>
> I did this a while back. It is easy enough to do:
>
> 1) Have Xen load OVMF_code.fd rather than the combined blob.
>
> 2) Tweak the location in guest memory where hvmloader loads the blob.
>
> 3) Tweak QEMU to load the OVMF_vars.fd blob at the correct location.
>
> 4) Start QEMU with emulated flash for OVMF_vars.fd only.
>
> Tweaking the locations ensures that the various parts of OVMF are at 
> the location it expects. If you want, I could probably dig up some 
> patches (hacks) which do this.

     Yes, this is the _exact_ solution I had in mind.

     I would very much like to see your patches so I can see how you did 
it (which would assist me in learning the code). If you wouldn't mind, 
could please send the patches to me privately? I'd greatly appreciate 
it. Thanks!!

>
>>
>>    Or any other suggested approaches on how to fix this issue?
>
> However... I did not like this approach for two reasons:
>
> 1) Having an emulated flash blob is difficult to manage outside of the 
> guest (i.e. populating initial state, updating variables if needed). 
> For XenServer, we want more flexibility.
>
> 2) While Secure Boot can be enabled with this implementation, it is 
> not sufficiently secure because the guest is able to write anything it 
> wants to the emulated flash. KVM solved this problem with SMM mode but 
> I don't like that solution either.
>
> So I am busy implementing:
>
> A UEFI driver frontend which implements variable services by proxying 
> requests to a backend running in dom0 (could be part of QEMU). This 
> ensures security because the guest cannot directly write to the 
> variable storage and the authentication checks are done outside of 
> guest context. It allows flexibility because the backend can store the 
> variables in any form (e.g. an sqlite database) rather than being 
> restricted to an emulated flash blob.
>
> So far I've got the normal variable services implemented and am busy 
> with the authenticated variable services.
>
> Let me know if you have any questions. I hope to present this approach 
> at the QEMU/KVM forum later this year.
>
> Hope that helps,

   Thank you very much for the info.

    Look forward to your solution!

   -Aaron


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel