SVSM initiated early attestation / guest secrets injection

SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

In the previous discussion about vTPMs and their persistence:

https://lore.kernel.org/linux-coco/4a5bde7e-c473-0fdc-3c3f-e08321e0b911@linux.ibm.com/T/#e92f5ec757664eec5c993d4c5d85b11a13ac59d00

it was suggested we split the work into several distinct pieces

[quote]
  1. Ephemeral vTPM with attestation retrieved from guest
  2. Attestation and injection API from SVSM to host/guest end point
  3. SVSM API for saving TPM state
[/quote]

I've been thinking about items 2 and 3 on that list. Beyond the desire to
support persistence of the vTPM, I think there are other use cases that would
benefit from early attestation. The overarching appealing aspect though is
that we can largely isolate the guest OS userspace from having to know / do
anything about SEV-SNP. This is a benefit because the more we can make a guest
OS work just like it would on any other (non-confidential) VM / bare metal,
the less burden there is for the guest owner and guest OS vendor alike. Every
time there is a different code path / action sequence required for CVMs, we
increase the testing burden and knowledge burden. Infrequently used code paths
are invariably more likely to bitrot.

If the guest OS can unlock its disk secrets from a persistent vTPM without
having todo any attestation from the initrd, this simplifies the early boot
process for the guest. Especially not having to bring up networking at this
point in boot is a win, as that would add many failure scenarios, and we're
limited in per-VM configuration options until after the guest's root disk is
unlocked too.

Aside from a desire to persist vTPM, it would also be desirable to support
persistence of OVMF NVRAM variables. Traditional VMs have the split CODE.fd /
VARS.fd arrangement where VARS.fd is writable per-VM, but we don't use that
with confidential VMs as it wouldn't be covered by the SEV launch measurements.
This is a difference from non-CVM environments and there could be use cases
where OVMF NVRAM persistence is useful to support.

Lets assume as a starting point that the guest owner has designated an
attestation server for their CVM, and pre-loaded it with a set of secrets to
be released upon successful attestation of their CVM The attestation server
could be run by the cloud provider, or run by the guest owner themselves,
whichever suits their confidentiality needs best.


Just as we don't want to deal with the complexity of networking in the initrd,
we also don't want that in SVSM. Instead it is desirable to rely on the host
OS to act as a proxy between the attestation server and SVSM, while maintaining
confidentiality against this intentional MITM. A plain old serial UART can be
exposed to the guest OS for communication with the host attestation proxy.

The sequence of operations/communications would follow:

 1. Guest owner designates an attestation server to use with the CVM
    and provides its address and a public key associated with the attestation
    server to the hypervisor

 2. Hypervisor injects the attestation server public key to the CVM

 3. SVSM generates a public/private key pair using a public key algorithm to
    be specified

 4. SVSM requests an attestation report from SEV-SNP firmware, embedding a
    hash of the attestation server public key and its own public key.

 5. SVSM transmits the attestation report and the two public keys on the ISA
    serial port

 6. Host attestation proxy receives the attestation report and public key
    on QEMU serial port backend

 7. Host attestation proxy makes a HTTPS request to the designated attestation
    server associated with the CVM instance

 8. Remote attestation server validates the attestation report which
    establishes that

     * SVSM is running in a SEV-SNP VM
     * The hash of the public keys comes from SVSM

 8. Remote attestation server extracts the public keys hash from the
    validated attestation report
 
 9. Remote attestation server validates the two public keys which
    establishes that

      * SVSM has a valid public key for the attestation server
      * It can encrypt data to send back to SVSM

 10. Remote attestation server acquires secrets to be released to the CVM,
     encrypts them with SVSM’s public key, and signs them with its own
     private key

 11. Remote attestation server sends the HTTPS reply with the encrypted
     secrets as payload

 12. Host attestation proxy receives the HTTPS reply from the attestation
     server and transmits the payload on the QEMU serial port backend

 13. SVSM receives the attestation result payload from the ISA serial port

 14. SVSM verifies the signature of the attestation server which
     establishes that:

       * The data came from an attestation server associated
         with the public key is holds
       * Establishes that the data has not been tampered with
         in transit from the attestation server

 15. SVSM decrypts the secrets received using its private key

Use of a plain UART is suggested primarily for simplicity. The amount of
data to be transferred is just a few 10's of KB, so a high data rate channel
is not required. Further SVSM would block between steps 5 and 13 while waiting
for the attestation response, before launching the guest firmware. The serial
port would not be used again once the guest firmware is launched. Using
virtio-serial or similar feels like overkill, extra code complexity for little
obvious benefit. We could designate any of the traditional COM1/COM2/COM3/COM4
ioports, or designate a new IO port specifically for this purpose but still use
the simple UART protocol

It is suggested that the secrets released from the attestation server be
structured using the EFI secret table defined previously for SEV/SEV-ES
in

   https://github.com/torvalds/linux/blob/master/drivers/virt/coco/efi_secret/efi_secret.c

Thus when SVSM decrypts the secrets it can place them into guest RAM in the
address specified by OVMF, such that they are accessible by the efi_secret.ko
guest driver (or equivalent for non-Linux).

This on its own would allow for booting of encrypted root disks from the
initrd, without even needing any vTPM integration. For example, systemd
crypttab supports pointing to a plain file to acquire the LUKS passphrase
and can thus be pointed to the efi_secret file in sysfs.

This mechanism could also be used for injecting a host identity such as an
SSH server keypair. Injecting the SSH identity is a way to mitigate the main
risk in the above communications handshake where the malicious host proxy
doesn't talk to the user's attestation server, and just replies with a bunch
of secrets of its own choosing. If it did this the SSH identity would not
match what the guest owner expects and thus it could tell the VM is not
trustworthy on first use. Equivalently a HTTP server cert would serve much
the same purpose if the CVM were exposing an HTTP service.

This can also be used as a way to supply data for systemd credentials
(https://systemd.io/CREDENTIALS/), as a confidential alternative to SMBIOS OEM
strings or fw_cfg which are used in non-CVMs. It is important to be able to
get these injected at the start of boot, as various systemd startup units
can be influenced by this data. This avoids having to integrate support for
CVM attestation servers directly with systemd.


In addition to placing the secrets into guest RAM, however, SVSM can
designate certain secret UUIDs for its own usage, and extract those before
puitting the remaining secret table entries into guest RAM. One UUID could
identify a key to use for encrypting/decrypting the vTPM persistent state.
Another UUID could identify a key to use for encrypting / decrypting OVMF
NVRAM variable state.

In terms of how we actually provide vTPM / OVMF NVRAM state, it looks quite
straightforward to just use the existing pflash support in QEMU. On first
boot of the CVM, SVSM would have to initialize the pflash using the
designate encryption key, erasing any data the hypervisor may be exposed.
Either expose 2 pflash devices, one of vTPM and one for OVMF NVRAM, or just
split the 1 pflash into two parts.

With such a setup, the vTPM would by default be fully ephemeral. Only if the
attestation server provided a vTPM encryption secret, would the pflash be
used to persistent the vTPM state.

For the OVMF NVRAM persistent, SVSM would have to expose a service to
OVMF for the storage of NVRAM keys. I've not considered in any detail.

Even if SVSM does an attestation to acquire secrets and inject them into
the guest, the guest owner can repeat the attestation at any time when the
guest OS is running, even in early boot if so desired. The SVSM initiated
attestation provides a good default out of the box experiance though,
avoiding the need to make most guest OS images aware of fact that they're
running inside a CVM, and allows easy portability to other execution
environments for the VM image.

The scope of changes to SVSM to achieve the above looks fairly small. 
RSA key gen, some encrypt/decrypt ops, and expand the serial port impl
to support reading data.

There would be minimal (if any) change to QEMU, we just need to provide
the attestation server public key in some manner, perhaps fwcfg/SMBIOS,
since it doesn't need to be confidential.

We would need to write an attestation server proxy to run on the host,
connected to QEMU's serial port backend. This is likely the largest
amount of new code, and would need to know how to talk to any of the
various attestation server impls that are relevant.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

Hi Daniel,

On Thu, Jan 12, 2023 at 02:39:24PM +0000, Daniel P. Berrangé wrote:
>  4. SVSM requests an attestation report from SEV-SNP firmware, embedding a
>     hash of the attestation server public key and its own public key.
> 
>  5. SVSM transmits the attestation report and the two public keys on the ISA
>     serial port

This basically emulates the secret injection mechanism that was
implemented for SEV and SEV-ES by the firmware, right?

I see a problem here which allows a potential host owner to steal
the secrets from the attestation server. Maybe I am wrong, but I think
the following is possible with the above sequence:

	1. Host owner sets up a CVM like the guest owner would do, with
	   the same SVSM and Firmware binaries, same initial state and
	   so on, so that the initial measurement is the same as if the
	   VM was setup by the guest owner.

	2. Host owner attaches a different disk image with malicious
	   content, e.g. a boot loader that sends the secrets to the host
	   owner.

	3. SVSM and attestation server have no way of detecting this,
	   because the disk image is not part of the initial measurement
	   from the SNP firmware. So above sequence would complete and
	   SVSM gets the secrets from the attestation server.

	4. The malicious software loaded from the disk image gets access
	   to the secrets and sends them to the host owner.

	4. Host owner can use the guest owners secrets to steal data
	   from the encrypted disk image the guest owner provided.

To prevent this it is necessary that the measurement sent to the
attestation server includes all software and data which is
executed/loaded from unencrypted storage. For example, in a common boot
flow it needs to include the Grub binary and Grub configuration.

But these parts are not included in the initial measurement that the
SVSM gets from the SNP firmware.

Regards,

	Joerg

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Fri, 2023-01-13 at 18:22 +0100, Jörg Rödel wrote:
[...]
>         2. Host owner attaches a different disk image with malicious
>            content, e.g. a boot loader that sends the secrets to the
> host
>            owner.


This attack was discussed in the initial encrypted image prototype for
SEV and SEV-ES.  The idea is that either the disk is encrypted with the
injected encryption key (i.e. decodes correctly) or it isn't, in which
case it's up to the mounting component (grub in the initial prototype
and initrd in this proposal) to declare failure and destroy the
secrets.

In the original proposal, grub was combined with ovmf to produce
attestation covering the mounting component.  In this new scheme, you
could use measured direct boot to ensure that the initial attestation
covers the initrd since the boot partition isn't encrypted.

James

Re: SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

On Fri, Jan 13, 2023 at 06:22:38PM +0100, Jörg Rödel wrote:
> Hi Daniel,
> 
> On Thu, Jan 12, 2023 at 02:39:24PM +0000, Daniel P. Berrangé wrote:
> >  4. SVSM requests an attestation report from SEV-SNP firmware, embedding a
> >     hash of the attestation server public key and its own public key.
> > 
> >  5. SVSM transmits the attestation report and the two public keys on the ISA
> >     serial port
> 
> This basically emulates the secret injection mechanism that was
> implemented for SEV and SEV-ES by the firmware, right?
> 
> I see a problem here which allows a potential host owner to steal
> the secrets from the attestation server. Maybe I am wrong, but I think
> the following is possible with the above sequence:
> 
> 	1. Host owner sets up a CVM like the guest owner would do, with
> 	   the same SVSM and Firmware binaries, same initial state and
> 	   so on, so that the initial measurement is the same as if the
> 	   VM was setup by the guest owner.
> 
> 	2. Host owner attaches a different disk image with malicious
> 	   content, e.g. a boot loader that sends the secrets to the host
> 	   owner.
> 
> 	3. SVSM and attestation server have no way of detecting this,
> 	   because the disk image is not part of the initial measurement
> 	   from the SNP firmware. So above sequence would complete and
> 	   SVSM gets the secrets from the attestation server.
> 
> 	4. The malicious software loaded from the disk image gets access
> 	   to the secrets and sends them to the host owner.
> 
> 	4. Host owner can use the guest owners secrets to steal data
> 	   from the encrypted disk image the guest owner provided.
> 
> To prevent this it is necessary that the measurement sent to the
> attestation server includes all software and data which is
> executed/loaded from unencrypted storage. For example, in a common boot
> flow it needs to include the Grub binary and Grub configuration.
> 
> But these parts are not included in the initial measurement that the
> SVSM gets from the SNP firmware.

Aside from what James' says, one option is to lockdown the OVMF and
secureboot chain. eg OVMF built with SecureBoot=on, and instead of
having the generic Microsoft keys enrolled, have a distro specific
key enrolled. That distro key would have to be one that is only
used for signing UKIs (Unified Kernel Images), and the initrd embedded
in the UKI would need to be designed abort if it finds the disk is
not encrypted.

The main challenge here is that you don't have a single OVMF anymore.
You have many OVMFs, one for each distinct set of distro SecureBoot
keys, and the attestation server has to decide which one it wants
based on the distro the VM is expected to use.

If we don't do any of this, then the early attestation and secret
fetch would have to be limited to /only/ the secret for unlocking
the vTPM state, and not injected into the guest OS. The OS would
have to do everything else with the vTPM and PCR sealed data to
achieve the same end results of only getting access to secrets if
the vTPM sees the set of PCR values corresponding to the desired
UKI secureboot keys.

Ultimately the latter is probably the more common case and more
flexible.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[Dionna Amalie Glaze]

>
> Aside from what James' says, one option is to lockdown the OVMF and
> secureboot chain. eg OVMF built with SecureBoot=on, and instead of
> having the generic Microsoft keys enrolled, have a distro specific
> key enrolled. That distro key would have to be one that is only
> used for signing UKIs (Unified Kernel Images), and the initrd embedded
> in the UKI would need to be designed abort if it finds the disk is
> not encrypted.
>

IIUC, the disk doesn't need to be encrypted, it needs integrity. We
can't treat booting to an encrypted disk as booting to the expected
encrypted disk.
Everything still depends on rooting your core secrets to your
workload's identity. This relies on an integrity-protected initrd that
has its root hash measured as part of the kernel_cmdline.

The secret-holding server should only ever release secrets encrypted
by an attested vTPM binary (SVSM-generated public key) that has sealed
the secret to the workload PCRs.
This kicks the problem to the server and VM having pre-agreed on
what's the expected combination of OVMF and SVSM. So yes the vTPM gets
its core secret transferred to it before the PCRs get set to the
expected values, but that's part of the attestation dance with the
initial registration with the secret-holding server.

> The main challenge here is that you don't have a single OVMF anymore.
> You have many OVMFs, one for each distinct set of distro SecureBoot
> keys, and the attestation server has to decide which one it wants
> based on the distro the VM is expected to use.
>
> If we don't do any of this, then the early attestation and secret
> fetch would have to be limited to /only/ the secret for unlocking
> the vTPM state, and not injected into the guest OS. The OS would
> have to do everything else with the vTPM and PCR sealed data to
> achieve the same end results of only getting access to secrets if
> the vTPM sees the set of PCR values corresponding to the desired
> UKI secureboot keys.
>
> Ultimately the latter is probably the more common case and more
> flexible.
>

I don't think there can ever be a one-size-fits all base measurement
to seal secrets to, since there's always the final variable of the
workload.


-- 
-Dionna Glaze, PhD (she/her)

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Fri, Jan 13, 2023 at 01:02:30PM -0500, James Bottomley wrote:
> In the original proposal, grub was combined with ovmf to produce
> attestation covering the mounting component.  In this new scheme, you
> could use measured direct boot to ensure that the initial attestation
> covers the initrd since the boot partition isn't encrypted.

This can work in some environments, but for a generic distro it is not
very useful because it make is difficult to do image updates on the live
system. Well, except you use the first kernel and initrd (which is
measuresd) as the boot-loader to kexec into the distro kernel later,
which is loaded from the encrypted disk. This scheme is used on S390
Confidential VMs to some extent.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Fri, Jan 13, 2023 at 06:28:32PM +0000, Daniel P. Berrangé wrote:
> Aside from what James' says, one option is to lockdown the OVMF and
> secureboot chain. eg OVMF built with SecureBoot=on, and instead of
> having the generic Microsoft keys enrolled, have a distro specific
> key enrolled. That distro key would have to be one that is only
> used for signing UKIs (Unified Kernel Images), and the initrd embedded
> in the UKI would need to be designed abort if it finds the disk is
> not encrypted.
> 
> The main challenge here is that you don't have a single OVMF anymore.
> You have many OVMFs, one for each distinct set of distro SecureBoot
> keys, and the attestation server has to decide which one it wants
> based on the distro the VM is expected to use.

Yeah, as you said, this requires OS-vendor specific firmware in the VM,
I doubt the CSPs will implement offerings this way. More likely is
that a CSP will deploy its forked OVMF BIOS with a set of secure boot
keys, which will likely also include keys from the CSP (especially if
the CSP also provides its own distro). So you still need to trust the
CSP that it will not fake one of your VMs to steal secrets.

As James also said, the measurement to unlock secrets need to include
all software/data components up to the point where the encrypted disk
gets mounted.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Sat, 2023-01-14 at 18:08 +0100, Jörg Rödel wrote:

[...]
> As James also said, the measurement to unlock secrets need to include
> all software/data components up to the point where the encrypted disk
> gets mounted.

Well, we have a prototype in IBM Research using keylime to do this
based on the vTPM measurements.  We currently bring up a network
interface inside the initrd to run the keylime agent, but if you're
already inventing a non-network method for attestation, there's no
reason we couldn't transport TPM quotes over it as well.

James
 

Re: SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

On Fri, Jan 13, 2023 at 10:52:57AM -0800, Dionna Amalie Glaze wrote:
> >
> > Aside from what James' says, one option is to lockdown the OVMF and
> > secureboot chain. eg OVMF built with SecureBoot=on, and instead of
> > having the generic Microsoft keys enrolled, have a distro specific
> > key enrolled. That distro key would have to be one that is only
> > used for signing UKIs (Unified Kernel Images), and the initrd embedded
> > in the UKI would need to be designed abort if it finds the disk is
> > not encrypted.
> >
> 
> IIUC, the disk doesn't need to be encrypted, it needs integrity. We
> can't treat booting to an encrypted disk as booting to the expected
> encrypted disk.

The attacker does not know the keyslot passphrase for the expected
encrypted disk. Thus if the initrd successfully unlocks the root
disk keyslot, we know it was presented the expected one. The only
serious attack is for the encrypted disk to be substituted with an
unencrypted disk, which can be handled by having the initrd refuse
to boot with any unencrypted root disk.

Other attacks on the encrypted disk such as tampering with the
cipher text would be mere denial of service. Use of LUKS AEAD
ciphers would mitigate that, but other integrity checking layers
can work too.

> The secret-holding server should only ever release secrets encrypted
> by an attested vTPM binary (SVSM-generated public key) that has sealed
> the secret to the workload PCRs.
> This kicks the problem to the server and VM having pre-agreed on
> what's the expected combination of OVMF and SVSM. So yes the vTPM gets
> its core secret transferred to it before the PCRs get set to the
> expected values, but that's part of the attestation dance with the
> initial registration with the secret-holding server.

To be clear, I think the vTPM is a good thing, but it does not look
like it is mandatory to me. If you lock down your firmware and initrd
sufficiently strict in its impl, a vTPM is not a blocker. What a vTPM
does is make the system much more flexible, since you can come up with
a wide variety of PCR policies for tieing unlock to. When not having a
vTPM you're effectively having to rely on one very specific policy that
is hardcoded through initrd/firmware impl.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Sat, Jan 14, 2023 at 01:22:41PM -0500, James Bottomley wrote:
> On Sat, 2023-01-14 at 18:08 +0100, Jörg Rödel wrote:
> 
> [...]
> > As James also said, the measurement to unlock secrets need to include
> > all software/data components up to the point where the encrypted disk
> > gets mounted.
> 
> Well, we have a prototype in IBM Research using keylime to do this
> based on the vTPM measurements.  We currently bring up a network
> interface inside the initrd to run the keylime agent, but if you're
> already inventing a non-network method for attestation, there's no
> reason we couldn't transport TPM quotes over it as well.

So you are unlocking the disk via keylime remote attestation. This means
you need a measured initrd, right?

We were thinking about doing disk-unlocking via TPM in Grub, but that
has the problem that we need to securely deploy the TPM state at SVSM
init-time.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Mon, 2023-01-16 at 17:55 +0100, Jörg Rödel wrote:
> On Sat, Jan 14, 2023 at 01:22:41PM -0500, James Bottomley wrote:
> > On Sat, 2023-01-14 at 18:08 +0100, Jörg Rödel wrote:
> > 
> > [...]
> > > As James also said, the measurement to unlock secrets need to
> > > include all software/data components up to the point where the
> > > encrypted disk gets mounted.
> > 
> > Well, we have a prototype in IBM Research using keylime to do this
> > based on the vTPM measurements.  We currently bring up a network
> > interface inside the initrd to run the keylime agent, but if you're
> > already inventing a non-network method for attestation, there's no
> > reason we couldn't transport TPM quotes over it as well.
> 
> So you are unlocking the disk via keylime remote attestation. This
> means you need a measured initrd, right?

Right, the current measured boot for a standard machine includes a
measurement of the initrd.

> We were thinking about doing disk-unlocking via TPM in Grub, but that
> has the problem that we need to securely deploy the TPM state at SVSM
> init-time.

Not necessarily.  We're prototyping it with an intermediate parent key
which can be injected after measurement using the keylime facility.

James

Re: SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

On Mon, Jan 16, 2023 at 05:55:52PM +0100, Jörg Rödel wrote:
> On Sat, Jan 14, 2023 at 01:22:41PM -0500, James Bottomley wrote:
> > On Sat, 2023-01-14 at 18:08 +0100, Jörg Rödel wrote:
> > 
> > [...]
> > > As James also said, the measurement to unlock secrets need to include
> > > all software/data components up to the point where the encrypted disk
> > > gets mounted.
> > 
> > Well, we have a prototype in IBM Research using keylime to do this
> > based on the vTPM measurements.  We currently bring up a network
> > interface inside the initrd to run the keylime agent, but if you're
> > already inventing a non-network method for attestation, there's no
> > reason we couldn't transport TPM quotes over it as well.
> 
> So you are unlocking the disk via keylime remote attestation. This means
> you need a measured initrd, right?
> 
> We were thinking about doing disk-unlocking via TPM in Grub, but that
> has the problem that we need to securely deploy the TPM state at SVSM
> init-time.

IMHO it is undesirable to build a reliance on a specific bootloader.

My desire is that we have the stateful TPM in SVSM, such that once
the UEFI firmware starts everything functions identically to how
it would in bare metal or non-confidential VMs with a TPM. eg the
ability to use all the normal Linux tools, and especially the
standard systemd integration with LUKS and TPMs.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Mon, Jan 16, 2023 at 11:59:17AM -0500, James Bottomley wrote:
> Not necessarily.  We're prototyping it with an intermediate parent key
> which can be injected after measurement using the keylime facility.

Okay, that way we can defer TPM secret deployment to the time grub runs,
at least in our case where /boot is encrypted.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Mon, Jan 16, 2023 at 05:13:00PM +0000, Daniel P. Berrangé wrote:
> IMHO it is undesirable to build a reliance on a specific bootloader.
> 
> My desire is that we have the stateful TPM in SVSM, such that once
> the UEFI firmware starts everything functions identically to how
> it would in bare metal or non-confidential VMs with a TPM. eg the
> ability to use all the normal Linux tools, and especially the
> standard systemd integration with LUKS and TPMs.

Yes, a secure stateful TPM is my goal as well. Btw, I thought a bit more
about your proposal, and if the injected secret at SVSM init time is the
TPM state, then the SVSM will keep the actual secret (== LUKS key) sealed
until the boot measurements (bootloader + configuration) happened.

This invalidates the attack I described earlier in this thread, as
injected software from the CSP still has no way to fake these
measurments to do the unsealing.

After the bootloader gets the LUKS key it can load kernel and initrd
from encrypted storage, so that there is no immediate need to measure
those.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[Christophe de Dinechin Dupont de Dinechin]

> On 13 Jan 2023, at 19:02, James Bottomley <jejb@linux.ibm.com> wrote:
> 
> On Fri, 2023-01-13 at 18:22 +0100, Jörg Rödel wrote:
> [...]
>>         2. Host owner attaches a different disk image with malicious
>>            content, e.g. a boot loader that sends the secrets to the
>> host
>>            owner.
> 
> 
> This attack was discussed in the initial encrypted image prototype for
> SEV and SEV-ES.  The idea is that either the disk is encrypted with the
> injected encryption key (i.e. decodes correctly) or it isn't, in which
> case it's up to the mounting component (grub in the initial prototype
> and initrd in this proposal) to declare failure and destroy the
> secrets.

However, I recall discussions about the possibility that the host
owner could trigger a fallback back that mounts an *unencrypted*
disk successfully (or even discovers it). I remember discussions
on such cases with Daniel (about auto-discovery of boot partitions)
and Gerd (about fallback path for “compatibility” reasons).

So I believe that we need to clarify how we prevent the discovery
and mounting of non-encrypted partitions, no?

> 
> In the original proposal, grub was combined with ovmf to produce
> attestation covering the mounting component.  In this new scheme, you
> could use measured direct boot to ensure that the initial attestation
> covers the initrd since the boot partition isn't encrypted.
> 
> James
> 
> 

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Thu, 2023-01-19 at 15:05 +0100, Christophe de Dinechin Dupont de
Dinechin wrote:
> 
> 
> > On 13 Jan 2023, at 19:02, James Bottomley <jejb@linux.ibm.com>
> > wrote:
> > 
> > On Fri, 2023-01-13 at 18:22 +0100, Jörg Rödel wrote:
> > [...]
> > >         2. Host owner attaches a different disk image with
> > > malicious content, e.g. a boot loader that sends the secrets to
> > > the host owner.
> > 
> > 
> > This attack was discussed in the initial encrypted image prototype
> > for SEV and SEV-ES.  The idea is that either the disk is encrypted
> > with the injected encryption key (i.e. decodes correctly) or it
> > isn't, in which case it's up to the mounting component (grub in the
> > initial prototype and initrd in this proposal) to declare failure
> > and destroy the secrets.
> 
> However, I recall discussions about the possibility that the host
> owner could trigger a fallback back that mounts an *unencrypted*
> disk successfully (or even discovers it). I remember discussions
> on such cases with Daniel (about auto-discovery of boot partitions)
> and Gerd (about fallback path for “compatibility” reasons).
> 
> So I believe that we need to clarify how we prevent the discovery
> and mounting of non-encrypted partitions, no?

Yes, you have to think about this.  Possessing the secrets is the
problem if you don't trust what you've mounted, so in the original
prototype the secrets got destroyed if they weren't used (as in you
couldn't crypto mount the disk), so you can fallback to a potentially
untrusted disk if you can ensure the secrets can't be leaked.  I
suppose secret destruction and fallback should be a configurable policy
of the system instead of hard coded as I did in the initial prototype.

James

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Thu, Jan 19, 2023 at 09:10:48AM -0500, James Bottomley wrote:
> Yes, you have to think about this.  Possessing the secrets is the
> problem if you don't trust what you've mounted, so in the original
> prototype the secrets got destroyed if they weren't used (as in you
> couldn't crypto mount the disk), so you can fallback to a potentially
> untrusted disk if you can ensure the secrets can't be leaked.  I
> suppose secret destruction and fallback should be a configurable policy
> of the system instead of hard coded as I did in the initial prototype.

Is this still an issue in the SVSM version of this? The secrets will
stay in SVSM memory at VMPL0 and the higher VMPLs will not be able to
access them until they proved that only trusted software was loaded.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Thu, 2023-01-19 at 22:18 +0100, Jörg Rödel wrote:
> On Thu, Jan 19, 2023 at 09:10:48AM -0500, James Bottomley wrote:
> > Yes, you have to think about this.  Possessing the secrets is the
> > problem if you don't trust what you've mounted, so in the original
> > prototype the secrets got destroyed if they weren't used (as in you
> > couldn't crypto mount the disk), so you can fallback to a
> > potentially untrusted disk if you can ensure the secrets can't be
> > leaked.  I suppose secret destruction and fallback should be a
> > configurable policy of the system instead of hard coded as I did in
> > the initial prototype.
> 
> Is this still an issue in the SVSM version of this? The secrets will
> stay in SVSM memory at VMPL0 and the higher VMPLs will not be able to
> access them until they proved that only trusted software was loaded.

How can they stay there?  Even if the SVSM is the point of first
contact to receive the secret, it must give the secret to higher VMPLs
to try the mount, so the higher VMPLs have to destroy the secret they
were given on failure.

James

Re: SVSM initiated early attestation / guest secrets injection

[Jörg Rödel]

On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley wrote:
> How can they stay there?  Even if the SVSM is the point of first
> contact to receive the secret, it must give the secret to higher VMPLs
> to try the mount, so the higher VMPLs have to destroy the secret they
> were given on failure.

My thinking was that the SVSM will only hand out the secrets once the
code in higher VMPLs has proven itself to be trusted. But it is possible
for an attacker to create an image with the expected parts to get the
measurements right and then use a fall-back mount if decryption fails to
steal the secret.

But such mount fallbacks a generally a bad idea in a CVM.

Regards,

-- 
Jörg Rödel
jroedel@suse.de

SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany

(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman

Re: SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

On Fri, Jan 20, 2023 at 09:37:35AM +0100, Jörg Rödel wrote:
> On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley wrote:
> > How can they stay there?  Even if the SVSM is the point of first
> > contact to receive the secret, it must give the secret to higher VMPLs
> > to try the mount, so the higher VMPLs have to destroy the secret they
> > were given on failure.
> 
> My thinking was that the SVSM will only hand out the secrets once the
> code in higher VMPLs has proven itself to be trusted. But it is possible
> for an attacker to create an image with the expected parts to get the
> measurements right and then use a fall-back mount if decryption fails to
> steal the secret.

With a vTPM, and systemd-pcrphase.service, it is possible to seal
secrets to prevent such an attack, by including phase=initrd in
the sealing PCRs. IOW, nothing after the initrd can unseal the
secrets, even if a different filesystem mount was substituted.

Secrets that are needed /after/ the root FS is mounted (eg SSH
host key, Apache HTTP certs) can be sealed with a PCR set that
covers the LUKS master key hash, so that they can only be acccessed
if the correct rootfs was unlocked.

In the vTPM case, the guest owner is deciding policy when they
seal secrets to the vTPM during guest provisioning initially.
If we try to support a scenario without a vTPM, then the guest
owner has to determine policy by using a carefully built initrd
that does not do any fallbacks, and they have to rely on having
a known firmware with a particular set to certs enrolled, which
has support implications for public clouds. Much less flexible.
 
> But such mount fallbacks a generally a bad idea in a CVM.

Yep, but hopefully not fatal, if the user picks a suitable set
of PCRs for sealing data.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Fri, 2023-01-20 at 09:37 +0100, Jörg Rödel wrote:
> On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley wrote:
> > How can they stay there?  Even if the SVSM is the point of first
> > contact to receive the secret, it must give the secret to higher
> > VMPLs to try the mount, so the higher VMPLs have to destroy the
> > secret they were given on failure.
> 
> My thinking was that the SVSM will only hand out the secrets once the
> code in higher VMPLs has proven itself to be trusted.

Yes, that's quite correct, but not inconsistent with the need to pass
secrets up before you know there's a problem.

The posited situation is that the trusted initrd/OS needs to mount a
cloud supplied encrypted disk.  There's no assessment the SVSM can make
of that *before* the mount is actually tried, so the SVSM has to
release the secrets based on the fact that the measured initrd/OS is
trusted to execute the correct policy on mount failure. i.e. destroy
the secrets.

>  But it is possible for an attacker to create an image with the
> expected parts to get the measurements right and then use a fall-back
> mount if decryption fails to steal the secret.

No, that's the principle of chain of trust.  The attacker can't insert
the bad component to replace any of the measured parts, so if you
measure the part that uses the secret and you trust it to handle the
secret correctly there's no substitution attack which can exfiltrate
it.

> But such mount fallbacks a generally a bad idea in a CVM.

I won't dispute that, but even if you don't fallback, for security's
sake you still have to destroy the secret ASAP because the longer you
run with it the wider the window of exposure to an exploit attack in a
measured component.

James

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Fri, 2023-01-20 at 08:57 +0000, Daniel P. Berrangé wrote:
> On Fri, Jan 20, 2023 at 09:37:35AM +0100, Jörg Rödel wrote:
> > On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley wrote:
> > > How can they stay there?  Even if the SVSM is the point of first
> > > contact to receive the secret, it must give the secret to higher
> > > VMPLs
> > > to try the mount, so the higher VMPLs have to destroy the secret
> > > they
> > > were given on failure.
> > 
> > My thinking was that the SVSM will only hand out the secrets once
> > the
> > code in higher VMPLs has proven itself to be trusted. But it is
> > possible
> > for an attacker to create an image with the expected parts to get
> > the
> > measurements right and then use a fall-back mount if decryption
> > fails to
> > steal the secret.
> 
> With a vTPM, and systemd-pcrphase.service, it is possible to seal
> secrets to prevent such an attack, by including phase=initrd in
> the sealing PCRs. IOW, nothing after the initrd can unseal the
> secrets, even if a different filesystem mount was substituted.
> 
> Secrets that are needed /after/ the root FS is mounted (eg SSH
> host key, Apache HTTP certs) can be sealed with a PCR set that
> covers the LUKS master key hash, so that they can only be acccessed
> if the correct rootfs was unlocked.

But that provides no additional security: the LUKS master hash is just
a hash of an area on the disk which anyone controlling the device can
see.  I can duplicate this area and still substitute my own volume
(which would still fail to decrypt when presented with the key), so the
trusted mounting component sill has to correctly handle the failure
case.

> In the vTPM case, the guest owner is deciding policy when they
> seal secrets to the vTPM during guest provisioning initially.
> If we try to support a scenario without a vTPM, then the guest
> owner has to determine policy by using a carefully built initrd
> that does not do any fallbacks, and they have to rely on having
> a known firmware with a particular set to certs enrolled, which
> has support implications for public clouds. Much less flexible.
>  
> > But such mount fallbacks a generally a bad idea in a CVM.
> 
> Yep, but hopefully not fatal, if the user picks a suitable set
> of PCRs for sealing data.

You can't rely only on PCRs for this because they don't contain enough
information, you must code the correct policy into the trusted mount
component as well.

James

Re: SVSM initiated early attestation / guest secrets injection

[Daniel P. Berrangé]

On Fri, Jan 20, 2023 at 07:39:30AM -0500, James Bottomley wrote:
> On Fri, 2023-01-20 at 08:57 +0000, Daniel P. Berrangé wrote:
> > On Fri, Jan 20, 2023 at 09:37:35AM +0100, Jörg Rödel wrote:
> > > On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley wrote:
> > > > How can they stay there?  Even if the SVSM is the point of first
> > > > contact to receive the secret, it must give the secret to higher
> > > > VMPLs
> > > > to try the mount, so the higher VMPLs have to destroy the secret
> > > > they
> > > > were given on failure.
> > > 
> > > My thinking was that the SVSM will only hand out the secrets once
> > > the
> > > code in higher VMPLs has proven itself to be trusted. But it is
> > > possible
> > > for an attacker to create an image with the expected parts to get
> > > the
> > > measurements right and then use a fall-back mount if decryption
> > > fails to
> > > steal the secret.
> > 
> > With a vTPM, and systemd-pcrphase.service, it is possible to seal
> > secrets to prevent such an attack, by including phase=initrd in
> > the sealing PCRs. IOW, nothing after the initrd can unseal the
> > secrets, even if a different filesystem mount was substituted.
> > 
> > Secrets that are needed /after/ the root FS is mounted (eg SSH
> > host key, Apache HTTP certs) can be sealed with a PCR set that
> > covers the LUKS master key hash, so that they can only be acccessed
> > if the correct rootfs was unlocked.
> 
> But that provides no additional security: the LUKS master hash is just
> a hash of an area on the disk which anyone controlling the device can
> see.  I can duplicate this area and still substitute my own volume
> (which would still fail to decrypt when presented with the key), so the
> trusted mounting component sill has to correctly handle the failure
> case.

Yes, it would have to be a hash that is different from the
hash that's exposed in clear text in the LUKS header. For
example it could be hash(master_key || uuid). 


> > In the vTPM case, the guest owner is deciding policy when they
> > seal secrets to the vTPM during guest provisioning initially.
> > If we try to support a scenario without a vTPM, then the guest
> > owner has to determine policy by using a carefully built initrd
> > that does not do any fallbacks, and they have to rely on having
> > a known firmware with a particular set to certs enrolled, which
> > has support implications for public clouds. Much less flexible.
> >  
> > > But such mount fallbacks a generally a bad idea in a CVM.
> > 
> > Yep, but hopefully not fatal, if the user picks a suitable set
> > of PCRs for sealing data.
> 
> You can't rely only on PCRs for this because they don't contain enough
> information, you must code the correct policy into the trusted mount
> component as well.

Well it entirely depends on what info is being measured into the
PCRs. If what systemd measures into PCRs in the initrd is not
sufficient to allow safe sealing policies, we can request further
info.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: SVSM initiated early attestation / guest secrets injection

[James Bottomley]

On Fri, 2023-01-20 at 12:51 +0000, Daniel P. Berrangé wrote:
> On Fri, Jan 20, 2023 at 07:39:30AM -0500, James Bottomley wrote:
> > On Fri, 2023-01-20 at 08:57 +0000, Daniel P. Berrangé wrote:
> > > On Fri, Jan 20, 2023 at 09:37:35AM +0100, Jörg Rödel wrote:
> > > > On Thu, Jan 19, 2023 at 04:29:23PM -0500, James Bottomley
> > > > wrote:
> > > > > How can they stay there?  Even if the SVSM is the point of
> > > > > first contact to receive the secret, it must give the secret
> > > > > to higher VMPLs to try the mount, so the higher VMPLs have to
> > > > > destroy the secret they were given on failure.
> > > > 
> > > > My thinking was that the SVSM will only hand out the secrets
> > > > once the code in higher VMPLs has proven itself to be trusted.
> > > > But it is possible for an attacker to create an image with the
> > > > expected parts to get the measurements right and then use a
> > > > fall-back mount if decryption fails to steal the secret.
> > > 
> > > With a vTPM, and systemd-pcrphase.service, it is possible to seal
> > > secrets to prevent such an attack, by including phase=initrd in
> > > the sealing PCRs. IOW, nothing after the initrd can unseal the
> > > secrets, even if a different filesystem mount was substituted.
> > > 
> > > Secrets that are needed /after/ the root FS is mounted (eg SSH
> > > host key, Apache HTTP certs) can be sealed with a PCR set that
> > > covers the LUKS master key hash, so that they can only be
> > > acccessed if the correct rootfs was unlocked.
> > 
> > But that provides no additional security: the LUKS master hash is
> > just a hash of an area on the disk which anyone controlling the
> > device can see.  I can duplicate this area and still substitute my
> > own volume (which would still fail to decrypt when presented with
> > the key), so the trusted mounting component sill has to correctly
> > handle the failure case.
> 
> Yes, it would have to be a hash that is different from the
> hash that's exposed in clear text in the LUKS header.

True, but whatever thing you has hash to be available to the trusted
mounting entity before the secret is released, so all it can know are
what an observer can see ... there's no other thing you can hash that
the trusted mounting component knows that an observer can't see as
well.

>  For example it could be hash(master_key || uuid). 

The hash could only contain master_key if the secret is already
released, so you can't condition the key release on this hash.

James