linux-coco.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* Runtime Memory Validation in Intel-TDX and AMD-SNP
@ 2021-07-19 12:58 Joerg Roedel
  2021-07-19 13:07 ` Matthew Wilcox
                   ` (6 more replies)
  0 siblings, 7 replies; 50+ messages in thread
From: Joerg Roedel @ 2021-07-19 12:58 UTC (permalink / raw)
  To: David Rientjes, Borislav Petkov, Andy Lutomirski,
	Sean Christopherson, Andrew Morton, Vlastimil Babka,
	Kirill A. Shutemov, Andi Kleen, Brijesh Singh, Tom Lendacky,
	Jon Grimm, Thomas Gleixner, Peter Zijlstra, Paolo Bonzini,
	Ingo Molnar, Kaplan, David, Varad Gautam, Dario Faggioli,
	Joerg Roedel
  Cc: x86, linux-mm, linux-coco

Hi,

I'd like to get some movement again into the discussion around how to
implement runtime memory validation for confidential guests and wrote up
some thoughts on it.
Below are the results in form of a proposal I put together. Please let
me know your thoughts on it and whether it fits everyones requirements.

Thanks,

	Joerg

Proposal for Runtime Memory Validation in Secure Guests on x86
==============================================================

This proposal describes a method and protocol for runtime validation of
memory in virtualization guests running with Intel Trusted Domain
Extensions (Intel-TDX) or AMD Secure Nested Paging (AMD-SNP).

AMD-SNP and Intel-TDX use different terms to discuss memory page states.
In AMD-SNP memory has to be 'validated' while in Intel-TDX is will be
'accepted'. This document uses the term 'validated' for both.

Problem Statement
-----------------

Virtualization guests which run with AMD-SNP or Intel-TDX need to
validate their memory before using it. The validation assigns a hardware
state to each page which allows the guest to detect when the hypervisor
tries to maliciously access or remap a guest-private page. The guest can
only access validated pages.

There are three ways the guest memory can be validated:

	I.   The firmware validates all of guest memory at boot time. This
	     is the simplest method which requires the least changes to
	     the Linux kernel. But this method is also very slow and
	     causes unwanted delays in the boot process, as verification
	     can take several seconds (depending on guest memory size).

	II.  The firmware only validates its own memory and memory
	     validation happens as the memory is used. This significantly
	     improves the boot time, but needs more intrusive changes to
	     the Linux kernel and its boot process.


	III. Approach I. and II. can be combined. The firmware only
	     validates the first X MB/GB of guest memory and the rest is
	     validated on-demand.

For method II. and III. the guest needs to track which pages have
already been validated to detect hypervisor attacks. This information
needs to be carried through the whole boot process.

This poses challenges on the Linux boot process, as there is currently
no way to forward information about validated memory up the boot chain.
This proposal tries to describe a way to solve these challenges.

Memory Validation through the Boot Process and in the Running System
--------------------------------------------------------------------

The memory is validated throughout the boot process as described below.
These steps assume a firmware is present, but this proposal does not
strictly require a firmware. The tasks done be the firmware can also be
done by the hypervisor before starting the guest. The steps are:

	1. The firmware validates all memory which will not be owned by
	   the boot loader or the OS.

	2. The firmware also validates the first X MB of memory, just
	   enough to run a boot loader and to load the compressed Linux
	   kernel image. X is not expected to be very large, 64 or 128
	   MB should be enough. This pre-validation should not cause
	   significant delays in the boot process.

	3. The validated memory is marked E820-Usable in struct
	   boot_params for the Linux decompressor. The rest of the
	   memory is also passed to Linux via new special E820 entries
	   which mark the memory as Usable-but-Invalid.

	4. When the Linux decompressor takes over control, it evaluates
	   the E820 table and calculates to total amount of memory
	   available to Linux (valid and invalid memory).

	   The decompressor allocates a physically contiguous data
	   structure at a random memory location which is big enough to
	   hold the the validation states of all 4kb pages available to
	   the guest. This data structure will be called the Validation
	   Bitmap through the rest of this document. The Validation
	   Bitmap is indexed by page frame numbers. 

	   It still needs to be determined how many bits are required
	   per page. This depends on the necessity to track validation
	   page-sizes. Two bits per page are enough to track the 3
	   page-sizes currently available on the x86 architecture.

	   The decompressor initializes the Validation Bitmap by first
	   validating its backing memory and then updating it with the
	   information from the E820 table. It will also update the
	   table if it changes the state of pages from invalid to valid
	   (and vice versa, e.g. for mapping a GHCB page).

	5. The 'struct boot_params' is extended to carry the location
	   and size of the Validation Bitmap to the extracted kernel
	   image.
	   In fact, since the decompressor already receives a 'struct
	   boot_params', it will check if it carries a Validation
	   Bitmap. If it does, the decompressor uses the existing one
	   instead of allocating a new one.

	6. When the extracted kernel image takes over control, it will
	   make sure the Validation Bitmap is up to date when memory
	   needs to be validated.

	7. When set up, the memblock and page allocators have to check
	   whether the memory they return is already validated, and
	   validate it if not.

	   This should happen after the memory is allocated and all
	   allocator-locks are dropped, but before the memory is
	   returned to the caller. This way the access to the
	   validation bitmap can be implemented without locking and only
	   using atomic instructions.

	   Under no circumstances the Linux kernel is allowed to
	   validate a page more than once. Doing this might create
	   attack vectors for the Hypervisor towards the guest.

	8. When memory is returned to the memblock or page allocators,
	   it is _not_ invalidated. In fact, all memory which is freed
	   need to be valid. If it was marked invalid in the meantime
	   (e.g. if it the memory was used for DMA buffers), the code
	   owning the memory needs to validate it again before freeing
	   it.

	   The benefit of doing memory validation at allocation time is
	   that it keeps the exception handler for invalid memory
	   simple, because no exceptions of this kind are expected under
	   normal operation.

The Validation Bitmap
---------------------

This document proposes the use of a Validation Bitmap to store the
validation state of guest pages. This section discusses the benefits of
this approach.

The Linux kernel already has an array to store various state for each
memory page in the system: The struct page array. While this would be a
natural place to also store page validation information, the Validation
Bitmap is chosen because having the information separated has some clear
benefits:

	- The Validation Bitmap is allocated in the Linux decompressor
	  and already available long before the struct page array is
	  initialized.

	- Since it is a simple in-memory data structure which is
	  physically contiguous, it can be passed along through the
	  various stages of the boot process.

	- It can even be passed to a new kernel booted via kexec/kdump,
	  making it trivial to enable these features for AMD-SNP and
	  Intel-TDX.

	- When memory validation happens in the memblock and page
	  allocators, there is no need for locking when making changes
	  to the Validation Bitmap, because:
	  
	    - Nobody will try to concurrently access the same bits, as
	      the code-path doing the validation is the only owner of
	      the memory.

	    - Updates can happen via atomic cmpxchg instructions
	      when multiple bits are used per page. If only one bit is
	      needed, atomic bit manipulation instructions will suffice.

	- NUMA-locality is not considered to be a problem for the
	  Validation Bitmap. Since memory is not invalidated upon free,
	  the data structure will become read-mostly over time.

Final Notes
-----------

This proposal does not introduce requirements about the firmware that
has to be used to run Intel-TDX or AMD-SNP guests. It works with UEFI
and non-UEFI firmwares, or with no firmware at all. This is important
for use-cases like Confidential Containers running in VMs, which often
use a very small firmware (or no firmware at all) for reducing boot
times.

^ permalink raw reply	[flat|nested] 50+ messages in thread

end of thread, other threads:[~2021-08-02 18:47 UTC | newest]

Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-19 12:58 Runtime Memory Validation in Intel-TDX and AMD-SNP Joerg Roedel
2021-07-19 13:07 ` Matthew Wilcox
2021-07-19 15:02   ` Joerg Roedel
2021-07-19 20:39 ` Andi Kleen
2021-07-20  8:55   ` Joerg Roedel
2021-07-20  9:34     ` Dr. David Alan Gilbert
2021-07-20 11:50       ` Joerg Roedel
2021-07-20  0:26 ` Andy Lutomirski
     [not found]   ` <CAAYXXYwFzrf8uY-PFkMRSG28+HztfGdJft8kB3Y3keWCx9K8TQ@mail.gmail.com>
2021-07-20  2:00     ` Erdem Aktas
2021-07-20  5:17     ` Andi Kleen
2021-07-20  9:11       ` Joerg Roedel
2021-07-20 17:32         ` Andi Kleen
2021-07-20 23:09       ` Erdem Aktas
2021-07-21  0:38         ` Andi Kleen
2021-07-22 17:31       ` Marc Orr
2021-07-26 18:55         ` Joerg Roedel
     [not found]     ` <eacb9c1f-2c61-4a7f-b5a3-7bf579e6cbf6@www.fastmail.com>
2021-07-20 19:54       ` Erdem Aktas
2021-07-20 22:01         ` Andi Kleen
2021-07-20 23:55           ` Erdem Aktas
2021-07-21  0:35             ` Andi Kleen
2021-07-21  8:51           ` Joerg Roedel
2021-07-20  8:44   ` Joerg Roedel
2021-07-20 14:14   ` Dave Hansen
2021-07-20 17:30 ` Kirill A. Shutemov
2021-07-21  9:20   ` Mike Rapoport
2021-07-21 10:02     ` Kirill A. Shutemov
2021-07-21 10:22       ` Mike Rapoport
2021-07-21 10:53       ` Joerg Roedel
2021-07-21  9:25   ` Joerg Roedel
2021-07-21 10:25     ` Kirill A. Shutemov
2021-07-21 10:48       ` Joerg Roedel
2021-07-22 15:46   ` David Hildenbrand
2021-07-26 19:02     ` Joerg Roedel
2021-07-27  9:34       ` David Hildenbrand
2021-08-02 10:19         ` Joerg Roedel
2021-08-02 18:47           ` David Hildenbrand
2021-07-22 15:57 ` David Hildenbrand
2021-07-22 19:51 ` Kirill A. Shutemov
2021-07-23 15:23   ` Mike Rapoport
2021-07-23 16:29     ` Kirill A. Shutemov
2021-07-25  9:16       ` Mike Rapoport
2021-07-25 18:28         ` Kirill A. Shutemov
2021-07-26 10:00           ` Mike Rapoport
2021-07-26 11:53             ` Kirill A. Shutemov
2021-07-26 19:13   ` Joerg Roedel
2021-07-26 23:02   ` Erdem Aktas
2021-07-26 23:54     ` Kirill A. Shutemov
2021-07-27  1:35       ` Erdem Aktas
2021-07-23 11:04 ` Varad Gautam
2021-07-23 14:34   ` Kaplan, David

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).