Unmapping KVM Guest Memory from Host Kernel

["Gowans, James" <jgowans@amazon.com>]

Hello KVM, MM and memfd_secret folks,

Currently when using anonymous memory for KVM guest RAM, the memory all
remains mapped into the kernel direct map. We are looking at options to
get KVM guest memory out of the kernel’s direct map as a principled
approach to mitigating speculative execution issues in the host kernel.
Our goal is to more completely address the class of issues whose leak
origin is categorized as "Mapped memory" [1].

We currently have downstream-only solutions to this, but we want to move
to purely upstream code.

So far we have been looking at using memfd_secret, which seems to be
designed exactly for usecases where it is undesirable to have some
memory range accessible through the kernel’s direct map.

However, memfd_secret doesn’t work out the box for KVM guest memory; the
main reason seems to be that the GUP path is intentionally disabled for
memfd_secret, so if we use a memfd_secret backed VMA for a memslot then
KVM is not able to fault the memory in. If it’s been pre-faulted in by
userspace then it seems to work.

There are a few other issues around when KVM accesses the guest memory.
For example the KVM PV clock code goes directly to the PFN via the
pfncache, and that also breaks if the PFN is not in the direct map, so
we’d need to change that sort of thing, perhaps going via userspace
addresses.

If we remove the memfd_secret check from the GUP path, and disable KVM’s
pvclock from userspace via KVM_CPUID_FEATURES, we are able to boot a
simple Linux initrd using a Firecracker VMM modified to use
memfd_secret.

We are also aware of ongoing work on guest_memfd. The current
implementation unmaps guest memory from VMM address space, but leaves it
in the kernel’s direct map. We’re not looking at unmapping from VMM
userspace yet; we still need guest RAM there for PV drivers like virtio
to continue to work. So KVM’s gmem doesn’t seem like the right solution?

With this in mind, what’s the best way to solve getting guest RAM out of
the direct map? Is memfd_secret integration with KVM the way to go, or
should we build a solution on top of guest_memfd, for example via some
flag that causes it to leave memory in the host userspace’s page tables,
but removes it from the direct map? 

We are keen to help contribute to getting this working, we’re just
looking for guidance from maintainers on what the correct way to solve
this is.

Cheers,
James + colleagues Derek and Patrick