Re: [PATCH v8 00/18] Add AMD SEV guest live migration support

[Steve Rutherford <srutherford@google.com>]

Are these likely to get merged into 5.9?


On Wed, Jun 3, 2020 at 3:14 PM Ashish Kalra <ashish.kalra@amd.com> wrote:
>
> Hello Steve,
>
> On Mon, Jun 01, 2020 at 01:02:23PM -0700, Steve Rutherford wrote:
> > On Mon, May 18, 2020 at 12:07 PM Ashish Kalra <ashish.kalra@amd.com> wrote:
> > >
> > > Hello All,
> > >
> > > Any other feedback, review or comments on this patch-set ?
> > >
> > > Thanks,
> > > Ashish
> > >
> > > On Tue, May 05, 2020 at 09:13:49PM +0000, Ashish Kalra wrote:
> > > > From: Ashish Kalra <ashish.kalra@amd.com>
> > > >
> > > > The series add support for AMD SEV guest live migration commands. To protect the
> > > > confidentiality of an SEV protected guest memory while in transit we need to
> > > > use the SEV commands defined in SEV API spec [1].
> > > >
> > > > SEV guest VMs have the concept of private and shared memory. Private memory
> > > > is encrypted with the guest-specific key, while shared memory may be encrypted
> > > > with hypervisor key. The commands provided by the SEV FW are meant to be used
> > > > for the private memory only. The patch series introduces a new hypercall.
> > > > The guest OS can use this hypercall to notify the page encryption status.
> > > > If the page is encrypted with guest specific-key then we use SEV command during
> > > > the migration. If page is not encrypted then fallback to default.
> > > >
> > > > The patch adds new ioctls KVM_{SET,GET}_PAGE_ENC_BITMAP. The ioctl can be used
> > > > by the qemu to get the page encrypted bitmap. Qemu can consult this bitmap
> > > > during the migration to know whether the page is encrypted.
> > > >
> > > > This section descibes how the SEV live migration feature is negotiated
> > > > between the host and guest, the host indicates this feature support via
> > > > KVM_FEATURE_CPUID. The guest firmware (OVMF) detects this feature and
> > > > sets a UEFI enviroment variable indicating OVMF support for live
> > > > migration, the guest kernel also detects the host support for this
> > > > feature via cpuid and in case of an EFI boot verifies if OVMF also
> > > > supports this feature by getting the UEFI enviroment variable and if it
> > > > set then enables live migration feature on host by writing to a custom
> > > > MSR, if not booted under EFI, then it simply enables the feature by
> > > > again writing to the custom MSR. The host returns error as part of
> > > > SET_PAGE_ENC_BITMAP ioctl if guest has not enabled live migration.
> > > >
> > > > A branch containing these patches is available here:
> > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAMDESE%2Flinux%2Ftree%2Fsev-migration-v8&amp;data=02%7C01%7Cashish.kalra%40amd.com%7Cb7da54c6f7784a548ed208d80666c99b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266386411473155&amp;sdata=igztZXTZl1i18e5T4DTlNJw07h6z3aBNCAD6%2BE7r9Ik%3D&amp;reserved=0
> > > >
> > > > [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F55766.PDF&amp;data=02%7C01%7Cashish.kalra%40amd.com%7Cb7da54c6f7784a548ed208d80666c99b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266386411473155&amp;sdata=GBgV6HAd2AXZzjK3hp%2F396tDaHlYtN%2FL3Zfny3GaSoU%3D&amp;reserved=0
> > > >
> > > > Changes since v7:
> > > > - Removed the hypervisor specific hypercall/paravirt callback for
> > > >   SEV live migration and moved back to calling kvm_sev_hypercall3
> > > >   directly.
> > > > - Fix build errors as
> > > >   Reported-by: kbuild test robot <lkp@intel.com>, specifically fixed
> > > >   build error when CONFIG_HYPERVISOR_GUEST=y and
> > > >   CONFIG_AMD_MEM_ENCRYPT=n.
> > > > - Implicitly enabled live migration for incoming VM(s) to handle
> > > >   A->B->C->... VM migrations.
> > > > - Fixed Documentation as per comments on v6 patches.
> > > > - Fixed error return path in sev_send_update_data() as per comments
> > > >   on v6 patches.
> > > >
> > > > Changes since v6:
> > > > - Rebasing to mainline and refactoring to the new split SVM
> > > >   infrastructre.
> > > > - Move to static allocation of the unified Page Encryption bitmap
> > > >   instead of the dynamic resizing of the bitmap, the static allocation
> > > >   is done implicitly by extending kvm_arch_commit_memory_region() callack
> > > >   to add svm specific x86_ops which can read the userspace provided memory
> > > >   region/memslots and calculate the amount of guest RAM managed by the KVM
> > > >   and grow the bitmap.
> > > > - Fixed KVM_SET_PAGE_ENC_BITMAP ioctl to set the whole bitmap instead
> > > >   of simply clearing specific bits.
> > > > - Removed KVM_PAGE_ENC_BITMAP_RESET ioctl, which is now performed using
> > > >   KVM_SET_PAGE_ENC_BITMAP.
> > > > - Extended guest support for enabling Live Migration feature by adding a
> > > >   check for UEFI environment variable indicating OVMF support for Live
> > > >   Migration feature and additionally checking for KVM capability for the
> > > >   same feature. If not booted under EFI, then we simply check for KVM
> > > >   capability.
> > > > - Add hypervisor specific hypercall for SEV live migration by adding
> > > >   a new paravirt callback as part of x86_hyper_runtime.
> > > >   (x86 hypervisor specific runtime callbacks)
> > > > - Moving MSR handling for MSR_KVM_SEV_LIVE_MIG_EN into svm/sev code
> > > >   and adding check for SEV live migration enabled by guest in the
> > > >   KVM_GET_PAGE_ENC_BITMAP ioctl.
> > > > - Instead of the complete __bss_decrypted section, only specific variables
> > > >   such as hv_clock_boot and wall_clock are marked as decrypted in the
> > > >   page encryption bitmap
> > > >
> > > > Changes since v5:
> > > > - Fix build errors as
> > > >   Reported-by: kbuild test robot <lkp@intel.com>
> > > >
> > > > Changes since v4:
> > > > - Host support has been added to extend KVM capabilities/feature bits to
> > > >   include a new KVM_FEATURE_SEV_LIVE_MIGRATION, which the guest can
> > > >   query for host-side support for SEV live migration and a new custom MSR
> > > >   MSR_KVM_SEV_LIVE_MIG_EN is added for guest to enable the SEV live
> > > >   migration feature.
> > > > - Ensure that _bss_decrypted section is marked as decrypted in the
> > > >   page encryption bitmap.
> > > > - Fixing KVM_GET_PAGE_ENC_BITMAP ioctl to return the correct bitmap
> > > >   as per the number of pages being requested by the user. Ensure that
> > > >   we only copy bmap->num_pages bytes in the userspace buffer, if
> > > >   bmap->num_pages is not byte aligned we read the trailing bits
> > > >   from the userspace and copy those bits as is. This fixes guest
> > > >   page(s) corruption issues observed after migration completion.
> > > > - Add kexec support for SEV Live Migration to reset the host's
> > > >   page encryption bitmap related to kernel specific page encryption
> > > >   status settings before we load a new kernel by kexec. We cannot
> > > >   reset the complete page encryption bitmap here as we need to
> > > >   retain the UEFI/OVMF firmware specific settings.
> > > >
> > > > Changes since v3:
> > > > - Rebasing to mainline and testing.
> > > > - Adding a new KVM_PAGE_ENC_BITMAP_RESET ioctl, which resets the
> > > >   page encryption bitmap on a guest reboot event.
> > > > - Adding a more reliable sanity check for GPA range being passed to
> > > >   the hypercall to ensure that guest MMIO ranges are also marked
> > > >   in the page encryption bitmap.
> > > >
> > > > Changes since v2:
> > > >  - reset the page encryption bitmap on vcpu reboot
> > > >
> > > > Changes since v1:
> > > >  - Add support to share the page encryption between the source and target
> > > >    machine.
> > > >  - Fix review feedbacks from Tom Lendacky.
> > > >  - Add check to limit the session blob length.
> > > >  - Update KVM_GET_PAGE_ENC_BITMAP icotl to use the base_gfn instead of
> > > >    the memory slot when querying the bitmap.
> > > >
> > > > Ashish Kalra (7):
> > > >   KVM: SVM: Add support for static allocation of unified Page Encryption
> > > >     Bitmap.
> > > >   KVM: x86: Introduce new KVM_FEATURE_SEV_LIVE_MIGRATION feature &
> > > >     Custom MSR.
> > > >   EFI: Introduce the new AMD Memory Encryption GUID.
> > > >   KVM: x86: Add guest support for detecting and enabling SEV Live
> > > >     Migration feature.
> > > >   KVM: x86: Mark _bss_decrypted section variables as decrypted in page
> > > >     encryption bitmap.
> > > >   KVM: x86: Add kexec support for SEV Live Migration.
> > > >   KVM: SVM: Enable SEV live migration feature implicitly on Incoming
> > > >     VM(s).
> > > >
> > > > Brijesh Singh (11):
> > > >   KVM: SVM: Add KVM_SEV SEND_START command
> > > >   KVM: SVM: Add KVM_SEND_UPDATE_DATA command
> > > >   KVM: SVM: Add KVM_SEV_SEND_FINISH command
> > > >   KVM: SVM: Add support for KVM_SEV_RECEIVE_START command
> > > >   KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command
> > > >   KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command
> > > >   KVM: x86: Add AMD SEV specific Hypercall3
> > > >   KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall
> > > >   KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl
> > > >   mm: x86: Invoke hypercall when page encryption status is changed
> > > >   KVM: x86: Introduce KVM_SET_PAGE_ENC_BITMAP ioctl
> > > >
> > > >  .../virt/kvm/amd-memory-encryption.rst        | 120 +++
> > > >  Documentation/virt/kvm/api.rst                |  71 ++
> > > >  Documentation/virt/kvm/cpuid.rst              |   5 +
> > > >  Documentation/virt/kvm/hypercalls.rst         |  15 +
> > > >  Documentation/virt/kvm/msr.rst                |  10 +
> > > >  arch/x86/include/asm/kvm_host.h               |   7 +
> > > >  arch/x86/include/asm/kvm_para.h               |  12 +
> > > >  arch/x86/include/asm/mem_encrypt.h            |  11 +
> > > >  arch/x86/include/asm/paravirt.h               |  10 +
> > > >  arch/x86/include/asm/paravirt_types.h         |   2 +
> > > >  arch/x86/include/uapi/asm/kvm_para.h          |   5 +
> > > >  arch/x86/kernel/kvm.c                         |  90 +++
> > > >  arch/x86/kernel/kvmclock.c                    |  12 +
> > > >  arch/x86/kernel/paravirt.c                    |   1 +
> > > >  arch/x86/kvm/svm/sev.c                        | 732 +++++++++++++++++-
> > > >  arch/x86/kvm/svm/svm.c                        |  21 +
> > > >  arch/x86/kvm/svm/svm.h                        |   9 +
> > > >  arch/x86/kvm/vmx/vmx.c                        |   1 +
> > > >  arch/x86/kvm/x86.c                            |  35 +
> > > >  arch/x86/mm/mem_encrypt.c                     |  68 +-
> > > >  arch/x86/mm/pat/set_memory.c                  |   7 +
> > > >  include/linux/efi.h                           |   1 +
> > > >  include/linux/psp-sev.h                       |   8 +-
> > > >  include/uapi/linux/kvm.h                      |  52 ++
> > > >  include/uapi/linux/kvm_para.h                 |   1 +
> > > >  25 files changed, 1297 insertions(+), 9 deletions(-)
> > > >
> > > > --
> > > > 2.17.1
> > > >
> >
> > Hey all,
> > These patches look pretty reasonable at this point. What's the next
> > step for getting them merged?
>
> I believe i have incorporated all your main comments and feedback, for
> example, no more dynamic resizing of the page encryption bitmap and
> static allocation of the same, fixing SET_PAGE_ENC_BITMAP ioctl to set
> the whole bitmap and also merging RESET_PAGE_ENC_BITMAP ioctl into it
> and other stuff like kexec support, etc.
>
> I know you have some additional comments and i am waiting for some more
> feedback and comments from others on the mailing list.
>
> But otherwise i believe these patches are fully ready to be merged at
> this point and i am looking forward to the same.
>
> Thanks,
> Ashish
>
> > Thanks,
> > Steve