KVM Archive on lore.kernel.org
 help / color / Atom feed
From: Paolo Bonzini <pbonzini@redhat.com>
To: Ashish Kalra <Ashish.Kalra@amd.com>
Cc: tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com,
	joro@8bytes.org, bp@suse.de, thomas.lendacky@amd.com,
	x86@kernel.org, kvm@vger.kernel.org,
	linux-kernel@vger.kernel.org, srutherford@google.com,
	seanjc@google.com, venu.busireddy@oracle.com,
	brijesh.singh@amd.com
Subject: Re: [PATCH v13 00/12] Add AMD SEV guest live migration support
Date: Tue, 20 Apr 2021 13:11:31 +0200
Message-ID: <65ebdd0c-3224-742b-d0dd-5003309d1d62@redhat.com> (raw)
In-Reply-To: <cover.1618498113.git.ashish.kalra@amd.com>

On 15/04/21 17:52, Ashish Kalra wrote:
> From: Ashish Kalra <ashish.kalra@amd.com>
> 
> The series add support for AMD SEV guest live migration commands. To protect the
> confidentiality of an SEV protected guest memory while in transit we need to
> use the SEV commands defined in SEV API spec [1].
> 
> SEV guest VMs have the concept of private and shared memory. Private memory
> is encrypted with the guest-specific key, while shared memory may be encrypted
> with hypervisor key. The commands provided by the SEV FW are meant to be used
> for the private memory only. The patch series introduces a new hypercall.
> The guest OS can use this hypercall to notify the page encryption status.
> If the page is encrypted with guest specific-key then we use SEV command during
> the migration. If page is not encrypted then fallback to default.
> 
> The patch uses the KVM_EXIT_HYPERCALL exitcode and hypercall to
> userspace exit functionality as a common interface from the guest back to the
> VMM and passing on the guest shared/unencrypted page information to the
> userspace VMM/Qemu. Qemu can consult this information during migration to know
> whether the page is encrypted.
> 
> This section descibes how the SEV live migration feature is negotiated
> between the host and guest, the host indicates this feature support via
> KVM_FEATURE_CPUID. The guest firmware (OVMF) detects this feature and
> sets a UEFI enviroment variable indicating OVMF support for live
> migration, the guest kernel also detects the host support for this
> feature via cpuid and in case of an EFI boot verifies if OVMF also
> supports this feature by getting the UEFI enviroment variable and if it
> set then enables live migration feature on host by writing to a custom
> MSR, if not booted under EFI, then it simply enables the feature by
> again writing to the custom MSR. The MSR is also handled by the
> userspace VMM/Qemu.
> 
> A branch containing these patches is available here:
> https://github.com/AMDESE/linux/tree/sev-migration-v13
> 
> [1] https://developer.amd.com/wp-content/resources/55766.PDF

I have queued patches 1-6.

For patches 8 and 10 I will post my own version based on my review and 
feedback.

For guest patches, please repost separately so that x86 maintainers will 
notice them and ack them.

Paolo

> Changes since v12:
> - Reset page encryption status during early boot instead of just
>    before the kexec to avoid SMP races during kvm_pv_guest_cpu_reboot().
> - Remove incorrect log message in case of non-EFI boot and implicit
>    enabling of SEV live migration feature.
> 
> Changes since v11:
> - Clean up and remove kvm_x86_ops callback for page_enc_status_hc and
>    instead add a new per-VM flag to support/enable the page encryption
>    status hypercall.
> - Remove KVM_EXIT_DMA_SHARE/KVM_EXIT_DMA_UNSHARE exitcodes and instead
>    use the KVM_EXIT_HYPERCALL exitcode for page encryption status
>    hypercall to userspace functionality.
> 
> Changes since v10:
> - Adds new KVM_EXIT_DMA_SHARE/KVM_EXIT_DMA_UNSHARE hypercall to
>    userspace exit functionality as a common interface from the guest back to the
>    KVM and passing on the guest shared/unencrypted region information to the
>    userspace VMM/Qemu. KVM/host kernel does not maintain the guest shared
>    memory regions information anymore.
> - Remove implicit enabling of SEV live migration feature for an SEV
>    guest, now this is explicitly in control of the userspace VMM/Qemu.
> - Custom MSR handling is also now moved into userspace VMM/Qemu.
> - As KVM does not maintain the guest shared memory region information
>    anymore, sev_dbg_crypt() cannot bypass unencrypted guest memory
>    regions without support from userspace VMM/Qemu.
> 
> Changes since v9:
> - Transitioning from page encryption bitmap to the shared pages list
>    to keep track of guest's shared/unencrypted memory regions.
> - Move back to marking the complete _bss_decrypted section as
>    decrypted in the shared pages list.
> - Invoke a new function check_kvm_sev_migration() via kvm_init_platform()
>    for guest to query for host-side support for SEV live migration
>    and to enable the SEV live migration feature, to avoid
>    #ifdefs in code
> - Rename MSR_KVM_SEV_LIVE_MIG_EN to MSR_KVM_SEV_LIVE_MIGRATION.
> - Invoke a new function handle_unencrypted_region() from
>    sev_dbg_crypt() to bypass unencrypted guest memory regions.
> 
> Changes since v8:
> - Rebasing to kvm next branch.
> - Fixed and added comments as per review feedback on v8 patches.
> - Removed implicitly enabling live migration for incoming VMs in
>    in KVM_SET_PAGE_ENC_BITMAP, it is now done via KVM_SET_MSR ioctl.
> - Adds support for bypassing unencrypted guest memory regions for
>    DBG_DECRYPT API calls, guest memory region encryption status in
>    sev_dbg_decrypt() is referenced using the page encryption bitmap.
> 
> Changes since v7:
> - Removed the hypervisor specific hypercall/paravirt callback for
>    SEV live migration and moved back to calling kvm_sev_hypercall3
>    directly.
> - Fix build errors as
>    Reported-by: kbuild test robot <lkp@intel.com>, specifically fixed
>    build error when CONFIG_HYPERVISOR_GUEST=y and
>    CONFIG_AMD_MEM_ENCRYPT=n.
> - Implicitly enabled live migration for incoming VM(s) to handle
>    A->B->C->... VM migrations.
> - Fixed Documentation as per comments on v6 patches.
> - Fixed error return path in sev_send_update_data() as per comments
>    on v6 patches.
> 
> Changes since v6:
> - Rebasing to mainline and refactoring to the new split SVM
>    infrastructre.
> - Move to static allocation of the unified Page Encryption bitmap
>    instead of the dynamic resizing of the bitmap, the static allocation
>    is done implicitly by extending kvm_arch_commit_memory_region() callack
>    to add svm specific x86_ops which can read the userspace provided memory
>    region/memslots and calculate the amount of guest RAM managed by the KVM
>    and grow the bitmap.
> - Fixed KVM_SET_PAGE_ENC_BITMAP ioctl to set the whole bitmap instead
>    of simply clearing specific bits.
> - Removed KVM_PAGE_ENC_BITMAP_RESET ioctl, which is now performed using
>    KVM_SET_PAGE_ENC_BITMAP.
> - Extended guest support for enabling Live Migration feature by adding a
>    check for UEFI environment variable indicating OVMF support for Live
>    Migration feature and additionally checking for KVM capability for the
>    same feature. If not booted under EFI, then we simply check for KVM
>    capability.
> - Add hypervisor specific hypercall for SEV live migration by adding
>    a new paravirt callback as part of x86_hyper_runtime.
>    (x86 hypervisor specific runtime callbacks)
> - Moving MSR handling for MSR_KVM_SEV_LIVE_MIG_EN into svm/sev code
>    and adding check for SEV live migration enabled by guest in the
>    KVM_GET_PAGE_ENC_BITMAP ioctl.
> - Instead of the complete __bss_decrypted section, only specific variables
>    such as hv_clock_boot and wall_clock are marked as decrypted in the
>    page encryption bitmap
> 
> Changes since v5:
> - Fix build errors as
>    Reported-by: kbuild test robot <lkp@intel.com>
> 
> Changes since v4:
> - Host support has been added to extend KVM capabilities/feature bits to
>    include a new KVM_FEATURE_SEV_LIVE_MIGRATION, which the guest can
>    query for host-side support for SEV live migration and a new custom MSR
>    MSR_KVM_SEV_LIVE_MIG_EN is added for guest to enable the SEV live
>    migration feature.
> - Ensure that _bss_decrypted section is marked as decrypted in the
>    page encryption bitmap.
> - Fixing KVM_GET_PAGE_ENC_BITMAP ioctl to return the correct bitmap
>    as per the number of pages being requested by the user. Ensure that
>    we only copy bmap->num_pages bytes in the userspace buffer, if
>    bmap->num_pages is not byte aligned we read the trailing bits
>    from the userspace and copy those bits as is. This fixes guest
>    page(s) corruption issues observed after migration completion.
> - Add kexec support for SEV Live Migration to reset the host's
>    page encryption bitmap related to kernel specific page encryption
>    status settings before we load a new kernel by kexec. We cannot
>    reset the complete page encryption bitmap here as we need to
>    retain the UEFI/OVMF firmware specific settings.
> 
> Changes since v3:
> - Rebasing to mainline and testing.
> - Adding a new KVM_PAGE_ENC_BITMAP_RESET ioctl, which resets the
>    page encryption bitmap on a guest reboot event.
> - Adding a more reliable sanity check for GPA range being passed to
>    the hypercall to ensure that guest MMIO ranges are also marked
>    in the page encryption bitmap.
> 
> Changes since v2:
>   - reset the page encryption bitmap on vcpu reboot
> 
> Changes since v1:
>   - Add support to share the page encryption between the source and target
>     machine.
>   - Fix review feedbacks from Tom Lendacky.
>   - Add check to limit the session blob length.
>   - Update KVM_GET_PAGE_ENC_BITMAP icotl to use the base_gfn instead of
>     the memory slot when querying the bitmap.
> 
> Ashish Kalra (4):
>    KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall
>    KVM: x86: Introduce new KVM_FEATURE_SEV_LIVE_MIGRATION feature &
>      Custom MSR.
>    EFI: Introduce the new AMD Memory Encryption GUID.
>    x86/kvm: Add guest support for detecting and enabling SEV Live
>      Migration feature.
> 
> Brijesh Singh (8):
>    KVM: SVM: Add KVM_SEV SEND_START command
>    KVM: SVM: Add KVM_SEND_UPDATE_DATA command
>    KVM: SVM: Add KVM_SEV_SEND_FINISH command
>    KVM: SVM: Add support for KVM_SEV_RECEIVE_START command
>    KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command
>    KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command
>    KVM: x86: Add AMD SEV specific Hypercall3
>    mm: x86: Invoke hypercall when page encryption status is changed
> 
>   .../virt/kvm/amd-memory-encryption.rst        | 120 +++++
>   Documentation/virt/kvm/cpuid.rst              |   5 +
>   Documentation/virt/kvm/hypercalls.rst         |  15 +
>   Documentation/virt/kvm/msr.rst                |  12 +
>   arch/x86/include/asm/kvm_host.h               |   2 +
>   arch/x86/include/asm/kvm_para.h               |  12 +
>   arch/x86/include/asm/mem_encrypt.h            |   8 +
>   arch/x86/include/asm/paravirt.h               |  10 +
>   arch/x86/include/asm/paravirt_types.h         |   2 +
>   arch/x86/include/uapi/asm/kvm_para.h          |   4 +
>   arch/x86/kernel/kvm.c                         |  55 +++
>   arch/x86/kernel/paravirt.c                    |   1 +
>   arch/x86/kvm/cpuid.c                          |   3 +-
>   arch/x86/kvm/svm/sev.c                        | 454 ++++++++++++++++++
>   arch/x86/kvm/x86.c                            |  29 ++
>   arch/x86/mm/mem_encrypt.c                     | 121 ++++-
>   arch/x86/mm/pat/set_memory.c                  |   7 +
>   include/linux/efi.h                           |   1 +
>   include/linux/psp-sev.h                       |   8 +-
>   include/uapi/linux/kvm.h                      |  39 ++
>   include/uapi/linux/kvm_para.h                 |   1 +
>   21 files changed, 903 insertions(+), 6 deletions(-)
> 


  parent reply index

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-15 15:52 Ashish Kalra
2021-04-15 15:53 ` [PATCH v13 01/12] KVM: SVM: Add KVM_SEV SEND_START command Ashish Kalra
2021-04-20  8:50   ` Paolo Bonzini
2021-04-15 15:53 ` [PATCH v13 02/12] KVM: SVM: Add KVM_SEND_UPDATE_DATA command Ashish Kalra
2021-04-15 15:54 ` [PATCH v13 03/12] KVM: SVM: Add KVM_SEV_SEND_FINISH command Ashish Kalra
2021-04-15 15:54 ` [PATCH v13 04/12] KVM: SVM: Add support for KVM_SEV_RECEIVE_START command Ashish Kalra
2021-04-20  8:38   ` Paolo Bonzini
2021-04-20  9:18     ` Paolo Bonzini
2021-04-15 15:55 ` [PATCH v13 05/12] KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command Ashish Kalra
2021-04-20  8:40   ` Paolo Bonzini
2021-04-20  8:43     ` Paolo Bonzini
2021-04-15 15:55 ` [PATCH v13 06/12] KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command Ashish Kalra
2021-04-15 15:56 ` [PATCH v13 07/12] KVM: x86: Add AMD SEV specific Hypercall3 Ashish Kalra
2021-04-15 15:57 ` [PATCH v13 08/12] KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall Ashish Kalra
2021-04-20 11:10   ` Paolo Bonzini
2021-04-20 17:24     ` Sean Christopherson
2021-04-15 15:57 ` [PATCH v13 09/12] mm: x86: Invoke hypercall when page encryption status is changed Ashish Kalra
2021-04-20  9:39   ` Paolo Bonzini
2021-04-21 10:05   ` Borislav Petkov
2021-04-21 12:00     ` Paolo Bonzini
2021-04-21 14:09       ` Borislav Petkov
2021-04-21 12:12     ` Ashish Kalra
2021-04-21 13:50       ` Brijesh Singh
2021-04-21 13:52       ` Borislav Petkov
2021-04-15 15:58 ` [PATCH v13 10/12] KVM: x86: Introduce new KVM_FEATURE_SEV_LIVE_MIGRATION feature & Custom MSR Ashish Kalra
2021-04-19 23:06   ` Sean Christopherson
2021-04-20 10:49     ` Paolo Bonzini
2021-04-20  9:47   ` Paolo Bonzini
2021-04-15 15:58 ` [PATCH v13 11/12] EFI: Introduce the new AMD Memory Encryption GUID Ashish Kalra
2021-04-15 16:01 ` [PATCH v13 12/12] x86/kvm: Add guest support for detecting and enabling SEV Live Migration feature Ashish Kalra
2021-04-20 10:52   ` Paolo Bonzini
2021-04-21 14:44   ` Borislav Petkov
2021-04-21 15:22     ` Ashish Kalra
2021-04-21 15:32       ` Borislav Petkov
2021-04-21 15:38     ` Paolo Bonzini
2021-04-21 18:48       ` Ashish Kalra
2021-04-21 19:19         ` Ashish Kalra
2021-04-16 21:43 ` [PATCH v13 00/12] Add AMD SEV guest live migration support Steve Rutherford
2021-04-19 14:40   ` Ashish Kalra
2021-04-20 11:11 ` Paolo Bonzini [this message]
2021-04-20 18:51   ` Borislav Petkov
2021-04-20 19:08     ` Paolo Bonzini
2021-04-20 20:28       ` Borislav Petkov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=65ebdd0c-3224-742b-d0dd-5003309d1d62@redhat.com \
    --to=pbonzini@redhat.com \
    --cc=Ashish.Kalra@amd.com \
    --cc=bp@suse.de \
    --cc=brijesh.singh@amd.com \
    --cc=hpa@zytor.com \
    --cc=joro@8bytes.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=seanjc@google.com \
    --cc=srutherford@google.com \
    --cc=tglx@linutronix.de \
    --cc=thomas.lendacky@amd.com \
    --cc=venu.busireddy@oracle.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

KVM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvm/0 kvm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvm kvm/ https://lore.kernel.org/kvm \
		kvm@vger.kernel.org
	public-inbox-index kvm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.kvm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git