From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35356C64EC7 for ; Tue, 28 Feb 2023 20:47:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E6956B0074; Tue, 28 Feb 2023 15:47:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 397BB6B0075; Tue, 28 Feb 2023 15:47:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238AC6B0078; Tue, 28 Feb 2023 15:47:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 14AB16B0074 for ; Tue, 28 Feb 2023 15:47:38 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D8F29AB1D7 for ; Tue, 28 Feb 2023 20:47:37 +0000 (UTC) X-FDA: 80517886554.03.AAAE800 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf25.hostedemail.com (Postfix) with ESMTP id CE56EA000D for ; Tue, 28 Feb 2023 20:47:34 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EF6ipXnc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of zhi.wang.linux@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=zhi.wang.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677617255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3L1QLt7UDDPEkqiycna07teZMPhk4C+uHRwLWaiwTyc=; b=fSdS+snFSA1dU6mSfYzcGSxkQlxqI87CeeiwX11QgVHzQfjf209oIK/YvTqYBtAHLZATyc +r+gVw8emvw9Q/KYOGtBC+cIR/hTkKYx+F8C1VpRGP20n0uw492RRcXM2H/JyKjgtIxc+s XCVhUN03FRr08KZbV8X0qqUFN5Ne7jc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EF6ipXnc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of zhi.wang.linux@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=zhi.wang.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677617255; a=rsa-sha256; cv=none; b=Eq4IdSjTQxF48KxN0fSDu7NRE0N8T2JawhQbLrz7av+3dVCMdHfRrxYjpIe/o1iQn7RhYZ ZZMJ6ds8hf+LdoiKTTsVfRlvCGjs2DLi33yzLXRAqab7OFXxV7ENQKzrqwAX3/SlRPywJg ZKZHKPrd3zPI3w697LCSMhhaibfKG2s= Received: by mail-lf1-f43.google.com with SMTP id m7so14902500lfj.8 for ; Tue, 28 Feb 2023 12:47:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1677617253; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:date:from:from:to:cc:subject:date :message-id:reply-to; bh=3L1QLt7UDDPEkqiycna07teZMPhk4C+uHRwLWaiwTyc=; b=EF6ipXncGR87Hyi7yuIkIWlrhtTuMv9ZwhTzqku5AcU+7xbIZO6LNpXbphdVN/4YTC 3gHjUpjN3/gvCJa92kzGi7UJiihYkqSxEOEztuC4Y+aG2M6yb9V+VeovWBmIA1OsuUAb 3SK9TUvY500zI+99tIz5zFv9ukjRv5jeS7Q+4SxyqCUbE9dNb5JsB13QfKf3u29XDdMj O8/KtllWWAK6CVQanDx1FYpD3YBdhau9KiGWWEGFqMhalWLx3YIm5tJahqV6ilqqMelD VRorG8yZYlbEENSGAReaYvRaH26bYF5QJd3SkU7U14t1Vi8qX9W/UJSnKrIzGThs+lL/ 0FAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677617253; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3L1QLt7UDDPEkqiycna07teZMPhk4C+uHRwLWaiwTyc=; b=H1HhAytZU0li/6YmI/tJm2PWJn6FRlQx9L/eJLFgZAxmd6BUHZm9Xjl1MVkUHZdP4I dBUAB5Wi4dgN27KjuPH8ht1y8NELKSbgJCuNfG2ci+xeAoh2e4gy66OGP5z8js0z8Ydo PpCagc2cGG8dp4/ZHXlec6M5Sd8dxwP9o9rn78j3ww+KIv5ES91dFPh33J7NOM2qEybS anzPcVKsDSgorkoUsnoOYcZgn1KDIL9G40LaqVtw6TzOrYXOgUDd5WC/eFQtfmA/WmdU P9PCWKQimID+WlU2R51FuWer5xIN5lN58LmHG6ScJcadruiEP1ywY1tp0Vu7DkKeYxIE db5A== X-Gm-Message-State: AO0yUKXOpQvhwEoC34Z2tQYHbZpSv9SNWFlYe9bOftknEZpdnSaXrFsr t8bB+G9KnSISrch7m1HQUL0= X-Google-Smtp-Source: AK7set8v7E0yifOxFvbZFDxajwr0/Pny5hwCohRYpmE+xovY8y4YPAZGRDqcnWw/eas+nkSXmzpyLg== X-Received: by 2002:ac2:4c21:0:b0:4b5:a7c7:9dc4 with SMTP id u1-20020ac24c21000000b004b5a7c79dc4mr805829lfq.3.1677617252760; Tue, 28 Feb 2023 12:47:32 -0800 (PST) Received: from localhost (88-115-161-74.elisa-laajakaista.fi. [88.115.161.74]) by smtp.gmail.com with ESMTPSA id t4-20020ac25484000000b004b592043413sm1478655lfk.12.2023.02.28.12.47.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Feb 2023 12:47:32 -0800 (PST) From: Zhi Wang X-Google-Original-From: Zhi Wang Date: Tue, 28 Feb 2023 22:47:30 +0200 To: Alexander Graf Cc: Michael Roth , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Brijesh Singh Subject: Re: [PATCH RFC v8 47/56] KVM: SVM: Support SEV-SNP AP Creation NAE event Message-ID: <20230228224730.00007d21@intel.com> In-Reply-To: <09696af0-b72d-29e7-862b-22ae4a630299@amazon.com> References: <20230220183847.59159-1-michael.roth@amd.com> <20230220183847.59159-48-michael.roth@amd.com> <09696af0-b72d-29e7-862b-22ae4a630299@amazon.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: CE56EA000D X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: cknxmfqsp41pjwn8gsujtyjz9m8hunui X-HE-Tag: 1677617254-130583 X-HE-Meta: U2FsdGVkX18blFAJZYMKbMGOKMf53bk9Amy4SxtqmdG+dLeOuFqu1NTV3GKe9xhPPefg6b1yrrDn6n1JEWBOqTloWmjCOUbF41kxUVm32+z6KGgK0D/t6g9NPxC1QHsKW+INvNAhP3uKuYuSCEQ14HJg30TZccV+47dQZwiIf3jJisKVM1utVwfPwDyeEkktYUNWXFXlAHZefMG/u+Urx9yJf1P0tbiwkUbjDjWhmiOtJ5hBO2TguU8Zq1avd2T8Ikad/vXzl36eJ9WQsbMBmlCi6tfeFijvYI1V2Ptz7z7rrrGbV7En725Mhx3wE928Btm9/iJwnFwQ/Ab+d2v8wKacSLDi/ZbICoj7IX2JVbyPmOWCqQlK4/ke6FRUeIQg8qYjW1A2wklDDJ5/e4tQ1eFKPqZt4B82HkrpOcggcysqYVxrUOOa7w80K4sbPcUHSQmsAaPWguScOsENn/6TAV53Fz2iiOVfZYH50EzhYLTy1lMxUah5aHpWQ3tH5FZwGjhJAf8eqi8UlI/w/WavGt5NUkpwb9/j7ePOPVzysgmy3ru3MX8zcDkB34ZWMn2j4jej7InaI5jZLjizV/H9DEJgwl2+01r5PGFvBA4k6+3qO9OwRfwErz0Ojopuj51m9k+IbADAJSdpEc+dKUQeVoqnL0Cvob0q/jA1BkKaMCFhl1atwVjez4rnhHyHTz4PpyF7sNGQV5/0jj8hq8oL/EigHC0m8bBY0D2njnnJtQSmHDPHOdvNsVROHV3lET3KDIF6byJ7UIYI7HcLNFaZ6gcZKeoddfMofz8/xi/FzlYmEkUSjIrXsMqEn41EvKrMq71aYbcffVp0FWmVWgGKhC3oDpBdpJ/nn9IfEO7NGP+2RwqHH7shQCGzi9qVwLUI176UOngVsQZy6KnPGXEgO6OVoZGIflzHV1MVohVi9dUfSjLic8UZmNSaMiEQ+cpjaOchLrwypPwvDKJOilY f7wWZSWu QsCoWJ70hQacy6VMwTifF6cHd445YV7yzBdjfRpiG679k1pcH20zl/iN84r4sM4S4CzkPIYOWp1SNyj8kJnnH2unHZCyCkR/42fIf8nwPzEneD2NCu1xGq2mzxvzj1ylb2gB/gIC+QgxEtC83UnppBC0R087EEzAuo92qu0HEIJSzyHQr+sI1vcTUfK5Tp2aD7+uLGhy+7zgTkbn00++WHAa96Uy798lwtWnvptO9qhHne6opB6oYnTo1pA1qtCSNgNZt2p4tbulwhJNtnop6HdYquztkuRFAMleKYISHb01RN5FEOlNXEqMKMXegAcLsGlsM7DOp7Q7kzhZKE7hNVaHe3Lh5veVFur+zYx4/iOe5f/VVwnlna7+95TMxgVdjd65QPQETB3UUQmWBXTkwnNLnL+Q0QJ+FG8qrvBtT1KnuKtsodMgwJ075oe0zlBzZ4vONzxmWOihtbTW3NoNdkRU8Ahud/O2Bwpgb6l3l/P90sYU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 24 Feb 2023 13:37:48 +0100 Alexander Graf wrote: > > On 20.02.23 19:38, Michael Roth wrote: > > From: Tom Lendacky > > > > Add support for the SEV-SNP AP Creation NAE event. This allows SEV-SNP > > guests to alter the register state of the APs on their own. This allows > > the guest a way of simulating INIT-SIPI. > > > > A new event, KVM_REQ_UPDATE_PROTECTED_GUEST_STATE, is created and used > > so as to avoid updating the VMSA pointer while the vCPU is running. > > > > For CREATE > > The guest supplies the GPA of the VMSA to be used for the vCPU with > > the specified APIC ID. The GPA is saved in the svm struct of the > > target vCPU, the KVM_REQ_UPDATE_PROTECTED_GUEST_STATE event is added > > to the vCPU and then the vCPU is kicked. > > > > For CREATE_ON_INIT: > > The guest supplies the GPA of the VMSA to be used for the vCPU with > > the specified APIC ID the next time an INIT is performed. The GPA is > > saved in the svm struct of the target vCPU. > > > > For DESTROY: > > The guest indicates it wishes to stop the vCPU. The GPA is cleared > > from the svm struct, the KVM_REQ_UPDATE_PROTECTED_GUEST_STATE event is > > added to vCPU and then the vCPU is kicked. > > > > The KVM_REQ_UPDATE_PROTECTED_GUEST_STATE event handler will be invoked > > as a result of the event or as a result of an INIT. The handler sets the > > vCPU to the KVM_MP_STATE_UNINITIALIZED state, so that any errors will > > leave the vCPU as not runnable. Any previous VMSA pages that were > > installed as part of an SEV-SNP AP Creation NAE event are un-pinned. If > > a new VMSA is to be installed, the VMSA guest page is pinned and set as > > the VMSA in the vCPU VMCB and the vCPU state is set to > > KVM_MP_STATE_RUNNABLE. If a new VMSA is not to be installed, the VMSA is > > cleared in the vCPU VMCB and the vCPU state is left as > > KVM_MP_STATE_UNINITIALIZED to prevent it from being run. > > > > Signed-off-by: Tom Lendacky > > Signed-off-by: Brijesh Singh > > Signed-off-by: Ashish Kalra > > [mdr: add handling for restrictedmem] > > Signed-off-by: Michael Roth > > > What is the intended boot sequence for SEV-SNP guests? FWIW with this > interface in place, guests will typically use in-guest VMSA pages to > hold secondary vcpu state. But that means we're now allocating 4kb of > memory for every vcpu that we create that will be for most of the > guest's lifetime superfluous. > > Wouldn't it make more sense to have a model where we only allocate the > VMSA for the boot CPU and leave secondary allocation to the guest? We > already need firmware changes for SEV-SNP - may as well make this one more. > > [...] > > > + > > +static int sev_snp_ap_creation(struct vcpu_svm *svm) > > +{ > > + struct kvm_sev_info *sev = &to_kvm_svm(svm->vcpu.kvm)->sev_info; > > + struct kvm_vcpu *vcpu = &svm->vcpu; > > + struct kvm_vcpu *target_vcpu; > > + struct vcpu_svm *target_svm; > > + unsigned int request; > > + unsigned int apic_id; > > + bool kick; > > + int ret; > > + > > + request = lower_32_bits(svm->vmcb->control.exit_info_1); > > + apic_id = upper_32_bits(svm->vmcb->control.exit_info_1); > > + > > + /* Validate the APIC ID */ > > + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, apic_id); > > > Out of curiosity: The target CPU can be my own vCPU, right? > > > > + if (!target_vcpu) { > > + vcpu_unimpl(vcpu, "vmgexit: invalid AP APIC ID [%#x] from guest\n", > > + apic_id); > > + return -EINVAL; > > + } > > + > > + ret = 0; > > + > > + target_svm = to_svm(target_vcpu); > > + > > + /* > > + * The target vCPU is valid, so the vCPU will be kicked unless the > > + * request is for CREATE_ON_INIT. For any errors at this stage, the > > + * kick will place the vCPU in an non-runnable state. > > + */ > > + kick = true; > > + > > + mutex_lock(&target_svm->sev_es.snp_vmsa_mutex); > > + > > + target_svm->sev_es.snp_vmsa_gpa = INVALID_PAGE; > > + target_svm->sev_es.snp_ap_create = true; > > + > > + /* Interrupt injection mode shouldn't change for AP creation */ > > + if (request < SVM_VMGEXIT_AP_DESTROY) { > > + u64 sev_features; > > + > > + sev_features = vcpu->arch.regs[VCPU_REGS_RAX]; > > + sev_features ^= sev->sev_features; > > + if (sev_features & SVM_SEV_FEAT_INT_INJ_MODES) { > > + vcpu_unimpl(vcpu, "vmgexit: invalid AP injection mode [%#lx] from guest\n", > > + vcpu->arch.regs[VCPU_REGS_RAX]); > > + ret = -EINVAL; > > + goto out; > > + } > > + } > > + > > + switch (request) { > > + case SVM_VMGEXIT_AP_CREATE_ON_INIT: > > + kick = false; > > + fallthrough; > > + case SVM_VMGEXIT_AP_CREATE: > > + if (!page_address_valid(vcpu, svm->vmcb->control.exit_info_2)) { > > + vcpu_unimpl(vcpu, "vmgexit: invalid AP VMSA address [%#llx] from guest\n", > > + svm->vmcb->control.exit_info_2); > > + ret = -EINVAL; > > + goto out; > > + } > > + > > + /* > > + * Malicious guest can RMPADJUST a large page into VMSA which > > + * will hit the SNP erratum where the CPU will incorrectly signal > > + * an RMP violation #PF if a hugepage collides with the RMP entry > > + * of VMSA page, reject the AP CREATE request if VMSA address from > > + * guest is 2M aligned. > > > This will break genuine current Linux kernels that just happen to > allocate a guest page, no? In fact, given enough vCPUs you're almost > guaranteed to hit an aligned structure somewhere. What is the guest > supposed to do in that situation? > > > > + */ > > + if (IS_ALIGNED(svm->vmcb->control.exit_info_2, PMD_SIZE)) { > > + vcpu_unimpl(vcpu, > > + "vmgexit: AP VMSA address [%llx] from guest is unsafe as it is 2M aligned\n", > > + svm->vmcb->control.exit_info_2); > > + ret = -EINVAL; > > + goto out; > > + } > > + > > + target_svm->sev_es.snp_vmsa_gpa = svm->vmcb->control.exit_info_2; > > + break; > > + case SVM_VMGEXIT_AP_DESTROY: > > > I don't understand the destroy path. Why does this case destroy anything? > > > > + break; > > + default: > > + vcpu_unimpl(vcpu, "vmgexit: invalid AP creation request [%#x] from guest\n", > > + request); > > + ret = -EINVAL; > > + break; > > + } > > + > > +out: > > + if (kick) { > > + if (target_vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED) > > + target_vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; > > > What if the guest AP goes through a create -> destroy -> create cycle? > Will it stay runnable while destroyed? The code is not very straightforward. 1) target_svm->sev_es.snp_vmsa_gpa is set as INVALID_PAGE in the beginning of this function. 2) If a DESTROY is hit in this function, target_svm->sev_es.snp_vmsa_gpa will be left as INVALID_PAGE. 3) At the end of this function, it calls kvm_make_request(KVM_REQ_UPDATE_PROTECTED_GUEST_STATE). 4) In the vcpu_enter_guest(), the kvm_vcpu_reset()->sev_snp_init_protected_guest_state() ->__sev_snp_init_protected_guest_state() is called. 5) The mp_state is set to KVM_MP_STATE_STOPPED by default and the runtime VMSA is cleared. Then the it will be initialized according to the guest's configuration. 6) As the snp_vmsa_gpa is set as INVALID_PAGE in 1, the mp_state will be left as KVM_MP_STATE_STOPPED. 7) With this code piece: + kvm_vcpu_reset(vcpu, true); + if (vcpu->arch.mp_state != KVM_MP_STATE_RUNNABLE) + goto out; vcpu_enter_guest() bails out. > > > Alex > > > + > > + kvm_make_request(KVM_REQ_UPDATE_PROTECTED_GUEST_STATE, target_vcpu); > > + kvm_vcpu_kick(target_vcpu); > > + } > > + > > + mutex_unlock(&target_svm->sev_es.snp_vmsa_mutex); > > + > > + return ret; > > +} > > + > > static int sev_handle_vmgexit_msr_protocol(struct vcpu_svm *svm) > > { > > struct vmcb_control_area *control = &svm->vmcb->control; > > > > Amazon Development Center Germany GmbH > Krausenstr. 38 > 10117 Berlin > Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss > Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B > Sitz: Berlin > Ust-ID: DE 289 237 879 > >