From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95A02C1B0E3 for ; Thu, 10 Dec 2020 16:14:39 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 35BA123118 for ; Thu, 10 Dec 2020 16:14:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 35BA123118 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YayhJ40pzwCFsgT5x4g2PG2o3V3wB40GZPIk+Hdzblw=; b=3QWI7gYjKx6ioVTqL5uiO/fIB 80Sk9qV2ZJ7Coq0kIVpF+Cj0IKeueLynBw8Pw9Opo/LNhY70EkLNj1GKhHuD4LB6SCfpOI3kEFlsi imKK+HQgUzhEH/bRS54J/Bc1mmJfBJmE+52U37wItW+COUmVMmvRE0b29l3GQzIk0GoXjrBiTqPHq P1DZIyjiHL062iuUUNCRhIIyZtCXqG1oLpB36+59P62czgZzOjpuK3A0EsBUGeoWXuxF7rhE2V4cD twEOtpEAnasUMUCgvmp8/9isXxrZ3Zll+HBUVWQ3+OQZ7/phMfPaTt89rMpCjcXEQItqLA9dPsoeO AySed0V9g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1knOYl-0000OA-HX; Thu, 10 Dec 2020 16:12:51 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1knOQK-0007Bz-Rf for linux-arm-kernel@lists.infradead.org; Thu, 10 Dec 2020 16:04:12 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8C3F523D5A; Thu, 10 Dec 2020 16:04:06 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1knONo-0008Di-Hx; Thu, 10 Dec 2020 16:01:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Subject: [PATCH v3 61/66] KVM: arm64: nv: Synchronize PSTATE early on exit Date: Thu, 10 Dec 2020 15:59:57 +0000 Message-Id: <20201210160002.1407373-62-maz@kernel.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201210160002.1407373-1-maz@kernel.org> References: <20201210160002.1407373-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201210_110409_250763_12021C87 X-CRM114-Status: GOOD ( 24.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kernel-team@android.com, Suzuki K Poulose , Andre Przywara , Christoffer Dall , James Morse , Alexandru Elisei , Jintack Lim , Julien Thierry Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The NV code relies on predicates such as is_hyp_ctxt() being reliable. In turn, is_hyp_ctxt() relies on things like PSTATE and the virtual HCR_EL2 being accurate. But with ARMv8.4-NV removing trapping for a large part of the EL2 system registers (among which HCR_EL2), we can't use such trapping to synchronize the rest of the state. Let's look at the following sequence for a VHE guest: (1) enter guest in host EL0 (2) guest traps to guest vEL2 (no hypervisor intervention) (3) guest clears virtual HCR_EL2.TGE (no trap either) (4) host interrupt fires, exit (5) is_hyp_ctxt() now says "guest" (PSTATE.M==EL1 and TGE==0) It is obvious that such behaviour would be rather unfortunate, and lead to interesting, difficult to catch bugs specially if preemption kicks in (yes, I wasted a whole week chasing this one). In order to preserve the invariant that a guest entered in host context must exit in the same context, we must make sure that is_hyp_ctxt() works correctly. Since we can always observe the guest value of HCR_EL2.{E2H,TGE} in the VNCR_EL2 page, we solely need to synchronize PSTATE as early as possible. This basically amounts to moving from_hw_pstate() as close as possible to the guest exit point, and fixup_guest_exit() seems as good a place as any. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/switch.h | 16 ++++-- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 26 ++-------- arch/arm64/kvm/hyp/nvhe/switch.c | 8 ++- arch/arm64/kvm/hyp/vhe/switch.c | 57 +++++++++++++++++++++- 4 files changed, 78 insertions(+), 29 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index e5e201314c87..3b56841eb328 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -407,11 +407,11 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) } /* - * Return true when we were able to fixup the guest exit and should return to - * the guest, false when we should restore the host state and return to the - * main run loop. + * Prologue for the guest fixup, populating ESR_EL2 and fixing up PC + * if required. */ -static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline void fixup_guest_exit_prologue(struct kvm_vcpu *vcpu, + u64 *exit_code) { if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); @@ -430,7 +430,15 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) if (esr_ec == ESR_ELx_EC_HVC32 || esr_ec == ESR_ELx_EC_HVC64) write_sysreg_el2(read_sysreg_el2(SYS_ELR) - 4, SYS_ELR); } +} +/* + * Return true when we were able to fixup the guest exit and should return to + * the guest, false when we should restore the host state and return to the + * main run loop. + */ +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +{ /* * We're using the raw exception code in order to only process * the trap if no SError is pending. We will come back to the diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 92715fa01e88..1931c8667d52 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -51,32 +51,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, SPSR_EL1) = read_sysreg_el1(SYS_SPSR); } -static inline u64 from_hw_pstate(const struct kvm_cpu_context *ctxt) -{ - u64 reg = read_sysreg_el2(SYS_SPSR); - - if (__is_hyp_ctxt(ctxt)) { - u64 mode = reg & (PSR_MODE_MASK | PSR_MODE32_BIT); - - switch (mode) { - case PSR_MODE_EL1t: - mode = PSR_MODE_EL2t; - break; - case PSR_MODE_EL1h: - mode = PSR_MODE_EL2h; - break; - } - - return (reg & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; - } - - return reg; -} - static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { + /* On VHE, PSTATE is saved in fixup_guest_exit_vhe() */ + if (!has_vhe()) + ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); ctxt->regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->regs.pstate = from_hw_pstate(ctxt); if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 07bf5c03631b..fb49757d3446 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -166,6 +166,12 @@ static void __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt) write_sysreg(pmu->events_host, pmcntenset_el0); } +static bool fixup_guest_exit_nvhe(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + fixup_guest_exit_prologue(vcpu, exit_code); + return fixup_guest_exit(vcpu, exit_code); +} + /* Switch to the guest for legacy non-VHE systems */ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) { @@ -219,7 +225,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) exit_code = __guest_enter(vcpu); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit_nvhe(vcpu, &exit_code)); __sysreg_save_state_nvhe(guest_ctxt); __sysreg32_save_state(vcpu); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index d9dc470c7790..4d80596e32a5 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -156,12 +156,60 @@ void deactivate_traps_vhe_put(void) __deactivate_traps_common(); } +static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code, + bool hyp_ctxt) +{ + u64 pstate = read_sysreg_el2(SYS_SPSR); + + /* + * Sync pstate back as early as possible, so that is_hyp_ctxt() + * reflects the exact context. It is otherwise possible to get + * confused with a VHE guest and ARMv8.4-NV, such as: + * + * (1) enter guest in host EL0 + * (2) guest traps to guest vEL2 (no hypervisor intervention) + * (3) guest clears virtual HCR_EL2.TGE (no trap either) + * (4) host interrupt fires, exit + * (5) is_hyp_ctxt() now says "guest" (pstate.M==EL1 and TGE==0) + * + * If host preemption occurs, vcpu_load/put() will be very confused. + * + * Consider this as the prologue before the fixup prologue... + */ + + if (unlikely(hyp_ctxt)) { + u64 mode = pstate & PSR_MODE_MASK; + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + pstate = (pstate & ~PSR_MODE_MASK) | mode; + } + + *vcpu_cpsr(vcpu) = pstate; + + fixup_guest_exit_prologue(vcpu, exit_code); + + if (*exit_code == ARM_EXCEPTION_TRAP) { + /* more to come here */ + } + + return fixup_guest_exit(vcpu, exit_code); +} + /* Switch to the guest for VHE systems running in EL2 */ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *host_ctxt; struct kvm_cpu_context *guest_ctxt; u64 exit_code; + bool hyp_ctxt; host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; @@ -188,12 +236,19 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); + /* + * Being in HYP context or not is an invariant here. If we enter in + * a given context, we exit in the same context. We can thus only + * sample the context once. + */ + WRITE_ONCE(hyp_ctxt, is_hyp_ctxt(vcpu)); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit_vhe(vcpu, &exit_code, READ_ONCE(hyp_ctxt))); sysreg_save_guest_state_vhe(guest_ctxt); -- 2.29.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel