From: Marc Zyngier <maz@kernel.org>
To: linux-arm-kernel@lists.infradead.org,
kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org
Cc: Andre Przywara <andre.przywara@arm.com>,
Christoffer Dall <christoffer.dall@arm.com>,
Jintack Lim <jintack@cs.columbia.edu>,
Haibo Xu <haibo.xu@linaro.org>, James Morse <james.morse@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Alexandru Elisei <alexandru.elisei@arm.com>,
kernel-team@android.com
Subject: [PATCH v4 61/66] KVM: arm64: nv: Synchronize PSTATE early on exit
Date: Mon, 10 May 2021 17:59:15 +0100 [thread overview]
Message-ID: <20210510165920.1913477-62-maz@kernel.org> (raw)
In-Reply-To: <20210510165920.1913477-1-maz@kernel.org>
The NV code relies on predicates such as is_hyp_ctxt() being
reliable. In turn, is_hyp_ctxt() relies on things like PSTATE
and the virtual HCR_EL2 being accurate.
But with ARMv8.4-NV removing trapping for a large part of the
EL2 system registers (among which HCR_EL2), we can't use such
trapping to synchronize the rest of the state.
Let's look at the following sequence for a VHE guest:
(1) enter guest in host EL0
(2) guest traps to guest vEL2 (no hypervisor intervention)
(3) guest clears virtual HCR_EL2.TGE (no trap either)
(4) host interrupt fires, exit
(5) is_hyp_ctxt() now says "guest" (PSTATE.M==EL1 and TGE==0)
It is obvious that such behaviour would be rather unfortunate,
and lead to interesting, difficult to catch bugs specially if
preemption kicks in (yes, I wasted a whole week chasing this one).
In order to preserve the invariant that a guest entered in host
context must exit in the same context, we must make sure that
is_hyp_ctxt() works correctly. Since we can always observe the
guest value of HCR_EL2.{E2H,TGE} in the VNCR_EL2 page, we solely
need to synchronize PSTATE as early as possible.
This basically amounts to moving from_hw_pstate() as close
as possible to the guest exit point, and fixup_guest_exit()
seems as good a place as any.
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
arch/arm64/kvm/hyp/include/hyp/switch.h | 16 ++++--
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 26 ++--------
arch/arm64/kvm/hyp/nvhe/switch.c | 8 ++-
arch/arm64/kvm/hyp/vhe/switch.c | 57 +++++++++++++++++++++-
4 files changed, 78 insertions(+), 29 deletions(-)
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 0790eb2b7545..6011f32fdb32 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -408,11 +408,11 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)
}
/*
- * Return true when we were able to fixup the guest exit and should return to
- * the guest, false when we should restore the host state and return to the
- * main run loop.
+ * Prologue for the guest fixup, populating ESR_EL2 and fixing up PC
+ * if required.
*/
-static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
+static inline void fixup_guest_exit_prologue(struct kvm_vcpu *vcpu,
+ u64 *exit_code)
{
if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ)
vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR);
@@ -431,7 +431,15 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
if (esr_ec == ESR_ELx_EC_HVC32 || esr_ec == ESR_ELx_EC_HVC64)
write_sysreg_el2(read_sysreg_el2(SYS_ELR) - 4, SYS_ELR);
}
+}
+/*
+ * Return true when we were able to fixup the guest exit and should return to
+ * the guest, false when we should restore the host state and return to the
+ * main run loop.
+ */
+static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
+{
/*
* We're using the raw exception code in order to only process
* the trap if no SError is pending. We will come back to the
diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
index 92715fa01e88..1931c8667d52 100644
--- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
@@ -51,32 +51,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt_sys_reg(ctxt, SPSR_EL1) = read_sysreg_el1(SYS_SPSR);
}
-static inline u64 from_hw_pstate(const struct kvm_cpu_context *ctxt)
-{
- u64 reg = read_sysreg_el2(SYS_SPSR);
-
- if (__is_hyp_ctxt(ctxt)) {
- u64 mode = reg & (PSR_MODE_MASK | PSR_MODE32_BIT);
-
- switch (mode) {
- case PSR_MODE_EL1t:
- mode = PSR_MODE_EL2t;
- break;
- case PSR_MODE_EL1h:
- mode = PSR_MODE_EL2h;
- break;
- }
-
- return (reg & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode;
- }
-
- return reg;
-}
-
static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
{
+ /* On VHE, PSTATE is saved in fixup_guest_exit_vhe() */
+ if (!has_vhe())
+ ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR);
ctxt->regs.pc = read_sysreg_el2(SYS_ELR);
- ctxt->regs.pstate = from_hw_pstate(ctxt);
if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2);
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 2b0f8675fe3b..b9b9c8e0a9f2 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -166,6 +166,12 @@ static void __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt)
write_sysreg(pmu->events_host, pmcntenset_el0);
}
+static bool fixup_guest_exit_nvhe(struct kvm_vcpu *vcpu, u64 *exit_code)
+{
+ fixup_guest_exit_prologue(vcpu, exit_code);
+ return fixup_guest_exit(vcpu, exit_code);
+}
+
/* Switch to the guest for legacy non-VHE systems */
int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
{
@@ -227,7 +233,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
exit_code = __guest_enter(vcpu);
/* And we're baaack! */
- } while (fixup_guest_exit(vcpu, &exit_code));
+ } while (fixup_guest_exit_nvhe(vcpu, &exit_code));
__sysreg_save_state_nvhe(guest_ctxt);
__sysreg32_save_state(vcpu);
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 2725dc62ab09..b665a3cc288e 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -154,12 +154,60 @@ void deactivate_traps_vhe_put(void)
__deactivate_traps_common();
}
+static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code,
+ bool hyp_ctxt)
+{
+ u64 pstate = read_sysreg_el2(SYS_SPSR);
+
+ /*
+ * Sync pstate back as early as possible, so that is_hyp_ctxt()
+ * reflects the exact context. It is otherwise possible to get
+ * confused with a VHE guest and ARMv8.4-NV, such as:
+ *
+ * (1) enter guest in host EL0
+ * (2) guest traps to guest vEL2 (no hypervisor intervention)
+ * (3) guest clears virtual HCR_EL2.TGE (no trap either)
+ * (4) host interrupt fires, exit
+ * (5) is_hyp_ctxt() now says "guest" (pstate.M==EL1 and TGE==0)
+ *
+ * If host preemption occurs, vcpu_load/put() will be very confused.
+ *
+ * Consider this as the prologue before the fixup prologue...
+ */
+
+ if (unlikely(hyp_ctxt)) {
+ u64 mode = pstate & PSR_MODE_MASK;
+
+ switch (mode) {
+ case PSR_MODE_EL1t:
+ mode = PSR_MODE_EL2t;
+ break;
+ case PSR_MODE_EL1h:
+ mode = PSR_MODE_EL2h;
+ break;
+ }
+
+ pstate = (pstate & ~PSR_MODE_MASK) | mode;
+ }
+
+ *vcpu_cpsr(vcpu) = pstate;
+
+ fixup_guest_exit_prologue(vcpu, exit_code);
+
+ if (*exit_code == ARM_EXCEPTION_TRAP) {
+ /* more to come here */
+ }
+
+ return fixup_guest_exit(vcpu, exit_code);
+}
+
/* Switch to the guest for VHE systems running in EL2 */
static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
+ bool hyp_ctxt;
host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
@@ -186,12 +234,19 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
+ /*
+ * Being in HYP context or not is an invariant here. If we enter in
+ * a given context, we exit in the same context. We can thus only
+ * sample the context once.
+ */
+ WRITE_ONCE(hyp_ctxt, is_hyp_ctxt(vcpu));
+
do {
/* Jump in the fire! */
exit_code = __guest_enter(vcpu);
/* And we're baaack! */
- } while (fixup_guest_exit(vcpu, &exit_code));
+ } while (fixup_guest_exit_vhe(vcpu, &exit_code, READ_ONCE(hyp_ctxt)));
sysreg_save_guest_state_vhe(guest_ctxt);
--
2.29.2
next prev parent reply other threads:[~2021-05-10 17:29 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-10 16:58 [PATCH v4 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 01/66] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Marc Zyngier
2021-05-20 13:32 ` Zenghui Yu
2021-05-10 16:58 ` [PATCH v4 02/66] KVM: arm64: nv: Introduce nested virtualization VCPU feature Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 03/66] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 04/66] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 05/66] KVM: arm64: nv: Add EL2 system registers to vcpu context Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 06/66] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 07/66] KVM: arm64: nv: Handle HCR_EL2.NV system register traps Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 08/66] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 09/66] KVM: arm64: nv: Support virtual EL2 exceptions Marc Zyngier
2021-05-20 12:55 ` Zenghui Yu
2021-05-10 16:58 ` [PATCH v4 10/66] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 11/66] KVM: arm64: nv: Handle trapped ERET from " Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 12/66] KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 13/66] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 14/66] KVM: arm64: nv: Handle SPSR_EL2 specially Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 15/66] KVM: arm64: nv: Handle HCR_EL2.E2H specially Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 16/66] KVM: arm64: nv: Save/Restore vEL2 sysregs Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 17/66] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 18/66] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 19/66] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from " Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 20/66] KVM: arm64: nv: Trap CPACR_EL1 access in " Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 21/66] KVM: arm64: nv: Handle PSCI call via smc from the guest Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 22/66] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 23/66] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 24/66] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 25/66] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 26/66] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 27/66] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 28/66] KVM: arm64: nv: Forward debug traps to the nested guest Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 29/66] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 30/66] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 31/66] KVM: arm64: nv: Filter out unsupported features from ID regs Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 32/66] KVM: arm64: nv: Hide RAS from nested guests Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 33/66] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 34/66] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 35/66] KVM: arm64: nv: Handle shadow stage 2 page faults Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 36/66] KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 37/66] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 38/66] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 39/66] KVM: arm64: nv: Set a handler for the system instruction traps Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 40/66] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 41/66] KVM: arm64: nv: Trap and emulate TLBI " Marc Zyngier
2021-07-14 16:40 ` Chase Conklin
2021-11-29 18:12 ` Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 42/66] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 43/66] KVM: arm64: nv: arch_timer: Support hyp timer emulation Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 44/66] KVM: arm64: nv: Add handling of EL2-specific timer registers Marc Zyngier
2021-05-10 16:58 ` [PATCH v4 45/66] KVM: arm64: nv: Load timer before the GIC Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 46/66] KVM: arm64: nv: Nested GICv3 Support Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 47/66] KVM: arm64: nv: Don't load the GICv4 context on entering a nested guest Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 48/66] KVM: arm64: nv: vgic: Emulate the HW bit in software Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 49/66] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 50/66] KVM: arm64: nv: Implement maintenance interrupt forwarding Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 51/66] KVM: arm64: nv: Add nested GICv3 tracepoints Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 52/66] KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 53/66] KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 54/66] KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like information Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 55/66] KVM: arm64: Allow populating S2 SW bits Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 56/66] KVM: arm64: nv: Tag shadow S2 entries with nested level Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 57/66] KVM: arm64: nv: Add include containing the VNCR_EL2 offsets Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 58/66] KVM: arm64: Map VNCR-capable registers to a separate page Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 59/66] KVM: arm64: nv: Move nested vgic state into the sysreg file Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 60/66] KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature Marc Zyngier
2021-05-10 16:59 ` Marc Zyngier [this message]
2021-05-10 16:59 ` [PATCH v4 62/66] KVM: arm64: nv: Sync nested timer state with ARMv8.4 Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 63/66] KVM: arm64: nv: Allocate VNCR page when required Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 64/66] KVM: arm64: nv: Enable ARMv8.4-NV support Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 65/66] KVM: arm64: nv: Fast-track 'InHost' exception returns Marc Zyngier
2021-05-10 16:59 ` [PATCH v4 66/66] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Marc Zyngier
2021-06-03 7:07 ` [PATCH v4 00/66] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Jamie Iles
2021-06-03 8:39 ` Marc Zyngier
2021-06-03 11:08 ` Marc Zyngier
2021-06-07 9:59 ` Jamie Iles
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210510165920.1913477-62-maz@kernel.org \
--to=maz@kernel.org \
--cc=alexandru.elisei@arm.com \
--cc=andre.przywara@arm.com \
--cc=christoffer.dall@arm.com \
--cc=haibo.xu@linaro.org \
--cc=james.morse@arm.com \
--cc=jintack@cs.columbia.edu \
--cc=kernel-team@android.com \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=suzuki.poulose@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).