KVM ARM Archive on lore.kernel.org
 help / color / Atom feed
From: Marc Zyngier <maz@kernel.org>
To: linux-arm-kernel@lists.infradead.org,
	kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org
Cc: Andre Przywara <andre.przywara@arm.com>,
	Dave Martin <Dave.Martin@arm.com>
Subject: [PATCH v2 93/94] KVM: arm64: nv: Fast-track 'InHost' exception returns
Date: Tue, 11 Feb 2020 17:49:37 +0000
Message-ID: <20200211174938.27809-94-maz@kernel.org> (raw)
In-Reply-To: <20200211174938.27809-1-maz@kernel.org>

A significant part of the ARMv8.3-NV extension is to trap ERET
instructions so that the hypervisor gets a chance to switch
from a vEL2 L1 guest to an EL1 L2 guest.

But this also has the unfortunate consequence of trapping ERET
in unsuspecting circumstances, such as staying at vEL2 (interrupt
handling while being in the guest hypervisor), or returning to host
userspace in the case of a VHE guest.

Although we already make some effort to handle these ERET quicker
by not doing the put/load dance, it is still way too far down the
line for it to be efficient enough.

For these cases, it would ideal to ERET directly, no question asked.
Of course, we can't do that. But the next best thing is to do it as
early as possible, in fixup_guest_exit(), much as we would handle
FPSIMD exceptions.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/emulate-nested.c | 26 ++--------------
 arch/arm64/kvm/hyp/switch.c     | 54 +++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 97af080b9b52..eef7394e5256 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -78,8 +78,7 @@ static u64 get_el2_except_vector(struct kvm_vcpu *vcpu,
 
 void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
 {
-	u64 spsr, elr, mode;
-	bool direct_eret;
+	u64 spsr, elr;
 
 	/*
 	 * Forward this trap to the virtual EL2 if the virtual
@@ -88,31 +87,10 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
 	if (forward_nv_traps(vcpu))
 		return;
 
-	/*
-	 * Going through the whole put/load motions is a waste of time
-	 * if this is a VHE guest hypervisor returning to its own
-	 * userspace, or the hypervisor performing a local exception
-	 * return. No need to save/restore registers, no need to
-	 * switch S2 MMU. Just do the canonical ERET.
-	 */
-	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
-	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
-
-	direct_eret  = (mode == PSR_MODE_EL0t &&
-			vcpu_el2_e2h_is_set(vcpu) &&
-			vcpu_el2_tge_is_set(vcpu));
-	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
-
-	if (direct_eret) {
-		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
-		*vcpu_cpsr(vcpu) = spsr;
-		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
-		return;
-	}
-
 	preempt_disable();
 	kvm_arch_vcpu_put(vcpu);
 
+	spsr = __vcpu_sys_reg(vcpu, SPSR_EL2);
 	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
 
 	trace_kvm_nested_eret(vcpu, elr, spsr);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 2eca04ca96b6..d8aae4b5f089 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -489,6 +489,57 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu)
 	return true;
 }
 
+static bool __hyp_text __hyp_handle_eret(struct kvm_vcpu *vcpu)
+{
+	struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt;
+	u64 spsr, mode;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET. Unless the trap
+	 * has to be forwarded further down the line, of course...
+	 */
+	if (kvm_vcpu_trap_get_class(vcpu) != ESR_ELx_EC_ERET)
+		return false;
+
+	/*
+	 * Let the trap forwarding be handled by the normal exception
+	 * handling code.
+	 */
+	if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV)
+		return false;
+
+	spsr = read_sysreg_el1(SYS_SPSR);
+	spsr = __fixup_spsr_el2_read(ctxt, spsr);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	switch (mode) {
+	case PSR_MODE_EL0t:
+		if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)))
+			return false;
+		break;
+	case PSR_MODE_EL2t:
+		mode = PSR_MODE_EL1t;
+		break;
+	case PSR_MODE_EL2h:
+		mode = PSR_MODE_EL1h;
+		break;
+	default:
+		return false;
+	}
+
+	spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode;
+
+	write_sysreg_el2(spsr, SYS_SPSR);
+	write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR);
+
+	return true;
+}
+
+
 static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu)
 {
 	u32 sysreg = esr_sys64_to_sysreg(kvm_vcpu_get_hsr(vcpu));
@@ -615,6 +666,9 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code,
 	if (__hyp_handle_fpsimd(vcpu))
 		return true;
 
+	if (__hyp_handle_eret(vcpu))
+		return true;
+
 	if (!__populate_fault_info(vcpu))
 		return true;
 
-- 
2.20.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

  parent reply index

Thread overview: 104+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-11 17:48 [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 01/94] KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 02/94] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 03/94] KVM: arm64: nv: Introduce nested virtualization VCPU feature Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 04/94] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 05/94] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 06/94] KVM: arm64: nv: Add EL2 system registers to vcpu context Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 07/94] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 08/94] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 09/94] KVM: arm64: nv: Support virtual EL2 exceptions Marc Zyngier
2020-02-17 12:52   ` Mark Rutland
2020-02-17 13:41     ` Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 10/94] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 11/94] KVM: arm64: nv: Handle trapped ERET from " Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 12/94] KVM: arm64: nv: Add EL2->EL1 translation helpers Marc Zyngier
2020-02-17 14:56   ` Mark Rutland
2020-02-17 17:43     ` Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 13/94] KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 14/94] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 15/94] KVM: arm64: nv: Handle SPSR_EL2 specially Marc Zyngier
2020-02-17 15:04   ` Mark Rutland
2020-02-11 17:48 ` [PATCH v2 16/94] KVM: arm64: nv: Handle HCR_EL2.E2H specially Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 17/94] KVM: arm64: nv: Save/Restore vEL2 sysregs Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 18/94] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 19/94] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 20/94] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from " Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 21/94] KVM: arm64: nv: Trap CPACR_EL1 access in " Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 22/94] KVM: arm64: nv: Handle PSCI call via smc from the guest Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 23/94] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 24/94] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP, FPEN} settings Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 25/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 26/94] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 27/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 28/94] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 29/94] KVM: arm64: nv: Forward debug traps to the nested guest Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 30/94] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 31/94] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Marc Zyngier
2020-02-17 15:13   ` Mark Rutland
2020-02-11 17:48 ` [PATCH v2 32/94] KVM: arm64: nv: Filter out unsupported features from ID regs Marc Zyngier
2020-02-17 15:17   ` Mark Rutland
2020-02-11 17:48 ` [PATCH v2 33/94] KVM: arm64: nv: Hide RAS from nested guests Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 34/94] KVM: arm64: nv: Use ARMv8.5-GTG to advertise supported Stage-2 page sizes Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 35/94] KVM: arm64: Check advertised Stage-2 page size capability Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 36/94] KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 37/94] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 38/94] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 39/94] KVM: arm64: nv: Handle shadow stage 2 page faults Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 40/94] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 41/94] KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 42/94] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 43/94] KVM: arm64: nv: Set a handler for the system instruction traps Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 44/94] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 45/94] KVM: arm64: nv: Trap and emulate TLBI " Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 46/94] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 47/94] KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 48/94] KVM: arm64: nv: arch_timer: Support hyp timer emulation Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 49/94] KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 50/94] KVM: arm64: nv: Load timer before the GIC Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 51/94] KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 52/94] KVM: arm64: nv: Nested GICv3 Support Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 53/94] KVM: arm64: nv: vgic: Emulate the HW bit in software Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 54/94] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Marc Zyngier
2020-02-11 17:48 ` [PATCH v2 55/94] KVM: arm64: nv: Implement maintenance interrupt forwarding Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 56/94] KVM: arm64: nv: Add nested GICv3 tracepoints Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 57/94] arm64: KVM: nv: Add handling of EL2-specific timer registers Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 58/94] arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 59/94] arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 60/94] arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 61/94] arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 62/94] arm64: Detect the ARMv8.4 TTL feature Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 63/94] arm64: KVM: nv: Add handling of ARMv8.4-TTL TLB invalidation Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 64/94] arm64: KVM: nv: Invalidate TLBs based on shadow S2 TTL-like information Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 65/94] arm64: KVM: nv: Tag shadow S2 entries with nested level Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 66/94] arm64: Add SW reserved PTE/PMD bits Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 67/94] arm64: Add level-hinted TLB invalidation helper Marc Zyngier
2020-03-25 10:38   ` Zhenyu Ye
2020-03-25 11:28     ` Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 68/94] arm64: KVM: Add a level hint to __kvm_tlb_flush_vmid_ipa Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 69/94] arm64: KVM: Use TTL hint in when invalidating stage-2 translations Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 70/94] arm64: KVM: nv: Add include containing the VNCR_EL2 offsets Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 71/94] KVM: arm64: Introduce accessor for ctxt->sys_reg Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 72/94] KVM: arm64: sysreg: Use ctxt_sys_reg() instead of raw sys_regs access Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 73/94] KVM: arm64: sve: Use __vcpu_sys_reg() " Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 74/94] KVM: arm64: pauth: Use ctxt_sys_reg() " Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 75/94] KVM: arm64: debug: " Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 76/94] KVM: arm64: Add missing reset handlers for PMU emulation Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 77/94] KVM: arm64: nv: Move sysreg reset check to boot time Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 78/94] KVM: arm64: Map VNCR-capable registers to a separate page Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 79/94] KVM: arm64: nv: Move nested vgic state into the sysreg file Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 80/94] KVM: arm64: Use accessors for timer ctl/cval/offset Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 81/94] KVM: arm64: Add VNCR-capable timer accessors for arm64 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 82/94] KVM: arm64: Make struct kvm_regs userspace-only Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 83/94] KVM: arm64: VNCR-ize ELR_EL1 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 84/94] KVM: arm64: VNCR-ize SP_EL1 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 85/94] KVM: arm64: Disintegrate SPSR array Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 86/94] KVM: arm64: aarch32: Use __vcpu_sys_reg() instead of raw sys_regs access Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 87/94] KVM: arm64: VNCR-ize SPSR_EL1 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 88/94] KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 89/94] KVM: arm64: nv: Synchronize PSTATE early on exit Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 90/94] KVM: arm64: nv: Sync nested timer state with ARMv8.4 Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 91/94] KVM: arm64: nv: Allocate VNCR page when required Marc Zyngier
2020-02-11 17:49 ` [PATCH v2 92/94] KVM: arm64: nv: Enable ARMv8.4-NV support Marc Zyngier
2020-02-11 17:49 ` Marc Zyngier [this message]
2020-02-11 17:49 ` [PATCH v2 94/94] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200211174938.27809-94-maz@kernel.org \
    --to=maz@kernel.org \
    --cc=Dave.Martin@arm.com \
    --cc=andre.przywara@arm.com \
    --cc=kvm@vger.kernel.org \
    --cc=kvmarm@lists.cs.columbia.edu \
    --cc=linux-arm-kernel@lists.infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

KVM ARM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvmarm/0 kvmarm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvmarm kvmarm/ https://lore.kernel.org/kvmarm \
		kvmarm@lists.cs.columbia.edu
	public-inbox-index kvmarm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/edu.columbia.cs.lists.kvmarm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git