From: Marc Zyngier <maz@kernel.org> To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara <andre.przywara@arm.com>, Dave Martin <Dave.Martin@arm.com> Subject: [PATCH v2 94/94] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Date: Tue, 11 Feb 2020 17:49:38 +0000 Message-ID: <20200211174938.27809-95-maz@kernel.org> (raw) In-Reply-To: <20200211174938.27809-1-maz@kernel.org> Due to the way ARMv8.4-NV suppresses traps when accessing EL2 system registers, we can't track when the guest changes its HCR_EL2.TGE setting. This means we alwats trap EL1 TLBIs, even if they don't affect any guest. This obviously has a huge impact on performance, as we handle TLBI traps as a normal exit, and a normal VHE host issues thousands of TLBIs when booting (and quite a few when running userspace). A cheap way to reduce the overhead is to handle the limited case of {E2H,TGE}=={1,1} as a guest fixup, as we already have the right mmu configuration in place. Just execute the decoded instruction right away and return to the guest. Signed-off-by: Marc Zyngier <maz@kernel.org> --- arch/arm64/kvm/hyp/switch.c | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/tlb.c | 6 ++++-- arch/arm64/kvm/sys_regs.c | 28 +++++++++------------------- 3 files changed, 48 insertions(+), 21 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index d8aae4b5f089..c892fdfecf56 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -539,6 +539,38 @@ static bool __hyp_text __hyp_handle_eret(struct kvm_vcpu *vcpu) return true; } +static bool __hyp_text __hyp_handle_tlbi_el1(struct kvm_vcpu *vcpu) +{ + u32 instr; + u64 val; + + /* + * Ideally, we would never trap on EL1 TLB invalidations when the + * guest's HCR_EL2.{E2H,TGE} == {1,1}. But "thanks" to ARMv8.4, we + * don't trap writes to HCR_EL2, meaning that we can't track + * changes to the virtual TGE bit. So we leave HCR_EL2.TTLB set on + * the host. Oopsie... + * + * In order to speed-up EL1 TLBIs from the vEL2 guest when TGE is + * set, try and handle these invalidation as quickly as possible, + * without fully exiting (unless this needs forwarding). + */ + if (kvm_vcpu_trap_get_class(vcpu) != ESR_ELx_EC_SYS64 || + !vcpu_mode_el2(vcpu) || + (__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) + return false; + + instr = esr_sys64_to_sysreg(kvm_vcpu_get_hsr(vcpu)); + if (sys_reg_Op0(instr) != TLBI_Op0 || + sys_reg_Op1(instr) != TLBI_Op1_EL1) + return false; + + val = vcpu_get_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu)); + __kvm_tlb_el1_instr(NULL, val, instr); + __kvm_skip_instr(vcpu); + + return true; +} static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) { @@ -669,6 +701,9 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code, if (__hyp_handle_eret(vcpu)) return true; + if (__hyp_handle_tlbi_el1(vcpu)) + return true; + if (!__populate_fault_info(vcpu)) return true; diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 71e4e86e0981..ef52c5a58866 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -278,7 +278,8 @@ void __hyp_text __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_enc dsb(ishst); /* Switch to requested VMID */ - __tlb_switch_to_guest(mmu, &cxt); + if (mmu) + __tlb_switch_to_guest(mmu, &cxt); /* * Execute the same instruction as the guest hypervisor did, @@ -317,5 +318,6 @@ void __hyp_text __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_enc dsb(ish); isb(); - __tlb_switch_to_host(mmu, &cxt); + if (mmu) + __tlb_switch_to_host(mmu, &cxt); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f20f5975633f..c464bc3d8dad 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2688,6 +2688,8 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; /* * If we're here, this is because we've trapped on a EL1 TLBI @@ -2706,27 +2708,15 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, mutex_lock(&vcpu->kvm->lock); - if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { - u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - struct kvm_s2_mmu *mmu; - - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); - if (mmu) - kvm_call_hyp(__kvm_tlb_el1_instr, - mmu, p->regval, sys_encoding); + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + kvm_call_hyp(__kvm_tlb_el1_instr, + mmu, p->regval, sys_encoding); - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); - if (mmu) - kvm_call_hyp(__kvm_tlb_el1_instr, - mmu, p->regval, sys_encoding); - } else { - /* - * ARMv8.4-NV allows the guest to change TGE behind - * our back, so we always trap EL1 TLBIs from vEL2... - */ + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) kvm_call_hyp(__kvm_tlb_el1_instr, - &vcpu->kvm->arch.mmu, p->regval, sys_encoding); - } + mmu, p->regval, sys_encoding); mutex_unlock(&vcpu->kvm->lock); -- 2.20.1 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
prev parent reply index Thread overview: 104+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-02-11 17:48 [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 01/94] KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 02/94] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 03/94] KVM: arm64: nv: Introduce nested virtualization VCPU feature Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 04/94] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 05/94] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 06/94] KVM: arm64: nv: Add EL2 system registers to vcpu context Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 07/94] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 08/94] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 09/94] KVM: arm64: nv: Support virtual EL2 exceptions Marc Zyngier 2020-02-17 12:52 ` Mark Rutland 2020-02-17 13:41 ` Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 10/94] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 11/94] KVM: arm64: nv: Handle trapped ERET from " Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 12/94] KVM: arm64: nv: Add EL2->EL1 translation helpers Marc Zyngier 2020-02-17 14:56 ` Mark Rutland 2020-02-17 17:43 ` Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 13/94] KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 14/94] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 15/94] KVM: arm64: nv: Handle SPSR_EL2 specially Marc Zyngier 2020-02-17 15:04 ` Mark Rutland 2020-02-11 17:48 ` [PATCH v2 16/94] KVM: arm64: nv: Handle HCR_EL2.E2H specially Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 17/94] KVM: arm64: nv: Save/Restore vEL2 sysregs Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 18/94] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 19/94] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 20/94] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from " Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 21/94] KVM: arm64: nv: Trap CPACR_EL1 access in " Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 22/94] KVM: arm64: nv: Handle PSCI call via smc from the guest Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 23/94] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 24/94] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP, FPEN} settings Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 25/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 26/94] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 27/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 28/94] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 29/94] KVM: arm64: nv: Forward debug traps to the nested guest Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 30/94] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 31/94] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Marc Zyngier 2020-02-17 15:13 ` Mark Rutland 2020-02-11 17:48 ` [PATCH v2 32/94] KVM: arm64: nv: Filter out unsupported features from ID regs Marc Zyngier 2020-02-17 15:17 ` Mark Rutland 2020-02-11 17:48 ` [PATCH v2 33/94] KVM: arm64: nv: Hide RAS from nested guests Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 34/94] KVM: arm64: nv: Use ARMv8.5-GTG to advertise supported Stage-2 page sizes Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 35/94] KVM: arm64: Check advertised Stage-2 page size capability Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 36/94] KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 37/94] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 38/94] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 39/94] KVM: arm64: nv: Handle shadow stage 2 page faults Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 40/94] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 41/94] KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 42/94] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 43/94] KVM: arm64: nv: Set a handler for the system instruction traps Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 44/94] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 45/94] KVM: arm64: nv: Trap and emulate TLBI " Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 46/94] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 47/94] KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 48/94] KVM: arm64: nv: arch_timer: Support hyp timer emulation Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 49/94] KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 50/94] KVM: arm64: nv: Load timer before the GIC Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 51/94] KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 52/94] KVM: arm64: nv: Nested GICv3 Support Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 53/94] KVM: arm64: nv: vgic: Emulate the HW bit in software Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 54/94] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Marc Zyngier 2020-02-11 17:48 ` [PATCH v2 55/94] KVM: arm64: nv: Implement maintenance interrupt forwarding Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 56/94] KVM: arm64: nv: Add nested GICv3 tracepoints Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 57/94] arm64: KVM: nv: Add handling of EL2-specific timer registers Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 58/94] arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 59/94] arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 60/94] arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 61/94] arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 62/94] arm64: Detect the ARMv8.4 TTL feature Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 63/94] arm64: KVM: nv: Add handling of ARMv8.4-TTL TLB invalidation Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 64/94] arm64: KVM: nv: Invalidate TLBs based on shadow S2 TTL-like information Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 65/94] arm64: KVM: nv: Tag shadow S2 entries with nested level Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 66/94] arm64: Add SW reserved PTE/PMD bits Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 67/94] arm64: Add level-hinted TLB invalidation helper Marc Zyngier 2020-03-25 10:38 ` Zhenyu Ye 2020-03-25 11:28 ` Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 68/94] arm64: KVM: Add a level hint to __kvm_tlb_flush_vmid_ipa Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 69/94] arm64: KVM: Use TTL hint in when invalidating stage-2 translations Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 70/94] arm64: KVM: nv: Add include containing the VNCR_EL2 offsets Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 71/94] KVM: arm64: Introduce accessor for ctxt->sys_reg Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 72/94] KVM: arm64: sysreg: Use ctxt_sys_reg() instead of raw sys_regs access Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 73/94] KVM: arm64: sve: Use __vcpu_sys_reg() " Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 74/94] KVM: arm64: pauth: Use ctxt_sys_reg() " Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 75/94] KVM: arm64: debug: " Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 76/94] KVM: arm64: Add missing reset handlers for PMU emulation Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 77/94] KVM: arm64: nv: Move sysreg reset check to boot time Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 78/94] KVM: arm64: Map VNCR-capable registers to a separate page Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 79/94] KVM: arm64: nv: Move nested vgic state into the sysreg file Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 80/94] KVM: arm64: Use accessors for timer ctl/cval/offset Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 81/94] KVM: arm64: Add VNCR-capable timer accessors for arm64 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 82/94] KVM: arm64: Make struct kvm_regs userspace-only Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 83/94] KVM: arm64: VNCR-ize ELR_EL1 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 84/94] KVM: arm64: VNCR-ize SP_EL1 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 85/94] KVM: arm64: Disintegrate SPSR array Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 86/94] KVM: arm64: aarch32: Use __vcpu_sys_reg() instead of raw sys_regs access Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 87/94] KVM: arm64: VNCR-ize SPSR_EL1 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 88/94] KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 89/94] KVM: arm64: nv: Synchronize PSTATE early on exit Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 90/94] KVM: arm64: nv: Sync nested timer state with ARMv8.4 Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 91/94] KVM: arm64: nv: Allocate VNCR page when required Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 92/94] KVM: arm64: nv: Enable ARMv8.4-NV support Marc Zyngier 2020-02-11 17:49 ` [PATCH v2 93/94] KVM: arm64: nv: Fast-track 'InHost' exception returns Marc Zyngier 2020-02-11 17:49 ` Marc Zyngier [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200211174938.27809-95-maz@kernel.org \ --to=maz@kernel.org \ --cc=Dave.Martin@arm.com \ --cc=andre.przywara@arm.com \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.cs.columbia.edu \ --cc=linux-arm-kernel@lists.infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
KVM ARM Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/kvmarm/0 kvmarm/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 kvmarm kvmarm/ https://lore.kernel.org/kvmarm \ kvmarm@lists.cs.columbia.edu public-inbox-index kvmarm Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/edu.columbia.cs.lists.kvmarm AGPL code for this site: git clone https://public-inbox.org/public-inbox.git