* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
This series redesigns parts of KVM/ARM to optimize the performance on
VHE systems. The general approach is to try to do as little work as
possible when transitioning between the VM and the hypervisor. This has
the benefit of lower latency when waiting for interrupts and delivering
virtual interrupts, and reduces the overhead of emulating behavior and
I/O in the host kernel.
Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
that can be generally improved. We then add infrastructure to move more
logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
registers.
We then introduce a new world-switch function for VHE systems, which we
can tweak and optimize for VHE systems. To do that, we rework a lot of
the system register save/restore handling and emulation code that may
need access to system registers, so that we can defer as many system
register save/restore operations to vcpu_load and vcpu_put, and move
this logic out of the VHE world switch function.
We then optimize the configuration of traps. On non-VHE systems, both
the host and VM kernels run in EL1, but because the host kernel should
have full access to the underlying hardware, but the VM kernel should
not, we essentially make the host kernel more privileged than the VM
kernel despite them both running at the same privilege level by enabling
VE traps when entering the VM and disabling those traps when exiting the
VM. On VHE systems, the host kernel runs in EL2 and has full access to
the hardware (as much as allowed by secure side software), and is
unaffected by the trap configuration. That means we can configure the
traps for VMs running in EL1 once, and don't have to switch them on and
off for every entry/exit to/from the VM.
Finally, we improve our VGIC handling by moving all save/restore logic
out of the VHE world-switch, and we make it possible to truly only
evaluate if the AP list is empty and not do *any* VGIC work if that is
the case, and only do the minimal amount of work required in the course
of the VGIC processing when we have virtual interrupts in flight.
The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
[1], the level-triggered mapped interrupts support series [2], and the
first five patches of James' SDEI series [3], a single SVE patch that
moves the CPU ID reg trap setup out of the world-switch path, and v3 of
my vcpu load/put series [4].
I've given the patches a fair amount of testing on Thunder-X, Mustang,
Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
functionality on the Foundation model, running both 64-bit VMs and
32-bit VMs side-by-side and using both GICv3-on-GICv3 and
GICv2-on-GICv3.
The patches are also available in the vhe-optimize-v2 branch on my
kernel.org repository [5].
Changes since v1:
- Rebased on v4.15-rc1 and newer versions of other dependencies,
including the vcpu load/put approach taken for KVM.
- Addressed review comments from v1 (detailed changelogs are in the
individual patches).
Thanks,
-Christoffer
[1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
[2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
[3]: git://linux-arm.org/linux-jm.git sdei/v5/base
[4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
[5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
Christoffer Dall (35):
KVM: arm64: Avoid storing the vcpu pointer on the stack
KVM: arm64: Rework hyp_panic for VHE and non-VHE
KVM: arm/arm64: Get rid of vcpu->arch.irq_lines
KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
KVM: arm64: Defer restoring host VFP state to vcpu_put
KVM: arm64: Move debug dirty flag calculation out of world switch
KVM: arm64: Slightly improve debug save/restore functions
KVM: arm64: Improve debug register save/restore flow
KVM: arm64: Factor out fault info population and gic workarounds
KVM: arm64: Introduce VHE-specific kvm_vcpu_run
KVM: arm64: Remove kern_hyp_va() use in VHE switch function
KVM: arm64: Don't deactivate VM on VHE systems
KVM: arm64: Remove noop calls to timer save/restore from VHE switch
KVM: arm64: Move userspace system registers into separate function
KVM: arm64: Rewrite sysreg alternatives to static keys
KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore
functions
KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
KVM: arm64: Change 32-bit handling of VM system registers
KVM: arm64: Prepare to handle traps on deferred VM sysregs
KVM: arm64: Prepare to handle traps on deferred EL0 sysregs
KVM: arm64: Prepare to handle traps on remaining deferred EL1 sysregs
KVM: arm64: Prepare to handle traps on deferred AArch32 sysregs
KVM: arm64: Defer saving/restoring system registers to vcpu load/put
on VHE
KVM: arm64: Move common VHE/non-VHE trap config in separate functions
KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
KVM: arm64: Configure c15, PMU, and debug register traps on cpu
load/put for VHE
KVM: arm64: Separate activate_traps and deactive_traps for VHE and
non-VHE
KVM: arm/arm64: Get rid of vgic_elrsr
KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code
KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64
KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on
VHE
KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load
KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs
Shih-Wei Li (1):
KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag
arch/arm/include/asm/kvm_asm.h | 5 +-
arch/arm/include/asm/kvm_emulate.h | 25 +-
arch/arm/include/asm/kvm_host.h | 8 +-
arch/arm/include/asm/kvm_hyp.h | 4 +
arch/arm/kvm/emulate.c | 2 +-
arch/arm/kvm/hyp/Makefile | 1 -
arch/arm/kvm/hyp/switch.c | 16 +-
arch/arm64/include/asm/kvm_arm.h | 4 +-
arch/arm64/include/asm/kvm_asm.h | 19 +-
arch/arm64/include/asm/kvm_emulate.h | 64 +++-
arch/arm64/include/asm/kvm_host.h | 36 +-
arch/arm64/include/asm/kvm_hyp.h | 29 +-
arch/arm64/kernel/asm-offsets.c | 2 +
arch/arm64/kvm/debug.c | 5 +
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/debug-sr.c | 88 ++---
arch/arm64/kvm/hyp/entry.S | 9 +-
arch/arm64/kvm/hyp/hyp-entry.S | 41 ++-
arch/arm64/kvm/hyp/switch.c | 399 ++++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 179 ++++++++--
{virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c | 81 -----
arch/arm64/kvm/inject_fault.c | 21 +-
arch/arm64/kvm/sys_regs.c | 75 +++-
arch/arm64/kvm/sys_regs_generic_v8.c | 5 +-
include/kvm/arm_vgic.h | 2 -
virt/kvm/arm/aarch32.c | 22 +-
virt/kvm/arm/arm.c | 18 +-
virt/kvm/arm/hyp/timer-sr.c | 36 +-
virt/kvm/arm/hyp/vgic-v3-sr.c | 244 +++++++------
virt/kvm/arm/mmu.c | 6 +-
virt/kvm/arm/vgic/vgic-v2.c | 61 +++-
virt/kvm/arm/vgic/vgic-v3.c | 12 +-
virt/kvm/arm/vgic/vgic.c | 21 ++
virt/kvm/arm/vgic/vgic.h | 3 +
34 files changed, 978 insertions(+), 567 deletions(-)
rename {virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c (50%)
--
2.14.2
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
This series redesigns parts of KVM/ARM to optimize the performance on
VHE systems. The general approach is to try to do as little work as
possible when transitioning between the VM and the hypervisor. This has
the benefit of lower latency when waiting for interrupts and delivering
virtual interrupts, and reduces the overhead of emulating behavior and
I/O in the host kernel.
Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
that can be generally improved. We then add infrastructure to move more
logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
registers.
We then introduce a new world-switch function for VHE systems, which we
can tweak and optimize for VHE systems. To do that, we rework a lot of
the system register save/restore handling and emulation code that may
need access to system registers, so that we can defer as many system
register save/restore operations to vcpu_load and vcpu_put, and move
this logic out of the VHE world switch function.
We then optimize the configuration of traps. On non-VHE systems, both
the host and VM kernels run in EL1, but because the host kernel should
have full access to the underlying hardware, but the VM kernel should
not, we essentially make the host kernel more privileged than the VM
kernel despite them both running at the same privilege level by enabling
VE traps when entering the VM and disabling those traps when exiting the
VM. On VHE systems, the host kernel runs in EL2 and has full access to
the hardware (as much as allowed by secure side software), and is
unaffected by the trap configuration. That means we can configure the
traps for VMs running in EL1 once, and don't have to switch them on and
off for every entry/exit to/from the VM.
Finally, we improve our VGIC handling by moving all save/restore logic
out of the VHE world-switch, and we make it possible to truly only
evaluate if the AP list is empty and not do *any* VGIC work if that is
the case, and only do the minimal amount of work required in the course
of the VGIC processing when we have virtual interrupts in flight.
The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
[1], the level-triggered mapped interrupts support series [2], and the
first five patches of James' SDEI series [3], a single SVE patch that
moves the CPU ID reg trap setup out of the world-switch path, and v3 of
my vcpu load/put series [4].
I've given the patches a fair amount of testing on Thunder-X, Mustang,
Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
functionality on the Foundation model, running both 64-bit VMs and
32-bit VMs side-by-side and using both GICv3-on-GICv3 and
GICv2-on-GICv3.
The patches are also available in the vhe-optimize-v2 branch on my
kernel.org repository [5].
Changes since v1:
- Rebased on v4.15-rc1 and newer versions of other dependencies,
including the vcpu load/put approach taken for KVM.
- Addressed review comments from v1 (detailed changelogs are in the
individual patches).
Thanks,
-Christoffer
[1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
[2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
[3]: git://linux-arm.org/linux-jm.git sdei/v5/base
[4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
[5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
Christoffer Dall (35):
KVM: arm64: Avoid storing the vcpu pointer on the stack
KVM: arm64: Rework hyp_panic for VHE and non-VHE
KVM: arm/arm64: Get rid of vcpu->arch.irq_lines
KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
KVM: arm64: Defer restoring host VFP state to vcpu_put
KVM: arm64: Move debug dirty flag calculation out of world switch
KVM: arm64: Slightly improve debug save/restore functions
KVM: arm64: Improve debug register save/restore flow
KVM: arm64: Factor out fault info population and gic workarounds
KVM: arm64: Introduce VHE-specific kvm_vcpu_run
KVM: arm64: Remove kern_hyp_va() use in VHE switch function
KVM: arm64: Don't deactivate VM on VHE systems
KVM: arm64: Remove noop calls to timer save/restore from VHE switch
KVM: arm64: Move userspace system registers into separate function
KVM: arm64: Rewrite sysreg alternatives to static keys
KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore
functions
KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
KVM: arm64: Change 32-bit handling of VM system registers
KVM: arm64: Prepare to handle traps on deferred VM sysregs
KVM: arm64: Prepare to handle traps on deferred EL0 sysregs
KVM: arm64: Prepare to handle traps on remaining deferred EL1 sysregs
KVM: arm64: Prepare to handle traps on deferred AArch32 sysregs
KVM: arm64: Defer saving/restoring system registers to vcpu load/put
on VHE
KVM: arm64: Move common VHE/non-VHE trap config in separate functions
KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
KVM: arm64: Configure c15, PMU, and debug register traps on cpu
load/put for VHE
KVM: arm64: Separate activate_traps and deactive_traps for VHE and
non-VHE
KVM: arm/arm64: Get rid of vgic_elrsr
KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code
KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64
KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on
VHE
KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load
KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs
Shih-Wei Li (1):
KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag
arch/arm/include/asm/kvm_asm.h | 5 +-
arch/arm/include/asm/kvm_emulate.h | 25 +-
arch/arm/include/asm/kvm_host.h | 8 +-
arch/arm/include/asm/kvm_hyp.h | 4 +
arch/arm/kvm/emulate.c | 2 +-
arch/arm/kvm/hyp/Makefile | 1 -
arch/arm/kvm/hyp/switch.c | 16 +-
arch/arm64/include/asm/kvm_arm.h | 4 +-
arch/arm64/include/asm/kvm_asm.h | 19 +-
arch/arm64/include/asm/kvm_emulate.h | 64 +++-
arch/arm64/include/asm/kvm_host.h | 36 +-
arch/arm64/include/asm/kvm_hyp.h | 29 +-
arch/arm64/kernel/asm-offsets.c | 2 +
arch/arm64/kvm/debug.c | 5 +
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/debug-sr.c | 88 ++---
arch/arm64/kvm/hyp/entry.S | 9 +-
arch/arm64/kvm/hyp/hyp-entry.S | 41 ++-
arch/arm64/kvm/hyp/switch.c | 399 ++++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 179 ++++++++--
{virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c | 81 -----
arch/arm64/kvm/inject_fault.c | 21 +-
arch/arm64/kvm/sys_regs.c | 75 +++-
arch/arm64/kvm/sys_regs_generic_v8.c | 5 +-
include/kvm/arm_vgic.h | 2 -
virt/kvm/arm/aarch32.c | 22 +-
virt/kvm/arm/arm.c | 18 +-
virt/kvm/arm/hyp/timer-sr.c | 36 +-
virt/kvm/arm/hyp/vgic-v3-sr.c | 244 +++++++------
virt/kvm/arm/mmu.c | 6 +-
virt/kvm/arm/vgic/vgic-v2.c | 61 +++-
virt/kvm/arm/vgic/vgic-v3.c | 12 +-
virt/kvm/arm/vgic/vgic.c | 21 ++
virt/kvm/arm/vgic/vgic.h | 3 +
34 files changed, 978 insertions(+), 567 deletions(-)
rename {virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c (50%)
--
2.14.2
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:05 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: kvm, Ard Biesheuvel, Marc Zyngier, Shih-Wei Li
We already have the percpu area for the host cpu state, which points to
the VCPU, so there's no need to store the VCPU pointer on the stack on
every context switch. We can be a little more clever and just use
tpidr_el2 for the percpu offset and load the VCPU pointer from the host
context.
This does require us to calculate the percpu offset without including
the offset from the kernel mapping of the percpu array to the linaro
mapping of the array (which is what we store in tpidr_el1), because a
PC-relative generated address in EL2 is already giving us the hyp alias
of the linear mapping of a kernel address. We do this in
__cpu_init_hyp_mode() by using kvm_ksym_ref().
This change also requires us to have a scratch register, so we take the
chance to rearrange some of the el1_sync code to only look at the
vttbr_el2 to determine if this is a trap from the guest or an HVC from
the host. We do add an extra check to call the panic code if the kernel
is configured with debugging enabled and we saw a trap from the host
which wasn't an HVC, indicating that we left some EL2 trap configured by
mistake.
The code that accesses ESR_EL2 was previously using an alternative to
use the _EL1 accessor on VHE systems, but this was actually unnecessary
as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
accessor does the same thing on both systems.
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Use PC-relative addressing to access per-cpu variables instead of
using a load from the literal pool.
- Remove stale comments as pointed out by Marc
- Reworded the commit message as suggested by Drew
arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kvm/hyp/entry.S | 6 +-----
arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
arch/arm64/kvm/hyp/switch.c | 5 +----
arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
7 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index ab4d0a926043..33e0edc6f8be 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -33,6 +33,7 @@
#define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
#define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
+/* Translate a kernel address of @sym into its equivalent linear mapping */
#define kvm_ksym_ref(sym) \
({ \
void *val = &sym; \
@@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
#endif
+#ifdef __ASSEMBLY__
+.macro get_host_ctxt reg, tmp
+ adr_l \reg, kvm_host_cpu_state
+ mrs \tmp, tpidr_el2
+ add \reg, \reg, \tmp
+.endm
+
+.macro get_vcpu vcpu, ctxt
+ ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
+ kern_hyp_va \vcpu
+.endm
+
+#endif
+
#endif /* __ARM_KVM_ASM_H__ */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 7ee72b402907..af58deb6ec3c 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
+extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
+DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
+
static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
unsigned long hyp_stack_ptr,
unsigned long vector_ptr)
{
+ u64 tpidr_el2;
+
/*
* Call initialization code, and switch to the full blown HYP code.
* If the cpucaps haven't been finalized yet, something has gone very
@@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
*/
BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
__kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
+
+ /*
+ * Calculate the raw per-cpu offset without a translation from the
+ * kernel's mapping to the linear mapping, and store it in tpidr_el2
+ * so that we can use adr_l to access per-cpu variables in EL2.
+ */
+ tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
+ - (u64)kvm_ksym_ref(kvm_host_cpu_state);
+
+ kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
}
static inline void kvm_arch_hardware_unsetup(void) {}
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 71bf088f1e4b..612021dce84f 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -135,6 +135,7 @@ int main(void)
DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
+ DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
#endif
#ifdef CONFIG_CPU_PM
DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index 9a8ab5dddd9e..a360ac6e89e9 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -62,9 +62,6 @@ ENTRY(__guest_enter)
// Store the host regs
save_callee_saved_regs x1
- // Store host_ctxt and vcpu for use at exit time
- stp x1, x0, [sp, #-16]!
-
add x18, x0, #VCPU_CONTEXT
// Restore guest regs x0-x17
@@ -118,8 +115,7 @@ ENTRY(__guest_exit)
// Store the guest regs x19-x29, lr
save_callee_saved_regs x1
- // Restore the host_ctxt from the stack
- ldr x2, [sp], #16
+ get_host_ctxt x2, x3
// Now restore the host regs
restore_callee_saved_regs x2
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index e4f37b9dd47c..71b4cc92895e 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
el1_sync: // Guest trapped into EL2
stp x0, x1, [sp, #-16]!
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs x1, esr_el2
-alternative_else
- mrs x1, esr_el1
-alternative_endif
- lsr x0, x1, #ESR_ELx_EC_SHIFT
+ mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
+ cbnz x1, el1_trap // from the guest
- cmp x0, #ESR_ELx_EC_HVC64
- b.ne el1_trap
-
- mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
- cbnz x1, el1_trap // called HVC
+#ifdef CONFIG_DEBUG
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
+ cmp x0, #ESR_ELx_EC_HVC64
+ b.ne __hyp_panic
+#endif
/* Here, we're pretty sure the host called HVC. */
ldp x0, x1, [sp], #16
@@ -101,10 +98,15 @@ alternative_endif
eret
el1_trap:
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
+
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
/*
* x0: ESR_EC
+ * x1: vcpu pointer
*/
- ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
/*
* We trap the first access to the FP/SIMD to save the host context
@@ -122,13 +124,15 @@ alternative_else_nop_endif
el1_irq:
stp x0, x1, [sp, #-16]!
- ldr x1, [sp, #16 + 8]
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
mov x0, #ARM_EXCEPTION_IRQ
b __guest_exit
el1_error:
stp x0, x1, [sp, #-16]!
- ldr x1, [sp, #16 + 8]
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
mov x0, #ARM_EXCEPTION_EL1_SERROR
b __guest_exit
@@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
ENDPROC(__hyp_do_panic)
ENTRY(__hyp_panic)
- /*
- * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
- * not be accessible by this address from EL2, hyp_panic() converts
- * it with kern_hyp_va() before use.
- */
- ldr x0, =kvm_host_cpu_state
- mrs x1, tpidr_el2
- add x0, x0, x1
+ get_host_ctxt x0, x1
b hyp_panic
ENDPROC(__hyp_panic)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f7307b6b42f0..6fcb37e220b5 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
__hyp_call_panic_nvhe, __hyp_call_panic_vhe,
ARM64_HAS_VIRT_HOST_EXTN);
-void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
+void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
{
struct kvm_vcpu *vcpu = NULL;
@@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
u64 par = read_sysreg(par_el1);
if (read_sysreg(vttbr_el2)) {
- struct kvm_cpu_context *host_ctxt;
-
- host_ctxt = kern_hyp_va(__host_ctxt);
vcpu = host_ctxt->__hyp_running_vcpu;
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index c54cc2afb92b..e19d89cabf2a 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
}
+
+void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
+{
+ asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
+}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
We already have the percpu area for the host cpu state, which points to
the VCPU, so there's no need to store the VCPU pointer on the stack on
every context switch. We can be a little more clever and just use
tpidr_el2 for the percpu offset and load the VCPU pointer from the host
context.
This does require us to calculate the percpu offset without including
the offset from the kernel mapping of the percpu array to the linaro
mapping of the array (which is what we store in tpidr_el1), because a
PC-relative generated address in EL2 is already giving us the hyp alias
of the linear mapping of a kernel address. We do this in
__cpu_init_hyp_mode() by using kvm_ksym_ref().
This change also requires us to have a scratch register, so we take the
chance to rearrange some of the el1_sync code to only look at the
vttbr_el2 to determine if this is a trap from the guest or an HVC from
the host. We do add an extra check to call the panic code if the kernel
is configured with debugging enabled and we saw a trap from the host
which wasn't an HVC, indicating that we left some EL2 trap configured by
mistake.
The code that accesses ESR_EL2 was previously using an alternative to
use the _EL1 accessor on VHE systems, but this was actually unnecessary
as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
accessor does the same thing on both systems.
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Use PC-relative addressing to access per-cpu variables instead of
using a load from the literal pool.
- Remove stale comments as pointed out by Marc
- Reworded the commit message as suggested by Drew
arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kvm/hyp/entry.S | 6 +-----
arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
arch/arm64/kvm/hyp/switch.c | 5 +----
arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
7 files changed, 57 insertions(+), 31 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index ab4d0a926043..33e0edc6f8be 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -33,6 +33,7 @@
#define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
#define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
+/* Translate a kernel address of @sym into its equivalent linear mapping */
#define kvm_ksym_ref(sym) \
({ \
void *val = &sym; \
@@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
#endif
+#ifdef __ASSEMBLY__
+.macro get_host_ctxt reg, tmp
+ adr_l \reg, kvm_host_cpu_state
+ mrs \tmp, tpidr_el2
+ add \reg, \reg, \tmp
+.endm
+
+.macro get_vcpu vcpu, ctxt
+ ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
+ kern_hyp_va \vcpu
+.endm
+
+#endif
+
#endif /* __ARM_KVM_ASM_H__ */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 7ee72b402907..af58deb6ec3c 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
+extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
+DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
+
static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
unsigned long hyp_stack_ptr,
unsigned long vector_ptr)
{
+ u64 tpidr_el2;
+
/*
* Call initialization code, and switch to the full blown HYP code.
* If the cpucaps haven't been finalized yet, something has gone very
@@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
*/
BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
__kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
+
+ /*
+ * Calculate the raw per-cpu offset without a translation from the
+ * kernel's mapping to the linear mapping, and store it in tpidr_el2
+ * so that we can use adr_l to access per-cpu variables in EL2.
+ */
+ tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
+ - (u64)kvm_ksym_ref(kvm_host_cpu_state);
+
+ kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
}
static inline void kvm_arch_hardware_unsetup(void) {}
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 71bf088f1e4b..612021dce84f 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -135,6 +135,7 @@ int main(void)
DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
+ DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
#endif
#ifdef CONFIG_CPU_PM
DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index 9a8ab5dddd9e..a360ac6e89e9 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -62,9 +62,6 @@ ENTRY(__guest_enter)
// Store the host regs
save_callee_saved_regs x1
- // Store host_ctxt and vcpu for use@exit time
- stp x1, x0, [sp, #-16]!
-
add x18, x0, #VCPU_CONTEXT
// Restore guest regs x0-x17
@@ -118,8 +115,7 @@ ENTRY(__guest_exit)
// Store the guest regs x19-x29, lr
save_callee_saved_regs x1
- // Restore the host_ctxt from the stack
- ldr x2, [sp], #16
+ get_host_ctxt x2, x3
// Now restore the host regs
restore_callee_saved_regs x2
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index e4f37b9dd47c..71b4cc92895e 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
el1_sync: // Guest trapped into EL2
stp x0, x1, [sp, #-16]!
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs x1, esr_el2
-alternative_else
- mrs x1, esr_el1
-alternative_endif
- lsr x0, x1, #ESR_ELx_EC_SHIFT
+ mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
+ cbnz x1, el1_trap // from the guest
- cmp x0, #ESR_ELx_EC_HVC64
- b.ne el1_trap
-
- mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
- cbnz x1, el1_trap // called HVC
+#ifdef CONFIG_DEBUG
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
+ cmp x0, #ESR_ELx_EC_HVC64
+ b.ne __hyp_panic
+#endif
/* Here, we're pretty sure the host called HVC. */
ldp x0, x1, [sp], #16
@@ -101,10 +98,15 @@ alternative_endif
eret
el1_trap:
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
+
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
/*
* x0: ESR_EC
+ * x1: vcpu pointer
*/
- ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
/*
* We trap the first access to the FP/SIMD to save the host context
@@ -122,13 +124,15 @@ alternative_else_nop_endif
el1_irq:
stp x0, x1, [sp, #-16]!
- ldr x1, [sp, #16 + 8]
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
mov x0, #ARM_EXCEPTION_IRQ
b __guest_exit
el1_error:
stp x0, x1, [sp, #-16]!
- ldr x1, [sp, #16 + 8]
+ get_host_ctxt x0, x1
+ get_vcpu x1, x0
mov x0, #ARM_EXCEPTION_EL1_SERROR
b __guest_exit
@@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
ENDPROC(__hyp_do_panic)
ENTRY(__hyp_panic)
- /*
- * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
- * not be accessible by this address from EL2, hyp_panic() converts
- * it with kern_hyp_va() before use.
- */
- ldr x0, =kvm_host_cpu_state
- mrs x1, tpidr_el2
- add x0, x0, x1
+ get_host_ctxt x0, x1
b hyp_panic
ENDPROC(__hyp_panic)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f7307b6b42f0..6fcb37e220b5 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
__hyp_call_panic_nvhe, __hyp_call_panic_vhe,
ARM64_HAS_VIRT_HOST_EXTN);
-void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
+void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
{
struct kvm_vcpu *vcpu = NULL;
@@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
u64 par = read_sysreg(par_el1);
if (read_sysreg(vttbr_el2)) {
- struct kvm_cpu_context *host_ctxt;
-
- host_ctxt = kern_hyp_va(__host_ctxt);
vcpu = host_ctxt->__hyp_running_vcpu;
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index c54cc2afb92b..e19d89cabf2a 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
}
+
+void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
+{
+ asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
+}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 02/36] KVM: arm64: Rework hyp_panic for VHE and non-VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:05 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
VHE actually doesn't rely on clearing the VTTBR when returning to the
host kernel, and that is the current key mechanism of hyp_panic to
figure out how to attempt to return to a state good enough to print a
panic statement.
Therefore, we split the hyp_panic function into two functions, a VHE and
a non-VHE, keeping the non-VHE version intact, but changing the VHE
behavior.
The vttbr_el2 check on VHE doesn't really make that much sense, because
the only situation where we can get here on VHE is when the hypervisor
assembly code actually called into hyp_panic, which only happens when
VBAR_EL2 has been set to the KVM exception vectors. On VHE, we can
always safely disable the traps and restore the host registers at this
point, so we simply do that unconditionally and call into the panic
function directly.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fixed typos in the commit message
- Still use the generic __deactivte_traps() function in the hyp panic
code until we rework that logic later.
arch/arm64/kvm/hyp/switch.c | 42 +++++++++++++++++++++++-------------------
1 file changed, 23 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 6fcb37e220b5..71700ecee308 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -419,10 +419,20 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
static const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
- struct kvm_vcpu *vcpu)
+ struct kvm_cpu_context *__host_ctxt)
{
+ struct kvm_vcpu *vcpu;
unsigned long str_va;
+ vcpu = __host_ctxt->__hyp_running_vcpu;
+
+ if (read_sysreg(vttbr_el2)) {
+ __timer_disable_traps(vcpu);
+ __deactivate_traps(vcpu);
+ __deactivate_vm(vcpu);
+ __sysreg_restore_host_state(__host_ctxt);
+ }
+
/*
* Force the panic string to be loaded from the literal pool,
* making sure it is a kernel address and not a PC-relative
@@ -436,37 +446,31 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
read_sysreg(hpfar_el2), par, vcpu);
}
-static void __hyp_text __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
- struct kvm_vcpu *vcpu)
+static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
+ struct kvm_cpu_context *host_ctxt)
{
+ struct kvm_vcpu *vcpu;
+ vcpu = host_ctxt->__hyp_running_vcpu;
+
+ __deactivate_traps(vcpu);
+ __sysreg_restore_host_state(host_ctxt);
+
panic(__hyp_panic_string,
spsr, elr,
read_sysreg_el2(esr), read_sysreg_el2(far),
read_sysreg(hpfar_el2), par, vcpu);
}
-static hyp_alternate_select(__hyp_call_panic,
- __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
{
- struct kvm_vcpu *vcpu = NULL;
-
u64 spsr = read_sysreg_el2(spsr);
u64 elr = read_sysreg_el2(elr);
u64 par = read_sysreg(par_el1);
- if (read_sysreg(vttbr_el2)) {
- vcpu = host_ctxt->__hyp_running_vcpu;
- __timer_disable_traps(vcpu);
- __deactivate_traps(vcpu);
- __deactivate_vm(vcpu);
- __sysreg_restore_host_state(host_ctxt);
- }
-
- /* Call panic for real */
- __hyp_call_panic()(spsr, elr, par, vcpu);
+ if (!has_vhe())
+ __hyp_call_panic_nvhe(spsr, elr, par, host_ctxt);
+ else
+ __hyp_call_panic_vhe(spsr, elr, par, host_ctxt);
unreachable();
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 02/36] KVM: arm64: Rework hyp_panic for VHE and non-VHE
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
VHE actually doesn't rely on clearing the VTTBR when returning to the
host kernel, and that is the current key mechanism of hyp_panic to
figure out how to attempt to return to a state good enough to print a
panic statement.
Therefore, we split the hyp_panic function into two functions, a VHE and
a non-VHE, keeping the non-VHE version intact, but changing the VHE
behavior.
The vttbr_el2 check on VHE doesn't really make that much sense, because
the only situation where we can get here on VHE is when the hypervisor
assembly code actually called into hyp_panic, which only happens when
VBAR_EL2 has been set to the KVM exception vectors. On VHE, we can
always safely disable the traps and restore the host registers at this
point, so we simply do that unconditionally and call into the panic
function directly.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fixed typos in the commit message
- Still use the generic __deactivte_traps() function in the hyp panic
code until we rework that logic later.
arch/arm64/kvm/hyp/switch.c | 42 +++++++++++++++++++++++-------------------
1 file changed, 23 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 6fcb37e220b5..71700ecee308 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -419,10 +419,20 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
static const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
- struct kvm_vcpu *vcpu)
+ struct kvm_cpu_context *__host_ctxt)
{
+ struct kvm_vcpu *vcpu;
unsigned long str_va;
+ vcpu = __host_ctxt->__hyp_running_vcpu;
+
+ if (read_sysreg(vttbr_el2)) {
+ __timer_disable_traps(vcpu);
+ __deactivate_traps(vcpu);
+ __deactivate_vm(vcpu);
+ __sysreg_restore_host_state(__host_ctxt);
+ }
+
/*
* Force the panic string to be loaded from the literal pool,
* making sure it is a kernel address and not a PC-relative
@@ -436,37 +446,31 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
read_sysreg(hpfar_el2), par, vcpu);
}
-static void __hyp_text __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
- struct kvm_vcpu *vcpu)
+static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
+ struct kvm_cpu_context *host_ctxt)
{
+ struct kvm_vcpu *vcpu;
+ vcpu = host_ctxt->__hyp_running_vcpu;
+
+ __deactivate_traps(vcpu);
+ __sysreg_restore_host_state(host_ctxt);
+
panic(__hyp_panic_string,
spsr, elr,
read_sysreg_el2(esr), read_sysreg_el2(far),
read_sysreg(hpfar_el2), par, vcpu);
}
-static hyp_alternate_select(__hyp_call_panic,
- __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
{
- struct kvm_vcpu *vcpu = NULL;
-
u64 spsr = read_sysreg_el2(spsr);
u64 elr = read_sysreg_el2(elr);
u64 par = read_sysreg(par_el1);
- if (read_sysreg(vttbr_el2)) {
- vcpu = host_ctxt->__hyp_running_vcpu;
- __timer_disable_traps(vcpu);
- __deactivate_traps(vcpu);
- __deactivate_vm(vcpu);
- __sysreg_restore_host_state(host_ctxt);
- }
-
- /* Call panic for real */
- __hyp_call_panic()(spsr, elr, par, vcpu);
+ if (!has_vhe())
+ __hyp_call_panic_nvhe(spsr, elr, par, host_ctxt);
+ else
+ __hyp_call_panic_vhe(spsr, elr, par, host_ctxt);
unreachable();
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 03/36] KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:05 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
From: Shih-Wei Li <shihwei@cs.columbia.edu>
We always set the IMO and FMO bits in the HCR_EL2 when running the
guest, regardless if we use the vgic or not. By moving these flags to
HCR_GUEST_FLAGS we can avoid one of the extra save/restore operations of
HCR_EL2 in the world switch code, and we can also soon get rid of the
other one.
This is safe, because even though the IMO and FMO bits control both
taking the interrupts to EL2 and remapping ICC_*_EL1 to ICV_*_EL1
executed at EL1, as long as we ensure that these bits are clear when
running the EL1 host, as defined in the HCR_HOST_[VHE_]FLAGS, we're OK.
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Shih-Wei Li <shihwei@cs.columbia.edu>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_arm.h | 4 ++--
arch/arm64/kvm/hyp/switch.c | 3 ---
2 files changed, 2 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 715d395ef45b..656deeb17bf2 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -79,9 +79,9 @@
*/
#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
HCR_TVM | HCR_BSU_IS | HCR_FB | HCR_TAC | \
- HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW)
+ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | \
+ HCR_FMO | HCR_IMO)
#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
-#define HCR_INT_OVERRIDE (HCR_FMO | HCR_IMO)
#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)
/* TCR_EL2 Registers bits */
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 71700ecee308..f6189d08753e 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -167,8 +167,6 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
__vgic_v3_save_state(vcpu);
else
__vgic_v2_save_state(vcpu);
-
- write_sysreg(read_sysreg(hcr_el2) & ~HCR_INT_OVERRIDE, hcr_el2);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
@@ -176,7 +174,6 @@ static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
u64 val;
val = read_sysreg(hcr_el2);
- val |= HCR_INT_OVERRIDE;
val |= vcpu->arch.irq_lines;
write_sysreg(val, hcr_el2);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 03/36] KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
From: Shih-Wei Li <shihwei@cs.columbia.edu>
We always set the IMO and FMO bits in the HCR_EL2 when running the
guest, regardless if we use the vgic or not. By moving these flags to
HCR_GUEST_FLAGS we can avoid one of the extra save/restore operations of
HCR_EL2 in the world switch code, and we can also soon get rid of the
other one.
This is safe, because even though the IMO and FMO bits control both
taking the interrupts to EL2 and remapping ICC_*_EL1 to ICV_*_EL1
executed at EL1, as long as we ensure that these bits are clear when
running the EL1 host, as defined in the HCR_HOST_[VHE_]FLAGS, we're OK.
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Shih-Wei Li <shihwei@cs.columbia.edu>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_arm.h | 4 ++--
arch/arm64/kvm/hyp/switch.c | 3 ---
2 files changed, 2 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 715d395ef45b..656deeb17bf2 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -79,9 +79,9 @@
*/
#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
HCR_TVM | HCR_BSU_IS | HCR_FB | HCR_TAC | \
- HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW)
+ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | \
+ HCR_FMO | HCR_IMO)
#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
-#define HCR_INT_OVERRIDE (HCR_FMO | HCR_IMO)
#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)
/* TCR_EL2 Registers bits */
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 71700ecee308..f6189d08753e 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -167,8 +167,6 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
__vgic_v3_save_state(vcpu);
else
__vgic_v2_save_state(vcpu);
-
- write_sysreg(read_sysreg(hcr_el2) & ~HCR_INT_OVERRIDE, hcr_el2);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
@@ -176,7 +174,6 @@ static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
u64 val;
val = read_sysreg(hcr_el2);
- val |= HCR_INT_OVERRIDE;
val |= vcpu->arch.irq_lines;
write_sysreg(val, hcr_el2);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 04/36] KVM: arm/arm64: Get rid of vcpu->arch.irq_lines
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:05 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
We currently have a separate read-modify-write of the HCR_EL2 on entry
to the guest for the sole purpose of setting the VF and VI bits, if set.
Since this is most rarely the case (only when using userspace IRQ chip
and interrupts are in flight), let's get rid of this operation and
instead modify the bits in the vcpu->arch.hcr[_el2] directly when
needed.
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Julien Thierry <julien.thierry@arm.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_emulate.h | 9 ++-------
arch/arm/include/asm/kvm_host.h | 3 ---
arch/arm/kvm/emulate.c | 2 +-
arch/arm/kvm/hyp/switch.c | 2 +-
arch/arm64/include/asm/kvm_emulate.h | 9 ++-------
arch/arm64/include/asm/kvm_host.h | 3 ---
arch/arm64/kvm/hyp/switch.c | 6 ------
arch/arm64/kvm/inject_fault.c | 2 +-
virt/kvm/arm/arm.c | 11 ++++++-----
virt/kvm/arm/mmu.c | 6 +++---
10 files changed, 16 insertions(+), 37 deletions(-)
diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h
index 3d22eb87f919..d5e1b8bf6422 100644
--- a/arch/arm/include/asm/kvm_emulate.h
+++ b/arch/arm/include/asm/kvm_emulate.h
@@ -92,14 +92,9 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
vcpu->arch.hcr = HCR_GUEST_MASK;
}
-static inline unsigned long vcpu_get_hcr(const struct kvm_vcpu *vcpu)
+static inline unsigned long *vcpu_hcr(const struct kvm_vcpu *vcpu)
{
- return vcpu->arch.hcr;
-}
-
-static inline void vcpu_set_hcr(struct kvm_vcpu *vcpu, unsigned long hcr)
-{
- vcpu->arch.hcr = hcr;
+ return (unsigned long *)&vcpu->arch.hcr;
}
static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index a9f7d3f47134..b92d2bcb3c08 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -153,9 +153,6 @@ struct kvm_vcpu_arch {
/* HYP trapping configuration */
u32 hcr;
- /* Interrupt related fields */
- u32 irq_lines; /* IRQ and FIQ levels */
-
/* Exception Information */
struct kvm_vcpu_fault_info fault;
diff --git a/arch/arm/kvm/emulate.c b/arch/arm/kvm/emulate.c
index cdff963f133a..fa501bf437f3 100644
--- a/arch/arm/kvm/emulate.c
+++ b/arch/arm/kvm/emulate.c
@@ -174,5 +174,5 @@ unsigned long *vcpu_spsr(struct kvm_vcpu *vcpu)
*/
void kvm_inject_vabt(struct kvm_vcpu *vcpu)
{
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) | HCR_VA);
+ *vcpu_hcr(vcpu) |= HCR_VA;
}
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 330c9ce34ba5..c3b9799e2e13 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -43,7 +43,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu, u32 *fpexc_host)
isb();
}
- write_sysreg(vcpu->arch.hcr | vcpu->arch.irq_lines, HCR);
+ write_sysreg(vcpu->arch.hcr, HCR);
/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
write_sysreg(HSTR_T(15), HSTR);
write_sysreg(HCPTR_TTA | HCPTR_TCP(10) | HCPTR_TCP(11), HCPTR);
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 8ff5aef44656..b36aaa1fe332 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -62,14 +62,9 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
vcpu->arch.hcr_el2 |= HCR_TID3;
}
-static inline unsigned long vcpu_get_hcr(struct kvm_vcpu *vcpu)
+static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
{
- return vcpu->arch.hcr_el2;
-}
-
-static inline void vcpu_set_hcr(struct kvm_vcpu *vcpu, unsigned long hcr)
-{
- vcpu->arch.hcr_el2 = hcr;
+ return (unsigned long *)&vcpu->arch.hcr_el2;
}
static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index af58deb6ec3c..d85c1e6ec53f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -267,9 +267,6 @@ struct kvm_vcpu_arch {
/* IO related fields */
struct kvm_decode mmio_decode;
- /* Interrupt related fields */
- u64 irq_lines; /* IRQ and FIQ levels */
-
/* Cache some mmu pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f6189d08753e..11ec1c6f3b84 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -171,12 +171,6 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- u64 val;
-
- val = read_sysreg(hcr_el2);
- val |= vcpu->arch.irq_lines;
- write_sysreg(val, hcr_el2);
-
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
else
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index 8ecbcb40e317..2d38ede2eff0 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -173,5 +173,5 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)
*/
void kvm_inject_vabt(struct kvm_vcpu *vcpu)
{
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) | HCR_VSE);
+ *vcpu_hcr(vcpu) |= HCR_VSE;
}
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 0065f4adc251..e5a5cf0d17fe 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -415,7 +415,8 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
*/
int kvm_arch_vcpu_runnable(struct kvm_vcpu *v)
{
- return ((!!v->arch.irq_lines || kvm_vgic_vcpu_pending_irq(v))
+ bool irq_lines = *vcpu_hcr(v) & (HCR_VI | HCR_VF);
+ return ((irq_lines || kvm_vgic_vcpu_pending_irq(v))
&& !v->arch.power_off && !v->arch.pause);
}
@@ -786,18 +787,18 @@ static int vcpu_interrupt_line(struct kvm_vcpu *vcpu, int number, bool level)
{
int bit_index;
bool set;
- unsigned long *ptr;
+ unsigned long *hcr;
if (number == KVM_ARM_IRQ_CPU_IRQ)
bit_index = __ffs(HCR_VI);
else /* KVM_ARM_IRQ_CPU_FIQ */
bit_index = __ffs(HCR_VF);
- ptr = (unsigned long *)&vcpu->arch.irq_lines;
+ hcr = vcpu_hcr(vcpu);
if (level)
- set = test_and_set_bit(bit_index, ptr);
+ set = test_and_set_bit(bit_index, hcr);
else
- set = test_and_clear_bit(bit_index, ptr);
+ set = test_and_clear_bit(bit_index, hcr);
/*
* If we didn't change anything, no need to wake up or kick other CPUs
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b36945d49986..d93d56d4cc5b 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1987,7 +1987,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
*/
void kvm_set_way_flush(struct kvm_vcpu *vcpu)
{
- unsigned long hcr = vcpu_get_hcr(vcpu);
+ unsigned long hcr = *vcpu_hcr(vcpu);
/*
* If this is the first time we do a S/W operation
@@ -2002,7 +2002,7 @@ void kvm_set_way_flush(struct kvm_vcpu *vcpu)
trace_kvm_set_way_flush(*vcpu_pc(vcpu),
vcpu_has_cache_enabled(vcpu));
stage2_flush_vm(vcpu->kvm);
- vcpu_set_hcr(vcpu, hcr | HCR_TVM);
+ *vcpu_hcr(vcpu) = hcr | HCR_TVM;
}
}
@@ -2020,7 +2020,7 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
/* Caches are now on, stop trapping VM ops (until a S/W op) */
if (now_enabled)
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) & ~HCR_TVM);
+ *vcpu_hcr(vcpu) &= ~HCR_TVM;
trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 04/36] KVM: arm/arm64: Get rid of vcpu->arch.irq_lines
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
We currently have a separate read-modify-write of the HCR_EL2 on entry
to the guest for the sole purpose of setting the VF and VI bits, if set.
Since this is most rarely the case (only when using userspace IRQ chip
and interrupts are in flight), let's get rid of this operation and
instead modify the bits in the vcpu->arch.hcr[_el2] directly when
needed.
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Julien Thierry <julien.thierry@arm.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_emulate.h | 9 ++-------
arch/arm/include/asm/kvm_host.h | 3 ---
arch/arm/kvm/emulate.c | 2 +-
arch/arm/kvm/hyp/switch.c | 2 +-
arch/arm64/include/asm/kvm_emulate.h | 9 ++-------
arch/arm64/include/asm/kvm_host.h | 3 ---
arch/arm64/kvm/hyp/switch.c | 6 ------
arch/arm64/kvm/inject_fault.c | 2 +-
virt/kvm/arm/arm.c | 11 ++++++-----
virt/kvm/arm/mmu.c | 6 +++---
10 files changed, 16 insertions(+), 37 deletions(-)
diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h
index 3d22eb87f919..d5e1b8bf6422 100644
--- a/arch/arm/include/asm/kvm_emulate.h
+++ b/arch/arm/include/asm/kvm_emulate.h
@@ -92,14 +92,9 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
vcpu->arch.hcr = HCR_GUEST_MASK;
}
-static inline unsigned long vcpu_get_hcr(const struct kvm_vcpu *vcpu)
+static inline unsigned long *vcpu_hcr(const struct kvm_vcpu *vcpu)
{
- return vcpu->arch.hcr;
-}
-
-static inline void vcpu_set_hcr(struct kvm_vcpu *vcpu, unsigned long hcr)
-{
- vcpu->arch.hcr = hcr;
+ return (unsigned long *)&vcpu->arch.hcr;
}
static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index a9f7d3f47134..b92d2bcb3c08 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -153,9 +153,6 @@ struct kvm_vcpu_arch {
/* HYP trapping configuration */
u32 hcr;
- /* Interrupt related fields */
- u32 irq_lines; /* IRQ and FIQ levels */
-
/* Exception Information */
struct kvm_vcpu_fault_info fault;
diff --git a/arch/arm/kvm/emulate.c b/arch/arm/kvm/emulate.c
index cdff963f133a..fa501bf437f3 100644
--- a/arch/arm/kvm/emulate.c
+++ b/arch/arm/kvm/emulate.c
@@ -174,5 +174,5 @@ unsigned long *vcpu_spsr(struct kvm_vcpu *vcpu)
*/
void kvm_inject_vabt(struct kvm_vcpu *vcpu)
{
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) | HCR_VA);
+ *vcpu_hcr(vcpu) |= HCR_VA;
}
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 330c9ce34ba5..c3b9799e2e13 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -43,7 +43,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu, u32 *fpexc_host)
isb();
}
- write_sysreg(vcpu->arch.hcr | vcpu->arch.irq_lines, HCR);
+ write_sysreg(vcpu->arch.hcr, HCR);
/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
write_sysreg(HSTR_T(15), HSTR);
write_sysreg(HCPTR_TTA | HCPTR_TCP(10) | HCPTR_TCP(11), HCPTR);
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 8ff5aef44656..b36aaa1fe332 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -62,14 +62,9 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
vcpu->arch.hcr_el2 |= HCR_TID3;
}
-static inline unsigned long vcpu_get_hcr(struct kvm_vcpu *vcpu)
+static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
{
- return vcpu->arch.hcr_el2;
-}
-
-static inline void vcpu_set_hcr(struct kvm_vcpu *vcpu, unsigned long hcr)
-{
- vcpu->arch.hcr_el2 = hcr;
+ return (unsigned long *)&vcpu->arch.hcr_el2;
}
static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index af58deb6ec3c..d85c1e6ec53f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -267,9 +267,6 @@ struct kvm_vcpu_arch {
/* IO related fields */
struct kvm_decode mmio_decode;
- /* Interrupt related fields */
- u64 irq_lines; /* IRQ and FIQ levels */
-
/* Cache some mmu pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f6189d08753e..11ec1c6f3b84 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -171,12 +171,6 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- u64 val;
-
- val = read_sysreg(hcr_el2);
- val |= vcpu->arch.irq_lines;
- write_sysreg(val, hcr_el2);
-
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
else
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index 8ecbcb40e317..2d38ede2eff0 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -173,5 +173,5 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)
*/
void kvm_inject_vabt(struct kvm_vcpu *vcpu)
{
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) | HCR_VSE);
+ *vcpu_hcr(vcpu) |= HCR_VSE;
}
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 0065f4adc251..e5a5cf0d17fe 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -415,7 +415,8 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
*/
int kvm_arch_vcpu_runnable(struct kvm_vcpu *v)
{
- return ((!!v->arch.irq_lines || kvm_vgic_vcpu_pending_irq(v))
+ bool irq_lines = *vcpu_hcr(v) & (HCR_VI | HCR_VF);
+ return ((irq_lines || kvm_vgic_vcpu_pending_irq(v))
&& !v->arch.power_off && !v->arch.pause);
}
@@ -786,18 +787,18 @@ static int vcpu_interrupt_line(struct kvm_vcpu *vcpu, int number, bool level)
{
int bit_index;
bool set;
- unsigned long *ptr;
+ unsigned long *hcr;
if (number == KVM_ARM_IRQ_CPU_IRQ)
bit_index = __ffs(HCR_VI);
else /* KVM_ARM_IRQ_CPU_FIQ */
bit_index = __ffs(HCR_VF);
- ptr = (unsigned long *)&vcpu->arch.irq_lines;
+ hcr = vcpu_hcr(vcpu);
if (level)
- set = test_and_set_bit(bit_index, ptr);
+ set = test_and_set_bit(bit_index, hcr);
else
- set = test_and_clear_bit(bit_index, ptr);
+ set = test_and_clear_bit(bit_index, hcr);
/*
* If we didn't change anything, no need to wake up or kick other CPUs
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b36945d49986..d93d56d4cc5b 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1987,7 +1987,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
*/
void kvm_set_way_flush(struct kvm_vcpu *vcpu)
{
- unsigned long hcr = vcpu_get_hcr(vcpu);
+ unsigned long hcr = *vcpu_hcr(vcpu);
/*
* If this is the first time we do a S/W operation
@@ -2002,7 +2002,7 @@ void kvm_set_way_flush(struct kvm_vcpu *vcpu)
trace_kvm_set_way_flush(*vcpu_pc(vcpu),
vcpu_has_cache_enabled(vcpu));
stage2_flush_vm(vcpu->kvm);
- vcpu_set_hcr(vcpu, hcr | HCR_TVM);
+ *vcpu_hcr(vcpu) = hcr | HCR_TVM;
}
}
@@ -2020,7 +2020,7 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
/* Caches are now on, stop trapping VM ops (until a S/W op) */
if (now_enabled)
- vcpu_set_hcr(vcpu, vcpu_get_hcr(vcpu) & ~HCR_TVM);
+ *vcpu_hcr(vcpu) &= ~HCR_TVM;
trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 05/36] KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:05 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
As we are about to move a bunch of save/restore logic for VHE kernels to
the load and put functions, we need some infrastructure to do this.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Reworded comments as suggested by Drew
arch/arm/include/asm/kvm_host.h | 3 +++
arch/arm64/include/asm/kvm_host.h | 3 +++
arch/arm64/kvm/hyp/sysreg-sr.c | 30 ++++++++++++++++++++++++++++++
virt/kvm/arm/arm.c | 2 ++
4 files changed, 38 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index b92d2bcb3c08..8fce576199e0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -298,4 +298,7 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,
/* All host FP/SIMD state is restored on guest exit, so nothing to save: */
static inline void kvm_fpsimd_flush_cpu_state(void) {}
+static inline void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu) {}
+static inline void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu) {}
+
#endif /* __ARM_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index d85c1e6ec53f..20fab9194794 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -410,4 +410,7 @@ static inline void kvm_fpsimd_flush_cpu_state(void)
sve_flush_cpu_state();
}
+void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu);
+void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu);
+
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index e19d89cabf2a..cbbcd6f410a8 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -184,6 +184,36 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
}
+/**
+ * kvm_vcpu_load_sysregs - Load guest system registers to the physical CPU
+ *
+ * @vcpu: The VCPU pointer
+ *
+ * Load system registers that do not affect the host's execution, for
+ * example EL1 system registers on a VHE system where the host kernel
+ * runs at EL2. This function is called from KVM's vcpu_load() function
+ * and loading system register state early avoids having to load them on
+ * every entry to the VM.
+ */
+void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
+{
+}
+
+/**
+ * kvm_vcpu_put_sysregs - Restore host system registers to the physical CPU
+ *
+ * @vcpu: The VCPU pointer
+ *
+ * Save guest system registers that do not affect the host's execution, for
+ * example EL1 system registers on a VHE system where the host kernel
+ * runs at EL2. This function is called from KVM's vcpu_put() function
+ * and deferring saving system register state until we're no longer running the
+ * VCPU avoids having to save them on every exit from the VM.
+ */
+void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
+{
+}
+
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
{
asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index e5a5cf0d17fe..3e10343374a1 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -357,10 +357,12 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
kvm_arm_set_running_vcpu(vcpu);
kvm_vgic_load(vcpu);
kvm_timer_vcpu_load(vcpu);
+ kvm_vcpu_load_sysregs(vcpu);
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
+ kvm_vcpu_put_sysregs(vcpu);
kvm_timer_vcpu_put(vcpu);
kvm_vgic_put(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 05/36] KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
@ 2017-12-07 17:05 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:05 UTC (permalink / raw)
To: linux-arm-kernel
As we are about to move a bunch of save/restore logic for VHE kernels to
the load and put functions, we need some infrastructure to do this.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Reworded comments as suggested by Drew
arch/arm/include/asm/kvm_host.h | 3 +++
arch/arm64/include/asm/kvm_host.h | 3 +++
arch/arm64/kvm/hyp/sysreg-sr.c | 30 ++++++++++++++++++++++++++++++
virt/kvm/arm/arm.c | 2 ++
4 files changed, 38 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index b92d2bcb3c08..8fce576199e0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -298,4 +298,7 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,
/* All host FP/SIMD state is restored on guest exit, so nothing to save: */
static inline void kvm_fpsimd_flush_cpu_state(void) {}
+static inline void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu) {}
+static inline void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu) {}
+
#endif /* __ARM_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index d85c1e6ec53f..20fab9194794 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -410,4 +410,7 @@ static inline void kvm_fpsimd_flush_cpu_state(void)
sve_flush_cpu_state();
}
+void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu);
+void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu);
+
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index e19d89cabf2a..cbbcd6f410a8 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -184,6 +184,36 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
}
+/**
+ * kvm_vcpu_load_sysregs - Load guest system registers to the physical CPU
+ *
+ * @vcpu: The VCPU pointer
+ *
+ * Load system registers that do not affect the host's execution, for
+ * example EL1 system registers on a VHE system where the host kernel
+ * runs at EL2. This function is called from KVM's vcpu_load() function
+ * and loading system register state early avoids having to load them on
+ * every entry to the VM.
+ */
+void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
+{
+}
+
+/**
+ * kvm_vcpu_put_sysregs - Restore host system registers to the physical CPU
+ *
+ * @vcpu: The VCPU pointer
+ *
+ * Save guest system registers that do not affect the host's execution, for
+ * example EL1 system registers on a VHE system where the host kernel
+ * runs at EL2. This function is called from KVM's vcpu_put() function
+ * and deferring saving system register state until we're no longer running the
+ * VCPU avoids having to save them on every exit from the VM.
+ */
+void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
+{
+}
+
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
{
asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index e5a5cf0d17fe..3e10343374a1 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -357,10 +357,12 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
kvm_arm_set_running_vcpu(vcpu);
kvm_vgic_load(vcpu);
kvm_timer_vcpu_load(vcpu);
+ kvm_vcpu_load_sysregs(vcpu);
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
+ kvm_vcpu_put_sysregs(vcpu);
kvm_timer_vcpu_put(vcpu);
kvm_vgic_put(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
Avoid saving the guest VFP registers and restoring the host VFP
registers on every exit from the VM. Only when we're about to run
userspace or other threads in the kernel do we really have to switch the
state back to the host state.
We still initially configure the VFP registers to trap when entering the
VM, but the difference is that we now leave the guest state in the
hardware registers as long as we're running this VCPU, even if we
occasionally trap to the host, and we only restore the host state when
we return to user space or when scheduling another thread.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Cosmetic changes
- Change the flags variable to a u8
- Expanded the commit message
arch/arm64/include/asm/kvm_emulate.h | 5 ++++
arch/arm64/include/asm/kvm_host.h | 3 +++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kvm/hyp/entry.S | 3 +++
arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
6 files changed, 46 insertions(+), 36 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index b36aaa1fe332..635137e6ed1c 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
return (unsigned long *)&vcpu->arch.hcr_el2;
}
+static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
+{
+ return !(vcpu->arch.hcr_el2 & HCR_RW);
+}
+
static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
{
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 20fab9194794..c841eeeeb5c5 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
/* Guest debug state */
u64 debug_flags;
+ /* 1 if the guest VFP state is loaded into the hardware */
+ u8 guest_vfp_loaded;
+
/*
* We maintain more than a single set of debug registers to support
* debugging the guest from the host and to maintain separate host and
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 612021dce84f..99467327c043 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -133,6 +133,7 @@ int main(void)
DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
+ DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index a360ac6e89e9..53652287a236 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -184,6 +184,9 @@ alternative_endif
add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
bl __fpsimd_restore_state
+ mov x0, #1
+ strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
+
// Skip restoring fpexc32 for AArch64 guests
mrs x1, hcr_el2
tbnz x1, #HCR_RW_SHIFT, 1f
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 11ec1c6f3b84..f5d53ef9ca79 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,43 +24,32 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
-static bool __hyp_text __fpsimd_enabled_nvhe(void)
-{
- return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
-}
-
-static bool __hyp_text __fpsimd_enabled_vhe(void)
-{
- return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
-}
-
-static hyp_alternate_select(__fpsimd_is_enabled,
- __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-bool __hyp_text __fpsimd_enabled(void)
-{
- return __fpsimd_is_enabled()();
-}
-
-static void __hyp_text __activate_traps_vhe(void)
+static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
{
u64 val;
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
- val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
+ val &= ~CPACR_EL1_ZEN;
+ if (vcpu->arch.guest_vfp_loaded)
+ val |= CPACR_EL1_FPEN;
+ else
+ val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
write_sysreg(__kvm_hyp_vector, vbar_el1);
}
-static void __hyp_text __activate_traps_nvhe(void)
+static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
val = CPTR_EL2_DEFAULT;
- val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
+ val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
+ if (vcpu->arch.guest_vfp_loaded)
+ val &= ~CPTR_EL2_TFP;
+ else
+ val |= CPTR_EL2_TFP;
write_sysreg(val, cptr_el2);
}
@@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
*/
val = vcpu->arch.hcr_el2;
- if (!(val & HCR_RW) && system_supports_fpsimd()) {
+ if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
+ !vcpu->arch.guest_vfp_loaded) {
write_sysreg(1 << 30, fpexc32_el2);
isb();
}
@@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
write_sysreg(0, pmselr_el0);
write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
- __activate_traps_arch()();
+ __activate_traps_arch()(vcpu);
}
static void __hyp_text __deactivate_traps_vhe(void)
@@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
- bool fp_enabled;
u64 exit_code;
vcpu = kern_hyp_va(vcpu);
@@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
/* 0 falls through to be handled out of EL2 */
}
- fp_enabled = __fpsimd_enabled();
-
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
@@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_host_state(host_ctxt);
- if (fp_enabled) {
- __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
- __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
- }
-
__debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index cbbcd6f410a8..68a7d164e5e1 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -19,6 +19,7 @@
#include <linux/kvm_host.h>
#include <asm/kvm_asm.h>
+#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
/* Yes, this does nothing, on purpose */
@@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
__sysreg_restore_common_state(ctxt);
}
+static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
+{
+ ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
+}
+
void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
@@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
- if (__fpsimd_enabled())
- sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
-
if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
}
@@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
*/
void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
+ struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+
+ /* Restore host FP/SIMD state */
+ if (vcpu->arch.guest_vfp_loaded) {
+ if (vcpu_el1_is_32bit(vcpu)) {
+ kvm_call_hyp(__fpsimd32_save_state,
+ kern_hyp_va(guest_ctxt));
+ }
+ __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
+ __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
+ vcpu->arch.guest_vfp_loaded = 0;
+ }
}
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Avoid saving the guest VFP registers and restoring the host VFP
registers on every exit from the VM. Only when we're about to run
userspace or other threads in the kernel do we really have to switch the
state back to the host state.
We still initially configure the VFP registers to trap when entering the
VM, but the difference is that we now leave the guest state in the
hardware registers as long as we're running this VCPU, even if we
occasionally trap to the host, and we only restore the host state when
we return to user space or when scheduling another thread.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Cosmetic changes
- Change the flags variable to a u8
- Expanded the commit message
arch/arm64/include/asm/kvm_emulate.h | 5 ++++
arch/arm64/include/asm/kvm_host.h | 3 +++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kvm/hyp/entry.S | 3 +++
arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
6 files changed, 46 insertions(+), 36 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index b36aaa1fe332..635137e6ed1c 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
return (unsigned long *)&vcpu->arch.hcr_el2;
}
+static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
+{
+ return !(vcpu->arch.hcr_el2 & HCR_RW);
+}
+
static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
{
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 20fab9194794..c841eeeeb5c5 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
/* Guest debug state */
u64 debug_flags;
+ /* 1 if the guest VFP state is loaded into the hardware */
+ u8 guest_vfp_loaded;
+
/*
* We maintain more than a single set of debug registers to support
* debugging the guest from the host and to maintain separate host and
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 612021dce84f..99467327c043 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -133,6 +133,7 @@ int main(void)
DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
+ DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index a360ac6e89e9..53652287a236 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -184,6 +184,9 @@ alternative_endif
add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
bl __fpsimd_restore_state
+ mov x0, #1
+ strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
+
// Skip restoring fpexc32 for AArch64 guests
mrs x1, hcr_el2
tbnz x1, #HCR_RW_SHIFT, 1f
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 11ec1c6f3b84..f5d53ef9ca79 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,43 +24,32 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
-static bool __hyp_text __fpsimd_enabled_nvhe(void)
-{
- return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
-}
-
-static bool __hyp_text __fpsimd_enabled_vhe(void)
-{
- return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
-}
-
-static hyp_alternate_select(__fpsimd_is_enabled,
- __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-bool __hyp_text __fpsimd_enabled(void)
-{
- return __fpsimd_is_enabled()();
-}
-
-static void __hyp_text __activate_traps_vhe(void)
+static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
{
u64 val;
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
- val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
+ val &= ~CPACR_EL1_ZEN;
+ if (vcpu->arch.guest_vfp_loaded)
+ val |= CPACR_EL1_FPEN;
+ else
+ val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
write_sysreg(__kvm_hyp_vector, vbar_el1);
}
-static void __hyp_text __activate_traps_nvhe(void)
+static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
val = CPTR_EL2_DEFAULT;
- val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
+ val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
+ if (vcpu->arch.guest_vfp_loaded)
+ val &= ~CPTR_EL2_TFP;
+ else
+ val |= CPTR_EL2_TFP;
write_sysreg(val, cptr_el2);
}
@@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
*/
val = vcpu->arch.hcr_el2;
- if (!(val & HCR_RW) && system_supports_fpsimd()) {
+ if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
+ !vcpu->arch.guest_vfp_loaded) {
write_sysreg(1 << 30, fpexc32_el2);
isb();
}
@@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
write_sysreg(0, pmselr_el0);
write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
- __activate_traps_arch()();
+ __activate_traps_arch()(vcpu);
}
static void __hyp_text __deactivate_traps_vhe(void)
@@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
- bool fp_enabled;
u64 exit_code;
vcpu = kern_hyp_va(vcpu);
@@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
/* 0 falls through to be handled out of EL2 */
}
- fp_enabled = __fpsimd_enabled();
-
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
@@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_host_state(host_ctxt);
- if (fp_enabled) {
- __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
- __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
- }
-
__debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index cbbcd6f410a8..68a7d164e5e1 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -19,6 +19,7 @@
#include <linux/kvm_host.h>
#include <asm/kvm_asm.h>
+#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
/* Yes, this does nothing, on purpose */
@@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
__sysreg_restore_common_state(ctxt);
}
+static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
+{
+ ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
+}
+
void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
@@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
- if (__fpsimd_enabled())
- sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
-
if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
}
@@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
*/
void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
+ struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+
+ /* Restore host FP/SIMD state */
+ if (vcpu->arch.guest_vfp_loaded) {
+ if (vcpu_el1_is_32bit(vcpu)) {
+ kvm_call_hyp(__fpsimd32_save_state,
+ kern_hyp_va(guest_ctxt));
+ }
+ __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
+ __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
+ vcpu->arch.guest_vfp_loaded = 0;
+ }
}
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 07/36] KVM: arm64: Move debug dirty flag calculation out of world switch
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
There is no need to figure out inside the world-switch if we should
save/restore the debug registers or not, we can might as well do that in
the higher level debug setup code, making it easier to optimize down the
line.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/debug.c | 5 +++++
arch/arm64/kvm/hyp/debug-sr.c | 6 ------
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c
index fa63b28c65e0..feedb877cff8 100644
--- a/arch/arm64/kvm/debug.c
+++ b/arch/arm64/kvm/debug.c
@@ -193,6 +193,11 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)
if (trap_debug)
vcpu->arch.mdcr_el2 |= MDCR_EL2_TDA;
+ /* If KDE or MDE are set, perform a full save/restore cycle. */
+ if ((vcpu_sys_reg(vcpu, MDSCR_EL1) & DBG_MDSCR_KDE) ||
+ (vcpu_sys_reg(vcpu, MDSCR_EL1) & DBG_MDSCR_MDE))
+ vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY;
+
trace_kvm_arm_set_dreg32("MDCR_EL2", vcpu->arch.mdcr_el2);
trace_kvm_arm_set_dreg32("MDSCR_EL1", vcpu_sys_reg(vcpu, MDSCR_EL1));
}
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 321c9c05dd9e..406829b6a43e 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -162,12 +162,6 @@ void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
{
- /* If any of KDE, MDE or KVM_ARM64_DEBUG_DIRTY is set, perform
- * a full save/restore cycle. */
- if ((vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_KDE) ||
- (vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_MDE))
- vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY;
-
__debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
__debug_save_spe()(&vcpu->arch.host_debug_state.pmscr_el1);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 07/36] KVM: arm64: Move debug dirty flag calculation out of world switch
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There is no need to figure out inside the world-switch if we should
save/restore the debug registers or not, we can might as well do that in
the higher level debug setup code, making it easier to optimize down the
line.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/debug.c | 5 +++++
arch/arm64/kvm/hyp/debug-sr.c | 6 ------
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c
index fa63b28c65e0..feedb877cff8 100644
--- a/arch/arm64/kvm/debug.c
+++ b/arch/arm64/kvm/debug.c
@@ -193,6 +193,11 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu)
if (trap_debug)
vcpu->arch.mdcr_el2 |= MDCR_EL2_TDA;
+ /* If KDE or MDE are set, perform a full save/restore cycle. */
+ if ((vcpu_sys_reg(vcpu, MDSCR_EL1) & DBG_MDSCR_KDE) ||
+ (vcpu_sys_reg(vcpu, MDSCR_EL1) & DBG_MDSCR_MDE))
+ vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY;
+
trace_kvm_arm_set_dreg32("MDCR_EL2", vcpu->arch.mdcr_el2);
trace_kvm_arm_set_dreg32("MDSCR_EL1", vcpu_sys_reg(vcpu, MDSCR_EL1));
}
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 321c9c05dd9e..406829b6a43e 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -162,12 +162,6 @@ void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
{
- /* If any of KDE, MDE or KVM_ARM64_DEBUG_DIRTY is set, perform
- * a full save/restore cycle. */
- if ((vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_KDE) ||
- (vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_MDE))
- vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY;
-
__debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
__debug_save_spe()(&vcpu->arch.host_debug_state.pmscr_el1);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 08/36] KVM: arm64: Slightly improve debug save/restore functions
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
The debug save/restore functions can be improved by using the has_vhe()
static key instead of the instruction alternative. Using the static key
uses the same paradigm as we're going to use elsewhere, it makes the
code more readable, and it generates slightly better code (no
stack setups and function calls unless necessary).
We also use a static key on the restore path, because it will be
marginally faster than loading a value from memory.
Finally, we don't have to conditionally clear the debug dirty flag if
it's set, we can just clear it.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Change dot to comma in comment
- Rename __debug_restore_spe to __debug_restore_spe_nvhe
arch/arm64/kvm/hyp/debug-sr.c | 26 ++++++++++++--------------
1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 406829b6a43e..81b8ad44f9e0 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -65,11 +65,6 @@
default: write_debug(ptr[0], reg, 0); \
}
-static void __hyp_text __debug_save_spe_vhe(u64 *pmscr_el1)
-{
- /* The vcpu can run. but it can't hide. */
-}
-
static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)
{
u64 reg;
@@ -99,11 +94,7 @@ static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)
dsb(nsh);
}
-static hyp_alternate_select(__debug_save_spe,
- __debug_save_spe_nvhe, __debug_save_spe_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-static void __hyp_text __debug_restore_spe(u64 pmscr_el1)
+static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1)
{
if (!pmscr_el1)
return;
@@ -164,17 +155,24 @@ void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
{
__debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
- __debug_save_spe()(&vcpu->arch.host_debug_state.pmscr_el1);
+
+ /*
+ * Non-VHE: Disable and flush SPE data generation
+ * VHE: The vcpu can run, but it can't hide.
+ */
+ if (!has_vhe())
+ __debug_save_spe_nvhe(&vcpu->arch.host_debug_state.pmscr_el1);
}
void __hyp_text __debug_cond_restore_host_state(struct kvm_vcpu *vcpu)
{
- __debug_restore_spe(vcpu->arch.host_debug_state.pmscr_el1);
+ if (!has_vhe())
+ __debug_restore_spe_nvhe(vcpu->arch.host_debug_state.pmscr_el1);
+
__debug_restore_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
+ vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
u32 __hyp_text __kvm_get_mdcr_el2(void)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 08/36] KVM: arm64: Slightly improve debug save/restore functions
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The debug save/restore functions can be improved by using the has_vhe()
static key instead of the instruction alternative. Using the static key
uses the same paradigm as we're going to use elsewhere, it makes the
code more readable, and it generates slightly better code (no
stack setups and function calls unless necessary).
We also use a static key on the restore path, because it will be
marginally faster than loading a value from memory.
Finally, we don't have to conditionally clear the debug dirty flag if
it's set, we can just clear it.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Change dot to comma in comment
- Rename __debug_restore_spe to __debug_restore_spe_nvhe
arch/arm64/kvm/hyp/debug-sr.c | 26 ++++++++++++--------------
1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 406829b6a43e..81b8ad44f9e0 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -65,11 +65,6 @@
default: write_debug(ptr[0], reg, 0); \
}
-static void __hyp_text __debug_save_spe_vhe(u64 *pmscr_el1)
-{
- /* The vcpu can run. but it can't hide. */
-}
-
static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)
{
u64 reg;
@@ -99,11 +94,7 @@ static void __hyp_text __debug_save_spe_nvhe(u64 *pmscr_el1)
dsb(nsh);
}
-static hyp_alternate_select(__debug_save_spe,
- __debug_save_spe_nvhe, __debug_save_spe_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-static void __hyp_text __debug_restore_spe(u64 pmscr_el1)
+static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1)
{
if (!pmscr_el1)
return;
@@ -164,17 +155,24 @@ void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
{
__debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
- __debug_save_spe()(&vcpu->arch.host_debug_state.pmscr_el1);
+
+ /*
+ * Non-VHE: Disable and flush SPE data generation
+ * VHE: The vcpu can run, but it can't hide.
+ */
+ if (!has_vhe())
+ __debug_save_spe_nvhe(&vcpu->arch.host_debug_state.pmscr_el1);
}
void __hyp_text __debug_cond_restore_host_state(struct kvm_vcpu *vcpu)
{
- __debug_restore_spe(vcpu->arch.host_debug_state.pmscr_el1);
+ if (!has_vhe())
+ __debug_restore_spe_nvhe(vcpu->arch.host_debug_state.pmscr_el1);
+
__debug_restore_state(vcpu, &vcpu->arch.host_debug_state.regs,
kern_hyp_va(vcpu->arch.host_cpu_context));
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
+ vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
u32 __hyp_text __kvm_get_mdcr_el2(void)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 09/36] KVM: arm64: Improve debug register save/restore flow
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
Instead of having multiple calls from the world switch path to the debug
logic, each figuring out if the dirty bit is set and if we should
save/restore the debug registers, let's just provide two hooks to the
debug save/restore functionality, one for switching to the guest
context, and one for switching to the host context, and we get the
benefit of only having to evaluate the dirty flag once on each path,
plus we give the compiler some more room to inline some of this
functionality.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Remove leading underscores from local variables
arch/arm64/include/asm/kvm_hyp.h | 10 ++-----
arch/arm64/kvm/hyp/debug-sr.c | 56 +++++++++++++++++++++++++++-------------
arch/arm64/kvm/hyp/switch.c | 6 ++---
3 files changed, 42 insertions(+), 30 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 08d3bb66c8b7..a0e5a7038237 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -139,14 +139,8 @@ void __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt);
void __sysreg32_save_state(struct kvm_vcpu *vcpu);
void __sysreg32_restore_state(struct kvm_vcpu *vcpu);
-void __debug_save_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt);
-void __debug_restore_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt);
-void __debug_cond_save_host_state(struct kvm_vcpu *vcpu);
-void __debug_cond_restore_host_state(struct kvm_vcpu *vcpu);
+void __debug_switch_to_guest(struct kvm_vcpu *vcpu);
+void __debug_switch_to_host(struct kvm_vcpu *vcpu);
void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 81b8ad44f9e0..ee87115eb12f 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -106,16 +106,13 @@ static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1)
write_sysreg_s(pmscr_el1, SYS_PMSCR_EL1);
}
-void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt)
+static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
+ struct kvm_guest_debug_arch *dbg,
+ struct kvm_cpu_context *ctxt)
{
u64 aa64dfr0;
int brps, wrps;
- if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
- return;
-
aa64dfr0 = read_sysreg(id_aa64dfr0_el1);
brps = (aa64dfr0 >> 12) & 0xf;
wrps = (aa64dfr0 >> 20) & 0xf;
@@ -128,16 +125,13 @@ void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
ctxt->sys_regs[MDCCINT_EL1] = read_sysreg(mdccint_el1);
}
-void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt)
+static void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
+ struct kvm_guest_debug_arch *dbg,
+ struct kvm_cpu_context *ctxt)
{
u64 aa64dfr0;
int brps, wrps;
- if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
- return;
-
aa64dfr0 = read_sysreg(id_aa64dfr0_el1);
brps = (aa64dfr0 >> 12) & 0xf;
@@ -151,10 +145,12 @@ void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
write_sysreg(ctxt->sys_regs[MDCCINT_EL1], mdccint_el1);
}
-void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
+void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu)
{
- __debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
- kern_hyp_va(vcpu->arch.host_cpu_context));
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ struct kvm_guest_debug_arch *host_dbg;
+ struct kvm_guest_debug_arch *guest_dbg;
/*
* Non-VHE: Disable and flush SPE data generation
@@ -162,15 +158,39 @@ void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
*/
if (!has_vhe())
__debug_save_spe_nvhe(&vcpu->arch.host_debug_state.pmscr_el1);
+
+ if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
+ return;
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ guest_ctxt = &vcpu->arch.ctxt;
+ host_dbg = &vcpu->arch.host_debug_state.regs;
+ guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
+
+ __debug_save_state(vcpu, host_dbg, host_ctxt);
+ __debug_restore_state(vcpu, guest_dbg, guest_ctxt);
}
-void __hyp_text __debug_cond_restore_host_state(struct kvm_vcpu *vcpu)
+void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ struct kvm_guest_debug_arch *host_dbg;
+ struct kvm_guest_debug_arch *guest_dbg;
+
if (!has_vhe())
__debug_restore_spe_nvhe(vcpu->arch.host_debug_state.pmscr_el1);
- __debug_restore_state(vcpu, &vcpu->arch.host_debug_state.regs,
- kern_hyp_va(vcpu->arch.host_cpu_context));
+ if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
+ return;
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ guest_ctxt = &vcpu->arch.ctxt;
+ host_dbg = &vcpu->arch.host_debug_state.regs;
+ guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
+
+ __debug_save_state(vcpu, guest_dbg, guest_ctxt);
+ __debug_restore_state(vcpu, host_dbg, host_ctxt);
vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f5d53ef9ca79..6982392745f5 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -287,7 +287,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
guest_ctxt = &vcpu->arch.ctxt;
__sysreg_save_host_state(host_ctxt);
- __debug_cond_save_host_state(vcpu);
__activate_traps(vcpu);
__activate_vm(vcpu);
@@ -301,7 +300,7 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
*/
__sysreg32_restore_state(vcpu);
__sysreg_restore_guest_state(guest_ctxt);
- __debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
+ __debug_switch_to_guest(vcpu);
/* Jump in the fire! */
again:
@@ -379,12 +378,11 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_host_state(host_ctxt);
- __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
* system may enable SPE here and make use of the TTBRs.
*/
- __debug_cond_restore_host_state(vcpu);
+ __debug_switch_to_host(vcpu);
return exit_code;
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 09/36] KVM: arm64: Improve debug register save/restore flow
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Instead of having multiple calls from the world switch path to the debug
logic, each figuring out if the dirty bit is set and if we should
save/restore the debug registers, let's just provide two hooks to the
debug save/restore functionality, one for switching to the guest
context, and one for switching to the host context, and we get the
benefit of only having to evaluate the dirty flag once on each path,
plus we give the compiler some more room to inline some of this
functionality.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Remove leading underscores from local variables
arch/arm64/include/asm/kvm_hyp.h | 10 ++-----
arch/arm64/kvm/hyp/debug-sr.c | 56 +++++++++++++++++++++++++++-------------
arch/arm64/kvm/hyp/switch.c | 6 ++---
3 files changed, 42 insertions(+), 30 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 08d3bb66c8b7..a0e5a7038237 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -139,14 +139,8 @@ void __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt);
void __sysreg32_save_state(struct kvm_vcpu *vcpu);
void __sysreg32_restore_state(struct kvm_vcpu *vcpu);
-void __debug_save_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt);
-void __debug_restore_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt);
-void __debug_cond_save_host_state(struct kvm_vcpu *vcpu);
-void __debug_cond_restore_host_state(struct kvm_vcpu *vcpu);
+void __debug_switch_to_guest(struct kvm_vcpu *vcpu);
+void __debug_switch_to_host(struct kvm_vcpu *vcpu);
void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 81b8ad44f9e0..ee87115eb12f 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -106,16 +106,13 @@ static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1)
write_sysreg_s(pmscr_el1, SYS_PMSCR_EL1);
}
-void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt)
+static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
+ struct kvm_guest_debug_arch *dbg,
+ struct kvm_cpu_context *ctxt)
{
u64 aa64dfr0;
int brps, wrps;
- if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
- return;
-
aa64dfr0 = read_sysreg(id_aa64dfr0_el1);
brps = (aa64dfr0 >> 12) & 0xf;
wrps = (aa64dfr0 >> 20) & 0xf;
@@ -128,16 +125,13 @@ void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu,
ctxt->sys_regs[MDCCINT_EL1] = read_sysreg(mdccint_el1);
}
-void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
- struct kvm_guest_debug_arch *dbg,
- struct kvm_cpu_context *ctxt)
+static void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
+ struct kvm_guest_debug_arch *dbg,
+ struct kvm_cpu_context *ctxt)
{
u64 aa64dfr0;
int brps, wrps;
- if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
- return;
-
aa64dfr0 = read_sysreg(id_aa64dfr0_el1);
brps = (aa64dfr0 >> 12) & 0xf;
@@ -151,10 +145,12 @@ void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,
write_sysreg(ctxt->sys_regs[MDCCINT_EL1], mdccint_el1);
}
-void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
+void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu)
{
- __debug_save_state(vcpu, &vcpu->arch.host_debug_state.regs,
- kern_hyp_va(vcpu->arch.host_cpu_context));
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ struct kvm_guest_debug_arch *host_dbg;
+ struct kvm_guest_debug_arch *guest_dbg;
/*
* Non-VHE: Disable and flush SPE data generation
@@ -162,15 +158,39 @@ void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu)
*/
if (!has_vhe())
__debug_save_spe_nvhe(&vcpu->arch.host_debug_state.pmscr_el1);
+
+ if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
+ return;
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ guest_ctxt = &vcpu->arch.ctxt;
+ host_dbg = &vcpu->arch.host_debug_state.regs;
+ guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
+
+ __debug_save_state(vcpu, host_dbg, host_ctxt);
+ __debug_restore_state(vcpu, guest_dbg, guest_ctxt);
}
-void __hyp_text __debug_cond_restore_host_state(struct kvm_vcpu *vcpu)
+void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ struct kvm_guest_debug_arch *host_dbg;
+ struct kvm_guest_debug_arch *guest_dbg;
+
if (!has_vhe())
__debug_restore_spe_nvhe(vcpu->arch.host_debug_state.pmscr_el1);
- __debug_restore_state(vcpu, &vcpu->arch.host_debug_state.regs,
- kern_hyp_va(vcpu->arch.host_cpu_context));
+ if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY))
+ return;
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ guest_ctxt = &vcpu->arch.ctxt;
+ host_dbg = &vcpu->arch.host_debug_state.regs;
+ guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
+
+ __debug_save_state(vcpu, guest_dbg, guest_ctxt);
+ __debug_restore_state(vcpu, host_dbg, host_ctxt);
vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f5d53ef9ca79..6982392745f5 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -287,7 +287,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
guest_ctxt = &vcpu->arch.ctxt;
__sysreg_save_host_state(host_ctxt);
- __debug_cond_save_host_state(vcpu);
__activate_traps(vcpu);
__activate_vm(vcpu);
@@ -301,7 +300,7 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
*/
__sysreg32_restore_state(vcpu);
__sysreg_restore_guest_state(guest_ctxt);
- __debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
+ __debug_switch_to_guest(vcpu);
/* Jump in the fire! */
again:
@@ -379,12 +378,11 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_host_state(host_ctxt);
- __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
* system may enable SPE here and make use of the TTBRs.
*/
- __debug_cond_restore_host_state(vcpu);
+ __debug_switch_to_host(vcpu);
return exit_code;
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 10/36] KVM: arm64: Factor out fault info population and gic workarounds
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
The current world-switch function has functionality to detect a number
of cases where we need to fixup some part of the exit condition and
possibly run the guest again, before having restored the host state.
This includes populating missing fault info, emulating GICv2 CPU
interface accesses when mapped at unaligned addresses, and emulating
the GICv3 CPU interface on systems that need it.
As we are about to have an alternative switch function for VHE systems,
but VHE systems still need the same early fixup logic, factor out this
logic into a separate function that can be shared by both switch
functions.
No functional change.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fixed typos in commit message
- Changed comment in fixup_guest_exit
- Us do-while instead of jumping to a label
arch/arm64/kvm/hyp/switch.c | 99 ++++++++++++++++++++++++---------------------
1 file changed, 54 insertions(+), 45 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 6982392745f5..845e3bece399 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -274,50 +274,24 @@ static bool __hyp_text __skip_instr(struct kvm_vcpu *vcpu)
}
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+/*
+ * Return true when we were able to fixup the guest exit and should return to
+ * the guest, false when we should restore the host state and return to the
+ * main run loop.
+ */
+static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
{
- struct kvm_cpu_context *host_ctxt;
- struct kvm_cpu_context *guest_ctxt;
- u64 exit_code;
-
- vcpu = kern_hyp_va(vcpu);
-
- host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
- host_ctxt->__hyp_running_vcpu = vcpu;
- guest_ctxt = &vcpu->arch.ctxt;
-
- __sysreg_save_host_state(host_ctxt);
-
- __activate_traps(vcpu);
- __activate_vm(vcpu);
-
- __vgic_restore_state(vcpu);
- __timer_enable_traps(vcpu);
-
- /*
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
- __debug_switch_to_guest(vcpu);
-
- /* Jump in the fire! */
-again:
- exit_code = __guest_enter(vcpu, host_ctxt);
- /* And we're baaack! */
-
/*
* We're using the raw exception code in order to only process
* the trap if no SError is pending. We will come back to the
* same PC once the SError has been injected, and replay the
* trapping instruction.
*/
- if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
- goto again;
+ if (*exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
+ return true;
if (static_branch_unlikely(&vgic_v2_cpuif_trap) &&
- exit_code == ARM_EXCEPTION_TRAP) {
+ *exit_code == ARM_EXCEPTION_TRAP) {
bool valid;
valid = kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_DABT_LOW &&
@@ -331,9 +305,9 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
if (ret == 1) {
if (__skip_instr(vcpu))
- goto again;
+ return true;
else
- exit_code = ARM_EXCEPTION_TRAP;
+ *exit_code = ARM_EXCEPTION_TRAP;
}
if (ret == -1) {
@@ -345,29 +319,64 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
*/
if (!__skip_instr(vcpu))
*vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
- exit_code = ARM_EXCEPTION_EL1_SERROR;
+ *exit_code = ARM_EXCEPTION_EL1_SERROR;
}
-
- /* 0 falls through to be handler out of EL2 */
}
}
if (static_branch_unlikely(&vgic_v3_cpuif_trap) &&
- exit_code == ARM_EXCEPTION_TRAP &&
+ *exit_code == ARM_EXCEPTION_TRAP &&
(kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 ||
kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_CP15_32)) {
int ret = __vgic_v3_perform_cpuif_access(vcpu);
if (ret == 1) {
if (__skip_instr(vcpu))
- goto again;
+ return true;
else
- exit_code = ARM_EXCEPTION_TRAP;
+ *exit_code = ARM_EXCEPTION_TRAP;
}
-
- /* 0 falls through to be handled out of EL2 */
}
+ /* Return to the host kernel and handle the exit */
+ return false;
+}
+
+int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ u64 exit_code;
+
+ vcpu = kern_hyp_va(vcpu);
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt->__hyp_running_vcpu = vcpu;
+ guest_ctxt = &vcpu->arch.ctxt;
+
+ __sysreg_save_host_state(host_ctxt);
+
+ __activate_traps(vcpu);
+ __activate_vm(vcpu);
+
+ __vgic_restore_state(vcpu);
+ __timer_enable_traps(vcpu);
+
+ /*
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_guest_state(guest_ctxt);
+ __debug_switch_to_guest(vcpu);
+
+ do {
+ /* Jump in the fire! */
+ exit_code = __guest_enter(vcpu, host_ctxt);
+
+ /* And we're baaack! */
+ } while (fixup_guest_exit(vcpu, &exit_code));
+
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 10/36] KVM: arm64: Factor out fault info population and gic workarounds
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The current world-switch function has functionality to detect a number
of cases where we need to fixup some part of the exit condition and
possibly run the guest again, before having restored the host state.
This includes populating missing fault info, emulating GICv2 CPU
interface accesses when mapped at unaligned addresses, and emulating
the GICv3 CPU interface on systems that need it.
As we are about to have an alternative switch function for VHE systems,
but VHE systems still need the same early fixup logic, factor out this
logic into a separate function that can be shared by both switch
functions.
No functional change.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fixed typos in commit message
- Changed comment in fixup_guest_exit
- Us do-while instead of jumping to a label
arch/arm64/kvm/hyp/switch.c | 99 ++++++++++++++++++++++++---------------------
1 file changed, 54 insertions(+), 45 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 6982392745f5..845e3bece399 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -274,50 +274,24 @@ static bool __hyp_text __skip_instr(struct kvm_vcpu *vcpu)
}
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+/*
+ * Return true when we were able to fixup the guest exit and should return to
+ * the guest, false when we should restore the host state and return to the
+ * main run loop.
+ */
+static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
{
- struct kvm_cpu_context *host_ctxt;
- struct kvm_cpu_context *guest_ctxt;
- u64 exit_code;
-
- vcpu = kern_hyp_va(vcpu);
-
- host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
- host_ctxt->__hyp_running_vcpu = vcpu;
- guest_ctxt = &vcpu->arch.ctxt;
-
- __sysreg_save_host_state(host_ctxt);
-
- __activate_traps(vcpu);
- __activate_vm(vcpu);
-
- __vgic_restore_state(vcpu);
- __timer_enable_traps(vcpu);
-
- /*
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
- __debug_switch_to_guest(vcpu);
-
- /* Jump in the fire! */
-again:
- exit_code = __guest_enter(vcpu, host_ctxt);
- /* And we're baaack! */
-
/*
* We're using the raw exception code in order to only process
* the trap if no SError is pending. We will come back to the
* same PC once the SError has been injected, and replay the
* trapping instruction.
*/
- if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
- goto again;
+ if (*exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu))
+ return true;
if (static_branch_unlikely(&vgic_v2_cpuif_trap) &&
- exit_code == ARM_EXCEPTION_TRAP) {
+ *exit_code == ARM_EXCEPTION_TRAP) {
bool valid;
valid = kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_DABT_LOW &&
@@ -331,9 +305,9 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
if (ret == 1) {
if (__skip_instr(vcpu))
- goto again;
+ return true;
else
- exit_code = ARM_EXCEPTION_TRAP;
+ *exit_code = ARM_EXCEPTION_TRAP;
}
if (ret == -1) {
@@ -345,29 +319,64 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
*/
if (!__skip_instr(vcpu))
*vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
- exit_code = ARM_EXCEPTION_EL1_SERROR;
+ *exit_code = ARM_EXCEPTION_EL1_SERROR;
}
-
- /* 0 falls through to be handler out of EL2 */
}
}
if (static_branch_unlikely(&vgic_v3_cpuif_trap) &&
- exit_code == ARM_EXCEPTION_TRAP &&
+ *exit_code == ARM_EXCEPTION_TRAP &&
(kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 ||
kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_CP15_32)) {
int ret = __vgic_v3_perform_cpuif_access(vcpu);
if (ret == 1) {
if (__skip_instr(vcpu))
- goto again;
+ return true;
else
- exit_code = ARM_EXCEPTION_TRAP;
+ *exit_code = ARM_EXCEPTION_TRAP;
}
-
- /* 0 falls through to be handled out of EL2 */
}
+ /* Return to the host kernel and handle the exit */
+ return false;
+}
+
+int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ u64 exit_code;
+
+ vcpu = kern_hyp_va(vcpu);
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt->__hyp_running_vcpu = vcpu;
+ guest_ctxt = &vcpu->arch.ctxt;
+
+ __sysreg_save_host_state(host_ctxt);
+
+ __activate_traps(vcpu);
+ __activate_vm(vcpu);
+
+ __vgic_restore_state(vcpu);
+ __timer_enable_traps(vcpu);
+
+ /*
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_guest_state(guest_ctxt);
+ __debug_switch_to_guest(vcpu);
+
+ do {
+ /* Jump in the fire! */
+ exit_code = __guest_enter(vcpu, host_ctxt);
+
+ /* And we're baaack! */
+ } while (fixup_guest_exit(vcpu, &exit_code));
+
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
So far this is just a copy of the legacy non-VHE switch function, where
we only change the existing calls to has_vhe() in both the original and
new functions.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
__kvm_vcpu_run_nvhe
- Removed stray whitespace line
arch/arm/include/asm/kvm_asm.h | 5 +++-
arch/arm/kvm/hyp/switch.c | 2 +-
arch/arm64/include/asm/kvm_asm.h | 4 ++-
arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
virt/kvm/arm/arm.c | 5 +++-
5 files changed, 69 insertions(+), 5 deletions(-)
diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
index 36dd2962a42d..4ac717276543 100644
--- a/arch/arm/include/asm/kvm_asm.h
+++ b/arch/arm/include/asm/kvm_asm.h
@@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
-extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+/* no VHE on 32-bit :( */
+static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
+
+extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
extern void __init_stage2_translation(void);
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index c3b9799e2e13..7b2bd25e3b10 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
return true;
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 33e0edc6f8be..adaf1db12271 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
-extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
+
+extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
extern u64 __vgic_v3_get_ich_vtr_el2(void);
extern u64 __vgic_v3_read_vmcr(void);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 845e3bece399..8f32f8dcab65 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
return false;
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+/* Switch to the guest for VHE systems running in EL2 */
+int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ u64 exit_code;
+
+ vcpu = kern_hyp_va(vcpu);
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt->__hyp_running_vcpu = vcpu;
+ guest_ctxt = &vcpu->arch.ctxt;
+
+ __sysreg_save_host_state(host_ctxt);
+
+ __activate_traps(vcpu);
+ __activate_vm(vcpu);
+
+ __vgic_restore_state(vcpu);
+ __timer_enable_traps(vcpu);
+
+ /*
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_guest_state(guest_ctxt);
+ __debug_switch_to_guest(vcpu);
+
+ do {
+ /* Jump in the fire! */
+ exit_code = __guest_enter(vcpu, host_ctxt);
+
+ /* And we're baaack! */
+ } while (fixup_guest_exit(vcpu, &exit_code));
+
+ __sysreg_save_guest_state(guest_ctxt);
+ __sysreg32_save_state(vcpu);
+ __timer_disable_traps(vcpu);
+ __vgic_save_state(vcpu);
+
+ __deactivate_traps(vcpu);
+ __deactivate_vm(vcpu);
+
+ __sysreg_restore_host_state(host_ctxt);
+
+ /*
+ * This must come after restoring the host sysregs, since a non-VHE
+ * system may enable SPE here and make use of the TTBRs.
+ */
+ __debug_switch_to_host(vcpu);
+
+ return exit_code;
+}
+
+/* Switch to the guest for legacy non-VHE systems */
+int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 3e10343374a1..104ee524c75a 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
trace_kvm_entry(*vcpu_pc(vcpu));
guest_enter_irqoff();
- ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
+ if (has_vhe())
+ ret = kvm_vcpu_run_vhe(vcpu);
+ else
+ ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
vcpu->mode = OUTSIDE_GUEST_MODE;
vcpu->stat.exits++;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
So far this is just a copy of the legacy non-VHE switch function, where
we only change the existing calls to has_vhe() in both the original and
new functions.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
__kvm_vcpu_run_nvhe
- Removed stray whitespace line
arch/arm/include/asm/kvm_asm.h | 5 +++-
arch/arm/kvm/hyp/switch.c | 2 +-
arch/arm64/include/asm/kvm_asm.h | 4 ++-
arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
virt/kvm/arm/arm.c | 5 +++-
5 files changed, 69 insertions(+), 5 deletions(-)
diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
index 36dd2962a42d..4ac717276543 100644
--- a/arch/arm/include/asm/kvm_asm.h
+++ b/arch/arm/include/asm/kvm_asm.h
@@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
-extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+/* no VHE on 32-bit :( */
+static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
+
+extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
extern void __init_stage2_translation(void);
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index c3b9799e2e13..7b2bd25e3b10 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
return true;
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 33e0edc6f8be..adaf1db12271 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
-extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
+
+extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
extern u64 __vgic_v3_get_ich_vtr_el2(void);
extern u64 __vgic_v3_read_vmcr(void);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 845e3bece399..8f32f8dcab65 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
return false;
}
-int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
+/* Switch to the guest for VHE systems running in EL2 */
+int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpu_context *host_ctxt;
+ struct kvm_cpu_context *guest_ctxt;
+ u64 exit_code;
+
+ vcpu = kern_hyp_va(vcpu);
+
+ host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt->__hyp_running_vcpu = vcpu;
+ guest_ctxt = &vcpu->arch.ctxt;
+
+ __sysreg_save_host_state(host_ctxt);
+
+ __activate_traps(vcpu);
+ __activate_vm(vcpu);
+
+ __vgic_restore_state(vcpu);
+ __timer_enable_traps(vcpu);
+
+ /*
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_guest_state(guest_ctxt);
+ __debug_switch_to_guest(vcpu);
+
+ do {
+ /* Jump in the fire! */
+ exit_code = __guest_enter(vcpu, host_ctxt);
+
+ /* And we're baaack! */
+ } while (fixup_guest_exit(vcpu, &exit_code));
+
+ __sysreg_save_guest_state(guest_ctxt);
+ __sysreg32_save_state(vcpu);
+ __timer_disable_traps(vcpu);
+ __vgic_save_state(vcpu);
+
+ __deactivate_traps(vcpu);
+ __deactivate_vm(vcpu);
+
+ __sysreg_restore_host_state(host_ctxt);
+
+ /*
+ * This must come after restoring the host sysregs, since a non-VHE
+ * system may enable SPE here and make use of the TTBRs.
+ */
+ __debug_switch_to_host(vcpu);
+
+ return exit_code;
+}
+
+/* Switch to the guest for legacy non-VHE systems */
+int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
struct kvm_cpu_context *guest_ctxt;
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 3e10343374a1..104ee524c75a 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
trace_kvm_entry(*vcpu_pc(vcpu));
guest_enter_irqoff();
- ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
+ if (has_vhe())
+ ret = kvm_vcpu_run_vhe(vcpu);
+ else
+ ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
vcpu->mode = OUTSIDE_GUEST_MODE;
vcpu->stat.exits++;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 12/36] KVM: arm64: Remove kern_hyp_va() use in VHE switch function
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
VHE kernels run completely in EL2 and therefore don't have a notion of
kernel and hyp addresses, they are all just kernel addresses. Therefore
don't call kern_hyp_va() in the VHE switch function.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 8f32f8dcab65..eac2202a179c 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -349,9 +349,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
- vcpu = kern_hyp_va(vcpu);
-
- host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt = vcpu->arch.host_cpu_context;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 12/36] KVM: arm64: Remove kern_hyp_va() use in VHE switch function
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
VHE kernels run completely in EL2 and therefore don't have a notion of
kernel and hyp addresses, they are all just kernel addresses. Therefore
don't call kern_hyp_va() in the VHE switch function.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 8f32f8dcab65..eac2202a179c 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -349,9 +349,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
- vcpu = kern_hyp_va(vcpu);
-
- host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt = vcpu->arch.host_cpu_context;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 13/36] KVM: arm64: Don't deactivate VM on VHE systems
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
There is no need to reset the VTTBR to zero when exiting the guest on
VHE systems. VHE systems don't use stage 2 translations for the EL2&0
translation regime used by the host.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Changed __activate_vm to take a kvm pointer
- No longer adding inline attributes to functions
arch/arm64/kvm/hyp/switch.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index eac2202a179c..e783e2371b7c 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -140,9 +140,8 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
write_sysreg(0, pmuserenr_el0);
}
-static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu)
+static void __hyp_text __activate_vm(struct kvm *kvm)
{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
write_sysreg(kvm->arch.vttbr, vttbr_el2);
}
@@ -356,7 +355,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__sysreg_save_host_state(host_ctxt);
__activate_traps(vcpu);
- __activate_vm(vcpu);
+ __activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
__timer_enable_traps(vcpu);
@@ -382,7 +381,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
- __deactivate_vm(vcpu);
__sysreg_restore_host_state(host_ctxt);
@@ -411,7 +409,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__sysreg_save_host_state(host_ctxt);
__activate_traps(vcpu);
- __activate_vm(vcpu);
+ __activate_vm(kern_hyp_va(vcpu->kvm));
__vgic_restore_state(vcpu);
__timer_enable_traps(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 13/36] KVM: arm64: Don't deactivate VM on VHE systems
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There is no need to reset the VTTBR to zero when exiting the guest on
VHE systems. VHE systems don't use stage 2 translations for the EL2&0
translation regime used by the host.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Changed __activate_vm to take a kvm pointer
- No longer adding inline attributes to functions
arch/arm64/kvm/hyp/switch.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index eac2202a179c..e783e2371b7c 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -140,9 +140,8 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
write_sysreg(0, pmuserenr_el0);
}
-static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu)
+static void __hyp_text __activate_vm(struct kvm *kvm)
{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
write_sysreg(kvm->arch.vttbr, vttbr_el2);
}
@@ -356,7 +355,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__sysreg_save_host_state(host_ctxt);
__activate_traps(vcpu);
- __activate_vm(vcpu);
+ __activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
__timer_enable_traps(vcpu);
@@ -382,7 +381,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
- __deactivate_vm(vcpu);
__sysreg_restore_host_state(host_ctxt);
@@ -411,7 +409,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__sysreg_save_host_state(host_ctxt);
__activate_traps(vcpu);
- __activate_vm(vcpu);
+ __activate_vm(kern_hyp_va(vcpu->kvm));
__vgic_restore_state(vcpu);
__timer_enable_traps(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
The VHE switch function calls __timer_enable_traps and
__timer_disable_traps which don't do anything on VHE systems.
Therefore, simply remove these calls from the VHE switch function and
make the functions non-conditional as they are now only called from the
non-VHE switch path.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 2 --
virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
2 files changed, 14 insertions(+), 24 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index e783e2371b7c..09aafa0470f7 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
- __timer_enable_traps(vcpu);
/*
* We must restore the 32-bit state before the sysregs, thanks
@@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
- __timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
index f24404b3c8df..752b37f9133c 100644
--- a/virt/kvm/arm/hyp/timer-sr.c
+++ b/virt/kvm/arm/hyp/timer-sr.c
@@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
{
- /*
- * We don't need to do this for VHE since the host kernel runs in EL2
- * with HCR_EL2.TGE ==1, which makes those bits have no impact.
- */
- if (!has_vhe()) {
- u64 val;
+ u64 val;
- /* Allow physical timer/counter access for the host */
- val = read_sysreg(cnthctl_el2);
- val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
- write_sysreg(val, cnthctl_el2);
- }
+ /* Allow physical timer/counter access for the host */
+ val = read_sysreg(cnthctl_el2);
+ val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
+ write_sysreg(val, cnthctl_el2);
}
void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
{
- if (!has_vhe()) {
- u64 val;
+ u64 val;
- /*
- * Disallow physical timer access for the guest
- * Physical counter access is allowed
- */
- val = read_sysreg(cnthctl_el2);
- val &= ~CNTHCTL_EL1PCEN;
- val |= CNTHCTL_EL1PCTEN;
- write_sysreg(val, cnthctl_el2);
- }
+ /*
+ * Disallow physical timer access for the guest
+ * Physical counter access is allowed
+ */
+ val = read_sysreg(cnthctl_el2);
+ val &= ~CNTHCTL_EL1PCEN;
+ val |= CNTHCTL_EL1PCTEN;
+ write_sysreg(val, cnthctl_el2);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The VHE switch function calls __timer_enable_traps and
__timer_disable_traps which don't do anything on VHE systems.
Therefore, simply remove these calls from the VHE switch function and
make the functions non-conditional as they are now only called from the
non-VHE switch path.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 2 --
virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
2 files changed, 14 insertions(+), 24 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index e783e2371b7c..09aafa0470f7 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
- __timer_enable_traps(vcpu);
/*
* We must restore the 32-bit state before the sysregs, thanks
@@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__sysreg_save_guest_state(guest_ctxt);
__sysreg32_save_state(vcpu);
- __timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
index f24404b3c8df..752b37f9133c 100644
--- a/virt/kvm/arm/hyp/timer-sr.c
+++ b/virt/kvm/arm/hyp/timer-sr.c
@@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
{
- /*
- * We don't need to do this for VHE since the host kernel runs in EL2
- * with HCR_EL2.TGE ==1, which makes those bits have no impact.
- */
- if (!has_vhe()) {
- u64 val;
+ u64 val;
- /* Allow physical timer/counter access for the host */
- val = read_sysreg(cnthctl_el2);
- val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
- write_sysreg(val, cnthctl_el2);
- }
+ /* Allow physical timer/counter access for the host */
+ val = read_sysreg(cnthctl_el2);
+ val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
+ write_sysreg(val, cnthctl_el2);
}
void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
{
- if (!has_vhe()) {
- u64 val;
+ u64 val;
- /*
- * Disallow physical timer access for the guest
- * Physical counter access is allowed
- */
- val = read_sysreg(cnthctl_el2);
- val &= ~CNTHCTL_EL1PCEN;
- val |= CNTHCTL_EL1PCTEN;
- write_sysreg(val, cnthctl_el2);
- }
+ /*
+ * Disallow physical timer access for the guest
+ * Physical counter access is allowed
+ */
+ val = read_sysreg(cnthctl_el2);
+ val &= ~CNTHCTL_EL1PCEN;
+ val |= CNTHCTL_EL1PCTEN;
+ write_sysreg(val, cnthctl_el2);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
There's a semantic difference between the EL1 registers that control
operation of a kernel running in EL1 and EL1 registers that only control
userspace execution in EL0. Since we can defer saving/restoring the
latter, move them into their own function.
We also take this chance to rename the function saving/restoring the
remaining system register to make it clear this function deals with
the EL1 system registers.
No functional change.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Added comment about sp_el0 to common save sysreg save/restore functions
arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
1 file changed, 33 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 68a7d164e5e1..bbfb4d01af88 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
*/
static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
+{
+ ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
+
+ /*
+ * The host arm64 Linux uses sp_el0 to point to 'current' and it must
+ * therefore be saved/restored on every entry/exit to/from the guest.
+ */
+ ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
+}
+
+static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
{
ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
- ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
- ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
}
-static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
+static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
{
ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
@@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
}
static hyp_alternate_select(__sysreg_call_save_host_state,
- __sysreg_save_state, __sysreg_do_nothing,
+ __sysreg_save_el1_state, __sysreg_do_nothing,
ARM64_HAS_VIRT_HOST_EXTN);
void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
{
__sysreg_call_save_host_state()(ctxt);
__sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
}
void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_save_state(ctxt);
+ __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
}
static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
{
- write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
- write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
- write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
+
+ /*
+ * The host arm64 Linux uses sp_el0 to point to 'current' and it must
+ * therefore be saved/restored on every entry/exit to/from the guest.
+ */
write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
}
-static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
+static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
+{
+ write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
+ write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
+ write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
+}
+
+static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
{
write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
@@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
}
static hyp_alternate_select(__sysreg_call_restore_host_state,
- __sysreg_restore_state, __sysreg_do_nothing,
+ __sysreg_restore_el1_state, __sysreg_do_nothing,
ARM64_HAS_VIRT_HOST_EXTN);
void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
{
__sysreg_call_restore_host_state()(ctxt);
__sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
}
void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_restore_state(ctxt);
+ __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
}
static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There's a semantic difference between the EL1 registers that control
operation of a kernel running in EL1 and EL1 registers that only control
userspace execution in EL0. Since we can defer saving/restoring the
latter, move them into their own function.
We also take this chance to rename the function saving/restoring the
remaining system register to make it clear this function deals with
the EL1 system registers.
No functional change.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Added comment about sp_el0 to common save sysreg save/restore functions
arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
1 file changed, 33 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 68a7d164e5e1..bbfb4d01af88 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
*/
static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
+{
+ ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
+
+ /*
+ * The host arm64 Linux uses sp_el0 to point to 'current' and it must
+ * therefore be saved/restored on every entry/exit to/from the guest.
+ */
+ ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
+}
+
+static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
{
ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
- ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
- ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
}
-static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
+static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
{
ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
@@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
}
static hyp_alternate_select(__sysreg_call_save_host_state,
- __sysreg_save_state, __sysreg_do_nothing,
+ __sysreg_save_el1_state, __sysreg_do_nothing,
ARM64_HAS_VIRT_HOST_EXTN);
void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
{
__sysreg_call_save_host_state()(ctxt);
__sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
}
void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_save_state(ctxt);
+ __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
}
static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
{
- write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
- write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
- write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
+
+ /*
+ * The host arm64 Linux uses sp_el0 to point to 'current' and it must
+ * therefore be saved/restored on every entry/exit to/from the guest.
+ */
write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
}
-static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
+static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
+{
+ write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
+ write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
+ write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
+}
+
+static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
{
write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
@@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
}
static hyp_alternate_select(__sysreg_call_restore_host_state,
- __sysreg_restore_state, __sysreg_do_nothing,
+ __sysreg_restore_el1_state, __sysreg_do_nothing,
ARM64_HAS_VIRT_HOST_EXTN);
void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
{
__sysreg_call_restore_host_state()(ctxt);
__sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
}
void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_restore_state(ctxt);
+ __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
}
static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 16/36] KVM: arm64: Rewrite sysreg alternatives to static keys
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
As we are about to move calls around in the sysreg save/restore logic,
let's first rewrite the alternative function callers, because it is
going to make the next patches much easier to read.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 17 ++++-------------
1 file changed, 4 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index bbfb4d01af88..e747f2fd31be 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -22,9 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-/* Yes, this does nothing, on purpose */
-static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
-
/*
* Non-VHE: Both host and guest must save everything.
*
@@ -78,13 +75,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-static hyp_alternate_select(__sysreg_call_save_host_state,
- __sysreg_save_el1_state, __sysreg_do_nothing,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_call_save_host_state()(ctxt);
+ if (!has_vhe())
+ __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
}
@@ -142,13 +136,10 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-static hyp_alternate_select(__sysreg_call_restore_host_state,
- __sysreg_restore_el1_state, __sysreg_do_nothing,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_call_restore_host_state()(ctxt);
+ if (!has_vhe())
+ __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 16/36] KVM: arm64: Rewrite sysreg alternatives to static keys
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
As we are about to move calls around in the sysreg save/restore logic,
let's first rewrite the alternative function callers, because it is
going to make the next patches much easier to read.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 17 ++++-------------
1 file changed, 4 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index bbfb4d01af88..e747f2fd31be 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -22,9 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-/* Yes, this does nothing, on purpose */
-static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
-
/*
* Non-VHE: Both host and guest must save everything.
*
@@ -78,13 +75,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-static hyp_alternate_select(__sysreg_call_save_host_state,
- __sysreg_save_el1_state, __sysreg_do_nothing,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_call_save_host_state()(ctxt);
+ if (!has_vhe())
+ __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
}
@@ -142,13 +136,10 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-static hyp_alternate_select(__sysreg_call_restore_host_state,
- __sysreg_restore_el1_state, __sysreg_do_nothing,
- ARM64_HAS_VIRT_HOST_EXTN);
-
void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
{
- __sysreg_call_restore_host_state()(ctxt);
+ if (!has_vhe())
+ __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 17/36] KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functions
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
As we are about to handle system registers quite differently between VHE
and non-VHE systems. In preparation for that, we need to split some of
the handling functions between VHE and non-VHE functionality.
For now, we simply copy the non-VHE functions, but we do change the use
of static keys for VHE and non-VHE functionality now that we have
separate functions.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 12 ++++++++----
arch/arm64/kvm/hyp/switch.c | 20 ++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 40 ++++++++++++++++++++++++++++++++--------
3 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index a0e5a7038237..998152da9b66 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -132,10 +132,14 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
void __timer_disable_traps(struct kvm_vcpu *vcpu);
-void __sysreg_save_host_state(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_host_state(struct kvm_cpu_context *ctxt);
-void __sysreg_save_guest_state(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt);
+void __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt);
void __sysreg32_save_state(struct kvm_vcpu *vcpu);
void __sysreg32_restore_state(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 09aafa0470f7..13c990c6eb64 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -352,7 +352,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state(host_ctxt);
+ sysreg_save_host_state_vhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(vcpu->kvm);
@@ -364,7 +364,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
+ sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -374,13 +374,13 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state(guest_ctxt);
+ sysreg_save_guest_state_vhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ sysreg_restore_host_state_vhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -404,7 +404,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state(host_ctxt);
+ __sysreg_save_host_state_nvhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
@@ -417,7 +417,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
+ __sysreg_restore_guest_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -427,7 +427,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state(guest_ctxt);
+ __sysreg_save_guest_state_nvhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
@@ -435,7 +435,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ __sysreg_restore_host_state_nvhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -460,7 +460,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state(__host_ctxt);
+ __sysreg_restore_host_state_nvhe(__host_ctxt);
}
/*
@@ -483,7 +483,7 @@ static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
vcpu = host_ctxt->__hyp_running_vcpu;
__deactivate_traps(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ sysreg_restore_host_state_vhe(host_ctxt);
panic(__hyp_panic_string,
spsr, elr,
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index e747f2fd31be..268c1cd25459 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -75,15 +75,27 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_save_el1_state(ctxt);
+ __sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
+}
+
+void __hyp_text __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_save_el1_state(ctxt);
+ __sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
+}
+
+void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
{
- if (!has_vhe())
- __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
}
-void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
+void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
@@ -136,15 +148,27 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_restore_el1_state(ctxt);
+ __sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
+}
+
+void __hyp_text __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_restore_el1_state(ctxt);
+ __sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
+}
+
+void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
{
- if (!has_vhe())
- __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
}
-void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
+void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 17/36] KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functions
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
As we are about to handle system registers quite differently between VHE
and non-VHE systems. In preparation for that, we need to split some of
the handling functions between VHE and non-VHE functionality.
For now, we simply copy the non-VHE functions, but we do change the use
of static keys for VHE and non-VHE functionality now that we have
separate functions.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 12 ++++++++----
arch/arm64/kvm/hyp/switch.c | 20 ++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 40 ++++++++++++++++++++++++++++++++--------
3 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index a0e5a7038237..998152da9b66 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -132,10 +132,14 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
void __timer_disable_traps(struct kvm_vcpu *vcpu);
-void __sysreg_save_host_state(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_host_state(struct kvm_cpu_context *ctxt);
-void __sysreg_save_guest_state(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt);
+void __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
+void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt);
void __sysreg32_save_state(struct kvm_vcpu *vcpu);
void __sysreg32_restore_state(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 09aafa0470f7..13c990c6eb64 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -352,7 +352,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state(host_ctxt);
+ sysreg_save_host_state_vhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(vcpu->kvm);
@@ -364,7 +364,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
+ sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -374,13 +374,13 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state(guest_ctxt);
+ sysreg_save_guest_state_vhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ sysreg_restore_host_state_vhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -404,7 +404,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state(host_ctxt);
+ __sysreg_save_host_state_nvhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
@@ -417,7 +417,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state(guest_ctxt);
+ __sysreg_restore_guest_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -427,7 +427,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state(guest_ctxt);
+ __sysreg_save_guest_state_nvhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
@@ -435,7 +435,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ __sysreg_restore_host_state_nvhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -460,7 +460,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state(__host_ctxt);
+ __sysreg_restore_host_state_nvhe(__host_ctxt);
}
/*
@@ -483,7 +483,7 @@ static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
vcpu = host_ctxt->__hyp_running_vcpu;
__deactivate_traps(vcpu);
- __sysreg_restore_host_state(host_ctxt);
+ sysreg_restore_host_state_vhe(host_ctxt);
panic(__hyp_panic_string,
spsr, elr,
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index e747f2fd31be..268c1cd25459 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -75,15 +75,27 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_save_el1_state(ctxt);
+ __sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
+}
+
+void __hyp_text __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_save_el1_state(ctxt);
+ __sysreg_save_common_state(ctxt);
+ __sysreg_save_user_state(ctxt);
+}
+
+void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
{
- if (!has_vhe())
- __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
}
-void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
+void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
@@ -136,15 +148,27 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_restore_el1_state(ctxt);
+ __sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
+}
+
+void __hyp_text __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+{
+ __sysreg_restore_el1_state(ctxt);
+ __sysreg_restore_common_state(ctxt);
+ __sysreg_restore_user_state(ctxt);
+}
+
+void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
{
- if (!has_vhe())
- __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
}
-void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
+void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 18/36] KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
The comment only applied to SPE on non-VHE systems, so we simply remove
it.
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 13c990c6eb64..9e5bc490ef0f 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -382,10 +382,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_restore_host_state_vhe(host_ctxt);
- /*
- * This must come after restoring the host sysregs, since a non-VHE
- * system may enable SPE here and make use of the TTBRs.
- */
__debug_switch_to_host(vcpu);
return exit_code;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 18/36] KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The comment only applied to SPE on non-VHE systems, so we simply remove
it.
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 13c990c6eb64..9e5bc490ef0f 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -382,10 +382,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_restore_host_state_vhe(host_ctxt);
- /*
- * This must come after restoring the host sysregs, since a non-VHE
- * system may enable SPE here and make use of the TTBRs.
- */
__debug_switch_to_host(vcpu);
return exit_code;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 19/36] KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
There is no need to have multiple identical functions with different
names for saving host and guest state. When saving and restoring state
for the host and guest, the state is the same for both contexts, and
that's why we have the kvm_cpu_context structure. Delete one
version and rename the other into simply save/restore.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 6 ++----
arch/arm64/kvm/hyp/switch.c | 10 +++++-----
arch/arm64/kvm/hyp/sysreg-sr.c | 18 ++----------------
3 files changed, 9 insertions(+), 25 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 998152da9b66..3f54c55f77a1 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -132,10 +132,8 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
void __timer_disable_traps(struct kvm_vcpu *vcpu);
-void __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt);
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 9e5bc490ef0f..c3e1a9c65bc1 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -400,7 +400,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state_nvhe(host_ctxt);
+ __sysreg_save_state_nvhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
@@ -413,7 +413,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state_nvhe(guest_ctxt);
+ __sysreg_restore_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -423,7 +423,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state_nvhe(guest_ctxt);
+ __sysreg_save_state_nvhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
@@ -431,7 +431,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state_nvhe(host_ctxt);
+ __sysreg_restore_state_nvhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -456,7 +456,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state_nvhe(__host_ctxt);
+ __sysreg_restore_state_nvhe(__host_ctxt);
}
/*
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 268c1cd25459..a12112494f75 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -75,14 +75,7 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-void __hyp_text __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt)
-{
- __sysreg_save_el1_state(ctxt);
- __sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
-}
-
-void __hyp_text __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
@@ -148,14 +141,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-void __hyp_text __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt)
-{
- __sysreg_restore_el1_state(ctxt);
- __sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
-}
-
-void __hyp_text __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 19/36] KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There is no need to have multiple identical functions with different
names for saving host and guest state. When saving and restoring state
for the host and guest, the state is the same for both contexts, and
that's why we have the kvm_cpu_context structure. Delete one
version and rename the other into simply save/restore.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 6 ++----
arch/arm64/kvm/hyp/switch.c | 10 +++++-----
arch/arm64/kvm/hyp/sysreg-sr.c | 18 ++----------------
3 files changed, 9 insertions(+), 25 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 998152da9b66..3f54c55f77a1 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -132,10 +132,8 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
void __timer_disable_traps(struct kvm_vcpu *vcpu);
-void __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt);
-void __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt);
+void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt);
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 9e5bc490ef0f..c3e1a9c65bc1 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -400,7 +400,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
- __sysreg_save_host_state_nvhe(host_ctxt);
+ __sysreg_save_state_nvhe(host_ctxt);
__activate_traps(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
@@ -413,7 +413,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
__sysreg32_restore_state(vcpu);
- __sysreg_restore_guest_state_nvhe(guest_ctxt);
+ __sysreg_restore_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
do {
@@ -423,7 +423,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
/* And we're baaack! */
} while (fixup_guest_exit(vcpu, &exit_code));
- __sysreg_save_guest_state_nvhe(guest_ctxt);
+ __sysreg_save_state_nvhe(guest_ctxt);
__sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
@@ -431,7 +431,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state_nvhe(host_ctxt);
+ __sysreg_restore_state_nvhe(host_ctxt);
/*
* This must come after restoring the host sysregs, since a non-VHE
@@ -456,7 +456,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
__timer_disable_traps(vcpu);
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
- __sysreg_restore_host_state_nvhe(__host_ctxt);
+ __sysreg_restore_state_nvhe(__host_ctxt);
}
/*
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 268c1cd25459..a12112494f75 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -75,14 +75,7 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
-void __hyp_text __sysreg_save_host_state_nvhe(struct kvm_cpu_context *ctxt)
-{
- __sysreg_save_el1_state(ctxt);
- __sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
-}
-
-void __hyp_text __sysreg_save_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
@@ -148,14 +141,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
-void __hyp_text __sysreg_restore_host_state_nvhe(struct kvm_cpu_context *ctxt)
-{
- __sysreg_restore_el1_state(ctxt);
- __sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
-}
-
-void __hyp_text __sysreg_restore_guest_state_nvhe(struct kvm_cpu_context *ctxt)
+void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
can return to the host in EL1 in the same state and location where we
issued a hypercall to EL2, but these registers don't contain anything
important on VHE, because all of the host runs in EL2. Therefore,
factor out these registers into separate save/restore functions, making
it easy to exclude them from the VHE world-switch path later on.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index a12112494f75..479de0f0dd07 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
+}
+
+static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
+{
ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
@@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
+ __sysreg_save_el2_return_state(ctxt);
}
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
@@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
+ __sysreg_save_el2_return_state(ctxt);
}
static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
@@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
+}
+
+static void __hyp_text
+__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
+{
write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
@@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
+ __sysreg_restore_el2_return_state(ctxt);
}
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
@@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
+ __sysreg_restore_el2_return_state(ctxt);
}
static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
can return to the host in EL1 in the same state and location where we
issued a hypercall to EL2, but these registers don't contain anything
important on VHE, because all of the host runs in EL2. Therefore,
factor out these registers into separate save/restore functions, making
it easy to exclude them from the VHE world-switch path later on.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index a12112494f75..479de0f0dd07 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
+}
+
+static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
+{
ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
@@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
+ __sysreg_save_el2_return_state(ctxt);
}
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
@@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
__sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
__sysreg_save_user_state(ctxt);
+ __sysreg_save_el2_return_state(ctxt);
}
static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
@@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
+}
+
+static void __hyp_text
+__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
+{
write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
@@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
+ __sysreg_restore_el2_return_state(ctxt);
}
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
@@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
__sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
__sysreg_restore_user_state(ctxt);
+ __sysreg_restore_el2_return_state(ctxt);
}
static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 21/36] KVM: arm64: Change 32-bit handling of VM system registers
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
We currently handle 32-bit accesses to trapped VM system registers using
the 32-bit index into the coproc array on the vcpu structure, which is a
union of the coproc array and the sysreg array.
Since all the 32-bit coproc indicies are created to correspond to the
architectural mapping between 64-bit system registers and 32-bit
coprocessor registers, and because the AArch64 system registers are the
double in size of the AArch32 coprocessor registers, we can always find
the system register entry that we must update by dividing the 32-bit
coproc index by 2.
This is going to make our lives much easier when we have to start
accessing system registers that use deferred save/restore and might
have to be read directly from the physical CPU.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_host.h | 8 --------
arch/arm64/kvm/sys_regs.c | 20 +++++++++++++++-----
2 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index c841eeeeb5c5..de0d55b30b61 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -290,14 +290,6 @@ struct kvm_vcpu_arch {
#define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
#define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
-#ifdef CONFIG_CPU_BIG_ENDIAN
-#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r))
-#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r) + 1)
-#else
-#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r) + 1)
-#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r))
-#endif
-
struct kvm_vm_stat {
ulong remote_tlb_flush;
};
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 1830ebc227d1..62c12ab9e6c4 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -121,16 +121,26 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *r)
{
bool was_enabled = vcpu_has_cache_enabled(vcpu);
+ u64 val;
+ int reg = r->reg;
BUG_ON(!p->is_write);
- if (!p->is_aarch32) {
- vcpu_sys_reg(vcpu, r->reg) = p->regval;
+ /* See the 32bit mapping in kvm_host.h */
+ if (p->is_aarch32)
+ reg = r->reg / 2;
+
+ if (!p->is_aarch32 || !p->is_32bit) {
+ val = p->regval;
} else {
- if (!p->is_32bit)
- vcpu_cp15_64_high(vcpu, r->reg) = upper_32_bits(p->regval);
- vcpu_cp15_64_low(vcpu, r->reg) = lower_32_bits(p->regval);
+ val = vcpu_sys_reg(vcpu, reg);
+ if (r->reg % 2)
+ val = (p->regval << 32) | (u64)lower_32_bits(val);
+ else
+ val = ((u64)upper_32_bits(val) << 32) |
+ (u64)lower_32_bits(p->regval);
}
+ vcpu_sys_reg(vcpu, reg) = val;
kvm_toggle_cache(vcpu, was_enabled);
return true;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 21/36] KVM: arm64: Change 32-bit handling of VM system registers
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
We currently handle 32-bit accesses to trapped VM system registers using
the 32-bit index into the coproc array on the vcpu structure, which is a
union of the coproc array and the sysreg array.
Since all the 32-bit coproc indicies are created to correspond to the
architectural mapping between 64-bit system registers and 32-bit
coprocessor registers, and because the AArch64 system registers are the
double in size of the AArch32 coprocessor registers, we can always find
the system register entry that we must update by dividing the 32-bit
coproc index by 2.
This is going to make our lives much easier when we have to start
accessing system registers that use deferred save/restore and might
have to be read directly from the physical CPU.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_host.h | 8 --------
arch/arm64/kvm/sys_regs.c | 20 +++++++++++++++-----
2 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index c841eeeeb5c5..de0d55b30b61 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -290,14 +290,6 @@ struct kvm_vcpu_arch {
#define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
#define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
-#ifdef CONFIG_CPU_BIG_ENDIAN
-#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r))
-#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r) + 1)
-#else
-#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r) + 1)
-#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r))
-#endif
-
struct kvm_vm_stat {
ulong remote_tlb_flush;
};
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 1830ebc227d1..62c12ab9e6c4 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -121,16 +121,26 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *r)
{
bool was_enabled = vcpu_has_cache_enabled(vcpu);
+ u64 val;
+ int reg = r->reg;
BUG_ON(!p->is_write);
- if (!p->is_aarch32) {
- vcpu_sys_reg(vcpu, r->reg) = p->regval;
+ /* See the 32bit mapping in kvm_host.h */
+ if (p->is_aarch32)
+ reg = r->reg / 2;
+
+ if (!p->is_aarch32 || !p->is_32bit) {
+ val = p->regval;
} else {
- if (!p->is_32bit)
- vcpu_cp15_64_high(vcpu, r->reg) = upper_32_bits(p->regval);
- vcpu_cp15_64_low(vcpu, r->reg) = lower_32_bits(p->regval);
+ val = vcpu_sys_reg(vcpu, reg);
+ if (r->reg % 2)
+ val = (p->regval << 32) | (u64)lower_32_bits(val);
+ else
+ val = ((u64)upper_32_bits(val) << 32) |
+ (u64)lower_32_bits(p->regval);
}
+ vcpu_sys_reg(vcpu, reg) = val;
kvm_toggle_cache(vcpu, was_enabled);
return true;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
When we defer the save/restore of system registers to vcpu_load and
vcpu_put, we need to take care of the emulation code that handles traps
to these registers, since simply reading the memory array will return
stale data.
Therefore, introduce two functions to directly read/write the registers
from the physical CPU when we're on a VHE system that has loaded the
system registers onto the physical CPU.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Removed spurious white space
arch/arm64/include/asm/kvm_host.h | 4 +++
arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index de0d55b30b61..f6afe685a280 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
/* Detect first run of a vcpu */
bool has_run_once;
+
+ /* True when deferrable sysregs are loaded on the physical CPU,
+ * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
+ bool sysregs_loaded_on_cpu;
};
#define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 62c12ab9e6c4..80adbec933de 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -35,6 +35,7 @@
#include <asm/kvm_coproc.h>
#include <asm/kvm_emulate.h>
#include <asm/kvm_host.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
#include <asm/perf_event.h>
#include <asm/sysreg.h>
@@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
return true;
}
+static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu) {
+ switch (reg) {
+ case SCTLR_EL1: return read_sysreg_el1(sctlr);
+ case TTBR0_EL1: return read_sysreg_el1(ttbr0);
+ case TTBR1_EL1: return read_sysreg_el1(ttbr1);
+ case TCR_EL1: return read_sysreg_el1(tcr);
+ case ESR_EL1: return read_sysreg_el1(esr);
+ case FAR_EL1: return read_sysreg_el1(far);
+ case AFSR0_EL1: return read_sysreg_el1(afsr0);
+ case AFSR1_EL1: return read_sysreg_el1(afsr1);
+ case MAIR_EL1: return read_sysreg_el1(mair);
+ case AMAIR_EL1: return read_sysreg_el1(amair);
+ case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
+ case DACR32_EL2: return read_sysreg(dacr32_el2);
+ case IFSR32_EL2: return read_sysreg(ifsr32_el2);
+ default: BUG();
+ }
+ }
+
+ return vcpu_sys_reg(vcpu, reg);
+}
+
+static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu) {
+ switch (reg) {
+ case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
+ case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
+ case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
+ case TCR_EL1: write_sysreg_el1(val, tcr); return;
+ case ESR_EL1: write_sysreg_el1(val, esr); return;
+ case FAR_EL1: write_sysreg_el1(val, far); return;
+ case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
+ case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
+ case MAIR_EL1: write_sysreg_el1(val, mair); return;
+ case AMAIR_EL1: write_sysreg_el1(val, amair); return;
+ case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
+ case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
+ case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
+ default: BUG();
+ }
+ }
+
+ vcpu_sys_reg(vcpu, reg) = val;
+}
+
/*
* Generic accessor for VM registers. Only called as long as HCR_TVM
* is set. If the guest enables the MMU, we stop trapping the VM
@@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
if (!p->is_aarch32 || !p->is_32bit) {
val = p->regval;
} else {
- val = vcpu_sys_reg(vcpu, reg);
+ val = read_deferrable_vm_reg(vcpu, reg);
if (r->reg % 2)
val = (p->regval << 32) | (u64)lower_32_bits(val);
else
val = ((u64)upper_32_bits(val) << 32) |
(u64)lower_32_bits(p->regval);
}
- vcpu_sys_reg(vcpu, reg) = val;
+ write_deferrable_vm_reg(vcpu, reg, val);
kvm_toggle_cache(vcpu, was_enabled);
return true;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
When we defer the save/restore of system registers to vcpu_load and
vcpu_put, we need to take care of the emulation code that handles traps
to these registers, since simply reading the memory array will return
stale data.
Therefore, introduce two functions to directly read/write the registers
from the physical CPU when we're on a VHE system that has loaded the
system registers onto the physical CPU.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Removed spurious white space
arch/arm64/include/asm/kvm_host.h | 4 +++
arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index de0d55b30b61..f6afe685a280 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
/* Detect first run of a vcpu */
bool has_run_once;
+
+ /* True when deferrable sysregs are loaded on the physical CPU,
+ * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
+ bool sysregs_loaded_on_cpu;
};
#define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 62c12ab9e6c4..80adbec933de 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -35,6 +35,7 @@
#include <asm/kvm_coproc.h>
#include <asm/kvm_emulate.h>
#include <asm/kvm_host.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
#include <asm/perf_event.h>
#include <asm/sysreg.h>
@@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
return true;
}
+static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu) {
+ switch (reg) {
+ case SCTLR_EL1: return read_sysreg_el1(sctlr);
+ case TTBR0_EL1: return read_sysreg_el1(ttbr0);
+ case TTBR1_EL1: return read_sysreg_el1(ttbr1);
+ case TCR_EL1: return read_sysreg_el1(tcr);
+ case ESR_EL1: return read_sysreg_el1(esr);
+ case FAR_EL1: return read_sysreg_el1(far);
+ case AFSR0_EL1: return read_sysreg_el1(afsr0);
+ case AFSR1_EL1: return read_sysreg_el1(afsr1);
+ case MAIR_EL1: return read_sysreg_el1(mair);
+ case AMAIR_EL1: return read_sysreg_el1(amair);
+ case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
+ case DACR32_EL2: return read_sysreg(dacr32_el2);
+ case IFSR32_EL2: return read_sysreg(ifsr32_el2);
+ default: BUG();
+ }
+ }
+
+ return vcpu_sys_reg(vcpu, reg);
+}
+
+static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu) {
+ switch (reg) {
+ case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
+ case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
+ case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
+ case TCR_EL1: write_sysreg_el1(val, tcr); return;
+ case ESR_EL1: write_sysreg_el1(val, esr); return;
+ case FAR_EL1: write_sysreg_el1(val, far); return;
+ case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
+ case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
+ case MAIR_EL1: write_sysreg_el1(val, mair); return;
+ case AMAIR_EL1: write_sysreg_el1(val, amair); return;
+ case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
+ case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
+ case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
+ default: BUG();
+ }
+ }
+
+ vcpu_sys_reg(vcpu, reg) = val;
+}
+
/*
* Generic accessor for VM registers. Only called as long as HCR_TVM
* is set. If the guest enables the MMU, we stop trapping the VM
@@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
if (!p->is_aarch32 || !p->is_32bit) {
val = p->regval;
} else {
- val = vcpu_sys_reg(vcpu, reg);
+ val = read_deferrable_vm_reg(vcpu, reg);
if (r->reg % 2)
val = (p->regval << 32) | (u64)lower_32_bits(val);
else
val = ((u64)upper_32_bits(val) << 32) |
(u64)lower_32_bits(p->regval);
}
- vcpu_sys_reg(vcpu, reg) = val;
+ write_deferrable_vm_reg(vcpu, reg, val);
kvm_toggle_cache(vcpu, was_enabled);
return true;
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 23/36] KVM: arm64: Prepare to handle traps on deferred EL0 sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
We can trap access to ACTLR_EL1 which we can later defer to only
save/restore during vcpu_load and vcpu_put, so let's read the value
directly from the CPU when necessary.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fix bug in access_actlr that read the actlr_el1 and threw away the
value. Whoops.
arch/arm64/kvm/sys_regs_generic_v8.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/kvm/sys_regs_generic_v8.c b/arch/arm64/kvm/sys_regs_generic_v8.c
index 969ade1d333d..8df4b4301696 100644
--- a/arch/arm64/kvm/sys_regs_generic_v8.c
+++ b/arch/arm64/kvm/sys_regs_generic_v8.c
@@ -38,7 +38,10 @@ static bool access_actlr(struct kvm_vcpu *vcpu,
if (p->is_write)
return ignore_write(vcpu, p);
- p->regval = vcpu_sys_reg(vcpu, ACTLR_EL1);
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ p->regval = read_sysreg(actlr_el1);
+ else
+ p->regval = vcpu_sys_reg(vcpu, ACTLR_EL1);
return true;
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 23/36] KVM: arm64: Prepare to handle traps on deferred EL0 sysregs
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
We can trap access to ACTLR_EL1 which we can later defer to only
save/restore during vcpu_load and vcpu_put, so let's read the value
directly from the CPU when necessary.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Fix bug in access_actlr that read the actlr_el1 and threw away the
value. Whoops.
arch/arm64/kvm/sys_regs_generic_v8.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/kvm/sys_regs_generic_v8.c b/arch/arm64/kvm/sys_regs_generic_v8.c
index 969ade1d333d..8df4b4301696 100644
--- a/arch/arm64/kvm/sys_regs_generic_v8.c
+++ b/arch/arm64/kvm/sys_regs_generic_v8.c
@@ -38,7 +38,10 @@ static bool access_actlr(struct kvm_vcpu *vcpu,
if (p->is_write)
return ignore_write(vcpu, p);
- p->regval = vcpu_sys_reg(vcpu, ACTLR_EL1);
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ p->regval = read_sysreg(actlr_el1);
+ else
+ p->regval = vcpu_sys_reg(vcpu, ACTLR_EL1);
return true;
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 24/36] KVM: arm64: Prepare to handle traps on remaining deferred EL1 sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
Handle accesses during traps to any remaining EL1 registers which can be
deferred to vcpu_load and vcpu_put, by either accessing them directly on
the physical CPU when the latest version is stored there, or by
synchronizing the memory representation with the CPU state.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_emulate.h | 16 ++++++++++++
arch/arm/include/asm/kvm_host.h | 2 ++
arch/arm64/include/asm/kvm_emulate.h | 49 +++++++++++++++++++++++++-----------
arch/arm64/include/asm/kvm_host.h | 2 ++
arch/arm64/kvm/inject_fault.c | 19 ++++++++++----
arch/arm64/kvm/sys_regs.c | 6 ++++-
virt/kvm/arm/aarch32.c | 22 +++++++++++++---
7 files changed, 93 insertions(+), 23 deletions(-)
diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h
index d5e1b8bf6422..47efa953460a 100644
--- a/arch/arm/include/asm/kvm_emulate.h
+++ b/arch/arm/include/asm/kvm_emulate.h
@@ -55,6 +55,22 @@ static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
*vcpu_reg(vcpu, reg_num) = val;
}
+/* Set the SPSR for the current mode */
+static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, unsigned long val)
+{
+ *vcpu_spsr(vcpu) = val;
+}
+
+static inline unsigned long vcpu_get_vbar(struct kvm_vcpu *vcpu)
+{
+ return vcpu_cp15(vcpu, c12_VBAR);
+}
+
+static inline u32 vcpu_get_c1_sctlr(struct kvm_vcpu *vcpu)
+{
+ return vcpu_cp15(vcpu, c1_SCTLR);
+}
+
bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
void kvm_skip_instr32(struct kvm_vcpu *vcpu, bool is_wide_instr);
void kvm_inject_undef32(struct kvm_vcpu *vcpu);
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 8fce576199e0..997c0568bfa3 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -203,6 +203,8 @@ struct kvm_vcpu_stat {
#define vcpu_cp15(v,r) (v)->arch.ctxt.cp15[r]
+#define vcpu_sysregs_loaded(_v) (false)
+
int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 635137e6ed1c..3f765b9de94d 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -26,6 +26,7 @@
#include <asm/esr.h>
#include <asm/kvm_arm.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmio.h>
#include <asm/ptrace.h>
#include <asm/cputype.h>
@@ -77,11 +78,6 @@ static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
}
-static inline unsigned long *vcpu_elr_el1(const struct kvm_vcpu *vcpu)
-{
- return (unsigned long *)&vcpu_gp_regs(vcpu)->elr_el1;
-}
-
static inline unsigned long *vcpu_cpsr(const struct kvm_vcpu *vcpu)
{
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pstate;
@@ -92,6 +88,40 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
return !!(*vcpu_cpsr(vcpu) & PSR_MODE32_BIT);
}
+/* Set the SPSR for the current mode */
+static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
+{
+ if (vcpu_mode_is_32bit(vcpu))
+ *vcpu_spsr32(vcpu) = val;
+
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ write_sysreg_el1(val, spsr);
+ else
+ vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1] = val;
+}
+
+static inline unsigned long vcpu_get_vbar(struct kvm_vcpu *vcpu)
+{
+ unsigned long vbar;
+
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ vbar = read_sysreg_el1(vbar);
+ else
+ vbar = vcpu_sys_reg(vcpu, VBAR_EL1);
+
+ if (vcpu_el1_is_32bit(vcpu))
+ return lower_32_bits(vbar);
+ return vbar;
+}
+
+static inline u32 vcpu_get_c1_sctlr(struct kvm_vcpu *vcpu)
+{
+ if (vcpu_sysregs_loaded(vcpu))
+ return lower_32_bits(read_sysreg_el1(sctlr));
+ else
+ return vcpu_cp15(vcpu, c1_SCTLR);
+}
+
static inline bool kvm_condition_valid(const struct kvm_vcpu *vcpu)
{
if (vcpu_mode_is_32bit(vcpu))
@@ -131,15 +161,6 @@ static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
vcpu_gp_regs(vcpu)->regs.regs[reg_num] = val;
}
-/* Get vcpu SPSR for current mode */
-static inline unsigned long *vcpu_spsr(const struct kvm_vcpu *vcpu)
-{
- if (vcpu_mode_is_32bit(vcpu))
- return vcpu_spsr32(vcpu);
-
- return (unsigned long *)&vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1];
-}
-
static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)
{
u32 mode;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index f6afe685a280..992c19816893 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -294,6 +294,8 @@ struct kvm_vcpu_arch {
#define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
#define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
+#define vcpu_sysregs_loaded(_v) ((_v)->arch.sysregs_loaded_on_cpu)
+
struct kvm_vm_stat {
ulong remote_tlb_flush;
};
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index 2d38ede2eff0..1d941e8e8102 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -23,6 +23,7 @@
#include <linux/kvm_host.h>
#include <asm/kvm_emulate.h>
+#include <asm/kvm_hyp.h>
#include <asm/esr.h>
#define PSTATE_FAULT_BITS_64 (PSR_MODE_EL1h | PSR_A_BIT | PSR_F_BIT | \
@@ -33,6 +34,14 @@
#define LOWER_EL_AArch64_VECTOR 0x400
#define LOWER_EL_AArch32_VECTOR 0x600
+static void vcpu_set_elr_el1(struct kvm_vcpu *vcpu, u64 val)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ write_sysreg_el1(val, elr);
+ else
+ vcpu_gp_regs(vcpu)->elr_el1 = val;
+}
+
enum exception_type {
except_type_sync = 0,
except_type_irq = 0x80,
@@ -58,7 +67,7 @@ static u64 get_except_vector(struct kvm_vcpu *vcpu, enum exception_type type)
exc_offset = LOWER_EL_AArch32_VECTOR;
}
- return vcpu_sys_reg(vcpu, VBAR_EL1) + exc_offset + type;
+ return vcpu_get_vbar(vcpu) + exc_offset + type;
}
static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
@@ -67,11 +76,11 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr
bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
u32 esr = 0;
- *vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ vcpu_set_elr_el1(vcpu, *vcpu_pc(vcpu));
*vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_spsr(vcpu) = cpsr;
+ vcpu_set_spsr(vcpu, cpsr);
vcpu_sys_reg(vcpu, FAR_EL1) = addr;
@@ -102,11 +111,11 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
unsigned long cpsr = *vcpu_cpsr(vcpu);
u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
- *vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ vcpu_set_elr_el1(vcpu, *vcpu_pc(vcpu));
*vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_spsr(vcpu) = cpsr;
+ vcpu_set_spsr(vcpu, cpsr);
/*
* Build an unknown exception, depending on the instruction
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 80adbec933de..6109dc8d5fb7 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -87,12 +87,16 @@ static u32 cache_levels;
static u32 get_ccsidr(u32 csselr)
{
u32 ccsidr;
+ u32 csselr_preserve;
- /* Make sure noone else changes CSSELR during this! */
+ /* Make sure noone else changes CSSELR during this and preserve any
+ * existing value in the CSSELR! */
local_irq_disable();
+ csselr_preserve = read_sysreg(csselr_el1);
write_sysreg(csselr, csselr_el1);
isb();
ccsidr = read_sysreg(ccsidr_el1);
+ write_sysreg(csselr_preserve, csselr_el1);
local_irq_enable();
return ccsidr;
diff --git a/virt/kvm/arm/aarch32.c b/virt/kvm/arm/aarch32.c
index 8bc479fa37e6..67b62ff79b6f 100644
--- a/virt/kvm/arm/aarch32.c
+++ b/virt/kvm/arm/aarch32.c
@@ -166,7 +166,7 @@ static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
unsigned long new_spsr_value = *vcpu_cpsr(vcpu);
bool is_thumb = (new_spsr_value & COMPAT_PSR_T_BIT);
u32 return_offset = return_offsets[vect_offset >> 2][is_thumb];
- u32 sctlr = vcpu_cp15(vcpu, c1_SCTLR);
+ u32 sctlr = vcpu_get_c1_sctlr(vcpu);
cpsr = mode | COMPAT_PSR_I_BIT;
@@ -178,14 +178,14 @@ static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
*vcpu_cpsr(vcpu) = cpsr;
/* Note: These now point to the banked copies */
- *vcpu_spsr(vcpu) = new_spsr_value;
+ vcpu_set_spsr(vcpu, new_spsr_value);
*vcpu_reg32(vcpu, 14) = *vcpu_pc(vcpu) + return_offset;
/* Branch to exception vector */
if (sctlr & (1 << 13))
vect_offset += 0xffff0000;
else /* always have security exceptions */
- vect_offset += vcpu_cp15(vcpu, c12_VBAR);
+ vect_offset += vcpu_get_vbar(vcpu);
*vcpu_pc(vcpu) = vect_offset;
}
@@ -206,6 +206,19 @@ static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt,
u32 *far, *fsr;
bool is_lpae;
+ /*
+ * The emulation code here is going to modify several system
+ * registers, so on arm64 with VHE we want to load them into memory
+ * and store them back into registers, ensuring that we observe the
+ * most recent values and that we expose the right values back to the
+ * guest.
+ *
+ * We disable preemption to avoid racing with another vcpu_put/load
+ * operation.
+ */
+ preempt_disable();
+ kvm_vcpu_put_sysregs(vcpu);
+
if (is_pabt) {
vect_offset = 12;
far = &vcpu_cp15(vcpu, c6_IFAR);
@@ -226,6 +239,9 @@ static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt,
*fsr = 1 << 9 | 0x34;
else
*fsr = 0x14;
+
+ kvm_vcpu_load_sysregs(vcpu);
+ preempt_enable();
}
void kvm_inject_dabt32(struct kvm_vcpu *vcpu, unsigned long addr)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 24/36] KVM: arm64: Prepare to handle traps on remaining deferred EL1 sysregs
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Handle accesses during traps to any remaining EL1 registers which can be
deferred to vcpu_load and vcpu_put, by either accessing them directly on
the physical CPU when the latest version is stored there, or by
synchronizing the memory representation with the CPU state.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_emulate.h | 16 ++++++++++++
arch/arm/include/asm/kvm_host.h | 2 ++
arch/arm64/include/asm/kvm_emulate.h | 49 +++++++++++++++++++++++++-----------
arch/arm64/include/asm/kvm_host.h | 2 ++
arch/arm64/kvm/inject_fault.c | 19 ++++++++++----
arch/arm64/kvm/sys_regs.c | 6 ++++-
virt/kvm/arm/aarch32.c | 22 +++++++++++++---
7 files changed, 93 insertions(+), 23 deletions(-)
diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h
index d5e1b8bf6422..47efa953460a 100644
--- a/arch/arm/include/asm/kvm_emulate.h
+++ b/arch/arm/include/asm/kvm_emulate.h
@@ -55,6 +55,22 @@ static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
*vcpu_reg(vcpu, reg_num) = val;
}
+/* Set the SPSR for the current mode */
+static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, unsigned long val)
+{
+ *vcpu_spsr(vcpu) = val;
+}
+
+static inline unsigned long vcpu_get_vbar(struct kvm_vcpu *vcpu)
+{
+ return vcpu_cp15(vcpu, c12_VBAR);
+}
+
+static inline u32 vcpu_get_c1_sctlr(struct kvm_vcpu *vcpu)
+{
+ return vcpu_cp15(vcpu, c1_SCTLR);
+}
+
bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
void kvm_skip_instr32(struct kvm_vcpu *vcpu, bool is_wide_instr);
void kvm_inject_undef32(struct kvm_vcpu *vcpu);
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 8fce576199e0..997c0568bfa3 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -203,6 +203,8 @@ struct kvm_vcpu_stat {
#define vcpu_cp15(v,r) (v)->arch.ctxt.cp15[r]
+#define vcpu_sysregs_loaded(_v) (false)
+
int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 635137e6ed1c..3f765b9de94d 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -26,6 +26,7 @@
#include <asm/esr.h>
#include <asm/kvm_arm.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmio.h>
#include <asm/ptrace.h>
#include <asm/cputype.h>
@@ -77,11 +78,6 @@ static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
}
-static inline unsigned long *vcpu_elr_el1(const struct kvm_vcpu *vcpu)
-{
- return (unsigned long *)&vcpu_gp_regs(vcpu)->elr_el1;
-}
-
static inline unsigned long *vcpu_cpsr(const struct kvm_vcpu *vcpu)
{
return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pstate;
@@ -92,6 +88,40 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
return !!(*vcpu_cpsr(vcpu) & PSR_MODE32_BIT);
}
+/* Set the SPSR for the current mode */
+static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
+{
+ if (vcpu_mode_is_32bit(vcpu))
+ *vcpu_spsr32(vcpu) = val;
+
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ write_sysreg_el1(val, spsr);
+ else
+ vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1] = val;
+}
+
+static inline unsigned long vcpu_get_vbar(struct kvm_vcpu *vcpu)
+{
+ unsigned long vbar;
+
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ vbar = read_sysreg_el1(vbar);
+ else
+ vbar = vcpu_sys_reg(vcpu, VBAR_EL1);
+
+ if (vcpu_el1_is_32bit(vcpu))
+ return lower_32_bits(vbar);
+ return vbar;
+}
+
+static inline u32 vcpu_get_c1_sctlr(struct kvm_vcpu *vcpu)
+{
+ if (vcpu_sysregs_loaded(vcpu))
+ return lower_32_bits(read_sysreg_el1(sctlr));
+ else
+ return vcpu_cp15(vcpu, c1_SCTLR);
+}
+
static inline bool kvm_condition_valid(const struct kvm_vcpu *vcpu)
{
if (vcpu_mode_is_32bit(vcpu))
@@ -131,15 +161,6 @@ static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
vcpu_gp_regs(vcpu)->regs.regs[reg_num] = val;
}
-/* Get vcpu SPSR for current mode */
-static inline unsigned long *vcpu_spsr(const struct kvm_vcpu *vcpu)
-{
- if (vcpu_mode_is_32bit(vcpu))
- return vcpu_spsr32(vcpu);
-
- return (unsigned long *)&vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1];
-}
-
static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)
{
u32 mode;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index f6afe685a280..992c19816893 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -294,6 +294,8 @@ struct kvm_vcpu_arch {
#define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
#define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
+#define vcpu_sysregs_loaded(_v) ((_v)->arch.sysregs_loaded_on_cpu)
+
struct kvm_vm_stat {
ulong remote_tlb_flush;
};
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index 2d38ede2eff0..1d941e8e8102 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -23,6 +23,7 @@
#include <linux/kvm_host.h>
#include <asm/kvm_emulate.h>
+#include <asm/kvm_hyp.h>
#include <asm/esr.h>
#define PSTATE_FAULT_BITS_64 (PSR_MODE_EL1h | PSR_A_BIT | PSR_F_BIT | \
@@ -33,6 +34,14 @@
#define LOWER_EL_AArch64_VECTOR 0x400
#define LOWER_EL_AArch32_VECTOR 0x600
+static void vcpu_set_elr_el1(struct kvm_vcpu *vcpu, u64 val)
+{
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ write_sysreg_el1(val, elr);
+ else
+ vcpu_gp_regs(vcpu)->elr_el1 = val;
+}
+
enum exception_type {
except_type_sync = 0,
except_type_irq = 0x80,
@@ -58,7 +67,7 @@ static u64 get_except_vector(struct kvm_vcpu *vcpu, enum exception_type type)
exc_offset = LOWER_EL_AArch32_VECTOR;
}
- return vcpu_sys_reg(vcpu, VBAR_EL1) + exc_offset + type;
+ return vcpu_get_vbar(vcpu) + exc_offset + type;
}
static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
@@ -67,11 +76,11 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr
bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
u32 esr = 0;
- *vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ vcpu_set_elr_el1(vcpu, *vcpu_pc(vcpu));
*vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_spsr(vcpu) = cpsr;
+ vcpu_set_spsr(vcpu, cpsr);
vcpu_sys_reg(vcpu, FAR_EL1) = addr;
@@ -102,11 +111,11 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
unsigned long cpsr = *vcpu_cpsr(vcpu);
u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
- *vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ vcpu_set_elr_el1(vcpu, *vcpu_pc(vcpu));
*vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_spsr(vcpu) = cpsr;
+ vcpu_set_spsr(vcpu, cpsr);
/*
* Build an unknown exception, depending on the instruction
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 80adbec933de..6109dc8d5fb7 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -87,12 +87,16 @@ static u32 cache_levels;
static u32 get_ccsidr(u32 csselr)
{
u32 ccsidr;
+ u32 csselr_preserve;
- /* Make sure noone else changes CSSELR during this! */
+ /* Make sure noone else changes CSSELR during this and preserve any
+ * existing value in the CSSELR! */
local_irq_disable();
+ csselr_preserve = read_sysreg(csselr_el1);
write_sysreg(csselr, csselr_el1);
isb();
ccsidr = read_sysreg(ccsidr_el1);
+ write_sysreg(csselr_preserve, csselr_el1);
local_irq_enable();
return ccsidr;
diff --git a/virt/kvm/arm/aarch32.c b/virt/kvm/arm/aarch32.c
index 8bc479fa37e6..67b62ff79b6f 100644
--- a/virt/kvm/arm/aarch32.c
+++ b/virt/kvm/arm/aarch32.c
@@ -166,7 +166,7 @@ static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
unsigned long new_spsr_value = *vcpu_cpsr(vcpu);
bool is_thumb = (new_spsr_value & COMPAT_PSR_T_BIT);
u32 return_offset = return_offsets[vect_offset >> 2][is_thumb];
- u32 sctlr = vcpu_cp15(vcpu, c1_SCTLR);
+ u32 sctlr = vcpu_get_c1_sctlr(vcpu);
cpsr = mode | COMPAT_PSR_I_BIT;
@@ -178,14 +178,14 @@ static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
*vcpu_cpsr(vcpu) = cpsr;
/* Note: These now point to the banked copies */
- *vcpu_spsr(vcpu) = new_spsr_value;
+ vcpu_set_spsr(vcpu, new_spsr_value);
*vcpu_reg32(vcpu, 14) = *vcpu_pc(vcpu) + return_offset;
/* Branch to exception vector */
if (sctlr & (1 << 13))
vect_offset += 0xffff0000;
else /* always have security exceptions */
- vect_offset += vcpu_cp15(vcpu, c12_VBAR);
+ vect_offset += vcpu_get_vbar(vcpu);
*vcpu_pc(vcpu) = vect_offset;
}
@@ -206,6 +206,19 @@ static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt,
u32 *far, *fsr;
bool is_lpae;
+ /*
+ * The emulation code here is going to modify several system
+ * registers, so on arm64 with VHE we want to load them into memory
+ * and store them back into registers, ensuring that we observe the
+ * most recent values and that we expose the right values back to the
+ * guest.
+ *
+ * We disable preemption to avoid racing with another vcpu_put/load
+ * operation.
+ */
+ preempt_disable();
+ kvm_vcpu_put_sysregs(vcpu);
+
if (is_pabt) {
vect_offset = 12;
far = &vcpu_cp15(vcpu, c6_IFAR);
@@ -226,6 +239,9 @@ static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt,
*fsr = 1 << 9 | 0x34;
else
*fsr = 0x14;
+
+ kvm_vcpu_load_sysregs(vcpu);
+ preempt_enable();
}
void kvm_inject_dabt32(struct kvm_vcpu *vcpu, unsigned long addr)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 25/36] KVM: arm64: Prepare to handle traps on deferred AArch32 sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
Handle accesses to any AArch32 EL1 system registers where we can defer
saving and restoring them to vcpu_load and vcpu_put, and which are
stored in special EL2 registers only used support 32-bit guests.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_emulate.h | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 3f765b9de94d..e7234f823cba 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -91,9 +91,16 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
/* Set the SPSR for the current mode */
static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
{
- if (vcpu_mode_is_32bit(vcpu))
+ if (vcpu_mode_is_32bit(vcpu)) {
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ __sysreg32_save_state(vcpu);
+
*vcpu_spsr32(vcpu) = val;
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ __sysreg32_restore_state(vcpu);
+ }
+
if (vcpu->arch.sysregs_loaded_on_cpu)
write_sysreg_el1(val, spsr);
else
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 25/36] KVM: arm64: Prepare to handle traps on deferred AArch32 sysregs
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Handle accesses to any AArch32 EL1 system registers where we can defer
saving and restoring them to vcpu_load and vcpu_put, and which are
stored in special EL2 registers only used support 32-bit guests.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_emulate.h | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 3f765b9de94d..e7234f823cba 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -91,9 +91,16 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
/* Set the SPSR for the current mode */
static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
{
- if (vcpu_mode_is_32bit(vcpu))
+ if (vcpu_mode_is_32bit(vcpu)) {
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ __sysreg32_save_state(vcpu);
+
*vcpu_spsr32(vcpu) = val;
+ if (vcpu->arch.sysregs_loaded_on_cpu)
+ __sysreg32_restore_state(vcpu);
+ }
+
if (vcpu->arch.sysregs_loaded_on_cpu)
write_sysreg_el1(val, spsr);
else
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
Some system registers do not affect the host kernel's execution and can
therefore be loaded when we are about to run a VCPU and we don't have to
restore the host state to the hardware before the time when we are
actually about to return to userspace or schedule out the VCPU thread.
The EL1 system registers and the userspace state registers, which only
affect EL0 execution, do not affect the host kernel's execution.
The 32-bit system registers are not used by a VHE host kernel and
therefore don't need to be saved/restored on every entry/exit to/from
the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
We have already prepared the trap handling code which accesses any of
these registers to directly access the registers on the physical CPU or
to sync the registers when needed.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 6 ------
arch/arm64/kvm/hyp/sysreg-sr.c | 46 ++++++++++++++++++++++++++++++++++--------
2 files changed, 38 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index c3e1a9c65bc1..2ac8af354de0 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -359,11 +359,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__vgic_restore_state(vcpu);
- /*
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -375,7 +370,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
sysreg_save_guest_state_vhe(guest_ctxt);
- __sysreg32_save_state(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 479de0f0dd07..65abf1aeba59 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -25,8 +25,12 @@
/*
* Non-VHE: Both host and guest must save everything.
*
- * VHE: Host must save tpidr*_el0, actlr_el1, mdscr_el1, sp_el0,
- * and guest must save everything.
+ * VHE: Host and guest must save mdscr_el1 and sp_el0 (and the PC and pstate,
+ * which are handled as part of the el2 return state) on every switch.
+ * tpidr_el0, tpidrro_el0, and actlr_el1 only need to be switched when going
+ * to host userspace or a different VCPU. EL1 registers only need to be
+ * switched when potentially going to run a different VCPU. The latter two
+ * classes are handled as part of kvm_arch_vcpu_load and kvm_arch_vcpu_put.
*/
static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
@@ -90,14 +94,11 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
}
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
- __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
__sysreg_save_el2_return_state(ctxt);
}
@@ -163,14 +164,11 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
}
void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
- __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
__sysreg_restore_el2_return_state(ctxt);
}
@@ -236,6 +234,26 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
*/
void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
+ struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+
+ if (!has_vhe())
+ return;
+
+ __sysreg_save_user_state(host_ctxt);
+
+
+ /*
+ * Load guest EL1 and user state
+ *
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_user_state(guest_ctxt);
+ __sysreg_restore_el1_state(guest_ctxt);
+
+ vcpu->arch.sysregs_loaded_on_cpu = true;
}
/**
@@ -264,6 +282,18 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
__fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
vcpu->arch.guest_vfp_loaded = 0;
}
+
+ if (!has_vhe())
+ return;
+
+ __sysreg_save_el1_state(guest_ctxt);
+ __sysreg_save_user_state(guest_ctxt);
+ __sysreg32_save_state(vcpu);
+
+ /* Restore host user state */
+ __sysreg_restore_user_state(host_ctxt);
+
+ vcpu->arch.sysregs_loaded_on_cpu = false;
}
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Some system registers do not affect the host kernel's execution and can
therefore be loaded when we are about to run a VCPU and we don't have to
restore the host state to the hardware before the time when we are
actually about to return to userspace or schedule out the VCPU thread.
The EL1 system registers and the userspace state registers, which only
affect EL0 execution, do not affect the host kernel's execution.
The 32-bit system registers are not used by a VHE host kernel and
therefore don't need to be saved/restored on every entry/exit to/from
the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
We have already prepared the trap handling code which accesses any of
these registers to directly access the registers on the physical CPU or
to sync the registers when needed.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 6 ------
arch/arm64/kvm/hyp/sysreg-sr.c | 46 ++++++++++++++++++++++++++++++++++--------
2 files changed, 38 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index c3e1a9c65bc1..2ac8af354de0 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -359,11 +359,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
__vgic_restore_state(vcpu);
- /*
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -375,7 +370,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
sysreg_save_guest_state_vhe(guest_ctxt);
- __sysreg32_save_state(vcpu);
__vgic_save_state(vcpu);
__deactivate_traps(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 479de0f0dd07..65abf1aeba59 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -25,8 +25,12 @@
/*
* Non-VHE: Both host and guest must save everything.
*
- * VHE: Host must save tpidr*_el0, actlr_el1, mdscr_el1, sp_el0,
- * and guest must save everything.
+ * VHE: Host and guest must save mdscr_el1 and sp_el0 (and the PC and pstate,
+ * which are handled as part of the el2 return state) on every switch.
+ * tpidr_el0, tpidrro_el0, and actlr_el1 only need to be switched when going
+ * to host userspace or a different VCPU. EL1 registers only need to be
+ * switched when potentially going to run a different VCPU. The latter two
+ * classes are handled as part of kvm_arch_vcpu_load and kvm_arch_vcpu_put.
*/
static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
@@ -90,14 +94,11 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
}
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
- __sysreg_save_el1_state(ctxt);
__sysreg_save_common_state(ctxt);
- __sysreg_save_user_state(ctxt);
__sysreg_save_el2_return_state(ctxt);
}
@@ -163,14 +164,11 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
{
__sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
}
void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
{
- __sysreg_restore_el1_state(ctxt);
__sysreg_restore_common_state(ctxt);
- __sysreg_restore_user_state(ctxt);
__sysreg_restore_el2_return_state(ctxt);
}
@@ -236,6 +234,26 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
*/
void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
+ struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+
+ if (!has_vhe())
+ return;
+
+ __sysreg_save_user_state(host_ctxt);
+
+
+ /*
+ * Load guest EL1 and user state
+ *
+ * We must restore the 32-bit state before the sysregs, thanks
+ * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
+ */
+ __sysreg32_restore_state(vcpu);
+ __sysreg_restore_user_state(guest_ctxt);
+ __sysreg_restore_el1_state(guest_ctxt);
+
+ vcpu->arch.sysregs_loaded_on_cpu = true;
}
/**
@@ -264,6 +282,18 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
__fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
vcpu->arch.guest_vfp_loaded = 0;
}
+
+ if (!has_vhe())
+ return;
+
+ __sysreg_save_el1_state(guest_ctxt);
+ __sysreg_save_user_state(guest_ctxt);
+ __sysreg32_save_state(vcpu);
+
+ /* Restore host user state */
+ __sysreg_restore_user_state(host_ctxt);
+
+ vcpu->arch.sysregs_loaded_on_cpu = false;
}
void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 27/36] KVM: arm64: Move common VHE/non-VHE trap config in separate functions
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
As we are about to be more lazy with some of the trap configuration
register read/writes for VHE systems, move the logic that is currently
shared between VHE and non-VHE into a separate function which can be
called from either the world-switch path or from vcpu_load/vcpu_put.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 72 ++++++++++++++++++++++++---------------------
1 file changed, 39 insertions(+), 33 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 2ac8af354de0..c01bcfc3fb52 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,6 +24,43 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
+static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+{
+ /*
+ * We are about to set CPTR_EL2.TFP to trap all floating point
+ * register accesses to EL2, however, the ARM ARM clearly states that
+ * traps are only taken to EL2 if the operation would not otherwise
+ * trap to EL1. Therefore, always make sure that for 32-bit guests,
+ * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
+ * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
+ * it will cause an exception.
+ */
+ if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
+ !vcpu->arch.guest_vfp_loaded) {
+ write_sysreg(1 << 30, fpexc32_el2);
+ isb();
+ }
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
+
+ /* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
+ write_sysreg(1 << 15, hstr_el2);
+ /*
+ * Make sure we trap PMU access from EL0 to EL2. Also sanitize
+ * PMSELR_EL0 to make sure it never contains the cycle
+ * counter, which could make a PMXEVCNTR_EL0 access UNDEF at
+ * EL1 instead of being trapped to EL2.
+ */
+ write_sysreg(0, pmselr_el0);
+ write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
+ write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
+}
+
+static void __hyp_text __deactivate_traps_common(void)
+{
+ write_sysreg(0, hstr_el2);
+ write_sysreg(0, pmuserenr_el0);
+}
+
static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
{
u64 val;
@@ -59,37 +96,7 @@ static hyp_alternate_select(__activate_traps_arch,
static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
{
- u64 val;
-
- /*
- * We are about to set CPTR_EL2.TFP to trap all floating point
- * register accesses to EL2, however, the ARM ARM clearly states that
- * traps are only taken to EL2 if the operation would not otherwise
- * trap to EL1. Therefore, always make sure that for 32-bit guests,
- * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
- * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
- * it will cause an exception.
- */
- val = vcpu->arch.hcr_el2;
-
- if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
- !vcpu->arch.guest_vfp_loaded) {
- write_sysreg(1 << 30, fpexc32_el2);
- isb();
- }
- write_sysreg(val, hcr_el2);
-
- /* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
- write_sysreg(1 << 15, hstr_el2);
- /*
- * Make sure we trap PMU access from EL0 to EL2. Also sanitize
- * PMSELR_EL0 to make sure it never contains the cycle
- * counter, which could make a PMXEVCNTR_EL0 access UNDEF at
- * EL1 instead of being trapped to EL2.
- */
- write_sysreg(0, pmselr_el0);
- write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
- write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
+ __activate_traps_common(vcpu);
__activate_traps_arch()(vcpu);
}
@@ -135,9 +142,8 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
if (vcpu->arch.hcr_el2 & HCR_VSE)
vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
+ __deactivate_traps_common();
__deactivate_traps_arch()();
- write_sysreg(0, hstr_el2);
- write_sysreg(0, pmuserenr_el0);
}
static void __hyp_text __activate_vm(struct kvm *kvm)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 27/36] KVM: arm64: Move common VHE/non-VHE trap config in separate functions
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
As we are about to be more lazy with some of the trap configuration
register read/writes for VHE systems, move the logic that is currently
shared between VHE and non-VHE into a separate function which can be
called from either the world-switch path or from vcpu_load/vcpu_put.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 72 ++++++++++++++++++++++++---------------------
1 file changed, 39 insertions(+), 33 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 2ac8af354de0..c01bcfc3fb52 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,6 +24,43 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
+static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+{
+ /*
+ * We are about to set CPTR_EL2.TFP to trap all floating point
+ * register accesses to EL2, however, the ARM ARM clearly states that
+ * traps are only taken to EL2 if the operation would not otherwise
+ * trap to EL1. Therefore, always make sure that for 32-bit guests,
+ * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
+ * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
+ * it will cause an exception.
+ */
+ if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
+ !vcpu->arch.guest_vfp_loaded) {
+ write_sysreg(1 << 30, fpexc32_el2);
+ isb();
+ }
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
+
+ /* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
+ write_sysreg(1 << 15, hstr_el2);
+ /*
+ * Make sure we trap PMU access from EL0 to EL2. Also sanitize
+ * PMSELR_EL0 to make sure it never contains the cycle
+ * counter, which could make a PMXEVCNTR_EL0 access UNDEF at
+ * EL1 instead of being trapped to EL2.
+ */
+ write_sysreg(0, pmselr_el0);
+ write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
+ write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
+}
+
+static void __hyp_text __deactivate_traps_common(void)
+{
+ write_sysreg(0, hstr_el2);
+ write_sysreg(0, pmuserenr_el0);
+}
+
static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
{
u64 val;
@@ -59,37 +96,7 @@ static hyp_alternate_select(__activate_traps_arch,
static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
{
- u64 val;
-
- /*
- * We are about to set CPTR_EL2.TFP to trap all floating point
- * register accesses to EL2, however, the ARM ARM clearly states that
- * traps are only taken to EL2 if the operation would not otherwise
- * trap to EL1. Therefore, always make sure that for 32-bit guests,
- * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
- * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
- * it will cause an exception.
- */
- val = vcpu->arch.hcr_el2;
-
- if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
- !vcpu->arch.guest_vfp_loaded) {
- write_sysreg(1 << 30, fpexc32_el2);
- isb();
- }
- write_sysreg(val, hcr_el2);
-
- /* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
- write_sysreg(1 << 15, hstr_el2);
- /*
- * Make sure we trap PMU access from EL0 to EL2. Also sanitize
- * PMSELR_EL0 to make sure it never contains the cycle
- * counter, which could make a PMXEVCNTR_EL0 access UNDEF at
- * EL1 instead of being trapped to EL2.
- */
- write_sysreg(0, pmselr_el0);
- write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
- write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
+ __activate_traps_common(vcpu);
__activate_traps_arch()(vcpu);
}
@@ -135,9 +142,8 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
if (vcpu->arch.hcr_el2 & HCR_VSE)
vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
+ __deactivate_traps_common();
__deactivate_traps_arch()();
- write_sysreg(0, hstr_el2);
- write_sysreg(0, pmuserenr_el0);
}
static void __hyp_text __activate_vm(struct kvm *kvm)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 28/36] KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
There is no need to enable/disable traps to FP registers on every switch
to/from the VM, because the host kernel does not use this resource
without calling vcpu_put. We can therefore move things around enough
that we still always write FPEXC32_EL2 before programming CPTR_EL2 but
only program these during vcpu load/put.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 3 +++
arch/arm64/kvm/hyp/switch.c | 34 ++++++++++++++++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++++
3 files changed, 31 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 3f54c55f77a1..28d5f3cb4001 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -148,6 +148,9 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
bool __fpsimd_enabled(void);
+void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
+void deactivate_traps_vhe_put(void);
+
u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
void __noreturn __hyp_do_panic(unsigned long, ...);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index c01bcfc3fb52..44aae69a7fec 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,22 +24,25 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
-static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
{
/*
- * We are about to set CPTR_EL2.TFP to trap all floating point
- * register accesses to EL2, however, the ARM ARM clearly states that
- * traps are only taken to EL2 if the operation would not otherwise
- * trap to EL1. Therefore, always make sure that for 32-bit guests,
- * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
- * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
- * it will cause an exception.
+ * We are about to trap all floating point register accesses to EL2,
+ * however, traps are only taken to EL2 if the operation would not
+ * otherwise trap to EL1. Therefore, always make sure that for 32-bit
+ * guests, we set FPEXC.EN to prevent traps to EL1, when setting the
+ * TFP bit. If FP/ASIMD is not implemented, FPEXC is UNDEFINED and
+ * any access to it will cause an exception.
*/
if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
!vcpu->arch.guest_vfp_loaded) {
write_sysreg(1 << 30, fpexc32_el2);
isb();
}
+}
+
+static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+{
write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
@@ -61,10 +64,12 @@ static void __hyp_text __deactivate_traps_common(void)
write_sysreg(0, pmuserenr_el0);
}
-static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
{
u64 val;
+ __activate_traps_fpsimd32(vcpu);
+
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
val &= ~CPACR_EL1_ZEN;
@@ -73,7 +78,15 @@ static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
else
val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
+}
+void deactivate_traps_vhe_put(void)
+{
+ write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
+}
+
+static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+{
write_sysreg(__kvm_hyp_vector, vbar_el1);
}
@@ -81,6 +94,8 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
+ __activate_traps_fpsimd32(vcpu);
+
val = CPTR_EL2_DEFAULT;
val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
if (vcpu->arch.guest_vfp_loaded)
@@ -111,7 +126,6 @@ static void __hyp_text __deactivate_traps_vhe(void)
write_sysreg(mdcr_el2, mdcr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
- write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
write_sysreg(vectors, vbar_el1);
}
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 65abf1aeba59..b647fea93fdc 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -254,6 +254,8 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
__sysreg_restore_el1_state(guest_ctxt);
vcpu->arch.sysregs_loaded_on_cpu = true;
+
+ activate_traps_vhe_load(vcpu);
}
/**
@@ -286,6 +288,8 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
if (!has_vhe())
return;
+ deactivate_traps_vhe_put();
+
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);
__sysreg32_save_state(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 28/36] KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There is no need to enable/disable traps to FP registers on every switch
to/from the VM, because the host kernel does not use this resource
without calling vcpu_put. We can therefore move things around enough
that we still always write FPEXC32_EL2 before programming CPTR_EL2 but
only program these during vcpu load/put.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/include/asm/kvm_hyp.h | 3 +++
arch/arm64/kvm/hyp/switch.c | 34 ++++++++++++++++++++++++----------
arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++++
3 files changed, 31 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 3f54c55f77a1..28d5f3cb4001 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -148,6 +148,9 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
bool __fpsimd_enabled(void);
+void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
+void deactivate_traps_vhe_put(void);
+
u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
void __noreturn __hyp_do_panic(unsigned long, ...);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index c01bcfc3fb52..44aae69a7fec 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -24,22 +24,25 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
-static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
{
/*
- * We are about to set CPTR_EL2.TFP to trap all floating point
- * register accesses to EL2, however, the ARM ARM clearly states that
- * traps are only taken to EL2 if the operation would not otherwise
- * trap to EL1. Therefore, always make sure that for 32-bit guests,
- * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
- * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
- * it will cause an exception.
+ * We are about to trap all floating point register accesses to EL2,
+ * however, traps are only taken to EL2 if the operation would not
+ * otherwise trap to EL1. Therefore, always make sure that for 32-bit
+ * guests, we set FPEXC.EN to prevent traps to EL1, when setting the
+ * TFP bit. If FP/ASIMD is not implemented, FPEXC is UNDEFINED and
+ * any access to it will cause an exception.
*/
if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
!vcpu->arch.guest_vfp_loaded) {
write_sysreg(1 << 30, fpexc32_el2);
isb();
}
+}
+
+static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
+{
write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
@@ -61,10 +64,12 @@ static void __hyp_text __deactivate_traps_common(void)
write_sysreg(0, pmuserenr_el0);
}
-static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
{
u64 val;
+ __activate_traps_fpsimd32(vcpu);
+
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
val &= ~CPACR_EL1_ZEN;
@@ -73,7 +78,15 @@ static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
else
val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
+}
+void deactivate_traps_vhe_put(void)
+{
+ write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
+}
+
+static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+{
write_sysreg(__kvm_hyp_vector, vbar_el1);
}
@@ -81,6 +94,8 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
+ __activate_traps_fpsimd32(vcpu);
+
val = CPTR_EL2_DEFAULT;
val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
if (vcpu->arch.guest_vfp_loaded)
@@ -111,7 +126,6 @@ static void __hyp_text __deactivate_traps_vhe(void)
write_sysreg(mdcr_el2, mdcr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
- write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
write_sysreg(vectors, vbar_el1);
}
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 65abf1aeba59..b647fea93fdc 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -254,6 +254,8 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
__sysreg_restore_el1_state(guest_ctxt);
vcpu->arch.sysregs_loaded_on_cpu = true;
+
+ activate_traps_vhe_load(vcpu);
}
/**
@@ -286,6 +288,8 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
if (!has_vhe())
return;
+ deactivate_traps_vhe_put();
+
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);
__sysreg32_save_state(vcpu);
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 29/36] KVM: arm64: Configure c15, PMU, and debug register traps on cpu load/put for VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
We do not have to change the c15 trap setting on each switch to/from the
guest on VHE systems, because this setting only affects EL0.
The PMU and debug trap configuration can also be done on vcpu load/put
instead, because they don't affect how the host kernel can access the
debug registers while executing KVM kernel code and KVM doesn't use
floating point itself.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 35 ++++++++++++++++++++++++++---------
1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 44aae69a7fec..f039aad86c27 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -43,8 +43,6 @@ static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
{
- write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
-
/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
write_sysreg(1 << 15, hstr_el2);
/*
@@ -64,12 +62,15 @@ static void __hyp_text __deactivate_traps_common(void)
write_sysreg(0, pmuserenr_el0);
}
+/* Activate the traps we can during vcpu_load with VHE */
void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
{
u64 val;
+ /* Make sure 32-bit guests trap VFP */
__activate_traps_fpsimd32(vcpu);
+ /* Trap VFP accesses on a VHE system */
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
val &= ~CPACR_EL1_ZEN;
@@ -78,11 +79,28 @@ void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
else
val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
+
+ /* Activate traps on impdef sysregs, PMU, and debug */
+ __activate_traps_common(vcpu);
}
+/* Deactivate the traps we can during vcpu_put with VHE */
void deactivate_traps_vhe_put(void)
{
+ u64 mdcr_el2;
+
+ /* Re-enable host VFP and SVE access */
write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
+
+ /* Re-enable host access to impdef sysregs and the PMU */
+ __deactivate_traps_common();
+
+ /* Re-enable host access to the debug regs */
+ mdcr_el2 = read_sysreg(mdcr_el2);
+ mdcr_el2 &= MDCR_EL2_HPMN_MASK |
+ MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT |
+ MDCR_EL2_TPMS;
+ write_sysreg(mdcr_el2, mdcr_el2);
}
static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
@@ -94,8 +112,13 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
+ /* Activate traps on impdef sysregs, PMU, and debug */
+ __activate_traps_common(vcpu);
+
+ /* Make sure 32-bit guests trap VFP */
__activate_traps_fpsimd32(vcpu);
+ /* Trap VFP accesses on a non-VHE system */
val = CPTR_EL2_DEFAULT;
val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
if (vcpu->arch.guest_vfp_loaded)
@@ -111,20 +134,14 @@ static hyp_alternate_select(__activate_traps_arch,
static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
{
- __activate_traps_common(vcpu);
__activate_traps_arch()(vcpu);
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
}
static void __hyp_text __deactivate_traps_vhe(void)
{
extern char vectors[]; /* kernel exception vectors */
- u64 mdcr_el2 = read_sysreg(mdcr_el2);
- mdcr_el2 &= MDCR_EL2_HPMN_MASK |
- MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT |
- MDCR_EL2_TPMS;
-
- write_sysreg(mdcr_el2, mdcr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
write_sysreg(vectors, vbar_el1);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 29/36] KVM: arm64: Configure c15, PMU, and debug register traps on cpu load/put for VHE
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
We do not have to change the c15 trap setting on each switch to/from the
guest on VHE systems, because this setting only affects EL0.
The PMU and debug trap configuration can also be done on vcpu load/put
instead, because they don't affect how the host kernel can access the
debug registers while executing KVM kernel code and KVM doesn't use
floating point itself.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 35 ++++++++++++++++++++++++++---------
1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 44aae69a7fec..f039aad86c27 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -43,8 +43,6 @@ static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
{
- write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
-
/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
write_sysreg(1 << 15, hstr_el2);
/*
@@ -64,12 +62,15 @@ static void __hyp_text __deactivate_traps_common(void)
write_sysreg(0, pmuserenr_el0);
}
+/* Activate the traps we can during vcpu_load with VHE */
void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
{
u64 val;
+ /* Make sure 32-bit guests trap VFP */
__activate_traps_fpsimd32(vcpu);
+ /* Trap VFP accesses on a VHE system */
val = read_sysreg(cpacr_el1);
val |= CPACR_EL1_TTA;
val &= ~CPACR_EL1_ZEN;
@@ -78,11 +79,28 @@ void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
else
val &= ~CPACR_EL1_FPEN;
write_sysreg(val, cpacr_el1);
+
+ /* Activate traps on impdef sysregs, PMU, and debug */
+ __activate_traps_common(vcpu);
}
+/* Deactivate the traps we can during vcpu_put with VHE */
void deactivate_traps_vhe_put(void)
{
+ u64 mdcr_el2;
+
+ /* Re-enable host VFP and SVE access */
write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
+
+ /* Re-enable host access to impdef sysregs and the PMU */
+ __deactivate_traps_common();
+
+ /* Re-enable host access to the debug regs */
+ mdcr_el2 = read_sysreg(mdcr_el2);
+ mdcr_el2 &= MDCR_EL2_HPMN_MASK |
+ MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT |
+ MDCR_EL2_TPMS;
+ write_sysreg(mdcr_el2, mdcr_el2);
}
static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
@@ -94,8 +112,13 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
{
u64 val;
+ /* Activate traps on impdef sysregs, PMU, and debug */
+ __activate_traps_common(vcpu);
+
+ /* Make sure 32-bit guests trap VFP */
__activate_traps_fpsimd32(vcpu);
+ /* Trap VFP accesses on a non-VHE system */
val = CPTR_EL2_DEFAULT;
val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
if (vcpu->arch.guest_vfp_loaded)
@@ -111,20 +134,14 @@ static hyp_alternate_select(__activate_traps_arch,
static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
{
- __activate_traps_common(vcpu);
__activate_traps_arch()(vcpu);
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
}
static void __hyp_text __deactivate_traps_vhe(void)
{
extern char vectors[]; /* kernel exception vectors */
- u64 mdcr_el2 = read_sysreg(mdcr_el2);
- mdcr_el2 &= MDCR_EL2_HPMN_MASK |
- MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT |
- MDCR_EL2_TPMS;
-
- write_sysreg(mdcr_el2, mdcr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
write_sysreg(vectors, vbar_el1);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 30/36] KVM: arm64: Separate activate_traps and deactive_traps for VHE and non-VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
To make the code more readable and to avoid the overhead of a function
call, let's get rid of a pair of the alternative function selectors and
explicitly call the VHE and non-VHE functions instead, telling the
compiler to try to inline the static function if it can.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 78 +++++++++++++++++++++------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f039aad86c27..f9f104bfc27b 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -103,9 +103,27 @@ void deactivate_traps_vhe_put(void)
write_sysreg(mdcr_el2, mdcr_el2);
}
-static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+static inline void activate_traps_vhe(struct kvm_vcpu *vcpu)
{
- write_sysreg(__kvm_hyp_vector, vbar_el1);
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
+ write_sysreg_el2(__kvm_hyp_vector, vbar);
+}
+
+static inline void deactivate_traps_vhe(struct kvm_vcpu *vcpu)
+{
+ extern char vectors[]; /* kernel exception vectors */
+
+ /*
+ * If we pended a virtual abort, preserve it until it gets
+ * cleared. See D1.14.3 (Virtual Interrupts) for details, but
+ * the crucial bit is "On taking a vSError interrupt,
+ * HCR_EL2.VSE is cleared to 0."
+ */
+ if (vcpu->arch.hcr_el2 & HCR_VSE)
+ vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
+
+ write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
+ write_sysreg(vectors, vbar_el1);
}
static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
@@ -126,44 +144,15 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
else
val |= CPTR_EL2_TFP;
write_sysreg(val, cptr_el2);
-}
-static hyp_alternate_select(__activate_traps_arch,
- __activate_traps_nvhe, __activate_traps_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
-{
- __activate_traps_arch()(vcpu);
+ /* Configure all other hypervisor traps and features */
write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
}
-static void __hyp_text __deactivate_traps_vhe(void)
-{
- extern char vectors[]; /* kernel exception vectors */
-
- write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
- write_sysreg(vectors, vbar_el1);
-}
-
-static void __hyp_text __deactivate_traps_nvhe(void)
+static void __hyp_text __deactivate_traps_nvhe(struct kvm_vcpu *vcpu)
{
- u64 mdcr_el2 = read_sysreg(mdcr_el2);
-
- mdcr_el2 &= MDCR_EL2_HPMN_MASK;
- mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT;
-
- write_sysreg(mdcr_el2, mdcr_el2);
- write_sysreg(HCR_RW, hcr_el2);
- write_sysreg(CPTR_EL2_DEFAULT, cptr_el2);
-}
-
-static hyp_alternate_select(__deactivate_traps_arch,
- __deactivate_traps_nvhe, __deactivate_traps_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
+ u64 mdcr_el2;
-static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
-{
/*
* If we pended a virtual abort, preserve it until it gets
* cleared. See D1.14.3 (Virtual Interrupts) for details, but
@@ -174,7 +163,14 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
__deactivate_traps_common();
- __deactivate_traps_arch()();
+
+ mdcr_el2 = read_sysreg(mdcr_el2);
+ mdcr_el2 &= MDCR_EL2_HPMN_MASK;
+ mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT;
+
+ write_sysreg(mdcr_el2, mdcr_el2);
+ write_sysreg(HCR_RW, hcr_el2);
+ write_sysreg(CPTR_EL2_DEFAULT, cptr_el2);
}
static void __hyp_text __activate_vm(struct kvm *kvm)
@@ -391,7 +387,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_host_state_vhe(host_ctxt);
- __activate_traps(vcpu);
+ activate_traps_vhe(vcpu);
__activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
@@ -409,7 +405,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_guest_state_vhe(guest_ctxt);
__vgic_save_state(vcpu);
- __deactivate_traps(vcpu);
+ deactivate_traps_vhe(vcpu);
sysreg_restore_host_state_vhe(host_ctxt);
@@ -433,7 +429,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__sysreg_save_state_nvhe(host_ctxt);
- __activate_traps(vcpu);
+ __activate_traps_nvhe(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
__vgic_restore_state(vcpu);
@@ -459,7 +455,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
- __deactivate_traps(vcpu);
+ __deactivate_traps_nvhe(vcpu);
__deactivate_vm(vcpu);
__sysreg_restore_state_nvhe(host_ctxt);
@@ -485,7 +481,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
if (read_sysreg(vttbr_el2)) {
__timer_disable_traps(vcpu);
- __deactivate_traps(vcpu);
+ __deactivate_traps_nvhe(vcpu);
__deactivate_vm(vcpu);
__sysreg_restore_state_nvhe(__host_ctxt);
}
@@ -509,7 +505,7 @@ static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
struct kvm_vcpu *vcpu;
vcpu = host_ctxt->__hyp_running_vcpu;
- __deactivate_traps(vcpu);
+ deactivate_traps_vhe(vcpu);
sysreg_restore_host_state_vhe(host_ctxt);
panic(__hyp_panic_string,
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 30/36] KVM: arm64: Separate activate_traps and deactive_traps for VHE and non-VHE
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
To make the code more readable and to avoid the overhead of a function
call, let's get rid of a pair of the alternative function selectors and
explicitly call the VHE and non-VHE functions instead, telling the
compiler to try to inline the static function if it can.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 78 +++++++++++++++++++++------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f039aad86c27..f9f104bfc27b 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -103,9 +103,27 @@ void deactivate_traps_vhe_put(void)
write_sysreg(mdcr_el2, mdcr_el2);
}
-static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
+static inline void activate_traps_vhe(struct kvm_vcpu *vcpu)
{
- write_sysreg(__kvm_hyp_vector, vbar_el1);
+ write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
+ write_sysreg_el2(__kvm_hyp_vector, vbar);
+}
+
+static inline void deactivate_traps_vhe(struct kvm_vcpu *vcpu)
+{
+ extern char vectors[]; /* kernel exception vectors */
+
+ /*
+ * If we pended a virtual abort, preserve it until it gets
+ * cleared. See D1.14.3 (Virtual Interrupts) for details, but
+ * the crucial bit is "On taking a vSError interrupt,
+ * HCR_EL2.VSE is cleared to 0."
+ */
+ if (vcpu->arch.hcr_el2 & HCR_VSE)
+ vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
+
+ write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
+ write_sysreg(vectors, vbar_el1);
}
static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
@@ -126,44 +144,15 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
else
val |= CPTR_EL2_TFP;
write_sysreg(val, cptr_el2);
-}
-static hyp_alternate_select(__activate_traps_arch,
- __activate_traps_nvhe, __activate_traps_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
-
-static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
-{
- __activate_traps_arch()(vcpu);
+ /* Configure all other hypervisor traps and features */
write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
}
-static void __hyp_text __deactivate_traps_vhe(void)
-{
- extern char vectors[]; /* kernel exception vectors */
-
- write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
- write_sysreg(vectors, vbar_el1);
-}
-
-static void __hyp_text __deactivate_traps_nvhe(void)
+static void __hyp_text __deactivate_traps_nvhe(struct kvm_vcpu *vcpu)
{
- u64 mdcr_el2 = read_sysreg(mdcr_el2);
-
- mdcr_el2 &= MDCR_EL2_HPMN_MASK;
- mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT;
-
- write_sysreg(mdcr_el2, mdcr_el2);
- write_sysreg(HCR_RW, hcr_el2);
- write_sysreg(CPTR_EL2_DEFAULT, cptr_el2);
-}
-
-static hyp_alternate_select(__deactivate_traps_arch,
- __deactivate_traps_nvhe, __deactivate_traps_vhe,
- ARM64_HAS_VIRT_HOST_EXTN);
+ u64 mdcr_el2;
-static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
-{
/*
* If we pended a virtual abort, preserve it until it gets
* cleared. See D1.14.3 (Virtual Interrupts) for details, but
@@ -174,7 +163,14 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
vcpu->arch.hcr_el2 = read_sysreg(hcr_el2);
__deactivate_traps_common();
- __deactivate_traps_arch()();
+
+ mdcr_el2 = read_sysreg(mdcr_el2);
+ mdcr_el2 &= MDCR_EL2_HPMN_MASK;
+ mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT;
+
+ write_sysreg(mdcr_el2, mdcr_el2);
+ write_sysreg(HCR_RW, hcr_el2);
+ write_sysreg(CPTR_EL2_DEFAULT, cptr_el2);
}
static void __hyp_text __activate_vm(struct kvm *kvm)
@@ -391,7 +387,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_host_state_vhe(host_ctxt);
- __activate_traps(vcpu);
+ activate_traps_vhe(vcpu);
__activate_vm(vcpu->kvm);
__vgic_restore_state(vcpu);
@@ -409,7 +405,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_guest_state_vhe(guest_ctxt);
__vgic_save_state(vcpu);
- __deactivate_traps(vcpu);
+ deactivate_traps_vhe(vcpu);
sysreg_restore_host_state_vhe(host_ctxt);
@@ -433,7 +429,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__sysreg_save_state_nvhe(host_ctxt);
- __activate_traps(vcpu);
+ __activate_traps_nvhe(vcpu);
__activate_vm(kern_hyp_va(vcpu->kvm));
__vgic_restore_state(vcpu);
@@ -459,7 +455,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
- __deactivate_traps(vcpu);
+ __deactivate_traps_nvhe(vcpu);
__deactivate_vm(vcpu);
__sysreg_restore_state_nvhe(host_ctxt);
@@ -485,7 +481,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
if (read_sysreg(vttbr_el2)) {
__timer_disable_traps(vcpu);
- __deactivate_traps(vcpu);
+ __deactivate_traps_nvhe(vcpu);
__deactivate_vm(vcpu);
__sysreg_restore_state_nvhe(__host_ctxt);
}
@@ -509,7 +505,7 @@ static void __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
struct kvm_vcpu *vcpu;
vcpu = host_ctxt->__hyp_running_vcpu;
- __deactivate_traps(vcpu);
+ deactivate_traps_vhe(vcpu);
sysreg_restore_host_state_vhe(host_ctxt);
panic(__hyp_panic_string,
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 31/36] KVM: arm/arm64: Get rid of vgic_elrsr
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
There is really no need to store the vgic_elrsr on the VGIC data
structures as the only need we have for the elrsr is to figure out if an
LR is inactive when we save the VGIC state upon returning from the
guest. We can might as well store this in a temporary local variable.
This also gets rid of the endianness conversion in the VGIC save
function, which is completely unnecessary and would actually result in
incorrect functionality on big-endian systems, because we are only using
typed values here and not converting pointers and reading different
types here.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Moved patch up the queue before we start moving code around to avoid moving
potentially broken code.
include/kvm/arm_vgic.h | 2 --
virt/kvm/arm/hyp/vgic-v2-sr.c | 28 +++++++---------------------
virt/kvm/arm/hyp/vgic-v3-sr.c | 6 +++---
virt/kvm/arm/vgic/vgic-v2.c | 1 -
virt/kvm/arm/vgic/vgic-v3.c | 1 -
5 files changed, 10 insertions(+), 28 deletions(-)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index cdbd142ca7f2..ac98ae46bfb7 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -263,7 +263,6 @@ struct vgic_dist {
struct vgic_v2_cpu_if {
u32 vgic_hcr;
u32 vgic_vmcr;
- u64 vgic_elrsr; /* Saved only */
u32 vgic_apr;
u32 vgic_lr[VGIC_V2_MAX_LRS];
};
@@ -272,7 +271,6 @@ struct vgic_v3_cpu_if {
u32 vgic_hcr;
u32 vgic_vmcr;
u32 vgic_sre; /* Restored only, change ignored */
- u32 vgic_elrsr; /* Saved only */
u32 vgic_ap0r[4];
u32 vgic_ap1r[4];
u64 vgic_lr[VGIC_V3_MAX_LRS];
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index d7fd46fe9efb..c536e3d87942 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -22,29 +22,19 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-static void __hyp_text save_elrsr(struct kvm_vcpu *vcpu, void __iomem *base)
-{
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- int nr_lr = (kern_hyp_va(&kvm_vgic_global_state))->nr_lr;
- u32 elrsr0, elrsr1;
-
- elrsr0 = readl_relaxed(base + GICH_ELRSR0);
- if (unlikely(nr_lr > 32))
- elrsr1 = readl_relaxed(base + GICH_ELRSR1);
- else
- elrsr1 = 0;
-
- cpu_if->vgic_elrsr = ((u64)elrsr1 << 32) | elrsr0;
-}
-
static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
{
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- int i;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ u64 elrsr;
+ int i;
+
+ elrsr = readl_relaxed(base + GICH_ELRSR0);
+ if (unlikely(used_lrs > 32))
+ elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
for (i = 0; i < used_lrs; i++) {
- if (cpu_if->vgic_elrsr & (1UL << i))
+ if (elrsr & (1UL << i))
cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
else
cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
@@ -67,13 +57,9 @@ void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
-
- save_elrsr(vcpu, base);
save_lrs(vcpu, base);
-
writel_relaxed(0, base + GICH_HCR);
} else {
- cpu_if->vgic_elrsr = ~0UL;
cpu_if->vgic_apr = 0;
}
}
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index f5c3d6d7019e..9abf2f3c12b5 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -222,15 +222,16 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
int i;
u32 nr_pre_bits;
+ u32 elrsr;
- cpu_if->vgic_elrsr = read_gicreg(ICH_ELSR_EL2);
+ elrsr = read_gicreg(ICH_ELSR_EL2);
write_gicreg(0, ICH_HCR_EL2);
val = read_gicreg(ICH_VTR_EL2);
nr_pre_bits = vtr_to_nr_pre_bits(val);
for (i = 0; i < used_lrs; i++) {
- if (cpu_if->vgic_elrsr & (1 << i))
+ if (elrsr & (1 << i))
cpu_if->vgic_lr[i] &= ~ICH_LR_STATE;
else
cpu_if->vgic_lr[i] = __gic_v3_get_lr(i);
@@ -262,7 +263,6 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
cpu_if->its_vpe.its_vm)
write_gicreg(0, ICH_HCR_EL2);
- cpu_if->vgic_elrsr = 0xffff;
cpu_if->vgic_ap0r[0] = 0;
cpu_if->vgic_ap0r[1] = 0;
cpu_if->vgic_ap0r[2] = 0;
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index c32d7b93ffd1..bb305d49cfdd 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -265,7 +265,6 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu)
* anyway.
*/
vcpu->arch.vgic_cpu.vgic_v2.vgic_vmcr = 0;
- vcpu->arch.vgic_cpu.vgic_v2.vgic_elrsr = ~0;
/* Get the show on the road... */
vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 6b329414e57a..b76e21f3e6bd 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -267,7 +267,6 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
* anyway.
*/
vgic_v3->vgic_vmcr = 0;
- vgic_v3->vgic_elrsr = ~0;
/*
* If we are emulating a GICv3, we do it in an non-GICv2-compatible
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 31/36] KVM: arm/arm64: Get rid of vgic_elrsr
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
There is really no need to store the vgic_elrsr on the VGIC data
structures as the only need we have for the elrsr is to figure out if an
LR is inactive when we save the VGIC state upon returning from the
guest. We can might as well store this in a temporary local variable.
This also gets rid of the endianness conversion in the VGIC save
function, which is completely unnecessary and would actually result in
incorrect functionality on big-endian systems, because we are only using
typed values here and not converting pointers and reading different
types here.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Moved patch up the queue before we start moving code around to avoid moving
potentially broken code.
include/kvm/arm_vgic.h | 2 --
virt/kvm/arm/hyp/vgic-v2-sr.c | 28 +++++++---------------------
virt/kvm/arm/hyp/vgic-v3-sr.c | 6 +++---
virt/kvm/arm/vgic/vgic-v2.c | 1 -
virt/kvm/arm/vgic/vgic-v3.c | 1 -
5 files changed, 10 insertions(+), 28 deletions(-)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index cdbd142ca7f2..ac98ae46bfb7 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -263,7 +263,6 @@ struct vgic_dist {
struct vgic_v2_cpu_if {
u32 vgic_hcr;
u32 vgic_vmcr;
- u64 vgic_elrsr; /* Saved only */
u32 vgic_apr;
u32 vgic_lr[VGIC_V2_MAX_LRS];
};
@@ -272,7 +271,6 @@ struct vgic_v3_cpu_if {
u32 vgic_hcr;
u32 vgic_vmcr;
u32 vgic_sre; /* Restored only, change ignored */
- u32 vgic_elrsr; /* Saved only */
u32 vgic_ap0r[4];
u32 vgic_ap1r[4];
u64 vgic_lr[VGIC_V3_MAX_LRS];
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index d7fd46fe9efb..c536e3d87942 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -22,29 +22,19 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-static void __hyp_text save_elrsr(struct kvm_vcpu *vcpu, void __iomem *base)
-{
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- int nr_lr = (kern_hyp_va(&kvm_vgic_global_state))->nr_lr;
- u32 elrsr0, elrsr1;
-
- elrsr0 = readl_relaxed(base + GICH_ELRSR0);
- if (unlikely(nr_lr > 32))
- elrsr1 = readl_relaxed(base + GICH_ELRSR1);
- else
- elrsr1 = 0;
-
- cpu_if->vgic_elrsr = ((u64)elrsr1 << 32) | elrsr0;
-}
-
static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
{
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- int i;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ u64 elrsr;
+ int i;
+
+ elrsr = readl_relaxed(base + GICH_ELRSR0);
+ if (unlikely(used_lrs > 32))
+ elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
for (i = 0; i < used_lrs; i++) {
- if (cpu_if->vgic_elrsr & (1UL << i))
+ if (elrsr & (1UL << i))
cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
else
cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
@@ -67,13 +57,9 @@ void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
-
- save_elrsr(vcpu, base);
save_lrs(vcpu, base);
-
writel_relaxed(0, base + GICH_HCR);
} else {
- cpu_if->vgic_elrsr = ~0UL;
cpu_if->vgic_apr = 0;
}
}
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index f5c3d6d7019e..9abf2f3c12b5 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -222,15 +222,16 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
int i;
u32 nr_pre_bits;
+ u32 elrsr;
- cpu_if->vgic_elrsr = read_gicreg(ICH_ELSR_EL2);
+ elrsr = read_gicreg(ICH_ELSR_EL2);
write_gicreg(0, ICH_HCR_EL2);
val = read_gicreg(ICH_VTR_EL2);
nr_pre_bits = vtr_to_nr_pre_bits(val);
for (i = 0; i < used_lrs; i++) {
- if (cpu_if->vgic_elrsr & (1 << i))
+ if (elrsr & (1 << i))
cpu_if->vgic_lr[i] &= ~ICH_LR_STATE;
else
cpu_if->vgic_lr[i] = __gic_v3_get_lr(i);
@@ -262,7 +263,6 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
cpu_if->its_vpe.its_vm)
write_gicreg(0, ICH_HCR_EL2);
- cpu_if->vgic_elrsr = 0xffff;
cpu_if->vgic_ap0r[0] = 0;
cpu_if->vgic_ap0r[1] = 0;
cpu_if->vgic_ap0r[2] = 0;
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index c32d7b93ffd1..bb305d49cfdd 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -265,7 +265,6 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu)
* anyway.
*/
vcpu->arch.vgic_cpu.vgic_v2.vgic_vmcr = 0;
- vcpu->arch.vgic_cpu.vgic_v2.vgic_elrsr = ~0;
/* Get the show on the road... */
vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 6b329414e57a..b76e21f3e6bd 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -267,7 +267,6 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
* anyway.
*/
vgic_v3->vgic_vmcr = 0;
- vgic_v3->vgic_elrsr = ~0;
/*
* If we are emulating a GICv3, we do it in an non-GICv2-compatible
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 32/36] KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
We can program the GICv2 hypervisor control interface logic directly
from the core vgic code and can instead do the save/restore directly
from the flush/sync functions, which can lead to a number of future
optimizations.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Removed unnecessary kvm_hyp.h include
- Adapted the patch based on having gotten rid of storing the elrsr
prior to this patch.
- No longer change the interrupt handling of the maintenance interrupt
handler. That seems to have been a leftover from an earlier version
of the timer patches where we were syncing the vgic state after
having enabled interrupts, leading to the maintenance interrupt firing.
It may be possible to move the vgic sync function out to an
interrupts enabled section later on, which would require
re-introducing logic to disable the VGIC maintenance interrupt in the
maintenance interrupt handler, but we leave this for future work as
the immediate benefit is not clear.
arch/arm/kvm/hyp/switch.c | 4 ---
arch/arm64/include/asm/kvm_hyp.h | 2 --
arch/arm64/kvm/hyp/switch.c | 4 ---
virt/kvm/arm/hyp/vgic-v2-sr.c | 65 ----------------------------------------
virt/kvm/arm/vgic/vgic-v2.c | 63 ++++++++++++++++++++++++++++++++++++++
virt/kvm/arm/vgic/vgic.c | 19 +++++++++++-
virt/kvm/arm/vgic/vgic.h | 3 ++
7 files changed, 84 insertions(+), 76 deletions(-)
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 7b2bd25e3b10..214187446e63 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -91,16 +91,12 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_save_state(vcpu);
- else
- __vgic_v2_save_state(vcpu);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
- else
- __vgic_v2_restore_state(vcpu);
}
static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 28d5f3cb4001..bd3fe6446728 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -121,8 +121,6 @@ typeof(orig) * __hyp_text fname(void) \
return val; \
}
-void __vgic_v2_save_state(struct kvm_vcpu *vcpu);
-void __vgic_v2_restore_state(struct kvm_vcpu *vcpu);
int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f9f104bfc27b..a7de1436a0e6 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -187,16 +187,12 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_save_state(vcpu);
- else
- __vgic_v2_save_state(vcpu);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
- else
- __vgic_v2_restore_state(vcpu);
}
static bool __hyp_text __true_value(void)
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index c536e3d87942..b433257f4348 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -22,71 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
-{
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 elrsr;
- int i;
-
- elrsr = readl_relaxed(base + GICH_ELRSR0);
- if (unlikely(used_lrs > 32))
- elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
-
- for (i = 0; i < used_lrs; i++) {
- if (elrsr & (1UL << i))
- cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
- else
- cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
-
- writel_relaxed(0, base + GICH_LR0 + (i * 4));
- }
-}
-
-/* vcpu is already in the HYP VA space */
-void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
-{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- struct vgic_dist *vgic = &kvm->arch.vgic;
- void __iomem *base = kern_hyp_va(vgic->vctrl_base);
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
-
- if (!base)
- return;
-
- if (used_lrs) {
- cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
- save_lrs(vcpu, base);
- writel_relaxed(0, base + GICH_HCR);
- } else {
- cpu_if->vgic_apr = 0;
- }
-}
-
-/* vcpu is already in the HYP VA space */
-void __hyp_text __vgic_v2_restore_state(struct kvm_vcpu *vcpu)
-{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- struct vgic_dist *vgic = &kvm->arch.vgic;
- void __iomem *base = kern_hyp_va(vgic->vctrl_base);
- int i;
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
-
- if (!base)
- return;
-
- if (used_lrs) {
- writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
- writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
- for (i = 0; i < used_lrs; i++) {
- writel_relaxed(cpu_if->vgic_lr[i],
- base + GICH_LR0 + (i * 4));
- }
- }
-}
-
#ifdef CONFIG_ARM64
/*
* __vgic_v2_perform_cpuif_access -- perform a GICV access on behalf of the
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index bb305d49cfdd..1e5f3eb6973d 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -421,6 +421,69 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
return ret;
}
+static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
+{
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ u64 elrsr;
+ int i;
+
+ elrsr = readl_relaxed(base + GICH_ELRSR0);
+ if (unlikely(used_lrs > 32))
+ elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
+
+ for (i = 0; i < used_lrs; i++) {
+ if (elrsr & (1UL << i))
+ cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
+ else
+ cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
+
+ writel_relaxed(0, base + GICH_LR0 + (i * 4));
+ }
+}
+
+void vgic_v2_save_state(struct kvm_vcpu *vcpu)
+{
+ struct kvm *kvm = vcpu->kvm;
+ struct vgic_dist *vgic = &kvm->arch.vgic;
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ void __iomem *base = vgic->vctrl_base;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+
+ if (!base)
+ return;
+
+ if (used_lrs) {
+ cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
+ save_lrs(vcpu, base);
+ writel_relaxed(0, base + GICH_HCR);
+ } else {
+ cpu_if->vgic_apr = 0;
+ }
+}
+
+void vgic_v2_restore_state(struct kvm_vcpu *vcpu)
+{
+ struct kvm *kvm = vcpu->kvm;
+ struct vgic_dist *vgic = &kvm->arch.vgic;
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ void __iomem *base = vgic->vctrl_base;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ int i;
+
+ if (!base)
+ return;
+
+ if (used_lrs) {
+ writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
+ writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
+ for (i = 0; i < used_lrs; i++) {
+ writel_relaxed(cpu_if->vgic_lr[i],
+ base + GICH_LR0 + (i * 4));
+ }
+ }
+}
+
void vgic_v2_load(struct kvm_vcpu *vcpu)
{
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index f6299eb1998f..5bf0804e79b4 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -750,11 +750,19 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu)
vgic_clear_lr(vcpu, count);
}
+static inline void vgic_save_state(struct kvm_vcpu *vcpu)
+{
+ if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ vgic_v2_save_state(vcpu);
+}
+
/* Sync back the hardware VGIC state into our emulation after a guest's run. */
void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+ vgic_save_state(vcpu);
+
WARN_ON(vgic_v4_sync_hwstate(vcpu));
/* An empty ap_list_head implies used_lrs == 0 */
@@ -766,6 +774,12 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
vgic_prune_ap_list(vcpu);
}
+static inline void vgic_restore_state(struct kvm_vcpu *vcpu)
+{
+ if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ vgic_v2_restore_state(vcpu);
+}
+
/* Flush our emulation state into the GIC hardware before entering the guest. */
void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
{
@@ -781,13 +795,16 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
* this.
*/
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
- return;
+ goto out;
DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
spin_lock(&vcpu->arch.vgic_cpu.ap_list_lock);
vgic_flush_lr_state(vcpu);
spin_unlock(&vcpu->arch.vgic_cpu.ap_list_lock);
+
+out:
+ vgic_restore_state(vcpu);
}
void kvm_vgic_load(struct kvm_vcpu *vcpu)
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 12c37b89f7a3..89b9547fba27 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -176,6 +176,9 @@ void vgic_v2_init_lrs(void);
void vgic_v2_load(struct kvm_vcpu *vcpu);
void vgic_v2_put(struct kvm_vcpu *vcpu);
+void vgic_v2_save_state(struct kvm_vcpu *vcpu);
+void vgic_v2_restore_state(struct kvm_vcpu *vcpu);
+
static inline void vgic_get_irq_kref(struct vgic_irq *irq)
{
if (irq->intid < VGIC_MIN_LPI)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 32/36] KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
We can program the GICv2 hypervisor control interface logic directly
from the core vgic code and can instead do the save/restore directly
from the flush/sync functions, which can lead to a number of future
optimizations.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
Notes:
Changes since v1:
- Removed unnecessary kvm_hyp.h include
- Adapted the patch based on having gotten rid of storing the elrsr
prior to this patch.
- No longer change the interrupt handling of the maintenance interrupt
handler. That seems to have been a leftover from an earlier version
of the timer patches where we were syncing the vgic state after
having enabled interrupts, leading to the maintenance interrupt firing.
It may be possible to move the vgic sync function out to an
interrupts enabled section later on, which would require
re-introducing logic to disable the VGIC maintenance interrupt in the
maintenance interrupt handler, but we leave this for future work as
the immediate benefit is not clear.
arch/arm/kvm/hyp/switch.c | 4 ---
arch/arm64/include/asm/kvm_hyp.h | 2 --
arch/arm64/kvm/hyp/switch.c | 4 ---
virt/kvm/arm/hyp/vgic-v2-sr.c | 65 ----------------------------------------
virt/kvm/arm/vgic/vgic-v2.c | 63 ++++++++++++++++++++++++++++++++++++++
virt/kvm/arm/vgic/vgic.c | 19 +++++++++++-
virt/kvm/arm/vgic/vgic.h | 3 ++
7 files changed, 84 insertions(+), 76 deletions(-)
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 7b2bd25e3b10..214187446e63 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -91,16 +91,12 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_save_state(vcpu);
- else
- __vgic_v2_save_state(vcpu);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
- else
- __vgic_v2_restore_state(vcpu);
}
static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 28d5f3cb4001..bd3fe6446728 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -121,8 +121,6 @@ typeof(orig) * __hyp_text fname(void) \
return val; \
}
-void __vgic_v2_save_state(struct kvm_vcpu *vcpu);
-void __vgic_v2_restore_state(struct kvm_vcpu *vcpu);
int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index f9f104bfc27b..a7de1436a0e6 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -187,16 +187,12 @@ static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_save_state(vcpu);
- else
- __vgic_v2_save_state(vcpu);
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
__vgic_v3_restore_state(vcpu);
- else
- __vgic_v2_restore_state(vcpu);
}
static bool __hyp_text __true_value(void)
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index c536e3d87942..b433257f4348 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -22,71 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
-{
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 elrsr;
- int i;
-
- elrsr = readl_relaxed(base + GICH_ELRSR0);
- if (unlikely(used_lrs > 32))
- elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
-
- for (i = 0; i < used_lrs; i++) {
- if (elrsr & (1UL << i))
- cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
- else
- cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
-
- writel_relaxed(0, base + GICH_LR0 + (i * 4));
- }
-}
-
-/* vcpu is already in the HYP VA space */
-void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
-{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- struct vgic_dist *vgic = &kvm->arch.vgic;
- void __iomem *base = kern_hyp_va(vgic->vctrl_base);
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
-
- if (!base)
- return;
-
- if (used_lrs) {
- cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
- save_lrs(vcpu, base);
- writel_relaxed(0, base + GICH_HCR);
- } else {
- cpu_if->vgic_apr = 0;
- }
-}
-
-/* vcpu is already in the HYP VA space */
-void __hyp_text __vgic_v2_restore_state(struct kvm_vcpu *vcpu)
-{
- struct kvm *kvm = kern_hyp_va(vcpu->kvm);
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
- struct vgic_dist *vgic = &kvm->arch.vgic;
- void __iomem *base = kern_hyp_va(vgic->vctrl_base);
- int i;
- u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
-
- if (!base)
- return;
-
- if (used_lrs) {
- writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
- writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
- for (i = 0; i < used_lrs; i++) {
- writel_relaxed(cpu_if->vgic_lr[i],
- base + GICH_LR0 + (i * 4));
- }
- }
-}
-
#ifdef CONFIG_ARM64
/*
* __vgic_v2_perform_cpuif_access -- perform a GICV access on behalf of the
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index bb305d49cfdd..1e5f3eb6973d 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -421,6 +421,69 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
return ret;
}
+static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
+{
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ u64 elrsr;
+ int i;
+
+ elrsr = readl_relaxed(base + GICH_ELRSR0);
+ if (unlikely(used_lrs > 32))
+ elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
+
+ for (i = 0; i < used_lrs; i++) {
+ if (elrsr & (1UL << i))
+ cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
+ else
+ cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
+
+ writel_relaxed(0, base + GICH_LR0 + (i * 4));
+ }
+}
+
+void vgic_v2_save_state(struct kvm_vcpu *vcpu)
+{
+ struct kvm *kvm = vcpu->kvm;
+ struct vgic_dist *vgic = &kvm->arch.vgic;
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ void __iomem *base = vgic->vctrl_base;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+
+ if (!base)
+ return;
+
+ if (used_lrs) {
+ cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
+ save_lrs(vcpu, base);
+ writel_relaxed(0, base + GICH_HCR);
+ } else {
+ cpu_if->vgic_apr = 0;
+ }
+}
+
+void vgic_v2_restore_state(struct kvm_vcpu *vcpu)
+{
+ struct kvm *kvm = vcpu->kvm;
+ struct vgic_dist *vgic = &kvm->arch.vgic;
+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+ void __iomem *base = vgic->vctrl_base;
+ u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+ int i;
+
+ if (!base)
+ return;
+
+ if (used_lrs) {
+ writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
+ writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
+ for (i = 0; i < used_lrs; i++) {
+ writel_relaxed(cpu_if->vgic_lr[i],
+ base + GICH_LR0 + (i * 4));
+ }
+ }
+}
+
void vgic_v2_load(struct kvm_vcpu *vcpu)
{
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index f6299eb1998f..5bf0804e79b4 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -750,11 +750,19 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu)
vgic_clear_lr(vcpu, count);
}
+static inline void vgic_save_state(struct kvm_vcpu *vcpu)
+{
+ if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ vgic_v2_save_state(vcpu);
+}
+
/* Sync back the hardware VGIC state into our emulation after a guest's run. */
void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+ vgic_save_state(vcpu);
+
WARN_ON(vgic_v4_sync_hwstate(vcpu));
/* An empty ap_list_head implies used_lrs == 0 */
@@ -766,6 +774,12 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
vgic_prune_ap_list(vcpu);
}
+static inline void vgic_restore_state(struct kvm_vcpu *vcpu)
+{
+ if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ vgic_v2_restore_state(vcpu);
+}
+
/* Flush our emulation state into the GIC hardware before entering the guest. */
void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
{
@@ -781,13 +795,16 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
* this.
*/
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
- return;
+ goto out;
DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
spin_lock(&vcpu->arch.vgic_cpu.ap_list_lock);
vgic_flush_lr_state(vcpu);
spin_unlock(&vcpu->arch.vgic_cpu.ap_list_lock);
+
+out:
+ vgic_restore_state(vcpu);
}
void kvm_vgic_load(struct kvm_vcpu *vcpu)
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 12c37b89f7a3..89b9547fba27 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -176,6 +176,9 @@ void vgic_v2_init_lrs(void);
void vgic_v2_load(struct kvm_vcpu *vcpu);
void vgic_v2_put(struct kvm_vcpu *vcpu);
+void vgic_v2_save_state(struct kvm_vcpu *vcpu);
+void vgic_v2_restore_state(struct kvm_vcpu *vcpu);
+
static inline void vgic_get_irq_kref(struct vgic_irq *irq)
{
if (irq->intid < VGIC_MIN_LPI)
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 33/36] KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
The vgic-v2-sr.c file now only contains the logic to replay unaligned
accesses to the virtual CPU interface on 16K and 64K page systems, which
is only relevant on 64-bit platforms. Therefore move this file to the
arm64 KVM tree, remove the compile directive from the 32-bit side
makefile, and remove the ifdef in the C file.
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/kvm/hyp/Makefile | 1 -
arch/arm64/kvm/hyp/Makefile | 2 +-
{virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c | 2 --
3 files changed, 1 insertion(+), 4 deletions(-)
rename {virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c (98%)
diff --git a/arch/arm/kvm/hyp/Makefile b/arch/arm/kvm/hyp/Makefile
index 5638ce0c9524..1964111c984a 100644
--- a/arch/arm/kvm/hyp/Makefile
+++ b/arch/arm/kvm/hyp/Makefile
@@ -7,7 +7,6 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING
KVM=../../../../virt/kvm
-obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o
diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
index f04400d494b7..7e8d41210288 100644
--- a/arch/arm64/kvm/hyp/Makefile
+++ b/arch/arm64/kvm/hyp/Makefile
@@ -7,10 +7,10 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING
KVM=../../../../virt/kvm
-obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o
+obj-$(CONFIG_KVM_ARM_HOST) += vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += sysreg-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += debug-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += entry.o
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/arch/arm64/kvm/hyp/vgic-v2-sr.c
similarity index 98%
rename from virt/kvm/arm/hyp/vgic-v2-sr.c
rename to arch/arm64/kvm/hyp/vgic-v2-sr.c
index b433257f4348..fcd7b4eff927 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v2-sr.c
@@ -22,7 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-#ifdef CONFIG_ARM64
/*
* __vgic_v2_perform_cpuif_access -- perform a GICV access on behalf of the
* guest.
@@ -76,4 +75,3 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
return 1;
}
-#endif
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 33/36] KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The vgic-v2-sr.c file now only contains the logic to replay unaligned
accesses to the virtual CPU interface on 16K and 64K page systems, which
is only relevant on 64-bit platforms. Therefore move this file to the
arm64 KVM tree, remove the compile directive from the 32-bit side
makefile, and remove the ifdef in the C file.
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/kvm/hyp/Makefile | 1 -
arch/arm64/kvm/hyp/Makefile | 2 +-
{virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c | 2 --
3 files changed, 1 insertion(+), 4 deletions(-)
rename {virt/kvm/arm => arch/arm64/kvm}/hyp/vgic-v2-sr.c (98%)
diff --git a/arch/arm/kvm/hyp/Makefile b/arch/arm/kvm/hyp/Makefile
index 5638ce0c9524..1964111c984a 100644
--- a/arch/arm/kvm/hyp/Makefile
+++ b/arch/arm/kvm/hyp/Makefile
@@ -7,7 +7,6 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING
KVM=../../../../virt/kvm
-obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o
diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
index f04400d494b7..7e8d41210288 100644
--- a/arch/arm64/kvm/hyp/Makefile
+++ b/arch/arm64/kvm/hyp/Makefile
@@ -7,10 +7,10 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING
KVM=../../../../virt/kvm
-obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o
+obj-$(CONFIG_KVM_ARM_HOST) += vgic-v2-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += sysreg-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += debug-sr.o
obj-$(CONFIG_KVM_ARM_HOST) += entry.o
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/arch/arm64/kvm/hyp/vgic-v2-sr.c
similarity index 98%
rename from virt/kvm/arm/hyp/vgic-v2-sr.c
rename to arch/arm64/kvm/hyp/vgic-v2-sr.c
index b433257f4348..fcd7b4eff927 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v2-sr.c
@@ -22,7 +22,6 @@
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
-#ifdef CONFIG_ARM64
/*
* __vgic_v2_perform_cpuif_access -- perform a GICV access on behalf of the
* guest.
@@ -76,4 +75,3 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
return 1;
}
-#endif
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 34/36] KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
Just like we can program the GICv2 hypervisor control interface directly
from the core vgic code, we can do the same for the GICv3 hypervisor
control interface on VHE systems.
We do this by simply calling the save/restore functions when we have VHE
and we can then get rid of the save/restore function calls from the VHE
world switch function.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 3 ---
virt/kvm/arm/vgic/vgic.c | 5 +++++
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index a7de1436a0e6..947684312079 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -386,8 +386,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
activate_traps_vhe(vcpu);
__activate_vm(vcpu->kvm);
- __vgic_restore_state(vcpu);
-
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -399,7 +397,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
sysreg_save_guest_state_vhe(guest_ctxt);
- __vgic_save_state(vcpu);
deactivate_traps_vhe(vcpu);
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 5bf0804e79b4..636200cace55 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -19,6 +19,7 @@
#include <linux/list_sort.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
+#include <asm/kvm_hyp.h>
#include "vgic.h"
@@ -754,6 +755,8 @@ static inline void vgic_save_state(struct kvm_vcpu *vcpu)
{
if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
vgic_v2_save_state(vcpu);
+ else if (has_vhe())
+ __vgic_v3_save_state(vcpu);
}
/* Sync back the hardware VGIC state into our emulation after a guest's run. */
@@ -778,6 +781,8 @@ static inline void vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
vgic_v2_restore_state(vcpu);
+ else if (has_vhe())
+ __vgic_v3_restore_state(vcpu);
}
/* Flush our emulation state into the GIC hardware before entering the guest. */
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 34/36] KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on VHE
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
Just like we can program the GICv2 hypervisor control interface directly
from the core vgic code, we can do the same for the GICv3 hypervisor
control interface on VHE systems.
We do this by simply calling the save/restore functions when we have VHE
and we can then get rid of the save/restore function calls from the VHE
world switch function.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm64/kvm/hyp/switch.c | 3 ---
virt/kvm/arm/vgic/vgic.c | 5 +++++
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index a7de1436a0e6..947684312079 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -386,8 +386,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
activate_traps_vhe(vcpu);
__activate_vm(vcpu->kvm);
- __vgic_restore_state(vcpu);
-
sysreg_restore_guest_state_vhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -399,7 +397,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
sysreg_save_guest_state_vhe(guest_ctxt);
- __vgic_save_state(vcpu);
deactivate_traps_vhe(vcpu);
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 5bf0804e79b4..636200cace55 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -19,6 +19,7 @@
#include <linux/list_sort.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
+#include <asm/kvm_hyp.h>
#include "vgic.h"
@@ -754,6 +755,8 @@ static inline void vgic_save_state(struct kvm_vcpu *vcpu)
{
if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
vgic_v2_save_state(vcpu);
+ else if (has_vhe())
+ __vgic_v3_save_state(vcpu);
}
/* Sync back the hardware VGIC state into our emulation after a guest's run. */
@@ -778,6 +781,8 @@ static inline void vgic_restore_state(struct kvm_vcpu *vcpu)
{
if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
vgic_v2_restore_state(vcpu);
+ else if (has_vhe())
+ __vgic_v3_restore_state(vcpu);
}
/* Flush our emulation state into the GIC hardware before entering the guest. */
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 35/36] KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel
Cc: kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones, Christoffer Dall
The APRs can only have bits set when the guest acknowledges an interrupt
in the LR and can only have a bit cleared when the guest EOIs an
interrupt in the LR. Therefore, if we have no LRs with any
pending/active interrupts, the APR cannot change value and there is no
need to clear it on every exit from the VM (hint: it will have already
been cleared when we exited the guest the last time with the LRs all
EOIed).
The only case we need to take care of is when we migrate the VCPU away
from a CPU or migrate a new VCPU onto a CPU, or when we return to
userspace to capture the state of the VCPU for migration. To make sure
this works, factor out the APR save/restore functionality into separate
functions called from the VCPU (and by extension VGIC) put/load hooks.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_hyp.h | 2 +
arch/arm64/include/asm/kvm_hyp.h | 2 +
virt/kvm/arm/hyp/vgic-v3-sr.c | 123 +++++++++++++++++++++------------------
virt/kvm/arm/vgic/vgic-v2.c | 7 +--
virt/kvm/arm/vgic/vgic-v3.c | 5 ++
5 files changed, 77 insertions(+), 62 deletions(-)
diff --git a/arch/arm/include/asm/kvm_hyp.h b/arch/arm/include/asm/kvm_hyp.h
index ab20ffa8b9e7..b3dd4f4304f5 100644
--- a/arch/arm/include/asm/kvm_hyp.h
+++ b/arch/arm/include/asm/kvm_hyp.h
@@ -109,6 +109,8 @@ void __sysreg_restore_state(struct kvm_cpu_context *ctxt);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
+void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
asmlinkage void __vfp_save_state(struct vfp_hard_struct *vfp);
asmlinkage void __vfp_restore_state(struct vfp_hard_struct *vfp);
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index bd3fe6446728..62872ceefa33 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -125,6 +125,8 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
+void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index 9abf2f3c12b5..811b42c8441d 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -221,14 +221,11 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
int i;
- u32 nr_pre_bits;
u32 elrsr;
elrsr = read_gicreg(ICH_ELSR_EL2);
write_gicreg(0, ICH_HCR_EL2);
- val = read_gicreg(ICH_VTR_EL2);
- nr_pre_bits = vtr_to_nr_pre_bits(val);
for (i = 0; i < used_lrs; i++) {
if (elrsr & (1 << i))
@@ -238,39 +235,10 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
__gic_v3_set_lr(0, i);
}
-
- switch (nr_pre_bits) {
- case 7:
- cpu_if->vgic_ap0r[3] = __vgic_v3_read_ap0rn(3);
- cpu_if->vgic_ap0r[2] = __vgic_v3_read_ap0rn(2);
- case 6:
- cpu_if->vgic_ap0r[1] = __vgic_v3_read_ap0rn(1);
- default:
- cpu_if->vgic_ap0r[0] = __vgic_v3_read_ap0rn(0);
- }
-
- switch (nr_pre_bits) {
- case 7:
- cpu_if->vgic_ap1r[3] = __vgic_v3_read_ap1rn(3);
- cpu_if->vgic_ap1r[2] = __vgic_v3_read_ap1rn(2);
- case 6:
- cpu_if->vgic_ap1r[1] = __vgic_v3_read_ap1rn(1);
- default:
- cpu_if->vgic_ap1r[0] = __vgic_v3_read_ap1rn(0);
- }
} else {
if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
cpu_if->its_vpe.its_vm)
write_gicreg(0, ICH_HCR_EL2);
-
- cpu_if->vgic_ap0r[0] = 0;
- cpu_if->vgic_ap0r[1] = 0;
- cpu_if->vgic_ap0r[2] = 0;
- cpu_if->vgic_ap0r[3] = 0;
- cpu_if->vgic_ap1r[0] = 0;
- cpu_if->vgic_ap1r[1] = 0;
- cpu_if->vgic_ap1r[2] = 0;
- cpu_if->vgic_ap1r[3] = 0;
}
val = read_gicreg(ICC_SRE_EL2);
@@ -287,8 +255,6 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
{
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 val;
- u32 nr_pre_bits;
int i;
/*
@@ -306,32 +272,9 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
write_gicreg(cpu_if->vgic_vmcr, ICH_VMCR_EL2);
}
- val = read_gicreg(ICH_VTR_EL2);
- nr_pre_bits = vtr_to_nr_pre_bits(val);
-
if (used_lrs) {
write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
- switch (nr_pre_bits) {
- case 7:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[3], 3);
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[2], 2);
- case 6:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[1], 1);
- default:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[0], 0);
- }
-
- switch (nr_pre_bits) {
- case 7:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[3], 3);
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[2], 2);
- case 6:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[1], 1);
- default:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[0], 0);
- }
-
for (i = 0; i < used_lrs; i++)
__gic_v3_set_lr(cpu_if->vgic_lr[i], i);
} else {
@@ -364,6 +307,72 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
ICC_SRE_EL2);
}
+void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if;
+ u64 val;
+ u32 nr_pre_bits;
+
+ vcpu = kern_hyp_va(vcpu);
+ cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
+ val = read_gicreg(ICH_VTR_EL2);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
+
+ switch (nr_pre_bits) {
+ case 7:
+ cpu_if->vgic_ap0r[3] = __vgic_v3_read_ap0rn(3);
+ cpu_if->vgic_ap0r[2] = __vgic_v3_read_ap0rn(2);
+ case 6:
+ cpu_if->vgic_ap0r[1] = __vgic_v3_read_ap0rn(1);
+ default:
+ cpu_if->vgic_ap0r[0] = __vgic_v3_read_ap0rn(0);
+ }
+
+ switch (nr_pre_bits) {
+ case 7:
+ cpu_if->vgic_ap1r[3] = __vgic_v3_read_ap1rn(3);
+ cpu_if->vgic_ap1r[2] = __vgic_v3_read_ap1rn(2);
+ case 6:
+ cpu_if->vgic_ap1r[1] = __vgic_v3_read_ap1rn(1);
+ default:
+ cpu_if->vgic_ap1r[0] = __vgic_v3_read_ap1rn(0);
+ }
+}
+
+void __hyp_text __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if;
+ u64 val;
+ u32 nr_pre_bits;
+
+ vcpu = kern_hyp_va(vcpu);
+ cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
+ val = read_gicreg(ICH_VTR_EL2);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
+
+ switch (nr_pre_bits) {
+ case 7:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[3], 3);
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[2], 2);
+ case 6:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[1], 1);
+ default:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[0], 0);
+ }
+
+ switch (nr_pre_bits) {
+ case 7:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[3], 3);
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[2], 2);
+ case 6:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[1], 1);
+ default:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[0], 0);
+ }
+}
+
void __hyp_text __vgic_v3_init_lrs(void)
{
int max_lr_idx = vtr_to_max_lr_idx(read_gicreg(ICH_VTR_EL2));
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 1e5f3eb6973d..ca7cfee9f353 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -446,7 +446,6 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
struct vgic_dist *vgic = &kvm->arch.vgic;
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
void __iomem *base = vgic->vctrl_base;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
@@ -454,11 +453,8 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)
return;
if (used_lrs) {
- cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
save_lrs(vcpu, base);
writel_relaxed(0, base + GICH_HCR);
- } else {
- cpu_if->vgic_apr = 0;
}
}
@@ -476,7 +472,6 @@ void vgic_v2_restore_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
- writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
for (i = 0; i < used_lrs; i++) {
writel_relaxed(cpu_if->vgic_lr[i],
base + GICH_LR0 + (i * 4));
@@ -490,6 +485,7 @@ void vgic_v2_load(struct kvm_vcpu *vcpu)
struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
writel_relaxed(cpu_if->vgic_vmcr, vgic->vctrl_base + GICH_VMCR);
+ writel_relaxed(cpu_if->vgic_apr, vgic->vctrl_base + GICH_APR);
}
void vgic_v2_put(struct kvm_vcpu *vcpu)
@@ -498,4 +494,5 @@ void vgic_v2_put(struct kvm_vcpu *vcpu)
struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
cpu_if->vgic_vmcr = readl_relaxed(vgic->vctrl_base + GICH_VMCR);
+ cpu_if->vgic_apr = readl_relaxed(vgic->vctrl_base + GICH_APR);
}
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index b76e21f3e6bd..4bafcd1e6bb8 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -16,6 +16,7 @@
#include <linux/kvm.h>
#include <linux/kvm_host.h>
#include <kvm/arm_vgic.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
#include <asm/kvm_asm.h>
@@ -587,6 +588,8 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
*/
if (likely(cpu_if->vgic_sre))
kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr);
+
+ kvm_call_hyp(__vgic_v3_restore_aprs, vcpu);
}
void vgic_v3_put(struct kvm_vcpu *vcpu)
@@ -595,4 +598,6 @@ void vgic_v3_put(struct kvm_vcpu *vcpu)
if (likely(cpu_if->vgic_sre))
cpu_if->vgic_vmcr = kvm_call_hyp(__vgic_v3_read_vmcr);
+
+ kvm_call_hyp(__vgic_v3_save_aprs, vcpu);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 35/36] KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
The APRs can only have bits set when the guest acknowledges an interrupt
in the LR and can only have a bit cleared when the guest EOIs an
interrupt in the LR. Therefore, if we have no LRs with any
pending/active interrupts, the APR cannot change value and there is no
need to clear it on every exit from the VM (hint: it will have already
been cleared when we exited the guest the last time with the LRs all
EOIed).
The only case we need to take care of is when we migrate the VCPU away
from a CPU or migrate a new VCPU onto a CPU, or when we return to
userspace to capture the state of the VCPU for migration. To make sure
this works, factor out the APR save/restore functionality into separate
functions called from the VCPU (and by extension VGIC) put/load hooks.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_hyp.h | 2 +
arch/arm64/include/asm/kvm_hyp.h | 2 +
virt/kvm/arm/hyp/vgic-v3-sr.c | 123 +++++++++++++++++++++------------------
virt/kvm/arm/vgic/vgic-v2.c | 7 +--
virt/kvm/arm/vgic/vgic-v3.c | 5 ++
5 files changed, 77 insertions(+), 62 deletions(-)
diff --git a/arch/arm/include/asm/kvm_hyp.h b/arch/arm/include/asm/kvm_hyp.h
index ab20ffa8b9e7..b3dd4f4304f5 100644
--- a/arch/arm/include/asm/kvm_hyp.h
+++ b/arch/arm/include/asm/kvm_hyp.h
@@ -109,6 +109,8 @@ void __sysreg_restore_state(struct kvm_cpu_context *ctxt);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
+void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
asmlinkage void __vfp_save_state(struct vfp_hard_struct *vfp);
asmlinkage void __vfp_restore_state(struct vfp_hard_struct *vfp);
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index bd3fe6446728..62872ceefa33 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -125,6 +125,8 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
+void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __timer_enable_traps(struct kvm_vcpu *vcpu);
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index 9abf2f3c12b5..811b42c8441d 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -221,14 +221,11 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
int i;
- u32 nr_pre_bits;
u32 elrsr;
elrsr = read_gicreg(ICH_ELSR_EL2);
write_gicreg(0, ICH_HCR_EL2);
- val = read_gicreg(ICH_VTR_EL2);
- nr_pre_bits = vtr_to_nr_pre_bits(val);
for (i = 0; i < used_lrs; i++) {
if (elrsr & (1 << i))
@@ -238,39 +235,10 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
__gic_v3_set_lr(0, i);
}
-
- switch (nr_pre_bits) {
- case 7:
- cpu_if->vgic_ap0r[3] = __vgic_v3_read_ap0rn(3);
- cpu_if->vgic_ap0r[2] = __vgic_v3_read_ap0rn(2);
- case 6:
- cpu_if->vgic_ap0r[1] = __vgic_v3_read_ap0rn(1);
- default:
- cpu_if->vgic_ap0r[0] = __vgic_v3_read_ap0rn(0);
- }
-
- switch (nr_pre_bits) {
- case 7:
- cpu_if->vgic_ap1r[3] = __vgic_v3_read_ap1rn(3);
- cpu_if->vgic_ap1r[2] = __vgic_v3_read_ap1rn(2);
- case 6:
- cpu_if->vgic_ap1r[1] = __vgic_v3_read_ap1rn(1);
- default:
- cpu_if->vgic_ap1r[0] = __vgic_v3_read_ap1rn(0);
- }
} else {
if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
cpu_if->its_vpe.its_vm)
write_gicreg(0, ICH_HCR_EL2);
-
- cpu_if->vgic_ap0r[0] = 0;
- cpu_if->vgic_ap0r[1] = 0;
- cpu_if->vgic_ap0r[2] = 0;
- cpu_if->vgic_ap0r[3] = 0;
- cpu_if->vgic_ap1r[0] = 0;
- cpu_if->vgic_ap1r[1] = 0;
- cpu_if->vgic_ap1r[2] = 0;
- cpu_if->vgic_ap1r[3] = 0;
}
val = read_gicreg(ICC_SRE_EL2);
@@ -287,8 +255,6 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
{
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 val;
- u32 nr_pre_bits;
int i;
/*
@@ -306,32 +272,9 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
write_gicreg(cpu_if->vgic_vmcr, ICH_VMCR_EL2);
}
- val = read_gicreg(ICH_VTR_EL2);
- nr_pre_bits = vtr_to_nr_pre_bits(val);
-
if (used_lrs) {
write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
- switch (nr_pre_bits) {
- case 7:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[3], 3);
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[2], 2);
- case 6:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[1], 1);
- default:
- __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[0], 0);
- }
-
- switch (nr_pre_bits) {
- case 7:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[3], 3);
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[2], 2);
- case 6:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[1], 1);
- default:
- __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[0], 0);
- }
-
for (i = 0; i < used_lrs; i++)
__gic_v3_set_lr(cpu_if->vgic_lr[i], i);
} else {
@@ -364,6 +307,72 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
ICC_SRE_EL2);
}
+void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if;
+ u64 val;
+ u32 nr_pre_bits;
+
+ vcpu = kern_hyp_va(vcpu);
+ cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
+ val = read_gicreg(ICH_VTR_EL2);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
+
+ switch (nr_pre_bits) {
+ case 7:
+ cpu_if->vgic_ap0r[3] = __vgic_v3_read_ap0rn(3);
+ cpu_if->vgic_ap0r[2] = __vgic_v3_read_ap0rn(2);
+ case 6:
+ cpu_if->vgic_ap0r[1] = __vgic_v3_read_ap0rn(1);
+ default:
+ cpu_if->vgic_ap0r[0] = __vgic_v3_read_ap0rn(0);
+ }
+
+ switch (nr_pre_bits) {
+ case 7:
+ cpu_if->vgic_ap1r[3] = __vgic_v3_read_ap1rn(3);
+ cpu_if->vgic_ap1r[2] = __vgic_v3_read_ap1rn(2);
+ case 6:
+ cpu_if->vgic_ap1r[1] = __vgic_v3_read_ap1rn(1);
+ default:
+ cpu_if->vgic_ap1r[0] = __vgic_v3_read_ap1rn(0);
+ }
+}
+
+void __hyp_text __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if;
+ u64 val;
+ u32 nr_pre_bits;
+
+ vcpu = kern_hyp_va(vcpu);
+ cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
+ val = read_gicreg(ICH_VTR_EL2);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
+
+ switch (nr_pre_bits) {
+ case 7:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[3], 3);
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[2], 2);
+ case 6:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[1], 1);
+ default:
+ __vgic_v3_write_ap0rn(cpu_if->vgic_ap0r[0], 0);
+ }
+
+ switch (nr_pre_bits) {
+ case 7:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[3], 3);
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[2], 2);
+ case 6:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[1], 1);
+ default:
+ __vgic_v3_write_ap1rn(cpu_if->vgic_ap1r[0], 0);
+ }
+}
+
void __hyp_text __vgic_v3_init_lrs(void)
{
int max_lr_idx = vtr_to_max_lr_idx(read_gicreg(ICH_VTR_EL2));
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 1e5f3eb6973d..ca7cfee9f353 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -446,7 +446,6 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
struct vgic_dist *vgic = &kvm->arch.vgic;
- struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
void __iomem *base = vgic->vctrl_base;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
@@ -454,11 +453,8 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)
return;
if (used_lrs) {
- cpu_if->vgic_apr = readl_relaxed(base + GICH_APR);
save_lrs(vcpu, base);
writel_relaxed(0, base + GICH_HCR);
- } else {
- cpu_if->vgic_apr = 0;
}
}
@@ -476,7 +472,6 @@ void vgic_v2_restore_state(struct kvm_vcpu *vcpu)
if (used_lrs) {
writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
- writel_relaxed(cpu_if->vgic_apr, base + GICH_APR);
for (i = 0; i < used_lrs; i++) {
writel_relaxed(cpu_if->vgic_lr[i],
base + GICH_LR0 + (i * 4));
@@ -490,6 +485,7 @@ void vgic_v2_load(struct kvm_vcpu *vcpu)
struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
writel_relaxed(cpu_if->vgic_vmcr, vgic->vctrl_base + GICH_VMCR);
+ writel_relaxed(cpu_if->vgic_apr, vgic->vctrl_base + GICH_APR);
}
void vgic_v2_put(struct kvm_vcpu *vcpu)
@@ -498,4 +494,5 @@ void vgic_v2_put(struct kvm_vcpu *vcpu)
struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
cpu_if->vgic_vmcr = readl_relaxed(vgic->vctrl_base + GICH_VMCR);
+ cpu_if->vgic_apr = readl_relaxed(vgic->vctrl_base + GICH_APR);
}
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index b76e21f3e6bd..4bafcd1e6bb8 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -16,6 +16,7 @@
#include <linux/kvm.h>
#include <linux/kvm_host.h>
#include <kvm/arm_vgic.h>
+#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
#include <asm/kvm_asm.h>
@@ -587,6 +588,8 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
*/
if (likely(cpu_if->vgic_sre))
kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr);
+
+ kvm_call_hyp(__vgic_v3_restore_aprs, vcpu);
}
void vgic_v3_put(struct kvm_vcpu *vcpu)
@@ -595,4 +598,6 @@ void vgic_v3_put(struct kvm_vcpu *vcpu)
if (likely(cpu_if->vgic_sre))
cpu_if->vgic_vmcr = kvm_call_hyp(__vgic_v3_read_vmcr);
+
+ kvm_call_hyp(__vgic_v3_save_aprs, vcpu);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 36/36] KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-07 17:06 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: kvmarm, linux-arm-kernel; +Cc: Marc Zyngier, Shih-Wei Li, kvm
We can finally get completely rid of any calls to the VGICv3
save/restore functions when the AP lists are empty on VHE systems. This
requires carefully factoring out trap configuration from saving and
restoring state, and carefully choosing what to do on the VHE and
non-VHE path.
One of the challenges is that we cannot save/restore the VMCR lazily
because we can only write the VMCR when ICC_SRE_EL1.SRE is cleared when
emulating a GICv2-on-GICv3, since otherwise all Group-0 interrupts end
up being delivered as FIQ.
To solve this problem, and still provide fast performance in the fast
path of exiting a VM when no interrupts are pending (which also
optimized the latency for actually delivering virtual interrupts coming
from physical interrupts), we orchestrate a dance of only doing the
activate/deactivate traps in vgic load/put for VHE systems (which can
have ICC_SRE_EL1.SRE cleared when running in the host), and doing the
configuration on every round-trip on non-VHE systems.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_hyp.h | 2 +
arch/arm/kvm/hyp/switch.c | 8 ++-
arch/arm64/include/asm/kvm_hyp.h | 2 +
arch/arm64/kvm/hyp/switch.c | 8 ++-
virt/kvm/arm/hyp/vgic-v3-sr.c | 121 +++++++++++++++++++++++++--------------
virt/kvm/arm/vgic/vgic-v3.c | 6 ++
virt/kvm/arm/vgic/vgic.c | 7 +--
7 files changed, 103 insertions(+), 51 deletions(-)
diff --git a/arch/arm/include/asm/kvm_hyp.h b/arch/arm/include/asm/kvm_hyp.h
index b3dd4f4304f5..d01676e5b816 100644
--- a/arch/arm/include/asm/kvm_hyp.h
+++ b/arch/arm/include/asm/kvm_hyp.h
@@ -109,6 +109,8 @@ void __sysreg_restore_state(struct kvm_cpu_context *ctxt);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu);
+void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu);
void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 214187446e63..337c76230885 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -89,14 +89,18 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
__vgic_v3_save_state(vcpu);
+ __vgic_v3_deactivate_traps(vcpu);
+ }
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
+ __vgic_v3_activate_traps(vcpu);
__vgic_v3_restore_state(vcpu);
+ }
}
static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 62872ceefa33..1c21a706fc83 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -125,6 +125,8 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu);
+void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu);
void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 947684312079..832aa3e928fe 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -185,14 +185,18 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
__vgic_v3_save_state(vcpu);
+ __vgic_v3_deactivate_traps(vcpu);
+ }
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
+ __vgic_v3_activate_traps(vcpu);
__vgic_v3_restore_state(vcpu);
+ }
}
static bool __hyp_text __true_value(void)
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index 811b42c8441d..e5f3bc7582b6 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -208,15 +208,15 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
{
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 val;
/*
* Make sure stores to the GIC via the memory mapped interface
- * are now visible to the system register interface.
+ * are now visible to the system register interface when reading the
+ * LRs, and when reading back the VMCR on non-VHE systems.
*/
- if (!cpu_if->vgic_sre) {
- dsb(st);
- cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
+ if (used_lrs || !has_vhe()) {
+ if (!cpu_if->vgic_sre)
+ dsb(st);
}
if (used_lrs) {
@@ -225,7 +225,7 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
elrsr = read_gicreg(ICH_ELSR_EL2);
- write_gicreg(0, ICH_HCR_EL2);
+ write_gicreg(cpu_if->vgic_hcr & ~ICH_HCR_EN, ICH_HCR_EL2);
for (i = 0; i < used_lrs; i++) {
if (elrsr & (1 << i))
@@ -235,19 +235,6 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
__gic_v3_set_lr(0, i);
}
- } else {
- if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
- cpu_if->its_vpe.its_vm)
- write_gicreg(0, ICH_HCR_EL2);
- }
-
- val = read_gicreg(ICC_SRE_EL2);
- write_gicreg(val | ICC_SRE_EL2_ENABLE, ICC_SRE_EL2);
-
- if (!cpu_if->vgic_sre) {
- /* Make sure ENABLE is set at EL2 before setting SRE at EL1 */
- isb();
- write_gicreg(1, ICC_SRE_EL1);
}
}
@@ -257,6 +244,31 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
int i;
+ if (used_lrs) {
+ write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+
+ for (i = 0; i < used_lrs; i++)
+ __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
+ }
+
+ /*
+ * Ensure that writes to the LRs, and on non-VHE systems ensure that
+ * the write to the VMCR in __vgic_v3_activate_traps(), will have
+ * reached the (re)distributors. This ensure the guest will read the
+ * correct values from the memory-mapped interface.
+ */
+ if (used_lrs || !has_vhe()) {
+ if (!cpu_if->vgic_sre) {
+ isb();
+ dsb(sy);
+ }
+ }
+}
+
+void __hyp_text __vgic_v3_activate_traps(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
/*
* VFIQEn is RES1 if ICC_SRE_EL1.SRE is 1. This causes a
* Group0 interrupt (as generated in GICv2 mode) to be
@@ -264,47 +276,70 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
* consequences. So we must make sure that ICC_SRE_EL1 has
* been actually programmed with the value we want before
* starting to mess with the rest of the GIC, and VMCR_EL2 in
- * particular.
+ * particular. This logic must be called before
+ * __vgic_v3_restore_state().
*/
if (!cpu_if->vgic_sre) {
write_gicreg(0, ICC_SRE_EL1);
isb();
write_gicreg(cpu_if->vgic_vmcr, ICH_VMCR_EL2);
- }
- if (used_lrs) {
- write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
- for (i = 0; i < used_lrs; i++)
- __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
- } else {
- /*
- * If we need to trap system registers, we must write
- * ICH_HCR_EL2 anyway, even if no interrupts are being
- * injected. Same thing if GICv4 is used, as VLPI
- * delivery is gated by ICH_HCR_EL2.En.
- */
- if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
- cpu_if->its_vpe.its_vm)
- write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+ if (has_vhe()) {
+ /*
+ * Ensure that the write to the VMCR will have reached
+ * the (re)distributors. This ensure the guest will
+ * read the correct values from the memory-mapped
+ * interface.
+ */
+ isb();
+ dsb(sy);
+ }
}
/*
- * Ensures that the above will have reached the
- * (re)distributors. This ensure the guest will read the
- * correct values from the memory-mapped interface.
+ * Prevent the guest from touching the GIC system registers if
+ * SRE isn't enabled for GICv3 emulation.
+ */
+ write_gicreg(read_gicreg(ICC_SRE_EL2) & ~ICC_SRE_EL2_ENABLE,
+ ICC_SRE_EL2);
+
+ /*
+ * If we need to trap system registers, we must write
+ * ICH_HCR_EL2 anyway, even if no interrupts are being
+ * injected,
*/
+ if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
+ cpu_if->its_vpe.its_vm)
+ write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+}
+
+void __hyp_text __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+ u64 val;
+
if (!cpu_if->vgic_sre) {
+ cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
+ }
+
+
+ val = read_gicreg(ICC_SRE_EL2);
+ write_gicreg(val | ICC_SRE_EL2_ENABLE, ICC_SRE_EL2);
+
+ if (!cpu_if->vgic_sre) {
+ /* Make sure ENABLE is set at EL2 before setting SRE at EL1 */
isb();
- dsb(sy);
+ write_gicreg(1, ICC_SRE_EL1);
}
/*
- * Prevent the guest from touching the GIC system registers if
- * SRE isn't enabled for GICv3 emulation.
+ * If we were trapping system registers, we enabled the VGIC even if
+ * no interrupts were being injected, and we disable it again here.
*/
- write_gicreg(read_gicreg(ICC_SRE_EL2) & ~ICC_SRE_EL2_ENABLE,
- ICC_SRE_EL2);
+ if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
+ cpu_if->its_vpe.its_vm)
+ write_gicreg(0, ICH_HCR_EL2);
}
void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu)
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 4bafcd1e6bb8..4200657694f0 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -590,6 +590,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr);
kvm_call_hyp(__vgic_v3_restore_aprs, vcpu);
+
+ if (has_vhe())
+ __vgic_v3_activate_traps(vcpu);
}
void vgic_v3_put(struct kvm_vcpu *vcpu)
@@ -600,4 +603,7 @@ void vgic_v3_put(struct kvm_vcpu *vcpu)
cpu_if->vgic_vmcr = kvm_call_hyp(__vgic_v3_read_vmcr);
kvm_call_hyp(__vgic_v3_save_aprs, vcpu);
+
+ if (has_vhe())
+ __vgic_v3_deactivate_traps(vcpu);
}
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 636200cace55..352eb09d8fab 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -764,14 +764,14 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
- vgic_save_state(vcpu);
-
WARN_ON(vgic_v4_sync_hwstate(vcpu));
/* An empty ap_list_head implies used_lrs == 0 */
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
return;
+ vgic_save_state(vcpu);
+
if (vgic_cpu->used_lrs)
vgic_fold_lr_state(vcpu);
vgic_prune_ap_list(vcpu);
@@ -800,7 +800,7 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
* this.
*/
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
- goto out;
+ return;
DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
@@ -808,7 +808,6 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
vgic_flush_lr_state(vcpu);
spin_unlock(&vcpu->arch.vgic_cpu.ap_list_lock);
-out:
vgic_restore_state(vcpu);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 36/36] KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs
@ 2017-12-07 17:06 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-07 17:06 UTC (permalink / raw)
To: linux-arm-kernel
We can finally get completely rid of any calls to the VGICv3
save/restore functions when the AP lists are empty on VHE systems. This
requires carefully factoring out trap configuration from saving and
restoring state, and carefully choosing what to do on the VHE and
non-VHE path.
One of the challenges is that we cannot save/restore the VMCR lazily
because we can only write the VMCR when ICC_SRE_EL1.SRE is cleared when
emulating a GICv2-on-GICv3, since otherwise all Group-0 interrupts end
up being delivered as FIQ.
To solve this problem, and still provide fast performance in the fast
path of exiting a VM when no interrupts are pending (which also
optimized the latency for actually delivering virtual interrupts coming
from physical interrupts), we orchestrate a dance of only doing the
activate/deactivate traps in vgic load/put for VHE systems (which can
have ICC_SRE_EL1.SRE cleared when running in the host), and doing the
configuration on every round-trip on non-VHE systems.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
---
arch/arm/include/asm/kvm_hyp.h | 2 +
arch/arm/kvm/hyp/switch.c | 8 ++-
arch/arm64/include/asm/kvm_hyp.h | 2 +
arch/arm64/kvm/hyp/switch.c | 8 ++-
virt/kvm/arm/hyp/vgic-v3-sr.c | 121 +++++++++++++++++++++++++--------------
virt/kvm/arm/vgic/vgic-v3.c | 6 ++
virt/kvm/arm/vgic/vgic.c | 7 +--
7 files changed, 103 insertions(+), 51 deletions(-)
diff --git a/arch/arm/include/asm/kvm_hyp.h b/arch/arm/include/asm/kvm_hyp.h
index b3dd4f4304f5..d01676e5b816 100644
--- a/arch/arm/include/asm/kvm_hyp.h
+++ b/arch/arm/include/asm/kvm_hyp.h
@@ -109,6 +109,8 @@ void __sysreg_restore_state(struct kvm_cpu_context *ctxt);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu);
+void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu);
void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
index 214187446e63..337c76230885 100644
--- a/arch/arm/kvm/hyp/switch.c
+++ b/arch/arm/kvm/hyp/switch.c
@@ -89,14 +89,18 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
__vgic_v3_save_state(vcpu);
+ __vgic_v3_deactivate_traps(vcpu);
+ }
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
+ __vgic_v3_activate_traps(vcpu);
__vgic_v3_restore_state(vcpu);
+ }
}
static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 62872ceefa33..1c21a706fc83 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -125,6 +125,8 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
void __vgic_v3_save_state(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_state(struct kvm_vcpu *vcpu);
+void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu);
+void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu);
void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu);
void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu);
int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 947684312079..832aa3e928fe 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -185,14 +185,18 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu)
static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
__vgic_v3_save_state(vcpu);
+ __vgic_v3_deactivate_traps(vcpu);
+ }
}
static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)
{
- if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
+ if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
+ __vgic_v3_activate_traps(vcpu);
__vgic_v3_restore_state(vcpu);
+ }
}
static bool __hyp_text __true_value(void)
diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c
index 811b42c8441d..e5f3bc7582b6 100644
--- a/virt/kvm/arm/hyp/vgic-v3-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v3-sr.c
@@ -208,15 +208,15 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
{
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
- u64 val;
/*
* Make sure stores to the GIC via the memory mapped interface
- * are now visible to the system register interface.
+ * are now visible to the system register interface when reading the
+ * LRs, and when reading back the VMCR on non-VHE systems.
*/
- if (!cpu_if->vgic_sre) {
- dsb(st);
- cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
+ if (used_lrs || !has_vhe()) {
+ if (!cpu_if->vgic_sre)
+ dsb(st);
}
if (used_lrs) {
@@ -225,7 +225,7 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
elrsr = read_gicreg(ICH_ELSR_EL2);
- write_gicreg(0, ICH_HCR_EL2);
+ write_gicreg(cpu_if->vgic_hcr & ~ICH_HCR_EN, ICH_HCR_EL2);
for (i = 0; i < used_lrs; i++) {
if (elrsr & (1 << i))
@@ -235,19 +235,6 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu)
__gic_v3_set_lr(0, i);
}
- } else {
- if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
- cpu_if->its_vpe.its_vm)
- write_gicreg(0, ICH_HCR_EL2);
- }
-
- val = read_gicreg(ICC_SRE_EL2);
- write_gicreg(val | ICC_SRE_EL2_ENABLE, ICC_SRE_EL2);
-
- if (!cpu_if->vgic_sre) {
- /* Make sure ENABLE is set at EL2 before setting SRE at EL1 */
- isb();
- write_gicreg(1, ICC_SRE_EL1);
}
}
@@ -257,6 +244,31 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
int i;
+ if (used_lrs) {
+ write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+
+ for (i = 0; i < used_lrs; i++)
+ __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
+ }
+
+ /*
+ * Ensure that writes to the LRs, and on non-VHE systems ensure that
+ * the write to the VMCR in __vgic_v3_activate_traps(), will have
+ * reached the (re)distributors. This ensure the guest will read the
+ * correct values from the memory-mapped interface.
+ */
+ if (used_lrs || !has_vhe()) {
+ if (!cpu_if->vgic_sre) {
+ isb();
+ dsb(sy);
+ }
+ }
+}
+
+void __hyp_text __vgic_v3_activate_traps(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+
/*
* VFIQEn is RES1 if ICC_SRE_EL1.SRE is 1. This causes a
* Group0 interrupt (as generated in GICv2 mode) to be
@@ -264,47 +276,70 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu)
* consequences. So we must make sure that ICC_SRE_EL1 has
* been actually programmed with the value we want before
* starting to mess with the rest of the GIC, and VMCR_EL2 in
- * particular.
+ * particular. This logic must be called before
+ * __vgic_v3_restore_state().
*/
if (!cpu_if->vgic_sre) {
write_gicreg(0, ICC_SRE_EL1);
isb();
write_gicreg(cpu_if->vgic_vmcr, ICH_VMCR_EL2);
- }
- if (used_lrs) {
- write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
- for (i = 0; i < used_lrs; i++)
- __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
- } else {
- /*
- * If we need to trap system registers, we must write
- * ICH_HCR_EL2 anyway, even if no interrupts are being
- * injected. Same thing if GICv4 is used, as VLPI
- * delivery is gated by ICH_HCR_EL2.En.
- */
- if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
- cpu_if->its_vpe.its_vm)
- write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+ if (has_vhe()) {
+ /*
+ * Ensure that the write to the VMCR will have reached
+ * the (re)distributors. This ensure the guest will
+ * read the correct values from the memory-mapped
+ * interface.
+ */
+ isb();
+ dsb(sy);
+ }
}
/*
- * Ensures that the above will have reached the
- * (re)distributors. This ensure the guest will read the
- * correct values from the memory-mapped interface.
+ * Prevent the guest from touching the GIC system registers if
+ * SRE isn't enabled for GICv3 emulation.
+ */
+ write_gicreg(read_gicreg(ICC_SRE_EL2) & ~ICC_SRE_EL2_ENABLE,
+ ICC_SRE_EL2);
+
+ /*
+ * If we need to trap system registers, we must write
+ * ICH_HCR_EL2 anyway, even if no interrupts are being
+ * injected,
*/
+ if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
+ cpu_if->its_vpe.its_vm)
+ write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
+}
+
+void __hyp_text __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu)
+{
+ struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
+ u64 val;
+
if (!cpu_if->vgic_sre) {
+ cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
+ }
+
+
+ val = read_gicreg(ICC_SRE_EL2);
+ write_gicreg(val | ICC_SRE_EL2_ENABLE, ICC_SRE_EL2);
+
+ if (!cpu_if->vgic_sre) {
+ /* Make sure ENABLE is set at EL2 before setting SRE at EL1 */
isb();
- dsb(sy);
+ write_gicreg(1, ICC_SRE_EL1);
}
/*
- * Prevent the guest from touching the GIC system registers if
- * SRE isn't enabled for GICv3 emulation.
+ * If we were trapping system registers, we enabled the VGIC even if
+ * no interrupts were being injected, and we disable it again here.
*/
- write_gicreg(read_gicreg(ICC_SRE_EL2) & ~ICC_SRE_EL2_ENABLE,
- ICC_SRE_EL2);
+ if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
+ cpu_if->its_vpe.its_vm)
+ write_gicreg(0, ICH_HCR_EL2);
}
void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu)
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 4bafcd1e6bb8..4200657694f0 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -590,6 +590,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr);
kvm_call_hyp(__vgic_v3_restore_aprs, vcpu);
+
+ if (has_vhe())
+ __vgic_v3_activate_traps(vcpu);
}
void vgic_v3_put(struct kvm_vcpu *vcpu)
@@ -600,4 +603,7 @@ void vgic_v3_put(struct kvm_vcpu *vcpu)
cpu_if->vgic_vmcr = kvm_call_hyp(__vgic_v3_read_vmcr);
kvm_call_hyp(__vgic_v3_save_aprs, vcpu);
+
+ if (has_vhe())
+ __vgic_v3_deactivate_traps(vcpu);
}
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 636200cace55..352eb09d8fab 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -764,14 +764,14 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
- vgic_save_state(vcpu);
-
WARN_ON(vgic_v4_sync_hwstate(vcpu));
/* An empty ap_list_head implies used_lrs == 0 */
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
return;
+ vgic_save_state(vcpu);
+
if (vgic_cpu->used_lrs)
vgic_fold_lr_state(vcpu);
vgic_prune_ap_list(vcpu);
@@ -800,7 +800,7 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
* this.
*/
if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
- goto out;
+ return;
DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
@@ -808,7 +808,6 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)
vgic_flush_lr_state(vcpu);
spin_unlock(&vcpu->arch.vgic_cpu.ap_list_lock);
-out:
vgic_restore_state(vcpu);
}
--
2.14.2
^ permalink raw reply related [flat|nested] 158+ messages in thread
* Re: [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-09 17:19 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:19 UTC (permalink / raw)
To: Christoffer Dall
Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones, Ard Biesheuvel
On Thu, 07 Dec 2017 17:05:55 +0000,
Christoffer Dall wrote:
>
> We already have the percpu area for the host cpu state, which points to
> the VCPU, so there's no need to store the VCPU pointer on the stack on
> every context switch. We can be a little more clever and just use
> tpidr_el2 for the percpu offset and load the VCPU pointer from the host
> context.
>
> This does require us to calculate the percpu offset without including
> the offset from the kernel mapping of the percpu array to the linaro
"linaro"?
> mapping of the array (which is what we store in tpidr_el1), because a
> PC-relative generated address in EL2 is already giving us the hyp alias
> of the linear mapping of a kernel address. We do this in
> __cpu_init_hyp_mode() by using kvm_ksym_ref().
>
> This change also requires us to have a scratch register, so we take the
> chance to rearrange some of the el1_sync code to only look at the
> vttbr_el2 to determine if this is a trap from the guest or an HVC from
> the host. We do add an extra check to call the panic code if the kernel
> is configured with debugging enabled and we saw a trap from the host
> which wasn't an HVC, indicating that we left some EL2 trap configured by
> mistake.
>
> The code that accesses ESR_EL2 was previously using an alternative to
> use the _EL1 accessor on VHE systems, but this was actually unnecessary
> as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
> accessor does the same thing on both systems.
>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Use PC-relative addressing to access per-cpu variables instead of
> using a load from the literal pool.
> - Remove stale comments as pointed out by Marc
> - Reworded the commit message as suggested by Drew
>
> arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
> arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
> arch/arm64/kernel/asm-offsets.c | 1 +
> arch/arm64/kvm/hyp/entry.S | 6 +-----
> arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
> arch/arm64/kvm/hyp/switch.c | 5 +----
> arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
> 7 files changed, 57 insertions(+), 31 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index ab4d0a926043..33e0edc6f8be 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -33,6 +33,7 @@
> #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
> #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
>
> +/* Translate a kernel address of @sym into its equivalent linear mapping */
> #define kvm_ksym_ref(sym) \
> ({ \
> void *val = &sym; \
> @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
>
> #endif
>
> +#ifdef __ASSEMBLY__
You could turn this and the previous #endif into a simple #else.
> +.macro get_host_ctxt reg, tmp
> + adr_l \reg, kvm_host_cpu_state
> + mrs \tmp, tpidr_el2
> + add \reg, \reg, \tmp
> +.endm
> +
> +.macro get_vcpu vcpu, ctxt
> + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
> + kern_hyp_va \vcpu
> +.endm
> +
> +#endif
> +
> #endif /* __ARM_KVM_ASM_H__ */
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 7ee72b402907..af58deb6ec3c 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
>
> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>
> +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
Is there any advantage in having this prototype in kvm_host.h, instead
of putting it in kvm_hyp.h (which feels more appropriate)?
> +DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
> +
> static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> unsigned long hyp_stack_ptr,
> unsigned long vector_ptr)
> {
> + u64 tpidr_el2;
> +
> /*
> * Call initialization code, and switch to the full blown HYP code.
> * If the cpucaps haven't been finalized yet, something has gone very
> @@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> */
> BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
> __kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
> +
> + /*
> + * Calculate the raw per-cpu offset without a translation from the
> + * kernel's mapping to the linear mapping, and store it in tpidr_el2
> + * so that we can use adr_l to access per-cpu variables in EL2.
> + */
> + tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
> + - (u64)kvm_ksym_ref(kvm_host_cpu_state);
> +
> + kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
> }
>
> static inline void kvm_arch_hardware_unsetup(void) {}
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 71bf088f1e4b..612021dce84f 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -135,6 +135,7 @@ int main(void)
> DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> + DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> #endif
> #ifdef CONFIG_CPU_PM
> DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
> diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> index 9a8ab5dddd9e..a360ac6e89e9 100644
> --- a/arch/arm64/kvm/hyp/entry.S
> +++ b/arch/arm64/kvm/hyp/entry.S
> @@ -62,9 +62,6 @@ ENTRY(__guest_enter)
> // Store the host regs
> save_callee_saved_regs x1
>
> - // Store host_ctxt and vcpu for use at exit time
> - stp x1, x0, [sp, #-16]!
> -
> add x18, x0, #VCPU_CONTEXT
>
> // Restore guest regs x0-x17
> @@ -118,8 +115,7 @@ ENTRY(__guest_exit)
> // Store the guest regs x19-x29, lr
> save_callee_saved_regs x1
>
> - // Restore the host_ctxt from the stack
> - ldr x2, [sp], #16
> + get_host_ctxt x2, x3
>
> // Now restore the host regs
> restore_callee_saved_regs x2
> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> index e4f37b9dd47c..71b4cc92895e 100644
> --- a/arch/arm64/kvm/hyp/hyp-entry.S
> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> @@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
> el1_sync: // Guest trapped into EL2
> stp x0, x1, [sp, #-16]!
>
> -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> - mrs x1, esr_el2
> -alternative_else
> - mrs x1, esr_el1
> -alternative_endif
> - lsr x0, x1, #ESR_ELx_EC_SHIFT
> + mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
> + cbnz x1, el1_trap // from the guest
>
> - cmp x0, #ESR_ELx_EC_HVC64
> - b.ne el1_trap
> -
> - mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
> - cbnz x1, el1_trap // called HVC
> +#ifdef CONFIG_DEBUG
> + mrs x0, esr_el2
> + lsr x0, x0, #ESR_ELx_EC_SHIFT
> + cmp x0, #ESR_ELx_EC_HVC64
> + b.ne __hyp_panic
> +#endif
>
> /* Here, we're pretty sure the host called HVC. */
> ldp x0, x1, [sp], #16
> @@ -101,10 +98,15 @@ alternative_endif
> eret
>
> el1_trap:
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> +
> + mrs x0, esr_el2
> + lsr x0, x0, #ESR_ELx_EC_SHIFT
> /*
> * x0: ESR_EC
> + * x1: vcpu pointer
> */
> - ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
>
> /*
> * We trap the first access to the FP/SIMD to save the host context
> @@ -122,13 +124,15 @@ alternative_else_nop_endif
>
> el1_irq:
> stp x0, x1, [sp, #-16]!
> - ldr x1, [sp, #16 + 8]
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> mov x0, #ARM_EXCEPTION_IRQ
> b __guest_exit
>
> el1_error:
> stp x0, x1, [sp, #-16]!
> - ldr x1, [sp, #16 + 8]
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> mov x0, #ARM_EXCEPTION_EL1_SERROR
> b __guest_exit
>
> @@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
> ENDPROC(__hyp_do_panic)
>
> ENTRY(__hyp_panic)
> - /*
> - * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
> - * not be accessible by this address from EL2, hyp_panic() converts
> - * it with kern_hyp_va() before use.
> - */
> - ldr x0, =kvm_host_cpu_state
> - mrs x1, tpidr_el2
> - add x0, x0, x1
> + get_host_ctxt x0, x1
> b hyp_panic
> ENDPROC(__hyp_panic)
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index f7307b6b42f0..6fcb37e220b5 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
> __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> -void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> +void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
> {
> struct kvm_vcpu *vcpu = NULL;
>
> @@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> u64 par = read_sysreg(par_el1);
>
> if (read_sysreg(vttbr_el2)) {
> - struct kvm_cpu_context *host_ctxt;
> -
> - host_ctxt = kern_hyp_va(__host_ctxt);
> vcpu = host_ctxt->__hyp_running_vcpu;
> __timer_disable_traps(vcpu);
> __deactivate_traps(vcpu);
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index c54cc2afb92b..e19d89cabf2a 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
> if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
> }
> +
> +void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> +{
> + asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
> +}
> --
> 2.14.2
>
Other than the couple of nits above:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
@ 2017-12-09 17:19 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:19 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:05:55 +0000,
Christoffer Dall wrote:
>
> We already have the percpu area for the host cpu state, which points to
> the VCPU, so there's no need to store the VCPU pointer on the stack on
> every context switch. We can be a little more clever and just use
> tpidr_el2 for the percpu offset and load the VCPU pointer from the host
> context.
>
> This does require us to calculate the percpu offset without including
> the offset from the kernel mapping of the percpu array to the linaro
"linaro"?
> mapping of the array (which is what we store in tpidr_el1), because a
> PC-relative generated address in EL2 is already giving us the hyp alias
> of the linear mapping of a kernel address. We do this in
> __cpu_init_hyp_mode() by using kvm_ksym_ref().
>
> This change also requires us to have a scratch register, so we take the
> chance to rearrange some of the el1_sync code to only look at the
> vttbr_el2 to determine if this is a trap from the guest or an HVC from
> the host. We do add an extra check to call the panic code if the kernel
> is configured with debugging enabled and we saw a trap from the host
> which wasn't an HVC, indicating that we left some EL2 trap configured by
> mistake.
>
> The code that accesses ESR_EL2 was previously using an alternative to
> use the _EL1 accessor on VHE systems, but this was actually unnecessary
> as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
> accessor does the same thing on both systems.
>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Use PC-relative addressing to access per-cpu variables instead of
> using a load from the literal pool.
> - Remove stale comments as pointed out by Marc
> - Reworded the commit message as suggested by Drew
>
> arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
> arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
> arch/arm64/kernel/asm-offsets.c | 1 +
> arch/arm64/kvm/hyp/entry.S | 6 +-----
> arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
> arch/arm64/kvm/hyp/switch.c | 5 +----
> arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
> 7 files changed, 57 insertions(+), 31 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index ab4d0a926043..33e0edc6f8be 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -33,6 +33,7 @@
> #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
> #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
>
> +/* Translate a kernel address of @sym into its equivalent linear mapping */
> #define kvm_ksym_ref(sym) \
> ({ \
> void *val = &sym; \
> @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
>
> #endif
>
> +#ifdef __ASSEMBLY__
You could turn this and the previous #endif into a simple #else.
> +.macro get_host_ctxt reg, tmp
> + adr_l \reg, kvm_host_cpu_state
> + mrs \tmp, tpidr_el2
> + add \reg, \reg, \tmp
> +.endm
> +
> +.macro get_vcpu vcpu, ctxt
> + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
> + kern_hyp_va \vcpu
> +.endm
> +
> +#endif
> +
> #endif /* __ARM_KVM_ASM_H__ */
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 7ee72b402907..af58deb6ec3c 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
>
> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>
> +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
Is there any advantage in having this prototype in kvm_host.h, instead
of putting it in kvm_hyp.h (which feels more appropriate)?
> +DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
> +
> static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> unsigned long hyp_stack_ptr,
> unsigned long vector_ptr)
> {
> + u64 tpidr_el2;
> +
> /*
> * Call initialization code, and switch to the full blown HYP code.
> * If the cpucaps haven't been finalized yet, something has gone very
> @@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> */
> BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
> __kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
> +
> + /*
> + * Calculate the raw per-cpu offset without a translation from the
> + * kernel's mapping to the linear mapping, and store it in tpidr_el2
> + * so that we can use adr_l to access per-cpu variables in EL2.
> + */
> + tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
> + - (u64)kvm_ksym_ref(kvm_host_cpu_state);
> +
> + kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
> }
>
> static inline void kvm_arch_hardware_unsetup(void) {}
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 71bf088f1e4b..612021dce84f 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -135,6 +135,7 @@ int main(void)
> DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> + DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> #endif
> #ifdef CONFIG_CPU_PM
> DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
> diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> index 9a8ab5dddd9e..a360ac6e89e9 100644
> --- a/arch/arm64/kvm/hyp/entry.S
> +++ b/arch/arm64/kvm/hyp/entry.S
> @@ -62,9 +62,6 @@ ENTRY(__guest_enter)
> // Store the host regs
> save_callee_saved_regs x1
>
> - // Store host_ctxt and vcpu for use at exit time
> - stp x1, x0, [sp, #-16]!
> -
> add x18, x0, #VCPU_CONTEXT
>
> // Restore guest regs x0-x17
> @@ -118,8 +115,7 @@ ENTRY(__guest_exit)
> // Store the guest regs x19-x29, lr
> save_callee_saved_regs x1
>
> - // Restore the host_ctxt from the stack
> - ldr x2, [sp], #16
> + get_host_ctxt x2, x3
>
> // Now restore the host regs
> restore_callee_saved_regs x2
> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> index e4f37b9dd47c..71b4cc92895e 100644
> --- a/arch/arm64/kvm/hyp/hyp-entry.S
> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> @@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
> el1_sync: // Guest trapped into EL2
> stp x0, x1, [sp, #-16]!
>
> -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> - mrs x1, esr_el2
> -alternative_else
> - mrs x1, esr_el1
> -alternative_endif
> - lsr x0, x1, #ESR_ELx_EC_SHIFT
> + mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
> + cbnz x1, el1_trap // from the guest
>
> - cmp x0, #ESR_ELx_EC_HVC64
> - b.ne el1_trap
> -
> - mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
> - cbnz x1, el1_trap // called HVC
> +#ifdef CONFIG_DEBUG
> + mrs x0, esr_el2
> + lsr x0, x0, #ESR_ELx_EC_SHIFT
> + cmp x0, #ESR_ELx_EC_HVC64
> + b.ne __hyp_panic
> +#endif
>
> /* Here, we're pretty sure the host called HVC. */
> ldp x0, x1, [sp], #16
> @@ -101,10 +98,15 @@ alternative_endif
> eret
>
> el1_trap:
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> +
> + mrs x0, esr_el2
> + lsr x0, x0, #ESR_ELx_EC_SHIFT
> /*
> * x0: ESR_EC
> + * x1: vcpu pointer
> */
> - ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
>
> /*
> * We trap the first access to the FP/SIMD to save the host context
> @@ -122,13 +124,15 @@ alternative_else_nop_endif
>
> el1_irq:
> stp x0, x1, [sp, #-16]!
> - ldr x1, [sp, #16 + 8]
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> mov x0, #ARM_EXCEPTION_IRQ
> b __guest_exit
>
> el1_error:
> stp x0, x1, [sp, #-16]!
> - ldr x1, [sp, #16 + 8]
> + get_host_ctxt x0, x1
> + get_vcpu x1, x0
> mov x0, #ARM_EXCEPTION_EL1_SERROR
> b __guest_exit
>
> @@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
> ENDPROC(__hyp_do_panic)
>
> ENTRY(__hyp_panic)
> - /*
> - * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
> - * not be accessible by this address from EL2, hyp_panic() converts
> - * it with kern_hyp_va() before use.
> - */
> - ldr x0, =kvm_host_cpu_state
> - mrs x1, tpidr_el2
> - add x0, x0, x1
> + get_host_ctxt x0, x1
> b hyp_panic
> ENDPROC(__hyp_panic)
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index f7307b6b42f0..6fcb37e220b5 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
> __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> -void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> +void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
> {
> struct kvm_vcpu *vcpu = NULL;
>
> @@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> u64 par = read_sysreg(par_el1);
>
> if (read_sysreg(vttbr_el2)) {
> - struct kvm_cpu_context *host_ctxt;
> -
> - host_ctxt = kern_hyp_va(__host_ctxt);
> vcpu = host_ctxt->__hyp_running_vcpu;
> __timer_disable_traps(vcpu);
> __deactivate_traps(vcpu);
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index c54cc2afb92b..e19d89cabf2a 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
> if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
> }
> +
> +void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> +{
> + asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
> +}
> --
> 2.14.2
>
Other than the couple of nits above:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 02/36] KVM: arm64: Rework hyp_panic for VHE and non-VHE
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-09 17:24 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:24 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Thu, 07 Dec 2017 17:05:56 +0000,
Christoffer Dall wrote:
>
> VHE actually doesn't rely on clearing the VTTBR when returning to the
> host kernel, and that is the current key mechanism of hyp_panic to
> figure out how to attempt to return to a state good enough to print a
> panic statement.
>
> Therefore, we split the hyp_panic function into two functions, a VHE and
> a non-VHE, keeping the non-VHE version intact, but changing the VHE
> behavior.
>
> The vttbr_el2 check on VHE doesn't really make that much sense, because
> the only situation where we can get here on VHE is when the hypervisor
> assembly code actually called into hyp_panic, which only happens when
> VBAR_EL2 has been set to the KVM exception vectors. On VHE, we can
> always safely disable the traps and restore the host registers at this
> point, so we simply do that unconditionally and call into the panic
> function directly.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 02/36] KVM: arm64: Rework hyp_panic for VHE and non-VHE
@ 2017-12-09 17:24 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:24 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:05:56 +0000,
Christoffer Dall wrote:
>
> VHE actually doesn't rely on clearing the VTTBR when returning to the
> host kernel, and that is the current key mechanism of hyp_panic to
> figure out how to attempt to return to a state good enough to print a
> panic statement.
>
> Therefore, we split the hyp_panic function into two functions, a VHE and
> a non-VHE, keeping the non-VHE version intact, but changing the VHE
> behavior.
>
> The vttbr_el2 check on VHE doesn't really make that much sense, because
> the only situation where we can get here on VHE is when the hypervisor
> assembly code actually called into hyp_panic, which only happens when
> VBAR_EL2 has been set to the KVM exception vectors. On VHE, we can
> always safely disable the traps and restore the host registers at this
> point, so we simply do that unconditionally and call into the panic
> function directly.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 05/36] KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-09 17:30 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:30 UTC (permalink / raw)
To: Christoffer Dall; +Cc: Shih-Wei Li, kvmarm, linux-arm-kernel, kvm
On Thu, 07 Dec 2017 17:05:59 +0000,
Christoffer Dall wrote:
>
> As we are about to move a bunch of save/restore logic for VHE kernels to
> the load and put functions, we need some infrastructure to do this.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 05/36] KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
@ 2017-12-09 17:30 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:30 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:05:59 +0000,
Christoffer Dall wrote:
>
> As we are about to move a bunch of save/restore logic for VHE kernels to
> the load and put functions, we need some infrastructure to do this.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-09 17:37 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:37 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Thu, 07 Dec 2017 17:06:00 +0000,
Christoffer Dall wrote:
>
> Avoid saving the guest VFP registers and restoring the host VFP
> registers on every exit from the VM. Only when we're about to run
> userspace or other threads in the kernel do we really have to switch the
> state back to the host state.
>
> We still initially configure the VFP registers to trap when entering the
> VM, but the difference is that we now leave the guest state in the
> hardware registers as long as we're running this VCPU, even if we
> occasionally trap to the host, and we only restore the host state when
> we return to user space or when scheduling another thread.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Cosmetic changes
> - Change the flags variable to a u8
> - Expanded the commit message
>
> arch/arm64/include/asm/kvm_emulate.h | 5 ++++
> arch/arm64/include/asm/kvm_host.h | 3 +++
> arch/arm64/kernel/asm-offsets.c | 1 +
> arch/arm64/kvm/hyp/entry.S | 3 +++
> arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
> arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
> 6 files changed, 46 insertions(+), 36 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index b36aaa1fe332..635137e6ed1c 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
> return (unsigned long *)&vcpu->arch.hcr_el2;
> }
>
> +static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
> +{
> + return !(vcpu->arch.hcr_el2 & HCR_RW);
> +}
Since you now introduce this helper, could you use it to repaint
inject_fault.c which could make use of it too? This could actually be
a separate patch.
> +
> static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
> {
> return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 20fab9194794..c841eeeeb5c5 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
> /* Guest debug state */
> u64 debug_flags;
>
> + /* 1 if the guest VFP state is loaded into the hardware */
> + u8 guest_vfp_loaded;
> +
> /*
> * We maintain more than a single set of debug registers to support
> * debugging the guest from the host and to maintain separate host and
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 612021dce84f..99467327c043 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -133,6 +133,7 @@ int main(void)
> DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
> DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
> DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> + DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
> DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> index a360ac6e89e9..53652287a236 100644
> --- a/arch/arm64/kvm/hyp/entry.S
> +++ b/arch/arm64/kvm/hyp/entry.S
> @@ -184,6 +184,9 @@ alternative_endif
> add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
> bl __fpsimd_restore_state
>
> + mov x0, #1
> + strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
> +
> // Skip restoring fpexc32 for AArch64 guests
> mrs x1, hcr_el2
> tbnz x1, #HCR_RW_SHIFT, 1f
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 11ec1c6f3b84..f5d53ef9ca79 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -24,43 +24,32 @@
> #include <asm/fpsimd.h>
> #include <asm/debug-monitors.h>
>
> -static bool __hyp_text __fpsimd_enabled_nvhe(void)
> -{
> - return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
> -}
> -
> -static bool __hyp_text __fpsimd_enabled_vhe(void)
> -{
> - return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
> -}
> -
> -static hyp_alternate_select(__fpsimd_is_enabled,
> - __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
> - ARM64_HAS_VIRT_HOST_EXTN);
> -
> -bool __hyp_text __fpsimd_enabled(void)
> -{
> - return __fpsimd_is_enabled()();
> -}
> -
> -static void __hyp_text __activate_traps_vhe(void)
> +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> val = read_sysreg(cpacr_el1);
> val |= CPACR_EL1_TTA;
> - val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
> + val &= ~CPACR_EL1_ZEN;
> + if (vcpu->arch.guest_vfp_loaded)
> + val |= CPACR_EL1_FPEN;
> + else
> + val &= ~CPACR_EL1_FPEN;
> write_sysreg(val, cpacr_el1);
>
> write_sysreg(__kvm_hyp_vector, vbar_el1);
> }
>
> -static void __hyp_text __activate_traps_nvhe(void)
> +static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> val = CPTR_EL2_DEFAULT;
> - val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
> + val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> + if (vcpu->arch.guest_vfp_loaded)
> + val &= ~CPTR_EL2_TFP;
> + else
> + val |= CPTR_EL2_TFP;
> write_sysreg(val, cptr_el2);
> }
>
> @@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> */
> val = vcpu->arch.hcr_el2;
>
> - if (!(val & HCR_RW) && system_supports_fpsimd()) {
> + if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> + !vcpu->arch.guest_vfp_loaded) {
> write_sysreg(1 << 30, fpexc32_el2);
> isb();
> }
> @@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> write_sysreg(0, pmselr_el0);
> write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
> write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
> - __activate_traps_arch()();
> + __activate_traps_arch()(vcpu);
> }
>
> static void __hyp_text __deactivate_traps_vhe(void)
> @@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> - bool fp_enabled;
> u64 exit_code;
>
> vcpu = kern_hyp_va(vcpu);
> @@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> /* 0 falls through to be handled out of EL2 */
> }
>
> - fp_enabled = __fpsimd_enabled();
> -
> __sysreg_save_guest_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
> __timer_disable_traps(vcpu);
> @@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>
> __sysreg_restore_host_state(host_ctxt);
>
> - if (fp_enabled) {
> - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> - }
> -
> __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
> /*
> * This must come after restoring the host sysregs, since a non-VHE
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index cbbcd6f410a8..68a7d164e5e1 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -19,6 +19,7 @@
> #include <linux/kvm_host.h>
>
> #include <asm/kvm_asm.h>
> +#include <asm/kvm_emulate.h>
> #include <asm/kvm_hyp.h>
>
> /* Yes, this does nothing, on purpose */
> @@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> __sysreg_restore_common_state(ctxt);
> }
>
> +static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> +{
> + ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> +}
> +
> void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> {
> u64 *spsr, *sysreg;
> @@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
> sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
>
> - if (__fpsimd_enabled())
> - sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> -
> if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
> }
> @@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> */
> void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> {
> + struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
> + struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
> +
> + /* Restore host FP/SIMD state */
> + if (vcpu->arch.guest_vfp_loaded) {
> + if (vcpu_el1_is_32bit(vcpu)) {
> + kvm_call_hyp(__fpsimd32_save_state,
> + kern_hyp_va(guest_ctxt));
> + }
> + __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> + __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> + vcpu->arch.guest_vfp_loaded = 0;
> + }
> }
>
> void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> --
> 2.14.2
>
Otherwise,
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
@ 2017-12-09 17:37 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 17:37 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:06:00 +0000,
Christoffer Dall wrote:
>
> Avoid saving the guest VFP registers and restoring the host VFP
> registers on every exit from the VM. Only when we're about to run
> userspace or other threads in the kernel do we really have to switch the
> state back to the host state.
>
> We still initially configure the VFP registers to trap when entering the
> VM, but the difference is that we now leave the guest state in the
> hardware registers as long as we're running this VCPU, even if we
> occasionally trap to the host, and we only restore the host state when
> we return to user space or when scheduling another thread.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Cosmetic changes
> - Change the flags variable to a u8
> - Expanded the commit message
>
> arch/arm64/include/asm/kvm_emulate.h | 5 ++++
> arch/arm64/include/asm/kvm_host.h | 3 +++
> arch/arm64/kernel/asm-offsets.c | 1 +
> arch/arm64/kvm/hyp/entry.S | 3 +++
> arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
> arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
> 6 files changed, 46 insertions(+), 36 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index b36aaa1fe332..635137e6ed1c 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
> return (unsigned long *)&vcpu->arch.hcr_el2;
> }
>
> +static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
> +{
> + return !(vcpu->arch.hcr_el2 & HCR_RW);
> +}
Since you now introduce this helper, could you use it to repaint
inject_fault.c which could make use of it too? This could actually be
a separate patch.
> +
> static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
> {
> return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 20fab9194794..c841eeeeb5c5 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
> /* Guest debug state */
> u64 debug_flags;
>
> + /* 1 if the guest VFP state is loaded into the hardware */
> + u8 guest_vfp_loaded;
> +
> /*
> * We maintain more than a single set of debug registers to support
> * debugging the guest from the host and to maintain separate host and
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 612021dce84f..99467327c043 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -133,6 +133,7 @@ int main(void)
> DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
> DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
> DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> + DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
> DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> index a360ac6e89e9..53652287a236 100644
> --- a/arch/arm64/kvm/hyp/entry.S
> +++ b/arch/arm64/kvm/hyp/entry.S
> @@ -184,6 +184,9 @@ alternative_endif
> add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
> bl __fpsimd_restore_state
>
> + mov x0, #1
> + strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
> +
> // Skip restoring fpexc32 for AArch64 guests
> mrs x1, hcr_el2
> tbnz x1, #HCR_RW_SHIFT, 1f
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 11ec1c6f3b84..f5d53ef9ca79 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -24,43 +24,32 @@
> #include <asm/fpsimd.h>
> #include <asm/debug-monitors.h>
>
> -static bool __hyp_text __fpsimd_enabled_nvhe(void)
> -{
> - return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
> -}
> -
> -static bool __hyp_text __fpsimd_enabled_vhe(void)
> -{
> - return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
> -}
> -
> -static hyp_alternate_select(__fpsimd_is_enabled,
> - __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
> - ARM64_HAS_VIRT_HOST_EXTN);
> -
> -bool __hyp_text __fpsimd_enabled(void)
> -{
> - return __fpsimd_is_enabled()();
> -}
> -
> -static void __hyp_text __activate_traps_vhe(void)
> +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> val = read_sysreg(cpacr_el1);
> val |= CPACR_EL1_TTA;
> - val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
> + val &= ~CPACR_EL1_ZEN;
> + if (vcpu->arch.guest_vfp_loaded)
> + val |= CPACR_EL1_FPEN;
> + else
> + val &= ~CPACR_EL1_FPEN;
> write_sysreg(val, cpacr_el1);
>
> write_sysreg(__kvm_hyp_vector, vbar_el1);
> }
>
> -static void __hyp_text __activate_traps_nvhe(void)
> +static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> val = CPTR_EL2_DEFAULT;
> - val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
> + val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> + if (vcpu->arch.guest_vfp_loaded)
> + val &= ~CPTR_EL2_TFP;
> + else
> + val |= CPTR_EL2_TFP;
> write_sysreg(val, cptr_el2);
> }
>
> @@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> */
> val = vcpu->arch.hcr_el2;
>
> - if (!(val & HCR_RW) && system_supports_fpsimd()) {
> + if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> + !vcpu->arch.guest_vfp_loaded) {
> write_sysreg(1 << 30, fpexc32_el2);
> isb();
> }
> @@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> write_sysreg(0, pmselr_el0);
> write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
> write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
> - __activate_traps_arch()();
> + __activate_traps_arch()(vcpu);
> }
>
> static void __hyp_text __deactivate_traps_vhe(void)
> @@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> - bool fp_enabled;
> u64 exit_code;
>
> vcpu = kern_hyp_va(vcpu);
> @@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> /* 0 falls through to be handled out of EL2 */
> }
>
> - fp_enabled = __fpsimd_enabled();
> -
> __sysreg_save_guest_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
> __timer_disable_traps(vcpu);
> @@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>
> __sysreg_restore_host_state(host_ctxt);
>
> - if (fp_enabled) {
> - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> - }
> -
> __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
> /*
> * This must come after restoring the host sysregs, since a non-VHE
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index cbbcd6f410a8..68a7d164e5e1 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -19,6 +19,7 @@
> #include <linux/kvm_host.h>
>
> #include <asm/kvm_asm.h>
> +#include <asm/kvm_emulate.h>
> #include <asm/kvm_hyp.h>
>
> /* Yes, this does nothing, on purpose */
> @@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> __sysreg_restore_common_state(ctxt);
> }
>
> +static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> +{
> + ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> +}
> +
> void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> {
> u64 *spsr, *sysreg;
> @@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
> sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
>
> - if (__fpsimd_enabled())
> - sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> -
> if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
> }
> @@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> */
> void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> {
> + struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
> + struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
> +
> + /* Restore host FP/SIMD state */
> + if (vcpu->arch.guest_vfp_loaded) {
> + if (vcpu_el1_is_32bit(vcpu)) {
> + kvm_call_hyp(__fpsimd32_save_state,
> + kern_hyp_va(guest_ctxt));
> + }
> + __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> + __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> + vcpu->arch.guest_vfp_loaded = 0;
> + }
> }
>
> void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> --
> 2.14.2
>
Otherwise,
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 07/36] KVM: arm64: Move debug dirty flag calculation out of world switch
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-09 19:20 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 19:20 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Thu, 07 Dec 2017 17:06:01 +0000,
Christoffer Dall wrote:
>
> There is no need to figure out inside the world-switch if we should
> save/restore the debug registers or not, we can might as well do that in
> the higher level debug setup code, making it easier to optimize down the
> line.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 07/36] KVM: arm64: Move debug dirty flag calculation out of world switch
@ 2017-12-09 19:20 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 19:20 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:06:01 +0000,
Christoffer Dall wrote:
>
> There is no need to figure out inside the world-switch if we should
> save/restore the debug registers or not, we can might as well do that in
> the higher level debug setup code, making it easier to optimize down the
> line.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 08/36] KVM: arm64: Slightly improve debug save/restore functions
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-09 19:37 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 19:37 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Thu, 07 Dec 2017 17:06:02 +0000,
Christoffer Dall wrote:
>
> The debug save/restore functions can be improved by using the has_vhe()
> static key instead of the instruction alternative. Using the static key
> uses the same paradigm as we're going to use elsewhere, it makes the
> code more readable, and it generates slightly better code (no
> stack setups and function calls unless necessary).
>
> We also use a static key on the restore path, because it will be
> marginally faster than loading a value from memory.
>
> Finally, we don't have to conditionally clear the debug dirty flag if
> it's set, we can just clear it.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 08/36] KVM: arm64: Slightly improve debug save/restore functions
@ 2017-12-09 19:37 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-09 19:37 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, 07 Dec 2017 17:06:02 +0000,
Christoffer Dall wrote:
>
> The debug save/restore functions can be improved by using the has_vhe()
> static key instead of the instruction alternative. Using the static key
> uses the same paradigm as we're going to use elsewhere, it makes the
> code more readable, and it generates slightly better code (no
> stack setups and function calls unless necessary).
>
> We also use a static key on the restore path, because it will be
> marginally faster than loading a value from memory.
>
> Finally, we don't have to conditionally clear the debug dirty flag if
> it's set, we can just clear it.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead, it just smell funny.
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
2017-12-09 17:19 ` Marc Zyngier
@ 2017-12-11 9:30 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 9:30 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvm, Ard Biesheuvel, linux-arm-kernel, kvmarm, Shih-Wei Li
On Sat, Dec 09, 2017 at 05:19:41PM +0000, Marc Zyngier wrote:
> On Thu, 07 Dec 2017 17:05:55 +0000,
> Christoffer Dall wrote:
> >
> > We already have the percpu area for the host cpu state, which points to
> > the VCPU, so there's no need to store the VCPU pointer on the stack on
> > every context switch. We can be a little more clever and just use
> > tpidr_el2 for the percpu offset and load the VCPU pointer from the host
> > context.
> >
> > This does require us to calculate the percpu offset without including
> > the offset from the kernel mapping of the percpu array to the linaro
>
> "linaro"?
>
*linear* - spell check must have played a trick on me.
> > mapping of the array (which is what we store in tpidr_el1), because a
> > PC-relative generated address in EL2 is already giving us the hyp alias
> > of the linear mapping of a kernel address. We do this in
> > __cpu_init_hyp_mode() by using kvm_ksym_ref().
> >
> > This change also requires us to have a scratch register, so we take the
> > chance to rearrange some of the el1_sync code to only look at the
> > vttbr_el2 to determine if this is a trap from the guest or an HVC from
> > the host. We do add an extra check to call the panic code if the kernel
> > is configured with debugging enabled and we saw a trap from the host
> > which wasn't an HVC, indicating that we left some EL2 trap configured by
> > mistake.
> >
> > The code that accesses ESR_EL2 was previously using an alternative to
> > use the _EL1 accessor on VHE systems, but this was actually unnecessary
> > as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
> > accessor does the same thing on both systems.
> >
> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Use PC-relative addressing to access per-cpu variables instead of
> > using a load from the literal pool.
> > - Remove stale comments as pointed out by Marc
> > - Reworded the commit message as suggested by Drew
> >
> > arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
> > arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
> > arch/arm64/kernel/asm-offsets.c | 1 +
> > arch/arm64/kvm/hyp/entry.S | 6 +-----
> > arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
> > arch/arm64/kvm/hyp/switch.c | 5 +----
> > arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
> > 7 files changed, 57 insertions(+), 31 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> > index ab4d0a926043..33e0edc6f8be 100644
> > --- a/arch/arm64/include/asm/kvm_asm.h
> > +++ b/arch/arm64/include/asm/kvm_asm.h
> > @@ -33,6 +33,7 @@
> > #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
> > #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
> >
> > +/* Translate a kernel address of @sym into its equivalent linear mapping */
> > #define kvm_ksym_ref(sym) \
> > ({ \
> > void *val = &sym; \
> > @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
> >
> > #endif
> >
> > +#ifdef __ASSEMBLY__
>
> You could turn this and the previous #endif into a simple #else.
>
yup
> > +.macro get_host_ctxt reg, tmp
> > + adr_l \reg, kvm_host_cpu_state
> > + mrs \tmp, tpidr_el2
> > + add \reg, \reg, \tmp
> > +.endm
> > +
> > +.macro get_vcpu vcpu, ctxt
> > + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
> > + kern_hyp_va \vcpu
> > +.endm
> > +
> > +#endif
> > +
> > #endif /* __ARM_KVM_ASM_H__ */
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index 7ee72b402907..af58deb6ec3c 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
> >
> > struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
> >
> > +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
>
> Is there any advantage in having this prototype in kvm_host.h, instead
> of putting it in kvm_hyp.h (which feels more appropriate)?
>
asm-offsets.c pukes all over the place and I can't include asm/kvm_hyp.h
from kvm_host.h, because it depends on kvm_host.h, and I can't include
asm/kvm_hyp.h easily in asm-offsets.c because it pukes again from all
sorts of other missing things.
If you care strongly about this point, I can try to dig deeper in this
or refactor this somehow more substantially. Ideas?
> > +DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
> > +
> > static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> > unsigned long hyp_stack_ptr,
> > unsigned long vector_ptr)
> > {
> > + u64 tpidr_el2;
> > +
> > /*
> > * Call initialization code, and switch to the full blown HYP code.
> > * If the cpucaps haven't been finalized yet, something has gone very
> > @@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> > */
> > BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
> > __kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
> > +
> > + /*
> > + * Calculate the raw per-cpu offset without a translation from the
> > + * kernel's mapping to the linear mapping, and store it in tpidr_el2
> > + * so that we can use adr_l to access per-cpu variables in EL2.
> > + */
> > + tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
> > + - (u64)kvm_ksym_ref(kvm_host_cpu_state);
> > +
> > + kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
> > }
> >
> > static inline void kvm_arch_hardware_unsetup(void) {}
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 71bf088f1e4b..612021dce84f 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -135,6 +135,7 @@ int main(void)
> > DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> > DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> > DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> > + DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> > #endif
> > #ifdef CONFIG_CPU_PM
> > DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
> > diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> > index 9a8ab5dddd9e..a360ac6e89e9 100644
> > --- a/arch/arm64/kvm/hyp/entry.S
> > +++ b/arch/arm64/kvm/hyp/entry.S
> > @@ -62,9 +62,6 @@ ENTRY(__guest_enter)
> > // Store the host regs
> > save_callee_saved_regs x1
> >
> > - // Store host_ctxt and vcpu for use at exit time
> > - stp x1, x0, [sp, #-16]!
> > -
> > add x18, x0, #VCPU_CONTEXT
> >
> > // Restore guest regs x0-x17
> > @@ -118,8 +115,7 @@ ENTRY(__guest_exit)
> > // Store the guest regs x19-x29, lr
> > save_callee_saved_regs x1
> >
> > - // Restore the host_ctxt from the stack
> > - ldr x2, [sp], #16
> > + get_host_ctxt x2, x3
> >
> > // Now restore the host regs
> > restore_callee_saved_regs x2
> > diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> > index e4f37b9dd47c..71b4cc92895e 100644
> > --- a/arch/arm64/kvm/hyp/hyp-entry.S
> > +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> > @@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
> > el1_sync: // Guest trapped into EL2
> > stp x0, x1, [sp, #-16]!
> >
> > -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> > - mrs x1, esr_el2
> > -alternative_else
> > - mrs x1, esr_el1
> > -alternative_endif
> > - lsr x0, x1, #ESR_ELx_EC_SHIFT
> > + mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
> > + cbnz x1, el1_trap // from the guest
> >
> > - cmp x0, #ESR_ELx_EC_HVC64
> > - b.ne el1_trap
> > -
> > - mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
> > - cbnz x1, el1_trap // called HVC
> > +#ifdef CONFIG_DEBUG
> > + mrs x0, esr_el2
> > + lsr x0, x0, #ESR_ELx_EC_SHIFT
> > + cmp x0, #ESR_ELx_EC_HVC64
> > + b.ne __hyp_panic
> > +#endif
> >
> > /* Here, we're pretty sure the host called HVC. */
> > ldp x0, x1, [sp], #16
> > @@ -101,10 +98,15 @@ alternative_endif
> > eret
> >
> > el1_trap:
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > +
> > + mrs x0, esr_el2
> > + lsr x0, x0, #ESR_ELx_EC_SHIFT
> > /*
> > * x0: ESR_EC
> > + * x1: vcpu pointer
> > */
> > - ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
> >
> > /*
> > * We trap the first access to the FP/SIMD to save the host context
> > @@ -122,13 +124,15 @@ alternative_else_nop_endif
> >
> > el1_irq:
> > stp x0, x1, [sp, #-16]!
> > - ldr x1, [sp, #16 + 8]
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > mov x0, #ARM_EXCEPTION_IRQ
> > b __guest_exit
> >
> > el1_error:
> > stp x0, x1, [sp, #-16]!
> > - ldr x1, [sp, #16 + 8]
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > mov x0, #ARM_EXCEPTION_EL1_SERROR
> > b __guest_exit
> >
> > @@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
> > ENDPROC(__hyp_do_panic)
> >
> > ENTRY(__hyp_panic)
> > - /*
> > - * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
> > - * not be accessible by this address from EL2, hyp_panic() converts
> > - * it with kern_hyp_va() before use.
> > - */
> > - ldr x0, =kvm_host_cpu_state
> > - mrs x1, tpidr_el2
> > - add x0, x0, x1
> > + get_host_ctxt x0, x1
> > b hyp_panic
> > ENDPROC(__hyp_panic)
> >
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index f7307b6b42f0..6fcb37e220b5 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
> > __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > -void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> > +void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
> > {
> > struct kvm_vcpu *vcpu = NULL;
> >
> > @@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> > u64 par = read_sysreg(par_el1);
> >
> > if (read_sysreg(vttbr_el2)) {
> > - struct kvm_cpu_context *host_ctxt;
> > -
> > - host_ctxt = kern_hyp_va(__host_ctxt);
> > vcpu = host_ctxt->__hyp_running_vcpu;
> > __timer_disable_traps(vcpu);
> > __deactivate_traps(vcpu);
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index c54cc2afb92b..e19d89cabf2a 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
> > if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> > write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
> > }
> > +
> > +void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> > +{
> > + asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
> > +}
> > --
> > 2.14.2
> >
>
> Other than the couple of nits above:
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks!
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
@ 2017-12-11 9:30 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 9:30 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Dec 09, 2017 at 05:19:41PM +0000, Marc Zyngier wrote:
> On Thu, 07 Dec 2017 17:05:55 +0000,
> Christoffer Dall wrote:
> >
> > We already have the percpu area for the host cpu state, which points to
> > the VCPU, so there's no need to store the VCPU pointer on the stack on
> > every context switch. We can be a little more clever and just use
> > tpidr_el2 for the percpu offset and load the VCPU pointer from the host
> > context.
> >
> > This does require us to calculate the percpu offset without including
> > the offset from the kernel mapping of the percpu array to the linaro
>
> "linaro"?
>
*linear* - spell check must have played a trick on me.
> > mapping of the array (which is what we store in tpidr_el1), because a
> > PC-relative generated address in EL2 is already giving us the hyp alias
> > of the linear mapping of a kernel address. We do this in
> > __cpu_init_hyp_mode() by using kvm_ksym_ref().
> >
> > This change also requires us to have a scratch register, so we take the
> > chance to rearrange some of the el1_sync code to only look at the
> > vttbr_el2 to determine if this is a trap from the guest or an HVC from
> > the host. We do add an extra check to call the panic code if the kernel
> > is configured with debugging enabled and we saw a trap from the host
> > which wasn't an HVC, indicating that we left some EL2 trap configured by
> > mistake.
> >
> > The code that accesses ESR_EL2 was previously using an alternative to
> > use the _EL1 accessor on VHE systems, but this was actually unnecessary
> > as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
> > accessor does the same thing on both systems.
> >
> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Use PC-relative addressing to access per-cpu variables instead of
> > using a load from the literal pool.
> > - Remove stale comments as pointed out by Marc
> > - Reworded the commit message as suggested by Drew
> >
> > arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
> > arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
> > arch/arm64/kernel/asm-offsets.c | 1 +
> > arch/arm64/kvm/hyp/entry.S | 6 +-----
> > arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
> > arch/arm64/kvm/hyp/switch.c | 5 +----
> > arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
> > 7 files changed, 57 insertions(+), 31 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> > index ab4d0a926043..33e0edc6f8be 100644
> > --- a/arch/arm64/include/asm/kvm_asm.h
> > +++ b/arch/arm64/include/asm/kvm_asm.h
> > @@ -33,6 +33,7 @@
> > #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
> > #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
> >
> > +/* Translate a kernel address of @sym into its equivalent linear mapping */
> > #define kvm_ksym_ref(sym) \
> > ({ \
> > void *val = &sym; \
> > @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
> >
> > #endif
> >
> > +#ifdef __ASSEMBLY__
>
> You could turn this and the previous #endif into a simple #else.
>
yup
> > +.macro get_host_ctxt reg, tmp
> > + adr_l \reg, kvm_host_cpu_state
> > + mrs \tmp, tpidr_el2
> > + add \reg, \reg, \tmp
> > +.endm
> > +
> > +.macro get_vcpu vcpu, ctxt
> > + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
> > + kern_hyp_va \vcpu
> > +.endm
> > +
> > +#endif
> > +
> > #endif /* __ARM_KVM_ASM_H__ */
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index 7ee72b402907..af58deb6ec3c 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
> >
> > struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
> >
> > +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
>
> Is there any advantage in having this prototype in kvm_host.h, instead
> of putting it in kvm_hyp.h (which feels more appropriate)?
>
asm-offsets.c pukes all over the place and I can't include asm/kvm_hyp.h
from kvm_host.h, because it depends on kvm_host.h, and I can't include
asm/kvm_hyp.h easily in asm-offsets.c because it pukes again from all
sorts of other missing things.
If you care strongly about this point, I can try to dig deeper in this
or refactor this somehow more substantially. Ideas?
> > +DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
> > +
> > static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> > unsigned long hyp_stack_ptr,
> > unsigned long vector_ptr)
> > {
> > + u64 tpidr_el2;
> > +
> > /*
> > * Call initialization code, and switch to the full blown HYP code.
> > * If the cpucaps haven't been finalized yet, something has gone very
> > @@ -360,6 +365,16 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
> > */
> > BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
> > __kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
> > +
> > + /*
> > + * Calculate the raw per-cpu offset without a translation from the
> > + * kernel's mapping to the linear mapping, and store it in tpidr_el2
> > + * so that we can use adr_l to access per-cpu variables in EL2.
> > + */
> > + tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
> > + - (u64)kvm_ksym_ref(kvm_host_cpu_state);
> > +
> > + kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
> > }
> >
> > static inline void kvm_arch_hardware_unsetup(void) {}
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 71bf088f1e4b..612021dce84f 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -135,6 +135,7 @@ int main(void)
> > DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> > DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> > DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> > + DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> > #endif
> > #ifdef CONFIG_CPU_PM
> > DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
> > diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> > index 9a8ab5dddd9e..a360ac6e89e9 100644
> > --- a/arch/arm64/kvm/hyp/entry.S
> > +++ b/arch/arm64/kvm/hyp/entry.S
> > @@ -62,9 +62,6 @@ ENTRY(__guest_enter)
> > // Store the host regs
> > save_callee_saved_regs x1
> >
> > - // Store host_ctxt and vcpu for use at exit time
> > - stp x1, x0, [sp, #-16]!
> > -
> > add x18, x0, #VCPU_CONTEXT
> >
> > // Restore guest regs x0-x17
> > @@ -118,8 +115,7 @@ ENTRY(__guest_exit)
> > // Store the guest regs x19-x29, lr
> > save_callee_saved_regs x1
> >
> > - // Restore the host_ctxt from the stack
> > - ldr x2, [sp], #16
> > + get_host_ctxt x2, x3
> >
> > // Now restore the host regs
> > restore_callee_saved_regs x2
> > diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> > index e4f37b9dd47c..71b4cc92895e 100644
> > --- a/arch/arm64/kvm/hyp/hyp-entry.S
> > +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> > @@ -56,18 +56,15 @@ ENDPROC(__vhe_hyp_call)
> > el1_sync: // Guest trapped into EL2
> > stp x0, x1, [sp, #-16]!
> >
> > -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> > - mrs x1, esr_el2
> > -alternative_else
> > - mrs x1, esr_el1
> > -alternative_endif
> > - lsr x0, x1, #ESR_ELx_EC_SHIFT
> > + mrs x1, vttbr_el2 // If vttbr is valid, this is a trap
> > + cbnz x1, el1_trap // from the guest
> >
> > - cmp x0, #ESR_ELx_EC_HVC64
> > - b.ne el1_trap
> > -
> > - mrs x1, vttbr_el2 // If vttbr is valid, the 64bit guest
> > - cbnz x1, el1_trap // called HVC
> > +#ifdef CONFIG_DEBUG
> > + mrs x0, esr_el2
> > + lsr x0, x0, #ESR_ELx_EC_SHIFT
> > + cmp x0, #ESR_ELx_EC_HVC64
> > + b.ne __hyp_panic
> > +#endif
> >
> > /* Here, we're pretty sure the host called HVC. */
> > ldp x0, x1, [sp], #16
> > @@ -101,10 +98,15 @@ alternative_endif
> > eret
> >
> > el1_trap:
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > +
> > + mrs x0, esr_el2
> > + lsr x0, x0, #ESR_ELx_EC_SHIFT
> > /*
> > * x0: ESR_EC
> > + * x1: vcpu pointer
> > */
> > - ldr x1, [sp, #16 + 8] // vcpu stored by __guest_enter
> >
> > /*
> > * We trap the first access to the FP/SIMD to save the host context
> > @@ -122,13 +124,15 @@ alternative_else_nop_endif
> >
> > el1_irq:
> > stp x0, x1, [sp, #-16]!
> > - ldr x1, [sp, #16 + 8]
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > mov x0, #ARM_EXCEPTION_IRQ
> > b __guest_exit
> >
> > el1_error:
> > stp x0, x1, [sp, #-16]!
> > - ldr x1, [sp, #16 + 8]
> > + get_host_ctxt x0, x1
> > + get_vcpu x1, x0
> > mov x0, #ARM_EXCEPTION_EL1_SERROR
> > b __guest_exit
> >
> > @@ -164,14 +168,7 @@ ENTRY(__hyp_do_panic)
> > ENDPROC(__hyp_do_panic)
> >
> > ENTRY(__hyp_panic)
> > - /*
> > - * '=kvm_host_cpu_state' is a host VA from the constant pool, it may
> > - * not be accessible by this address from EL2, hyp_panic() converts
> > - * it with kern_hyp_va() before use.
> > - */
> > - ldr x0, =kvm_host_cpu_state
> > - mrs x1, tpidr_el2
> > - add x0, x0, x1
> > + get_host_ctxt x0, x1
> > b hyp_panic
> > ENDPROC(__hyp_panic)
> >
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index f7307b6b42f0..6fcb37e220b5 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -449,7 +449,7 @@ static hyp_alternate_select(__hyp_call_panic,
> > __hyp_call_panic_nvhe, __hyp_call_panic_vhe,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > -void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> > +void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
> > {
> > struct kvm_vcpu *vcpu = NULL;
> >
> > @@ -458,9 +458,6 @@ void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *__host_ctxt)
> > u64 par = read_sysreg(par_el1);
> >
> > if (read_sysreg(vttbr_el2)) {
> > - struct kvm_cpu_context *host_ctxt;
> > -
> > - host_ctxt = kern_hyp_va(__host_ctxt);
> > vcpu = host_ctxt->__hyp_running_vcpu;
> > __timer_disable_traps(vcpu);
> > __deactivate_traps(vcpu);
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index c54cc2afb92b..e19d89cabf2a 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
> > if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> > write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
> > }
> > +
> > +void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> > +{
> > + asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
> > +}
> > --
> > 2.14.2
> >
>
> Other than the couple of nits above:
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks!
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
2017-12-09 17:37 ` Marc Zyngier
@ 2017-12-11 9:31 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 9:31 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Sat, Dec 09, 2017 at 05:37:53PM +0000, Marc Zyngier wrote:
> On Thu, 07 Dec 2017 17:06:00 +0000,
> Christoffer Dall wrote:
> >
> > Avoid saving the guest VFP registers and restoring the host VFP
> > registers on every exit from the VM. Only when we're about to run
> > userspace or other threads in the kernel do we really have to switch the
> > state back to the host state.
> >
> > We still initially configure the VFP registers to trap when entering the
> > VM, but the difference is that we now leave the guest state in the
> > hardware registers as long as we're running this VCPU, even if we
> > occasionally trap to the host, and we only restore the host state when
> > we return to user space or when scheduling another thread.
> >
> > Reviewed-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Cosmetic changes
> > - Change the flags variable to a u8
> > - Expanded the commit message
> >
> > arch/arm64/include/asm/kvm_emulate.h | 5 ++++
> > arch/arm64/include/asm/kvm_host.h | 3 +++
> > arch/arm64/kernel/asm-offsets.c | 1 +
> > arch/arm64/kvm/hyp/entry.S | 3 +++
> > arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
> > arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
> > 6 files changed, 46 insertions(+), 36 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> > index b36aaa1fe332..635137e6ed1c 100644
> > --- a/arch/arm64/include/asm/kvm_emulate.h
> > +++ b/arch/arm64/include/asm/kvm_emulate.h
> > @@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
> > return (unsigned long *)&vcpu->arch.hcr_el2;
> > }
> >
> > +static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
> > +{
> > + return !(vcpu->arch.hcr_el2 & HCR_RW);
> > +}
>
> Since you now introduce this helper, could you use it to repaint
> inject_fault.c which could make use of it too? This could actually be
> a separate patch.
>
Yes, I'll do that first, and the have this patch follow.
> > +
> > static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
> > {
> > return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index 20fab9194794..c841eeeeb5c5 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
> > /* Guest debug state */
> > u64 debug_flags;
> >
> > + /* 1 if the guest VFP state is loaded into the hardware */
> > + u8 guest_vfp_loaded;
> > +
> > /*
> > * We maintain more than a single set of debug registers to support
> > * debugging the guest from the host and to maintain separate host and
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 612021dce84f..99467327c043 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -133,6 +133,7 @@ int main(void)
> > DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
> > DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
> > DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> > + DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
> > DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> > DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> > DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> > diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> > index a360ac6e89e9..53652287a236 100644
> > --- a/arch/arm64/kvm/hyp/entry.S
> > +++ b/arch/arm64/kvm/hyp/entry.S
> > @@ -184,6 +184,9 @@ alternative_endif
> > add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
> > bl __fpsimd_restore_state
> >
> > + mov x0, #1
> > + strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
> > +
> > // Skip restoring fpexc32 for AArch64 guests
> > mrs x1, hcr_el2
> > tbnz x1, #HCR_RW_SHIFT, 1f
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index 11ec1c6f3b84..f5d53ef9ca79 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -24,43 +24,32 @@
> > #include <asm/fpsimd.h>
> > #include <asm/debug-monitors.h>
> >
> > -static bool __hyp_text __fpsimd_enabled_nvhe(void)
> > -{
> > - return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
> > -}
> > -
> > -static bool __hyp_text __fpsimd_enabled_vhe(void)
> > -{
> > - return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
> > -}
> > -
> > -static hyp_alternate_select(__fpsimd_is_enabled,
> > - __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
> > - ARM64_HAS_VIRT_HOST_EXTN);
> > -
> > -bool __hyp_text __fpsimd_enabled(void)
> > -{
> > - return __fpsimd_is_enabled()();
> > -}
> > -
> > -static void __hyp_text __activate_traps_vhe(void)
> > +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> > {
> > u64 val;
> >
> > val = read_sysreg(cpacr_el1);
> > val |= CPACR_EL1_TTA;
> > - val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
> > + val &= ~CPACR_EL1_ZEN;
> > + if (vcpu->arch.guest_vfp_loaded)
> > + val |= CPACR_EL1_FPEN;
> > + else
> > + val &= ~CPACR_EL1_FPEN;
> > write_sysreg(val, cpacr_el1);
> >
> > write_sysreg(__kvm_hyp_vector, vbar_el1);
> > }
> >
> > -static void __hyp_text __activate_traps_nvhe(void)
> > +static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> > {
> > u64 val;
> >
> > val = CPTR_EL2_DEFAULT;
> > - val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
> > + val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> > + if (vcpu->arch.guest_vfp_loaded)
> > + val &= ~CPTR_EL2_TFP;
> > + else
> > + val |= CPTR_EL2_TFP;
> > write_sysreg(val, cptr_el2);
> > }
> >
> > @@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> > */
> > val = vcpu->arch.hcr_el2;
> >
> > - if (!(val & HCR_RW) && system_supports_fpsimd()) {
> > + if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> > + !vcpu->arch.guest_vfp_loaded) {
> > write_sysreg(1 << 30, fpexc32_el2);
> > isb();
> > }
> > @@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> > write_sysreg(0, pmselr_el0);
> > write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
> > write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
> > - __activate_traps_arch()();
> > + __activate_traps_arch()(vcpu);
> > }
> >
> > static void __hyp_text __deactivate_traps_vhe(void)
> > @@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > - bool fp_enabled;
> > u64 exit_code;
> >
> > vcpu = kern_hyp_va(vcpu);
> > @@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > /* 0 falls through to be handled out of EL2 */
> > }
> >
> > - fp_enabled = __fpsimd_enabled();
> > -
> > __sysreg_save_guest_state(guest_ctxt);
> > __sysreg32_save_state(vcpu);
> > __timer_disable_traps(vcpu);
> > @@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> >
> > __sysreg_restore_host_state(host_ctxt);
> >
> > - if (fp_enabled) {
> > - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> > - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> > - }
> > -
> > __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
> > /*
> > * This must come after restoring the host sysregs, since a non-VHE
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index cbbcd6f410a8..68a7d164e5e1 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -19,6 +19,7 @@
> > #include <linux/kvm_host.h>
> >
> > #include <asm/kvm_asm.h>
> > +#include <asm/kvm_emulate.h>
> > #include <asm/kvm_hyp.h>
> >
> > /* Yes, this does nothing, on purpose */
> > @@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_common_state(ctxt);
> > }
> >
> > +static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> > +{
> > + ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> > +}
> > +
> > void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> > {
> > u64 *spsr, *sysreg;
> > @@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> > sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
> > sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
> >
> > - if (__fpsimd_enabled())
> > - sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> > -
> > if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> > sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
> > }
> > @@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> > */
> > void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> > {
> > + struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
> > + struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
> > +
> > + /* Restore host FP/SIMD state */
> > + if (vcpu->arch.guest_vfp_loaded) {
> > + if (vcpu_el1_is_32bit(vcpu)) {
> > + kvm_call_hyp(__fpsimd32_save_state,
> > + kern_hyp_va(guest_ctxt));
> > + }
> > + __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> > + __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> > + vcpu->arch.guest_vfp_loaded = 0;
> > + }
> > }
> >
> > void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> > --
> > 2.14.2
> >
>
> Otherwise,
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks!
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put
@ 2017-12-11 9:31 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 9:31 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Dec 09, 2017 at 05:37:53PM +0000, Marc Zyngier wrote:
> On Thu, 07 Dec 2017 17:06:00 +0000,
> Christoffer Dall wrote:
> >
> > Avoid saving the guest VFP registers and restoring the host VFP
> > registers on every exit from the VM. Only when we're about to run
> > userspace or other threads in the kernel do we really have to switch the
> > state back to the host state.
> >
> > We still initially configure the VFP registers to trap when entering the
> > VM, but the difference is that we now leave the guest state in the
> > hardware registers as long as we're running this VCPU, even if we
> > occasionally trap to the host, and we only restore the host state when
> > we return to user space or when scheduling another thread.
> >
> > Reviewed-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Cosmetic changes
> > - Change the flags variable to a u8
> > - Expanded the commit message
> >
> > arch/arm64/include/asm/kvm_emulate.h | 5 ++++
> > arch/arm64/include/asm/kvm_host.h | 3 +++
> > arch/arm64/kernel/asm-offsets.c | 1 +
> > arch/arm64/kvm/hyp/entry.S | 3 +++
> > arch/arm64/kvm/hyp/switch.c | 48 +++++++++++-------------------------
> > arch/arm64/kvm/hyp/sysreg-sr.c | 22 ++++++++++++++---
> > 6 files changed, 46 insertions(+), 36 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> > index b36aaa1fe332..635137e6ed1c 100644
> > --- a/arch/arm64/include/asm/kvm_emulate.h
> > +++ b/arch/arm64/include/asm/kvm_emulate.h
> > @@ -67,6 +67,11 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
> > return (unsigned long *)&vcpu->arch.hcr_el2;
> > }
> >
> > +static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
> > +{
> > + return !(vcpu->arch.hcr_el2 & HCR_RW);
> > +}
>
> Since you now introduce this helper, could you use it to repaint
> inject_fault.c which could make use of it too? This could actually be
> a separate patch.
>
Yes, I'll do that first, and the have this patch follow.
> > +
> > static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
> > {
> > return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc;
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index 20fab9194794..c841eeeeb5c5 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -211,6 +211,9 @@ struct kvm_vcpu_arch {
> > /* Guest debug state */
> > u64 debug_flags;
> >
> > + /* 1 if the guest VFP state is loaded into the hardware */
> > + u8 guest_vfp_loaded;
> > +
> > /*
> > * We maintain more than a single set of debug registers to support
> > * debugging the guest from the host and to maintain separate host and
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 612021dce84f..99467327c043 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -133,6 +133,7 @@ int main(void)
> > DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
> > DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
> > DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
> > + DEFINE(VCPU_GUEST_VFP_LOADED, offsetof(struct kvm_vcpu, arch.guest_vfp_loaded));
> > DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
> > DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
> > DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
> > diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
> > index a360ac6e89e9..53652287a236 100644
> > --- a/arch/arm64/kvm/hyp/entry.S
> > +++ b/arch/arm64/kvm/hyp/entry.S
> > @@ -184,6 +184,9 @@ alternative_endif
> > add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS)
> > bl __fpsimd_restore_state
> >
> > + mov x0, #1
> > + strb w0, [x3, #VCPU_GUEST_VFP_LOADED]
> > +
> > // Skip restoring fpexc32 for AArch64 guests
> > mrs x1, hcr_el2
> > tbnz x1, #HCR_RW_SHIFT, 1f
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index 11ec1c6f3b84..f5d53ef9ca79 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -24,43 +24,32 @@
> > #include <asm/fpsimd.h>
> > #include <asm/debug-monitors.h>
> >
> > -static bool __hyp_text __fpsimd_enabled_nvhe(void)
> > -{
> > - return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP);
> > -}
> > -
> > -static bool __hyp_text __fpsimd_enabled_vhe(void)
> > -{
> > - return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN);
> > -}
> > -
> > -static hyp_alternate_select(__fpsimd_is_enabled,
> > - __fpsimd_enabled_nvhe, __fpsimd_enabled_vhe,
> > - ARM64_HAS_VIRT_HOST_EXTN);
> > -
> > -bool __hyp_text __fpsimd_enabled(void)
> > -{
> > - return __fpsimd_is_enabled()();
> > -}
> > -
> > -static void __hyp_text __activate_traps_vhe(void)
> > +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> > {
> > u64 val;
> >
> > val = read_sysreg(cpacr_el1);
> > val |= CPACR_EL1_TTA;
> > - val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
> > + val &= ~CPACR_EL1_ZEN;
> > + if (vcpu->arch.guest_vfp_loaded)
> > + val |= CPACR_EL1_FPEN;
> > + else
> > + val &= ~CPACR_EL1_FPEN;
> > write_sysreg(val, cpacr_el1);
> >
> > write_sysreg(__kvm_hyp_vector, vbar_el1);
> > }
> >
> > -static void __hyp_text __activate_traps_nvhe(void)
> > +static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> > {
> > u64 val;
> >
> > val = CPTR_EL2_DEFAULT;
> > - val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
> > + val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> > + if (vcpu->arch.guest_vfp_loaded)
> > + val &= ~CPTR_EL2_TFP;
> > + else
> > + val |= CPTR_EL2_TFP;
> > write_sysreg(val, cptr_el2);
> > }
> >
> > @@ -83,7 +72,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> > */
> > val = vcpu->arch.hcr_el2;
> >
> > - if (!(val & HCR_RW) && system_supports_fpsimd()) {
> > + if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> > + !vcpu->arch.guest_vfp_loaded) {
> > write_sysreg(1 << 30, fpexc32_el2);
> > isb();
> > }
> > @@ -100,7 +90,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
> > write_sysreg(0, pmselr_el0);
> > write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
> > write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
> > - __activate_traps_arch()();
> > + __activate_traps_arch()(vcpu);
> > }
> >
> > static void __hyp_text __deactivate_traps_vhe(void)
> > @@ -288,7 +278,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > - bool fp_enabled;
> > u64 exit_code;
> >
> > vcpu = kern_hyp_va(vcpu);
> > @@ -380,8 +369,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > /* 0 falls through to be handled out of EL2 */
> > }
> >
> > - fp_enabled = __fpsimd_enabled();
> > -
> > __sysreg_save_guest_state(guest_ctxt);
> > __sysreg32_save_state(vcpu);
> > __timer_disable_traps(vcpu);
> > @@ -392,11 +379,6 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> >
> > __sysreg_restore_host_state(host_ctxt);
> >
> > - if (fp_enabled) {
> > - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> > - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> > - }
> > -
> > __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
> > /*
> > * This must come after restoring the host sysregs, since a non-VHE
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index cbbcd6f410a8..68a7d164e5e1 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -19,6 +19,7 @@
> > #include <linux/kvm_host.h>
> >
> > #include <asm/kvm_asm.h>
> > +#include <asm/kvm_emulate.h>
> > #include <asm/kvm_hyp.h>
> >
> > /* Yes, this does nothing, on purpose */
> > @@ -137,6 +138,11 @@ void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_common_state(ctxt);
> > }
> >
> > +static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> > +{
> > + ctxt->sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> > +}
> > +
> > void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> > {
> > u64 *spsr, *sysreg;
> > @@ -155,9 +161,6 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
> > sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
> > sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
> >
> > - if (__fpsimd_enabled())
> > - sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2);
> > -
> > if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
> > sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
> > }
> > @@ -212,6 +215,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> > */
> > void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> > {
> > + struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
> > + struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
> > +
> > + /* Restore host FP/SIMD state */
> > + if (vcpu->arch.guest_vfp_loaded) {
> > + if (vcpu_el1_is_32bit(vcpu)) {
> > + kvm_call_hyp(__fpsimd32_save_state,
> > + kern_hyp_va(guest_ctxt));
> > + }
> > + __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
> > + __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs);
> > + vcpu->arch.guest_vfp_loaded = 0;
> > + }
> > }
> >
> > void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
> > --
> > 2.14.2
> >
>
> Otherwise,
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks!
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
2017-12-11 9:30 ` Christoffer Dall
@ 2017-12-11 9:35 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:35 UTC (permalink / raw)
To: Christoffer Dall
Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones, Ard Biesheuvel
On 11/12/17 09:30, Christoffer Dall wrote:
> On Sat, Dec 09, 2017 at 05:19:41PM +0000, Marc Zyngier wrote:
>> On Thu, 07 Dec 2017 17:05:55 +0000,
>> Christoffer Dall wrote:
>>>
>>> We already have the percpu area for the host cpu state, which points to
>>> the VCPU, so there's no need to store the VCPU pointer on the stack on
>>> every context switch. We can be a little more clever and just use
>>> tpidr_el2 for the percpu offset and load the VCPU pointer from the host
>>> context.
>>>
>>> This does require us to calculate the percpu offset without including
>>> the offset from the kernel mapping of the percpu array to the linaro
>>
>> "linaro"?
>>
>
> *linear* - spell check must have played a trick on me.
>
>>> mapping of the array (which is what we store in tpidr_el1), because a
>>> PC-relative generated address in EL2 is already giving us the hyp alias
>>> of the linear mapping of a kernel address. We do this in
>>> __cpu_init_hyp_mode() by using kvm_ksym_ref().
>>>
>>> This change also requires us to have a scratch register, so we take the
>>> chance to rearrange some of the el1_sync code to only look at the
>>> vttbr_el2 to determine if this is a trap from the guest or an HVC from
>>> the host. We do add an extra check to call the panic code if the kernel
>>> is configured with debugging enabled and we saw a trap from the host
>>> which wasn't an HVC, indicating that we left some EL2 trap configured by
>>> mistake.
>>>
>>> The code that accesses ESR_EL2 was previously using an alternative to
>>> use the _EL1 accessor on VHE systems, but this was actually unnecessary
>>> as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
>>> accessor does the same thing on both systems.
>>>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Use PC-relative addressing to access per-cpu variables instead of
>>> using a load from the literal pool.
>>> - Remove stale comments as pointed out by Marc
>>> - Reworded the commit message as suggested by Drew
>>>
>>> arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
>>> arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
>>> arch/arm64/kernel/asm-offsets.c | 1 +
>>> arch/arm64/kvm/hyp/entry.S | 6 +-----
>>> arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
>>> arch/arm64/kvm/hyp/switch.c | 5 +----
>>> arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
>>> 7 files changed, 57 insertions(+), 31 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
>>> index ab4d0a926043..33e0edc6f8be 100644
>>> --- a/arch/arm64/include/asm/kvm_asm.h
>>> +++ b/arch/arm64/include/asm/kvm_asm.h
>>> @@ -33,6 +33,7 @@
>>> #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
>>> #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
>>>
>>> +/* Translate a kernel address of @sym into its equivalent linear mapping */
>>> #define kvm_ksym_ref(sym) \
>>> ({ \
>>> void *val = &sym; \
>>> @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
>>>
>>> #endif
>>>
>>> +#ifdef __ASSEMBLY__
>>
>> You could turn this and the previous #endif into a simple #else.
>>
>
> yup
>
>>> +.macro get_host_ctxt reg, tmp
>>> + adr_l \reg, kvm_host_cpu_state
>>> + mrs \tmp, tpidr_el2
>>> + add \reg, \reg, \tmp
>>> +.endm
>>> +
>>> +.macro get_vcpu vcpu, ctxt
>>> + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
>>> + kern_hyp_va \vcpu
>>> +.endm
>>> +
>>> +#endif
>>> +
>>> #endif /* __ARM_KVM_ASM_H__ */
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index 7ee72b402907..af58deb6ec3c 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
>>>
>>> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>>>
>>> +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
>>
>> Is there any advantage in having this prototype in kvm_host.h, instead
>> of putting it in kvm_hyp.h (which feels more appropriate)?
>>
>
> asm-offsets.c pukes all over the place and I can't include asm/kvm_hyp.h
> from kvm_host.h, because it depends on kvm_host.h, and I can't include
> asm/kvm_hyp.h easily in asm-offsets.c because it pukes again from all
> sorts of other missing things.
>
> If you care strongly about this point, I can try to dig deeper in this
> or refactor this somehow more substantially. Ideas?
Yeah. I've fixed a couple of the asm-offsets crap (see my randomization
series), but it is really not worth adding a dependency on that. Forget
about it for now, we'll address it at a later time, if ever.
Thanks,
M./
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack
@ 2017-12-11 9:35 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:35 UTC (permalink / raw)
To: linux-arm-kernel
On 11/12/17 09:30, Christoffer Dall wrote:
> On Sat, Dec 09, 2017 at 05:19:41PM +0000, Marc Zyngier wrote:
>> On Thu, 07 Dec 2017 17:05:55 +0000,
>> Christoffer Dall wrote:
>>>
>>> We already have the percpu area for the host cpu state, which points to
>>> the VCPU, so there's no need to store the VCPU pointer on the stack on
>>> every context switch. We can be a little more clever and just use
>>> tpidr_el2 for the percpu offset and load the VCPU pointer from the host
>>> context.
>>>
>>> This does require us to calculate the percpu offset without including
>>> the offset from the kernel mapping of the percpu array to the linaro
>>
>> "linaro"?
>>
>
> *linear* - spell check must have played a trick on me.
>
>>> mapping of the array (which is what we store in tpidr_el1), because a
>>> PC-relative generated address in EL2 is already giving us the hyp alias
>>> of the linear mapping of a kernel address. We do this in
>>> __cpu_init_hyp_mode() by using kvm_ksym_ref().
>>>
>>> This change also requires us to have a scratch register, so we take the
>>> chance to rearrange some of the el1_sync code to only look at the
>>> vttbr_el2 to determine if this is a trap from the guest or an HVC from
>>> the host. We do add an extra check to call the panic code if the kernel
>>> is configured with debugging enabled and we saw a trap from the host
>>> which wasn't an HVC, indicating that we left some EL2 trap configured by
>>> mistake.
>>>
>>> The code that accesses ESR_EL2 was previously using an alternative to
>>> use the _EL1 accessor on VHE systems, but this was actually unnecessary
>>> as the _EL1 accessor aliases the ESR_EL2 register on VHE, and the _EL2
>>> accessor does the same thing on both systems.
>>>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Use PC-relative addressing to access per-cpu variables instead of
>>> using a load from the literal pool.
>>> - Remove stale comments as pointed out by Marc
>>> - Reworded the commit message as suggested by Drew
>>>
>>> arch/arm64/include/asm/kvm_asm.h | 15 ++++++++++++++
>>> arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++
>>> arch/arm64/kernel/asm-offsets.c | 1 +
>>> arch/arm64/kvm/hyp/entry.S | 6 +-----
>>> arch/arm64/kvm/hyp/hyp-entry.S | 41 ++++++++++++++++++---------------------
>>> arch/arm64/kvm/hyp/switch.c | 5 +----
>>> arch/arm64/kvm/hyp/sysreg-sr.c | 5 +++++
>>> 7 files changed, 57 insertions(+), 31 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
>>> index ab4d0a926043..33e0edc6f8be 100644
>>> --- a/arch/arm64/include/asm/kvm_asm.h
>>> +++ b/arch/arm64/include/asm/kvm_asm.h
>>> @@ -33,6 +33,7 @@
>>> #define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
>>> #define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
>>>
>>> +/* Translate a kernel address of @sym into its equivalent linear mapping */
>>> #define kvm_ksym_ref(sym) \
>>> ({ \
>>> void *val = &sym; \
>>> @@ -70,4 +71,18 @@ extern u32 __init_stage2_translation(void);
>>>
>>> #endif
>>>
>>> +#ifdef __ASSEMBLY__
>>
>> You could turn this and the previous #endif into a simple #else.
>>
>
> yup
>
>>> +.macro get_host_ctxt reg, tmp
>>> + adr_l \reg, kvm_host_cpu_state
>>> + mrs \tmp, tpidr_el2
>>> + add \reg, \reg, \tmp
>>> +.endm
>>> +
>>> +.macro get_vcpu vcpu, ctxt
>>> + ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
>>> + kern_hyp_va \vcpu
>>> +.endm
>>> +
>>> +#endif
>>> +
>>> #endif /* __ARM_KVM_ASM_H__ */
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index 7ee72b402907..af58deb6ec3c 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -348,10 +348,15 @@ int kvm_perf_teardown(void);
>>>
>>> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>>>
>>> +extern void __kvm_set_tpidr_el2(u64 tpidr_el2);
>>
>> Is there any advantage in having this prototype in kvm_host.h, instead
>> of putting it in kvm_hyp.h (which feels more appropriate)?
>>
>
> asm-offsets.c pukes all over the place and I can't include asm/kvm_hyp.h
> from kvm_host.h, because it depends on kvm_host.h, and I can't include
> asm/kvm_hyp.h easily in asm-offsets.c because it pukes again from all
> sorts of other missing things.
>
> If you care strongly about this point, I can try to dig deeper in this
> or refactor this somehow more substantially. Ideas?
Yeah. I've fixed a couple of the asm-offsets crap (see my randomization
series), but it is really not worth adding a dependency on that. Forget
about it for now, we'll address it at a later time, if ever.
Thanks,
M./
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 09/36] KVM: arm64: Improve debug register save/restore flow
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 9:40 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:40 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> Instead of having multiple calls from the world switch path to the debug
> logic, each figuring out if the dirty bit is set and if we should
> save/restore the debug registers, let's just provide two hooks to the
> debug save/restore functionality, one for switching to the guest
> context, and one for switching to the host context, and we get the
> benefit of only having to evaluate the dirty flag once on each path,
> plus we give the compiler some more room to inline some of this
> functionality.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 09/36] KVM: arm64: Improve debug register save/restore flow
@ 2017-12-11 9:40 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:40 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> Instead of having multiple calls from the world switch path to the debug
> logic, each figuring out if the dirty bit is set and if we should
> save/restore the debug registers, let's just provide two hooks to the
> debug save/restore functionality, one for switching to the guest
> context, and one for switching to the host context, and we get the
> benefit of only having to evaluate the dirty flag once on each path,
> plus we give the compiler some more room to inline some of this
> functionality.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 10/36] KVM: arm64: Factor out fault info population and gic workarounds
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 9:45 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:45 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> The current world-switch function has functionality to detect a number
> of cases where we need to fixup some part of the exit condition and
> possibly run the guest again, before having restored the host state.
>
> This includes populating missing fault info, emulating GICv2 CPU
> interface accesses when mapped at unaligned addresses, and emulating
> the GICv3 CPU interface on systems that need it.
>
> As we are about to have an alternative switch function for VHE systems,
> but VHE systems still need the same early fixup logic, factor out this
> logic into a separate function that can be shared by both switch
> functions.
>
> No functional change.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 10/36] KVM: arm64: Factor out fault info population and gic workarounds
@ 2017-12-11 9:45 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:45 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> The current world-switch function has functionality to detect a number
> of cases where we need to fixup some part of the exit condition and
> possibly run the guest again, before having restored the host state.
>
> This includes populating missing fault info, emulating GICv2 CPU
> interface accesses when mapped at unaligned addresses, and emulating
> the GICv3 CPU interface on systems that need it.
>
> As we are about to have an alternative switch function for VHE systems,
> but VHE systems still need the same early fixup logic, factor out this
> logic into a separate function that can be shared by both switch
> functions.
>
> No functional change.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 9:53 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:53 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> So far this is just a copy of the legacy non-VHE switch function, where
> we only change the existing calls to has_vhe() in both the original and
> new functions.
I'm not sure I correctly parse the above. Do you want to say that you
use has_vhe() to select the right world switch back-end? Or did you mean
to drop some of the has_vhe() calls in the non-VHE case?
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
> __kvm_vcpu_run_nvhe
> - Removed stray whitespace line
>
> arch/arm/include/asm/kvm_asm.h | 5 +++-
> arch/arm/kvm/hyp/switch.c | 2 +-
> arch/arm64/include/asm/kvm_asm.h | 4 ++-
> arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
> virt/kvm/arm/arm.c | 5 +++-
> 5 files changed, 69 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
> index 36dd2962a42d..4ac717276543 100644
> --- a/arch/arm/include/asm/kvm_asm.h
> +++ b/arch/arm/include/asm/kvm_asm.h
> @@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
>
> extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
>
> -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> +/* no VHE on 32-bit :( */
> +static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
> +
> +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
>
> extern void __init_stage2_translation(void);
>
> diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
> index c3b9799e2e13..7b2bd25e3b10 100644
> --- a/arch/arm/kvm/hyp/switch.c
> +++ b/arch/arm/kvm/hyp/switch.c
> @@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
> return true;
> }
>
> -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index 33e0edc6f8be..adaf1db12271 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
>
> extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
>
> -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> +extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
> +
> +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
>
> extern u64 __vgic_v3_get_ich_vtr_el2(void);
> extern u64 __vgic_v3_read_vmcr(void);
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 845e3bece399..8f32f8dcab65 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
> return false;
> }
>
> -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> +/* Switch to the guest for VHE systems running in EL2 */
> +int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> +{
> + struct kvm_cpu_context *host_ctxt;
> + struct kvm_cpu_context *guest_ctxt;
> + u64 exit_code;
> +
> + vcpu = kern_hyp_va(vcpu);
> +
> + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
> + host_ctxt->__hyp_running_vcpu = vcpu;
> + guest_ctxt = &vcpu->arch.ctxt;
> +
> + __sysreg_save_host_state(host_ctxt);
> +
> + __activate_traps(vcpu);
> + __activate_vm(vcpu);
> +
> + __vgic_restore_state(vcpu);
> + __timer_enable_traps(vcpu);
> +
> + /*
> + * We must restore the 32-bit state before the sysregs, thanks
> + * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
> + */
> + __sysreg32_restore_state(vcpu);
> + __sysreg_restore_guest_state(guest_ctxt);
> + __debug_switch_to_guest(vcpu);
> +
> + do {
> + /* Jump in the fire! */
> + exit_code = __guest_enter(vcpu, host_ctxt);
> +
> + /* And we're baaack! */
> + } while (fixup_guest_exit(vcpu, &exit_code));
> +
> + __sysreg_save_guest_state(guest_ctxt);
> + __sysreg32_save_state(vcpu);
> + __timer_disable_traps(vcpu);
> + __vgic_save_state(vcpu);
> +
> + __deactivate_traps(vcpu);
> + __deactivate_vm(vcpu);
> +
> + __sysreg_restore_host_state(host_ctxt);
> +
> + /*
> + * This must come after restoring the host sysregs, since a non-VHE
> + * system may enable SPE here and make use of the TTBRs.
> + */
> + __debug_switch_to_host(vcpu);
> +
> + return exit_code;
> +}
> +
> +/* Switch to the guest for legacy non-VHE systems */
> +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 3e10343374a1..104ee524c75a 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
> trace_kvm_entry(*vcpu_pc(vcpu));
> guest_enter_irqoff();
>
> - ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
> + if (has_vhe())
> + ret = kvm_vcpu_run_vhe(vcpu);
> + else
> + ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
>
> vcpu->mode = OUTSIDE_GUEST_MODE;
> vcpu->stat.exits++;
>
Otherwise looks good to me.
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
@ 2017-12-11 9:53 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:53 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> So far this is just a copy of the legacy non-VHE switch function, where
> we only change the existing calls to has_vhe() in both the original and
> new functions.
I'm not sure I correctly parse the above. Do you want to say that you
use has_vhe() to select the right world switch back-end? Or did you mean
to drop some of the has_vhe() calls in the non-VHE case?
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
> __kvm_vcpu_run_nvhe
> - Removed stray whitespace line
>
> arch/arm/include/asm/kvm_asm.h | 5 +++-
> arch/arm/kvm/hyp/switch.c | 2 +-
> arch/arm64/include/asm/kvm_asm.h | 4 ++-
> arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
> virt/kvm/arm/arm.c | 5 +++-
> 5 files changed, 69 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
> index 36dd2962a42d..4ac717276543 100644
> --- a/arch/arm/include/asm/kvm_asm.h
> +++ b/arch/arm/include/asm/kvm_asm.h
> @@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
>
> extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
>
> -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> +/* no VHE on 32-bit :( */
> +static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
> +
> +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
>
> extern void __init_stage2_translation(void);
>
> diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
> index c3b9799e2e13..7b2bd25e3b10 100644
> --- a/arch/arm/kvm/hyp/switch.c
> +++ b/arch/arm/kvm/hyp/switch.c
> @@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
> return true;
> }
>
> -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index 33e0edc6f8be..adaf1db12271 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
>
> extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
>
> -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> +extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
> +
> +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
>
> extern u64 __vgic_v3_get_ich_vtr_el2(void);
> extern u64 __vgic_v3_read_vmcr(void);
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 845e3bece399..8f32f8dcab65 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
> return false;
> }
>
> -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> +/* Switch to the guest for VHE systems running in EL2 */
> +int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> +{
> + struct kvm_cpu_context *host_ctxt;
> + struct kvm_cpu_context *guest_ctxt;
> + u64 exit_code;
> +
> + vcpu = kern_hyp_va(vcpu);
> +
> + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
> + host_ctxt->__hyp_running_vcpu = vcpu;
> + guest_ctxt = &vcpu->arch.ctxt;
> +
> + __sysreg_save_host_state(host_ctxt);
> +
> + __activate_traps(vcpu);
> + __activate_vm(vcpu);
> +
> + __vgic_restore_state(vcpu);
> + __timer_enable_traps(vcpu);
> +
> + /*
> + * We must restore the 32-bit state before the sysregs, thanks
> + * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
> + */
> + __sysreg32_restore_state(vcpu);
> + __sysreg_restore_guest_state(guest_ctxt);
> + __debug_switch_to_guest(vcpu);
> +
> + do {
> + /* Jump in the fire! */
> + exit_code = __guest_enter(vcpu, host_ctxt);
> +
> + /* And we're baaack! */
> + } while (fixup_guest_exit(vcpu, &exit_code));
> +
> + __sysreg_save_guest_state(guest_ctxt);
> + __sysreg32_save_state(vcpu);
> + __timer_disable_traps(vcpu);
> + __vgic_save_state(vcpu);
> +
> + __deactivate_traps(vcpu);
> + __deactivate_vm(vcpu);
> +
> + __sysreg_restore_host_state(host_ctxt);
> +
> + /*
> + * This must come after restoring the host sysregs, since a non-VHE
> + * system may enable SPE here and make use of the TTBRs.
> + */
> + __debug_switch_to_host(vcpu);
> +
> + return exit_code;
> +}
> +
> +/* Switch to the guest for legacy non-VHE systems */
> +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> {
> struct kvm_cpu_context *host_ctxt;
> struct kvm_cpu_context *guest_ctxt;
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 3e10343374a1..104ee524c75a 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
> trace_kvm_entry(*vcpu_pc(vcpu));
> guest_enter_irqoff();
>
> - ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
> + if (has_vhe())
> + ret = kvm_vcpu_run_vhe(vcpu);
> + else
> + ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
>
> vcpu->mode = OUTSIDE_GUEST_MODE;
> vcpu->stat.exits++;
>
Otherwise looks good to me.
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 12/36] KVM: arm64: Remove kern_hyp_va() use in VHE switch function
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 9:54 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:54 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: Shih-Wei Li, kvm
On 07/12/17 17:06, Christoffer Dall wrote:
> VHE kernels run completely in EL2 and therefore don't have a notion of
> kernel and hyp addresses, they are all just kernel addresses. Therefore
> don't call kern_hyp_va() in the VHE switch function.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 12/36] KVM: arm64: Remove kern_hyp_va() use in VHE switch function
@ 2017-12-11 9:54 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:54 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> VHE kernels run completely in EL2 and therefore don't have a notion of
> kernel and hyp addresses, they are all just kernel addresses. Therefore
> don't call kern_hyp_va() in the VHE switch function.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 13/36] KVM: arm64: Don't deactivate VM on VHE systems
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 9:58 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:58 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to reset the VTTBR to zero when exiting the guest on
> VHE systems. VHE systems don't use stage 2 translations for the EL2&0
> translation regime used by the host.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 13/36] KVM: arm64: Don't deactivate VM on VHE systems
@ 2017-12-11 9:58 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 9:58 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to reset the VTTBR to zero when exiting the guest on
> VHE systems. VHE systems don't use stage 2 translations for the EL2&0
> translation regime used by the host.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:02 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:02 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: Shih-Wei Li, kvm
On 07/12/17 17:06, Christoffer Dall wrote:
> The VHE switch function calls __timer_enable_traps and
> __timer_disable_traps which don't do anything on VHE systems.
> Therefore, simply remove these calls from the VHE switch function and
> make the functions non-conditional as they are now only called from the
> non-VHE switch path.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/kvm/hyp/switch.c | 2 --
> virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
> 2 files changed, 14 insertions(+), 24 deletions(-)
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index e783e2371b7c..09aafa0470f7 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> __activate_vm(vcpu->kvm);
>
> __vgic_restore_state(vcpu);
> - __timer_enable_traps(vcpu);
>
> /*
> * We must restore the 32-bit state before the sysregs, thanks
> @@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
>
> __sysreg_save_guest_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
> - __timer_disable_traps(vcpu);
> __vgic_save_state(vcpu);
>
> __deactivate_traps(vcpu);
> diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
> index f24404b3c8df..752b37f9133c 100644
> --- a/virt/kvm/arm/hyp/timer-sr.c
> +++ b/virt/kvm/arm/hyp/timer-sr.c
> @@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
>
> void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
> {
> - /*
> - * We don't need to do this for VHE since the host kernel runs in EL2
> - * with HCR_EL2.TGE ==1, which makes those bits have no impact.
> - */
> - if (!has_vhe()) {
> - u64 val;
> + u64 val;
>
> - /* Allow physical timer/counter access for the host */
> - val = read_sysreg(cnthctl_el2);
> - val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> - write_sysreg(val, cnthctl_el2);
> - }
> + /* Allow physical timer/counter access for the host */
> + val = read_sysreg(cnthctl_el2);
> + val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> + write_sysreg(val, cnthctl_el2);
> }
>
> void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
> {
> - if (!has_vhe()) {
> - u64 val;
> + u64 val;
>
> - /*
> - * Disallow physical timer access for the guest
> - * Physical counter access is allowed
> - */
> - val = read_sysreg(cnthctl_el2);
> - val &= ~CNTHCTL_EL1PCEN;
> - val |= CNTHCTL_EL1PCTEN;
> - write_sysreg(val, cnthctl_el2);
> - }
> + /*
> + * Disallow physical timer access for the guest
> + * Physical counter access is allowed
> + */
> + val = read_sysreg(cnthctl_el2);
> + val &= ~CNTHCTL_EL1PCEN;
> + val |= CNTHCTL_EL1PCTEN;
> + write_sysreg(val, cnthctl_el2);
> }
>
Since we're not testing for !VHE anymore, can you add a small comment
saying that these two function are for the benefit of !VHE only and
shouldn't be called on VHE?
Otherwise,
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
@ 2017-12-11 10:02 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:02 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> The VHE switch function calls __timer_enable_traps and
> __timer_disable_traps which don't do anything on VHE systems.
> Therefore, simply remove these calls from the VHE switch function and
> make the functions non-conditional as they are now only called from the
> non-VHE switch path.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/kvm/hyp/switch.c | 2 --
> virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
> 2 files changed, 14 insertions(+), 24 deletions(-)
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index e783e2371b7c..09aafa0470f7 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> __activate_vm(vcpu->kvm);
>
> __vgic_restore_state(vcpu);
> - __timer_enable_traps(vcpu);
>
> /*
> * We must restore the 32-bit state before the sysregs, thanks
> @@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
>
> __sysreg_save_guest_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
> - __timer_disable_traps(vcpu);
> __vgic_save_state(vcpu);
>
> __deactivate_traps(vcpu);
> diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
> index f24404b3c8df..752b37f9133c 100644
> --- a/virt/kvm/arm/hyp/timer-sr.c
> +++ b/virt/kvm/arm/hyp/timer-sr.c
> @@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
>
> void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
> {
> - /*
> - * We don't need to do this for VHE since the host kernel runs in EL2
> - * with HCR_EL2.TGE ==1, which makes those bits have no impact.
> - */
> - if (!has_vhe()) {
> - u64 val;
> + u64 val;
>
> - /* Allow physical timer/counter access for the host */
> - val = read_sysreg(cnthctl_el2);
> - val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> - write_sysreg(val, cnthctl_el2);
> - }
> + /* Allow physical timer/counter access for the host */
> + val = read_sysreg(cnthctl_el2);
> + val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> + write_sysreg(val, cnthctl_el2);
> }
>
> void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
> {
> - if (!has_vhe()) {
> - u64 val;
> + u64 val;
>
> - /*
> - * Disallow physical timer access for the guest
> - * Physical counter access is allowed
> - */
> - val = read_sysreg(cnthctl_el2);
> - val &= ~CNTHCTL_EL1PCEN;
> - val |= CNTHCTL_EL1PCTEN;
> - write_sysreg(val, cnthctl_el2);
> - }
> + /*
> + * Disallow physical timer access for the guest
> + * Physical counter access is allowed
> + */
> + val = read_sysreg(cnthctl_el2);
> + val &= ~CNTHCTL_EL1PCEN;
> + val |= CNTHCTL_EL1PCTEN;
> + write_sysreg(val, cnthctl_el2);
> }
>
Since we're not testing for !VHE anymore, can you add a small comment
saying that these two function are for the benefit of !VHE only and
shouldn't be called on VHE?
Otherwise,
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:14 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:14 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> There's a semantic difference between the EL1 registers that control
> operation of a kernel running in EL1 and EL1 registers that only control
> userspace execution in EL0. Since we can defer saving/restoring the
> latter, move them into their own function.
>
> We also take this chance to rename the function saving/restoring the
> remaining system register to make it clear this function deals with
> the EL1 system registers.
>
> No functional change.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Added comment about sp_el0 to common save sysreg save/restore functions
>
> arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
> 1 file changed, 33 insertions(+), 11 deletions(-)
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index 68a7d164e5e1..bbfb4d01af88 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
> */
>
> static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
> +{
> + ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> +
> + /*
> + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> + * therefore be saved/restored on every entry/exit to/from the guest.
> + */
> + ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> +}
> +
> +static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
> {
> ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
What is the rational for keeping ACTLR_EL1 as part of the user state?
> ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
> ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
> - ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> - ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> }
>
> -static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> +static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> {
> ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
> ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
> @@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> }
>
> static hyp_alternate_select(__sysreg_call_save_host_state,
> - __sysreg_save_state, __sysreg_do_nothing,
> + __sysreg_save_el1_state, __sysreg_do_nothing,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
> {
> __sysreg_call_save_host_state()(ctxt);
> __sysreg_save_common_state(ctxt);
> + __sysreg_save_user_state(ctxt);
> }
>
> void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
> {
> - __sysreg_save_state(ctxt);
> + __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> + __sysreg_save_user_state(ctxt);
> }
>
> static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> {
> - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
> - write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> - write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
> +
> + /*
> + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> + * therefore be saved/restored on every entry/exit to/from the guest.
> + */
> write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
> }
>
> -static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> +static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
> +{
> + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
Same here.
> + write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> + write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> +}
> +
> +static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> {
> write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
> write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
> @@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> }
>
> static hyp_alternate_select(__sysreg_call_restore_host_state,
> - __sysreg_restore_state, __sysreg_do_nothing,
> + __sysreg_restore_el1_state, __sysreg_do_nothing,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
> {
> __sysreg_call_restore_host_state()(ctxt);
> __sysreg_restore_common_state(ctxt);
> + __sysreg_restore_user_state(ctxt);
> }
>
> void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> {
> - __sysreg_restore_state(ctxt);
> + __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> + __sysreg_restore_user_state(ctxt);
> }
>
> static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
>
I think we should move ACTLR_EL1 to the EL1 state, allowing it to be
lazily switched. See the note in D10.2.1 that recommends a VHE enabled
system to have ACTLR_EL1 as a guest-only register.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
@ 2017-12-11 10:14 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:14 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> There's a semantic difference between the EL1 registers that control
> operation of a kernel running in EL1 and EL1 registers that only control
> userspace execution in EL0. Since we can defer saving/restoring the
> latter, move them into their own function.
>
> We also take this chance to rename the function saving/restoring the
> remaining system register to make it clear this function deals with
> the EL1 system registers.
>
> No functional change.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Added comment about sp_el0 to common save sysreg save/restore functions
>
> arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
> 1 file changed, 33 insertions(+), 11 deletions(-)
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index 68a7d164e5e1..bbfb4d01af88 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
> */
>
> static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
> +{
> + ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> +
> + /*
> + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> + * therefore be saved/restored on every entry/exit to/from the guest.
> + */
> + ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> +}
> +
> +static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
> {
> ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
What is the rational for keeping ACTLR_EL1 as part of the user state?
> ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
> ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
> - ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> - ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> }
>
> -static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> +static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> {
> ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
> ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
> @@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> }
>
> static hyp_alternate_select(__sysreg_call_save_host_state,
> - __sysreg_save_state, __sysreg_do_nothing,
> + __sysreg_save_el1_state, __sysreg_do_nothing,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
> {
> __sysreg_call_save_host_state()(ctxt);
> __sysreg_save_common_state(ctxt);
> + __sysreg_save_user_state(ctxt);
> }
>
> void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
> {
> - __sysreg_save_state(ctxt);
> + __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> + __sysreg_save_user_state(ctxt);
> }
>
> static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> {
> - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
> - write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> - write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
> +
> + /*
> + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> + * therefore be saved/restored on every entry/exit to/from the guest.
> + */
> write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
> }
>
> -static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> +static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
> +{
> + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
Same here.
> + write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> + write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> +}
> +
> +static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> {
> write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
> write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
> @@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> }
>
> static hyp_alternate_select(__sysreg_call_restore_host_state,
> - __sysreg_restore_state, __sysreg_do_nothing,
> + __sysreg_restore_el1_state, __sysreg_do_nothing,
> ARM64_HAS_VIRT_HOST_EXTN);
>
> void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
> {
> __sysreg_call_restore_host_state()(ctxt);
> __sysreg_restore_common_state(ctxt);
> + __sysreg_restore_user_state(ctxt);
> }
>
> void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> {
> - __sysreg_restore_state(ctxt);
> + __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> + __sysreg_restore_user_state(ctxt);
> }
>
> static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
>
I think we should move ACTLR_EL1 to the EL1 state, allowing it to be
lazily switched. See the note in D10.2.1 that recommends a VHE enabled
system to have ACTLR_EL1 as a guest-only register.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 16/36] KVM: arm64: Rewrite sysreg alternatives to static keys
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:15 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:15 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to move calls around in the sysreg save/restore logic,
> let's first rewrite the alternative function callers, because it is
> going to make the next patches much easier to read.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 16/36] KVM: arm64: Rewrite sysreg alternatives to static keys
@ 2017-12-11 10:15 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:15 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to move calls around in the sysreg save/restore logic,
> let's first rewrite the alternative function callers, because it is
> going to make the next patches much easier to read.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 17/36] KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functions
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:22 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:22 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: Shih-Wei Li, kvm
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to handle system registers quite differently between VHE
> and non-VHE systems. In preparation for that, we need to split some of
> the handling functions between VHE and non-VHE functionality.
>
> For now, we simply copy the non-VHE functions, but we do change the use
> of static keys for VHE and non-VHE functionality now that we have
> separate functions.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 17/36] KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functions
@ 2017-12-11 10:22 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:22 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to handle system registers quite differently between VHE
> and non-VHE systems. In preparation for that, we need to split some of
> the handling functions between VHE and non-VHE functionality.
>
> For now, we simply copy the non-VHE functions, but we do change the use
> of static keys for VHE and non-VHE functionality now that we have
> separate functions.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 18/36] KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:30 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:30 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> The comment only applied to SPE on non-VHE systems, so we simply remove
> it.
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M. (Yeah, I made it halfway!)
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 18/36] KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
@ 2017-12-11 10:30 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:30 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> The comment only applied to SPE on non-VHE systems, so we simply remove
> it.
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
M. (Yeah, I made it halfway!)
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 19/36] KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:40 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:40 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to have multiple identical functions with different
> names for saving host and guest state. When saving and restoring state
> for the host and guest, the state is the same for both contexts, and
> that's why we have the kvm_cpu_context structure. Delete one
> version and rename the other into simply save/restore.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 19/36] KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
@ 2017-12-11 10:40 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:40 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to have multiple identical functions with different
> names for saving host and guest state. When saving and restoring state
> for the host and guest, the state is the same for both contexts, and
> that's why we have the kvm_cpu_context structure. Delete one
> version and rename the other into simply save/restore.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:44 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:44 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: Shih-Wei Li, kvm
On 07/12/17 17:06, Christoffer Dall wrote:
> On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
> can return to the host in EL1 in the same state and location where we
> issued a hypercall to EL2, but these registers don't contain anything
> important on VHE, because all of the host runs in EL2. Therefore,
If I may refine the rational: ELR_EL2 and SPSR_EL2 are not useful here
because we never enter a guest as a result of an exception entry that
would be directly handled by KVM. The kernel entry code already saves
ELR_EL1/SPSR_EL1 on exception entry, which is enough.
> factor out these registers into separate save/restore functions, making
> it easy to exclude them from the VHE world-switch path later on.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index a12112494f75..479de0f0dd07 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
> ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
> ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
> +}
> +
> +static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
> +{
> ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
> ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
> }
> @@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
> __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> __sysreg_save_user_state(ctxt);
> + __sysreg_save_el2_return_state(ctxt);
> }
>
> void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
> @@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
> __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> __sysreg_save_user_state(ctxt);
> + __sysreg_save_el2_return_state(ctxt);
> }
>
> static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> @@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
> write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
> write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
> +}
> +
> +static void __hyp_text
> +__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
> +{
> write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
> write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
> }
> @@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
> __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> __sysreg_restore_user_state(ctxt);
> + __sysreg_restore_el2_return_state(ctxt);
> }
>
> void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
> @@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
> __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> __sysreg_restore_user_state(ctxt);
> + __sysreg_restore_el2_return_state(ctxt);
> }
>
> static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
>
Otherwise:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
@ 2017-12-11 10:44 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:44 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
> can return to the host in EL1 in the same state and location where we
> issued a hypercall to EL2, but these registers don't contain anything
> important on VHE, because all of the host runs in EL2. Therefore,
If I may refine the rational: ELR_EL2 and SPSR_EL2 are not useful here
because we never enter a guest as a result of an exception entry that
would be directly handled by KVM. The kernel entry code already saves
ELR_EL1/SPSR_EL1 on exception entry, which is enough.
> factor out these registers into separate save/restore functions, making
> it easy to exclude them from the VHE world-switch path later on.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index a12112494f75..479de0f0dd07 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
> ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
> ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
> +}
> +
> +static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
> +{
> ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
> ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
> }
> @@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
> __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> __sysreg_save_user_state(ctxt);
> + __sysreg_save_el2_return_state(ctxt);
> }
>
> void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
> @@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
> __sysreg_save_el1_state(ctxt);
> __sysreg_save_common_state(ctxt);
> __sysreg_save_user_state(ctxt);
> + __sysreg_save_el2_return_state(ctxt);
> }
>
> static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> @@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
> write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
> write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
> +}
> +
> +static void __hyp_text
> +__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
> +{
> write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
> write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
> }
> @@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
> __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> __sysreg_restore_user_state(ctxt);
> + __sysreg_restore_el2_return_state(ctxt);
> }
>
> void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
> @@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
> __sysreg_restore_el1_state(ctxt);
> __sysreg_restore_common_state(ctxt);
> __sysreg_restore_user_state(ctxt);
> + __sysreg_restore_el2_return_state(ctxt);
> }
>
> static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
>
Otherwise:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 21/36] KVM: arm64: Change 32-bit handling of VM system registers
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 10:57 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:57 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: Shih-Wei Li, kvm
On 07/12/17 17:06, Christoffer Dall wrote:
> We currently handle 32-bit accesses to trapped VM system registers using
> the 32-bit index into the coproc array on the vcpu structure, which is a
> union of the coproc array and the sysreg array.
>
> Since all the 32-bit coproc indicies are created to correspond to the
indices?
> architectural mapping between 64-bit system registers and 32-bit
> coprocessor registers, and because the AArch64 system registers are the
> double in size of the AArch32 coprocessor registers, we can always find
> the system register entry that we must update by dividing the 32-bit
> coproc index by 2.
>
> This is going to make our lives much easier when we have to start
> accessing system registers that use deferred save/restore and might
> have to be read directly from the physical CPU.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/include/asm/kvm_host.h | 8 --------
> arch/arm64/kvm/sys_regs.c | 20 +++++++++++++++-----
> 2 files changed, 15 insertions(+), 13 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index c841eeeeb5c5..de0d55b30b61 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -290,14 +290,6 @@ struct kvm_vcpu_arch {
> #define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
> #define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
>
> -#ifdef CONFIG_CPU_BIG_ENDIAN
> -#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r))
> -#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r) + 1)
> -#else
> -#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r) + 1)
> -#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r))
> -#endif
> -
> struct kvm_vm_stat {
> ulong remote_tlb_flush;
> };
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 1830ebc227d1..62c12ab9e6c4 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -121,16 +121,26 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> const struct sys_reg_desc *r)
> {
> bool was_enabled = vcpu_has_cache_enabled(vcpu);
> + u64 val;
> + int reg = r->reg;
>
> BUG_ON(!p->is_write);
>
> - if (!p->is_aarch32) {
> - vcpu_sys_reg(vcpu, r->reg) = p->regval;
> + /* See the 32bit mapping in kvm_host.h */
> + if (p->is_aarch32)
> + reg = r->reg / 2;
> +
> + if (!p->is_aarch32 || !p->is_32bit) {
> + val = p->regval;
> } else {
> - if (!p->is_32bit)
> - vcpu_cp15_64_high(vcpu, r->reg) = upper_32_bits(p->regval);
> - vcpu_cp15_64_low(vcpu, r->reg) = lower_32_bits(p->regval);
> + val = vcpu_sys_reg(vcpu, reg);
> + if (r->reg % 2)
> + val = (p->regval << 32) | (u64)lower_32_bits(val);
> + else
> + val = ((u64)upper_32_bits(val) << 32) |
> + (u64)lower_32_bits(p->regval);
Nit: the u64 casts on lower_32_bits are superfluous.
> }
> + vcpu_sys_reg(vcpu, reg) = val;
>
> kvm_toggle_cache(vcpu, was_enabled);
> return true;
>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 21/36] KVM: arm64: Change 32-bit handling of VM system registers
@ 2017-12-11 10:57 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 10:57 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> We currently handle 32-bit accesses to trapped VM system registers using
> the 32-bit index into the coproc array on the vcpu structure, which is a
> union of the coproc array and the sysreg array.
>
> Since all the 32-bit coproc indicies are created to correspond to the
indices?
> architectural mapping between 64-bit system registers and 32-bit
> coprocessor registers, and because the AArch64 system registers are the
> double in size of the AArch32 coprocessor registers, we can always find
> the system register entry that we must update by dividing the 32-bit
> coproc index by 2.
>
> This is going to make our lives much easier when we have to start
> accessing system registers that use deferred save/restore and might
> have to be read directly from the physical CPU.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/include/asm/kvm_host.h | 8 --------
> arch/arm64/kvm/sys_regs.c | 20 +++++++++++++++-----
> 2 files changed, 15 insertions(+), 13 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index c841eeeeb5c5..de0d55b30b61 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -290,14 +290,6 @@ struct kvm_vcpu_arch {
> #define vcpu_cp14(v,r) ((v)->arch.ctxt.copro[(r)])
> #define vcpu_cp15(v,r) ((v)->arch.ctxt.copro[(r)])
>
> -#ifdef CONFIG_CPU_BIG_ENDIAN
> -#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r))
> -#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r) + 1)
> -#else
> -#define vcpu_cp15_64_high(v,r) vcpu_cp15((v),(r) + 1)
> -#define vcpu_cp15_64_low(v,r) vcpu_cp15((v),(r))
> -#endif
> -
> struct kvm_vm_stat {
> ulong remote_tlb_flush;
> };
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 1830ebc227d1..62c12ab9e6c4 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -121,16 +121,26 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> const struct sys_reg_desc *r)
> {
> bool was_enabled = vcpu_has_cache_enabled(vcpu);
> + u64 val;
> + int reg = r->reg;
>
> BUG_ON(!p->is_write);
>
> - if (!p->is_aarch32) {
> - vcpu_sys_reg(vcpu, r->reg) = p->regval;
> + /* See the 32bit mapping in kvm_host.h */
> + if (p->is_aarch32)
> + reg = r->reg / 2;
> +
> + if (!p->is_aarch32 || !p->is_32bit) {
> + val = p->regval;
> } else {
> - if (!p->is_32bit)
> - vcpu_cp15_64_high(vcpu, r->reg) = upper_32_bits(p->regval);
> - vcpu_cp15_64_low(vcpu, r->reg) = lower_32_bits(p->regval);
> + val = vcpu_sys_reg(vcpu, reg);
> + if (r->reg % 2)
> + val = (p->regval << 32) | (u64)lower_32_bits(val);
> + else
> + val = ((u64)upper_32_bits(val) << 32) |
> + (u64)lower_32_bits(p->regval);
Nit: the u64 casts on lower_32_bits are superfluous.
> }
> + vcpu_sys_reg(vcpu, reg) = val;
>
> kvm_toggle_cache(vcpu, was_enabled);
> return true;
>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 11:10 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 11:10 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> When we defer the save/restore of system registers to vcpu_load and
> vcpu_put, we need to take care of the emulation code that handles traps
> to these registers, since simply reading the memory array will return
> stale data.
>
> Therefore, introduce two functions to directly read/write the registers
> from the physical CPU when we're on a VHE system that has loaded the
> system registers onto the physical CPU.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Removed spurious white space
>
> arch/arm64/include/asm/kvm_host.h | 4 +++
> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> 2 files changed, 55 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index de0d55b30b61..f6afe685a280 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>
> /* Detect first run of a vcpu */
> bool has_run_once;
> +
> + /* True when deferrable sysregs are loaded on the physical CPU,
> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> + bool sysregs_loaded_on_cpu;
> };
>
> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 62c12ab9e6c4..80adbec933de 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -35,6 +35,7 @@
> #include <asm/kvm_coproc.h>
> #include <asm/kvm_emulate.h>
> #include <asm/kvm_host.h>
> +#include <asm/kvm_hyp.h>
> #include <asm/kvm_mmu.h>
> #include <asm/perf_event.h>
> #include <asm/sysreg.h>
> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> return true;
> }
>
> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> +{
> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> + switch (reg) {
> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> + case TCR_EL1: return read_sysreg_el1(tcr);
> + case ESR_EL1: return read_sysreg_el1(esr);
> + case FAR_EL1: return read_sysreg_el1(far);
> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> + case MAIR_EL1: return read_sysreg_el1(mair);
> + case AMAIR_EL1: return read_sysreg_el1(amair);
> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> + case DACR32_EL2: return read_sysreg(dacr32_el2);
> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> + default: BUG();
> + }
> + }
> +
> + return vcpu_sys_reg(vcpu, reg);
> +}
> +
> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> +{
> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> + switch (reg) {
> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> + case ESR_EL1: write_sysreg_el1(val, esr); return;
> + case FAR_EL1: write_sysreg_el1(val, far); return;
> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> + default: BUG();
> + }
> + }
> +
> + vcpu_sys_reg(vcpu, reg) = val;
> +}
> +
> /*
> * Generic accessor for VM registers. Only called as long as HCR_TVM
> * is set. If the guest enables the MMU, we stop trapping the VM
> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> if (!p->is_aarch32 || !p->is_32bit) {
> val = p->regval;
> } else {
> - val = vcpu_sys_reg(vcpu, reg);
> + val = read_deferrable_vm_reg(vcpu, reg);
> if (r->reg % 2)
> val = (p->regval << 32) | (u64)lower_32_bits(val);
> else
> val = ((u64)upper_32_bits(val) << 32) |
> (u64)lower_32_bits(p->regval);
> }
> - vcpu_sys_reg(vcpu, reg) = val;
> + write_deferrable_vm_reg(vcpu, reg, val);
>
> kvm_toggle_cache(vcpu, was_enabled);
> return true;
>
I'm slightly uneasy with this. It means that the rest of the KVM code
has to know whether a given register is deferrable or not (or face the
wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
magic in the vcpu_sys_reg() accessors.
Thoughts?
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-11 11:10 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 11:10 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> When we defer the save/restore of system registers to vcpu_load and
> vcpu_put, we need to take care of the emulation code that handles traps
> to these registers, since simply reading the memory array will return
> stale data.
>
> Therefore, introduce two functions to directly read/write the registers
> from the physical CPU when we're on a VHE system that has loaded the
> system registers onto the physical CPU.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
>
> Notes:
> Changes since v1:
> - Removed spurious white space
>
> arch/arm64/include/asm/kvm_host.h | 4 +++
> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> 2 files changed, 55 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index de0d55b30b61..f6afe685a280 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>
> /* Detect first run of a vcpu */
> bool has_run_once;
> +
> + /* True when deferrable sysregs are loaded on the physical CPU,
> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> + bool sysregs_loaded_on_cpu;
> };
>
> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 62c12ab9e6c4..80adbec933de 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -35,6 +35,7 @@
> #include <asm/kvm_coproc.h>
> #include <asm/kvm_emulate.h>
> #include <asm/kvm_host.h>
> +#include <asm/kvm_hyp.h>
> #include <asm/kvm_mmu.h>
> #include <asm/perf_event.h>
> #include <asm/sysreg.h>
> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> return true;
> }
>
> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> +{
> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> + switch (reg) {
> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> + case TCR_EL1: return read_sysreg_el1(tcr);
> + case ESR_EL1: return read_sysreg_el1(esr);
> + case FAR_EL1: return read_sysreg_el1(far);
> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> + case MAIR_EL1: return read_sysreg_el1(mair);
> + case AMAIR_EL1: return read_sysreg_el1(amair);
> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> + case DACR32_EL2: return read_sysreg(dacr32_el2);
> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> + default: BUG();
> + }
> + }
> +
> + return vcpu_sys_reg(vcpu, reg);
> +}
> +
> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> +{
> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> + switch (reg) {
> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> + case ESR_EL1: write_sysreg_el1(val, esr); return;
> + case FAR_EL1: write_sysreg_el1(val, far); return;
> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> + default: BUG();
> + }
> + }
> +
> + vcpu_sys_reg(vcpu, reg) = val;
> +}
> +
> /*
> * Generic accessor for VM registers. Only called as long as HCR_TVM
> * is set. If the guest enables the MMU, we stop trapping the VM
> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> if (!p->is_aarch32 || !p->is_32bit) {
> val = p->regval;
> } else {
> - val = vcpu_sys_reg(vcpu, reg);
> + val = read_deferrable_vm_reg(vcpu, reg);
> if (r->reg % 2)
> val = (p->regval << 32) | (u64)lower_32_bits(val);
> else
> val = ((u64)upper_32_bits(val) << 32) |
> (u64)lower_32_bits(p->regval);
> }
> - vcpu_sys_reg(vcpu, reg) = val;
> + write_deferrable_vm_reg(vcpu, reg, val);
>
> kvm_toggle_cache(vcpu, was_enabled);
> return true;
>
I'm slightly uneasy with this. It means that the rest of the KVM code
has to know whether a given register is deferrable or not (or face the
wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
magic in the vcpu_sys_reg() accessors.
Thoughts?
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-11 11:10 ` Marc Zyngier
@ 2017-12-11 11:24 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 11:24 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > When we defer the save/restore of system registers to vcpu_load and
> > vcpu_put, we need to take care of the emulation code that handles traps
> > to these registers, since simply reading the memory array will return
> > stale data.
> >
> > Therefore, introduce two functions to directly read/write the registers
> > from the physical CPU when we're on a VHE system that has loaded the
> > system registers onto the physical CPU.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Removed spurious white space
> >
> > arch/arm64/include/asm/kvm_host.h | 4 +++
> > arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> > 2 files changed, 55 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index de0d55b30b61..f6afe685a280 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
> >
> > /* Detect first run of a vcpu */
> > bool has_run_once;
> > +
> > + /* True when deferrable sysregs are loaded on the physical CPU,
> > + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> > + bool sysregs_loaded_on_cpu;
> > };
> >
> > #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> > diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> > index 62c12ab9e6c4..80adbec933de 100644
> > --- a/arch/arm64/kvm/sys_regs.c
> > +++ b/arch/arm64/kvm/sys_regs.c
> > @@ -35,6 +35,7 @@
> > #include <asm/kvm_coproc.h>
> > #include <asm/kvm_emulate.h>
> > #include <asm/kvm_host.h>
> > +#include <asm/kvm_hyp.h>
> > #include <asm/kvm_mmu.h>
> > #include <asm/perf_event.h>
> > #include <asm/sysreg.h>
> > @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> > return true;
> > }
> >
> > +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> > +{
> > + if (vcpu->arch.sysregs_loaded_on_cpu) {
> > + switch (reg) {
> > + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> > + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> > + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> > + case TCR_EL1: return read_sysreg_el1(tcr);
> > + case ESR_EL1: return read_sysreg_el1(esr);
> > + case FAR_EL1: return read_sysreg_el1(far);
> > + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> > + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> > + case MAIR_EL1: return read_sysreg_el1(mair);
> > + case AMAIR_EL1: return read_sysreg_el1(amair);
> > + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> > + case DACR32_EL2: return read_sysreg(dacr32_el2);
> > + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> > + default: BUG();
> > + }
> > + }
> > +
> > + return vcpu_sys_reg(vcpu, reg);
> > +}
> > +
> > +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> > +{
> > + if (vcpu->arch.sysregs_loaded_on_cpu) {
> > + switch (reg) {
> > + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> > + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> > + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> > + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> > + case ESR_EL1: write_sysreg_el1(val, esr); return;
> > + case FAR_EL1: write_sysreg_el1(val, far); return;
> > + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> > + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> > + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> > + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> > + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> > + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> > + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> > + default: BUG();
> > + }
> > + }
> > +
> > + vcpu_sys_reg(vcpu, reg) = val;
> > +}
> > +
> > /*
> > * Generic accessor for VM registers. Only called as long as HCR_TVM
> > * is set. If the guest enables the MMU, we stop trapping the VM
> > @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> > if (!p->is_aarch32 || !p->is_32bit) {
> > val = p->regval;
> > } else {
> > - val = vcpu_sys_reg(vcpu, reg);
> > + val = read_deferrable_vm_reg(vcpu, reg);
> > if (r->reg % 2)
> > val = (p->regval << 32) | (u64)lower_32_bits(val);
> > else
> > val = ((u64)upper_32_bits(val) << 32) |
> > (u64)lower_32_bits(p->regval);
> > }
> > - vcpu_sys_reg(vcpu, reg) = val;
> > + write_deferrable_vm_reg(vcpu, reg, val);
> >
> > kvm_toggle_cache(vcpu, was_enabled);
> > return true;
> >
>
> I'm slightly uneasy with this. It means that the rest of the KVM code
> has to know whether a given register is deferrable or not (or face the
> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
> magic in the vcpu_sys_reg() accessors.
>
> Thoughts?
>
Yes, this is the main reservation I also have with the series.
I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
vcpu_get_sys_reg" which hides this logic, and we may want to go back to
that.
That does mean that we need a giant switch statement which knows how to
read any deferrable EL1 (and EL0) system register from hardware, and
still BUG/WARN if someone adds a system register but forgets to add that
handler and test on VHE. Unless there's some fantastic auto-gen
mechanism that can take a hash define and figure out which sysreg
instruction to use - I couldn't think of that.
I'm happy to go back to that approach, but I didn't find it that much
nicer either.
How about I send you the small handful of patches that implement the
alternative approach and you have a look at that?
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-11 11:24 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 11:24 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > When we defer the save/restore of system registers to vcpu_load and
> > vcpu_put, we need to take care of the emulation code that handles traps
> > to these registers, since simply reading the memory array will return
> > stale data.
> >
> > Therefore, introduce two functions to directly read/write the registers
> > from the physical CPU when we're on a VHE system that has loaded the
> > system registers onto the physical CPU.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Removed spurious white space
> >
> > arch/arm64/include/asm/kvm_host.h | 4 +++
> > arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> > 2 files changed, 55 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index de0d55b30b61..f6afe685a280 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
> >
> > /* Detect first run of a vcpu */
> > bool has_run_once;
> > +
> > + /* True when deferrable sysregs are loaded on the physical CPU,
> > + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> > + bool sysregs_loaded_on_cpu;
> > };
> >
> > #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> > diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> > index 62c12ab9e6c4..80adbec933de 100644
> > --- a/arch/arm64/kvm/sys_regs.c
> > +++ b/arch/arm64/kvm/sys_regs.c
> > @@ -35,6 +35,7 @@
> > #include <asm/kvm_coproc.h>
> > #include <asm/kvm_emulate.h>
> > #include <asm/kvm_host.h>
> > +#include <asm/kvm_hyp.h>
> > #include <asm/kvm_mmu.h>
> > #include <asm/perf_event.h>
> > #include <asm/sysreg.h>
> > @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> > return true;
> > }
> >
> > +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> > +{
> > + if (vcpu->arch.sysregs_loaded_on_cpu) {
> > + switch (reg) {
> > + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> > + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> > + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> > + case TCR_EL1: return read_sysreg_el1(tcr);
> > + case ESR_EL1: return read_sysreg_el1(esr);
> > + case FAR_EL1: return read_sysreg_el1(far);
> > + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> > + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> > + case MAIR_EL1: return read_sysreg_el1(mair);
> > + case AMAIR_EL1: return read_sysreg_el1(amair);
> > + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> > + case DACR32_EL2: return read_sysreg(dacr32_el2);
> > + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> > + default: BUG();
> > + }
> > + }
> > +
> > + return vcpu_sys_reg(vcpu, reg);
> > +}
> > +
> > +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> > +{
> > + if (vcpu->arch.sysregs_loaded_on_cpu) {
> > + switch (reg) {
> > + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> > + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> > + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> > + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> > + case ESR_EL1: write_sysreg_el1(val, esr); return;
> > + case FAR_EL1: write_sysreg_el1(val, far); return;
> > + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> > + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> > + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> > + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> > + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> > + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> > + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> > + default: BUG();
> > + }
> > + }
> > +
> > + vcpu_sys_reg(vcpu, reg) = val;
> > +}
> > +
> > /*
> > * Generic accessor for VM registers. Only called as long as HCR_TVM
> > * is set. If the guest enables the MMU, we stop trapping the VM
> > @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> > if (!p->is_aarch32 || !p->is_32bit) {
> > val = p->regval;
> > } else {
> > - val = vcpu_sys_reg(vcpu, reg);
> > + val = read_deferrable_vm_reg(vcpu, reg);
> > if (r->reg % 2)
> > val = (p->regval << 32) | (u64)lower_32_bits(val);
> > else
> > val = ((u64)upper_32_bits(val) << 32) |
> > (u64)lower_32_bits(p->regval);
> > }
> > - vcpu_sys_reg(vcpu, reg) = val;
> > + write_deferrable_vm_reg(vcpu, reg, val);
> >
> > kvm_toggle_cache(vcpu, was_enabled);
> > return true;
> >
>
> I'm slightly uneasy with this. It means that the rest of the KVM code
> has to know whether a given register is deferrable or not (or face the
> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
> magic in the vcpu_sys_reg() accessors.
>
> Thoughts?
>
Yes, this is the main reservation I also have with the series.
I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
vcpu_get_sys_reg" which hides this logic, and we may want to go back to
that.
That does mean that we need a giant switch statement which knows how to
read any deferrable EL1 (and EL0) system register from hardware, and
still BUG/WARN if someone adds a system register but forgets to add that
handler and test on VHE. Unless there's some fantastic auto-gen
mechanism that can take a hash define and figure out which sysreg
instruction to use - I couldn't think of that.
I'm happy to go back to that approach, but I didn't find it that much
nicer either.
How about I send you the small handful of patches that implement the
alternative approach and you have a look at that?
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-11 11:24 ` Christoffer Dall
@ 2017-12-11 11:46 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 11:46 UTC (permalink / raw)
To: Christoffer Dall; +Cc: Shih-Wei Li, kvmarm, linux-arm-kernel, kvm
On 11/12/17 11:24, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> When we defer the save/restore of system registers to vcpu_load and
>>> vcpu_put, we need to take care of the emulation code that handles traps
>>> to these registers, since simply reading the memory array will return
>>> stale data.
>>>
>>> Therefore, introduce two functions to directly read/write the registers
>>> from the physical CPU when we're on a VHE system that has loaded the
>>> system registers onto the physical CPU.
>>>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Removed spurious white space
>>>
>>> arch/arm64/include/asm/kvm_host.h | 4 +++
>>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
>>> 2 files changed, 55 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index de0d55b30b61..f6afe685a280 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>>>
>>> /* Detect first run of a vcpu */
>>> bool has_run_once;
>>> +
>>> + /* True when deferrable sysregs are loaded on the physical CPU,
>>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
>>> + bool sysregs_loaded_on_cpu;
>>> };
>>>
>>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 62c12ab9e6c4..80adbec933de 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -35,6 +35,7 @@
>>> #include <asm/kvm_coproc.h>
>>> #include <asm/kvm_emulate.h>
>>> #include <asm/kvm_host.h>
>>> +#include <asm/kvm_hyp.h>
>>> #include <asm/kvm_mmu.h>
>>> #include <asm/perf_event.h>
>>> #include <asm/sysreg.h>
>>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
>>> return true;
>>> }
>>>
>>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
>>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
>>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
>>> + case TCR_EL1: return read_sysreg_el1(tcr);
>>> + case ESR_EL1: return read_sysreg_el1(esr);
>>> + case FAR_EL1: return read_sysreg_el1(far);
>>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
>>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
>>> + case MAIR_EL1: return read_sysreg_el1(mair);
>>> + case AMAIR_EL1: return read_sysreg_el1(amair);
>>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
>>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
>>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + return vcpu_sys_reg(vcpu, reg);
>>> +}
>>> +
>>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
>>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
>>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
>>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
>>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
>>> + case FAR_EL1: write_sysreg_el1(val, far); return;
>>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
>>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
>>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
>>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
>>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
>>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
>>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + vcpu_sys_reg(vcpu, reg) = val;
>>> +}
>>> +
>>> /*
>>> * Generic accessor for VM registers. Only called as long as HCR_TVM
>>> * is set. If the guest enables the MMU, we stop trapping the VM
>>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
>>> if (!p->is_aarch32 || !p->is_32bit) {
>>> val = p->regval;
>>> } else {
>>> - val = vcpu_sys_reg(vcpu, reg);
>>> + val = read_deferrable_vm_reg(vcpu, reg);
>>> if (r->reg % 2)
>>> val = (p->regval << 32) | (u64)lower_32_bits(val);
>>> else
>>> val = ((u64)upper_32_bits(val) << 32) |
>>> (u64)lower_32_bits(p->regval);
>>> }
>>> - vcpu_sys_reg(vcpu, reg) = val;
>>> + write_deferrable_vm_reg(vcpu, reg, val);
>>>
>>> kvm_toggle_cache(vcpu, was_enabled);
>>> return true;
>>>
>>
>> I'm slightly uneasy with this. It means that the rest of the KVM code
>> has to know whether a given register is deferrable or not (or face the
>> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
>> magic in the vcpu_sys_reg() accessors.
>>
>> Thoughts?
>>
>
> Yes, this is the main reservation I also have with the series.
>
> I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> that.
>
> That does mean that we need a giant switch statement which knows how to
> read any deferrable EL1 (and EL0) system register from hardware, and
> still BUG/WARN if someone adds a system register but forgets to add that
> handler and test on VHE. Unless there's some fantastic auto-gen
> mechanism that can take a hash define and figure out which sysreg
> instruction to use - I couldn't think of that.
>
> I'm happy to go back to that approach, but I didn't find it that much
> nicer either.
>
> How about I send you the small handful of patches that implement the
> alternative approach and you have a look at that?
Sure, feel free to post them. I wonder if we can take a similar approach
to the hack I used for the CP15 stuff on 32bit, where read/write_sysreg
are automagically turned into the right type of accessor...
I'll have a try.
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-11 11:46 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 11:46 UTC (permalink / raw)
To: linux-arm-kernel
On 11/12/17 11:24, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> When we defer the save/restore of system registers to vcpu_load and
>>> vcpu_put, we need to take care of the emulation code that handles traps
>>> to these registers, since simply reading the memory array will return
>>> stale data.
>>>
>>> Therefore, introduce two functions to directly read/write the registers
>>> from the physical CPU when we're on a VHE system that has loaded the
>>> system registers onto the physical CPU.
>>>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Removed spurious white space
>>>
>>> arch/arm64/include/asm/kvm_host.h | 4 +++
>>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
>>> 2 files changed, 55 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index de0d55b30b61..f6afe685a280 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>>>
>>> /* Detect first run of a vcpu */
>>> bool has_run_once;
>>> +
>>> + /* True when deferrable sysregs are loaded on the physical CPU,
>>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
>>> + bool sysregs_loaded_on_cpu;
>>> };
>>>
>>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 62c12ab9e6c4..80adbec933de 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -35,6 +35,7 @@
>>> #include <asm/kvm_coproc.h>
>>> #include <asm/kvm_emulate.h>
>>> #include <asm/kvm_host.h>
>>> +#include <asm/kvm_hyp.h>
>>> #include <asm/kvm_mmu.h>
>>> #include <asm/perf_event.h>
>>> #include <asm/sysreg.h>
>>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
>>> return true;
>>> }
>>>
>>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
>>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
>>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
>>> + case TCR_EL1: return read_sysreg_el1(tcr);
>>> + case ESR_EL1: return read_sysreg_el1(esr);
>>> + case FAR_EL1: return read_sysreg_el1(far);
>>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
>>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
>>> + case MAIR_EL1: return read_sysreg_el1(mair);
>>> + case AMAIR_EL1: return read_sysreg_el1(amair);
>>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
>>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
>>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + return vcpu_sys_reg(vcpu, reg);
>>> +}
>>> +
>>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
>>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
>>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
>>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
>>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
>>> + case FAR_EL1: write_sysreg_el1(val, far); return;
>>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
>>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
>>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
>>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
>>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
>>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
>>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + vcpu_sys_reg(vcpu, reg) = val;
>>> +}
>>> +
>>> /*
>>> * Generic accessor for VM registers. Only called as long as HCR_TVM
>>> * is set. If the guest enables the MMU, we stop trapping the VM
>>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
>>> if (!p->is_aarch32 || !p->is_32bit) {
>>> val = p->regval;
>>> } else {
>>> - val = vcpu_sys_reg(vcpu, reg);
>>> + val = read_deferrable_vm_reg(vcpu, reg);
>>> if (r->reg % 2)
>>> val = (p->regval << 32) | (u64)lower_32_bits(val);
>>> else
>>> val = ((u64)upper_32_bits(val) << 32) |
>>> (u64)lower_32_bits(p->regval);
>>> }
>>> - vcpu_sys_reg(vcpu, reg) = val;
>>> + write_deferrable_vm_reg(vcpu, reg, val);
>>>
>>> kvm_toggle_cache(vcpu, was_enabled);
>>> return true;
>>>
>>
>> I'm slightly uneasy with this. It means that the rest of the KVM code
>> has to know whether a given register is deferrable or not (or face the
>> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
>> magic in the vcpu_sys_reg() accessors.
>>
>> Thoughts?
>>
>
> Yes, this is the main reservation I also have with the series.
>
> I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> that.
>
> That does mean that we need a giant switch statement which knows how to
> read any deferrable EL1 (and EL0) system register from hardware, and
> still BUG/WARN if someone adds a system register but forgets to add that
> handler and test on VHE. Unless there's some fantastic auto-gen
> mechanism that can take a hash define and figure out which sysreg
> instruction to use - I couldn't think of that.
>
> I'm happy to go back to that approach, but I didn't find it that much
> nicer either.
>
> How about I send you the small handful of patches that implement the
> alternative approach and you have a look at that?
Sure, feel free to post them. I wonder if we can take a similar approach
to the hack I used for the CP15 stuff on 32bit, where read/write_sysreg
are automagically turned into the right type of accessor...
I'll have a try.
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 13:20 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 13:20 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> Some system registers do not affect the host kernel's execution and can
> therefore be loaded when we are about to run a VCPU and we don't have to
> restore the host state to the hardware before the time when we are
> actually about to return to userspace or schedule out the VCPU thread.
>
> The EL1 system registers and the userspace state registers, which only
> affect EL0 execution, do not affect the host kernel's execution.
>
> The 32-bit system registers are not used by a VHE host kernel and
> therefore don't need to be saved/restored on every entry/exit to/from
> the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
Note that they are not used by the !VHE host kernel either, and I
believe they could be deferred too, although that would imply a round
trip to HYP to save/restore them. We already have such a hook there when
configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
What do you think?
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
@ 2017-12-11 13:20 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 13:20 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> Some system registers do not affect the host kernel's execution and can
> therefore be loaded when we are about to run a VCPU and we don't have to
> restore the host state to the hardware before the time when we are
> actually about to return to userspace or schedule out the VCPU thread.
>
> The EL1 system registers and the userspace state registers, which only
> affect EL0 execution, do not affect the host kernel's execution.
>
> The 32-bit system registers are not used by a VHE host kernel and
> therefore don't need to be saved/restored on every entry/exit to/from
> the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
Note that they are not used by the !VHE host kernel either, and I
believe they could be deferred too, although that would imply a round
trip to HYP to save/restore them. We already have such a hook there when
configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
What do you think?
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 27/36] KVM: arm64: Move common VHE/non-VHE trap config in separate functions
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 13:53 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 13:53 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to be more lazy with some of the trap configuration
> register read/writes for VHE systems, move the logic that is currently
> shared between VHE and non-VHE into a separate function which can be
> called from either the world-switch path or from vcpu_load/vcpu_put.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 27/36] KVM: arm64: Move common VHE/non-VHE trap config in separate functions
@ 2017-12-11 13:53 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 13:53 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> As we are about to be more lazy with some of the trap configuration
> register read/writes for VHE systems, move the logic that is currently
> shared between VHE and non-VHE into a separate function which can be
> called from either the world-switch path or from vcpu_load/vcpu_put.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 28/36] KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
2017-12-07 17:06 ` Christoffer Dall
@ 2017-12-11 14:18 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 14:18 UTC (permalink / raw)
To: Christoffer Dall, kvmarm, linux-arm-kernel; +Cc: kvm, Shih-Wei Li, Andrew Jones
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to enable/disable traps to FP registers on every switch
> to/from the VM, because the host kernel does not use this resource
> without calling vcpu_put. We can therefore move things around enough
> that we still always write FPEXC32_EL2 before programming CPTR_EL2 but
> only program these during vcpu load/put.
Same as the 32bit registers. I don't see the point in limiting this to
be VHE only. Or am I missing something?
Thanks,
M.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/include/asm/kvm_hyp.h | 3 +++
> arch/arm64/kvm/hyp/switch.c | 34 ++++++++++++++++++++++++----------
> arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++++
> 3 files changed, 31 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
> index 3f54c55f77a1..28d5f3cb4001 100644
> --- a/arch/arm64/include/asm/kvm_hyp.h
> +++ b/arch/arm64/include/asm/kvm_hyp.h
> @@ -148,6 +148,9 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
> void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
> bool __fpsimd_enabled(void);
>
> +void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
> +void deactivate_traps_vhe_put(void);
> +
> u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
> void __noreturn __hyp_do_panic(unsigned long, ...);
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index c01bcfc3fb52..44aae69a7fec 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -24,22 +24,25 @@
> #include <asm/fpsimd.h>
> #include <asm/debug-monitors.h>
>
> -static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
> +static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
> {
> /*
> - * We are about to set CPTR_EL2.TFP to trap all floating point
> - * register accesses to EL2, however, the ARM ARM clearly states that
> - * traps are only taken to EL2 if the operation would not otherwise
> - * trap to EL1. Therefore, always make sure that for 32-bit guests,
> - * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
> - * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
> - * it will cause an exception.
> + * We are about to trap all floating point register accesses to EL2,
> + * however, traps are only taken to EL2 if the operation would not
> + * otherwise trap to EL1. Therefore, always make sure that for 32-bit
> + * guests, we set FPEXC.EN to prevent traps to EL1, when setting the
> + * TFP bit. If FP/ASIMD is not implemented, FPEXC is UNDEFINED and
> + * any access to it will cause an exception.
> */
> if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> !vcpu->arch.guest_vfp_loaded) {
> write_sysreg(1 << 30, fpexc32_el2);
> isb();
> }
> +}
> +
> +static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
> +{
> write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
>
> /* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
> @@ -61,10 +64,12 @@ static void __hyp_text __deactivate_traps_common(void)
> write_sysreg(0, pmuserenr_el0);
> }
>
> -static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> +void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> + __activate_traps_fpsimd32(vcpu);
> +
> val = read_sysreg(cpacr_el1);
> val |= CPACR_EL1_TTA;
> val &= ~CPACR_EL1_ZEN;
> @@ -73,7 +78,15 @@ static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> else
> val &= ~CPACR_EL1_FPEN;
> write_sysreg(val, cpacr_el1);
> +}
>
> +void deactivate_traps_vhe_put(void)
> +{
> + write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
> +}
> +
> +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> +{
> write_sysreg(__kvm_hyp_vector, vbar_el1);
> }
>
> @@ -81,6 +94,8 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> + __activate_traps_fpsimd32(vcpu);
> +
> val = CPTR_EL2_DEFAULT;
> val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> if (vcpu->arch.guest_vfp_loaded)
> @@ -111,7 +126,6 @@ static void __hyp_text __deactivate_traps_vhe(void)
>
> write_sysreg(mdcr_el2, mdcr_el2);
> write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
> - write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
> write_sysreg(vectors, vbar_el1);
> }
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index 65abf1aeba59..b647fea93fdc 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -254,6 +254,8 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> __sysreg_restore_el1_state(guest_ctxt);
>
> vcpu->arch.sysregs_loaded_on_cpu = true;
> +
> + activate_traps_vhe_load(vcpu);
> }
>
> /**
> @@ -286,6 +288,8 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> if (!has_vhe())
> return;
>
> + deactivate_traps_vhe_put();
> +
> __sysreg_save_el1_state(guest_ctxt);
> __sysreg_save_user_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
>
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 28/36] KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE
@ 2017-12-11 14:18 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 14:18 UTC (permalink / raw)
To: linux-arm-kernel
On 07/12/17 17:06, Christoffer Dall wrote:
> There is no need to enable/disable traps to FP registers on every switch
> to/from the VM, because the host kernel does not use this resource
> without calling vcpu_put. We can therefore move things around enough
> that we still always write FPEXC32_EL2 before programming CPTR_EL2 but
> only program these during vcpu load/put.
Same as the 32bit registers. I don't see the point in limiting this to
be VHE only. Or am I missing something?
Thanks,
M.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> ---
> arch/arm64/include/asm/kvm_hyp.h | 3 +++
> arch/arm64/kvm/hyp/switch.c | 34 ++++++++++++++++++++++++----------
> arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++++
> 3 files changed, 31 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
> index 3f54c55f77a1..28d5f3cb4001 100644
> --- a/arch/arm64/include/asm/kvm_hyp.h
> +++ b/arch/arm64/include/asm/kvm_hyp.h
> @@ -148,6 +148,9 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
> void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
> bool __fpsimd_enabled(void);
>
> +void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
> +void deactivate_traps_vhe_put(void);
> +
> u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
> void __noreturn __hyp_do_panic(unsigned long, ...);
>
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index c01bcfc3fb52..44aae69a7fec 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -24,22 +24,25 @@
> #include <asm/fpsimd.h>
> #include <asm/debug-monitors.h>
>
> -static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
> +static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
> {
> /*
> - * We are about to set CPTR_EL2.TFP to trap all floating point
> - * register accesses to EL2, however, the ARM ARM clearly states that
> - * traps are only taken to EL2 if the operation would not otherwise
> - * trap to EL1. Therefore, always make sure that for 32-bit guests,
> - * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit.
> - * If FP/ASIMD is not implemented, FPEXC is UNDEFINED and any access to
> - * it will cause an exception.
> + * We are about to trap all floating point register accesses to EL2,
> + * however, traps are only taken to EL2 if the operation would not
> + * otherwise trap to EL1. Therefore, always make sure that for 32-bit
> + * guests, we set FPEXC.EN to prevent traps to EL1, when setting the
> + * TFP bit. If FP/ASIMD is not implemented, FPEXC is UNDEFINED and
> + * any access to it will cause an exception.
> */
> if (vcpu_el1_is_32bit(vcpu) && system_supports_fpsimd() &&
> !vcpu->arch.guest_vfp_loaded) {
> write_sysreg(1 << 30, fpexc32_el2);
> isb();
> }
> +}
> +
> +static void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)
> +{
> write_sysreg(vcpu->arch.hcr_el2, hcr_el2);
>
> /* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
> @@ -61,10 +64,12 @@ static void __hyp_text __deactivate_traps_common(void)
> write_sysreg(0, pmuserenr_el0);
> }
>
> -static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> +void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> + __activate_traps_fpsimd32(vcpu);
> +
> val = read_sysreg(cpacr_el1);
> val |= CPACR_EL1_TTA;
> val &= ~CPACR_EL1_ZEN;
> @@ -73,7 +78,15 @@ static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> else
> val &= ~CPACR_EL1_FPEN;
> write_sysreg(val, cpacr_el1);
> +}
>
> +void deactivate_traps_vhe_put(void)
> +{
> + write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
> +}
> +
> +static void __hyp_text __activate_traps_vhe(struct kvm_vcpu *vcpu)
> +{
> write_sysreg(__kvm_hyp_vector, vbar_el1);
> }
>
> @@ -81,6 +94,8 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu)
> {
> u64 val;
>
> + __activate_traps_fpsimd32(vcpu);
> +
> val = CPTR_EL2_DEFAULT;
> val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
> if (vcpu->arch.guest_vfp_loaded)
> @@ -111,7 +126,6 @@ static void __hyp_text __deactivate_traps_vhe(void)
>
> write_sysreg(mdcr_el2, mdcr_el2);
> write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
> - write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
> write_sysreg(vectors, vbar_el1);
> }
>
> diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> index 65abf1aeba59..b647fea93fdc 100644
> --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> @@ -254,6 +254,8 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
> __sysreg_restore_el1_state(guest_ctxt);
>
> vcpu->arch.sysregs_loaded_on_cpu = true;
> +
> + activate_traps_vhe_load(vcpu);
> }
>
> /**
> @@ -286,6 +288,8 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
> if (!has_vhe())
> return;
>
> + deactivate_traps_vhe_put();
> +
> __sysreg_save_el1_state(guest_ctxt);
> __sysreg_save_user_state(guest_ctxt);
> __sysreg32_save_state(vcpu);
>
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
2017-12-07 17:05 ` Christoffer Dall
@ 2017-12-11 14:43 ` Yury Norov
-1 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 14:43 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvm, Marc Zyngier, linux-arm-kernel, kvmarm, Shih-Wei Li
Hi Christoffer,
On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> This series redesigns parts of KVM/ARM to optimize the performance on
> VHE systems. The general approach is to try to do as little work as
> possible when transitioning between the VM and the hypervisor. This has
> the benefit of lower latency when waiting for interrupts and delivering
> virtual interrupts, and reduces the overhead of emulating behavior and
> I/O in the host kernel.
>
> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> that can be generally improved. We then add infrastructure to move more
> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> registers.
>
> We then introduce a new world-switch function for VHE systems, which we
> can tweak and optimize for VHE systems. To do that, we rework a lot of
> the system register save/restore handling and emulation code that may
> need access to system registers, so that we can defer as many system
> register save/restore operations to vcpu_load and vcpu_put, and move
> this logic out of the VHE world switch function.
>
> We then optimize the configuration of traps. On non-VHE systems, both
> the host and VM kernels run in EL1, but because the host kernel should
> have full access to the underlying hardware, but the VM kernel should
> not, we essentially make the host kernel more privileged than the VM
> kernel despite them both running at the same privilege level by enabling
> VE traps when entering the VM and disabling those traps when exiting the
> VM. On VHE systems, the host kernel runs in EL2 and has full access to
> the hardware (as much as allowed by secure side software), and is
> unaffected by the trap configuration. That means we can configure the
> traps for VMs running in EL1 once, and don't have to switch them on and
> off for every entry/exit to/from the VM.
>
> Finally, we improve our VGIC handling by moving all save/restore logic
> out of the VHE world-switch, and we make it possible to truly only
> evaluate if the AP list is empty and not do *any* VGIC work if that is
> the case, and only do the minimal amount of work required in the course
> of the VGIC processing when we have virtual interrupts in flight.
>
> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> [1], the level-triggered mapped interrupts support series [2], and the
> first five patches of James' SDEI series [3], a single SVE patch that
> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> my vcpu load/put series [4].
>
> I've given the patches a fair amount of testing on Thunder-X, Mustang,
> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> functionality on the Foundation model, running both 64-bit VMs and
> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> GICv2-on-GICv3.
>
> The patches are also available in the vhe-optimize-v2 branch on my
> kernel.org repository [5].
>
> Changes since v1:
> - Rebased on v4.15-rc1 and newer versions of other dependencies,
> including the vcpu load/put approach taken for KVM.
> - Addressed review comments from v1 (detailed changelogs are in the
> individual patches).
>
> Thanks,
> -Christoffer
>
> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
I just submitted the benchmark I used to test your v1 and v2 series':
https://lkml.org/lkml/2017/12/11/364
On ThunderX2, 112 online CPUs test results for v1 are like this:
Host, v4.14:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 81 110
Broadcast IPI: 0 2106
Guest, v4.14:
Dry-run: 0 1
Self-IPI: 10 18
Normal IPI: 305 525
Broadcast IPI: 0 9729
Guest, v4.14 + VHE:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 176 343
Broadcast IPI: 0 9885
And for v2.
Host, v4.15:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 79 108
Broadcast IPI: 0 2102
Guest, v4.15-rc:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 291 526
Broadcast IPI: 0 10439
Guest, v4.15-rc + VHE:
Dry-run: 0 2
Self-IPI: 14 28
Normal IPI: 370 569
Broadcast IPI: 0 11688
All times are normalized to v1 host dry-run time. Smaller - better.
Results for v1 and v2 may vary because kernel version is changed.
What makes us worry is slowing down the "Normal IPI" test observed in
v2 series.
Nevertheless, if you find test relevant, for v1 and v2,
Tested-by: Yury Norov <ynorov@caviumnetworks.com>
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-11 14:43 ` Yury Norov
0 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 14:43 UTC (permalink / raw)
To: linux-arm-kernel
Hi Christoffer,
On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> This series redesigns parts of KVM/ARM to optimize the performance on
> VHE systems. The general approach is to try to do as little work as
> possible when transitioning between the VM and the hypervisor. This has
> the benefit of lower latency when waiting for interrupts and delivering
> virtual interrupts, and reduces the overhead of emulating behavior and
> I/O in the host kernel.
>
> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> that can be generally improved. We then add infrastructure to move more
> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> registers.
>
> We then introduce a new world-switch function for VHE systems, which we
> can tweak and optimize for VHE systems. To do that, we rework a lot of
> the system register save/restore handling and emulation code that may
> need access to system registers, so that we can defer as many system
> register save/restore operations to vcpu_load and vcpu_put, and move
> this logic out of the VHE world switch function.
>
> We then optimize the configuration of traps. On non-VHE systems, both
> the host and VM kernels run in EL1, but because the host kernel should
> have full access to the underlying hardware, but the VM kernel should
> not, we essentially make the host kernel more privileged than the VM
> kernel despite them both running at the same privilege level by enabling
> VE traps when entering the VM and disabling those traps when exiting the
> VM. On VHE systems, the host kernel runs in EL2 and has full access to
> the hardware (as much as allowed by secure side software), and is
> unaffected by the trap configuration. That means we can configure the
> traps for VMs running in EL1 once, and don't have to switch them on and
> off for every entry/exit to/from the VM.
>
> Finally, we improve our VGIC handling by moving all save/restore logic
> out of the VHE world-switch, and we make it possible to truly only
> evaluate if the AP list is empty and not do *any* VGIC work if that is
> the case, and only do the minimal amount of work required in the course
> of the VGIC processing when we have virtual interrupts in flight.
>
> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> [1], the level-triggered mapped interrupts support series [2], and the
> first five patches of James' SDEI series [3], a single SVE patch that
> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> my vcpu load/put series [4].
>
> I've given the patches a fair amount of testing on Thunder-X, Mustang,
> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> functionality on the Foundation model, running both 64-bit VMs and
> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> GICv2-on-GICv3.
>
> The patches are also available in the vhe-optimize-v2 branch on my
> kernel.org repository [5].
>
> Changes since v1:
> - Rebased on v4.15-rc1 and newer versions of other dependencies,
> including the vcpu load/put approach taken for KVM.
> - Addressed review comments from v1 (detailed changelogs are in the
> individual patches).
>
> Thanks,
> -Christoffer
>
> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
I just submitted the benchmark I used to test your v1 and v2 series':
https://lkml.org/lkml/2017/12/11/364
On ThunderX2, 112 online CPUs test results for v1 are like this:
Host, v4.14:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 81 110
Broadcast IPI: 0 2106
Guest, v4.14:
Dry-run: 0 1
Self-IPI: 10 18
Normal IPI: 305 525
Broadcast IPI: 0 9729
Guest, v4.14 + VHE:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 176 343
Broadcast IPI: 0 9885
And for v2.
Host, v4.15:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 79 108
Broadcast IPI: 0 2102
Guest, v4.15-rc:
Dry-run: 0 1
Self-IPI: 9 18
Normal IPI: 291 526
Broadcast IPI: 0 10439
Guest, v4.15-rc + VHE:
Dry-run: 0 2
Self-IPI: 14 28
Normal IPI: 370 569
Broadcast IPI: 0 11688
All times are normalized to v1 host dry-run time. Smaller - better.
Results for v1 and v2 may vary because kernel version is changed.
What makes us worry is slowing down the "Normal IPI" test observed in
v2 series.
Nevertheless, if you find test relevant, for v1 and v2,
Tested-by: Yury Norov <ynorov@caviumnetworks.com>
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
2017-12-11 14:43 ` Yury Norov
@ 2017-12-11 14:56 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 14:56 UTC (permalink / raw)
To: Yury Norov, Christoffer Dall
Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On 11/12/17 14:43, Yury Norov wrote:
> Hi Christoffer,
>
> On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
>> This series redesigns parts of KVM/ARM to optimize the performance on
>> VHE systems. The general approach is to try to do as little work as
>> possible when transitioning between the VM and the hypervisor. This has
>> the benefit of lower latency when waiting for interrupts and delivering
>> virtual interrupts, and reduces the overhead of emulating behavior and
>> I/O in the host kernel.
>>
>> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
>> that can be generally improved. We then add infrastructure to move more
>> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
>> registers.
>>
>> We then introduce a new world-switch function for VHE systems, which we
>> can tweak and optimize for VHE systems. To do that, we rework a lot of
>> the system register save/restore handling and emulation code that may
>> need access to system registers, so that we can defer as many system
>> register save/restore operations to vcpu_load and vcpu_put, and move
>> this logic out of the VHE world switch function.
>>
>> We then optimize the configuration of traps. On non-VHE systems, both
>> the host and VM kernels run in EL1, but because the host kernel should
>> have full access to the underlying hardware, but the VM kernel should
>> not, we essentially make the host kernel more privileged than the VM
>> kernel despite them both running at the same privilege level by enabling
>> VE traps when entering the VM and disabling those traps when exiting the
>> VM. On VHE systems, the host kernel runs in EL2 and has full access to
>> the hardware (as much as allowed by secure side software), and is
>> unaffected by the trap configuration. That means we can configure the
>> traps for VMs running in EL1 once, and don't have to switch them on and
>> off for every entry/exit to/from the VM.
>>
>> Finally, we improve our VGIC handling by moving all save/restore logic
>> out of the VHE world-switch, and we make it possible to truly only
>> evaluate if the AP list is empty and not do *any* VGIC work if that is
>> the case, and only do the minimal amount of work required in the course
>> of the VGIC processing when we have virtual interrupts in flight.
>>
>> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
>> [1], the level-triggered mapped interrupts support series [2], and the
>> first five patches of James' SDEI series [3], a single SVE patch that
>> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
>> my vcpu load/put series [4].
>>
>> I've given the patches a fair amount of testing on Thunder-X, Mustang,
>> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
>> functionality on the Foundation model, running both 64-bit VMs and
>> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
>> GICv2-on-GICv3.
>>
>> The patches are also available in the vhe-optimize-v2 branch on my
>> kernel.org repository [5].
>>
>> Changes since v1:
>> - Rebased on v4.15-rc1 and newer versions of other dependencies,
>> including the vcpu load/put approach taken for KVM.
>> - Addressed review comments from v1 (detailed changelogs are in the
>> individual patches).
>>
>> Thanks,
>> -Christoffer
>>
>> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
>> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
>> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
>> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
>> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
>
> I just submitted the benchmark I used to test your v1 and v2 series':
> https://lkml.org/lkml/2017/12/11/364
>
> On ThunderX2, 112 online CPUs test results for v1 are like this:
>
> Host, v4.14:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 81 110
> Broadcast IPI: 0 2106
>
> Guest, v4.14:
> Dry-run: 0 1
> Self-IPI: 10 18
> Normal IPI: 305 525
> Broadcast IPI: 0 9729
>
> Guest, v4.14 + VHE:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 176 343
> Broadcast IPI: 0 9885
>
> And for v2.
>
> Host, v4.15:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 79 108
> Broadcast IPI: 0 2102
>
> Guest, v4.15-rc:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 291 526
> Broadcast IPI: 0 10439
>
> Guest, v4.15-rc + VHE:
> Dry-run: 0 2
> Self-IPI: 14 28
> Normal IPI: 370 569
> Broadcast IPI: 0 11688
>
> All times are normalized to v1 host dry-run time. Smaller - better.
>
> Results for v1 and v2 may vary because kernel version is changed.
> What makes us worry is slowing down the "Normal IPI" test observed in
> v2 series.
It'd be interesting if you could profile your system to find our where
you're spending time. My own tests, with a different benchmark, did show
a 40% reduction in the number of *cycles*.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-11 14:56 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-11 14:56 UTC (permalink / raw)
To: linux-arm-kernel
On 11/12/17 14:43, Yury Norov wrote:
> Hi Christoffer,
>
> On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
>> This series redesigns parts of KVM/ARM to optimize the performance on
>> VHE systems. The general approach is to try to do as little work as
>> possible when transitioning between the VM and the hypervisor. This has
>> the benefit of lower latency when waiting for interrupts and delivering
>> virtual interrupts, and reduces the overhead of emulating behavior and
>> I/O in the host kernel.
>>
>> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
>> that can be generally improved. We then add infrastructure to move more
>> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
>> registers.
>>
>> We then introduce a new world-switch function for VHE systems, which we
>> can tweak and optimize for VHE systems. To do that, we rework a lot of
>> the system register save/restore handling and emulation code that may
>> need access to system registers, so that we can defer as many system
>> register save/restore operations to vcpu_load and vcpu_put, and move
>> this logic out of the VHE world switch function.
>>
>> We then optimize the configuration of traps. On non-VHE systems, both
>> the host and VM kernels run in EL1, but because the host kernel should
>> have full access to the underlying hardware, but the VM kernel should
>> not, we essentially make the host kernel more privileged than the VM
>> kernel despite them both running at the same privilege level by enabling
>> VE traps when entering the VM and disabling those traps when exiting the
>> VM. On VHE systems, the host kernel runs in EL2 and has full access to
>> the hardware (as much as allowed by secure side software), and is
>> unaffected by the trap configuration. That means we can configure the
>> traps for VMs running in EL1 once, and don't have to switch them on and
>> off for every entry/exit to/from the VM.
>>
>> Finally, we improve our VGIC handling by moving all save/restore logic
>> out of the VHE world-switch, and we make it possible to truly only
>> evaluate if the AP list is empty and not do *any* VGIC work if that is
>> the case, and only do the minimal amount of work required in the course
>> of the VGIC processing when we have virtual interrupts in flight.
>>
>> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
>> [1], the level-triggered mapped interrupts support series [2], and the
>> first five patches of James' SDEI series [3], a single SVE patch that
>> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
>> my vcpu load/put series [4].
>>
>> I've given the patches a fair amount of testing on Thunder-X, Mustang,
>> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
>> functionality on the Foundation model, running both 64-bit VMs and
>> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
>> GICv2-on-GICv3.
>>
>> The patches are also available in the vhe-optimize-v2 branch on my
>> kernel.org repository [5].
>>
>> Changes since v1:
>> - Rebased on v4.15-rc1 and newer versions of other dependencies,
>> including the vcpu load/put approach taken for KVM.
>> - Addressed review comments from v1 (detailed changelogs are in the
>> individual patches).
>>
>> Thanks,
>> -Christoffer
>>
>> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
>> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
>> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
>> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
>> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
>
> I just submitted the benchmark I used to test your v1 and v2 series':
> https://lkml.org/lkml/2017/12/11/364
>
> On ThunderX2, 112 online CPUs test results for v1 are like this:
>
> Host, v4.14:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 81 110
> Broadcast IPI: 0 2106
>
> Guest, v4.14:
> Dry-run: 0 1
> Self-IPI: 10 18
> Normal IPI: 305 525
> Broadcast IPI: 0 9729
>
> Guest, v4.14 + VHE:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 176 343
> Broadcast IPI: 0 9885
>
> And for v2.
>
> Host, v4.15:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 79 108
> Broadcast IPI: 0 2102
>
> Guest, v4.15-rc:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 291 526
> Broadcast IPI: 0 10439
>
> Guest, v4.15-rc + VHE:
> Dry-run: 0 2
> Self-IPI: 14 28
> Normal IPI: 370 569
> Broadcast IPI: 0 11688
>
> All times are normalized to v1 host dry-run time. Smaller - better.
>
> Results for v1 and v2 may vary because kernel version is changed.
> What makes us worry is slowing down the "Normal IPI" test observed in
> v2 series.
It'd be interesting if you could profile your system to find our where
you're spending time. My own tests, with a different benchmark, did show
a 40% reduction in the number of *cycles*.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
2017-12-11 14:56 ` Marc Zyngier
@ 2017-12-11 15:14 ` Yury Norov
-1 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 15:14 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvm, Shih-Wei Li, kvmarm, linux-arm-kernel
On Mon, Dec 11, 2017 at 02:56:01PM +0000, Marc Zyngier wrote:
> On 11/12/17 14:43, Yury Norov wrote:
> > Hi Christoffer,
> >
> > On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> >> This series redesigns parts of KVM/ARM to optimize the performance on
> >> VHE systems. The general approach is to try to do as little work as
> >> possible when transitioning between the VM and the hypervisor. This has
> >> the benefit of lower latency when waiting for interrupts and delivering
> >> virtual interrupts, and reduces the overhead of emulating behavior and
> >> I/O in the host kernel.
> >>
> >> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> >> that can be generally improved. We then add infrastructure to move more
> >> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> >> registers.
> >>
> >> We then introduce a new world-switch function for VHE systems, which we
> >> can tweak and optimize for VHE systems. To do that, we rework a lot of
> >> the system register save/restore handling and emulation code that may
> >> need access to system registers, so that we can defer as many system
> >> register save/restore operations to vcpu_load and vcpu_put, and move
> >> this logic out of the VHE world switch function.
> >>
> >> We then optimize the configuration of traps. On non-VHE systems, both
> >> the host and VM kernels run in EL1, but because the host kernel should
> >> have full access to the underlying hardware, but the VM kernel should
> >> not, we essentially make the host kernel more privileged than the VM
> >> kernel despite them both running at the same privilege level by enabling
> >> VE traps when entering the VM and disabling those traps when exiting the
> >> VM. On VHE systems, the host kernel runs in EL2 and has full access to
> >> the hardware (as much as allowed by secure side software), and is
> >> unaffected by the trap configuration. That means we can configure the
> >> traps for VMs running in EL1 once, and don't have to switch them on and
> >> off for every entry/exit to/from the VM.
> >>
> >> Finally, we improve our VGIC handling by moving all save/restore logic
> >> out of the VHE world-switch, and we make it possible to truly only
> >> evaluate if the AP list is empty and not do *any* VGIC work if that is
> >> the case, and only do the minimal amount of work required in the course
> >> of the VGIC processing when we have virtual interrupts in flight.
> >>
> >> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> >> [1], the level-triggered mapped interrupts support series [2], and the
> >> first five patches of James' SDEI series [3], a single SVE patch that
> >> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> >> my vcpu load/put series [4].
> >>
> >> I've given the patches a fair amount of testing on Thunder-X, Mustang,
> >> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> >> functionality on the Foundation model, running both 64-bit VMs and
> >> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> >> GICv2-on-GICv3.
> >>
> >> The patches are also available in the vhe-optimize-v2 branch on my
> >> kernel.org repository [5].
> >>
> >> Changes since v1:
> >> - Rebased on v4.15-rc1 and newer versions of other dependencies,
> >> including the vcpu load/put approach taken for KVM.
> >> - Addressed review comments from v1 (detailed changelogs are in the
> >> individual patches).
> >>
> >> Thanks,
> >> -Christoffer
> >>
> >> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> >> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> >> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> >> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> >> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
> >
> > I just submitted the benchmark I used to test your v1 and v2 series':
> > https://lkml.org/lkml/2017/12/11/364
> >
> > On ThunderX2, 112 online CPUs test results for v1 are like this:
> >
> > Host, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 81 110
> > Broadcast IPI: 0 2106
> >
> > Guest, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 10 18
> > Normal IPI: 305 525
> > Broadcast IPI: 0 9729
> >
> > Guest, v4.14 + VHE:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 176 343
> > Broadcast IPI: 0 9885
> >
> > And for v2.
> >
> > Host, v4.15:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 79 108
> > Broadcast IPI: 0 2102
> >
> > Guest, v4.15-rc:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 291 526
> > Broadcast IPI: 0 10439
> >
> > Guest, v4.15-rc + VHE:
> > Dry-run: 0 2
> > Self-IPI: 14 28
> > Normal IPI: 370 569
> > Broadcast IPI: 0 11688
> >
> > All times are normalized to v1 host dry-run time. Smaller - better.
> >
> > Results for v1 and v2 may vary because kernel version is changed.
> > What makes us worry is slowing down the "Normal IPI" test observed in
> > v2 series.
> It'd be interesting if you could profile your system to find our where
> you're spending time. My own tests, with a different benchmark, did show
> a 40% reduction in the number of *cycles*.
40% reduction is what I also observed for v1, to be specific - 42%.
So I was surprised when found v2 slower than vanilla kernel. Did you
observe 40% reduction for v2 or v1, or both?
I also think to switch to *cycles* as it (doubtly) might be CPU
frequency scaling issue, and do some profiling.
Yury
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-11 15:14 ` Yury Norov
0 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 15:14 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 02:56:01PM +0000, Marc Zyngier wrote:
> On 11/12/17 14:43, Yury Norov wrote:
> > Hi Christoffer,
> >
> > On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> >> This series redesigns parts of KVM/ARM to optimize the performance on
> >> VHE systems. The general approach is to try to do as little work as
> >> possible when transitioning between the VM and the hypervisor. This has
> >> the benefit of lower latency when waiting for interrupts and delivering
> >> virtual interrupts, and reduces the overhead of emulating behavior and
> >> I/O in the host kernel.
> >>
> >> Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> >> that can be generally improved. We then add infrastructure to move more
> >> logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> >> registers.
> >>
> >> We then introduce a new world-switch function for VHE systems, which we
> >> can tweak and optimize for VHE systems. To do that, we rework a lot of
> >> the system register save/restore handling and emulation code that may
> >> need access to system registers, so that we can defer as many system
> >> register save/restore operations to vcpu_load and vcpu_put, and move
> >> this logic out of the VHE world switch function.
> >>
> >> We then optimize the configuration of traps. On non-VHE systems, both
> >> the host and VM kernels run in EL1, but because the host kernel should
> >> have full access to the underlying hardware, but the VM kernel should
> >> not, we essentially make the host kernel more privileged than the VM
> >> kernel despite them both running at the same privilege level by enabling
> >> VE traps when entering the VM and disabling those traps when exiting the
> >> VM. On VHE systems, the host kernel runs in EL2 and has full access to
> >> the hardware (as much as allowed by secure side software), and is
> >> unaffected by the trap configuration. That means we can configure the
> >> traps for VMs running in EL1 once, and don't have to switch them on and
> >> off for every entry/exit to/from the VM.
> >>
> >> Finally, we improve our VGIC handling by moving all save/restore logic
> >> out of the VHE world-switch, and we make it possible to truly only
> >> evaluate if the AP list is empty and not do *any* VGIC work if that is
> >> the case, and only do the minimal amount of work required in the course
> >> of the VGIC processing when we have virtual interrupts in flight.
> >>
> >> The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> >> [1], the level-triggered mapped interrupts support series [2], and the
> >> first five patches of James' SDEI series [3], a single SVE patch that
> >> moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> >> my vcpu load/put series [4].
> >>
> >> I've given the patches a fair amount of testing on Thunder-X, Mustang,
> >> Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> >> functionality on the Foundation model, running both 64-bit VMs and
> >> 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> >> GICv2-on-GICv3.
> >>
> >> The patches are also available in the vhe-optimize-v2 branch on my
> >> kernel.org repository [5].
> >>
> >> Changes since v1:
> >> - Rebased on v4.15-rc1 and newer versions of other dependencies,
> >> including the vcpu load/put approach taken for KVM.
> >> - Addressed review comments from v1 (detailed changelogs are in the
> >> individual patches).
> >>
> >> Thanks,
> >> -Christoffer
> >>
> >> [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> >> [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> >> [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> >> [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> >> [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
> >
> > I just submitted the benchmark I used to test your v1 and v2 series':
> > https://lkml.org/lkml/2017/12/11/364
> >
> > On ThunderX2, 112 online CPUs test results for v1 are like this:
> >
> > Host, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 81 110
> > Broadcast IPI: 0 2106
> >
> > Guest, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 10 18
> > Normal IPI: 305 525
> > Broadcast IPI: 0 9729
> >
> > Guest, v4.14 + VHE:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 176 343
> > Broadcast IPI: 0 9885
> >
> > And for v2.
> >
> > Host, v4.15:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 79 108
> > Broadcast IPI: 0 2102
> >
> > Guest, v4.15-rc:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 291 526
> > Broadcast IPI: 0 10439
> >
> > Guest, v4.15-rc + VHE:
> > Dry-run: 0 2
> > Self-IPI: 14 28
> > Normal IPI: 370 569
> > Broadcast IPI: 0 11688
> >
> > All times are normalized to v1 host dry-run time. Smaller - better.
> >
> > Results for v1 and v2 may vary because kernel version is changed.
> > What makes us worry is slowing down the "Normal IPI" test observed in
> > v2 series.
> It'd be interesting if you could profile your system to find our where
> you're spending time. My own tests, with a different benchmark, did show
> a 40% reduction in the number of *cycles*.
40% reduction is what I also observed for v1, to be specific - 42%.
So I was surprised when found v2 slower than vanilla kernel. Did you
observe 40% reduction for v2 or v1, or both?
I also think to switch to *cycles* as it (doubtly) might be CPU
frequency scaling issue, and do some profiling.
Yury
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
2017-12-11 14:43 ` Yury Norov
@ 2017-12-11 15:34 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 15:34 UTC (permalink / raw)
To: Yury Norov
Cc: kvmarm, linux-arm-kernel, kvm, Marc Zyngier, Shih-Wei Li, Andrew Jones
Hi Yury,
On Mon, Dec 11, 2017 at 05:43:23PM +0300, Yury Norov wrote:
>
> On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> > This series redesigns parts of KVM/ARM to optimize the performance on
> > VHE systems. The general approach is to try to do as little work as
> > possible when transitioning between the VM and the hypervisor. This has
> > the benefit of lower latency when waiting for interrupts and delivering
> > virtual interrupts, and reduces the overhead of emulating behavior and
> > I/O in the host kernel.
> >
> > Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> > that can be generally improved. We then add infrastructure to move more
> > logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> > registers.
> >
> > We then introduce a new world-switch function for VHE systems, which we
> > can tweak and optimize for VHE systems. To do that, we rework a lot of
> > the system register save/restore handling and emulation code that may
> > need access to system registers, so that we can defer as many system
> > register save/restore operations to vcpu_load and vcpu_put, and move
> > this logic out of the VHE world switch function.
> >
> > We then optimize the configuration of traps. On non-VHE systems, both
> > the host and VM kernels run in EL1, but because the host kernel should
> > have full access to the underlying hardware, but the VM kernel should
> > not, we essentially make the host kernel more privileged than the VM
> > kernel despite them both running at the same privilege level by enabling
> > VE traps when entering the VM and disabling those traps when exiting the
> > VM. On VHE systems, the host kernel runs in EL2 and has full access to
> > the hardware (as much as allowed by secure side software), and is
> > unaffected by the trap configuration. That means we can configure the
> > traps for VMs running in EL1 once, and don't have to switch them on and
> > off for every entry/exit to/from the VM.
> >
> > Finally, we improve our VGIC handling by moving all save/restore logic
> > out of the VHE world-switch, and we make it possible to truly only
> > evaluate if the AP list is empty and not do *any* VGIC work if that is
> > the case, and only do the minimal amount of work required in the course
> > of the VGIC processing when we have virtual interrupts in flight.
> >
> > The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> > [1], the level-triggered mapped interrupts support series [2], and the
> > first five patches of James' SDEI series [3], a single SVE patch that
> > moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> > my vcpu load/put series [4].
> >
> > I've given the patches a fair amount of testing on Thunder-X, Mustang,
> > Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> > functionality on the Foundation model, running both 64-bit VMs and
> > 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> > GICv2-on-GICv3.
> >
> > The patches are also available in the vhe-optimize-v2 branch on my
> > kernel.org repository [5].
> >
> > Changes since v1:
> > - Rebased on v4.15-rc1 and newer versions of other dependencies,
> > including the vcpu load/put approach taken for KVM.
> > - Addressed review comments from v1 (detailed changelogs are in the
> > individual patches).
> >
> > Thanks,
> > -Christoffer
> >
> > [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> > [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> > [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> > [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> > [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
>
> I just submitted the benchmark I used to test your v1 and v2 series':
> https://lkml.org/lkml/2017/12/11/364
>
> On ThunderX2, 112 online CPUs test results for v1 are like this:
>
> Host, v4.14:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 81 110
> Broadcast IPI: 0 2106
>
> Guest, v4.14:
> Dry-run: 0 1
> Self-IPI: 10 18
> Normal IPI: 305 525
> Broadcast IPI: 0 9729
>
> Guest, v4.14 + VHE:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 176 343
> Broadcast IPI: 0 9885
>
> And for v2.
>
> Host, v4.15:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 79 108
> Broadcast IPI: 0 2102
>
> Guest, v4.15-rc:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 291 526
> Broadcast IPI: 0 10439
>
> Guest, v4.15-rc + VHE:
> Dry-run: 0 2
> Self-IPI: 14 28
> Normal IPI: 370 569
> Broadcast IPI: 0 11688
>
> All times are normalized to v1 host dry-run time. Smaller - better.
>
Thanks for running this.
> Results for v1 and v2 may vary because kernel version is changed.
> What makes us worry is slowing down the "Normal IPI" test observed in
> v2 series.
I'm wondering if this is not simply variability in your measurements.
How many times have you run this? The 100,000 iterations for each run
is not a lot if you consider the cost of migrating threads.
Is this workload pinned to a single CPU? Is the system otherwise idle
(both host and guest)? If you run this during boot or during kernel
module load, the results may be skewed by that.
Power management can greatly influence results as well.
Just so I'm sure we're reading these reults the same way, your "+ VHE"
notation means the VHE optimization series, but both the before and
after picture runs with VHE enabled, right?
Are you using the same guest kernel version and config for both your v1
and v2 results, and for both the before and after versions?
I can't easily come up with a scneario that explains the slowdown on the
normal IPI test, beyond some unfortunate bug introduced in v2.
>
> Nevertheless, if you find test relevant, for v1 and v2,
> Tested-by: Yury Norov <ynorov@caviumnetworks.com>
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-11 15:34 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-11 15:34 UTC (permalink / raw)
To: linux-arm-kernel
Hi Yury,
On Mon, Dec 11, 2017 at 05:43:23PM +0300, Yury Norov wrote:
>
> On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> > This series redesigns parts of KVM/ARM to optimize the performance on
> > VHE systems. The general approach is to try to do as little work as
> > possible when transitioning between the VM and the hypervisor. This has
> > the benefit of lower latency when waiting for interrupts and delivering
> > virtual interrupts, and reduces the overhead of emulating behavior and
> > I/O in the host kernel.
> >
> > Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> > that can be generally improved. We then add infrastructure to move more
> > logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> > registers.
> >
> > We then introduce a new world-switch function for VHE systems, which we
> > can tweak and optimize for VHE systems. To do that, we rework a lot of
> > the system register save/restore handling and emulation code that may
> > need access to system registers, so that we can defer as many system
> > register save/restore operations to vcpu_load and vcpu_put, and move
> > this logic out of the VHE world switch function.
> >
> > We then optimize the configuration of traps. On non-VHE systems, both
> > the host and VM kernels run in EL1, but because the host kernel should
> > have full access to the underlying hardware, but the VM kernel should
> > not, we essentially make the host kernel more privileged than the VM
> > kernel despite them both running at the same privilege level by enabling
> > VE traps when entering the VM and disabling those traps when exiting the
> > VM. On VHE systems, the host kernel runs in EL2 and has full access to
> > the hardware (as much as allowed by secure side software), and is
> > unaffected by the trap configuration. That means we can configure the
> > traps for VMs running in EL1 once, and don't have to switch them on and
> > off for every entry/exit to/from the VM.
> >
> > Finally, we improve our VGIC handling by moving all save/restore logic
> > out of the VHE world-switch, and we make it possible to truly only
> > evaluate if the AP list is empty and not do *any* VGIC work if that is
> > the case, and only do the minimal amount of work required in the course
> > of the VGIC processing when we have virtual interrupts in flight.
> >
> > The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> > [1], the level-triggered mapped interrupts support series [2], and the
> > first five patches of James' SDEI series [3], a single SVE patch that
> > moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> > my vcpu load/put series [4].
> >
> > I've given the patches a fair amount of testing on Thunder-X, Mustang,
> > Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> > functionality on the Foundation model, running both 64-bit VMs and
> > 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> > GICv2-on-GICv3.
> >
> > The patches are also available in the vhe-optimize-v2 branch on my
> > kernel.org repository [5].
> >
> > Changes since v1:
> > - Rebased on v4.15-rc1 and newer versions of other dependencies,
> > including the vcpu load/put approach taken for KVM.
> > - Addressed review comments from v1 (detailed changelogs are in the
> > individual patches).
> >
> > Thanks,
> > -Christoffer
> >
> > [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> > [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> > [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> > [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> > [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
>
> I just submitted the benchmark I used to test your v1 and v2 series':
> https://lkml.org/lkml/2017/12/11/364
>
> On ThunderX2, 112 online CPUs test results for v1 are like this:
>
> Host, v4.14:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 81 110
> Broadcast IPI: 0 2106
>
> Guest, v4.14:
> Dry-run: 0 1
> Self-IPI: 10 18
> Normal IPI: 305 525
> Broadcast IPI: 0 9729
>
> Guest, v4.14 + VHE:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 176 343
> Broadcast IPI: 0 9885
>
> And for v2.
>
> Host, v4.15:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 79 108
> Broadcast IPI: 0 2102
>
> Guest, v4.15-rc:
> Dry-run: 0 1
> Self-IPI: 9 18
> Normal IPI: 291 526
> Broadcast IPI: 0 10439
>
> Guest, v4.15-rc + VHE:
> Dry-run: 0 2
> Self-IPI: 14 28
> Normal IPI: 370 569
> Broadcast IPI: 0 11688
>
> All times are normalized to v1 host dry-run time. Smaller - better.
>
Thanks for running this.
> Results for v1 and v2 may vary because kernel version is changed.
> What makes us worry is slowing down the "Normal IPI" test observed in
> v2 series.
I'm wondering if this is not simply variability in your measurements.
How many times have you run this? The 100,000 iterations for each run
is not a lot if you consider the cost of migrating threads.
Is this workload pinned to a single CPU? Is the system otherwise idle
(both host and guest)? If you run this during boot or during kernel
module load, the results may be skewed by that.
Power management can greatly influence results as well.
Just so I'm sure we're reading these reults the same way, your "+ VHE"
notation means the VHE optimization series, but both the before and
after picture runs with VHE enabled, right?
Are you using the same guest kernel version and config for both your v1
and v2 results, and for both the before and after versions?
I can't easily come up with a scneario that explains the slowdown on the
normal IPI test, beyond some unfortunate bug introduced in v2.
>
> Nevertheless, if you find test relevant, for v1 and v2,
> Tested-by: Yury Norov <ynorov@caviumnetworks.com>
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
2017-12-11 15:34 ` Christoffer Dall
@ 2017-12-11 16:30 ` Yury Norov
-1 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 16:30 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvm, Marc Zyngier, linux-arm-kernel, kvmarm, Shih-Wei Li
On Mon, Dec 11, 2017 at 04:34:58PM +0100, Christoffer Dall wrote:
> Hi Yury,
>
> On Mon, Dec 11, 2017 at 05:43:23PM +0300, Yury Norov wrote:
> >
> > On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> > > This series redesigns parts of KVM/ARM to optimize the performance on
> > > VHE systems. The general approach is to try to do as little work as
> > > possible when transitioning between the VM and the hypervisor. This has
> > > the benefit of lower latency when waiting for interrupts and delivering
> > > virtual interrupts, and reduces the overhead of emulating behavior and
> > > I/O in the host kernel.
> > >
> > > Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> > > that can be generally improved. We then add infrastructure to move more
> > > logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> > > registers.
> > >
> > > We then introduce a new world-switch function for VHE systems, which we
> > > can tweak and optimize for VHE systems. To do that, we rework a lot of
> > > the system register save/restore handling and emulation code that may
> > > need access to system registers, so that we can defer as many system
> > > register save/restore operations to vcpu_load and vcpu_put, and move
> > > this logic out of the VHE world switch function.
> > >
> > > We then optimize the configuration of traps. On non-VHE systems, both
> > > the host and VM kernels run in EL1, but because the host kernel should
> > > have full access to the underlying hardware, but the VM kernel should
> > > not, we essentially make the host kernel more privileged than the VM
> > > kernel despite them both running at the same privilege level by enabling
> > > VE traps when entering the VM and disabling those traps when exiting the
> > > VM. On VHE systems, the host kernel runs in EL2 and has full access to
> > > the hardware (as much as allowed by secure side software), and is
> > > unaffected by the trap configuration. That means we can configure the
> > > traps for VMs running in EL1 once, and don't have to switch them on and
> > > off for every entry/exit to/from the VM.
> > >
> > > Finally, we improve our VGIC handling by moving all save/restore logic
> > > out of the VHE world-switch, and we make it possible to truly only
> > > evaluate if the AP list is empty and not do *any* VGIC work if that is
> > > the case, and only do the minimal amount of work required in the course
> > > of the VGIC processing when we have virtual interrupts in flight.
> > >
> > > The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> > > [1], the level-triggered mapped interrupts support series [2], and the
> > > first five patches of James' SDEI series [3], a single SVE patch that
> > > moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> > > my vcpu load/put series [4].
> > >
> > > I've given the patches a fair amount of testing on Thunder-X, Mustang,
> > > Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> > > functionality on the Foundation model, running both 64-bit VMs and
> > > 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> > > GICv2-on-GICv3.
> > >
> > > The patches are also available in the vhe-optimize-v2 branch on my
> > > kernel.org repository [5].
> > >
> > > Changes since v1:
> > > - Rebased on v4.15-rc1 and newer versions of other dependencies,
> > > including the vcpu load/put approach taken for KVM.
> > > - Addressed review comments from v1 (detailed changelogs are in the
> > > individual patches).
> > >
> > > Thanks,
> > > -Christoffer
> > >
> > > [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> > > [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> > > [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> > > [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> > > [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
> >
> > I just submitted the benchmark I used to test your v1 and v2 series':
> > https://lkml.org/lkml/2017/12/11/364
> >
> > On ThunderX2, 112 online CPUs test results for v1 are like this:
> >
> > Host, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 81 110
> > Broadcast IPI: 0 2106
> >
> > Guest, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 10 18
> > Normal IPI: 305 525
> > Broadcast IPI: 0 9729
> >
> > Guest, v4.14 + VHE:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 176 343
> > Broadcast IPI: 0 9885
> >
> > And for v2.
> >
> > Host, v4.15:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 79 108
> > Broadcast IPI: 0 2102
> >
> > Guest, v4.15-rc:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 291 526
> > Broadcast IPI: 0 10439
> >
> > Guest, v4.15-rc + VHE:
> > Dry-run: 0 2
> > Self-IPI: 14 28
> > Normal IPI: 370 569
> > Broadcast IPI: 0 11688
> >
> > All times are normalized to v1 host dry-run time. Smaller - better.
> >
>
> Thanks for running this.
>
> > Results for v1 and v2 may vary because kernel version is changed.
> > What makes us worry is slowing down the "Normal IPI" test observed in
> > v2 series.
>
> I'm wondering if this is not simply variability in your measurements.
> How many times have you run this? The 100,000 iterations for each run
> is not a lot if you consider the cost of migrating threads.
I ran it more than 100 times, maybe more than 200. Variablity exists,
but ~5% at max, much less than observed changes. I can run 1M iterations
version to handle this concern.
> Is this workload pinned to a single CPU?
No. We are interested in test close to real usecases, so I didn't pin
the test. Inside, sending IPI and waiting for acknowledge is pinned
using {get,put}_cpu(). Tomorrow I'll run test pinned to some CPU. Are
you OK with 'taskset -c 111 insmod ipi_benchmark.ko'?
> Is the system otherwise idle (both host and guest)?
Yes, this machine is in my exclusive usage, and I don't run something
heavy in background. And this is newly installed Ubuntu.
> If you run this during boot or during kernel module load, the results
> may be skewed by that.
Hmm... I do it at module load, but there are many tests that measure
performance like this... Anyway, I'll check that.
> Power management can greatly influence results as well.
That's true. I'll check this also. But as you see, all host numbers,
and guest dry-run and self-ipi numbers are stable, except v2 test...
> Just so I'm sure we're reading these reults the same way, your "+ VHE"
> notation means the VHE optimization series, but both the before and
> after picture runs with VHE enabled, right?
Yes.
> Are you using the same guest kernel version and config for both your v1
> and v2 results, and for both the before and after versions?
I rebased v1 on 4.14. For v2 I ran make olddefconfig, the rest is same
as on your branches. I used same kernel image for host and guest, ie
4.14 host + 4.14 guest for v1, and 4.15-rc host and guest for v2. I
also tested host with and without this series - no difference for both
versions.
> I can't easily come up with a scneario that explains the slowdown on the
> normal IPI test, beyond some unfortunate bug introduced in v2.
>
> >
> > Nevertheless, if you find test relevant, for v1 and v2,
> > Tested-by: Yury Norov <ynorov@caviumnetworks.com>
>
> Thanks,
> -Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 00/36] Optimize KVM/ARM for VHE systems
@ 2017-12-11 16:30 ` Yury Norov
0 siblings, 0 replies; 158+ messages in thread
From: Yury Norov @ 2017-12-11 16:30 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 04:34:58PM +0100, Christoffer Dall wrote:
> Hi Yury,
>
> On Mon, Dec 11, 2017 at 05:43:23PM +0300, Yury Norov wrote:
> >
> > On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> > > This series redesigns parts of KVM/ARM to optimize the performance on
> > > VHE systems. The general approach is to try to do as little work as
> > > possible when transitioning between the VM and the hypervisor. This has
> > > the benefit of lower latency when waiting for interrupts and delivering
> > > virtual interrupts, and reduces the overhead of emulating behavior and
> > > I/O in the host kernel.
> > >
> > > Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> > > that can be generally improved. We then add infrastructure to move more
> > > logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> > > registers.
> > >
> > > We then introduce a new world-switch function for VHE systems, which we
> > > can tweak and optimize for VHE systems. To do that, we rework a lot of
> > > the system register save/restore handling and emulation code that may
> > > need access to system registers, so that we can defer as many system
> > > register save/restore operations to vcpu_load and vcpu_put, and move
> > > this logic out of the VHE world switch function.
> > >
> > > We then optimize the configuration of traps. On non-VHE systems, both
> > > the host and VM kernels run in EL1, but because the host kernel should
> > > have full access to the underlying hardware, but the VM kernel should
> > > not, we essentially make the host kernel more privileged than the VM
> > > kernel despite them both running at the same privilege level by enabling
> > > VE traps when entering the VM and disabling those traps when exiting the
> > > VM. On VHE systems, the host kernel runs in EL2 and has full access to
> > > the hardware (as much as allowed by secure side software), and is
> > > unaffected by the trap configuration. That means we can configure the
> > > traps for VMs running in EL1 once, and don't have to switch them on and
> > > off for every entry/exit to/from the VM.
> > >
> > > Finally, we improve our VGIC handling by moving all save/restore logic
> > > out of the VHE world-switch, and we make it possible to truly only
> > > evaluate if the AP list is empty and not do *any* VGIC work if that is
> > > the case, and only do the minimal amount of work required in the course
> > > of the VGIC processing when we have virtual interrupts in flight.
> > >
> > > The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> > > [1], the level-triggered mapped interrupts support series [2], and the
> > > first five patches of James' SDEI series [3], a single SVE patch that
> > > moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> > > my vcpu load/put series [4].
> > >
> > > I've given the patches a fair amount of testing on Thunder-X, Mustang,
> > > Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> > > functionality on the Foundation model, running both 64-bit VMs and
> > > 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> > > GICv2-on-GICv3.
> > >
> > > The patches are also available in the vhe-optimize-v2 branch on my
> > > kernel.org repository [5].
> > >
> > > Changes since v1:
> > > - Rebased on v4.15-rc1 and newer versions of other dependencies,
> > > including the vcpu load/put approach taken for KVM.
> > > - Addressed review comments from v1 (detailed changelogs are in the
> > > individual patches).
> > >
> > > Thanks,
> > > -Christoffer
> > >
> > > [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> > > [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> > > [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> > > [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> > > [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
> >
> > I just submitted the benchmark I used to test your v1 and v2 series':
> > https://lkml.org/lkml/2017/12/11/364
> >
> > On ThunderX2, 112 online CPUs test results for v1 are like this:
> >
> > Host, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 81 110
> > Broadcast IPI: 0 2106
> >
> > Guest, v4.14:
> > Dry-run: 0 1
> > Self-IPI: 10 18
> > Normal IPI: 305 525
> > Broadcast IPI: 0 9729
> >
> > Guest, v4.14 + VHE:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 176 343
> > Broadcast IPI: 0 9885
> >
> > And for v2.
> >
> > Host, v4.15:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 79 108
> > Broadcast IPI: 0 2102
> >
> > Guest, v4.15-rc:
> > Dry-run: 0 1
> > Self-IPI: 9 18
> > Normal IPI: 291 526
> > Broadcast IPI: 0 10439
> >
> > Guest, v4.15-rc + VHE:
> > Dry-run: 0 2
> > Self-IPI: 14 28
> > Normal IPI: 370 569
> > Broadcast IPI: 0 11688
> >
> > All times are normalized to v1 host dry-run time. Smaller - better.
> >
>
> Thanks for running this.
>
> > Results for v1 and v2 may vary because kernel version is changed.
> > What makes us worry is slowing down the "Normal IPI" test observed in
> > v2 series.
>
> I'm wondering if this is not simply variability in your measurements.
> How many times have you run this? The 100,000 iterations for each run
> is not a lot if you consider the cost of migrating threads.
I ran it more than 100 times, maybe more than 200. Variablity exists,
but ~5% at max, much less than observed changes. I can run 1M iterations
version to handle this concern.
> Is this workload pinned to a single CPU?
No. We are interested in test close to real usecases, so I didn't pin
the test. Inside, sending IPI and waiting for acknowledge is pinned
using {get,put}_cpu(). Tomorrow I'll run test pinned to some CPU. Are
you OK with 'taskset -c 111 insmod ipi_benchmark.ko'?
> Is the system otherwise idle (both host and guest)?
Yes, this machine is in my exclusive usage, and I don't run something
heavy in background. And this is newly installed Ubuntu.
> If you run this during boot or during kernel module load, the results
> may be skewed by that.
Hmm... I do it at module load, but there are many tests that measure
performance like this... Anyway, I'll check that.
> Power management can greatly influence results as well.
That's true. I'll check this also. But as you see, all host numbers,
and guest dry-run and self-ipi numbers are stable, except v2 test...
> Just so I'm sure we're reading these reults the same way, your "+ VHE"
> notation means the VHE optimization series, but both the before and
> after picture runs with VHE enabled, right?
Yes.
> Are you using the same guest kernel version and config for both your v1
> and v2 results, and for both the before and after versions?
I rebased v1 on 4.14. For v2 I ran make olddefconfig, the rest is same
as on your branches. I used same kernel image for host and guest, ie
4.14 host + 4.14 guest for v1, and 4.15-rc host and guest for v2. I
also tested host with and without this series - no difference for both
versions.
> I can't easily come up with a scneario that explains the slowdown on the
> normal IPI test, beyond some unfortunate bug introduced in v2.
>
> >
> > Nevertheless, if you find test relevant, for v1 and v2,
> > Tested-by: Yury Norov <ynorov@caviumnetworks.com>
>
> Thanks,
> -Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-11 11:24 ` Christoffer Dall
@ 2017-12-12 13:08 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-12 13:08 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On 11/12/17 11:24, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> When we defer the save/restore of system registers to vcpu_load and
>>> vcpu_put, we need to take care of the emulation code that handles traps
>>> to these registers, since simply reading the memory array will return
>>> stale data.
>>>
>>> Therefore, introduce two functions to directly read/write the registers
>>> from the physical CPU when we're on a VHE system that has loaded the
>>> system registers onto the physical CPU.
>>>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Removed spurious white space
>>>
>>> arch/arm64/include/asm/kvm_host.h | 4 +++
>>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
>>> 2 files changed, 55 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index de0d55b30b61..f6afe685a280 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>>>
>>> /* Detect first run of a vcpu */
>>> bool has_run_once;
>>> +
>>> + /* True when deferrable sysregs are loaded on the physical CPU,
>>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
>>> + bool sysregs_loaded_on_cpu;
>>> };
>>>
>>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 62c12ab9e6c4..80adbec933de 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -35,6 +35,7 @@
>>> #include <asm/kvm_coproc.h>
>>> #include <asm/kvm_emulate.h>
>>> #include <asm/kvm_host.h>
>>> +#include <asm/kvm_hyp.h>
>>> #include <asm/kvm_mmu.h>
>>> #include <asm/perf_event.h>
>>> #include <asm/sysreg.h>
>>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
>>> return true;
>>> }
>>>
>>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
>>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
>>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
>>> + case TCR_EL1: return read_sysreg_el1(tcr);
>>> + case ESR_EL1: return read_sysreg_el1(esr);
>>> + case FAR_EL1: return read_sysreg_el1(far);
>>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
>>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
>>> + case MAIR_EL1: return read_sysreg_el1(mair);
>>> + case AMAIR_EL1: return read_sysreg_el1(amair);
>>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
>>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
>>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + return vcpu_sys_reg(vcpu, reg);
>>> +}
>>> +
>>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
>>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
>>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
>>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
>>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
>>> + case FAR_EL1: write_sysreg_el1(val, far); return;
>>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
>>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
>>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
>>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
>>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
>>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
>>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + vcpu_sys_reg(vcpu, reg) = val;
>>> +}
>>> +
>>> /*
>>> * Generic accessor for VM registers. Only called as long as HCR_TVM
>>> * is set. If the guest enables the MMU, we stop trapping the VM
>>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
>>> if (!p->is_aarch32 || !p->is_32bit) {
>>> val = p->regval;
>>> } else {
>>> - val = vcpu_sys_reg(vcpu, reg);
>>> + val = read_deferrable_vm_reg(vcpu, reg);
>>> if (r->reg % 2)
>>> val = (p->regval << 32) | (u64)lower_32_bits(val);
>>> else
>>> val = ((u64)upper_32_bits(val) << 32) |
>>> (u64)lower_32_bits(p->regval);
>>> }
>>> - vcpu_sys_reg(vcpu, reg) = val;
>>> + write_deferrable_vm_reg(vcpu, reg, val);
>>>
>>> kvm_toggle_cache(vcpu, was_enabled);
>>> return true;
>>>
>>
>> I'm slightly uneasy with this. It means that the rest of the KVM code
>> has to know whether a given register is deferrable or not (or face the
>> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
>> magic in the vcpu_sys_reg() accessors.
>>
>> Thoughts?
>>
>
> Yes, this is the main reservation I also have with the series.
>
> I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> that.
>
> That does mean that we need a giant switch statement which knows how to
> read any deferrable EL1 (and EL0) system register from hardware, and
> still BUG/WARN if someone adds a system register but forgets to add that
> handler and test on VHE. Unless there's some fantastic auto-gen
> mechanism that can take a hash define and figure out which sysreg
> instruction to use - I couldn't think of that.
Coming back to this, as I'm currently prototyping something.
What is the rational for the BUG()? It is not like we can add a random
sysreg and expect it to be deferred. It is a conscious decision to do
so, and I feel that the default should be that the sysreg should be
save/restored. What am I missing?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-12 13:08 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-12 13:08 UTC (permalink / raw)
To: linux-arm-kernel
On 11/12/17 11:24, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> When we defer the save/restore of system registers to vcpu_load and
>>> vcpu_put, we need to take care of the emulation code that handles traps
>>> to these registers, since simply reading the memory array will return
>>> stale data.
>>>
>>> Therefore, introduce two functions to directly read/write the registers
>>> from the physical CPU when we're on a VHE system that has loaded the
>>> system registers onto the physical CPU.
>>>
>>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>>> ---
>>>
>>> Notes:
>>> Changes since v1:
>>> - Removed spurious white space
>>>
>>> arch/arm64/include/asm/kvm_host.h | 4 +++
>>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
>>> 2 files changed, 55 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>>> index de0d55b30b61..f6afe685a280 100644
>>> --- a/arch/arm64/include/asm/kvm_host.h
>>> +++ b/arch/arm64/include/asm/kvm_host.h
>>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
>>>
>>> /* Detect first run of a vcpu */
>>> bool has_run_once;
>>> +
>>> + /* True when deferrable sysregs are loaded on the physical CPU,
>>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
>>> + bool sysregs_loaded_on_cpu;
>>> };
>>>
>>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 62c12ab9e6c4..80adbec933de 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -35,6 +35,7 @@
>>> #include <asm/kvm_coproc.h>
>>> #include <asm/kvm_emulate.h>
>>> #include <asm/kvm_host.h>
>>> +#include <asm/kvm_hyp.h>
>>> #include <asm/kvm_mmu.h>
>>> #include <asm/perf_event.h>
>>> #include <asm/sysreg.h>
>>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
>>> return true;
>>> }
>>>
>>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
>>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
>>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
>>> + case TCR_EL1: return read_sysreg_el1(tcr);
>>> + case ESR_EL1: return read_sysreg_el1(esr);
>>> + case FAR_EL1: return read_sysreg_el1(far);
>>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
>>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
>>> + case MAIR_EL1: return read_sysreg_el1(mair);
>>> + case AMAIR_EL1: return read_sysreg_el1(amair);
>>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
>>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
>>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + return vcpu_sys_reg(vcpu, reg);
>>> +}
>>> +
>>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
>>> +{
>>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
>>> + switch (reg) {
>>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
>>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
>>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
>>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
>>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
>>> + case FAR_EL1: write_sysreg_el1(val, far); return;
>>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
>>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
>>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
>>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
>>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
>>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
>>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
>>> + default: BUG();
>>> + }
>>> + }
>>> +
>>> + vcpu_sys_reg(vcpu, reg) = val;
>>> +}
>>> +
>>> /*
>>> * Generic accessor for VM registers. Only called as long as HCR_TVM
>>> * is set. If the guest enables the MMU, we stop trapping the VM
>>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
>>> if (!p->is_aarch32 || !p->is_32bit) {
>>> val = p->regval;
>>> } else {
>>> - val = vcpu_sys_reg(vcpu, reg);
>>> + val = read_deferrable_vm_reg(vcpu, reg);
>>> if (r->reg % 2)
>>> val = (p->regval << 32) | (u64)lower_32_bits(val);
>>> else
>>> val = ((u64)upper_32_bits(val) << 32) |
>>> (u64)lower_32_bits(p->regval);
>>> }
>>> - vcpu_sys_reg(vcpu, reg) = val;
>>> + write_deferrable_vm_reg(vcpu, reg, val);
>>>
>>> kvm_toggle_cache(vcpu, was_enabled);
>>> return true;
>>>
>>
>> I'm slightly uneasy with this. It means that the rest of the KVM code
>> has to know whether a given register is deferrable or not (or face the
>> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
>> magic in the vcpu_sys_reg() accessors.
>>
>> Thoughts?
>>
>
> Yes, this is the main reservation I also have with the series.
>
> I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> that.
>
> That does mean that we need a giant switch statement which knows how to
> read any deferrable EL1 (and EL0) system register from hardware, and
> still BUG/WARN if someone adds a system register but forgets to add that
> handler and test on VHE. Unless there's some fantastic auto-gen
> mechanism that can take a hash define and figure out which sysreg
> instruction to use - I couldn't think of that.
Coming back to this, as I'm currently prototyping something.
What is the rational for the BUG()? It is not like we can add a random
sysreg and expect it to be deferred. It is a conscious decision to do
so, and I feel that the default should be that the sysreg should be
save/restored. What am I missing?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
2017-12-12 13:08 ` Marc Zyngier
@ 2017-12-12 15:46 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-12 15:46 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Tue, Dec 12, 2017 at 01:08:30PM +0000, Marc Zyngier wrote:
> On 11/12/17 11:24, Christoffer Dall wrote:
> > On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
> >> On 07/12/17 17:06, Christoffer Dall wrote:
> >>> When we defer the save/restore of system registers to vcpu_load and
> >>> vcpu_put, we need to take care of the emulation code that handles traps
> >>> to these registers, since simply reading the memory array will return
> >>> stale data.
> >>>
> >>> Therefore, introduce two functions to directly read/write the registers
> >>> from the physical CPU when we're on a VHE system that has loaded the
> >>> system registers onto the physical CPU.
> >>>
> >>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >>> ---
> >>>
> >>> Notes:
> >>> Changes since v1:
> >>> - Removed spurious white space
> >>>
> >>> arch/arm64/include/asm/kvm_host.h | 4 +++
> >>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> >>> 2 files changed, 55 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> >>> index de0d55b30b61..f6afe685a280 100644
> >>> --- a/arch/arm64/include/asm/kvm_host.h
> >>> +++ b/arch/arm64/include/asm/kvm_host.h
> >>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
> >>>
> >>> /* Detect first run of a vcpu */
> >>> bool has_run_once;
> >>> +
> >>> + /* True when deferrable sysregs are loaded on the physical CPU,
> >>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> >>> + bool sysregs_loaded_on_cpu;
> >>> };
> >>>
> >>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> >>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> >>> index 62c12ab9e6c4..80adbec933de 100644
> >>> --- a/arch/arm64/kvm/sys_regs.c
> >>> +++ b/arch/arm64/kvm/sys_regs.c
> >>> @@ -35,6 +35,7 @@
> >>> #include <asm/kvm_coproc.h>
> >>> #include <asm/kvm_emulate.h>
> >>> #include <asm/kvm_host.h>
> >>> +#include <asm/kvm_hyp.h>
> >>> #include <asm/kvm_mmu.h>
> >>> #include <asm/perf_event.h>
> >>> #include <asm/sysreg.h>
> >>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> >>> return true;
> >>> }
> >>>
> >>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> >>> +{
> >>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> >>> + switch (reg) {
> >>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> >>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> >>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> >>> + case TCR_EL1: return read_sysreg_el1(tcr);
> >>> + case ESR_EL1: return read_sysreg_el1(esr);
> >>> + case FAR_EL1: return read_sysreg_el1(far);
> >>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> >>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> >>> + case MAIR_EL1: return read_sysreg_el1(mair);
> >>> + case AMAIR_EL1: return read_sysreg_el1(amair);
> >>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> >>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
> >>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> >>> + default: BUG();
> >>> + }
> >>> + }
> >>> +
> >>> + return vcpu_sys_reg(vcpu, reg);
> >>> +}
> >>> +
> >>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> >>> +{
> >>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> >>> + switch (reg) {
> >>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> >>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> >>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> >>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> >>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
> >>> + case FAR_EL1: write_sysreg_el1(val, far); return;
> >>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> >>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> >>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> >>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> >>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> >>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> >>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> >>> + default: BUG();
> >>> + }
> >>> + }
> >>> +
> >>> + vcpu_sys_reg(vcpu, reg) = val;
> >>> +}
> >>> +
> >>> /*
> >>> * Generic accessor for VM registers. Only called as long as HCR_TVM
> >>> * is set. If the guest enables the MMU, we stop trapping the VM
> >>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> >>> if (!p->is_aarch32 || !p->is_32bit) {
> >>> val = p->regval;
> >>> } else {
> >>> - val = vcpu_sys_reg(vcpu, reg);
> >>> + val = read_deferrable_vm_reg(vcpu, reg);
> >>> if (r->reg % 2)
> >>> val = (p->regval << 32) | (u64)lower_32_bits(val);
> >>> else
> >>> val = ((u64)upper_32_bits(val) << 32) |
> >>> (u64)lower_32_bits(p->regval);
> >>> }
> >>> - vcpu_sys_reg(vcpu, reg) = val;
> >>> + write_deferrable_vm_reg(vcpu, reg, val);
> >>>
> >>> kvm_toggle_cache(vcpu, was_enabled);
> >>> return true;
> >>>
> >>
> >> I'm slightly uneasy with this. It means that the rest of the KVM code
> >> has to know whether a given register is deferrable or not (or face the
> >> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
> >> magic in the vcpu_sys_reg() accessors.
> >>
> >> Thoughts?
> >>
> >
> > Yes, this is the main reservation I also have with the series.
> >
> > I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> > vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> > that.
> >
> > That does mean that we need a giant switch statement which knows how to
> > read any deferrable EL1 (and EL0) system register from hardware, and
> > still BUG/WARN if someone adds a system register but forgets to add that
> > handler and test on VHE. Unless there's some fantastic auto-gen
> > mechanism that can take a hash define and figure out which sysreg
> > instruction to use - I couldn't think of that.
>
> Coming back to this, as I'm currently prototyping something.
>
> What is the rational for the BUG()? It is not like we can add a random
> sysreg and expect it to be deferred. It is a conscious decision to do
> so, and I feel that the default should be that the sysreg should be
> save/restored. What am I missing?
>
The rationale is simply that as we're developing code, if we start
accessing registers while the sysregs are still loaded on the CPU, which
we haven't accessed before, and don't have an accessor for (new sysregs,
rework other parts of the flow etc.,), I wanted to have a clear point
where we realize that we forgot something. Especially if we mostly test
on non-VHE systems.
But, actually, BUG() may be too hard, as it would only ever affect the
VM, and not the host, so reads could be WARN+RAZ, and writes could be
WARN+WI.
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs
@ 2017-12-12 15:46 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-12 15:46 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Dec 12, 2017 at 01:08:30PM +0000, Marc Zyngier wrote:
> On 11/12/17 11:24, Christoffer Dall wrote:
> > On Mon, Dec 11, 2017 at 11:10:36AM +0000, Marc Zyngier wrote:
> >> On 07/12/17 17:06, Christoffer Dall wrote:
> >>> When we defer the save/restore of system registers to vcpu_load and
> >>> vcpu_put, we need to take care of the emulation code that handles traps
> >>> to these registers, since simply reading the memory array will return
> >>> stale data.
> >>>
> >>> Therefore, introduce two functions to directly read/write the registers
> >>> from the physical CPU when we're on a VHE system that has loaded the
> >>> system registers onto the physical CPU.
> >>>
> >>> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >>> ---
> >>>
> >>> Notes:
> >>> Changes since v1:
> >>> - Removed spurious white space
> >>>
> >>> arch/arm64/include/asm/kvm_host.h | 4 +++
> >>> arch/arm64/kvm/sys_regs.c | 53 +++++++++++++++++++++++++++++++++++++--
> >>> 2 files changed, 55 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> >>> index de0d55b30b61..f6afe685a280 100644
> >>> --- a/arch/arm64/include/asm/kvm_host.h
> >>> +++ b/arch/arm64/include/asm/kvm_host.h
> >>> @@ -279,6 +279,10 @@ struct kvm_vcpu_arch {
> >>>
> >>> /* Detect first run of a vcpu */
> >>> bool has_run_once;
> >>> +
> >>> + /* True when deferrable sysregs are loaded on the physical CPU,
> >>> + * see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
> >>> + bool sysregs_loaded_on_cpu;
> >>> };
> >>>
> >>> #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
> >>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> >>> index 62c12ab9e6c4..80adbec933de 100644
> >>> --- a/arch/arm64/kvm/sys_regs.c
> >>> +++ b/arch/arm64/kvm/sys_regs.c
> >>> @@ -35,6 +35,7 @@
> >>> #include <asm/kvm_coproc.h>
> >>> #include <asm/kvm_emulate.h>
> >>> #include <asm/kvm_host.h>
> >>> +#include <asm/kvm_hyp.h>
> >>> #include <asm/kvm_mmu.h>
> >>> #include <asm/perf_event.h>
> >>> #include <asm/sysreg.h>
> >>> @@ -111,6 +112,54 @@ static bool access_dcsw(struct kvm_vcpu *vcpu,
> >>> return true;
> >>> }
> >>>
> >>> +static u64 read_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg)
> >>> +{
> >>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> >>> + switch (reg) {
> >>> + case SCTLR_EL1: return read_sysreg_el1(sctlr);
> >>> + case TTBR0_EL1: return read_sysreg_el1(ttbr0);
> >>> + case TTBR1_EL1: return read_sysreg_el1(ttbr1);
> >>> + case TCR_EL1: return read_sysreg_el1(tcr);
> >>> + case ESR_EL1: return read_sysreg_el1(esr);
> >>> + case FAR_EL1: return read_sysreg_el1(far);
> >>> + case AFSR0_EL1: return read_sysreg_el1(afsr0);
> >>> + case AFSR1_EL1: return read_sysreg_el1(afsr1);
> >>> + case MAIR_EL1: return read_sysreg_el1(mair);
> >>> + case AMAIR_EL1: return read_sysreg_el1(amair);
> >>> + case CONTEXTIDR_EL1: return read_sysreg_el1(contextidr);
> >>> + case DACR32_EL2: return read_sysreg(dacr32_el2);
> >>> + case IFSR32_EL2: return read_sysreg(ifsr32_el2);
> >>> + default: BUG();
> >>> + }
> >>> + }
> >>> +
> >>> + return vcpu_sys_reg(vcpu, reg);
> >>> +}
> >>> +
> >>> +static void write_deferrable_vm_reg(struct kvm_vcpu *vcpu, int reg, u64 val)
> >>> +{
> >>> + if (vcpu->arch.sysregs_loaded_on_cpu) {
> >>> + switch (reg) {
> >>> + case SCTLR_EL1: write_sysreg_el1(val, sctlr); return;
> >>> + case TTBR0_EL1: write_sysreg_el1(val, ttbr0); return;
> >>> + case TTBR1_EL1: write_sysreg_el1(val, ttbr1); return;
> >>> + case TCR_EL1: write_sysreg_el1(val, tcr); return;
> >>> + case ESR_EL1: write_sysreg_el1(val, esr); return;
> >>> + case FAR_EL1: write_sysreg_el1(val, far); return;
> >>> + case AFSR0_EL1: write_sysreg_el1(val, afsr0); return;
> >>> + case AFSR1_EL1: write_sysreg_el1(val, afsr1); return;
> >>> + case MAIR_EL1: write_sysreg_el1(val, mair); return;
> >>> + case AMAIR_EL1: write_sysreg_el1(val, amair); return;
> >>> + case CONTEXTIDR_EL1: write_sysreg_el1(val, contextidr); return;
> >>> + case DACR32_EL2: write_sysreg(val, dacr32_el2); return;
> >>> + case IFSR32_EL2: write_sysreg(val, ifsr32_el2); return;
> >>> + default: BUG();
> >>> + }
> >>> + }
> >>> +
> >>> + vcpu_sys_reg(vcpu, reg) = val;
> >>> +}
> >>> +
> >>> /*
> >>> * Generic accessor for VM registers. Only called as long as HCR_TVM
> >>> * is set. If the guest enables the MMU, we stop trapping the VM
> >>> @@ -133,14 +182,14 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
> >>> if (!p->is_aarch32 || !p->is_32bit) {
> >>> val = p->regval;
> >>> } else {
> >>> - val = vcpu_sys_reg(vcpu, reg);
> >>> + val = read_deferrable_vm_reg(vcpu, reg);
> >>> if (r->reg % 2)
> >>> val = (p->regval << 32) | (u64)lower_32_bits(val);
> >>> else
> >>> val = ((u64)upper_32_bits(val) << 32) |
> >>> (u64)lower_32_bits(p->regval);
> >>> }
> >>> - vcpu_sys_reg(vcpu, reg) = val;
> >>> + write_deferrable_vm_reg(vcpu, reg, val);
> >>>
> >>> kvm_toggle_cache(vcpu, was_enabled);
> >>> return true;
> >>>
> >>
> >> I'm slightly uneasy with this. It means that the rest of the KVM code
> >> has to know whether a given register is deferrable or not (or face the
> >> wrath of the BUG). I'd be more inclined to hide the "loaded on cpu"
> >> magic in the vcpu_sys_reg() accessors.
> >>
> >> Thoughts?
> >>
> >
> > Yes, this is the main reservation I also have with the series.
> >
> > I did start out with a giant "rewrite everything to vcpu_get_sys_reg and
> > vcpu_get_sys_reg" which hides this logic, and we may want to go back to
> > that.
> >
> > That does mean that we need a giant switch statement which knows how to
> > read any deferrable EL1 (and EL0) system register from hardware, and
> > still BUG/WARN if someone adds a system register but forgets to add that
> > handler and test on VHE. Unless there's some fantastic auto-gen
> > mechanism that can take a hash define and figure out which sysreg
> > instruction to use - I couldn't think of that.
>
> Coming back to this, as I'm currently prototyping something.
>
> What is the rational for the BUG()? It is not like we can add a random
> sysreg and expect it to be deferred. It is a conscious decision to do
> so, and I feel that the default should be that the sysreg should be
> save/restored. What am I missing?
>
The rationale is simply that as we're developing code, if we start
accessing registers while the sysregs are still loaded on the CPU, which
we haven't accessed before, and don't have an accessor for (new sysregs,
rework other parts of the flow etc.,), I wanted to have a clear point
where we realize that we forgot something. Especially if we mostly test
on non-VHE systems.
But, actually, BUG() may be too hard, as it would only ever affect the
VM, and not the host, so reads could be WARN+RAZ, and writes could be
WARN+WI.
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
2017-12-11 9:53 ` Marc Zyngier
@ 2017-12-14 12:12 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:12 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 09:53:13AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > So far this is just a copy of the legacy non-VHE switch function, where
> > we only change the existing calls to has_vhe() in both the original and
> > new functions.
>
> I'm not sure I correctly parse the above. Do you want to say that you
> use has_vhe() to select the right world switch back-end? Or did you mean
> to drop some of the has_vhe() calls in the non-VHE case?
>
That's a leftover from a previous patch version. How about the
following instead?
So far this is just a copy of the legacy non-VHE switch function, but we
will start reworking these functions in separate directions to work on
VHE and non-VHE in the most optimal way in later patches.
Thanks,
-Christoffer
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
> > __kvm_vcpu_run_nvhe
> > - Removed stray whitespace line
> >
> > arch/arm/include/asm/kvm_asm.h | 5 +++-
> > arch/arm/kvm/hyp/switch.c | 2 +-
> > arch/arm64/include/asm/kvm_asm.h | 4 ++-
> > arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
> > virt/kvm/arm/arm.c | 5 +++-
> > 5 files changed, 69 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
> > index 36dd2962a42d..4ac717276543 100644
> > --- a/arch/arm/include/asm/kvm_asm.h
> > +++ b/arch/arm/include/asm/kvm_asm.h
> > @@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
> >
> > extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
> >
> > -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> > +/* no VHE on 32-bit :( */
> > +static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
> > +
> > +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
> >
> > extern void __init_stage2_translation(void);
> >
> > diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
> > index c3b9799e2e13..7b2bd25e3b10 100644
> > --- a/arch/arm/kvm/hyp/switch.c
> > +++ b/arch/arm/kvm/hyp/switch.c
> > @@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
> > return true;
> > }
> >
> > -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> > index 33e0edc6f8be..adaf1db12271 100644
> > --- a/arch/arm64/include/asm/kvm_asm.h
> > +++ b/arch/arm64/include/asm/kvm_asm.h
> > @@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
> >
> > extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
> >
> > -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> > +extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
> > +
> > +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
> >
> > extern u64 __vgic_v3_get_ich_vtr_el2(void);
> > extern u64 __vgic_v3_read_vmcr(void);
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index 845e3bece399..8f32f8dcab65 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
> > return false;
> > }
> >
> > -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > +/* Switch to the guest for VHE systems running in EL2 */
> > +int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> > +{
> > + struct kvm_cpu_context *host_ctxt;
> > + struct kvm_cpu_context *guest_ctxt;
> > + u64 exit_code;
> > +
> > + vcpu = kern_hyp_va(vcpu);
> > +
> > + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
> > + host_ctxt->__hyp_running_vcpu = vcpu;
> > + guest_ctxt = &vcpu->arch.ctxt;
> > +
> > + __sysreg_save_host_state(host_ctxt);
> > +
> > + __activate_traps(vcpu);
> > + __activate_vm(vcpu);
> > +
> > + __vgic_restore_state(vcpu);
> > + __timer_enable_traps(vcpu);
> > +
> > + /*
> > + * We must restore the 32-bit state before the sysregs, thanks
> > + * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
> > + */
> > + __sysreg32_restore_state(vcpu);
> > + __sysreg_restore_guest_state(guest_ctxt);
> > + __debug_switch_to_guest(vcpu);
> > +
> > + do {
> > + /* Jump in the fire! */
> > + exit_code = __guest_enter(vcpu, host_ctxt);
> > +
> > + /* And we're baaack! */
> > + } while (fixup_guest_exit(vcpu, &exit_code));
> > +
> > + __sysreg_save_guest_state(guest_ctxt);
> > + __sysreg32_save_state(vcpu);
> > + __timer_disable_traps(vcpu);
> > + __vgic_save_state(vcpu);
> > +
> > + __deactivate_traps(vcpu);
> > + __deactivate_vm(vcpu);
> > +
> > + __sysreg_restore_host_state(host_ctxt);
> > +
> > + /*
> > + * This must come after restoring the host sysregs, since a non-VHE
> > + * system may enable SPE here and make use of the TTBRs.
> > + */
> > + __debug_switch_to_host(vcpu);
> > +
> > + return exit_code;
> > +}
> > +
> > +/* Switch to the guest for legacy non-VHE systems */
> > +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> > index 3e10343374a1..104ee524c75a 100644
> > --- a/virt/kvm/arm/arm.c
> > +++ b/virt/kvm/arm/arm.c
> > @@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
> > trace_kvm_entry(*vcpu_pc(vcpu));
> > guest_enter_irqoff();
> >
> > - ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
> > + if (has_vhe())
> > + ret = kvm_vcpu_run_vhe(vcpu);
> > + else
> > + ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
> >
> > vcpu->mode = OUTSIDE_GUEST_MODE;
> > vcpu->stat.exits++;
> >
>
> Otherwise looks good to me.
>
> M.
> --
> Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
@ 2017-12-14 12:12 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:12 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 09:53:13AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > So far this is just a copy of the legacy non-VHE switch function, where
> > we only change the existing calls to has_vhe() in both the original and
> > new functions.
>
> I'm not sure I correctly parse the above. Do you want to say that you
> use has_vhe() to select the right world switch back-end? Or did you mean
> to drop some of the has_vhe() calls in the non-VHE case?
>
That's a leftover from a previous patch version. How about the
following instead?
So far this is just a copy of the legacy non-VHE switch function, but we
will start reworking these functions in separate directions to work on
VHE and non-VHE in the most optimal way in later patches.
Thanks,
-Christoffer
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Rename kvm_vcpu_run to kvm_vcpu_run_vhe and rename __kvm_vcpu_run to
> > __kvm_vcpu_run_nvhe
> > - Removed stray whitespace line
> >
> > arch/arm/include/asm/kvm_asm.h | 5 +++-
> > arch/arm/kvm/hyp/switch.c | 2 +-
> > arch/arm64/include/asm/kvm_asm.h | 4 ++-
> > arch/arm64/kvm/hyp/switch.c | 58 +++++++++++++++++++++++++++++++++++++++-
> > virt/kvm/arm/arm.c | 5 +++-
> > 5 files changed, 69 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
> > index 36dd2962a42d..4ac717276543 100644
> > --- a/arch/arm/include/asm/kvm_asm.h
> > +++ b/arch/arm/include/asm/kvm_asm.h
> > @@ -70,7 +70,10 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
> >
> > extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
> >
> > -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> > +/* no VHE on 32-bit :( */
> > +static inline int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { return 0; }
> > +
> > +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
> >
> > extern void __init_stage2_translation(void);
> >
> > diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c
> > index c3b9799e2e13..7b2bd25e3b10 100644
> > --- a/arch/arm/kvm/hyp/switch.c
> > +++ b/arch/arm/kvm/hyp/switch.c
> > @@ -153,7 +153,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu)
> > return true;
> > }
> >
> > -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> > index 33e0edc6f8be..adaf1db12271 100644
> > --- a/arch/arm64/include/asm/kvm_asm.h
> > +++ b/arch/arm64/include/asm/kvm_asm.h
> > @@ -58,7 +58,9 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);
> >
> > extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high);
> >
> > -extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
> > +extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu);
> > +
> > +extern int __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu);
> >
> > extern u64 __vgic_v3_get_ich_vtr_el2(void);
> > extern u64 __vgic_v3_read_vmcr(void);
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index 845e3bece399..8f32f8dcab65 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -342,7 +342,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
> > return false;
> > }
> >
> > -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
> > +/* Switch to the guest for VHE systems running in EL2 */
> > +int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> > +{
> > + struct kvm_cpu_context *host_ctxt;
> > + struct kvm_cpu_context *guest_ctxt;
> > + u64 exit_code;
> > +
> > + vcpu = kern_hyp_va(vcpu);
> > +
> > + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
> > + host_ctxt->__hyp_running_vcpu = vcpu;
> > + guest_ctxt = &vcpu->arch.ctxt;
> > +
> > + __sysreg_save_host_state(host_ctxt);
> > +
> > + __activate_traps(vcpu);
> > + __activate_vm(vcpu);
> > +
> > + __vgic_restore_state(vcpu);
> > + __timer_enable_traps(vcpu);
> > +
> > + /*
> > + * We must restore the 32-bit state before the sysregs, thanks
> > + * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
> > + */
> > + __sysreg32_restore_state(vcpu);
> > + __sysreg_restore_guest_state(guest_ctxt);
> > + __debug_switch_to_guest(vcpu);
> > +
> > + do {
> > + /* Jump in the fire! */
> > + exit_code = __guest_enter(vcpu, host_ctxt);
> > +
> > + /* And we're baaack! */
> > + } while (fixup_guest_exit(vcpu, &exit_code));
> > +
> > + __sysreg_save_guest_state(guest_ctxt);
> > + __sysreg32_save_state(vcpu);
> > + __timer_disable_traps(vcpu);
> > + __vgic_save_state(vcpu);
> > +
> > + __deactivate_traps(vcpu);
> > + __deactivate_vm(vcpu);
> > +
> > + __sysreg_restore_host_state(host_ctxt);
> > +
> > + /*
> > + * This must come after restoring the host sysregs, since a non-VHE
> > + * system may enable SPE here and make use of the TTBRs.
> > + */
> > + __debug_switch_to_host(vcpu);
> > +
> > + return exit_code;
> > +}
> > +
> > +/* Switch to the guest for legacy non-VHE systems */
> > +int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_cpu_context *host_ctxt;
> > struct kvm_cpu_context *guest_ctxt;
> > diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> > index 3e10343374a1..104ee524c75a 100644
> > --- a/virt/kvm/arm/arm.c
> > +++ b/virt/kvm/arm/arm.c
> > @@ -712,7 +712,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
> > trace_kvm_entry(*vcpu_pc(vcpu));
> > guest_enter_irqoff();
> >
> > - ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
> > + if (has_vhe())
> > + ret = kvm_vcpu_run_vhe(vcpu);
> > + else
> > + ret = kvm_call_hyp(__kvm_vcpu_run_nvhe, vcpu);
> >
> > vcpu->mode = OUTSIDE_GUEST_MODE;
> > vcpu->stat.exits++;
> >
>
> Otherwise looks good to me.
>
> M.
> --
> Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
2017-12-14 12:12 ` Christoffer Dall
@ 2017-12-14 12:17 ` Marc Zyngier
-1 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-14 12:17 UTC (permalink / raw)
To: Christoffer Dall; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On 14/12/17 12:12, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 09:53:13AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> So far this is just a copy of the legacy non-VHE switch function, where
>>> we only change the existing calls to has_vhe() in both the original and
>>> new functions.
>>
>> I'm not sure I correctly parse the above. Do you want to say that you
>> use has_vhe() to select the right world switch back-end? Or did you mean
>> to drop some of the has_vhe() calls in the non-VHE case?
>>
>
> That's a leftover from a previous patch version. How about the
> following instead?
>
> So far this is just a copy of the legacy non-VHE switch function, but we
> will start reworking these functions in separate directions to work on
> VHE and non-VHE in the most optimal way in later patches.
This makes more sense.With that change:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run
@ 2017-12-14 12:17 ` Marc Zyngier
0 siblings, 0 replies; 158+ messages in thread
From: Marc Zyngier @ 2017-12-14 12:17 UTC (permalink / raw)
To: linux-arm-kernel
On 14/12/17 12:12, Christoffer Dall wrote:
> On Mon, Dec 11, 2017 at 09:53:13AM +0000, Marc Zyngier wrote:
>> On 07/12/17 17:06, Christoffer Dall wrote:
>>> So far this is just a copy of the legacy non-VHE switch function, where
>>> we only change the existing calls to has_vhe() in both the original and
>>> new functions.
>>
>> I'm not sure I correctly parse the above. Do you want to say that you
>> use has_vhe() to select the right world switch back-end? Or did you mean
>> to drop some of the has_vhe() calls in the non-VHE case?
>>
>
> That's a leftover from a previous patch version. How about the
> following instead?
>
> So far this is just a copy of the legacy non-VHE switch function, but we
> will start reworking these functions in separate directions to work on
> VHE and non-VHE in the most optimal way in later patches.
This makes more sense.With that change:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
2017-12-11 10:02 ` Marc Zyngier
@ 2017-12-14 12:30 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:30 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 10:02:58AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > The VHE switch function calls __timer_enable_traps and
> > __timer_disable_traps which don't do anything on VHE systems.
> > Therefore, simply remove these calls from the VHE switch function and
> > make the functions non-conditional as they are now only called from the
> > non-VHE switch path.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> > arch/arm64/kvm/hyp/switch.c | 2 --
> > virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
> > 2 files changed, 14 insertions(+), 24 deletions(-)
> >
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index e783e2371b7c..09aafa0470f7 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> > __activate_vm(vcpu->kvm);
> >
> > __vgic_restore_state(vcpu);
> > - __timer_enable_traps(vcpu);
> >
> > /*
> > * We must restore the 32-bit state before the sysregs, thanks
> > @@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> >
> > __sysreg_save_guest_state(guest_ctxt);
> > __sysreg32_save_state(vcpu);
> > - __timer_disable_traps(vcpu);
> > __vgic_save_state(vcpu);
> >
> > __deactivate_traps(vcpu);
> > diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
> > index f24404b3c8df..752b37f9133c 100644
> > --- a/virt/kvm/arm/hyp/timer-sr.c
> > +++ b/virt/kvm/arm/hyp/timer-sr.c
> > @@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
> >
> > void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
> > {
> > - /*
> > - * We don't need to do this for VHE since the host kernel runs in EL2
> > - * with HCR_EL2.TGE ==1, which makes those bits have no impact.
> > - */
> > - if (!has_vhe()) {
> > - u64 val;
> > + u64 val;
> >
> > - /* Allow physical timer/counter access for the host */
> > - val = read_sysreg(cnthctl_el2);
> > - val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> > - write_sysreg(val, cnthctl_el2);
> > - }
> > + /* Allow physical timer/counter access for the host */
> > + val = read_sysreg(cnthctl_el2);
> > + val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> > + write_sysreg(val, cnthctl_el2);
> > }
> >
> > void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
> > {
> > - if (!has_vhe()) {
> > - u64 val;
> > + u64 val;
> >
> > - /*
> > - * Disallow physical timer access for the guest
> > - * Physical counter access is allowed
> > - */
> > - val = read_sysreg(cnthctl_el2);
> > - val &= ~CNTHCTL_EL1PCEN;
> > - val |= CNTHCTL_EL1PCTEN;
> > - write_sysreg(val, cnthctl_el2);
> > - }
> > + /*
> > + * Disallow physical timer access for the guest
> > + * Physical counter access is allowed
> > + */
> > + val = read_sysreg(cnthctl_el2);
> > + val &= ~CNTHCTL_EL1PCEN;
> > + val |= CNTHCTL_EL1PCTEN;
> > + write_sysreg(val, cnthctl_el2);
> > }
> >
>
> Since we're not testing for !VHE anymore, can you add a small comment
> saying that these two function are for the benefit of !VHE only and
> shouldn't be called on VHE?
Yes, absolutely:
diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
index 752b37f9133c..77754a62eb0c 100644
--- a/virt/kvm/arm/hyp/timer-sr.c
+++ b/virt/kvm/arm/hyp/timer-sr.c
@@ -27,6 +27,10 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
write_sysreg(cntvoff, cntvoff_el2);
}
+/*
+ * Should only be called on non-VHE systems.
+ * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe().
+ */
void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
{
u64 val;
@@ -37,6 +41,10 @@ void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
write_sysreg(val, cnthctl_el2);
}
+/*
+ * Should only be called on non-VHE systems.
+ * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe().
+ */
void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
{
u64 val;
>
> Otherwise,
>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks,
-Christoffer
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch
@ 2017-12-14 12:30 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:30 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 10:02:58AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > The VHE switch function calls __timer_enable_traps and
> > __timer_disable_traps which don't do anything on VHE systems.
> > Therefore, simply remove these calls from the VHE switch function and
> > make the functions non-conditional as they are now only called from the
> > non-VHE switch path.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> > arch/arm64/kvm/hyp/switch.c | 2 --
> > virt/kvm/arm/hyp/timer-sr.c | 36 ++++++++++++++----------------------
> > 2 files changed, 14 insertions(+), 24 deletions(-)
> >
> > diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> > index e783e2371b7c..09aafa0470f7 100644
> > --- a/arch/arm64/kvm/hyp/switch.c
> > +++ b/arch/arm64/kvm/hyp/switch.c
> > @@ -358,7 +358,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> > __activate_vm(vcpu->kvm);
> >
> > __vgic_restore_state(vcpu);
> > - __timer_enable_traps(vcpu);
> >
> > /*
> > * We must restore the 32-bit state before the sysregs, thanks
> > @@ -377,7 +376,6 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
> >
> > __sysreg_save_guest_state(guest_ctxt);
> > __sysreg32_save_state(vcpu);
> > - __timer_disable_traps(vcpu);
> > __vgic_save_state(vcpu);
> >
> > __deactivate_traps(vcpu);
> > diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
> > index f24404b3c8df..752b37f9133c 100644
> > --- a/virt/kvm/arm/hyp/timer-sr.c
> > +++ b/virt/kvm/arm/hyp/timer-sr.c
> > @@ -29,32 +29,24 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
> >
> > void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
> > {
> > - /*
> > - * We don't need to do this for VHE since the host kernel runs in EL2
> > - * with HCR_EL2.TGE ==1, which makes those bits have no impact.
> > - */
> > - if (!has_vhe()) {
> > - u64 val;
> > + u64 val;
> >
> > - /* Allow physical timer/counter access for the host */
> > - val = read_sysreg(cnthctl_el2);
> > - val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> > - write_sysreg(val, cnthctl_el2);
> > - }
> > + /* Allow physical timer/counter access for the host */
> > + val = read_sysreg(cnthctl_el2);
> > + val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN;
> > + write_sysreg(val, cnthctl_el2);
> > }
> >
> > void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
> > {
> > - if (!has_vhe()) {
> > - u64 val;
> > + u64 val;
> >
> > - /*
> > - * Disallow physical timer access for the guest
> > - * Physical counter access is allowed
> > - */
> > - val = read_sysreg(cnthctl_el2);
> > - val &= ~CNTHCTL_EL1PCEN;
> > - val |= CNTHCTL_EL1PCTEN;
> > - write_sysreg(val, cnthctl_el2);
> > - }
> > + /*
> > + * Disallow physical timer access for the guest
> > + * Physical counter access is allowed
> > + */
> > + val = read_sysreg(cnthctl_el2);
> > + val &= ~CNTHCTL_EL1PCEN;
> > + val |= CNTHCTL_EL1PCTEN;
> > + write_sysreg(val, cnthctl_el2);
> > }
> >
>
> Since we're not testing for !VHE anymore, can you add a small comment
> saying that these two function are for the benefit of !VHE only and
> shouldn't be called on VHE?
Yes, absolutely:
diff --git a/virt/kvm/arm/hyp/timer-sr.c b/virt/kvm/arm/hyp/timer-sr.c
index 752b37f9133c..77754a62eb0c 100644
--- a/virt/kvm/arm/hyp/timer-sr.c
+++ b/virt/kvm/arm/hyp/timer-sr.c
@@ -27,6 +27,10 @@ void __hyp_text __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high)
write_sysreg(cntvoff, cntvoff_el2);
}
+/*
+ * Should only be called on non-VHE systems.
+ * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe().
+ */
void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
{
u64 val;
@@ -37,6 +41,10 @@ void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)
write_sysreg(val, cnthctl_el2);
}
+/*
+ * Should only be called on non-VHE systems.
+ * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe().
+ */
void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu)
{
u64 val;
>
> Otherwise,
>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks,
-Christoffer
^ permalink raw reply related [flat|nested] 158+ messages in thread
* Re: [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
2017-12-11 10:14 ` Marc Zyngier
@ 2017-12-14 12:53 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:53 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 10:14:23AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > There's a semantic difference between the EL1 registers that control
> > operation of a kernel running in EL1 and EL1 registers that only control
> > userspace execution in EL0. Since we can defer saving/restoring the
> > latter, move them into their own function.
> >
> > We also take this chance to rename the function saving/restoring the
> > remaining system register to make it clear this function deals with
> > the EL1 system registers.
> >
> > No functional change.
> >
> > Reviewed-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Added comment about sp_el0 to common save sysreg save/restore functions
> >
> > arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
> > 1 file changed, 33 insertions(+), 11 deletions(-)
> >
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index 68a7d164e5e1..bbfb4d01af88 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
> > */
> >
> > static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
> > +{
> > + ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> > +
> > + /*
> > + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> > + * therefore be saved/restored on every entry/exit to/from the guest.
> > + */
> > + ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> > +}
> > +
> > +static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
> > {
> > ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
>
> What is the rational for keeping ACTLR_EL1 as part of the user state?
>
The rationale was that I missed the note you pointed me to below, and
therefore I figured that ACTLR_EL1 couldn't affect the host kernel,
because it runs in EL2, but could affect host userspace, which is
incorrect. So I was basically just being overlay cautious.
> > ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
> > ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
> > - ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> > - ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> > }
> >
> > -static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> > +static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> > {
> > ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
> > ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
> > @@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> > }
> >
> > static hyp_alternate_select(__sysreg_call_save_host_state,
> > - __sysreg_save_state, __sysreg_do_nothing,
> > + __sysreg_save_el1_state, __sysreg_do_nothing,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
> > {
> > __sysreg_call_save_host_state()(ctxt);
> > __sysreg_save_common_state(ctxt);
> > + __sysreg_save_user_state(ctxt);
> > }
> >
> > void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
> > {
> > - __sysreg_save_state(ctxt);
> > + __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > + __sysreg_save_user_state(ctxt);
> > }
> >
> > static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> > {
> > - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
> > - write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> > - write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> > write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
> > +
> > + /*
> > + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> > + * therefore be saved/restored on every entry/exit to/from the guest.
> > + */
> > write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
> > }
> >
> > -static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> > +static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
> > +{
> > + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
>
> Same here.
>
> > + write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> > + write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> > +}
> > +
> > +static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> > {
> > write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
> > write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
> > @@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> > }
> >
> > static hyp_alternate_select(__sysreg_call_restore_host_state,
> > - __sysreg_restore_state, __sysreg_do_nothing,
> > + __sysreg_restore_el1_state, __sysreg_do_nothing,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
> > {
> > __sysreg_call_restore_host_state()(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > + __sysreg_restore_user_state(ctxt);
> > }
> >
> > void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> > {
> > - __sysreg_restore_state(ctxt);
> > + __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > + __sysreg_restore_user_state(ctxt);
> > }
> >
> > static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> >
>
> I think we should move ACTLR_EL1 to the EL1 state, allowing it to be
> lazily switched. See the note in D10.2.1 that recommends a VHE enabled
> system to have ACTLR_EL1 as a guest-only register.
Thanks for this pointer. I will adjust the code as you suggest.
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function
@ 2017-12-14 12:53 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 12:53 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 10:14:23AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > There's a semantic difference between the EL1 registers that control
> > operation of a kernel running in EL1 and EL1 registers that only control
> > userspace execution in EL0. Since we can defer saving/restoring the
> > latter, move them into their own function.
> >
> > We also take this chance to rename the function saving/restoring the
> > remaining system register to make it clear this function deals with
> > the EL1 system registers.
> >
> > No functional change.
> >
> > Reviewed-by: Andrew Jones <drjones@redhat.com>
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> >
> > Notes:
> > Changes since v1:
> > - Added comment about sp_el0 to common save sysreg save/restore functions
> >
> > arch/arm64/kvm/hyp/sysreg-sr.c | 44 +++++++++++++++++++++++++++++++-----------
> > 1 file changed, 33 insertions(+), 11 deletions(-)
> >
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index 68a7d164e5e1..bbfb4d01af88 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -33,15 +33,24 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
> > */
> >
> > static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
> > +{
> > + ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> > +
> > + /*
> > + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> > + * therefore be saved/restored on every entry/exit to/from the guest.
> > + */
> > + ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> > +}
> > +
> > +static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
> > {
> > ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
>
> What is the rational for keeping ACTLR_EL1 as part of the user state?
>
The rationale was that I missed the note you pointed me to below, and
therefore I figured that ACTLR_EL1 couldn't affect the host kernel,
because it runs in EL2, but could affect host userspace, which is
incorrect. So I was basically just being overlay cautious.
> > ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
> > ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
> > - ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
> > - ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
> > }
> >
> > -static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> > +static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> > {
> > ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2);
> > ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1);
> > @@ -70,31 +79,42 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
> > }
> >
> > static hyp_alternate_select(__sysreg_call_save_host_state,
> > - __sysreg_save_state, __sysreg_do_nothing,
> > + __sysreg_save_el1_state, __sysreg_do_nothing,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > void __hyp_text __sysreg_save_host_state(struct kvm_cpu_context *ctxt)
> > {
> > __sysreg_call_save_host_state()(ctxt);
> > __sysreg_save_common_state(ctxt);
> > + __sysreg_save_user_state(ctxt);
> > }
> >
> > void __hyp_text __sysreg_save_guest_state(struct kvm_cpu_context *ctxt)
> > {
> > - __sysreg_save_state(ctxt);
> > + __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > + __sysreg_save_user_state(ctxt);
> > }
> >
> > static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> > {
> > - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
> > - write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> > - write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> > write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
> > +
> > + /*
> > + * The host arm64 Linux uses sp_el0 to point to 'current' and it must
> > + * therefore be saved/restored on every entry/exit to/from the guest.
> > + */
> > write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
> > }
> >
> > -static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> > +static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
> > +{
> > + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
>
> Same here.
>
> > + write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
> > + write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
> > +}
> > +
> > +static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> > {
> > write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
> > write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
> > @@ -123,19 +143,21 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
> > }
> >
> > static hyp_alternate_select(__sysreg_call_restore_host_state,
> > - __sysreg_restore_state, __sysreg_do_nothing,
> > + __sysreg_restore_el1_state, __sysreg_do_nothing,
> > ARM64_HAS_VIRT_HOST_EXTN);
> >
> > void __hyp_text __sysreg_restore_host_state(struct kvm_cpu_context *ctxt)
> > {
> > __sysreg_call_restore_host_state()(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > + __sysreg_restore_user_state(ctxt);
> > }
> >
> > void __hyp_text __sysreg_restore_guest_state(struct kvm_cpu_context *ctxt)
> > {
> > - __sysreg_restore_state(ctxt);
> > + __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > + __sysreg_restore_user_state(ctxt);
> > }
> >
> > static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> >
>
> I think we should move ACTLR_EL1 to the EL1 state, allowing it to be
> lazily switched. See the note in D10.2.1 that recommends a VHE enabled
> system to have ACTLR_EL1 as a guest-only register.
Thanks for this pointer. I will adjust the code as you suggest.
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
2017-12-11 10:44 ` Marc Zyngier
@ 2017-12-14 13:46 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 13:46 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 10:44:59AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
> > can return to the host in EL1 in the same state and location where we
> > issued a hypercall to EL2, but these registers don't contain anything
> > important on VHE, because all of the host runs in EL2. Therefore,
>
> If I may refine the rational: ELR_EL2 and SPSR_EL2 are not useful here
> because we never enter a guest as a result of an exception entry that
> would be directly handled by KVM. The kernel entry code already saves
> ELR_EL1/SPSR_EL1 on exception entry, which is enough.
>
Indeed, I was being a bit loosey-goosey here.
> > factor out these registers into separate save/restore functions, making
> > it easy to exclude them from the VHE world-switch path later on.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> > arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
> > 1 file changed, 13 insertions(+)
> >
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index a12112494f75..479de0f0dd07 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> > ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
> > ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
> > ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
> > +}
> > +
> > +static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
> > +{
> > ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
> > ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
> > }
> > @@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
> > __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > __sysreg_save_user_state(ctxt);
> > + __sysreg_save_el2_return_state(ctxt);
> > }
> >
> > void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
> > @@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
> > __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > __sysreg_save_user_state(ctxt);
> > + __sysreg_save_el2_return_state(ctxt);
> > }
> >
> > static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> > @@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> > write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
> > write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
> > write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
> > +}
> > +
> > +static void __hyp_text
> > +__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
> > +{
> > write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
> > write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
> > }
> > @@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > __sysreg_restore_user_state(ctxt);
> > + __sysreg_restore_el2_return_state(ctxt);
> > }
> >
> > void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
> > @@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > __sysreg_restore_user_state(ctxt);
> > + __sysreg_restore_el2_return_state(ctxt);
> > }
> >
> > static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> >
>
> Otherwise:
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
@ 2017-12-14 13:46 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-14 13:46 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 10:44:59AM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > On non-VHE systems we need to save the ELR_EL2 and SPSR_EL2 so that we
> > can return to the host in EL1 in the same state and location where we
> > issued a hypercall to EL2, but these registers don't contain anything
> > important on VHE, because all of the host runs in EL2. Therefore,
>
> If I may refine the rational: ELR_EL2 and SPSR_EL2 are not useful here
> because we never enter a guest as a result of an exception entry that
> would be directly handled by KVM. The kernel entry code already saves
> ELR_EL1/SPSR_EL1 on exception entry, which is enough.
>
Indeed, I was being a bit loosey-goosey here.
> > factor out these registers into separate save/restore functions, making
> > it easy to exclude them from the VHE world-switch path later on.
> >
> > Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> > ---
> > arch/arm64/kvm/hyp/sysreg-sr.c | 13 +++++++++++++
> > 1 file changed, 13 insertions(+)
> >
> > diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
> > index a12112494f75..479de0f0dd07 100644
> > --- a/arch/arm64/kvm/hyp/sysreg-sr.c
> > +++ b/arch/arm64/kvm/hyp/sysreg-sr.c
> > @@ -71,6 +71,10 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
> > ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
> > ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
> > ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
> > +}
> > +
> > +static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
> > +{
> > ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
> > ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
> > }
> > @@ -80,6 +84,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
> > __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > __sysreg_save_user_state(ctxt);
> > + __sysreg_save_el2_return_state(ctxt);
> > }
> >
> > void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt)
> > @@ -93,6 +98,7 @@ void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt)
> > __sysreg_save_el1_state(ctxt);
> > __sysreg_save_common_state(ctxt);
> > __sysreg_save_user_state(ctxt);
> > + __sysreg_save_el2_return_state(ctxt);
> > }
> >
> > static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
> > @@ -137,6 +143,11 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
> > write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
> > write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
> > write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
> > +}
> > +
> > +static void __hyp_text
> > +__sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
> > +{
> > write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
> > write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
> > }
> > @@ -146,6 +157,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > __sysreg_restore_user_state(ctxt);
> > + __sysreg_restore_el2_return_state(ctxt);
> > }
> >
> > void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt)
> > @@ -159,6 +171,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
> > __sysreg_restore_el1_state(ctxt);
> > __sysreg_restore_common_state(ctxt);
> > __sysreg_restore_user_state(ctxt);
> > + __sysreg_restore_el2_return_state(ctxt);
> > }
> >
> > static void __hyp_text __fpsimd32_save_state(struct kvm_cpu_context *ctxt)
> >
>
> Otherwise:
>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* Re: [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
2017-12-11 13:20 ` Marc Zyngier
@ 2017-12-15 16:29 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-15 16:29 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 01:20:03PM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > Some system registers do not affect the host kernel's execution and can
> > therefore be loaded when we are about to run a VCPU and we don't have to
> > restore the host state to the hardware before the time when we are
> > actually about to return to userspace or schedule out the VCPU thread.
> >
> > The EL1 system registers and the userspace state registers, which only
> > affect EL0 execution, do not affect the host kernel's execution.
> >
> > The 32-bit system registers are not used by a VHE host kernel and
> > therefore don't need to be saved/restored on every entry/exit to/from
> > the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
>
> Note that they are not used by the !VHE host kernel either, and I
> believe they could be deferred too, although that would imply a round
> trip to HYP to save/restore them. We already have such a hook there when
> configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
>
This turned out to be a bit trickier than I initial thought, and I think
it also revealed a bug around running 32-bit guests on VHE systems,
related to how DBGVCR32_EL2 is currently handled.
The result will look something like this (depending a bit on the rework
for the system register accesses discussed in the earlier patch):
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 0488841c6341..de98b99b1eec 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -92,13 +92,18 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
{
if (vcpu_mode_is_32bit(vcpu)) {
- if (vcpu->arch.sysregs_loaded_on_cpu)
- __sysreg32_save_state(vcpu);
+ bool loaded;
+
+ preempt_disable();
+ loaded = vcpu->arch.sysregs32_loaded_on_cpu;
+ if (loaded)
+ kvm_call_hyp(__sysreg32_save_state, vcpu);
*vcpu_spsr32(vcpu) = val;
- if (vcpu->arch.sysregs_loaded_on_cpu)
- __sysreg32_restore_state(vcpu);
+ if (loaded)
+ kvm_call_hyp(__sysreg32_restore_state, vcpu);
+ preempt_enable();
}
if (vcpu->arch.sysregs_loaded_on_cpu)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 992c19816893..bc116d6c8756 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -283,6 +283,7 @@ struct kvm_vcpu_arch {
/* True when deferrable sysregs are loaded on the physical CPU,
* see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
bool sysregs_loaded_on_cpu;
+ bool sysregs32_loaded_on_cpu;
};
#define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index ee87115eb12f..d80037b655b4 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -20,6 +20,7 @@
#include <asm/debug-monitors.h>
#include <asm/kvm_asm.h>
+#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
#define read_debug(r,n) read_sysreg(r##n##_el1)
@@ -169,6 +170,9 @@ void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu)
__debug_save_state(vcpu, host_dbg, host_ctxt);
__debug_restore_state(vcpu, guest_dbg, guest_ctxt);
+
+ if (vcpu_el1_is_32bit(vcpu))
+ write_sysreg(vcpu->arch.ctxt.sys_regs[DBGVCR32_EL2], dbgvcr32_el2);
}
void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
@@ -192,6 +196,9 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
__debug_save_state(vcpu, guest_dbg, guest_ctxt);
__debug_restore_state(vcpu, host_dbg, host_ctxt);
+ if (vcpu_el1_is_32bit(vcpu))
+ vcpu->arch.ctxt.sys_regs[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+
vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 05f266b505ce..48dc2c0b10d0 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -402,7 +402,6 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* We must restore the 32-bit state before the sysregs, thanks
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
- __sysreg32_restore_state(vcpu);
__sysreg_restore_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -414,7 +413,6 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
__sysreg_save_state_nvhe(guest_ctxt);
- __sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 3c62c1c14b22..42eb0cc68079 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -181,7 +181,9 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
- if (!vcpu_el1_is_32bit(vcpu))
+ vcpu = kern_hyp_va(vcpu);
+
+ if (!vcpu_el1_is_32bit(vcpu) || !vcpu->arch.sysregs32_loaded_on_cpu)
return;
spsr = vcpu->arch.ctxt.gp_regs.spsr;
@@ -195,15 +197,18 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+ sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+
+ vcpu->arch.sysregs32_loaded_on_cpu = false;
}
void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
- if (!vcpu_el1_is_32bit(vcpu))
+ vcpu = kern_hyp_va(vcpu);
+
+ if (!vcpu_el1_is_32bit(vcpu) || vcpu->arch.sysregs32_loaded_on_cpu)
return;
spsr = vcpu->arch.ctxt.gp_regs.spsr;
@@ -217,8 +222,9 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
write_sysreg(sysreg[DACR32_EL2], dacr32_el2);
write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2);
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
+ write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
+
+ vcpu->arch.sysregs32_loaded_on_cpu = true;
}
/**
@@ -237,19 +243,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+ /*
+ * Erratum #852523 (Cortex-A57) or #853709 (Cortex-A72) requires us to
+ * restore the 32-bit state before the sysregs, which will happen on
+ * both VHE (below) and on non-VHE in the world-switch path.
+ */
+ kvm_call_hyp(__sysreg32_restore_state, vcpu);
+
if (!has_vhe())
return;
__sysreg_save_user_state(host_ctxt);
-
- /*
- * Load guest EL1 and user state
- *
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
+ /* Load guest EL1 and user state */
__sysreg_restore_user_state(guest_ctxt);
__sysreg_restore_el1_state(guest_ctxt);
@@ -283,12 +289,13 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
vcpu->arch.guest_vfp_loaded = 0;
}
+ kvm_call_hyp(__sysreg32_save_state, vcpu);
+
if (!has_vhe())
return;
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);
- __sysreg32_save_state(vcpu);
/* Restore host user state */
__sysreg_restore_user_state(host_ctxt);
For now, I'll stash this as a separate patch as it will improve
readability and make it easier to bisect things.
Thanks,
-Christoffer
^ permalink raw reply related [flat|nested] 158+ messages in thread
* [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
@ 2017-12-15 16:29 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-15 16:29 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 01:20:03PM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > Some system registers do not affect the host kernel's execution and can
> > therefore be loaded when we are about to run a VCPU and we don't have to
> > restore the host state to the hardware before the time when we are
> > actually about to return to userspace or schedule out the VCPU thread.
> >
> > The EL1 system registers and the userspace state registers, which only
> > affect EL0 execution, do not affect the host kernel's execution.
> >
> > The 32-bit system registers are not used by a VHE host kernel and
> > therefore don't need to be saved/restored on every entry/exit to/from
> > the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
>
> Note that they are not used by the !VHE host kernel either, and I
> believe they could be deferred too, although that would imply a round
> trip to HYP to save/restore them. We already have such a hook there when
> configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
>
This turned out to be a bit trickier than I initial thought, and I think
it also revealed a bug around running 32-bit guests on VHE systems,
related to how DBGVCR32_EL2 is currently handled.
The result will look something like this (depending a bit on the rework
for the system register accesses discussed in the earlier patch):
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 0488841c6341..de98b99b1eec 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -92,13 +92,18 @@ static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
static inline void vcpu_set_spsr(struct kvm_vcpu *vcpu, u64 val)
{
if (vcpu_mode_is_32bit(vcpu)) {
- if (vcpu->arch.sysregs_loaded_on_cpu)
- __sysreg32_save_state(vcpu);
+ bool loaded;
+
+ preempt_disable();
+ loaded = vcpu->arch.sysregs32_loaded_on_cpu;
+ if (loaded)
+ kvm_call_hyp(__sysreg32_save_state, vcpu);
*vcpu_spsr32(vcpu) = val;
- if (vcpu->arch.sysregs_loaded_on_cpu)
- __sysreg32_restore_state(vcpu);
+ if (loaded)
+ kvm_call_hyp(__sysreg32_restore_state, vcpu);
+ preempt_enable();
}
if (vcpu->arch.sysregs_loaded_on_cpu)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 992c19816893..bc116d6c8756 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -283,6 +283,7 @@ struct kvm_vcpu_arch {
/* True when deferrable sysregs are loaded on the physical CPU,
* see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */
bool sysregs_loaded_on_cpu;
+ bool sysregs32_loaded_on_cpu;
};
#define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs)
diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index ee87115eb12f..d80037b655b4 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -20,6 +20,7 @@
#include <asm/debug-monitors.h>
#include <asm/kvm_asm.h>
+#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
#define read_debug(r,n) read_sysreg(r##n##_el1)
@@ -169,6 +170,9 @@ void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu)
__debug_save_state(vcpu, host_dbg, host_ctxt);
__debug_restore_state(vcpu, guest_dbg, guest_ctxt);
+
+ if (vcpu_el1_is_32bit(vcpu))
+ write_sysreg(vcpu->arch.ctxt.sys_regs[DBGVCR32_EL2], dbgvcr32_el2);
}
void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
@@ -192,6 +196,9 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu)
__debug_save_state(vcpu, guest_dbg, guest_ctxt);
__debug_restore_state(vcpu, host_dbg, host_ctxt);
+ if (vcpu_el1_is_32bit(vcpu))
+ vcpu->arch.ctxt.sys_regs[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+
vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY;
}
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 05f266b505ce..48dc2c0b10d0 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -402,7 +402,6 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
* We must restore the 32-bit state before the sysregs, thanks
* to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
*/
- __sysreg32_restore_state(vcpu);
__sysreg_restore_state_nvhe(guest_ctxt);
__debug_switch_to_guest(vcpu);
@@ -414,7 +413,6 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu)
} while (fixup_guest_exit(vcpu, &exit_code));
__sysreg_save_state_nvhe(guest_ctxt);
- __sysreg32_save_state(vcpu);
__timer_disable_traps(vcpu);
__vgic_save_state(vcpu);
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 3c62c1c14b22..42eb0cc68079 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -181,7 +181,9 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
- if (!vcpu_el1_is_32bit(vcpu))
+ vcpu = kern_hyp_va(vcpu);
+
+ if (!vcpu_el1_is_32bit(vcpu) || !vcpu->arch.sysregs32_loaded_on_cpu)
return;
spsr = vcpu->arch.ctxt.gp_regs.spsr;
@@ -195,15 +197,18 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu)
sysreg[DACR32_EL2] = read_sysreg(dacr32_el2);
sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2);
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+ sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2);
+
+ vcpu->arch.sysregs32_loaded_on_cpu = false;
}
void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
{
u64 *spsr, *sysreg;
- if (!vcpu_el1_is_32bit(vcpu))
+ vcpu = kern_hyp_va(vcpu);
+
+ if (!vcpu_el1_is_32bit(vcpu) || vcpu->arch.sysregs32_loaded_on_cpu)
return;
spsr = vcpu->arch.ctxt.gp_regs.spsr;
@@ -217,8 +222,9 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
write_sysreg(sysreg[DACR32_EL2], dacr32_el2);
write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2);
- if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
- write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
+ write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
+
+ vcpu->arch.sysregs32_loaded_on_cpu = true;
}
/**
@@ -237,19 +243,19 @@ void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *host_ctxt = vcpu->arch.host_cpu_context;
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
+ /*
+ * Erratum #852523 (Cortex-A57) or #853709 (Cortex-A72) requires us to
+ * restore the 32-bit state before the sysregs, which will happen on
+ * both VHE (below) and on non-VHE in the world-switch path.
+ */
+ kvm_call_hyp(__sysreg32_restore_state, vcpu);
+
if (!has_vhe())
return;
__sysreg_save_user_state(host_ctxt);
-
- /*
- * Load guest EL1 and user state
- *
- * We must restore the 32-bit state before the sysregs, thanks
- * to erratum #852523 (Cortex-A57) or #853709 (Cortex-A72).
- */
- __sysreg32_restore_state(vcpu);
+ /* Load guest EL1 and user state */
__sysreg_restore_user_state(guest_ctxt);
__sysreg_restore_el1_state(guest_ctxt);
@@ -283,12 +289,13 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu)
vcpu->arch.guest_vfp_loaded = 0;
}
+ kvm_call_hyp(__sysreg32_save_state, vcpu);
+
if (!has_vhe())
return;
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);
- __sysreg32_save_state(vcpu);
/* Restore host user state */
__sysreg_restore_user_state(host_ctxt);
For now, I'll stash this as a separate patch as it will improve
readability and make it easier to bisect things.
Thanks,
-Christoffer
^ permalink raw reply related [flat|nested] 158+ messages in thread
* Re: [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
2017-12-11 13:20 ` Marc Zyngier
@ 2017-12-29 16:39 ` Christoffer Dall
-1 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-29 16:39 UTC (permalink / raw)
To: Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm, Shih-Wei Li, Andrew Jones
On Mon, Dec 11, 2017 at 01:20:03PM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > Some system registers do not affect the host kernel's execution and can
> > therefore be loaded when we are about to run a VCPU and we don't have to
> > restore the host state to the hardware before the time when we are
> > actually about to return to userspace or schedule out the VCPU thread.
> >
> > The EL1 system registers and the userspace state registers, which only
> > affect EL0 execution, do not affect the host kernel's execution.
> >
> > The 32-bit system registers are not used by a VHE host kernel and
> > therefore don't need to be saved/restored on every entry/exit to/from
> > the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
>
> Note that they are not used by the !VHE host kernel either, and I
> believe they could be deferred too, although that would imply a round
> trip to HYP to save/restore them. We already have such a hook there when
> configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
>
> What do you think?
>
This turned out to be a bit of a mess. We can do this, but it conflicts
with the idea of hiding the sysregs_loaded_on_cpu handling based on
classifying a system register as deferrable or immediate.
What we'd need to do is to add a separate flag for the system registers
which are deferred on !VHE as well as VHE, for example
sysregs32_loaded_on_cpu (or something more generic like
nvhe_vhe_sysregs_loaded_on_cpu - in lack of a better name). We'd then
have to duplicate the DEFERRED macro for this other category of sysregs,
which checks our new flag instead of sysregs_loaded_on_cpu, and we'd
have to map all the accessors in Hyp, and do kvm_call_hyp() around all
the accessors (which could hurt normal performance) or expand the
deferred macros to output two functions, where the non-hyp calls into
the hyp one if the system register is loaded on the cpu.
Of course, it's arguable that all this is not needed because the 32-bit
system registers are not actually used yet for emulation while the state
could be loaded on the cpu, so it doesn't matter, but then we're back to
my original approach where all the code just has to know the state of a
register and when modifying KVM code we have to think carefully about
that.
Basically, I don't think the complexity is worth the effort given that
this only optimizes 32-bit guests, and I think we should just leave this
part out.
Thoughts?
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
* [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE
@ 2017-12-29 16:39 ` Christoffer Dall
0 siblings, 0 replies; 158+ messages in thread
From: Christoffer Dall @ 2017-12-29 16:39 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Dec 11, 2017 at 01:20:03PM +0000, Marc Zyngier wrote:
> On 07/12/17 17:06, Christoffer Dall wrote:
> > Some system registers do not affect the host kernel's execution and can
> > therefore be loaded when we are about to run a VCPU and we don't have to
> > restore the host state to the hardware before the time when we are
> > actually about to return to userspace or schedule out the VCPU thread.
> >
> > The EL1 system registers and the userspace state registers, which only
> > affect EL0 execution, do not affect the host kernel's execution.
> >
> > The 32-bit system registers are not used by a VHE host kernel and
> > therefore don't need to be saved/restored on every entry/exit to/from
> > the guest, but can be deferred to vcpu_load and vcpu_put, respectively.
>
> Note that they are not used by the !VHE host kernel either, and I
> believe they could be deferred too, although that would imply a round
> trip to HYP to save/restore them. We already have such a hook there when
> configuring ICH_VMCR_EL2, so we may not need much of a new infrastructure.
>
> What do you think?
>
This turned out to be a bit of a mess. We can do this, but it conflicts
with the idea of hiding the sysregs_loaded_on_cpu handling based on
classifying a system register as deferrable or immediate.
What we'd need to do is to add a separate flag for the system registers
which are deferred on !VHE as well as VHE, for example
sysregs32_loaded_on_cpu (or something more generic like
nvhe_vhe_sysregs_loaded_on_cpu - in lack of a better name). We'd then
have to duplicate the DEFERRED macro for this other category of sysregs,
which checks our new flag instead of sysregs_loaded_on_cpu, and we'd
have to map all the accessors in Hyp, and do kvm_call_hyp() around all
the accessors (which could hurt normal performance) or expand the
deferred macros to output two functions, where the non-hyp calls into
the hyp one if the system register is loaded on the cpu.
Of course, it's arguable that all this is not needed because the 32-bit
system registers are not actually used yet for emulation while the state
could be loaded on the cpu, so it doesn't matter, but then we're back to
my original approach where all the code just has to know the state of a
register and when modifying KVM code we have to think carefully about
that.
Basically, I don't think the complexity is worth the effort given that
this only optimizes 32-bit guests, and I think we should just leave this
part out.
Thoughts?
Thanks,
-Christoffer
^ permalink raw reply [flat|nested] 158+ messages in thread
end of thread, other threads:[~2017-12-29 16:39 UTC | newest]
Thread overview: 158+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-07 17:05 [PATCH v2 00/36] Optimize KVM/ARM for VHE systems Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-07 17:05 ` [PATCH v2 01/36] KVM: arm64: Avoid storing the vcpu pointer on the stack Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-09 17:19 ` Marc Zyngier
2017-12-09 17:19 ` Marc Zyngier
2017-12-11 9:30 ` Christoffer Dall
2017-12-11 9:30 ` Christoffer Dall
2017-12-11 9:35 ` Marc Zyngier
2017-12-11 9:35 ` Marc Zyngier
2017-12-07 17:05 ` [PATCH v2 02/36] KVM: arm64: Rework hyp_panic for VHE and non-VHE Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-09 17:24 ` Marc Zyngier
2017-12-09 17:24 ` Marc Zyngier
2017-12-07 17:05 ` [PATCH v2 03/36] KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-07 17:05 ` [PATCH v2 04/36] KVM: arm/arm64: Get rid of vcpu->arch.irq_lines Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-07 17:05 ` [PATCH v2 05/36] KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs Christoffer Dall
2017-12-07 17:05 ` Christoffer Dall
2017-12-09 17:30 ` Marc Zyngier
2017-12-09 17:30 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 06/36] KVM: arm64: Defer restoring host VFP state to vcpu_put Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-09 17:37 ` Marc Zyngier
2017-12-09 17:37 ` Marc Zyngier
2017-12-11 9:31 ` Christoffer Dall
2017-12-11 9:31 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 07/36] KVM: arm64: Move debug dirty flag calculation out of world switch Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-09 19:20 ` Marc Zyngier
2017-12-09 19:20 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 08/36] KVM: arm64: Slightly improve debug save/restore functions Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-09 19:37 ` Marc Zyngier
2017-12-09 19:37 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 09/36] KVM: arm64: Improve debug register save/restore flow Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 9:40 ` Marc Zyngier
2017-12-11 9:40 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 10/36] KVM: arm64: Factor out fault info population and gic workarounds Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 9:45 ` Marc Zyngier
2017-12-11 9:45 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 11/36] KVM: arm64: Introduce VHE-specific kvm_vcpu_run Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 9:53 ` Marc Zyngier
2017-12-11 9:53 ` Marc Zyngier
2017-12-14 12:12 ` Christoffer Dall
2017-12-14 12:12 ` Christoffer Dall
2017-12-14 12:17 ` Marc Zyngier
2017-12-14 12:17 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 12/36] KVM: arm64: Remove kern_hyp_va() use in VHE switch function Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 9:54 ` Marc Zyngier
2017-12-11 9:54 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 13/36] KVM: arm64: Don't deactivate VM on VHE systems Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 9:58 ` Marc Zyngier
2017-12-11 9:58 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 14/36] KVM: arm64: Remove noop calls to timer save/restore from VHE switch Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:02 ` Marc Zyngier
2017-12-11 10:02 ` Marc Zyngier
2017-12-14 12:30 ` Christoffer Dall
2017-12-14 12:30 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 15/36] KVM: arm64: Move userspace system registers into separate function Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:14 ` Marc Zyngier
2017-12-11 10:14 ` Marc Zyngier
2017-12-14 12:53 ` Christoffer Dall
2017-12-14 12:53 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 16/36] KVM: arm64: Rewrite sysreg alternatives to static keys Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:15 ` Marc Zyngier
2017-12-11 10:15 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 17/36] KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functions Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:22 ` Marc Zyngier
2017-12-11 10:22 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 18/36] KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:30 ` Marc Zyngier
2017-12-11 10:30 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 19/36] KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:40 ` Marc Zyngier
2017-12-11 10:40 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 20/36] KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:44 ` Marc Zyngier
2017-12-11 10:44 ` Marc Zyngier
2017-12-14 13:46 ` Christoffer Dall
2017-12-14 13:46 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 21/36] KVM: arm64: Change 32-bit handling of VM system registers Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 10:57 ` Marc Zyngier
2017-12-11 10:57 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 22/36] KVM: arm64: Prepare to handle traps on deferred VM sysregs Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 11:10 ` Marc Zyngier
2017-12-11 11:10 ` Marc Zyngier
2017-12-11 11:24 ` Christoffer Dall
2017-12-11 11:24 ` Christoffer Dall
2017-12-11 11:46 ` Marc Zyngier
2017-12-11 11:46 ` Marc Zyngier
2017-12-12 13:08 ` Marc Zyngier
2017-12-12 13:08 ` Marc Zyngier
2017-12-12 15:46 ` Christoffer Dall
2017-12-12 15:46 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 23/36] KVM: arm64: Prepare to handle traps on deferred EL0 sysregs Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 24/36] KVM: arm64: Prepare to handle traps on remaining deferred EL1 sysregs Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 25/36] KVM: arm64: Prepare to handle traps on deferred AArch32 sysregs Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 26/36] KVM: arm64: Defer saving/restoring system registers to vcpu load/put on VHE Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 13:20 ` Marc Zyngier
2017-12-11 13:20 ` Marc Zyngier
2017-12-15 16:29 ` Christoffer Dall
2017-12-15 16:29 ` Christoffer Dall
2017-12-29 16:39 ` Christoffer Dall
2017-12-29 16:39 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 27/36] KVM: arm64: Move common VHE/non-VHE trap config in separate functions Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 13:53 ` Marc Zyngier
2017-12-11 13:53 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 28/36] KVM: arm64: Configure FPSIMD traps on vcpu load/put for VHE Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 14:18 ` Marc Zyngier
2017-12-11 14:18 ` Marc Zyngier
2017-12-07 17:06 ` [PATCH v2 29/36] KVM: arm64: Configure c15, PMU, and debug register traps on cpu " Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 30/36] KVM: arm64: Separate activate_traps and deactive_traps for VHE and non-VHE Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 31/36] KVM: arm/arm64: Get rid of vgic_elrsr Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 32/36] KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 33/36] KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64 Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 34/36] KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on VHE Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 35/36] KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-07 17:06 ` [PATCH v2 36/36] KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs Christoffer Dall
2017-12-07 17:06 ` Christoffer Dall
2017-12-11 14:43 ` [PATCH v2 00/36] Optimize KVM/ARM for VHE systems Yury Norov
2017-12-11 14:43 ` Yury Norov
2017-12-11 14:56 ` Marc Zyngier
2017-12-11 14:56 ` Marc Zyngier
2017-12-11 15:14 ` Yury Norov
2017-12-11 15:14 ` Yury Norov
2017-12-11 15:34 ` Christoffer Dall
2017-12-11 15:34 ` Christoffer Dall
2017-12-11 16:30 ` Yury Norov
2017-12-11 16:30 ` Yury Norov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.