On Fri, 11 Dec 2020 at 00:00, Marc Zyngier wrote: > > This is a rework of the NV series that I posted 10 months ago[1], as a > lot of the KVM code has changed since, and the series apply anymore > (not that anybody really cares as the the HW is, as usual, made of > unobtainium...). > > From the previous version: > > - Integration with the new page-table code > - New exception injection code > - No more messing with the nVHE code > - No AArch32!!!! > - Rebased on v5.10-rc4 + kvmarm/next for 5.11 > > From a functionality perspective, you can expect a L2 guest to work, > but don't even think of L3, as we only partially emulate the > ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, > as well as anything that would require a Stage-1 PTW. What we want to > achieve is that with NV disabled, there is no performance overhead and > no regression. > > The series is roughly divided in 5 parts: exception handling, memory > virtualization, interrupts and timers for ARMv8.3, followed by the > ARMv8.4 support. There are of course some dependencies, but you'll > hopefully get the gist of it. > > For the most courageous of you, I've put out a branch[2]. Of course, > you'll need some userspace. Andre maintains a hacked version of > kvmtool[3] that takes a --nested option, allowing the guest to be > started at EL2. You can run the whole stack in the Foundation > model. Don't be in a hurry ;-). > Hi Marc, I got a kernel BUG message when booting the L2 guest kernel with the kvmtool on a FVP setup. Could you help have a look about the BUG message as well as my environment configuration? I think It probably caused by some local configurations of the FVP setup. Thanks, Haibo ------------------------------------------------------------------------------------------------------------------------------ 1 # lkvm run -k ./Image -m 320 -c 2 --name guest-229 2 [ 77.714334] kernel BUG at arch/arm64/kernel/traps.c:407! 3 [ 77.715758] Internal error: Oops - BUG: 0 [#1] SMP 4 [ 77.716900] Modules linked in: 5 [ 77.717754] CPU: 0 PID: 229 Comm: lkvm Not tainted 5.11.0-rc1+ #2 6 [ 77.719193] Hardware name: linux,dummy-virt (DT) 7 [ 77.720300] pstate: 00400009 (nzcv daif +PAN -UAO -TCO BTYPE=--) 8 [ 77.722006] pc : do_undefinstr+0x4d0/0x5b0 9 [ 77.722900] lr : do_undefinstr+0x380/0x5b0 10 [ 77.723800] sp : ffff8000122ab8c0 11 [ 77.724674] x29: ffff8000122ab8c0 x28: ffff0000048c49c0 12 [ 77.725824] x27: 0000000000000000 x26: 0000000000000000 13 [ 77.727050] x25: 0000000000000000 x24: ffff000004b9d520 14 [ 77.728217] x23: 0000000080400009 x22: ffff8000100e42d0 15 [ 77.729700] x21: ffff8000122abaa0 x20: ffff0000048c49c0 16 [ 77.730856] x19: ffff8000122ab950 x18: 0000000000000000 17 [ 77.732047] x17: 0000000000000000 x16: 0000000000000000 18 [ 77.733276] x15: 0000000000000000 x14: 0000000000000000 19 [ 77.734423] x13: 0000000000000000 x12: 0000000000000000 20 [ 77.735629] x11: 0000000000000000 x10: 0000000000000000 21 [ 77.737200] x9 : ffff8000102fb0a0 x8 : ffff000004b9d730 22 [ 77.738369] x7 : 0000000000000000 x6 : 0000000080000000 23 [ 77.739533] x5 : 0000000000000000 x4 : 000000000000001f 24 [ 77.740700] x3 : 0000000000000000 x2 : ffff800011a05b80 25 [ 77.741963] x1 : ffff800011c09110 x0 : 0000000080400009 26 [ 77.743142] Call trace: 27 [ 77.743700] do_undefinstr+0x4d0/0x5b0 28 [ 77.744662] el1_undef+0x2c/0x48 29 [ 77.745832] el1_sync_handler+0x80/0xb0 30 [ 77.746736] el1_sync+0x74/0x100 31 [ 77.747612] reset_pmcr+0x8/0x88 32 [ 77.748471] kvm_reset_vcpu+0x128/0x290 33 [ 77.749425] kvm_arch_vcpu_ioctl+0x698/0x6c8 34 [ 77.750561] kvm_vcpu_ioctl+0x3c8/0x5f0 35 [ 77.751472] __arm64_sys_ioctl+0xa8/0xe8 36 [ 77.752375] el0_svc_common.constprop.0+0x78/0x188 37 [ 77.753920] do_el0_svc+0x28/0x88 38 [ 77.754821] el0_svc+0x1c/0x28 39 [ 77.755723] el0_sync_handler+0xa8/0xb0 40 [ 77.756678] el0_sync+0x160/0x180 41 [ 77.757582] Code: d2801400 17ffffa9 a9025bf5 f9001bf7 (d4210000) 42 [ 77.758970] ---[ end trace a8cdeac7ff43b5a5 ]--- 43 [ 77.763012] ------------[ cut here ]------------ 44 [ 77.769730] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:632 rcu_eqs_enter.isra.0+0x68/0x70 45 [ 77.771552] Modules linked in: 46 [ 77.772420] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D 5.11.0-rc1+ #2 47 [ 77.774155] Hardware name: linux,dummy-virt (DT) 48 [ 77.775301] pstate: 204003c9 (nzCv DAIF +PAN -UAO -TCO BTYPE=--) 49 [ 77.776809] pc : rcu_eqs_enter.isra.0+0x68/0x70 50 [ 77.777999] lr : rcu_idle_enter+0x14/0x20 51 [ 77.778905] sp : ffff8000119f3e80 52 [ 77.779759] x29: ffff8000119f3e80 x28: ffff00001fefca00 53 [ 77.785541] x27: ffff800011a02ec0 x26: 0000000000000000 54 [ 77.786700] x25: 0000000000000000 x24: ffff8000119f9528 55 [ 77.787915] x23: ffff800011a02ec0 x22: ffff8000115837b8 56 [ 77.789166] x21: ffff8000119f9500 x20: 0000000000000000 57 [ 77.790345] x19: ffff80001156e000 x18: 0000000000000010 58 [ 77.791560] x17: 0000000000000000 x16: 0000000000000000 59 [ 77.792775] x15: 0000000000000000 x14: 0000000000000000 60 [ 77.793978] x13: 0000000000000002 x12: 0000000000000000 61 [ 77.795153] x11: 0000000000000001 x10: 0000000000000ae0 62 [ 77.805799] x9 : ffff800010db2d94 x8 : ffff800011a03a00 63 [ 77.813399] x7 : 0000000000000000 x6 : 000000166ce4e201 64 [ 77.821100] x5 : 00ffffffffffffff x4 : ffff80000e93e000 65 [ 77.828958] x3 : 4000000000000002 x2 : 4000000000000000 66 [ 77.836659] x1 : ffff800011585ac0 x0 : ffff00001fec3ac0 67 [ 77.844360] Call trace: 68 [ 77.847910] rcu_eqs_enter.isra.0+0x68/0x70 69 [ 77.851519] rcu_idle_enter+0x14/0x20 70 [ 77.854989] default_idle_call+0x3c/0x16c 71 [ 77.858460] do_idle+0x214/0x260 72 [ 77.862200] cpu_startup_entry+0x2c/0x90 73 [ 77.866046] rest_init+0xc4/0xd0 74 [ 77.869654] arch_call_rest_init+0x14/0x1c 75 [ 77.873334] start_kernel+0x80c/0x844 76 [ 77.877065] ---[ end trace a8cdeac7ff43b5a6 ]--- --------------------------------------------------------------------------------------------------------------- > And to be clear: although Jintack and Christoffer have written tons of > the stuff originaly, I'm the one responsible for breaking it! > > [1] https://lore.kernel.org/r/20200211174938.27809-1-maz@kernel.org > [2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-5.11.-WIP > [3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5 > > Andre Przywara (1): > KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ > > Christoffer Dall (15): > KVM: arm64: nv: Introduce nested virtualization VCPU feature > KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set > KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x > KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state > KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values > KVM: arm64: nv: Handle trapped ERET from virtual EL2 > KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor > KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 > KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 > changes > KVM: arm64: nv: Implement nested Stage-2 page table walk logic > KVM: arm64: nv: Unmap/flush shadow stage 2 page tables > KVM: arm64: nv: arch_timer: Support hyp timer emulation > KVM: arm64: nv: vgic: Emulate the HW bit in software > KVM: arm64: nv: Add nested GICv3 tracepoints > KVM: arm64: nv: Sync nested timer state with ARMv8.4 > > Jintack Lim (19): > arm64: Add ARM64_HAS_NESTED_VIRT cpufeature > KVM: arm64: nv: Handle HCR_EL2.NV system register traps > KVM: arm64: nv: Support virtual EL2 exceptions > KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 > KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 > KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 > KVM: arm64: nv: Handle PSCI call via smc from the guest > KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting > KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings > KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting > KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings > KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting > KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 > KVM: arm64: nv: Configure HCR_EL2 for nested virtualization > KVM: arm64: nv: Introduce sys_reg_desc.forward_trap > KVM: arm64: nv: Set a handler for the system instruction traps > KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 > KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 > KVM: arm64: nv: Nested GICv3 Support > > Marc Zyngier (31): > KVM: arm64: nv: Add EL2 system registers to vcpu context > KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers > KVM: arm64: nv: Handle virtual EL2 registers in > vcpu_read/write_sys_reg() > KVM: arm64: nv: Handle SPSR_EL2 specially > KVM: arm64: nv: Handle HCR_EL2.E2H specially > KVM: arm64: nv: Save/Restore vEL2 sysregs > KVM: arm64: nv: Forward debug traps to the nested guest > KVM: arm64: nv: Filter out unsupported features from ID regs > KVM: arm64: nv: Hide RAS from nested guests > KVM: arm64: nv: Support multiple nested Stage-2 mmu structures > KVM: arm64: nv: Handle shadow stage 2 page faults > KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's > KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's > KVM: arm64: nv: Add handling of EL2-specific timer registers > KVM: arm64: nv: Load timer before the GIC > KVM: arm64: nv: Don't load the GICv4 context on entering a nested > guest > KVM: arm64: nv: Implement maintenance interrupt forwarding > KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT > KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation > KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like > information > KVM: arm64: Allow populating S2 SW bits > KVM: arm64: nv: Tag shadow S2 entries with nested level > KVM: arm64: nv: Add include containing the VNCR_EL2 offsets > KVM: arm64: Map VNCR-capable registers to a separate page > KVM: arm64: nv: Move nested vgic state into the sysreg file > KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature > KVM: arm64: nv: Synchronize PSTATE early on exit > KVM: arm64: nv: Allocate VNCR page when required > KVM: arm64: nv: Enable ARMv8.4-NV support > KVM: arm64: nv: Fast-track 'InHost' exception returns > KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests > > .../admin-guide/kernel-parameters.txt | 4 + > .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- > arch/arm64/include/asm/cpucaps.h | 2 + > arch/arm64/include/asm/esr.h | 6 + > arch/arm64/include/asm/kvm_arm.h | 28 +- > arch/arm64/include/asm/kvm_asm.h | 4 + > arch/arm64/include/asm/kvm_emulate.h | 145 +- > arch/arm64/include/asm/kvm_host.h | 175 ++- > arch/arm64/include/asm/kvm_hyp.h | 2 + > arch/arm64/include/asm/kvm_mmu.h | 17 +- > arch/arm64/include/asm/kvm_nested.h | 152 ++ > arch/arm64/include/asm/kvm_pgtable.h | 10 + > arch/arm64/include/asm/sysreg.h | 104 +- > arch/arm64/include/asm/vncr_mapping.h | 73 + > arch/arm64/include/uapi/asm/kvm.h | 2 + > arch/arm64/kernel/cpufeature.c | 35 + > arch/arm64/kvm/Makefile | 4 +- > arch/arm64/kvm/arch_timer.c | 189 ++- > arch/arm64/kvm/arm.c | 34 +- > arch/arm64/kvm/at.c | 231 ++++ > arch/arm64/kvm/emulate-nested.c | 186 +++ > arch/arm64/kvm/guest.c | 6 + > arch/arm64/kvm/handle_exit.c | 81 +- > arch/arm64/kvm/hyp/exception.c | 44 +- > arch/arm64/kvm/hyp/include/hyp/switch.h | 31 +- > arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +- > arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- > arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- > arch/arm64/kvm/hyp/pgtable.c | 6 + > arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- > arch/arm64/kvm/hyp/vhe/switch.c | 207 ++- > arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 +- > arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ > arch/arm64/kvm/inject_fault.c | 62 +- > arch/arm64/kvm/mmu.c | 183 ++- > arch/arm64/kvm/nested.c | 908 ++++++++++++ > arch/arm64/kvm/reset.c | 14 +- > arch/arm64/kvm/sys_regs.c | 1226 ++++++++++++++++- > arch/arm64/kvm/sys_regs.h | 6 + > arch/arm64/kvm/trace_arm.h | 65 +- > arch/arm64/kvm/vgic/vgic-init.c | 30 + > arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 + > arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++ > arch/arm64/kvm/vgic/vgic-v3-nested.c | 240 ++++ > arch/arm64/kvm/vgic/vgic-v3.c | 39 +- > arch/arm64/kvm/vgic/vgic.c | 44 + > arch/arm64/kvm/vgic/vgic.h | 10 + > include/kvm/arm_arch_timer.h | 7 + > include/kvm/arm_vgic.h | 16 + > tools/arch/arm/include/uapi/asm/kvm.h | 1 + > 50 files changed, 4890 insertions(+), 160 deletions(-) > create mode 100644 arch/arm64/include/asm/kvm_nested.h > create mode 100644 arch/arm64/include/asm/vncr_mapping.h > create mode 100644 arch/arm64/kvm/at.c > create mode 100644 arch/arm64/kvm/emulate-nested.c > create mode 100644 arch/arm64/kvm/nested.c > create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h > create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c > > -- > 2.29.2 > > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm