From: Alexandru Elisei <alexandru.elisei@arm.com>
To: Marc Zyngier <Marc.Zyngier@arm.com>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Cc: Andre Przywara <Andre.Przywara@arm.com>,
Dave P Martin <Dave.Martin@arm.com>
Subject: Re: [PATCH 00/59] KVM: arm64: ARMv8.3 Nested Virtualization support
Date: Fri, 9 Aug 2019 11:01:51 +0100 [thread overview]
Message-ID: <0d9aa552-fa01-c482-41d7-587acf308259@arm.com> (raw)
In-Reply-To: <69cf1fe7-912c-1767-ff1b-dfcc7f549e44@arm.com>
On 8/2/19 11:11 AM, Alexandru Elisei wrote:
> Hi,
>
> On 6/21/19 10:37 AM, Marc Zyngier wrote:
>> I've taken over the maintenance of this series originally written by
>> Jintack and Christoffer. Since then, the series has been substantially
>> reworked, new features (and most probably bugs) have been added, and
>> the whole thing rebased multiple times. If anything breaks, please
>> blame me, and nobody else.
>>
>> As you can tell, this is quite big. It is also remarkably incomplete
>> (we're missing many critical bits for fully emulate EL2), but the idea
>> is to start merging things early in order to reduce the maintenance
>> headache. What we want to achieve is that with NV disabled, there is
>> no performance overhead and no regression. The only thing I intend to
>> merge ASAP is the first patch in the series, because it should have
>> zero effect and is a reasonable cleanup.
>>
>> The series is roughly divided in 4 parts: exception handling, memory
>> virtualization, interrupts and timers. There are of course some
>> dependencies, but you'll hopefully get the gist of it.
>>
>> For the most courageous of you, I've put out a branch[1] containing this
>> and a bit more. Of course, you'll need some userspace. Andre maintains
>> a hacked version of kvmtool[1] that takes a --nested option, allowing
>> the guest to be started at EL2. You can run the whole stack in the
>> Foundation model. Don't be in a hurry ;-).
>>
>> [1] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-wip-5.2-rc5
>> [2] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5
>>
>> Andre Przywara (4):
>> KVM: arm64: nv: Handle virtual EL2 registers in
>> vcpu_read/write_sys_reg()
>> KVM: arm64: nv: Save/Restore vEL2 sysregs
>> KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs
>> accessors
>> KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
>>
>> Christoffer Dall (16):
>> KVM: arm64: nv: Introduce nested virtualization VCPU feature
>> KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
>> KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
>> KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
>> KVM: arm64: nv: Handle trapped ERET from virtual EL2
>> KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
>> KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
>> KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
>> changes
>> KVM: arm/arm64: nv: Support multiple nested stage 2 mmu structures
>> KVM: arm64: nv: Implement nested Stage-2 page table walk logic
>> KVM: arm64: nv: Handle shadow stage 2 page faults
>> KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
>> KVM: arm64: nv: arch_timer: Support hyp timer emulation
>> KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu
>> KVM: arm64: nv: vgic: Emulate the HW bit in software
>> KVM: arm64: nv: Add nested GICv3 tracepoints
>>
>> Dave Martin (1):
>> KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
>>
>> Jintack Lim (21):
>> arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
>> KVM: arm64: nv: Add EL2 system registers to vcpu context
>> KVM: arm64: nv: Support virtual EL2 exceptions
>> KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
>> KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
>> KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
>> KVM: arm64: nv: Set a handler for the system instruction traps
>> KVM: arm64: nv: Handle PSCI call via smc from the guest
>> KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
>> KVM: arm64: nv: Respect virtual CPTR_EL2.TFP setting
>> KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
>> KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
>> KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
>> KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
>> KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
>> KVM: arm64: nv: Pretend we only support larger-than-host page sizes
>> KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
>> KVM: arm64: nv: Rework the system instruction emulation framework
>> KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
>> KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
>> KVM: arm64: nv: Nested GICv3 Support
>>
>> Marc Zyngier (17):
>> KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h
>> KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
>> KVM: arm64: nv: Handle SPSR_EL2 specially
>> KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg
>> KVM: arm64: nv: Don't expose SVE to nested guests
>> KVM: arm64: nv: Hide RAS from nested guests
>> KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm
>> KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu
>> KVM: arm64: nv: Don't always start an S2 MMU search from the beginning
>> KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer
>> KVM: arm64: nv: Load timer before the GIC
>> KVM: arm64: nv: Implement maintenance interrupt forwarding
>> arm64: KVM: nv: Add handling of EL2-specific timer registers
>> arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2
>> arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits
>> arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's
>> arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
>>
>> .../admin-guide/kernel-parameters.txt | 4 +
>> .../virtual/kvm/devices/arm-vgic-v3.txt | 9 +
>> arch/arm/include/asm/kvm_asm.h | 5 +-
>> arch/arm/include/asm/kvm_emulate.h | 3 +
>> arch/arm/include/asm/kvm_host.h | 31 +-
>> arch/arm/include/asm/kvm_hyp.h | 25 +-
>> arch/arm/include/asm/kvm_mmu.h | 83 +-
>> arch/arm/include/asm/kvm_nested.h | 9 +
>> arch/arm/include/uapi/asm/kvm.h | 1 +
>> arch/arm/kvm/hyp/switch.c | 11 +-
>> arch/arm/kvm/hyp/tlb.c | 13 +-
>> arch/arm64/include/asm/cpucaps.h | 3 +-
>> arch/arm64/include/asm/esr.h | 4 +-
>> arch/arm64/include/asm/kvm_arm.h | 28 +-
>> arch/arm64/include/asm/kvm_asm.h | 9 +-
>> arch/arm64/include/asm/kvm_coproc.h | 2 +-
>> arch/arm64/include/asm/kvm_emulate.h | 157 +-
>> arch/arm64/include/asm/kvm_host.h | 105 +-
>> arch/arm64/include/asm/kvm_hyp.h | 82 +-
>> arch/arm64/include/asm/kvm_mmu.h | 62 +-
>> arch/arm64/include/asm/kvm_nested.h | 68 +
>> arch/arm64/include/asm/sysreg.h | 143 +-
>> arch/arm64/include/uapi/asm/kvm.h | 2 +
>> arch/arm64/kernel/cpufeature.c | 26 +
>> arch/arm64/kvm/Makefile | 4 +
>> arch/arm64/kvm/emulate-nested.c | 223 +++
>> arch/arm64/kvm/guest.c | 6 +
>> arch/arm64/kvm/handle_exit.c | 76 +-
>> arch/arm64/kvm/hyp/Makefile | 1 +
>> arch/arm64/kvm/hyp/at.c | 217 +++
>> arch/arm64/kvm/hyp/switch.c | 86 +-
>> arch/arm64/kvm/hyp/sysreg-sr.c | 267 ++-
>> arch/arm64/kvm/hyp/tlb.c | 129 +-
>> arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 2 +-
>> arch/arm64/kvm/inject_fault.c | 12 -
>> arch/arm64/kvm/nested.c | 551 +++++++
>> arch/arm64/kvm/regmap.c | 4 +-
>> arch/arm64/kvm/reset.c | 7 +
>> arch/arm64/kvm/sys_regs.c | 1460 +++++++++++++++--
>> arch/arm64/kvm/sys_regs.h | 6 +
>> arch/arm64/kvm/trace.h | 58 +-
>> include/kvm/arm_arch_timer.h | 6 +
>> include/kvm/arm_vgic.h | 28 +-
>> virt/kvm/arm/arch_timer.c | 158 +-
>> virt/kvm/arm/arm.c | 62 +-
>> virt/kvm/arm/hyp/vgic-v3-sr.c | 35 +-
>> virt/kvm/arm/mmio.c | 12 +-
>> virt/kvm/arm/mmu.c | 445 +++--
>> virt/kvm/arm/trace.h | 6 +-
>> virt/kvm/arm/vgic/vgic-init.c | 30 +
>> virt/kvm/arm/vgic/vgic-kvm-device.c | 22 +
>> virt/kvm/arm/vgic/vgic-nested-trace.h | 137 ++
>> virt/kvm/arm/vgic/vgic-v2.c | 10 +-
>> virt/kvm/arm/vgic/vgic-v3-nested.c | 236 +++
>> virt/kvm/arm/vgic/vgic-v3.c | 40 +-
>> virt/kvm/arm/vgic/vgic.c | 74 +-
>> 56 files changed, 4683 insertions(+), 612 deletions(-)
>> create mode 100644 arch/arm/include/asm/kvm_nested.h
>> create mode 100644 arch/arm64/include/asm/kvm_nested.h
>> create mode 100644 arch/arm64/kvm/emulate-nested.c
>> create mode 100644 arch/arm64/kvm/hyp/at.c
>> create mode 100644 arch/arm64/kvm/nested.c
>> create mode 100644 virt/kvm/arm/vgic/vgic-nested-trace.h
>> create mode 100644 virt/kvm/arm/vgic/vgic-v3-nested.c
>>
> When working on adding support for EL2 to kvm-unit-tests I was able to trigger
> the following warning:
>
> # ./lkvm run -f psci.flat -m 128 -c 8 --console serial --irqchip gicv3 --nested
> # lkvm run --firmware psci.flat -m 128 -c 8 --name guest-151
> Info: Placing fdt at 0x80200000 - 0x80210000
> # Warning: The maximum recommended amount of VCPUs is 4
> chr_testdev_init: chr-testdev: can't find a virtio-console
> INFO: PSCI version 1.0
> PASS: invalid-function
> PASS: affinity-info-on
> PASS: affinity-info-off
> [ 24.381266] WARNING: CPU: 3 PID: 160 at
> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
> kvm_timer_irq_can_fire+0xc/0x30
> [ 24.381366] Modules linked in:
> [ 24.381466] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Not tainted
> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
> [ 24.381566] Hardware name: Foundation-v8A (DT)
> [ 24.381566] pstate: 40400009 (nZcv daif +PAN -UAO)
> [ 24.381666] pc : kvm_timer_irq_can_fire+0xc/0x30
> [ 24.381766] lr : timer_emulate+0x24/0x98
> [ 24.381766] sp : ffff000013d8b780
> [ 24.381866] x29: ffff000013d8b780 x28: ffff80087a639b80
> [ 24.381966] x27: ffff000010ba8648 x26: ffff000010b71b40
> [ 24.382066] x25: ffff80087a63a100 x24: 0000000000000000
> [ 24.382111] x23: 000080086ca54000 x22: ffff0000100ce260
> [ 24.382166] x21: ffff800875e7c918 x20: ffff800875e7a800
> [ 24.382275] x19: ffff800875e7ca08 x18: 0000000000000000
> [ 24.382366] x17: 0000000000000000 x16: 0000000000000000
> [ 24.382466] x15: 0000000000000000 x14: 0000000000002118
> [ 24.382566] x13: 0000000000002190 x12: 0000000000002280
> [ 24.382566] x11: 0000000000002208 x10: 0000000000000040
> [ 24.382666] x9 : ffff000012dc3b38 x8 : 0000000000000000
> [ 24.382766] x7 : 0000000000000000 x6 : ffff80087ac00248
> [ 24.382866] x5 : 000080086ca54000 x4 : 0000000000002118
> [ 24.382966] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
> [ 24.383066] x1 : 0000000000000001 x0 : ffff800875e7ca08
> [ 24.383066] Call trace:
> [ 24.383166] kvm_timer_irq_can_fire+0xc/0x30
> [ 24.383266] kvm_timer_vcpu_load+0x9c/0x1a0
> [ 24.383366] kvm_arch_vcpu_load+0xb0/0x1f0
> [ 24.383366] kvm_sched_in+0x1c/0x28
> [ 24.383466] finish_task_switch+0xd8/0x1d8
> [ 24.383566] __schedule+0x248/0x4a0
> [ 24.383666] preempt_schedule_irq+0x60/0x90
> [ 24.383666] el1_irq+0xd0/0x180
> [ 24.383766] kvm_handle_guest_abort+0x0/0x3a0
> [ 24.383866] kvm_arch_vcpu_ioctl_run+0x41c/0x688
> [ 24.383866] kvm_vcpu_ioctl+0x4c0/0x838
> [ 24.383966] do_vfs_ioctl+0xb8/0x878
> [ 24.384077] ksys_ioctl+0x84/0x90
> [ 24.384166] __arm64_sys_ioctl+0x18/0x28
> [ 24.384166] el0_svc_common.constprop.0+0xb0/0x168
> [ 24.384266] el0_svc_handler+0x28/0x78
> [ 24.384366] el0_svc+0x8/0xc
> [ 24.384366] ---[ end trace 37a32293e43ac12c ]---
> [ 24.384666] WARNING: CPU: 3 PID: 160 at
> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
> kvm_timer_irq_can_fire+0xc/0x30
> [ 24.384766] Modules linked in:
> [ 24.384866] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G W
> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
> [ 24.384966] Hardware name: Foundation-v8A (DT)
> [ 24.384966] pstate: 40400009 (nZcv daif +PAN -UAO)
> [ 24.385066] pc : kvm_timer_irq_can_fire+0xc/0x30
> [ 24.385166] lr : timer_emulate+0x24/0x98
> [ 24.385166] sp : ffff000013d8b780
> [ 24.385266] x29: ffff000013d8b780 x28: ffff80087a639b80
> [ 24.385366] x27: ffff000010ba8648 x26: ffff000010b71b40
> [ 24.385466] x25: ffff80087a63a100 x24: 0000000000000000
> [ 24.385466] x23: 000080086ca54000 x22: ffff0000100ce260
> [ 24.385566] x21: ffff800875e7c918 x20: ffff800875e7a800
> [ 24.385666] x19: ffff800875e7ca80 x18: 0000000000000000
> [ 24.385766] x17: 0000000000000000 x16: 0000000000000000
> [ 24.385866] x15: 0000000000000000 x14: 0000000000002118
> [ 24.385966] x13: 0000000000002190 x12: 0000000000002280
> [ 24.385966] x11: 0000000000002208 x10: 0000000000000040
> [ 24.386066] x9 : ffff000012dc3b38 x8 : 0000000000000000
> [ 24.386166] x7 : 0000000000000000 x6 : ffff80087ac00248
> [ 24.386266] x5 : 000080086ca54000 x4 : 0000000000002118
> [ 24.386366] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
> [ 24.386466] x1 : 0000000000000001 x0 : ffff800875e7ca80
> [ 24.386466] Call trace:
> [ 24.386566] kvm_timer_irq_can_fire+0xc/0x30
> [ 24.386666] kvm_timer_vcpu_load+0xa8/0x1a0
> [ 24.386666] kvm_arch_vcpu_load+0xb0/0x1f0
> [ 24.386898] kvm_sched_in+0x1c/0x28
> [ 24.386966] finish_task_switch+0xd8/0x1d8
> [ 24.387166] __schedule+0x248/0x4a0
> [ 24.387354] preempt_schedule_irq+0x60/0x90
> [ 24.387366] el1_irq+0xd0/0x180
> [ 24.387466] kvm_handle_guest_abort+0x0/0x3a0
> [ 24.387566] kvm_arch_vcpu_ioctl_run+0x41c/0x688
> [ 24.387566] kvm_vcpu_ioctl+0x4c0/0x838
> [ 24.387666] do_vfs_ioctl+0xb8/0x878
> [ 24.387766] ksys_ioctl+0x84/0x90
> [ 24.387866] __arm64_sys_ioctl+0x18/0x28
> [ 24.387866] el0_svc_common.constprop.0+0xb0/0x168
> [ 24.387966] el0_svc_handler+0x28/0x78
> [ 24.388066] el0_svc+0x8/0xc
> [ 24.388066] ---[ end trace 37a32293e43ac12d ]---
> PASS: cpu-on
> SUMMARY: 4 te[ 24.390266] WARNING: CPU: 3 PID: 160 at
> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
> kvm_timer_irq_can_fire+0xc/0x30
> s[ 24.390366] Modules linked in:
> ts[ 24.390366] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G W
> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
> [ 24.390566] Hardware name: Foundation-v8A (DT)
>
> [ 24.390795] pstate: 40400009 (nZcv daif +PAN -UAO)
> [ 24.390866] pc : kvm_timer_irq_can_fire+0xc/0x30
> [ 24.390966] lr : timer_emulate+0x24/0x98
> [ 24.391066] sp : ffff000013d8b780
> [ 24.391066] x29: ffff000013d8b780 x28: ffff80087a639b80
> [ 24.391166] x27: ffff000010ba8648 x26: ffff000010b71b40
> [ 24.391266] x25: ffff80087a63a100 x24: 0000000000000000
> [ 24.391366] x23: 000080086ca54000 x22: 0000000000000003
> [ 24.391466] x21: ffff800875e7c918 x20: ffff800875e7a800
> [ 24.391466] x19: ffff800875e7ca08 x18: 0000000000000000
> [ 24.391566] x17: 0000000000000000 x16: 0000000000000000
> [ 24.391666] x15: 0000000000000000 x14: 0000000000002118
> [ 24.391766] x13: 0000000000002190 x12: 0000000000002280
> [ 24.391866] x11: 0000000000002208 x10: 0000000000000040
> [ 24.391942] x9 : ffff000012dc3b38 x8 : 0000000000000000
> [ 24.391966] x7 : 0000000000000000 x6 : ffff80087ac00248
> [ 24.392066] x5 : 000080086ca54000 x4 : 0000000000002118
> [ 24.392166] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
> [ 24.392269] x1 : 0000000000000001 x0 : ffff800875e7ca08
> [ 24.392366] Call trace:
> [ 24.392433] kvm_timer_irq_can_fire+0xc/0x30
> [ 24.392466] kvm_timer_vcpu_load+0x9c/0x1a0
> [ 24.392597] kvm_arch_vcpu_load+0xb0/0x1f0
> [ 24.392666] kvm_sched_in+0x1c/0x28
> [ 24.392766] finish_task_switch+0xd8/0x1d8
> [ 24.392766] __schedule+0x248/0x4a0
> [ 24.392866] preempt_schedule_irq+0x60/0x90
> [ 24.392966] el1_irq+0xd0/0x180
> [ 24.392966] kvm_handle_guest_abort+0x0/0x3a0
> [ 24.393066] kvm_arch_vcpu_ioctl_run+0x41c/0x688
> [ 24.393166] kvm_vcpu_ioctl+0x4c0/0x838
> [ 24.393266] do_vfs_ioctl+0xb8/0x878
> [ 24.393266] ksys_ioctl+0x84/0x90
> [ 24.393366] __arm64_sys_ioctl+0x18/0x28
> [ 24.393466] el0_svc_common.constprop.0+0xb0/0x168
> [ 24.393566] el0_svc_handler+0x28/0x78
> [ 24.393566] el0_svc+0x8/0xc
> [ 24.393666] ---[ end trace 37a32293e43ac12e ]---
> [ 24.393866] WARNING: CPU: 3 PID: 160 at
> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
> kvm_timer_irq_can_fire+0xc/0x30
> [ 24.394066] Modules linked in:
> [ 24.394266] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G W
> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
> [ 24.394366] Hardware name: Foundation-v8A (DT)
> [ 24.394466] pstate: 40400009 (nZcv daif +PAN -UAO)
> [ 24.394466] pc : kvm_timer_irq_can_fire+0xc/0x30
> [ 24.394566] lr : timer_emulate+0x24/0x98
> [ 24.394666] sp : ffff000013d8b780
> [ 24.394727] x29: ffff000013d8b780 x28: ffff80087a639b80
> [ 24.394766] x27: ffff000010ba8648 x26: ffff000010b71b40
> [ 24.394866] x25: ffff80087a63a100 x24: 0000000000000000
> [ 24.394966] x23: 000080086ca54000 x22: 0000000000000003
> [ 24.394966] x21: ffff800875e7c918 x20: ffff800875e7a800
> [ 24.395066] x19: ffff800875e7ca80 x18: 0000000000000000
> [ 24.395166] x17: 0000000000000000 x16: 0000000000000000
> [ 24.395266] x15: 0000000000000000 x14: 0000000000002118
> [ 24.395383] x13: 0000000000002190 x12: 0000000000002280
> [ 24.395466] x11: 0000000000002208 x10: 0000000000000040
> [ 24.395547] x9 : ffff000012dc3b38 x8 : 0000000000000000
> [ 24.395666] x7 : 0000000000000000 x6 : ffff80087ac00248
> [ 24.395866] x5 : 000080086ca54000 x4 : 0000000000002118
> [ 24.395966] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
> [ 24.396066] x1 : 0000000000000001 x0 : ffff800875e7ca80
> [ 24.396066] Call trace:
> [ 24.396166] kvm_timer_irq_can_fire+0xc/0x30
> [ 24.396266] kvm_timer_vcpu_load+0xa8/0x1a0
> [ 24.396366] kvm_arch_vcpu_load+0xb0/0x1f0
> [ 24.396366] kvm_sched_in+0x1c/0x28
> [ 24.396466] finish_task_switch+0xd8/0x1d8
> [ 24.396566] __schedule+0x248/0x4a0
> [ 24.396666] preempt_schedule_irq+0x60/0x90
> [ 24.396666] el1_irq+0xd0/0x180
> [ 24.396766] kvm_handle_guest_abort+0x0/0x3a0
> [ 24.396866] kvm_arch_vcpu_ioctl_run+0x41c/0x688
> [ 24.396866] kvm_vcpu_ioctl+0x4c0/0x838
> [ 24.397021] do_vfs_ioctl+0xb8/0x878
> [ 24.397066] ksys_ioctl+0x84/0x90
> [ 24.397166] __arm64_sys_ioctl+0x18/0x28
> [ 24.397348] el0_svc_common.constprop.0+0xb0/0x168
> [ 24.397366] el0_svc_handler+0x28/0x78
> [ 24.397566] el0_svc+0x8/0xc
> [ 24.397676] ---[ end trace 37a32293e43ac12f ]---
>
> # KVM compatibility warning.
> virtio-9p device was not detected.
> While you have requested a virtio-9p device, the guest kernel did not
> initialize it.
> Please make sure that the guest kernel was compiled with
> CONFIG_NET_9P_VIRTIO=y enabled in .config.
>
> # KVM compatibility warning.
> virtio-net device was not detected.
> While you have requested a virtio-net device, the guest kernel did not
> initialize it.
> Please make sure that the guest kernel was compiled with CONFIG_VIRTIO_NET=y
> enabled in .config.
>
> [..]
Did some investigating and this was caused by a bug in kvm-unit-tests (the fix
for it will be part of the EL2 patches for kvm-unit-tests). The guest was trying
to fetch an instruction from address 0x200, which KVM interprets as a prefetch
abort on an I/O address and ends up calling kvm_inject_pabt. The code from
arch/arm64/kvm/inject_fault.c doesn't know anything about nested virtualization,
and it sets the VCPU mode directly to PSR_MODE_EL1h. This makes_hyp_ctxt return
false, and get_timer_map will return an incorrect mapping.
On next kvm_timer_vcpu_put, the direct timers will be {p,v}timer, and
h{p,v}timer->loaded will not be set to false. In the corresponding call to
kvm_timer_vcpu_load, KVM will try to emulate the hptimer and hvtimer, which
still have loaded = true. And this causes the warning I saw.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
next prev parent reply other threads:[~2019-08-09 10:02 UTC|newest]
Thread overview: 179+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-21 9:37 [PATCH 00/59] KVM: arm64: ARMv8.3 Nested Virtualization support Marc Zyngier
2019-06-21 9:37 ` [PATCH 01/59] KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s Marc Zyngier
2019-06-24 11:16 ` Dave Martin
2019-06-24 12:59 ` Alexandru Elisei
2019-07-03 12:32 ` Marc Zyngier
2019-06-21 9:37 ` [PATCH 02/59] KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h Marc Zyngier
2019-06-24 11:19 ` Dave Martin
2019-07-03 9:30 ` Marc Zyngier
2019-07-03 16:13 ` Dave Martin
2019-06-21 9:37 ` [PATCH 03/59] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Marc Zyngier
2019-06-21 13:08 ` Julien Thierry
2019-06-21 13:22 ` Marc Zyngier
2019-06-21 13:44 ` Suzuki K Poulose
2019-06-24 11:24 ` Dave Martin
2019-06-21 9:37 ` [PATCH 04/59] KVM: arm64: nv: Introduce nested virtualization VCPU feature Marc Zyngier
2019-06-21 13:08 ` Julien Thierry
2019-06-24 11:28 ` Dave Martin
2019-07-03 11:53 ` Marc Zyngier
2019-07-03 16:27 ` Dave Martin
2019-06-24 11:43 ` Dave Martin
2019-07-03 11:56 ` Marc Zyngier
2019-07-03 16:24 ` Dave Martin
2019-06-21 9:37 ` [PATCH 05/59] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Marc Zyngier
2019-06-24 10:19 ` Suzuki K Poulose
2019-06-24 11:38 ` Dave Martin
2019-06-21 9:37 ` [PATCH 06/59] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Marc Zyngier
2019-06-21 13:24 ` Julien Thierry
2019-06-21 13:50 ` Marc Zyngier
2019-06-24 12:48 ` Dave Martin
2019-07-03 9:21 ` Marc Zyngier
2019-07-04 10:00 ` Dave Martin
2019-06-21 9:37 ` [PATCH 07/59] KVM: arm64: nv: Add EL2 system registers to vcpu context Marc Zyngier
2019-06-24 12:54 ` Dave Martin
2019-07-03 12:20 ` Marc Zyngier
2019-07-03 16:31 ` Dave Martin
2019-06-24 15:47 ` Alexandru Elisei
2019-07-03 13:20 ` Marc Zyngier
2019-07-03 16:01 ` Marc Zyngier
2019-07-01 16:36 ` Suzuki K Poulose
2019-06-21 9:37 ` [PATCH 08/59] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Marc Zyngier
2019-06-24 12:59 ` Dave Martin
2019-06-21 9:37 ` [PATCH 09/59] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Marc Zyngier
2019-06-24 13:08 ` Dave Martin
2019-06-21 9:37 ` [PATCH 10/59] KVM: arm64: nv: Support virtual EL2 exceptions Marc Zyngier
2019-07-08 13:56 ` Steven Price
2019-06-21 9:37 ` [PATCH 11/59] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Marc Zyngier
2019-06-25 13:13 ` Alexandru Elisei
2019-07-03 14:16 ` Marc Zyngier
2019-07-30 14:08 ` Alexandru Elisei
2019-06-21 9:37 ` [PATCH 12/59] KVM: arm64: nv: Handle trapped ERET from " Marc Zyngier
2019-07-02 12:00 ` Alexandru Elisei
2019-06-21 9:37 ` [PATCH 13/59] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Marc Zyngier
2019-06-24 12:42 ` Julien Thierry
2019-06-25 14:02 ` Alexandru Elisei
2019-07-03 12:15 ` Marc Zyngier
2019-07-03 15:21 ` Julien Thierry
2019-06-25 15:18 ` Alexandru Elisei
2019-07-01 9:58 ` Alexandru Elisei
2019-07-03 15:59 ` Marc Zyngier
2019-07-03 16:32 ` Alexandru Elisei
2019-07-04 14:39 ` Marc Zyngier
2019-06-26 15:04 ` Alexandru Elisei
2019-07-04 15:05 ` Marc Zyngier
2019-07-01 12:10 ` Alexandru Elisei
2019-06-21 9:37 ` [PATCH 14/59] KVM: arm64: nv: Handle SPSR_EL2 specially Marc Zyngier
2019-06-21 9:37 ` [PATCH 15/59] KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg Marc Zyngier
2019-06-24 15:07 ` Julien Thierry
2019-07-03 13:09 ` Marc Zyngier
2019-06-27 9:21 ` Alexandru Elisei
2019-07-04 15:15 ` Marc Zyngier
2019-06-21 9:38 ` [PATCH 16/59] KVM: arm64: nv: Save/Restore vEL2 sysregs Marc Zyngier
2019-06-25 8:48 ` Julien Thierry
2019-07-03 13:42 ` Marc Zyngier
2019-07-01 12:09 ` Alexandru Elisei
2019-08-21 11:57 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 17/59] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Marc Zyngier
2019-06-21 9:38 ` [PATCH 18/59] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Marc Zyngier
2019-07-01 16:12 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 19/59] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from " Marc Zyngier
2019-06-21 9:38 ` [PATCH 20/59] KVM: arm64: nv: Trap CPACR_EL1 access in " Marc Zyngier
2019-07-01 16:40 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 21/59] KVM: arm64: nv: Set a handler for the system instruction traps Marc Zyngier
2019-06-25 12:55 ` Julien Thierry
2019-07-03 14:15 ` Marc Zyngier
2019-06-21 9:38 ` [PATCH 22/59] KVM: arm64: nv: Handle PSCI call via smc from the guest Marc Zyngier
2019-06-21 9:38 ` [PATCH 23/59] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Marc Zyngier
2019-06-25 14:19 ` Julien Thierry
2019-07-02 12:54 ` Alexandru Elisei
2019-07-03 14:18 ` Marc Zyngier
2019-06-21 9:38 ` [PATCH 24/59] KVM: arm64: nv: Respect virtual CPTR_EL2.TFP setting Marc Zyngier
2019-06-21 9:38 ` [PATCH 25/59] KVM: arm64: nv: Don't expose SVE to nested guests Marc Zyngier
2019-06-21 9:38 ` [PATCH 26/59] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Marc Zyngier
2019-06-26 5:31 ` Julien Thierry
2019-07-03 16:31 ` Marc Zyngier
2019-06-21 9:38 ` [PATCH 27/59] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Marc Zyngier
2019-06-26 6:55 ` Julien Thierry
2019-07-04 14:57 ` Marc Zyngier
2019-06-21 9:38 ` [PATCH 28/59] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Marc Zyngier
2019-06-26 7:23 ` Julien Thierry
2019-07-02 16:32 ` Alexandru Elisei
2019-07-03 9:10 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 29/59] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Marc Zyngier
2019-07-03 9:16 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 30/59] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Marc Zyngier
2019-06-21 9:38 ` [PATCH 31/59] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Marc Zyngier
2019-06-21 9:38 ` [PATCH 32/59] KVM: arm64: nv: Hide RAS from nested guests Marc Zyngier
2019-07-03 13:59 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 33/59] KVM: arm64: nv: Pretend we only support larger-than-host page sizes Marc Zyngier
2019-07-03 14:13 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 34/59] KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm Marc Zyngier
2019-07-03 15:52 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 35/59] KVM: arm/arm64: nv: Support multiple nested stage 2 mmu structures Marc Zyngier
2019-06-25 12:19 ` Alexandru Elisei
2019-07-03 13:47 ` Marc Zyngier
2019-06-27 13:15 ` Julien Thierry
2019-07-04 15:51 ` Alexandru Elisei
2020-01-05 11:35 ` Marc Zyngier
2020-01-06 16:31 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 36/59] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Marc Zyngier
2019-06-21 9:38 ` [PATCH 37/59] KVM: arm64: nv: Handle shadow stage 2 page faults Marc Zyngier
2019-07-05 14:28 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 38/59] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Marc Zyngier
2019-07-01 8:03 ` Julien Thierry
2019-06-21 9:38 ` [PATCH 39/59] KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu Marc Zyngier
2019-07-01 9:10 ` Julien Thierry
2019-07-05 15:28 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 40/59] KVM: arm64: nv: Don't always start an S2 MMU search from the beginning Marc Zyngier
2019-07-09 9:59 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 41/59] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Marc Zyngier
2019-06-21 9:38 ` [PATCH 42/59] KVM: arm64: nv: Rework the system instruction emulation framework Marc Zyngier
2019-06-21 9:38 ` [PATCH 43/59] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Marc Zyngier
2019-07-01 15:45 ` Julien Thierry
2019-07-09 13:20 ` Alexandru Elisei
2019-07-18 12:13 ` Tomasz Nowicki
2019-07-18 12:36 ` Alexandru Elisei
2019-07-18 12:56 ` Alexandru Elisei
2019-07-18 12:59 ` Tomasz Nowicki
2019-07-24 10:25 ` Tomasz Nowicki
2019-07-24 12:39 ` Marc Zyngier
2019-07-24 13:56 ` Tomasz Nowicki
2019-06-21 9:38 ` [PATCH 44/59] KVM: arm64: nv: Trap and emulate TLBI " Marc Zyngier
2019-07-02 12:37 ` Julien Thierry
2019-07-10 10:15 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 45/59] KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors Marc Zyngier
2019-06-21 9:38 ` [PATCH 46/59] KVM: arm64: nv: arch_timer: Support hyp timer emulation Marc Zyngier
2019-07-10 16:23 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 47/59] KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer Marc Zyngier
2019-08-08 9:34 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 48/59] KVM: arm64: nv: Load timer before the GIC Marc Zyngier
2019-07-11 13:17 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 49/59] KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu Marc Zyngier
2019-06-21 9:38 ` [PATCH 50/59] KVM: arm64: nv: Nested GICv3 Support Marc Zyngier
2019-07-16 11:41 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 51/59] KVM: arm64: nv: vgic: Emulate the HW bit in software Marc Zyngier
2019-06-21 9:38 ` [PATCH 52/59] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Marc Zyngier
2019-07-04 7:38 ` Julien Thierry
2019-07-04 9:01 ` Andre Przywara
2019-07-04 9:04 ` Julien Thierry
2019-06-21 9:38 ` [PATCH 53/59] KVM: arm64: nv: Implement maintenance interrupt forwarding Marc Zyngier
2019-07-04 8:06 ` Julien Thierry
2019-07-16 16:35 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 54/59] KVM: arm64: nv: Add nested GICv3 tracepoints Marc Zyngier
2019-06-21 9:38 ` [PATCH 55/59] arm64: KVM: nv: Add handling of EL2-specific timer registers Marc Zyngier
2019-07-11 12:35 ` Alexandru Elisei
2019-07-17 10:19 ` Alexandru Elisei
2019-06-21 9:38 ` [PATCH 56/59] arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 Marc Zyngier
2019-06-21 9:38 ` [PATCH 57/59] arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits Marc Zyngier
2019-06-21 9:38 ` [PATCH 58/59] arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's Marc Zyngier
2019-06-21 9:38 ` [PATCH 59/59] arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Marc Zyngier
2019-06-21 9:57 ` [PATCH 00/59] KVM: arm64: ARMv8.3 Nested Virtualization support Itaru Kitayama
2019-06-21 11:21 ` Marc Zyngier
2019-08-02 10:11 ` Alexandru Elisei
2019-08-02 10:30 ` Andrew Jones
2019-08-09 10:01 ` Alexandru Elisei [this message]
2019-08-09 11:44 ` Andrew Jones
2019-08-09 12:00 ` Alexandru Elisei
2019-08-09 13:00 ` Andrew Jones
2019-08-22 11:57 ` Alexandru Elisei
2019-08-22 15:32 ` Alexandru Elisei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0d9aa552-fa01-c482-41d7-587acf308259@arm.com \
--to=alexandru.elisei@arm.com \
--cc=Andre.Przywara@arm.com \
--cc=Dave.Martin@arm.com \
--cc=Marc.Zyngier@arm.com \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).