KVM ARM Archive on lore.kernel.org
 help / color / Atom feed
From: Alexandru Elisei <alexandru.elisei@arm.com>
To: Andrew Jones <drjones@redhat.com>
Cc: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	Marc Zyngier <Marc.Zyngier@arm.com>,
	Andre Przywara <Andre.Przywara@arm.com>,
	"kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
	Dave P Martin <Dave.Martin@arm.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH 00/59] KVM: arm64: ARMv8.3 Nested Virtualization support
Date: Fri, 9 Aug 2019 13:00:02 +0100
Message-ID: <6dafd748-257e-1d09-aecc-d5a2ab91bdc4@arm.com> (raw)
In-Reply-To: <20190809114455.w4jes6z2442vu3py@kamzik.brq.redhat.com>

Hi Andrew,

On 8/9/19 12:44 PM, Andrew Jones wrote:
> On Fri, Aug 09, 2019 at 11:01:51AM +0100, Alexandru Elisei wrote:
>> On 8/2/19 11:11 AM, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> On 6/21/19 10:37 AM, Marc Zyngier wrote:
>>>> I've taken over the maintenance of this series originally written by
>>>> Jintack and Christoffer. Since then, the series has been substantially
>>>> reworked, new features (and most probably bugs) have been added, and
>>>> the whole thing rebased multiple times. If anything breaks, please
>>>> blame me, and nobody else.
>>>>
>>>> As you can tell, this is quite big. It is also remarkably incomplete
>>>> (we're missing many critical bits for fully emulate EL2), but the idea
>>>> is to start merging things early in order to reduce the maintenance
>>>> headache. What we want to achieve is that with NV disabled, there is
>>>> no performance overhead and no regression. The only thing I intend to
>>>> merge ASAP is the first patch in the series, because it should have
>>>> zero effect and is a reasonable cleanup.
>>>>
>>>> The series is roughly divided in 4 parts: exception handling, memory
>>>> virtualization, interrupts and timers. There are of course some
>>>> dependencies, but you'll hopefully get the gist of it.
>>>>
>>>> For the most courageous of you, I've put out a branch[1] containing this
>>>> and a bit more. Of course, you'll need some userspace. Andre maintains
>>>> a hacked version of kvmtool[1] that takes a --nested option, allowing
>>>> the guest to be started at EL2. You can run the whole stack in the
>>>> Foundation model. Don't be in a hurry ;-).
>>>>
>>>> [1] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-wip-5.2-rc5
>>>> [2] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5
>>>>
>>>> Andre Przywara (4):
>>>>   KVM: arm64: nv: Handle virtual EL2 registers in
>>>>     vcpu_read/write_sys_reg()
>>>>   KVM: arm64: nv: Save/Restore vEL2 sysregs
>>>>   KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs
>>>>     accessors
>>>>   KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
>>>>
>>>> Christoffer Dall (16):
>>>>   KVM: arm64: nv: Introduce nested virtualization VCPU feature
>>>>   KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
>>>>   KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
>>>>   KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
>>>>   KVM: arm64: nv: Handle trapped ERET from virtual EL2
>>>>   KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
>>>>   KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
>>>>   KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
>>>>     changes
>>>>   KVM: arm/arm64: nv: Support multiple nested stage 2 mmu structures
>>>>   KVM: arm64: nv: Implement nested Stage-2 page table walk logic
>>>>   KVM: arm64: nv: Handle shadow stage 2 page faults
>>>>   KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
>>>>   KVM: arm64: nv: arch_timer: Support hyp timer emulation
>>>>   KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu
>>>>   KVM: arm64: nv: vgic: Emulate the HW bit in software
>>>>   KVM: arm64: nv: Add nested GICv3 tracepoints
>>>>
>>>> Dave Martin (1):
>>>>   KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
>>>>
>>>> Jintack Lim (21):
>>>>   arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
>>>>   KVM: arm64: nv: Add EL2 system registers to vcpu context
>>>>   KVM: arm64: nv: Support virtual EL2 exceptions
>>>>   KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
>>>>   KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
>>>>   KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
>>>>   KVM: arm64: nv: Set a handler for the system instruction traps
>>>>   KVM: arm64: nv: Handle PSCI call via smc from the guest
>>>>   KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
>>>>   KVM: arm64: nv: Respect virtual CPTR_EL2.TFP setting
>>>>   KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
>>>>   KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
>>>>   KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
>>>>   KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
>>>>   KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
>>>>   KVM: arm64: nv: Pretend we only support larger-than-host page sizes
>>>>   KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
>>>>   KVM: arm64: nv: Rework the system instruction emulation framework
>>>>   KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
>>>>   KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
>>>>   KVM: arm64: nv: Nested GICv3 Support
>>>>
>>>> Marc Zyngier (17):
>>>>   KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h
>>>>   KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
>>>>   KVM: arm64: nv: Handle SPSR_EL2 specially
>>>>   KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg
>>>>   KVM: arm64: nv: Don't expose SVE to nested guests
>>>>   KVM: arm64: nv: Hide RAS from nested guests
>>>>   KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm
>>>>   KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu
>>>>   KVM: arm64: nv: Don't always start an S2 MMU search from the beginning
>>>>   KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer
>>>>   KVM: arm64: nv: Load timer before the GIC
>>>>   KVM: arm64: nv: Implement maintenance interrupt forwarding
>>>>   arm64: KVM: nv: Add handling of EL2-specific timer registers
>>>>   arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2
>>>>   arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits
>>>>   arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's
>>>>   arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
>>>>
>>>>  .../admin-guide/kernel-parameters.txt         |    4 +
>>>>  .../virtual/kvm/devices/arm-vgic-v3.txt       |    9 +
>>>>  arch/arm/include/asm/kvm_asm.h                |    5 +-
>>>>  arch/arm/include/asm/kvm_emulate.h            |    3 +
>>>>  arch/arm/include/asm/kvm_host.h               |   31 +-
>>>>  arch/arm/include/asm/kvm_hyp.h                |   25 +-
>>>>  arch/arm/include/asm/kvm_mmu.h                |   83 +-
>>>>  arch/arm/include/asm/kvm_nested.h             |    9 +
>>>>  arch/arm/include/uapi/asm/kvm.h               |    1 +
>>>>  arch/arm/kvm/hyp/switch.c                     |   11 +-
>>>>  arch/arm/kvm/hyp/tlb.c                        |   13 +-
>>>>  arch/arm64/include/asm/cpucaps.h              |    3 +-
>>>>  arch/arm64/include/asm/esr.h                  |    4 +-
>>>>  arch/arm64/include/asm/kvm_arm.h              |   28 +-
>>>>  arch/arm64/include/asm/kvm_asm.h              |    9 +-
>>>>  arch/arm64/include/asm/kvm_coproc.h           |    2 +-
>>>>  arch/arm64/include/asm/kvm_emulate.h          |  157 +-
>>>>  arch/arm64/include/asm/kvm_host.h             |  105 +-
>>>>  arch/arm64/include/asm/kvm_hyp.h              |   82 +-
>>>>  arch/arm64/include/asm/kvm_mmu.h              |   62 +-
>>>>  arch/arm64/include/asm/kvm_nested.h           |   68 +
>>>>  arch/arm64/include/asm/sysreg.h               |  143 +-
>>>>  arch/arm64/include/uapi/asm/kvm.h             |    2 +
>>>>  arch/arm64/kernel/cpufeature.c                |   26 +
>>>>  arch/arm64/kvm/Makefile                       |    4 +
>>>>  arch/arm64/kvm/emulate-nested.c               |  223 +++
>>>>  arch/arm64/kvm/guest.c                        |    6 +
>>>>  arch/arm64/kvm/handle_exit.c                  |   76 +-
>>>>  arch/arm64/kvm/hyp/Makefile                   |    1 +
>>>>  arch/arm64/kvm/hyp/at.c                       |  217 +++
>>>>  arch/arm64/kvm/hyp/switch.c                   |   86 +-
>>>>  arch/arm64/kvm/hyp/sysreg-sr.c                |  267 ++-
>>>>  arch/arm64/kvm/hyp/tlb.c                      |  129 +-
>>>>  arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c      |    2 +-
>>>>  arch/arm64/kvm/inject_fault.c                 |   12 -
>>>>  arch/arm64/kvm/nested.c                       |  551 +++++++
>>>>  arch/arm64/kvm/regmap.c                       |    4 +-
>>>>  arch/arm64/kvm/reset.c                        |    7 +
>>>>  arch/arm64/kvm/sys_regs.c                     | 1460 +++++++++++++++--
>>>>  arch/arm64/kvm/sys_regs.h                     |    6 +
>>>>  arch/arm64/kvm/trace.h                        |   58 +-
>>>>  include/kvm/arm_arch_timer.h                  |    6 +
>>>>  include/kvm/arm_vgic.h                        |   28 +-
>>>>  virt/kvm/arm/arch_timer.c                     |  158 +-
>>>>  virt/kvm/arm/arm.c                            |   62 +-
>>>>  virt/kvm/arm/hyp/vgic-v3-sr.c                 |   35 +-
>>>>  virt/kvm/arm/mmio.c                           |   12 +-
>>>>  virt/kvm/arm/mmu.c                            |  445 +++--
>>>>  virt/kvm/arm/trace.h                          |    6 +-
>>>>  virt/kvm/arm/vgic/vgic-init.c                 |   30 +
>>>>  virt/kvm/arm/vgic/vgic-kvm-device.c           |   22 +
>>>>  virt/kvm/arm/vgic/vgic-nested-trace.h         |  137 ++
>>>>  virt/kvm/arm/vgic/vgic-v2.c                   |   10 +-
>>>>  virt/kvm/arm/vgic/vgic-v3-nested.c            |  236 +++
>>>>  virt/kvm/arm/vgic/vgic-v3.c                   |   40 +-
>>>>  virt/kvm/arm/vgic/vgic.c                      |   74 +-
>>>>  56 files changed, 4683 insertions(+), 612 deletions(-)
>>>>  create mode 100644 arch/arm/include/asm/kvm_nested.h
>>>>  create mode 100644 arch/arm64/include/asm/kvm_nested.h
>>>>  create mode 100644 arch/arm64/kvm/emulate-nested.c
>>>>  create mode 100644 arch/arm64/kvm/hyp/at.c
>>>>  create mode 100644 arch/arm64/kvm/nested.c
>>>>  create mode 100644 virt/kvm/arm/vgic/vgic-nested-trace.h
>>>>  create mode 100644 virt/kvm/arm/vgic/vgic-v3-nested.c
>>>>
>>> When working on adding support for EL2 to kvm-unit-tests I was able to trigger
>>> the following warning:
>>>
>>> # ./lkvm run -f psci.flat -m 128 -c 8 --console serial --irqchip gicv3 --nested
>>>   # lkvm run --firmware psci.flat -m 128 -c 8 --name guest-151
>>>   Info: Placing fdt at 0x80200000 - 0x80210000
>>>   # Warning: The maximum recommended amount of VCPUs is 4
>>> chr_testdev_init: chr-testdev: can't find a virtio-console
>>> INFO: PSCI version 1.0
>>> PASS: invalid-function
>>> PASS: affinity-info-on
>>> PASS: affinity-info-off
>>> [   24.381266] WARNING: CPU: 3 PID: 160 at
>>> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
>>> kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.381366] Modules linked in:
>>> [   24.381466] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Not tainted
>>> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
>>> [   24.381566] Hardware name: Foundation-v8A (DT)
>>> [   24.381566] pstate: 40400009 (nZcv daif +PAN -UAO)
>>> [   24.381666] pc : kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.381766] lr : timer_emulate+0x24/0x98
>>> [   24.381766] sp : ffff000013d8b780
>>> [   24.381866] x29: ffff000013d8b780 x28: ffff80087a639b80
>>> [   24.381966] x27: ffff000010ba8648 x26: ffff000010b71b40
>>> [   24.382066] x25: ffff80087a63a100 x24: 0000000000000000
>>> [   24.382111] x23: 000080086ca54000 x22: ffff0000100ce260
>>> [   24.382166] x21: ffff800875e7c918 x20: ffff800875e7a800
>>> [   24.382275] x19: ffff800875e7ca08 x18: 0000000000000000
>>> [   24.382366] x17: 0000000000000000 x16: 0000000000000000
>>> [   24.382466] x15: 0000000000000000 x14: 0000000000002118
>>> [   24.382566] x13: 0000000000002190 x12: 0000000000002280
>>> [   24.382566] x11: 0000000000002208 x10: 0000000000000040
>>> [   24.382666] x9 : ffff000012dc3b38 x8 : 0000000000000000
>>> [   24.382766] x7 : 0000000000000000 x6 : ffff80087ac00248
>>> [   24.382866] x5 : 000080086ca54000 x4 : 0000000000002118
>>> [   24.382966] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
>>> [   24.383066] x1 : 0000000000000001 x0 : ffff800875e7ca08
>>> [   24.383066] Call trace:
>>> [   24.383166]  kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.383266]  kvm_timer_vcpu_load+0x9c/0x1a0
>>> [   24.383366]  kvm_arch_vcpu_load+0xb0/0x1f0
>>> [   24.383366]  kvm_sched_in+0x1c/0x28
>>> [   24.383466]  finish_task_switch+0xd8/0x1d8
>>> [   24.383566]  __schedule+0x248/0x4a0
>>> [   24.383666]  preempt_schedule_irq+0x60/0x90
>>> [   24.383666]  el1_irq+0xd0/0x180
>>> [   24.383766]  kvm_handle_guest_abort+0x0/0x3a0
>>> [   24.383866]  kvm_arch_vcpu_ioctl_run+0x41c/0x688
>>> [   24.383866]  kvm_vcpu_ioctl+0x4c0/0x838
>>> [   24.383966]  do_vfs_ioctl+0xb8/0x878
>>> [   24.384077]  ksys_ioctl+0x84/0x90
>>> [   24.384166]  __arm64_sys_ioctl+0x18/0x28
>>> [   24.384166]  el0_svc_common.constprop.0+0xb0/0x168
>>> [   24.384266]  el0_svc_handler+0x28/0x78
>>> [   24.384366]  el0_svc+0x8/0xc
>>> [   24.384366] ---[ end trace 37a32293e43ac12c ]---
>>> [   24.384666] WARNING: CPU: 3 PID: 160 at
>>> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
>>> kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.384766] Modules linked in:
>>> [   24.384866] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G        W
>>> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
>>> [   24.384966] Hardware name: Foundation-v8A (DT)
>>> [   24.384966] pstate: 40400009 (nZcv daif +PAN -UAO)
>>> [   24.385066] pc : kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.385166] lr : timer_emulate+0x24/0x98
>>> [   24.385166] sp : ffff000013d8b780
>>> [   24.385266] x29: ffff000013d8b780 x28: ffff80087a639b80
>>> [   24.385366] x27: ffff000010ba8648 x26: ffff000010b71b40
>>> [   24.385466] x25: ffff80087a63a100 x24: 0000000000000000
>>> [   24.385466] x23: 000080086ca54000 x22: ffff0000100ce260
>>> [   24.385566] x21: ffff800875e7c918 x20: ffff800875e7a800
>>> [   24.385666] x19: ffff800875e7ca80 x18: 0000000000000000
>>> [   24.385766] x17: 0000000000000000 x16: 0000000000000000
>>> [   24.385866] x15: 0000000000000000 x14: 0000000000002118
>>> [   24.385966] x13: 0000000000002190 x12: 0000000000002280
>>> [   24.385966] x11: 0000000000002208 x10: 0000000000000040
>>> [   24.386066] x9 : ffff000012dc3b38 x8 : 0000000000000000
>>> [   24.386166] x7 : 0000000000000000 x6 : ffff80087ac00248
>>> [   24.386266] x5 : 000080086ca54000 x4 : 0000000000002118
>>> [   24.386366] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
>>> [   24.386466] x1 : 0000000000000001 x0 : ffff800875e7ca80
>>> [   24.386466] Call trace:
>>> [   24.386566]  kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.386666]  kvm_timer_vcpu_load+0xa8/0x1a0
>>> [   24.386666]  kvm_arch_vcpu_load+0xb0/0x1f0
>>> [   24.386898]  kvm_sched_in+0x1c/0x28
>>> [   24.386966]  finish_task_switch+0xd8/0x1d8
>>> [   24.387166]  __schedule+0x248/0x4a0
>>> [   24.387354]  preempt_schedule_irq+0x60/0x90
>>> [   24.387366]  el1_irq+0xd0/0x180
>>> [   24.387466]  kvm_handle_guest_abort+0x0/0x3a0
>>> [   24.387566]  kvm_arch_vcpu_ioctl_run+0x41c/0x688
>>> [   24.387566]  kvm_vcpu_ioctl+0x4c0/0x838
>>> [   24.387666]  do_vfs_ioctl+0xb8/0x878
>>> [   24.387766]  ksys_ioctl+0x84/0x90
>>> [   24.387866]  __arm64_sys_ioctl+0x18/0x28
>>> [   24.387866]  el0_svc_common.constprop.0+0xb0/0x168
>>> [   24.387966]  el0_svc_handler+0x28/0x78
>>> [   24.388066]  el0_svc+0x8/0xc
>>> [   24.388066] ---[ end trace 37a32293e43ac12d ]---
>>> PASS: cpu-on
>>> SUMMARY: 4 te[   24.390266] WARNING: CPU: 3 PID: 160 at
>>> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
>>> kvm_timer_irq_can_fire+0xc/0x30
>>> s[   24.390366] Modules linked in:
>>> ts[   24.390366] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G        W
>>> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
>>> [   24.390566] Hardware name: Foundation-v8A (DT)
>>>
>>> [   24.390795] pstate: 40400009 (nZcv daif +PAN -UAO)
>>> [   24.390866] pc : kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.390966] lr : timer_emulate+0x24/0x98
>>> [   24.391066] sp : ffff000013d8b780
>>> [   24.391066] x29: ffff000013d8b780 x28: ffff80087a639b80
>>> [   24.391166] x27: ffff000010ba8648 x26: ffff000010b71b40
>>> [   24.391266] x25: ffff80087a63a100 x24: 0000000000000000
>>> [   24.391366] x23: 000080086ca54000 x22: 0000000000000003
>>> [   24.391466] x21: ffff800875e7c918 x20: ffff800875e7a800
>>> [   24.391466] x19: ffff800875e7ca08 x18: 0000000000000000
>>> [   24.391566] x17: 0000000000000000 x16: 0000000000000000
>>> [   24.391666] x15: 0000000000000000 x14: 0000000000002118
>>> [   24.391766] x13: 0000000000002190 x12: 0000000000002280
>>> [   24.391866] x11: 0000000000002208 x10: 0000000000000040
>>> [   24.391942] x9 : ffff000012dc3b38 x8 : 0000000000000000
>>> [   24.391966] x7 : 0000000000000000 x6 : ffff80087ac00248
>>> [   24.392066] x5 : 000080086ca54000 x4 : 0000000000002118
>>> [   24.392166] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
>>> [   24.392269] x1 : 0000000000000001 x0 : ffff800875e7ca08
>>> [   24.392366] Call trace:
>>> [   24.392433]  kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.392466]  kvm_timer_vcpu_load+0x9c/0x1a0
>>> [   24.392597]  kvm_arch_vcpu_load+0xb0/0x1f0
>>> [   24.392666]  kvm_sched_in+0x1c/0x28
>>> [   24.392766]  finish_task_switch+0xd8/0x1d8
>>> [   24.392766]  __schedule+0x248/0x4a0
>>> [   24.392866]  preempt_schedule_irq+0x60/0x90
>>> [   24.392966]  el1_irq+0xd0/0x180
>>> [   24.392966]  kvm_handle_guest_abort+0x0/0x3a0
>>> [   24.393066]  kvm_arch_vcpu_ioctl_run+0x41c/0x688
>>> [   24.393166]  kvm_vcpu_ioctl+0x4c0/0x838
>>> [   24.393266]  do_vfs_ioctl+0xb8/0x878
>>> [   24.393266]  ksys_ioctl+0x84/0x90
>>> [   24.393366]  __arm64_sys_ioctl+0x18/0x28
>>> [   24.393466]  el0_svc_common.constprop.0+0xb0/0x168
>>> [   24.393566]  el0_svc_handler+0x28/0x78
>>> [   24.393566]  el0_svc+0x8/0xc
>>> [   24.393666] ---[ end trace 37a32293e43ac12e ]---
>>> [   24.393866] WARNING: CPU: 3 PID: 160 at
>>> arch/arm64/kvm/../../../virt/kvm/arm/arch_timer.c:170
>>> kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.394066] Modules linked in:
>>> [   24.394266] CPU: 3 PID: 160 Comm: kvm-vcpu-1 Tainted: G        W
>>> 5.2.0-rc5-00060-g7dbce63bd1c7 #145
>>> [   24.394366] Hardware name: Foundation-v8A (DT)
>>> [   24.394466] pstate: 40400009 (nZcv daif +PAN -UAO)
>>> [   24.394466] pc : kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.394566] lr : timer_emulate+0x24/0x98
>>> [   24.394666] sp : ffff000013d8b780
>>> [   24.394727] x29: ffff000013d8b780 x28: ffff80087a639b80
>>> [   24.394766] x27: ffff000010ba8648 x26: ffff000010b71b40
>>> [   24.394866] x25: ffff80087a63a100 x24: 0000000000000000
>>> [   24.394966] x23: 000080086ca54000 x22: 0000000000000003
>>> [   24.394966] x21: ffff800875e7c918 x20: ffff800875e7a800
>>> [   24.395066] x19: ffff800875e7ca80 x18: 0000000000000000
>>> [   24.395166] x17: 0000000000000000 x16: 0000000000000000
>>> [   24.395266] x15: 0000000000000000 x14: 0000000000002118
>>> [   24.395383] x13: 0000000000002190 x12: 0000000000002280
>>> [   24.395466] x11: 0000000000002208 x10: 0000000000000040
>>> [   24.395547] x9 : ffff000012dc3b38 x8 : 0000000000000000
>>> [   24.395666] x7 : 0000000000000000 x6 : ffff80087ac00248
>>> [   24.395866] x5 : 000080086ca54000 x4 : 0000000000002118
>>> [   24.395966] x3 : eeeeeeeeeeeeeeef x2 : ffff800875e7c918
>>> [   24.396066] x1 : 0000000000000001 x0 : ffff800875e7ca80
>>> [   24.396066] Call trace:
>>> [   24.396166]  kvm_timer_irq_can_fire+0xc/0x30
>>> [   24.396266]  kvm_timer_vcpu_load+0xa8/0x1a0
>>> [   24.396366]  kvm_arch_vcpu_load+0xb0/0x1f0
>>> [   24.396366]  kvm_sched_in+0x1c/0x28
>>> [   24.396466]  finish_task_switch+0xd8/0x1d8
>>> [   24.396566]  __schedule+0x248/0x4a0
>>> [   24.396666]  preempt_schedule_irq+0x60/0x90
>>> [   24.396666]  el1_irq+0xd0/0x180
>>> [   24.396766]  kvm_handle_guest_abort+0x0/0x3a0
>>> [   24.396866]  kvm_arch_vcpu_ioctl_run+0x41c/0x688
>>> [   24.396866]  kvm_vcpu_ioctl+0x4c0/0x838
>>> [   24.397021]  do_vfs_ioctl+0xb8/0x878
>>> [   24.397066]  ksys_ioctl+0x84/0x90
>>> [   24.397166]  __arm64_sys_ioctl+0x18/0x28
>>> [   24.397348]  el0_svc_common.constprop.0+0xb0/0x168
>>> [   24.397366]  el0_svc_handler+0x28/0x78
>>> [   24.397566]  el0_svc+0x8/0xc
>>> [   24.397676] ---[ end trace 37a32293e43ac12f ]---
>>>
>>>   # KVM compatibility warning.
>>>     virtio-9p device was not detected.
>>>     While you have requested a virtio-9p device, the guest kernel did not
>>> initialize it.
>>>     Please make sure that the guest kernel was compiled with
>>> CONFIG_NET_9P_VIRTIO=y enabled in .config.
>>>
>>>   # KVM compatibility warning.
>>>     virtio-net device was not detected.
>>>     While you have requested a virtio-net device, the guest kernel did not
>>> initialize it.
>>>     Please make sure that the guest kernel was compiled with CONFIG_VIRTIO_NET=y
>>> enabled in .config.
>>>
>>> [..]
>> Did some investigating and this was caused by a bug in kvm-unit-tests (the fix
>> for it will be part of the EL2 patches for kvm-unit-tests). The guest was trying
>> to fetch an instruction from address 0x200, which KVM interprets as a prefetch
>> abort on an I/O address and ends up calling kvm_inject_pabt. The code from
>> arch/arm64/kvm/inject_fault.c doesn't know anything about nested virtualization,
>> and it sets the VCPU mode directly to PSR_MODE_EL1h. This makes_hyp_ctxt return
>> false, and get_timer_map will return an incorrect mapping.
>>
>> On next kvm_timer_vcpu_put, the direct timers will be {p,v}timer, and
>> h{p,v}timer->loaded will not be set to false. In the corresponding call to
>> kvm_timer_vcpu_load, KVM will try to emulate the hptimer and hvtimer, which
>> still have loaded = true. And this causes the warning I saw.
>>
> Hi Alexandru,
>
> While a unit test in kvm-unit-tests may not do what it should in order to
> exercise the code it's targeting appropriately, and therefore need to be
> fixed in order to do that, I'd argue that if a guest can induce a host
> warning then that's a host bug. Indeed now that you've analyzed the
> issue you could write a kvm-unit-tests test to specifically reproduce the
> warning and then use that test to test any host fix candidates.
>
> Thanks,
> drew
>
It was a host bug triggered by a bug in kvm-unit-tests. The kvm-unit-tests bug
is a real bug because it goes against the intent of the psci test. It wasn't
discovered until now because with the upstream version of Linux we don't get any
messages about it. I'll post a patch for it as soon as I can and we can discuss
how we want to fix it :)

Thanks,
Alex
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

  reply index

Thread overview: 175+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-21  9:37 Marc Zyngier
2019-06-21  9:37 ` [PATCH 01/59] KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s Marc Zyngier
2019-06-24 11:16   ` Dave Martin
2019-06-24 12:59   ` Alexandru Elisei
2019-07-03 12:32     ` Marc Zyngier
2019-06-21  9:37 ` [PATCH 02/59] KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h Marc Zyngier
2019-06-24 11:19   ` Dave Martin
2019-07-03  9:30     ` Marc Zyngier
2019-07-03 16:13       ` Dave Martin
2019-06-21  9:37 ` [PATCH 03/59] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Marc Zyngier
2019-06-21 13:08   ` Julien Thierry
2019-06-21 13:22     ` Marc Zyngier
2019-06-21 13:44   ` Suzuki K Poulose
2019-06-24 11:24   ` Dave Martin
2019-06-21  9:37 ` [PATCH 04/59] KVM: arm64: nv: Introduce nested virtualization VCPU feature Marc Zyngier
2019-06-21 13:08   ` Julien Thierry
2019-06-24 11:28   ` Dave Martin
2019-07-03 11:53     ` Marc Zyngier
2019-07-03 16:27       ` Dave Martin
2019-06-24 11:43   ` Dave Martin
2019-07-03 11:56     ` Marc Zyngier
2019-07-03 16:24       ` Dave Martin
2019-06-21  9:37 ` [PATCH 05/59] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Marc Zyngier
2019-06-24 10:19   ` Suzuki K Poulose
2019-06-24 11:38   ` Dave Martin
2019-06-21  9:37 ` [PATCH 06/59] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Marc Zyngier
2019-06-21 13:24   ` Julien Thierry
2019-06-21 13:50     ` Marc Zyngier
2019-06-24 12:48       ` Dave Martin
2019-07-03  9:21         ` Marc Zyngier
2019-07-04 10:00           ` Dave Martin
2019-06-21  9:37 ` [PATCH 07/59] KVM: arm64: nv: Add EL2 system registers to vcpu context Marc Zyngier
2019-06-24 12:54   ` Dave Martin
2019-07-03 12:20     ` Marc Zyngier
2019-07-03 16:31       ` Dave Martin
2019-06-24 15:47   ` Alexandru Elisei
2019-07-03 13:20     ` Marc Zyngier
2019-07-03 16:01       ` Marc Zyngier
2019-07-01 16:36   ` Suzuki K Poulose
2019-06-21  9:37 ` [PATCH 08/59] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Marc Zyngier
2019-06-24 12:59   ` Dave Martin
2019-06-21  9:37 ` [PATCH 09/59] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Marc Zyngier
2019-06-24 13:08   ` Dave Martin
2019-06-21  9:37 ` [PATCH 10/59] KVM: arm64: nv: Support virtual EL2 exceptions Marc Zyngier
2019-07-08 13:56   ` Steven Price
2019-06-21  9:37 ` [PATCH 11/59] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Marc Zyngier
2019-06-25 13:13   ` Alexandru Elisei
2019-07-03 14:16     ` Marc Zyngier
2019-07-30 14:08     ` Alexandru Elisei
2019-06-21  9:37 ` [PATCH 12/59] KVM: arm64: nv: Handle trapped ERET from " Marc Zyngier
2019-07-02 12:00   ` Alexandru Elisei
2019-06-21  9:37 ` [PATCH 13/59] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Marc Zyngier
2019-06-24 12:42   ` Julien Thierry
2019-06-25 14:02     ` Alexandru Elisei
2019-07-03 12:15     ` Marc Zyngier
2019-07-03 15:21       ` Julien Thierry
2019-06-25 15:18   ` Alexandru Elisei
2019-07-01  9:58     ` Alexandru Elisei
2019-07-03 15:59     ` Marc Zyngier
2019-07-03 16:32       ` Alexandru Elisei
2019-07-04 14:39         ` Marc Zyngier
2019-06-26 15:04   ` Alexandru Elisei
2019-07-04 15:05     ` Marc Zyngier
2019-07-01 12:10   ` Alexandru Elisei
2019-06-21  9:37 ` [PATCH 14/59] KVM: arm64: nv: Handle SPSR_EL2 specially Marc Zyngier
2019-06-21  9:37 ` [PATCH 15/59] KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg Marc Zyngier
2019-06-24 15:07   ` Julien Thierry
2019-07-03 13:09     ` Marc Zyngier
2019-06-27  9:21   ` Alexandru Elisei
2019-07-04 15:15     ` Marc Zyngier
2019-06-21  9:38 ` [PATCH 16/59] KVM: arm64: nv: Save/Restore vEL2 sysregs Marc Zyngier
2019-06-25  8:48   ` Julien Thierry
2019-07-03 13:42     ` Marc Zyngier
2019-07-01 12:09   ` Alexandru Elisei
2019-08-21 11:57   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 17/59] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Marc Zyngier
2019-06-21  9:38 ` [PATCH 18/59] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Marc Zyngier
2019-07-01 16:12   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 19/59] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from " Marc Zyngier
2019-06-21  9:38 ` [PATCH 20/59] KVM: arm64: nv: Trap CPACR_EL1 access in " Marc Zyngier
2019-07-01 16:40   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 21/59] KVM: arm64: nv: Set a handler for the system instruction traps Marc Zyngier
2019-06-25 12:55   ` Julien Thierry
2019-07-03 14:15     ` Marc Zyngier
2019-06-21  9:38 ` [PATCH 22/59] KVM: arm64: nv: Handle PSCI call via smc from the guest Marc Zyngier
2019-06-21  9:38 ` [PATCH 23/59] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Marc Zyngier
2019-06-25 14:19   ` Julien Thierry
2019-07-02 12:54     ` Alexandru Elisei
2019-07-03 14:18     ` Marc Zyngier
2019-06-21  9:38 ` [PATCH 24/59] KVM: arm64: nv: Respect virtual CPTR_EL2.TFP setting Marc Zyngier
2019-06-21  9:38 ` [PATCH 25/59] KVM: arm64: nv: Don't expose SVE to nested guests Marc Zyngier
2019-06-21  9:38 ` [PATCH 26/59] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Marc Zyngier
2019-06-26  5:31   ` Julien Thierry
2019-07-03 16:31     ` Marc Zyngier
2019-06-21  9:38 ` [PATCH 27/59] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Marc Zyngier
2019-06-26  6:55   ` Julien Thierry
2019-07-04 14:57     ` Marc Zyngier
2019-06-21  9:38 ` [PATCH 28/59] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Marc Zyngier
2019-06-26  7:23   ` Julien Thierry
2019-07-02 16:32   ` Alexandru Elisei
2019-07-03  9:10     ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 29/59] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Marc Zyngier
2019-07-03  9:16   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 30/59] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Marc Zyngier
2019-06-21  9:38 ` [PATCH 31/59] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Marc Zyngier
2019-06-21  9:38 ` [PATCH 32/59] KVM: arm64: nv: Hide RAS from nested guests Marc Zyngier
2019-07-03 13:59   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 33/59] KVM: arm64: nv: Pretend we only support larger-than-host page sizes Marc Zyngier
2019-07-03 14:13   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 34/59] KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm Marc Zyngier
2019-07-03 15:52   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 35/59] KVM: arm/arm64: nv: Support multiple nested stage 2 mmu structures Marc Zyngier
2019-06-25 12:19   ` Alexandru Elisei
2019-07-03 13:47     ` Marc Zyngier
2019-06-27 13:15   ` Julien Thierry
2019-07-04 15:51   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 36/59] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Marc Zyngier
2019-06-21  9:38 ` [PATCH 37/59] KVM: arm64: nv: Handle shadow stage 2 page faults Marc Zyngier
2019-07-05 14:28   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 38/59] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Marc Zyngier
2019-07-01  8:03   ` Julien Thierry
2019-06-21  9:38 ` [PATCH 39/59] KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu Marc Zyngier
2019-07-01  9:10   ` Julien Thierry
2019-07-05 15:28   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 40/59] KVM: arm64: nv: Don't always start an S2 MMU search from the beginning Marc Zyngier
2019-07-09  9:59   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 41/59] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Marc Zyngier
2019-06-21  9:38 ` [PATCH 42/59] KVM: arm64: nv: Rework the system instruction emulation framework Marc Zyngier
2019-06-21  9:38 ` [PATCH 43/59] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Marc Zyngier
2019-07-01 15:45   ` Julien Thierry
2019-07-09 13:20   ` Alexandru Elisei
2019-07-18 12:13     ` Tomasz Nowicki
2019-07-18 12:36       ` Alexandru Elisei
2019-07-18 12:56         ` Alexandru Elisei
2019-07-18 12:59         ` Tomasz Nowicki
2019-07-24 10:25   ` Tomasz Nowicki
2019-07-24 12:39     ` Marc Zyngier
2019-07-24 13:56       ` Tomasz Nowicki
2019-06-21  9:38 ` [PATCH 44/59] KVM: arm64: nv: Trap and emulate TLBI " Marc Zyngier
2019-07-02 12:37   ` Julien Thierry
2019-07-10 10:15   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 45/59] KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors Marc Zyngier
2019-06-21  9:38 ` [PATCH 46/59] KVM: arm64: nv: arch_timer: Support hyp timer emulation Marc Zyngier
2019-07-10 16:23   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 47/59] KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer Marc Zyngier
2019-08-08  9:34   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 48/59] KVM: arm64: nv: Load timer before the GIC Marc Zyngier
2019-07-11 13:17   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 49/59] KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu Marc Zyngier
2019-06-21  9:38 ` [PATCH 50/59] KVM: arm64: nv: Nested GICv3 Support Marc Zyngier
2019-07-16 11:41   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 51/59] KVM: arm64: nv: vgic: Emulate the HW bit in software Marc Zyngier
2019-06-21  9:38 ` [PATCH 52/59] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Marc Zyngier
2019-07-04  7:38   ` Julien Thierry
2019-07-04  9:01     ` Andre Przywara
2019-07-04  9:04       ` Julien Thierry
2019-06-21  9:38 ` [PATCH 53/59] KVM: arm64: nv: Implement maintenance interrupt forwarding Marc Zyngier
2019-07-04  8:06   ` Julien Thierry
2019-07-16 16:35   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 54/59] KVM: arm64: nv: Add nested GICv3 tracepoints Marc Zyngier
2019-06-21  9:38 ` [PATCH 55/59] arm64: KVM: nv: Add handling of EL2-specific timer registers Marc Zyngier
2019-07-11 12:35   ` Alexandru Elisei
2019-07-17 10:19   ` Alexandru Elisei
2019-06-21  9:38 ` [PATCH 56/59] arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 Marc Zyngier
2019-06-21  9:38 ` [PATCH 57/59] arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits Marc Zyngier
2019-06-21  9:38 ` [PATCH 58/59] arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's Marc Zyngier
2019-06-21  9:38 ` [PATCH 59/59] arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Marc Zyngier
2019-06-21  9:57 ` [PATCH 00/59] KVM: arm64: ARMv8.3 Nested Virtualization support Itaru Kitayama
2019-06-21 11:21   ` Marc Zyngier
2019-08-02 10:11 ` Alexandru Elisei
2019-08-02 10:30   ` Andrew Jones
2019-08-09 10:01   ` Alexandru Elisei
2019-08-09 11:44     ` Andrew Jones
2019-08-09 12:00       ` Alexandru Elisei [this message]
2019-08-09 13:00         ` Andrew Jones

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6dafd748-257e-1d09-aecc-d5a2ab91bdc4@arm.com \
    --to=alexandru.elisei@arm.com \
    --cc=Andre.Przywara@arm.com \
    --cc=Dave.Martin@arm.com \
    --cc=Marc.Zyngier@arm.com \
    --cc=drjones@redhat.com \
    --cc=kvm@vger.kernel.org \
    --cc=kvmarm@lists.cs.columbia.edu \
    --cc=linux-arm-kernel@lists.infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

KVM ARM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvmarm/0 kvmarm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvmarm kvmarm/ https://lore.kernel.org/kvmarm \
		kvmarm@lists.cs.columbia.edu kvmarm@archiver.kernel.org
	public-inbox-index kvmarm


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/edu.columbia.cs.lists.kvmarm


AGPL code for this site: git clone https://public-inbox.org/ public-inbox