On Tue, 2021-03-30 at 12:59 -0400, Paolo Bonzini wrote:
> pvclock_gtod_sync_lock can be taken with interrupts disabled if the
> preempt notifier calls get_kvmclock_ns to update the Xen
> runstate information:
> 
>    spin_lock include/linux/spinlock.h:354 [inline]
>    get_kvmclock_ns+0x25/0x390 arch/x86/kvm/x86.c:2587
>    kvm_xen_update_runstate+0x3d/0x2c0 arch/x86/kvm/xen.c:69
>    kvm_xen_update_runstate_guest+0x74/0x320 arch/x86/kvm/xen.c:100
>    kvm_xen_runstate_set_preempted arch/x86/kvm/xen.h:96 [inline]
>    kvm_arch_vcpu_put+0x2d8/0x5a0 arch/x86/kvm/x86.c:4062
> 
> So change the users of the spinlock to spin_lock_irqsave and
> spin_unlock_irqrestore.

Apologies, I didn't spot this at the time. Looks sane enough (if we
ignore the elephant in the room that kvm_xen_update_runstate_guest() is
also writing to userspace with interrupts disabled on this preempted
code path, but I have a fix for that in the works¹).

However, in 5.15-rc5 I'm still seeing the warning below when I run
xen_shinfo_test. I confess I'm not entirely sure what it's telling me.


[   89.138354] =============================
[   89.138356] [ BUG: Invalid wait context ]
[   89.138358] 5.15.0-rc5+ #834 Tainted: G S        I E    
[   89.138360] -----------------------------
[   89.138361] xen_shinfo_test/2575 is trying to lock:
[   89.138363] ffffa34a0364efd8 (&kvm->arch.pvclock_gtod_sync_lock){....}-{3:3}, at: get_kvmclock_ns+0x1f/0x130 [kvm]
[   89.138442] other info that might help us debug this:
[   89.138444] context-{5:5}
[   89.138445] 4 locks held by xen_shinfo_test/2575:
[   89.138447]  #0: ffff972bdc3b8108 (&vcpu->mutex){+.+.}-{4:4}, at: kvm_vcpu_ioctl+0x77/0x6f0 [kvm]
[   89.138483]  #1: ffffa34a03662e90 (&kvm->srcu){....}-{0:0}, at: kvm_arch_vcpu_ioctl_run+0xdc/0x8b0 [kvm]
[   89.138526]  #2: ffff97331fdbac98 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0xff/0xbd0
[   89.138534]  #3: ffffa34a03662e90 (&kvm->srcu){....}-{0:0}, at: kvm_arch_vcpu_put+0x26/0x170 [kvm]
[   89.138576] stack backtrace:
[   89.138577] CPU: 27 PID: 2575 Comm: xen_shinfo_test Tainted: G S        I E     5.15.0-rc5+ #834
[   89.138580] Hardware name: Intel Corporation S2600CW/S2600CW, BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
[   89.138582] Call Trace:
[   89.138585]  dump_stack_lvl+0x6a/0x9a
[   89.138592]  __lock_acquire.cold+0x2ac/0x2d5
[   89.138597]  ? __lock_acquire+0x578/0x1f80
[   89.138604]  lock_acquire+0xc0/0x2d0
[   89.138608]  ? get_kvmclock_ns+0x1f/0x130 [kvm]
[   89.138648]  ? find_held_lock+0x2b/0x80
[   89.138653]  _raw_spin_lock_irqsave+0x48/0x60
[   89.138656]  ? get_kvmclock_ns+0x1f/0x130 [kvm]
[   89.138695]  get_kvmclock_ns+0x1f/0x130 [kvm]
[   89.138734]  kvm_xen_update_runstate+0x14/0x90 [kvm]
[   89.138783]  kvm_xen_update_runstate_guest+0x15/0xd0 [kvm]
[   89.138830]  kvm_arch_vcpu_put+0xe6/0x170 [kvm]
[   89.138870]  kvm_sched_out+0x2f/0x40 [kvm]
[   89.138900]  __schedule+0x5de/0xbd0
[   89.138904]  ? kvm_mmu_topup_memory_cache+0x21/0x70 [kvm]
[   89.138937]  __cond_resched+0x34/0x50
[   89.138941]  kmem_cache_alloc+0x228/0x2e0
[   89.138946]  kvm_mmu_topup_memory_cache+0x21/0x70 [kvm]
[   89.138979]  mmu_topup_memory_caches+0x1d/0x70 [kvm]
[   89.139024]  kvm_mmu_load+0x2d/0x750 [kvm]
[   89.139070]  ? kvm_cpu_has_extint+0x15/0x90 [kvm]
[   89.139113]  ? kvm_cpu_has_injectable_intr+0xe/0x50 [kvm]
[   89.139155]  vcpu_enter_guest+0xc77/0x1210 [kvm]
[   89.139195]  ? kvm_arch_vcpu_ioctl_run+0x146/0x8b0 [kvm]
[   89.139235]  kvm_arch_vcpu_ioctl_run+0x146/0x8b0 [kvm]
[   89.139274]  kvm_vcpu_ioctl+0x279/0x6f0 [kvm]
[   89.139306]  ? find_held_lock+0x2b/0x80
[   89.139312]  __x64_sys_ioctl+0x83/0xb0
[   89.139316]  do_syscall_64+0x3b/0x90
[   89.139320]  entry_SYSCALL_64_after_hwframe+0x44/0xae

¹ https://git.infradead.org/users/dwmw2/linux.git/commitdiff/ec22c08258