warning in kvm_hv_invalidate_tsc_page due to writes to guest memory from VM ioctl context

[Maxim Levitsky <mlevitsk@redhat.com>]

[102140.117649] WARNING: CPU: 10 PID: 579353 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:3161 mark_page_dirty_in_slot+0x6c/0x80 [kvm]
[102140.118063] Modules linked in: kvm_amd(O) ccp rng_core kvm(O) irqbypass tun xt_conntrack ip6table_filter ip6_tables tps6598x wmi_bmof snd_acp3x_pdm_dma snd_soc_dmic snd_acp3x_rn regmap_i2c
snd_soc_core ftdi_sio snd_ctl_led bfq snd_hda_codec_realtek iwlmvm btusb snd_hda_codec_generic snd_hda_codec_hdmi btrtl psmouse btbcm pcspkr btintel k10temp snd_hda_intel snd_intel_dspcfg tpm_crb
snd_hda_codec snd_hwdep snd_hda_core snd_pcm iwlwifi snd_rn_pci_acp3x i2c_piix4 tpm_tis tpm_tis_core thinkpad_acpi wmi platform_profile rtc_cmos i2c_multi_instantiate i2c_designware_platform
i2c_designware_core sch_fq_codel dm_crypt mmc_block hid_generic usbhid rtsx_pci_sdmmc mmc_core atkbd libps2 amdgpu drm_ttm_helper ttm gpu_sched i2c_algo_bit drm_kms_helper cfbfillrect syscopyarea
cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea nvme xhci_pci drm sp5100_tco nvme_core rtsx_pci ucsi_acpi xhci_hcd drm_panel_orientation_quirks mfd_core t10_pi typec_ucsi typec i8042
pinctrl_amd r8169 realtek 8250_pci
[102140.118133]  usbmon nbd fuse autofs4 [last unloaded: rng_core]
[102140.120708] CPU: 10 PID: 579353 Comm: qemu-system-x86 Tainted: G        W  O      5.16.0.stable #20
[102140.120971] Hardware name: LENOVO 20UF001CUS/20UF001CUS, BIOS R1CET65W(1.34 ) 06/17/2021
[102140.121206] RIP: 0010:mark_page_dirty_in_slot+0x6c/0x80 [kvm]
[102140.121441] Code: 21 00 00 0f bf b6 04 01 00 00 c1 e1 10 48 89 e5 09 ce e8 17 aa 00 00 5d c3 c3 48 8b 86 c0 00 00 00 48 63 d2 f0 48 0f ab 10 c3 <0f> 0b c3 0f 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00
0f 1f 00 0f 1f
[102140.121990] RSP: 0018:ffffc9000078fcb8 EFLAGS: 00010246
[102140.122152] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
[102140.122363] RDX: 0000000000108607 RSI: ffff88810a264600 RDI: ffffc90001e55000
[102140.122571] RBP: ffffc9000078fd08 R08: 0000000000000000 R09: 0000000000000000
[102140.122789] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88810a264600
[102140.123001] R13: 0000000000000004 R14: 0000000000108608 R15: ffffc90001e5e8cc
[102140.123210] FS:  00007ff0d9519d80(0000) GS:ffff8883ff880000(0000) knlGS:0000000000000000
[102140.123447] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102140.123618] CR2: 000055e6c099a0d8 CR3: 000000010a7f5000 CR4: 0000000000350ee0
[102140.123834] Call Trace:
[102140.123910]  <TASK>
[102140.123976]  ? kvm_write_guest+0x114/0x120 [kvm]
[102140.124183]  kvm_hv_invalidate_tsc_page+0x9e/0xf0 [kvm]
[102140.124396]  kvm_arch_vm_ioctl+0xa26/0xc50 [kvm]
[102140.124585]  ? schedule+0x4e/0xc0
[102140.124701]  ? __cond_resched+0x1a/0x50
[102140.124826]  ? futex_wait+0x166/0x250
[102140.124946]  ? __send_signal+0x1f1/0x3d0
[102140.125072]  kvm_vm_ioctl+0x747/0xda0 [kvm]
[102140.125264]  ? do_futex+0xa7/0x160
[102140.125370]  ? __x64_sys_futex+0x74/0x1b0
[102140.125493]  __x64_sys_ioctl+0x8e/0xc0
[102140.125612]  do_syscall_64+0x35/0x80
[102140.125738]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[102140.125893] RIP: 0033:0x7ff0dd6b72bb
[102140.126001] Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3d 2b 0f 00 f7
d8 64 89 01 48
[102140.126548] RSP: 002b:00007ffe28d41158 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[102140.126770] RAX: ffffffffffffffda RBX: 000055984440d610 RCX: 00007ff0dd6b72bb
[102140.126975] RDX: 00007ffe28d412a0 RSI: 000000004030ae7b RDI: 000000000000001e
[102140.127175] RBP: 00007ffe28d41250 R08: 0000000000000000 R09: 00000000ffffffff
[102140.127375] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ff0dec1c3a0
[102140.127574] R13: 0000559841ba2847 R14: 00005598442b03a0 R15: 0000559844077b10
[102140.127792]  </TASK>
[102140.127863] ---[ end trace b8a3ae7a8a53d467 ]---


This happens because kvm_hv_invalidate_tsc_page is called from kvm_vm_ioctl_set_clock
which is a VM wide ioctl and thus doesn't have to be called with an active vCPU.
 
But as I see the warring states that guest memory writes must alway be done
while a vCPU is active to allow the write to be recorded in its dirty track ring.
 
I _think_ it might be OK to drop this invalidation,
and rely on the fact that kvm_hv_setup_tsc_page will update it,
and it is called when vCPU0 processes KVM_REQ_CLOCK_UPDATE which is raised in the
kvm_vm_ioctl_set_clock eventually.
 
Vitaly, any thoughts on this?
 
For reference those are my HV flags:
 
    $cpu_flags: |
        $cpu_flags,
        hv_relaxed,hv_spinlocks=0x1fff,hv_vpindex,     # General HV features
        hv_runtime,hv_time,hv-frequencies,             # Clock stuff        
        hv_synic,hv_stimer,hv-stimer-direct,#hv-vapic, # APIC extensions
        #hv-tlbflush,hv-ipi,                           # IPI extensions
        hv-reenlightenment,                            # nested stuff
 
 
 
PS: unrelated question:
 
Vitaly, do you know why hv-evmcs needs hv-vapic?
 
 
I know that they stuffed the eVMCS pointer to HV_X64_MSR_VP_ASSIST_PAGE,
 
But can't we set HV_APIC_ACCESS_AVAILABLE but not HV_APIC_ACCESS_RECOMMENDED
so that guest would hopefully still know that HV assist page is available,
but should not use it for APIC acceleration?
 
Best regards,
	Maxim Levitsky