Re: [RFC PATCH] KVM: x86: Fix APIC page invalidation race

[Eiichi Tsukata <eiichi.tsukata@nutanix.com>]

> On Jun 8, 2020, at 22:13, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 06/06/20 06:26, Eiichi Tsukata wrote:
>> Commit b1394e745b94 ("KVM: x86: fix APIC page invalidation") tried to
>> fix inappropriate APIC page invalidation by re-introducing arch specific
>> kvm_arch_mmu_notifier_invalidate_range() and calling it from
>> kvm_mmu_notifier_invalidate_range_start. But threre could be the
>> following race because VMCS APIC address cache can be updated
>> *before* it is unmapped.
>> 
>> Race:
>>  (Invalidator) kvm_mmu_notifier_invalidate_range_start()
>>  (Invalidator) kvm_make_all_cpus_request(kvm, KVM_REQ_APIC_PAGE_RELOAD)
>>  (KVM VCPU) vcpu_enter_guest()
>>  (KVM VCPU) kvm_vcpu_reload_apic_access_page()
>>  (Invalidator) actually unmap page
>> 
>> Symptom:
>>  The above race can make Guest OS see already freed page and Guest OS
>> will see broken APIC register values.
> 
> This is not exactly the issue.  The values in the APIC-access page do
> not really matter, the problem is that the host physical address values
> won't match between the page tables and the APIC-access page address.
> Then the processor will not trap APIC accesses, and will instead show
> the raw contents of the APIC-access page (zeroes), and cause the crash
> as you mention below.
> 
> Still, the race explains the symptoms and the patch matches this text in
> include/linux/mmu_notifier.h:
> 
> 	 * If the subsystem
>         * can't guarantee that no additional references are taken to
>         * the pages in the range, it has to implement the
>         * invalidate_range() notifier to remove any references taken
>         * after invalidate_range_start().
> 
> where the "additional reference" is in the VMCS: because we have to
> account for kvm_vcpu_reload_apic_access_page running between
> invalidate_range_start() and invalidate_range_end(), we need to
> implement invalidate_range().
> 
> The patch seems good, but I'd like Andrea Arcangeli to take a look as
> well so I've CCed him.
> 
> Thank you very much!
> 
> Paolo
> 

Hello Paolo

Thanks for detailed explanation!
I’ll fix the commit message like this:

```
Symptom:
  The above race can cause mismatch between the page tables and the
APIC-access page address in VMCS.Then the processor will not trap APIC
accesses, and will instead show the raw contents of the APIC-access page
(zeroes). Especially, Windows OS checks LAPIC modification so it can cause
BSOD crash with BugCheck CRITICAL_STRUCTURE_CORRUPTION (109). These symptoms
are the same as we previously saw in
https://bugzilla.kernel.org/show_bug.cgi?id=197951
and we are currently seeing in
https://bugzilla.redhat.com/show_bug.cgi?id=1751017.

To prevent mismatch between page tables and APIC-access page address,
this patch calls kvm_arch_mmu_notifier_invalidate_range() from
kvm_mmu_notifier_invalidate_range() instead of ..._range_start().
We need to implement invalidate_range() because we have to
account for kvm_vcpu_reload_apic_access_page() running between
invalidate_range_start() and invalidate_range_end().
```


Best

Eiichi