Re: [PATCH 00/11] KVM: x86: track guest page access

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: gleb@kernel.org, mtosatti@redhat.com, kvm@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	Andrea Arcangeli <aarcange@redhat.com>
Subject: Re: [PATCH 00/11] KVM: x86: track guest page access
Date: Sun, 6 Dec 2015 00:56:46 +0800	[thread overview]
Message-ID: <5663174E.7080709@linux.intel.com> (raw)
In-Reply-To: <565DD218.4080902@linux.intel.com>

Ping...

Paolo, any comment?

On 12/02/2015 01:00 AM, Xiao Guangrong wrote:
>
>
> On 12/01/2015 06:17 PM, Paolo Bonzini wrote:
>>
>>
>> On 30/11/2015 19:26, Xiao Guangrong wrote:
>>> This patchset introduces the feature which allows us to track page
>>> access in guest. Currently, only write access tracking is implemented
>>> in this version.
>>>
>>> Four APIs are introduces:
>>> - kvm_page_track_add_page(kvm, gfn, mode), single guest page @gfn is
>>>    added into the track pool of the guest instance represented by @kvm,
>>>    @mode specifies which kind of access on the @gfn is tracked
>>>
>>> - kvm_page_track_remove_page(kvm, gfn, mode), is the opposed operation
>>>    of kvm_page_track_add_page() which removes @gfn from the tracking pool.
>>>    gfn is no tracked after its last user is gone
>>>
>>> - kvm_page_track_register_notifier(kvm, n), register a notifier so that
>>>    the event triggered by page tracking will be received, at that time,
>>>    the callback of n->track_write() will be called
>>>
>>> - kvm_page_track_unregister_notifier(kvm, n), does the opposed operation
>>>    of kvm_page_track_register_notifier(), which unlinks the notifier and
>>>    stops receiving the tracked event
>>>
>>> The first user of page track is non-leaf shadow page tables as they are
>>> always write protected. It also gains performance improvement because
>>> page track speeds up page fault handler for the tracked pages. The
>>> performance result of kernel building is as followings:
>>>
>>>     before           after
>>> real 461.63       real 455.48
>>> user 4529.55      user 4557.88
>>> sys 1995.39       sys 1922.57
>>
>> For KVM-GT, as far as I know Andrea Arcangeli is working on extending
>> userfaultfd to tracking write faults only.  Perhaps KVM-GT can do
>> something similar, where KVM gets the write tracking functionality for
>> free through the MMU notifiers.  Any thoughts on this?
>
> Userfaultfd is excellent and has the ability to notify write event indeed,
> however, it is not suitable for the use case of shadow page.
>
> For the performance, shadow GPU is performance critical and requires
> frequently being switched, it is not good to handle it in userspace. And
> windows guest has many GPU tables and updates it frequently, that means,
> we need to write protect huge number of pages which are single page based,
> I am afraid userfaultfd can not handle this case efficiently.
>
> For the functionality, userfaultfd can not fill the need of shadow page
> because:
> - the page is keeping readonly, userfaultfd can not fix the fault and let
>    the vcpu progress (write access causes writeable gup).
>
> - the access need to be emulated, however, userfaultfd/kernel does not have
>    the ability to emulate the access as the access is trigged by guest, the
>    instruction info is stored in VMCS so that only KVM can emulate it.
>
> - shadow page needs to be notified after the emulation is finished as it
>    should know the new data written to the page to update its page hierarchy.
>    (some hardwares lack the 'retry' ability so the shadow page table need to
>     reflect the table in guest at any time).
>
>>
>> Applying your technique to non-leaf shadow pages actually makes this
>> series quite interesting. :)  Shadow paging is still in use for nested
>> EPT, so it's always a good idea to speed it up.
>
> Yes. Very glad to see you like it. :)
>
>