From: Xiao Guangrong <guangrong.xiao@gmail.com> To: Paolo Bonzini <pbonzini@redhat.com>, mtosatti@redhat.com, avi.kivity@gmail.com, rkrcmar@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Xiao Guangrong <xiaoguangrong@tencent.com> Subject: Re: [PATCH 0/7] KVM: MMU: fast write protect Date: Wed, 3 May 2017 22:50:55 +0800 [thread overview] Message-ID: <ff732c49-de24-eebd-ae8e-2de00281f211@gmail.com> (raw) In-Reply-To: <e186ce8d-52fc-3eca-1343-1a2f18331a76@redhat.com> On 05/03/2017 08:28 PM, Paolo Bonzini wrote: > So if I understand correctly this relies on userspace doing: > > 1) KVM_GET_DIRTY_LOG without write protect > 2) KVM_WRITE_PROTECT_ALL_MEM > <only look now at the dirty log snapshot> > > Writes may happen between 1 and 2; they are not represented in the live > dirty bitmap but it's okay because they are in the snapshot and will > only be used after 2. This is similar to what the dirty page ring > buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very > similar to KVM_RESET_DIRTY_PAGES in those patches. > You are right. After 1) and 2), the page which has been modified either in the bitmap returned to userspace or in the bitmap of memslot, i.e, there is no dirty page lost. > On 03/05/2017 12:52, guangrong.xiao@gmail.com wrote: >> Comparing with the ordinary algorithm which >> write protects last level sptes based on the rmap one by one, >> it just simply updates the generation number to ask all vCPUs >> to reload its root page table, particularly, it can be done out >> of mmu-lock, so that it does not hurt vMMU's parallel. > > This is clever. > > For processors that have PML, write protecting is only done on large > pages and only for splitting purposes; not for dirty page tracking > process at 4k granularity. In this case, I think that you should do > nothing in the new write-protect-all ioctl? Good point, thanks for you pointing it out. Doing nothing in write-protect-all() is not acceptable as it breaks its semantic. :( Furthermore, userspace has no knowledge about if PML is enable (it can be required from sysfs, but it is a good way in QEMU), so it is difficult for the usespace to know when to use write-protect-all. Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if PML is enabled? > > Also, I wonder how the alternative write protection mechanism would > affect performance of the dirty page ring buffer patches. You would do > the write protection of all memory at the end of > kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate > ioctl, which is nice. On the other hand, checkpoints would be more > frequent and most pages would be write-protected, so it would be more > expensive to rebuild the shadow page tables... Yup, write-protect-all can improve reset_dirty_pages indeed, i will apply your idea after reset_dirty_pages is merged. However, we still prefer to have a separate ioctl for write-protect-all which cooperates with KVM_GET_DIRTY_LOG to improve live migration that should not always depend on checkpoint. > > Thanks, > > Paolo > >> @@ -490,6 +511,7 @@ static int kvm_physical_sync_dirty_bitmap(KVMMemoryListener *kml, >> memset(d.dirty_bitmap, 0, allocated_size); >> >> d.slot = mem->slot | (kml->as_id << 16); >> + d.flags = kvm_write_protect_all ? KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT : 0; >> if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, &d) == -1) { >> DPRINTF("ioctl failed %d\n", errno); >> ret = -1; > > How would this work when kvm_physical_sync_dirty_bitmap is called from > memory_region_sync_dirty_bitmap rather than > memory_region_global_dirty_log_sync? You are right, we did not consider the full cases carefully, will fix it when push it to QEMU formally. Thank you, Paolo!
WARNING: multiple messages have this Message-ID (diff)
From: Xiao Guangrong <guangrong.xiao@gmail.com> To: Paolo Bonzini <pbonzini@redhat.com>, mtosatti@redhat.com, avi.kivity@gmail.com, rkrcmar@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, Xiao Guangrong <xiaoguangrong@tencent.com> Subject: Re: [Qemu-devel] [PATCH 0/7] KVM: MMU: fast write protect Date: Wed, 3 May 2017 22:50:55 +0800 [thread overview] Message-ID: <ff732c49-de24-eebd-ae8e-2de00281f211@gmail.com> (raw) In-Reply-To: <e186ce8d-52fc-3eca-1343-1a2f18331a76@redhat.com> On 05/03/2017 08:28 PM, Paolo Bonzini wrote: > So if I understand correctly this relies on userspace doing: > > 1) KVM_GET_DIRTY_LOG without write protect > 2) KVM_WRITE_PROTECT_ALL_MEM > <only look now at the dirty log snapshot> > > Writes may happen between 1 and 2; they are not represented in the live > dirty bitmap but it's okay because they are in the snapshot and will > only be used after 2. This is similar to what the dirty page ring > buffer patches do; in fact, the KVM_WRITE_PROTECT_ALL_MEM ioctl is very > similar to KVM_RESET_DIRTY_PAGES in those patches. > You are right. After 1) and 2), the page which has been modified either in the bitmap returned to userspace or in the bitmap of memslot, i.e, there is no dirty page lost. > On 03/05/2017 12:52, guangrong.xiao@gmail.com wrote: >> Comparing with the ordinary algorithm which >> write protects last level sptes based on the rmap one by one, >> it just simply updates the generation number to ask all vCPUs >> to reload its root page table, particularly, it can be done out >> of mmu-lock, so that it does not hurt vMMU's parallel. > > This is clever. > > For processors that have PML, write protecting is only done on large > pages and only for splitting purposes; not for dirty page tracking > process at 4k granularity. In this case, I think that you should do > nothing in the new write-protect-all ioctl? Good point, thanks for you pointing it out. Doing nothing in write-protect-all() is not acceptable as it breaks its semantic. :( Furthermore, userspace has no knowledge about if PML is enable (it can be required from sysfs, but it is a good way in QEMU), so it is difficult for the usespace to know when to use write-protect-all. Maybe we can make KVM_CAP_X86_WRITE_PROTECT_ALL_MEM return false if PML is enabled? > > Also, I wonder how the alternative write protection mechanism would > affect performance of the dirty page ring buffer patches. You would do > the write protection of all memory at the end of > kvm_vm_ioctl_reset_dirty_pages. You wouldn't even need a separate > ioctl, which is nice. On the other hand, checkpoints would be more > frequent and most pages would be write-protected, so it would be more > expensive to rebuild the shadow page tables... Yup, write-protect-all can improve reset_dirty_pages indeed, i will apply your idea after reset_dirty_pages is merged. However, we still prefer to have a separate ioctl for write-protect-all which cooperates with KVM_GET_DIRTY_LOG to improve live migration that should not always depend on checkpoint. > > Thanks, > > Paolo > >> @@ -490,6 +511,7 @@ static int kvm_physical_sync_dirty_bitmap(KVMMemoryListener *kml, >> memset(d.dirty_bitmap, 0, allocated_size); >> >> d.slot = mem->slot | (kml->as_id << 16); >> + d.flags = kvm_write_protect_all ? KVM_DIRTY_LOG_WITHOUT_WRITE_PROTECT : 0; >> if (kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, &d) == -1) { >> DPRINTF("ioctl failed %d\n", errno); >> ret = -1; > > How would this work when kvm_physical_sync_dirty_bitmap is called from > memory_region_sync_dirty_bitmap rather than > memory_region_global_dirty_log_sync? You are right, we did not consider the full cases carefully, will fix it when push it to QEMU formally. Thank you, Paolo!
next prev parent reply other threads:[~2017-05-03 14:51 UTC|newest] Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-05-03 10:52 [PATCH 0/7] KVM: MMU: fast write protect guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 1/7] KVM: MMU: correct the behavior of mmu_spte_update_no_track guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 2/7] KVM: MMU: introduce possible_writable_spte_bitmap guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 3/7] KVM: MMU: introduce kvm_mmu_write_protect_all_pages guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 4/7] KVM: MMU: enable KVM_WRITE_PROTECT_ALL_MEM guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 5/7] KVM: MMU: allow dirty log without write protect guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 6/7] KVM: MMU: clarify fast_pf_fix_direct_spte guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 10:52 ` [PATCH 7/7] KVM: MMU: stop using mmu_spte_get_lockless under mmu-lock guangrong.xiao 2017-05-03 10:52 ` [Qemu-devel] " guangrong.xiao 2017-05-03 12:28 ` [PATCH 0/7] KVM: MMU: fast write protect Paolo Bonzini 2017-05-03 12:28 ` [Qemu-devel] " Paolo Bonzini 2017-05-03 14:50 ` Xiao Guangrong [this message] 2017-05-03 14:50 ` Xiao Guangrong 2017-05-03 14:57 ` Paolo Bonzini 2017-05-03 14:57 ` [Qemu-devel] " Paolo Bonzini 2017-05-04 3:36 ` Xiao Guangrong 2017-05-04 3:36 ` [Qemu-devel] " Xiao Guangrong 2017-05-04 7:06 ` Paolo Bonzini 2017-05-04 7:06 ` [Qemu-devel] " Paolo Bonzini 2017-05-23 2:23 ` Xiao Guangrong 2017-05-23 2:23 ` [Qemu-devel] " Xiao Guangrong 2017-05-29 16:48 ` Paolo Bonzini 2017-05-29 16:48 ` [Qemu-devel] " Paolo Bonzini 2017-06-09 3:19 ` Xiao Guangrong 2017-06-09 3:19 ` [Qemu-devel] " Xiao Guangrong 2017-06-05 7:36 ` Jay Zhou 2017-06-05 7:36 ` Jay Zhou 2017-06-06 2:56 ` Xiao Guangrong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=ff732c49-de24-eebd-ae8e-2de00281f211@gmail.com \ --to=guangrong.xiao@gmail.com \ --cc=avi.kivity@gmail.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mtosatti@redhat.com \ --cc=pbonzini@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=rkrcmar@redhat.com \ --cc=xiaoguangrong@tencent.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.