[PATCH v3 0/5] KVM: Ring-based dirty memory tracking for performant checkpointing solutions

[PATCH v3 0/5] KVM: Ring-based dirty memory tracking for performant checkpointing solutions

[Cao, Lei]

This patch series adds ring-based dirty memory tracking support for
performant checkpointing solutions. It can also be used by live migration
to improve predictability. 

Introduction

Brendan Cully's Remus project white paper is one of the best written on 
the subject of fault tolerance using checkpoint/rollback techniques and 
is the best place to start for a general background. 
(http://www.cs.ubc.ca/~andy/papers/remus-nsdi-final.pdf)  
It gives a great outline of the basic requirements and characteristics 
of a checkpointed system, including a few of the performance issues.  
But Remus did not go far enough in the area of system performance for 
commercial production.

This patch series addresses known bottleneck and limitation in a 
checkpointed system: use of large bitmaps to track dirty memory.
These bitmaps are copied to userspace when userspace queries KVM for
its dirty page information. The use of bitmaps makes sense in the
live-migration method, as it is possible for all of memory to be dirtied
from one log-dirty pass to another. But in a checkpointed system, the
number of dirty pages is bounded such that the VM is paused when it has
dirtied a pre-defined number of pages. Traversing a large, sparsely
populated bitmap to find set bits is time-consuming, as is copying the
bitmap to user-space.

The preferred data structure for performant checkpointing solutions is
a dense list of guest frame numbers (GFN). This patch series stores
the dirty list in kernel memory that can be memory mapped into 
userspace to allow speedy harvesting.

The modification and still more modifications to qemu have allowed us 
to run checkpoint cycles at rates up to 2500 per second,  while still 
allowing the VM to get useful work done.

Design Goals

The patch series does not change or remove any existing KVM functionality.
It represents only additional functions (ioctls) into KVM from user space 
and these changes coexist with the current dirty memory logging facilities. 
It is possible to run multiple guests such that some of the guests
perform live migration using the existing memory logging mechanism and 
others migrate or run in fault tolerant mode using the new memory tracking 
functions.  

Modifications

All modifications affect only the KVM instance where the primary (active) VM 
is running, and these modifications are not in play on the standby (passive) 
host, where a VM is created that matches the primary in its configuration, 
but it does not execute until a migration/failover event occurs.

Patch 1: Add support for capabilities that can be enabled in a generic way.
	 Instroduce new capability: ring-based dirty memory logging
Patch 2: Add new data type, struct kvm_gfn_ring, and support functions for
	 ring-based dirty memory logging. Add new ioctl,
	 KVM_RESET_DIRTY_PAGES, for dirty trap reset.
Patch 3: Modify kvm_write_guest_cached() and kvm_write_guest_offset_cached() to
	 take vcpu as a parameter instead kvm.
Patch 4: Add new exit reason KVM_EXIT_DIRTY_LOG_FULL for dirty ring full
	 conditions.
Patch 5: Implement ring-base dirty memory tracking.

 Documentation/virtual/kvm/api.txt |  94 +++++++++-
 arch/powerpc/kvm/powerpc.c        |  14 +-
 arch/s390/kvm/kvm-s390.c          |  11 +-
 arch/x86/include/asm/kvm_host.h   |   5 +
 arch/x86/kvm/Makefile             |   3 +-
 arch/x86/kvm/lapic.c              |   4 +-
 arch/x86/kvm/mmu.c                |   7 +
 arch/x86/kvm/vmx.c                |   7 +
 arch/x86/kvm/x86.c                |  36 ++--
 include/linux/kvm_gfn_ring.h      |  37 ++++
 include/linux/kvm_host.h          |  20 ++-
 include/uapi/linux/kvm.h          |  33 ++++
 virt/kvm/gfn_ring.c               | 100 +++++++++++
 virt/kvm/kvm_main.c               | 267 ++++++++++++++++++++++++++--
 14 files changed, 569 insertions(+), 69 deletions(-)

Re: [PATCH v3 0/5] KVM: Ring-based dirty memory tracking for performant checkpointing solutions

[Paolo Bonzini]

On 03/02/2017 11:58, Cao, Lei wrote:
> This patch series adds ring-based dirty memory tracking support for
> performant checkpointing solutions. It can also be used by live migration
> to improve predictability. 

Thanks, this looks almost ready.  I will include patch 3 already in 4.11.

The API also seems easy to use.  QEMU can use async_safe_run_on_cpu to
stop all CPUs and harvest the dirty pages from the ring buffers, and it
can also harvest them in kvm_cpu_exec so that vCPU ring buffers are
handled in parallel.

I'm sure v4 will be the one! :)

Paolo