Re: [RFC] KVM Fault Tolerance: Kemari for KVM

[Avi Kivity <avi@redhat.com>]

On 11/17/2009 01:04 PM, Yoshiaki Tamura wrote:
>> What I mean is:
>>
>> - choose synchronization point A
>> - start copying memory for synchronization point A
>>   - output is delayed
>> - choose synchronization point B
>> - copy memory for A and B
>>    if guest touches memory not yet copied for A, COW it
>> - once A copying is complete, release A output
>> - continue copying memory for B
>> - choose synchronization point B
>>
>> by keeping two synchronization points active, you don't have any 
>> pauses.  The cost is maintaining copy-on-write so we can copy dirty 
>> pages for A while keeping execution.
>
>
> The overall idea seems good, but if I'm understanding correctly, we 
> need a buffer for copying memory locally, and when it gets full, or 
> when we COW the memory for B, we still have to pause the guest to 
> prevent from overwriting. Correct?

Yes.  During COW the guest would not be able to access the page, but if 
other vcpus access other pages, they can still continue.  So generally 
synchronization would be pauseless.

> To make things simple, we would like to start with the synchronous 
> transmission first, and tackle asynchronous transmission later.

Of course.  I'm just worried that realistic workloads will drive the 
latency beyond acceptable limits.

>
>>>> How many pages do you copy per synchronization point for reasonably 
>>>> difficult workloads?
>>>
>>> That is very workload-dependent, but if you take a look at the examples
>>> below you can get a feeling of how Kemari behaves.
>>>
>>> IOzone            Kemari sync interval[ms]  dirtied pages
>>> ---------------------------------------------------------
>>> buffered + fsync                       400           3000
>>> O_SYNC                                  10             80
>>>
>>> In summary, if the guest executes few I/O operations, the interval
>>> between Kemari synchronizations points will increase and the number of
>>> dirtied pages will grow accordingly.
>>
>> In the example above, the externally observed latency grows to 400 
>> ms, yes?
>
> Not exactly.  The sync interval refers to the interval of 
> synchronization points captured when the workload is running.  In the 
> example above, when the observed sync interval is 400ms, it takes 
> about 150ms to sync VMs with 3000 dirtied pages.  Kemari resumes I/O 
> operations immediately once the synchronization is finished, and thus, 
> the externally observed latency is 150ms in this case.

Not sure I understand.

If a packet is output from a guest immediately after a synchronization 
point, doesn't it need to be delayed until the next synchronization 
point?  So it's not just the guest pause time that matters, but also the 
interval between sync points?

-- 
error compiling committee.c: too many arguments to function