[RFC][PATCH] Kemari v1.0 released

[RFC][PATCH] Kemari v1.0 released

[Yoshiaki Tamura]

Hi,

I would like to announce the first release of Kemari, an open-source
virtual machine synchronization mechanism for fault tolerance.
It offers a feasible approach to fault tolerance that does not
require the use of specific hardware or modification of applications/OS.
Kemari aims to keep VMs transparently running in times of hardware
failures. It transfers the state of the primary VM to the secondary
VM when the primary VM is about to send an event to devices such as
storage and networks. The source code and further information are
available at the following site.

http://www.osrg.net/kemari/

Older slides and video are also available at,

http://www.xen.org/community/projects.html

In addition to Linux with PV Drivers, we tested Windows with James
Harpers's GPL PV. The video
(http://www.osrg.net/kemari/download/kemari_windows.mov) demonstrates
that Windows continued transparently when power failure occurred to the
primary server.

Kemari is now listed on Xen Product Roadmap,
(http://www.xen.org/download/roadmap.html)
and we would like to ask for your reviews and comments.
Briefly, Kemari in VMM taps event channel, pauses the guest (not
suspend), prepares for transfer, and Kemari in userland transfers
the guest. On failover, Kemari on the secondary restores the guest,
and the backend drivers in dom0 set up the backend rings from the
state of the shared rings in the guest.

Could you please review the code? Any comments are appreciated.

We're also planning to merge with Remus from UBC to propose a better
solution for Xen 3.4 together.

Thanks,

Yoshi

-- 
TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: tamura.yoshiaki@lab.ntt.co.jp
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

Design document of Kemari

[Yoshiaki Tamura]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Hi,

We're currently preparing to submit patches of Kemari.
Since I didn't give presentation at the summit last week, I would like to share
the design document of Kemari, which is attached to this message. Any comments
are greatly appreciated. We'll submit patches based on this design this week.

You can also get the undivided patches from the following links.
http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch

If you need further information on the design or how it works, please visit
http://www.osrg.net/kemari/

Thanks,

Yoshi

-- 
TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: tamura.yoshiaki@lab.ntt.co.jp
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

[-- Attachment #2: kemari_design080302.pdf --]
[-- Type: application/pdf, Size: 489037 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Design document of Kemari

[Yoshiaki Tamura]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Hi,

We're currently preparing to submit patches of Kemari.
Since I didn't give presentation at the summit last week, I would like to share
the design document of Kemari, which is attached to this message. Any comments
are greatly appreciated. We'll submit patches based on this design this week.

You can also get the undivided patches from the following links.
http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch

If you need further information on the design or how it works, please visit
http://www.osrg.net/kemari/

Thanks,

Yoshi

-- 
TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: tamura.yoshiaki@lab.ntt.co.jp
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

[-- Attachment #2: kemari_design080302.pdf --]
[-- Type: application/pdf, Size: 489037 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Design document of Kemari

[Keir Fraser]

On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp>
wrote:

> You can also get the undivided patches from the following links.
> http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
> http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch

The two obvious things I note are the duplicated save/restore functions,
which will simply bitrot, and the additions to evtchn.c. In particular the
hack to relax a validity check in evtchn_bind_interdomain() looks pretty
suspicious. Apart from that there are various spurious white-space and
debug-tracing changes which I assume you would clean up anyway. Those are
just the things that jump out from a quick read through.

 -- Keir

Re: Design document of Kemari

[Yoshiaki Tamura]

Keir,

Thanks for your quick review.
I'm glad my messages have finally reached to the list.

Although the issues you pointed out here aren't fixed yet, let me post the 
divided patches which are updated a little. We'll keep up cleaning and updating 
the code.

Yoshi

Keir Fraser wrote:
> On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp>
> wrote:
> 
>> You can also get the undivided patches from the following links.
>> http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
>> http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch
> 
> The two obvious things I note are the duplicated save/restore functions,
> which will simply bitrot, and the additions to evtchn.c. In particular the
> hack to relax a validity check in evtchn_bind_interdomain() looks pretty
> suspicious. Apart from that there are various spurious white-space and
> debug-tracing changes which I assume you would clean up anyway. Those are
> just the things that jump out from a quick read through.
> 
>  -- Keir
> 
> 
> 
> 
> 

Re: Design document of Kemari

[John Levon]

On Fri, Mar 06, 2009 at 09:52:21AM +0900, Yoshiaki Tamura wrote:

> Thanks for your quick review.
> I'm glad my messages have finally reached to the list.

I wonder how Kemari differs from Remus?

regards
john

Re: Design document of Kemari

[Yoshiaki Tamura]

John Levon wrote:
> On Fri, Mar 06, 2009 at 09:52:21AM +0900, Yoshiaki Tamura wrote:
> 
>> Thanks for your quick review.
>> I'm glad my messages have finally reached to the list.
> 
> I wonder how Kemari differs from Remus?
> 
> regards
> john
> 
> 
> 
I believe Kemari and Remus shares a lot in common.
Stephen and Ian have described both Kemari and Remus on the roadmap document, 
and that might help for quick review.

http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/

In short, Kemari transfers the guest VM on event-driven, while Remus transfers 
on specific interval.

Thanks,

Yoshi

Re: Design document of Kemari

[John Levon]

On Sat, Mar 07, 2009 at 01:48:22AM +0900, Yoshiaki Tamura wrote:

> I believe Kemari and Remus shares a lot in common.
> Stephen and Ian have described both Kemari and Remus on the roadmap 
> document, and that might help for quick review.
> 
> http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/
> 
> In short, Kemari transfers the guest VM on event-driven, while Remus 
> transfers on specific interval.

Thanks. Do you have performance numbers? It sounds like your approach is
intended to give lower I/O response latencies, but Remus will give
better throughput with high I/O load as it doesn't sync on every single
outgoing I/O.

If both projects are indeed headed for the tree, people are going to
need some ideas of which to use...

regards
 john

Re: Design document of Kemari

[Yoshiaki Tamura]

John Levon wrote:
> On Sat, Mar 07, 2009 at 01:48:22AM +0900, Yoshiaki Tamura wrote:
> 
>> I believe Kemari and Remus shares a lot in common.
>> Stephen and Ian have described both Kemari and Remus on the roadmap 
>> document, and that might help for quick review.
>>
>> http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/
>>
>> In short, Kemari transfers the guest VM on event-driven, while Remus 
>> transfers on specific interval.
> 
> Thanks. Do you have performance numbers? It sounds like your approach is
> intended to give lower I/O response latencies, but Remus will give
> better throughput with high I/O load as it doesn't sync on every single
> outgoing I/O.

Thanks for your interest.
Although the numbers are old, you can get an extended abstract and slides which 
I used at Xen Summit from http://www.osrg.net/kemari

The patchset I sent today seems to perform a lot better, but I don't have papers 
yet.

Regarding the throughput, possible idea is to buffer some events to reduce the 
frequency, which would introduce latencies as a trade off. Although we haven't 
implemented this feature yet, it may be interesting how the numbers change.

> If both projects are indeed headed for the tree, people are going to
> need some ideas of which to use...

Well, I sent the following message when I open-sourced Kemari in November.

http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html

Thanks,

Yoshi

Re: Design document of Kemari

[John Levon]

On Sat, Mar 07, 2009 at 02:26:06AM +0900, Yoshiaki Tamura wrote:

> Although the numbers are old, you can get an extended abstract and slides 
> which I used at Xen Summit from http://www.osrg.net/kemari

Thanks, that has some more info. Seems like there's a good opportunity for
someone to do some more detailed comparisons.

> >If both projects are indeed headed for the tree, people are going to
> >need some ideas of which to use...
> 
> Well, I sent the following message when I open-sourced Kemari in November.
> 
> http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html

"We're also planning to merge with Remus from UBC to propose a better
solution for Xen 3.4 together."

That sounds very interesting.

regards
john

Re: Design document of Kemari

[Brendan Cully]

On Friday, 06 March 2009 at 12:36, John Levon wrote:
> On Sat, Mar 07, 2009 at 02:26:06AM +0900, Yoshiaki Tamura wrote:
> 
> > Although the numbers are old, you can get an extended abstract and slides 
> > which I used at Xen Summit from http://www.osrg.net/kemari
> 
> Thanks, that has some more info. Seems like there's a good opportunity for
> someone to do some more detailed comparisons.
> 
> > >If both projects are indeed headed for the tree, people are going to
> > >need some ideas of which to use...
> > 
> > Well, I sent the following message when I open-sourced Kemari in November.
> > 
> > http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html
> 
> "We're also planning to merge with Remus from UBC to propose a better
> solution for Xen 3.4 together."
> 
> That sounds very interesting.

Yes, Yoshi and I had a brief discussion a few months ago about
this. Kemari has been released, and I plan to release Remus as soon as
possible. Unfortunately, I've had a very busy start to 2009 (related
to the Tralfamadore project Andy presented at the last Xen summit) and
haven't had the cycles to forward port the code to unstable and test
it. I intend to do that ASAP, and no later than the end of the month.

The good news is that I believe all of the Xen and Linux support
necessary for Remus has already been merged, so the code I'll be
releasing is just tools patches (I should double check whether that
also holds for HVM Remus, but I believe it does). Once the code is
released we can start looking at how to share code between Kemari and
Remus. I think there is a lot of common ground between the two
approaches -- the heart of Remus is fine-grained checkpoints, and I
think that's true for Kemari as well, though I haven't yet looked
closely at it.

By the way, there's another project going on here at UBC that
complements both Kemari and Remus very well. Patrick Colp is working
on copy-on-write VM checkpoints, which could be a huge win for
checkpoint-based availability. He also presented some of his work at
the last Xen summit:

http://www.getxen.org/xensummit/xensummit_winter_2009.html

Re: Design document of Kemari

[Yoshisato YANAGISAWA]

Hi Keir,

On 2009/03/06 3:58, Keir Fraser wrote:
> On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp>
> wrote:
> 
>> You can also get the undivided patches from the following links.
>> http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
>> http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch
> 
> The two obvious things I note are the duplicated save/restore functions,
> which will simply bitrot, and the additions to evtchn.c. In particular the

Since I have implemented the save/restore functions, will you let me ask 
some?

I could not understand the meaning of "the duplicated save/restore 
functions".  Do you mean there are some duplicated code inside 
xc_dom_kemari_save.c and xc_dom_kemari_restore.c?  Or, do you mean there 
are some duplicated code, e.g. test_bit and clear_bit and so on, between 
xc_dom_kemari_save and xc_domain_save, and between xc_dom_kemari_restore 
and xc_domain_restore?

> suspicious. Apart from that there are various spurious white-space and
> debug-tracing changes which I assume you would clean up anyway. Those are
> just the things that jump out from a quick read through.

I will clean up those spurious white-space.

Thank you in advance,

-- 
Yoshisato YANAGISAWA, Ph.D.
Open Source Software Computing Project
NTT Cyber Space Laboratories
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
1-1 Hikarinooka, Yokosuka-Shi,
Kanagawa 239-0847, Japan

Re: Design document of Kemari

[Keir Fraser]

On 09/03/2009 02:05, "Yoshisato YANAGISAWA"
<yanagisawa.yoshisato@lab.ntt.co.jp> wrote:

> I could not understand the meaning of "the duplicated save/restore
> functions".  Do you mean there are some duplicated code inside
> xc_dom_kemari_save.c and xc_dom_kemari_restore.c?

xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite
a bit in common.

It will be interesting to see Remus posted and compare patch sizes and
approaches.

 -- Keir

Re: Design document of Kemari

[Yoshiaki Tamura]

Keir Fraser wrote:
> On 09/03/2009 02:05, "Yoshisato YANAGISAWA"
> <yanagisawa.yoshisato@lab.ntt.co.jp> wrote:
> 
>> I could not understand the meaning of "the duplicated save/restore
>> functions".  Do you mean there are some duplicated code inside
>> xc_dom_kemari_save.c and xc_dom_kemari_restore.c?
> 
> xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite
> a bit in common.

Yes they do. We kept the original xc_domain_{save,restore} to make kemari 
specific part less intrusive. If you suggest to integrate them, we would 
certainly do so. Would that be appropriate?

Yoshi

Re: Design document of Kemari

[Keir Fraser]

On 09/03/2009 22:28, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp>
wrote:

>> xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite
>> a bit in common.
> 
> Yes they do. We kept the original xc_domain_{save,restore} to make kemari
> specific part less intrusive. If you suggest to integrate them, we would
> certainly do so. Would that be appropriate?

Yes, we prefer a merged save/restore codebase.

 -- Keir

Re: Design document of Kemari

[Yoshisato YANAGISAWA]

On 2009/03/10 17:21, Keir Fraser wrote:
> On 09/03/2009 22:28, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp>
> wrote:
> 
>>> xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite
>>> a bit in common.
>> Yes they do. We kept the original xc_domain_{save,restore} to make kemari
>> specific part less intrusive. If you suggest to integrate them, we would
>> certainly do so. Would that be appropriate?
> 
> Yes, we prefer a merged save/restore codebase.

Since Kemari requires transaction-like feature, xc_dom_kemari_save 
should wait acknowledgment from xc_dom_kemari_restore. 
Xc_dom_kemari_restore should also buffer data until complete snapshot 
received from xc_dom_kemari_save.
Because of this reason, there might be large if-else statements if those 
code are merged.

Isn't it make xc_domain_save and xc_domain_restore too complexed?

-- 
Yoshisato YANAGISAWA, Ph.D.
Open Source Software Computing Project
NTT Cyber Space Laboratories
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
1-1 Hikarinooka, Yokosuka-Shi,
Kanagawa 239-0847, Japan

Re: Design document of Kemari

[Keir Fraser]

On 10/03/2009 10:16, "Yoshisato YANAGISAWA"
<yanagisawa.yoshisato@lab.ntt.co.jp> wrote:

>> Yes, we prefer a merged save/restore codebase.
> 
> Since Kemari requires transaction-like feature, xc_dom_kemari_save
> should wait acknowledgment from xc_dom_kemari_restore.
> Xc_dom_kemari_restore should also buffer data until complete snapshot
> received from xc_dom_kemari_save.
> Because of this reason, there might be large if-else statements if those
> code are merged.
> 
> Isn't it make xc_domain_save and xc_domain_restore too complexed?

Well, you can leave it for now pending further investigation. I also want to
see what the Remus approach is for this.

 -- Keir

Re: Design document of Kemari

[Brendan Cully]

On Tuesday, 10 March 2009 at 10:38, Keir Fraser wrote:
> On 10/03/2009 10:16, "Yoshisato YANAGISAWA"
> <yanagisawa.yoshisato@lab.ntt.co.jp> wrote:
> 
> >> Yes, we prefer a merged save/restore codebase.
> > 
> > Since Kemari requires transaction-like feature, xc_dom_kemari_save
> > should wait acknowledgment from xc_dom_kemari_restore.
> > Xc_dom_kemari_restore should also buffer data until complete snapshot
> > received from xc_dom_kemari_save.
> > Because of this reason, there might be large if-else statements if those
> > code are merged.
> > 
> > Isn't it make xc_domain_save and xc_domain_restore too complexed?
> 
> Well, you can leave it for now pending further investigation. I also want to
> see what the Remus approach is for this.

The main change to xc_domain_save is to split out the code that does a
round into its own function so that it can be called externally. I'll
have the patches together soon.