From mboxrd@z Thu Jan  1 00:00:00 1970
From: Stefan Hajnoczi <stefanha@gmail.com>
Subject: Re: [Qemu-devel] [PATCH 00/21][RFC] postcopy live migration
Date: Sun, 1 Jan 2012 16:27:56 +0000
Message-ID: <CAJSP0QWaZxp6ypWay6LRmp9TtvD2gwueb1RrfW08NWpHy1=2HQ@mail.gmail.com>
References: <cover.1325055139.git.yamahata@valinux.co.jp>
	<4EFCEC38.3080308@codemonkey.ws>
	<4F002AC6.7080007@redhat.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: QUOTED-PRINTABLE
Cc: Anthony Liguori <anthony@codemonkey.ws>, kvm@vger.kernel.org,
	satoshi.itoh@aist.go.jp, t.hirofuchi@aist.go.jp,
	Juan Quintela <quintela@redhat.com>,
	Michael Roth <mdroth@linux.vnet.ibm.com>,
	qemu-devel@nongnu.org, Isaku Yamahata <yamahata@valinux.co.jp>
To: Orit Wasserman <owasserm@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mail-we0-f174.google.com ([74.125.82.174]:51802 "EHLO
	mail-we0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751925Ab2AAQ2A convert rfc822-to-8bit (ORCPT
	<rfc822;kvm@vger.kernel.org>); Sun, 1 Jan 2012 11:28:00 -0500
Received: by werm1 with SMTP id m1so7162922wer.19
        for <kvm@vger.kernel.org>; Sun, 01 Jan 2012 08:27:59 -0800 (PST)
In-Reply-To: <4F002AC6.7080007@redhat.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

On Sun, Jan 1, 2012 at 9:43 AM, Orit Wasserman <owasserm@redhat.com> wr=
ote:
> On 12/30/2011 12:39 AM, Anthony Liguori wrote:
>> On 12/28/2011 07:25 PM, Isaku Yamahata wrote:
>>> Intro
>>> =3D=3D=3D=3D=3D
>>> This patch series implements postcopy live migration.[1]
>>> As discussed at KVM forum 2011, dedicated character device is used =
for
>>> distributed shared memory between migration source and destination.
>>> Now we can discuss/benchmark/compare with precopy. I believe there =
are
>>> much rooms for improvement.
>>>
>>> [1] http://wiki.qemu.org/Features/PostCopyLiveMigration
>>>
>>>
>>> Usage
>>> =3D=3D=3D=3D=3D
>>> You need load umem character device on the host before starting mig=
ration.
>>> Postcopy can be used for tcg and kvm accelarator. The implementatio=
n depend
>>> on only linux umem character device. But the driver dependent code =
is split
>>> into a file.
>>> I tested only host page size =3D=3D guest page size case, but the i=
mplementation
>>> allows host page size !=3D guest page size case.
>>>
>>> The following options are added with this patch series.
>>> - incoming part
>>> =A0 =A0command line options
>>> =A0 =A0-postcopy [-postcopy-flags<flags>]
>>> =A0 =A0where flags is for changing behavior for benchmark/debugging
>>> =A0 =A0Currently the following flags are available
>>> =A0 =A00: default
>>> =A0 =A01: enable touching page request
>>>
>>> =A0 =A0example:
>>> =A0 =A0qemu -postcopy -incoming tcp:0:4444 -monitor stdio -machine =
accel=3Dkvm
>>>
>>> - outging part
>>> =A0 =A0options for migrate command
>>> =A0 =A0migrate [-p [-n]] URI
>>> =A0 =A0-p: indicate postcopy migration
>>> =A0 =A0-n: disable background transferring pages: This is for bench=
mark/debugging
>>>
>>> =A0 =A0example:
>>> =A0 =A0migrate -p -n tcp:<dest ip address>:4444
>>>
>>>
>>> TODO
>>> =3D=3D=3D=3D
>>> - benchmark/evaluation. Especially how async page fault affects the=
 result.
>>
>> I'll review this series next week (Mike/Juan, please also review whe=
n you can).
>>
>> But we really need to think hard about whether this is the right thi=
ng to take into the tree. =A0I worry a lot about the fact that we don't=
 test pre-copy migration nearly enough and adding a second form just in=
troduces more things to test.
>>
>> It's also not clear to me why post-copy is better. =A0If you were go=
ing to sit down and explain to someone building a management tool when =
they should use pre-copy and when they should use post-copy, what would=
 you tell them?
>
> Start with pre-copy , if it doesn't converge switch to post-copy

Post-copy throttles the guest when page faults are encountered because
the destination machine waits for memory pages from the source
machine.  Is there a reason this page fault-based throttling cannot be
done on the source machine with pre-copy migration?  I'm not sure
post-copy provides new behavior in terms of convergence, we could do
the same with pre-copy migration.

Post-copy has other advantages though, it immediately frees logical
CPUs on the source machine (though RAM and network bandwidth is still
required until migration completes).

Stefan

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([140.186.70.92]:42542)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1RhOGQ-0003KA-DE
	for qemu-devel@nongnu.org; Sun, 01 Jan 2012 11:28:03 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1RhOGO-0005QD-I9
	for qemu-devel@nongnu.org; Sun, 01 Jan 2012 11:28:02 -0500
Received: from mail-we0-f173.google.com ([74.125.82.173]:37747)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1RhOGO-0005Q6-Dn
	for qemu-devel@nongnu.org; Sun, 01 Jan 2012 11:28:00 -0500
Received: by werb10 with SMTP id b10so8512063wer.4
	for <qemu-devel@nongnu.org>; Sun, 01 Jan 2012 08:27:59 -0800 (PST)
MIME-Version: 1.0
In-Reply-To: <4F002AC6.7080007@redhat.com>
References: <cover.1325055139.git.yamahata@valinux.co.jp>
	<4EFCEC38.3080308@codemonkey.ws> <4F002AC6.7080007@redhat.com>
Date: Sun, 1 Jan 2012 16:27:56 +0000
Message-ID: <CAJSP0QWaZxp6ypWay6LRmp9TtvD2gwueb1RrfW08NWpHy1=2HQ@mail.gmail.com>
From: Stefan Hajnoczi <stefanha@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [Qemu-devel] [PATCH 00/21][RFC] postcopy live migration
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Orit Wasserman <owasserm@redhat.com>
Cc: kvm@vger.kernel.org, Juan Quintela <quintela@redhat.com>, t.hirofuchi@aist.go.jp, satoshi.itoh@aist.go.jp, Michael Roth <mdroth@linux.vnet.ibm.com>, qemu-devel@nongnu.org, Isaku Yamahata <yamahata@valinux.co.jp>

On Sun, Jan 1, 2012 at 9:43 AM, Orit Wasserman <owasserm@redhat.com> wrote:
> On 12/30/2011 12:39 AM, Anthony Liguori wrote:
>> On 12/28/2011 07:25 PM, Isaku Yamahata wrote:
>>> Intro
>>> =3D=3D=3D=3D=3D
>>> This patch series implements postcopy live migration.[1]
>>> As discussed at KVM forum 2011, dedicated character device is used for
>>> distributed shared memory between migration source and destination.
>>> Now we can discuss/benchmark/compare with precopy. I believe there are
>>> much rooms for improvement.
>>>
>>> [1] http://wiki.qemu.org/Features/PostCopyLiveMigration
>>>
>>>
>>> Usage
>>> =3D=3D=3D=3D=3D
>>> You need load umem character device on the host before starting migrati=
on.
>>> Postcopy can be used for tcg and kvm accelarator. The implementation de=
pend
>>> on only linux umem character device. But the driver dependent code is s=
plit
>>> into a file.
>>> I tested only host page size =3D=3D guest page size case, but the imple=
mentation
>>> allows host page size !=3D guest page size case.
>>>
>>> The following options are added with this patch series.
>>> - incoming part
>>> =A0 =A0command line options
>>> =A0 =A0-postcopy [-postcopy-flags<flags>]
>>> =A0 =A0where flags is for changing behavior for benchmark/debugging
>>> =A0 =A0Currently the following flags are available
>>> =A0 =A00: default
>>> =A0 =A01: enable touching page request
>>>
>>> =A0 =A0example:
>>> =A0 =A0qemu -postcopy -incoming tcp:0:4444 -monitor stdio -machine acce=
l=3Dkvm
>>>
>>> - outging part
>>> =A0 =A0options for migrate command
>>> =A0 =A0migrate [-p [-n]] URI
>>> =A0 =A0-p: indicate postcopy migration
>>> =A0 =A0-n: disable background transferring pages: This is for benchmark=
/debugging
>>>
>>> =A0 =A0example:
>>> =A0 =A0migrate -p -n tcp:<dest ip address>:4444
>>>
>>>
>>> TODO
>>> =3D=3D=3D=3D
>>> - benchmark/evaluation. Especially how async page fault affects the res=
ult.
>>
>> I'll review this series next week (Mike/Juan, please also review when yo=
u can).
>>
>> But we really need to think hard about whether this is the right thing t=
o take into the tree. =A0I worry a lot about the fact that we don't test pr=
e-copy migration nearly enough and adding a second form just introduces mor=
e things to test.
>>
>> It's also not clear to me why post-copy is better. =A0If you were going =
to sit down and explain to someone building a management tool when they sho=
uld use pre-copy and when they should use post-copy, what would you tell th=
em?
>
> Start with pre-copy , if it doesn't converge switch to post-copy

Post-copy throttles the guest when page faults are encountered because
the destination machine waits for memory pages from the source
machine.  Is there a reason this page fault-based throttling cannot be
done on the source machine with pre-copy migration?  I'm not sure
post-copy provides new behavior in terms of convergence, we could do
the same with pre-copy migration.

Post-copy has other advantages though, it immediately frees logical
CPUs on the source machine (though RAM and network bandwidth is still
required until migration completes).

Stefan