[Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[lirans]

This series adds support for live migration without shared storage, means 
copy the storage while migrating. It was tested with KVM. Supports 2 ways 
to replicate the storage during migration:
1. Complete copy of storage to destination 
2. Assuming the storage is cow based, copy only the allocated 
data, time of the migration will be linear with the amount of allocated 
data (user responsibility to verify that the same backend file reside 
on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

Changes from v4:
- Register block dirty tracking for each block device.
- Block.c will now manage a dirty bitmap per device once 
bdrv_set_dirty_tracking() is called. Bitmap is polled by the upper 
layer (block-migration.c).
- Global variabels moved to a global state structure allocated dynamically.
- Coding style issues.

Patch summary:

Makefile          |    1 +
block-migration.c |  558 +++++++++++++++++++++++++++++++++++++++++++++++++++++
block-migration.h |   29 +++
block.c           |   78 ++++++++-
block.h           |    8 +-
block_int.h       |    2 +
buffered_file.c   |   10 +-
hw/hw.h           |    9 +-
migration-exec.c  |    9 +-
migration-fd.c    |    7 +-
migration-tcp.c   |    7 +-
migration-unix.c  |    7 +-
migration.c       |   23 ++-
migration.h       |   20 ++-
monitor.c         |   36 +++-
qemu-monitor.hx   |   19 ++-
savevm.c          |   62 +++++--
sysemu.h          |    2 +-
vl.c              |    9 +-
19 files changed, 841 insertions(+), 55 deletions(-)

Signed-off-by: Liran Schour <lirans@il.ibm.com>

Re: [Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[Avi Kivity]

On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
> This series adds support for live migration without shared storage, means
> copy the storage while migrating. It was tested with KVM. Supports 2 ways
> to replicate the storage during migration:
> 1. Complete copy of storage to destination
> 2. Assuming the storage is cow based, copy only the allocated
> data, time of the migration will be linear with the amount of allocated
> data (user responsibility to verify that the same backend file reside
> on src and destination).
>
> Live migration will work as follows:
> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
> (qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.
>    

I'd like to see the syntax generalized.  For one, the guest may have 
several disks; an install image cdrom might be available as an nfs image 
but the main storage is local.  Secondly, there can be several levels of 
cow and we want to control which one we copy.

I'll leave the exact details to the qpeople, but if we can specify a 
copy depth for each device, where 0=copy nothing, n=copy everything, 
1=copy the last level (equivalent to -d blk inc) I think we'll have 
covered everything.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[Liran Schour]

qemu-devel-bounces+lirans=il.ibm.com@nongnu.org wrote on 02/11/2009
15:40:25:
> Live migration will work as follows:
> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
> (qemu) migrate -d blk tcp:0:4444 # for live migration with complete
> storage copy
> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
> incremental storage copy, storage is cow based.
>
The up-to-date migrate command will work as follow:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d -b tcp:0:4444 # for live migration with complete storage
copy
(qemu) migrate -d -i tcp:0:4444 # for live migration with incremental
storage copy,
 storage is cow based.

- Liran

Re: [Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[Liran Schour]

- Liran

Avi Kivity <avi@redhat.com> wrote on 02/11/2009 20:47:34:
> On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
> > This series adds support for live migration without shared storage,
means
> > copy the storage while migrating. It was tested with KVM. Supports 2
ways
> > to replicate the storage during migration:
> > 1. Complete copy of storage to destination
> > 2. Assuming the storage is cow based, copy only the allocated
> > data, time of the migration will be linear with the amount of allocated
> > data (user responsibility to verify that the same backend file reside
> > on src and destination).
> >
> > Live migration will work as follows:
> > (qemu) migrate -d tcp:0:4444 # for ordinary live migration
> > (qemu) migrate -d blk tcp:0:4444 # for live migration with
> complete storage copy
> > (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
> incremental storage copy, storage is cow based.
> >
>
> I'd like to see the syntax generalized.  For one, the guest may have
> several disks; an install image cdrom might be available as an nfs image
> but the main storage is local.  Secondly, there can be several levels of
> cow and we want to control which one we copy.
>
> I'll leave the exact details to the qpeople, but if we can specify a
> copy depth for each device, where 0=copy nothing, n=copy everything,
> 1=copy the last level (equivalent to -d blk inc) I think we'll have
> covered everything.

I propose the following syntax:
migrate [-d] [-b [<device:copy_level> ...<device:copy_level>]]
tcp:<host>:<port>
Where device will be the name of the device: ide0-hd0. And copy_level will
be integer or n, 0=flat copy of the device, 1=copy the last level ...
n=copy all levels.
All HD devices that will not appear in the command will be treated as
copy_level=0. All devices != BDRV_TYPE_HD will be ignored.

What do you think?

- Liran

Re: [Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[Avi Kivity]

On 11/03/2009 11:40 AM, Liran Schour wrote:
> - Liran
>
> Avi Kivity<avi@redhat.com>  wrote on 02/11/2009 20:47:34:
>    
>> On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
>>      
>>> This series adds support for live migration without shared storage,
>>>        
> means
>    
>>> copy the storage while migrating. It was tested with KVM. Supports 2
>>>        
> ways
>    
>>> to replicate the storage during migration:
>>> 1. Complete copy of storage to destination
>>> 2. Assuming the storage is cow based, copy only the allocated
>>> data, time of the migration will be linear with the amount of allocated
>>> data (user responsibility to verify that the same backend file reside
>>> on src and destination).
>>>
>>> Live migration will work as follows:
>>> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
>>> (qemu) migrate -d blk tcp:0:4444 # for live migration with
>>>        
>> complete storage copy
>>      
>>> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
>>>        
>> incremental storage copy, storage is cow based.
>>      
>>>        
>> I'd like to see the syntax generalized.  For one, the guest may have
>> several disks; an install image cdrom might be available as an nfs image
>> but the main storage is local.  Secondly, there can be several levels of
>> cow and we want to control which one we copy.
>>
>> I'll leave the exact details to the qpeople, but if we can specify a
>> copy depth for each device, where 0=copy nothing, n=copy everything,
>> 1=copy the last level (equivalent to -d blk inc) I think we'll have
>> covered everything.
>>      
> I propose the following syntax:
> migrate [-d] [-b [<device:copy_level>  ...<device:copy_level>]]
> tcp:<host>:<port>
> Where device will be the name of the device: ide0-hd0. And copy_level will
> be integer or n, 0=flat copy of the device, 1=copy the last level ...
> n=copy all levels.
> All HD devices that will not appear in the command will be treated as
> copy_level=0. All devices != BDRV_TYPE_HD will be ignored.
>
>    

Looks good.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: [PATCH 0/3 v5] Live migration without shared storage

[Jan Kiszka]

Avi Kivity wrote:
> On 11/03/2009 11:40 AM, Liran Schour wrote:
>> - Liran
>>
>> Avi Kivity<avi@redhat.com>  wrote on 02/11/2009 20:47:34:
>>   
>>> On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
>>>     
>>>> This series adds support for live migration without shared storage,
>>>>        
>> means
>>   
>>>> copy the storage while migrating. It was tested with KVM. Supports 2
>>>>        
>> ways
>>   
>>>> to replicate the storage during migration:
>>>> 1. Complete copy of storage to destination
>>>> 2. Assuming the storage is cow based, copy only the allocated
>>>> data, time of the migration will be linear with the amount of allocated
>>>> data (user responsibility to verify that the same backend file reside
>>>> on src and destination).
>>>>
>>>> Live migration will work as follows:
>>>> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
>>>> (qemu) migrate -d blk tcp:0:4444 # for live migration with
>>>>        
>>> complete storage copy
>>>     
>>>> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
>>>>        
>>> incremental storage copy, storage is cow based.
>>>     
>>>>        
>>> I'd like to see the syntax generalized.  For one, the guest may have
>>> several disks; an install image cdrom might be available as an nfs image
>>> but the main storage is local.  Secondly, there can be several levels of
>>> cow and we want to control which one we copy.
>>>
>>> I'll leave the exact details to the qpeople, but if we can specify a
>>> copy depth for each device, where 0=copy nothing, n=copy everything,
>>> 1=copy the last level (equivalent to -d blk inc) I think we'll have
>>> covered everything.
>>>      
>> I propose the following syntax:
>> migrate [-d] [-b [<device:copy_level>  ...<device:copy_level>]]
>> tcp:<host>:<port>
>> Where device will be the name of the device: ide0-hd0. And copy_level
>> will
>> be integer or n, 0=flat copy of the device, 1=copy the last level ...
>> n=copy all levels.
>> All HD devices that will not appear in the command will be treated as
>> copy_level=0. All devices != BDRV_TYPE_HD will be ignored.
>>
>>    
> 
> Looks good.
> 

Do I get this correctly: this series is now "only" blocked by the
required rework of its user interface? Is there a schedule for this, a
v6 roll out? Will it be in time for qemu-0.12?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: [PATCH 0/3 v5] Live migration without shared storage

[Pierre Riteau]

On 23 nov. 2009, at 19:47, Jan Kiszka wrote:

> Avi Kivity wrote:
>> On 11/03/2009 11:40 AM, Liran Schour wrote:
>>> - Liran
>>> 
>>> Avi Kivity<avi@redhat.com>  wrote on 02/11/2009 20:47:34:
>>> 
>>>> On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
>>>> 
>>>>> This series adds support for live migration without shared storage,
>>>>> 
>>> means
>>> 
>>>>> copy the storage while migrating. It was tested with KVM. Supports 2
>>>>> 
>>> ways
>>> 
>>>>> to replicate the storage during migration:
>>>>> 1. Complete copy of storage to destination
>>>>> 2. Assuming the storage is cow based, copy only the allocated
>>>>> data, time of the migration will be linear with the amount of allocated
>>>>> data (user responsibility to verify that the same backend file reside
>>>>> on src and destination).
>>>>> 
>>>>> Live migration will work as follows:
>>>>> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
>>>>> (qemu) migrate -d blk tcp:0:4444 # for live migration with
>>>>> 
>>>> complete storage copy
>>>> 
>>>>> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
>>>>> 
>>>> incremental storage copy, storage is cow based.
>>>> 
>>>>> 
>>>> I'd like to see the syntax generalized.  For one, the guest may have
>>>> several disks; an install image cdrom might be available as an nfs image
>>>> but the main storage is local.  Secondly, there can be several levels of
>>>> cow and we want to control which one we copy.
>>>> 
>>>> I'll leave the exact details to the qpeople, but if we can specify a
>>>> copy depth for each device, where 0=copy nothing, n=copy everything,
>>>> 1=copy the last level (equivalent to -d blk inc) I think we'll have
>>>> covered everything.
>>>> 
>>> I propose the following syntax:
>>> migrate [-d] [-b [<device:copy_level>  ...<device:copy_level>]]
>>> tcp:<host>:<port>
>>> Where device will be the name of the device: ide0-hd0. And copy_level
>>> will
>>> be integer or n, 0=flat copy of the device, 1=copy the last level ...
>>> n=copy all levels.
>>> All HD devices that will not appear in the command will be treated as
>>> copy_level=0. All devices != BDRV_TYPE_HD will be ignored.
>>> 
>>> 
>> 
>> Looks good.
>> 
> 
> Do I get this correctly: this series is now "only" blocked by the
> required rework of its user interface? Is there a schedule for this, a
> v6 roll out? Will it be in time for qemu-0.12?
> 
> Jan

It was committed to the master branch a week ago.

-- 
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/

Re: [Qemu-devel] Re: [PATCH 0/3 v5] Live migration without shared storage

[Jan Kiszka]

Pierre Riteau wrote:
> On 23 nov. 2009, at 19:47, Jan Kiszka wrote:
> 
>> Avi Kivity wrote:
>>> On 11/03/2009 11:40 AM, Liran Schour wrote:
>>>> - Liran
>>>>
>>>> Avi Kivity<avi@redhat.com>  wrote on 02/11/2009 20:47:34:
>>>>
>>>>> On 11/02/2009 03:40 PM, lirans@il.ibm.com wrote:
>>>>>
>>>>>> This series adds support for live migration without shared storage,
>>>>>>
>>>> means
>>>>
>>>>>> copy the storage while migrating. It was tested with KVM. Supports 2
>>>>>>
>>>> ways
>>>>
>>>>>> to replicate the storage during migration:
>>>>>> 1. Complete copy of storage to destination
>>>>>> 2. Assuming the storage is cow based, copy only the allocated
>>>>>> data, time of the migration will be linear with the amount of allocated
>>>>>> data (user responsibility to verify that the same backend file reside
>>>>>> on src and destination).
>>>>>>
>>>>>> Live migration will work as follows:
>>>>>> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
>>>>>> (qemu) migrate -d blk tcp:0:4444 # for live migration with
>>>>>>
>>>>> complete storage copy
>>>>>
>>>>>> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
>>>>>>
>>>>> incremental storage copy, storage is cow based.
>>>>>
>>>>> I'd like to see the syntax generalized.  For one, the guest may have
>>>>> several disks; an install image cdrom might be available as an nfs image
>>>>> but the main storage is local.  Secondly, there can be several levels of
>>>>> cow and we want to control which one we copy.
>>>>>
>>>>> I'll leave the exact details to the qpeople, but if we can specify a
>>>>> copy depth for each device, where 0=copy nothing, n=copy everything,
>>>>> 1=copy the last level (equivalent to -d blk inc) I think we'll have
>>>>> covered everything.
>>>>>
>>>> I propose the following syntax:
>>>> migrate [-d] [-b [<device:copy_level>  ...<device:copy_level>]]
>>>> tcp:<host>:<port>
>>>> Where device will be the name of the device: ide0-hd0. And copy_level
>>>> will
>>>> be integer or n, 0=flat copy of the device, 1=copy the last level ...
>>>> n=copy all levels.
>>>> All HD devices that will not appear in the command will be treated as
>>>> copy_level=0. All devices != BDRV_TYPE_HD will be ignored.
>>>>
>>>>
>>> Looks good.
>>>
>> Do I get this correctly: this series is now "only" blocked by the
>> required rework of its user interface? Is there a schedule for this, a
>> v6 roll out? Will it be in time for qemu-0.12?
>>
>> Jan
> 
> It was committed to the master branch a week ago.

Oh, indeed, thanks. Due to the fact that this discussion suggested that
there are still open issues, I did not even checked git. And the missing
merge notifications contributed to this as well.

Anyway, good to see this upstream. /me goes testing.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: [PATCH 0/3 v5] Live migration without shared storage

[Anthony Liguori]

Jan Kiszka wrote:
> Oh, indeed, thanks. Due to the fact that this discussion suggested that
> there are still open issues, I did not even checked git.

They still need to be addressed.  However, I wanted to do that work in 
the tree vs. outside of the tree.  With all the qmp stuff happening, 
that seemed like the rationale thing to do.


-- 
Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [PATCH 0/3 v5] Live migration without shared storage

[Liran Schour]

"Anthony Liguori" <aliguori@linux.vnet.ibm.com> wrote on 24/11/2009
16:27:58:
> Jan Kiszka wrote:
> > Oh, indeed, thanks. Due to the fact that this discussion suggested that
> > there are still open issues, I did not even checked git.
>
> They still need to be addressed.  However, I wanted to do that work in
> the tree vs. outside of the tree.  With all the qmp stuff happening,
> that seemed like the rationale thing to do.
>

What do you think about the following syntax:
migrate [-d] URI
[block:default_copy_level:[<device:copy_level>,.....,<device:copy_level>]
copy_level: 0 - flat copy.
            1 - copy only first COW level.
            ..
            n - copy all COW levels.
default_copy_level: use this copy level for all devices that are not
mentioned in the migrate command.

I intend to implement it this week, I am planning to implement on the first
phase only copy levels 0 and 1.
It suppose to solve the major problem today that a guest has one block
device that needs to be copied flat and the other block device that needs
to be COW copy.

- Liran

Re: [Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

[jz4]

[Qemu-devel] [PATCH 0/3 v5] Live migration without shared storage

I tested the live migration without shared storage for the patch of  
the block migration. It succeeds for several times but sometimes it  
fails. I used the qemu-kvm-0.12.3 and the kvm 2.6.33 module on the  
ubuntu latest version.

In the failure case, i can see the shell on the destination site  
showed the correct content for the migrated VM, but it just hanged  
there without any response to the mouse and the keyboard. I can see it  
prints "migration  completes" twice. Is it a bug or did I do anything  
wrong?

Another question is about the command line for the destination site.  
For memory
migration, i will start the vm based on the shared image. For block  
migration, there is no shared image. What i did is creating an empty  
image with the same format (e.g. raw or qcow2) as the source migrated  
VM. Is it correct or it doesn't matter which format it is in the  
destination site?

For the memory migration, it emphasizes that the path for the VM  
should be the same as the source path. But for the block migration, i  
change the destination path to a different one and it also works. Does  
that mean path is not important now?

The command line must be "migrate -d -b ..". If it is "-b -d", the  
qemu-monitor doesn't understand. Does that intend to be designed like  
this or will it be modified in the future?

Thanks!

-Jie