[Qemu-devel] Tunneled Migration with Non-Shared Storage

[Qemu-devel] Tunneled Migration with Non-Shared Storage

[Gary R Hook]

What I really need to figure out is why, when performing a migration
using non-shared storage, an entire VM is copied into memory before it’s 
sent across the wire using this method. A copy-on-read operation
is performed first, then the disk is sent, then the dirty pages, then
the RAM.

The odd thing is that non-tunneled migrations use a completely different
code path that does _not_ execute a copy-on-read first.

The problem: VMs (VM disks) that are larger than available memory can't
be migrated due to space constraints.

I'd like to disable this COR opration but I think it's important to
understand why the code is written this way. I've yet to run across any
comments that explain the need for this "extra" copy, and am looking
for background and advice.

Any insights are welcome.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Dr. David Alan Gilbert]

* Gary R Hook (grhookatwork@gmail.com) wrote:
> What I really need to figure out is why, when performing a migration
> using non-shared storage, an entire VM is copied into memory before it???s
> sent across the wire using this method. A copy-on-read operation
> is performed first, then the disk is sent, then the dirty pages, then
> the RAM.
> 
> The odd thing is that non-tunneled migrations use a completely different
> code path that does _not_ execute a copy-on-read first.

This seems odd, since as far as I know the tunneling code is quite separate
to the migration code; I thought the only thing that the migration
code sees different is the file descriptors it gets past.
(Having said that, again I don't know storage stuff, so if this
is a storage special there may be something there...)

Do you have the libvirt commands you're using and ideally the logs
to show what qmp commands are being sent to qemu to see what's
actually doing the COR ?

> The problem: VMs (VM disks) that are larger than available memory can't
> be migrated due to space constraints.
> 
> I'd like to disable this COR opration but I think it's important to
> understand why the code is written this way. I've yet to run across any
> comments that explain the need for this "extra" copy, and am looking
> for background and advice.
> 
> Any insights are welcome.

Dave

> 
> -- 
> Gary R Hook
> Senior Kernel Engineer
> NIMBOXX, Inc
> 
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Paolo Bonzini]

On 18/11/2014 21:28, Dr. David Alan Gilbert wrote:
> This seems odd, since as far as I know the tunneling code is quite separate
> to the migration code; I thought the only thing that the migration
> code sees different is the file descriptors it gets past.
> (Having said that, again I don't know storage stuff, so if this
> is a storage special there may be something there...)

Tunnelled migration uses the old block-migration.c code.  Non-tunnelled
migration uses the NBD server and block/mirror.c.  The main problem with
the old code is that uses a possibly unbounded amount of memory in
mig_save_device_dirty and can have huge jitter if any serious workload
is running in the guest.

Paolo

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> 
> 
> On 18/11/2014 21:28, Dr. David Alan Gilbert wrote:
> > This seems odd, since as far as I know the tunneling code is quite separate
> > to the migration code; I thought the only thing that the migration
> > code sees different is the file descriptors it gets past.
> > (Having said that, again I don't know storage stuff, so if this
> > is a storage special there may be something there...)
> 
> Tunnelled migration uses the old block-migration.c code.  Non-tunnelled
> migration uses the NBD server and block/mirror.c. 

OK, that explains that.  Is that because the tunneling code can't 
deal with tunneling the NBD server connection?

> The main problem with
> the old code is that uses a possibly unbounded amount of memory in
> mig_save_device_dirty and can have huge jitter if any serious workload
> is running in the guest.

So that's sending dirty blocks iteratively? Not that I can see
when the allocations get freed; but is the amount allocated there
related to total disk size (as Gary suggested) or to the amount
of dirty blocks?

Dave

> 
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Paolo Bonzini]

On 19/11/2014 10:35, Dr. David Alan Gilbert wrote:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>
>>
>> On 18/11/2014 21:28, Dr. David Alan Gilbert wrote:
>>> This seems odd, since as far as I know the tunneling code is quite separate
>>> to the migration code; I thought the only thing that the migration
>>> code sees different is the file descriptors it gets past.
>>> (Having said that, again I don't know storage stuff, so if this
>>> is a storage special there may be something there...)
>>
>> Tunnelled migration uses the old block-migration.c code.  Non-tunnelled
>> migration uses the NBD server and block/mirror.c. 
> 
> OK, that explains that.  Is that because the tunneling code can't 
> deal with tunneling the NBD server connection?
> 
>> The main problem with
>> the old code is that uses a possibly unbounded amount of memory in
>> mig_save_device_dirty and can have huge jitter if any serious workload
>> is running in the guest.
> 
> So that's sending dirty blocks iteratively? Not that I can see
> when the allocations get freed; but is the amount allocated there
> related to total disk size (as Gary suggested) or to the amount
> of dirty blocks?

It should be related to the maximum rate limit (which can be set to
arbitrarily high values, however).

The reads are started, then the ones that are ready are sent and the
blocks are freed in flush_blks.  The jitter happens when the guest reads
a lot but only writes a few blocks.  In that case, the bdrv_drain_all in
mig_save_device_dirty can be called relatively often and it can be
expensive because it also waits for all guest-initiated reads to complete.

The bulk phase is similar, just with different functions (the reads are
done in mig_save_device_bulk).  With a high rate limit, the total
allocated memory can reach a few gigabytes indeed.

Depending on the scenario, a possible disadvantage of NBD migration is
that it can only throttle each disk separately, while the old code will
apply a single limit to all migrations.

Paolou

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Daniel P. Berrange]

On Wed, Nov 19, 2014 at 09:35:16AM +0000, Dr. David Alan Gilbert wrote:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> > 
> > 
> > On 18/11/2014 21:28, Dr. David Alan Gilbert wrote:
> > > This seems odd, since as far as I know the tunneling code is quite separate
> > > to the migration code; I thought the only thing that the migration
> > > code sees different is the file descriptors it gets past.
> > > (Having said that, again I don't know storage stuff, so if this
> > > is a storage special there may be something there...)
> > 
> > Tunnelled migration uses the old block-migration.c code.  Non-tunnelled
> > migration uses the NBD server and block/mirror.c. 
> 
> OK, that explains that.  Is that because the tunneling code can't 
> deal with tunneling the NBD server connection?

Yep, pretty much. The tunnelling code was built around the idea of a
single connection to be tunnelled.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] Tunneled Migration with Non-Shared Storage

[Gary R Hook]

On 11/18/14 2:28 PM, Dr. David Alan Gilbert wrote:
> Do you have the libvirt commands you're using and ideally the logs
> to show what qmp commands are being sent to qemu to see what's
> actually doing the COR ?

 From the libvirt log, here are all of the commands sent to move a small 
VM. The debug statement comes from qemuMonitorJSONCommandWithFd(), line 
289 in my code. I also have the XML that's sent to the destination, but 
not sure if that's helpful...

The formatting is unfortunate :-(


2014-11-19 18:52:32.241+0000 Send command 
'{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1
2014-11-19 18:52:32.243+0000 Send command 
'{"execute":"query-version","id":"libvirt-2"}' for write with FD -1
2014-11-19 18:52:32.246+0000 Send command 
'{"execute":"query-target","id":"libvirt-3"}' for write with FD -1
2014-11-19 18:52:32.248+0000 Send command 
'{"execute":"query-commands","id":"libvirt-4"}' for write with FD -1
2014-11-19 18:52:32.258+0000 Send command 
'{"execute":"add-fd","arguments":{"fdset-id":0,"opaque":"/dev/null"},"id":"libvirt-5"}' 
for write with FD 26
2014-11-19 18:52:32.260+0000 Send command 
'{"execute":"block-commit","arguments":{"device":"bogus","speed":0},"id":"libvirt-6"}' 
for write with FD -1
2014-11-19 18:52:32.263+0000 Send command 
'{"execute":"query-events","id":"libvirt-7"}' for write with FD -1
2014-11-19 18:52:32.268+0000 Send command 
'{"execute":"qom-list-types","id":"libvirt-8"}' for write with FD -1
2014-11-19 18:52:32.288+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-blk-pci"},"id":"libvirt-9"}' 
for write with FD -1
2014-11-19 18:52:32.293+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-net-pci"},"id":"libvirt-10"}' 
for write with FD -1
2014-11-19 18:52:32.297+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-blk-ccw"},"id":"libvirt-11"}' 
for write with FD -1
2014-11-19 18:52:32.299+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-net-ccw"},"id":"libvirt-12"}' 
for write with FD -1
2014-11-19 18:52:32.301+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-blk-s390"},"id":"libvirt-13"}' 
for write with FD -1
2014-11-19 18:52:32.304+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"virtio-net-s390"},"id":"libvirt-14"}' 
for write with FD -1
2014-11-19 18:52:32.307+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"pci-assign"},"id":"libvirt-15"}' 
for write with FD -1
2014-11-19 18:52:32.309+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"kvm-pci-assign"},"id":"libvirt-16"}' 
for write with FD -1
2014-11-19 18:52:32.313+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"vfio-pci"},"id":"libvirt-17"}' 
for write with FD -1
2014-11-19 18:52:32.316+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"scsi-disk"},"id":"libvirt-18"}' 
for write with FD -1
2014-11-19 18:52:32.321+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"ide-drive"},"id":"libvirt-19"}' 
for write with FD -1
2014-11-19 18:52:32.325+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"PIIX4_PM"},"id":"libvirt-20"}' 
for write with FD -1
2014-11-19 18:52:32.328+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"usb-redir"},"id":"libvirt-21"}' 
for write with FD -1
2014-11-19 18:52:32.331+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"usb-host"},"id":"libvirt-22"}' 
for write with FD -1
2014-11-19 18:52:32.334+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"scsi-generic"},"id":"libvirt-23"}' 
for write with FD -1
2014-11-19 18:52:32.336+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"i440FX-pcihost"},"id":"libvirt-24"}' 
for write with FD -1
2014-11-19 18:52:32.340+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"q35-pcihost"},"id":"libvirt-25"}' 
for write with FD -1
2014-11-19 18:52:32.343+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"usb-storage"},"id":"libvirt-26"}' 
for write with FD -1
2014-11-19 18:52:32.347+0000 Send command 
'{"execute":"device-list-properties","arguments":{"typename":"kvm-pit"},"id":"libvirt-27"}' 
for write with FD -1
2014-11-19 18:52:32.350+0000 Send command 
'{"execute":"query-machines","id":"libvirt-28"}' for write with FD -1
2014-11-19 18:52:32.355+0000 Send command 
'{"execute":"query-cpu-definitions","id":"libvirt-29"}' for write with FD -1
2014-11-19 18:52:32.358+0000 Send command 
'{"execute":"query-kvm","id":"libvirt-30"}' for write with FD -1
2014-11-19 18:52:32.360+0000 Send command 
'{"execute":"query-tpm-models","id":"libvirt-31"}' for write with FD -1
2014-11-19 18:52:32.362+0000 Send command 
'{"execute":"query-tpm-types","id":"libvirt-32"}' for write with FD -1
2014-11-19 18:52:32.364+0000 Send command 
'{"execute":"query-command-line-options","id":"libvirt-33"}' for write 
with FD -1
2014-11-19 18:52:32.398+0000 Send command 
'{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1
2014-11-19 18:52:32.404+0000 Send command 
'{"execute":"query-commands","id":"libvirt-2"}' for write with FD -1
2014-11-19 18:52:32.416+0000 Send command 
'{"execute":"add-fd","arguments":{"fdset-id":0,"opaque":"/dev/null"},"id":"libvirt-3"}' 
for write with FD 26
2014-11-19 18:52:32.418+0000 Send command 
'{"execute":"block-commit","arguments":{"device":"bogus","speed":0},"id":"libvirt-4"}' 
for write with FD -1
2014-11-19 18:52:32.978+0000 Send command 
'{"execute":"query-events","id":"libvirt-5"}' for write with FD -1
2014-11-19 18:52:32.984+0000 Send command 
'{"execute":"query-status","id":"libvirt-6"}' for write with FD -1
2014-11-19 18:52:32.988+0000 Send command 
'{"execute":"query-block","id":"libvirt-7"}' for write with FD -1
2014-11-19 18:53:10.639+0000 Send command 
'{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1
2014-11-19 18:53:10.649+0000 Send command 
'{"execute":"query-chardev","id":"libvirt-2"}' for write with FD -1
2014-11-19 18:53:10.651+0000 Send command 
'{"execute":"qom-list","arguments":{"path":"/machine/unattached/device[0]"},"id":"libvirt-3"}' 
for write with FD -1
2014-11-19 18:53:10.655+0000 Send command 
'{"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"feature-words"},"id":"libvirt-4"}' 
for write with FD -1
2014-11-19 18:53:10.659+0000 Send command 
'{"execute":"query-cpus","id":"libvirt-5"}' for write with FD -1
2014-11-19 18:53:10.674+0000 Send command 
'{"execute":"set_password","arguments":{"protocol":"vnc","password":"62b6dc26-b3ff-4fdf-9a05-8473237d284a","connected":"keep"},"id":"libvirt-6"}' 
for write with FD -1
2014-11-19 18:53:10.677+0000 Send command 
'{"execute":"expire_password","arguments":{"protocol":"vnc","time":"never"},"id":"libvirt-7"}' 
for write with FD -1
2014-11-19 18:53:10.679+0000 Send command 
'{"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-8"}' 
for write with FD -1
2014-11-19 18:53:10.682+0000 Send command 
'{"execute":"balloon","arguments":{"value":1073741824},"id":"libvirt-9"}' for 
write with FD -1
2014-11-19 18:53:10.685+0000 Send command 
'{"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":{"host":"::","port":"49153"}}},"id":"libvirt-10"}' 
for write with FD -1
2014-11-19 18:53:10.687+0000 Send command 
'{"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk0","writable":true},"id":"libvirt-11"}' 
for write with FD -1
2014-11-19 18:53:41.558+0000 Send command 
'{"execute":"cont","id":"libvirt-12"}' for write with FD -1
2014-11-19 18:53:41.564+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-13"}' for write with FD -1
...
2014-11-19 18:55:00.631+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-30"}' for write with FD -1
2014-11-19 18:55:01.025+0000 Send command 
'{"execute":"migrate_set_speed","arguments":{"value":9223372036853727232},"id":"libvirt-31"}' 
for write with FD -1
2014-11-19 18:55:01.028+0000 Send command 
'{"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-32"}' 
for write with FD 33
2014-11-19 18:55:01.030+0000 Send command 
'{"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":true,"uri":"fd:migrate"},"id":"libvirt-33"}' 
for write with FD -1
2014-11-19 18:55:01.033+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-34"}' for write with FD -1
...
2014-11-19 18:55:10.582+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-202"}' for write with FD -1
2014-11-19 18:55:10.626+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-203"}' for write with FD -1
2014-11-19 18:55:10.636+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-204"}' for write with FD -1
2014-11-19 18:55:10.643+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-205"}' for write with FD -1
...
2014-11-19 18:55:20.588+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-383"}' for write with FD -1
2014-11-19 18:55:20.624+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-384"}' for write with FD -1
2014-11-19 18:55:20.631+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-385"}' for write with FD -1
2014-11-19 18:55:20.642+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-386"}' for write with FD -1
...
2014-11-19 18:55:30.603+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-567"}' for write with FD -1
2014-11-19 18:55:30.623+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-568"}' for write with FD -1
2014-11-19 18:55:30.631+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-569"}' for write with FD -1
2014-11-19 18:55:30.658+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-570"}' for write with FD -1
...
2014-11-19 18:55:40.612+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-747"}' for write with FD -1
2014-11-19 18:55:40.639+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-748"}' for write with FD -1
2014-11-19 18:55:40.646+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-749"}' for write with FD -1
2014-11-19 18:55:40.667+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-750"}' for write with FD -1
...
2014-11-19 18:55:50.613+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-929"}' for write with FD -1
2014-11-19 18:55:50.625+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-930"}' for write with FD -1
2014-11-19 18:55:50.632+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-931"}' for write with FD -1
2014-11-19 18:55:50.669+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-932"}' for write with FD -1
...
2014-11-19 18:56:00.601+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1112"}' for write with FD -1
2014-11-19 18:56:00.625+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1113"}' for write with FD -1
2014-11-19 18:56:00.635+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1114"}' for write with FD -1
2014-11-19 18:56:00.657+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1115"}' for write with FD -1
...
2014-11-19 18:56:10.601+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1294"}' for write with FD -1
2014-11-19 18:56:10.624+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1295"}' for write with FD -1
2014-11-19 18:56:10.632+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1296"}' for write with FD -1
2014-11-19 18:56:10.658+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1297"}' for write with FD -1
...
2014-11-19 18:56:20.609+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1476"}' for write with FD -1
2014-11-19 18:56:20.624+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1477"}' for write with FD -1
2014-11-19 18:56:20.630+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1478"}' for write with FD -1
2014-11-19 18:56:20.663+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1479"}' for write with FD -1
...
2014-11-19 18:56:30.606+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1658"}' for write with FD -1
2014-11-19 18:56:30.624+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1659"}' for write with FD -1
2014-11-19 18:56:30.631+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1660"}' for write with FD -1
2014-11-19 18:56:30.660+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1661"}' for write with FD -1

2014-11-19 18:56:40.597+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1839"}' for write with FD -1
2014-11-19 18:56:40.634+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1840"}' for write with FD -1
2014-11-19 18:56:40.640+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-1841"}' for write with FD -1
2014-11-19 18:56:40.652+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-1842"}' for write with FD -1

2014-11-19 18:56:50.581+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-2021"}' for write with FD -1
2014-11-19 18:56:50.623+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-2022"}' for write with FD -1
2014-11-19 18:56:50.630+0000 Send command 
'{"execute":"query-blockstats","id":"libvirt-2023"}' for write with FD -1
2014-11-19 18:56:50.636+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-2024"}' for write with FD -1
...
2014-11-19 18:56:59.047+0000 Send command 
'{"execute":"query-migrate","id":"libvirt-2175"}' for write with FD -1
2014-11-19 18:56:59.258+0000 Send command 
'{"execute":"block-job-cancel","arguments":{"device":"drive-virtio-disk0"},"id":"libvirt-2176"}' 
for write with FD -1



-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc