[Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

Hi all.

We now have the following problem:

If dirty-bitmaps migration capability is enabled, persistance flag is 
dropped for all migrated bitmaps, to prevent their storing to the 
storage on inactivate. It works ok, persistence itself is migrated 
through the migration channel. But on source, bitmaps becomes not 
persistent, so if we, for some reasons, want to continue using source 
vm, we'll lose bitmaps on stop/start.

It's simple to fix it: just make bitmaps persistent again on invalidate 
[1].. But I have some questions.

1. What are possible cases? I think about the following:

a. migration cancel or fail, then invalidate
b. migration success, then qmp cont => invalidate
c. migration success, then stop/start (there was no invalidate, so [1] 
will not work)


2. Is it safe at all, saving bitmaps after inactivate, even without 
persistence?

Inactive disk implies, that it may be changed by somebody other, isn't 
it? Is it possible, that target will change the disk, and then we return 
control to the source? In this case bitmaps will be invalid. So, should 
not we drop all the bitmaps on inactivate?

-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Dr. David Alan Gilbert]

* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> Hi all.
> 
> We now have the following problem:
> 
> If dirty-bitmaps migration capability is enabled, persistance flag is
> dropped for all migrated bitmaps, to prevent their storing to the storage on
> inactivate. It works ok, persistence itself is migrated through the
> migration channel. But on source, bitmaps becomes not persistent, so if we,
> for some reasons, want to continue using source vm, we'll lose bitmaps on
> stop/start.
> 
> It's simple to fix it: just make bitmaps persistent again on invalidate
> [1].. But I have some questions.
> 
> 1. What are possible cases? I think about the following:
> 
> a. migration cancel or fail, then invalidate
> b. migration success, then qmp cont => invalidate
> c. migration success, then stop/start (there was no invalidate, so [1] will
> not work)
> 
> 
> 2. Is it safe at all, saving bitmaps after inactivate, even without
> persistence?
> 
> Inactive disk implies, that it may be changed by somebody other, isn't it?
> Is it possible, that target will change the disk, and then we return control
> to the source? In this case bitmaps will be invalid. So, should not we drop
> all the bitmaps on inactivate?

I don't know the full answers; but I do know it's valid to have the case
(b) - i.e. from QEMU's point of view the migration succeeds, but the
destination was run with -S and for some external failure reason, the
management layer decides to kill the destination and restart the source.
And you're right, that at that point there's no one holding the lock,
so the source can't be sure that no one snuck in and fiddled with the
disk before restarting.

Dave

> -- 
> Best regards,
> Vladimir
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[John Snow]

On 04/18/2018 10:00 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all.
> 
> We now have the following problem:
> 
> If dirty-bitmaps migration capability is enabled, persistance flag is
> dropped for all migrated bitmaps, to prevent their storing to the
> storage on inactivate. It works ok, persistence itself is migrated
> through the migration channel. But on source, bitmaps becomes not
> persistent, so if we, for some reasons, want to continue using source
> vm, we'll lose bitmaps on stop/start.
> 

Sorry for not following along more carefully, which kind of migration
are we talking about in this case?

> It's simple to fix it: just make bitmaps persistent again on invalidate
> [1].. But I have some questions.
> 
> 1. What are possible cases? I think about the following:
> 
> a. migration cancel or fail, then invalidate
> b. migration success, then qmp cont => invalidate
> c. migration success, then stop/start (there was no invalidate, so [1]
> will not work)
> 
> 
> 2. Is it safe at all, saving bitmaps after inactivate, even without
> persistence?
> 
> Inactive disk implies, that it may be changed by somebody other, isn't
> it? Is it possible, that target will change the disk, and then we return
> control to the source? In this case bitmaps will be invalid. So, should
> not we drop all the bitmaps on inactivate?
> 

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

12.05.2018 00:23, John Snow wrote:
>
> On 04/18/2018 10:00 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all.
>>
>> We now have the following problem:
>>
>> If dirty-bitmaps migration capability is enabled, persistance flag is
>> dropped for all migrated bitmaps, to prevent their storing to the
>> storage on inactivate. It works ok, persistence itself is migrated
>> through the migration channel. But on source, bitmaps becomes not
>> persistent, so if we, for some reasons, want to continue using source
>> vm, we'll lose bitmaps on stop/start.
>>
> Sorry for not following along more carefully, which kind of migration
> are we talking about in this case?

Any migration with dirty-bitmaps capability enabled..

>
>> It's simple to fix it: just make bitmaps persistent again on invalidate
>> [1].. But I have some questions.
>>
>> 1. What are possible cases? I think about the following:
>>
>> a. migration cancel or fail, then invalidate
>> b. migration success, then qmp cont => invalidate
>> c. migration success, then stop/start (there was no invalidate, so [1]
>> will not work)
>>
>>
>> 2. Is it safe at all, saving bitmaps after inactivate, even without
>> persistence?
>>
>> Inactive disk implies, that it may be changed by somebody other, isn't
>> it? Is it possible, that target will change the disk, and then we return
>> control to the source? In this case bitmaps will be invalid. So, should
>> not we drop all the bitmaps on inactivate?
>>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Fam Zheng]

On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
> Hi all.
> 
> We now have the following problem:
> 
> If dirty-bitmaps migration capability is enabled, persistance flag is
> dropped for all migrated bitmaps, to prevent their storing to the storage on
> inactivate. 

Why do we prevent source storing persistent bitmaps by clearing the flag instead
of making the bitmap code more BDRV_O_INACTIVE-aware, so it is _not_ stored
when/after inactivation?

> It works ok, persistence itself is migrated through the
> migration channel. But on source, bitmaps becomes not persistent, so if we,
> for some reasons, want to continue using source vm, we'll lose bitmaps on
> stop/start.
> 
> It's simple to fix it: just make bitmaps persistent again on invalidate
> [1].. But I have some questions.
> 
> 1. What are possible cases? I think about the following:
> 
> a. migration cancel or fail, then invalidate
> b. migration success, then qmp cont => invalidate

Is this "cont" on the source even though dst is already up? How will this work?
Isn't it expected that dst is using the image?

> c. migration success, then stop/start (there was no invalidate, so [1] will
> not work)
> 
> 
> 2. Is it safe at all, saving bitmaps after inactivate, even without
> persistence?

This is not safe. No I/O should be done to the image after inactivation.

> 
> Inactive disk implies, that it may be changed by somebody other, isn't it?

Yes.

> Is it possible, that target will change the disk, and then we return control
> to the source? In this case bitmaps will be invalid. So, should not we drop
> all the bitmaps on inactivate?

Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
fails to start, and we want to resume VM at src, we could (optionally?) reload
the persistent bitmaps, I guess.

> 
> -- 
> Best regards,
> Vladimir
> 

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

14.05.2018 09:41, Fam Zheng wrote:
> On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all.
>>
>> We now have the following problem:
>>
>> If dirty-bitmaps migration capability is enabled, persistance flag is
>> dropped for all migrated bitmaps, to prevent their storing to the storage on
>> inactivate.
> Why do we prevent source storing persistent bitmaps by clearing the flag instead
> of making the bitmap code more BDRV_O_INACTIVE-aware, so it is _not_ stored
> when/after inactivation?

Bitmaps are stored exactly on inactivation. So, we need some
flag, that on following inactivation, we don't want to save them. And 
this flag
is .persistent flag.

Of course, we can't save them if BDRV_O_INACTIVE is already set

>
>> It works ok, persistence itself is migrated through the
>> migration channel. But on source, bitmaps becomes not persistent, so if we,
>> for some reasons, want to continue using source vm, we'll lose bitmaps on
>> stop/start.
>>
>> It's simple to fix it: just make bitmaps persistent again on invalidate
>> [1].. But I have some questions.
>>
>> 1. What are possible cases? I think about the following:
>>
>> a. migration cancel or fail, then invalidate
>> b. migration success, then qmp cont => invalidate
> Is this "cont" on the source even though dst is already up? How will this work?
> Isn't it expected that dst is using the image?

Dr. David described this case, look at his answer.

>
>> c. migration success, then stop/start (there was no invalidate, so [1] will
>> not work)
>>
>>
>> 2. Is it safe at all, saving bitmaps after inactivate, even without
>> persistence?
> This is not safe. No I/O should be done to the image after inactivation.
>
>> Inactive disk implies, that it may be changed by somebody other, isn't it?
> Yes.
>
>> Is it possible, that target will change the disk, and then we return control
>> to the source? In this case bitmaps will be invalid. So, should not we drop
>> all the bitmaps on inactivate?
> Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
> fails to start, and we want to resume VM at src, we could (optionally?) reload
> the persistent bitmaps, I guess.

Reload from where? We didn't store them.

So, you agree, that dropping all bitmaps after inactivation is good 
idea.. The second question, is it possible to not drop them? Is there a 
way, to check that disk was not changed after pair of 
inactivate-invalidate? I have an idea:
we can create small-bitmap, which will not migrate through migration 
channel, but remain persistent. It will be very small (big granularity, 
to not increase downtime of migration), so after invalidate we can check 
this bitmap. If it is empty, we are happy, we can "activate" all our 
bitmaps and make them persistent again. If not, we have two ways:

1. drop all bitmaps
2. merge small-bitmap to all our bitmaps

>
>> -- 
>> Best regards,
>> Vladimir
>>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

14.05.2018 13:09, Vladimir Sementsov-Ogievskiy wrote:
> 14.05.2018 09:41, Fam Zheng wrote:
>> On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi all.
>>>
>>> We now have the following problem:
>>>
>>> If dirty-bitmaps migration capability is enabled, persistance flag is
>>> dropped for all migrated bitmaps, to prevent their storing to the 
>>> storage on
>>> inactivate.
>> Why do we prevent source storing persistent bitmaps by clearing the 
>> flag instead
>> of making the bitmap code more BDRV_O_INACTIVE-aware, so it is _not_ 
>> stored
>> when/after inactivation?
>
> Bitmaps are stored exactly on inactivation. So, we need some
> flag, that on following inactivation, we don't want to save them. And 
> this flag
> is .persistent flag.
>
> Of course, we can't save them if BDRV_O_INACTIVE is already set
>
>>
>>> It works ok, persistence itself is migrated through the
>>> migration channel. But on source, bitmaps becomes not persistent, so 
>>> if we,
>>> for some reasons, want to continue using source vm, we'll lose 
>>> bitmaps on
>>> stop/start.
>>>
>>> It's simple to fix it: just make bitmaps persistent again on invalidate
>>> [1].. But I have some questions.
>>>
>>> 1. What are possible cases? I think about the following:
>>>
>>> a. migration cancel or fail, then invalidate
>>> b. migration success, then qmp cont => invalidate
>> Is this "cont" on the source even though dst is already up? How will 
>> this work?
>> Isn't it expected that dst is using the image?
>
> Dr. David described this case, look at his answer.
>
>>
>>> c. migration success, then stop/start (there was no invalidate, so 
>>> [1] will
>>> not work)
>>>
>>>
>>> 2. Is it safe at all, saving bitmaps after inactivate, even without
>>> persistence?
>> This is not safe. No I/O should be done to the image after inactivation.
>>
>>> Inactive disk implies, that it may be changed by somebody other, 
>>> isn't it?
>> Yes.
>>
>>> Is it possible, that target will change the disk, and then we return 
>>> control
>>> to the source? In this case bitmaps will be invalid. So, should not 
>>> we drop
>>> all the bitmaps on inactivate?
>> Yes, dropping all live bitmaps upon inactivate sounds reasonable. If 
>> the dst
>> fails to start, and we want to resume VM at src, we could 
>> (optionally?) reload
>> the persistent bitmaps, I guess.
>
> Reload from where? We didn't store them.
>
> So, you agree, that dropping all bitmaps after inactivation is good 
> idea.. The second question, is it possible to not drop them? Is there 
> a way, to check that disk was not changed after pair of 
> inactivate-invalidate? I have an idea:
> we can create small-bitmap, which will not migrate through migration 
> channel, but remain persistent. It will be very small (big 
> granularity, to not increase downtime of migration), so after 
> invalidate we can check this bitmap. If it is empty, we are happy, we 
> can "activate" all our bitmaps and make them persistent again. If not, 
> we have two ways:
>
> 1. drop all bitmaps
> 2. merge small-bitmap to all our bitmaps

However, we must not start source, if disk was changed, as memory and 
devices states will not correspond to disk. So, such a small bitmap may 
be used to check, can we start source or not.

>
>>
>>> -- 
>>> Best regards,
>>> Vladimir
>>>
>
>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Fam Zheng]

On Mon, 05/14 13:23, Vladimir Sementsov-Ogievskiy wrote:
> > So, you agree, that dropping all bitmaps after inactivation is good
> > idea.. The second question, is it possible to not drop them? Is there a
> > way, to check that disk was not changed after pair of
> > inactivate-invalidate? I have an idea:
> > we can create small-bitmap, which will not migrate through migration
> > channel, but remain persistent. It will be very small (big granularity,
> > to not increase downtime of migration), so after invalidate we can check
> > this bitmap. If it is empty, we are happy, we can "activate" all our
> > bitmaps and make them persistent again. If not, we have two ways:
> > 
> > 1. drop all bitmaps
> > 2. merge small-bitmap to all our bitmaps
> 
> However, we must not start source, if disk was changed, as memory and
> devices states will not correspond to disk. So, such a small bitmap may be
> used to check, can we start source or not.

Or it can be a generation number/uuid that is updated when the image is changed
(upon the first write after each open), similar to VMDK's CID and VHDX GUIDs. We
can compare the on disk value with the known value at source QEMU, if they match
it means the image data is not touched.

Fam

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Kevin Wolf]

Am 14.05.2018 um 12:09 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 14.05.2018 09:41, Fam Zheng wrote:
> > On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
> > > Is it possible, that target will change the disk, and then we return control
> > > to the source? In this case bitmaps will be invalid. So, should not we drop
> > > all the bitmaps on inactivate?
> > Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
> > fails to start, and we want to resume VM at src, we could (optionally?) reload
> > the persistent bitmaps, I guess.
> 
> Reload from where? We didn't store them.

Maybe this just means that it turns out that not storing them was a bad
idea?

What was the motivation for not storing the bitmap? The additional
downtime? Is it really that bad, though? Bitmaps should be fairly small
for the usual image sizes and writing them out should be quick.

Kevin

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

16.05.2018 15:47, Kevin Wolf wrote:
> Am 14.05.2018 um 12:09 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 14.05.2018 09:41, Fam Zheng wrote:
>>> On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
>>>> Is it possible, that target will change the disk, and then we return control
>>>> to the source? In this case bitmaps will be invalid. So, should not we drop
>>>> all the bitmaps on inactivate?
>>> Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
>>> fails to start, and we want to resume VM at src, we could (optionally?) reload
>>> the persistent bitmaps, I guess.
>> Reload from where? We didn't store them.
> Maybe this just means that it turns out that not storing them was a bad
> idea?
>
> What was the motivation for not storing the bitmap? The additional
> downtime? Is it really that bad, though? Bitmaps should be fairly small
> for the usual image sizes and writing them out should be quick.
>
> Kevin

What are usual ones? A bitmap of standard granularity of 64k for 16Tb 
disk is ~30mb. If we have several such bitmaps it may be significant 
downtime.



-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Kevin Wolf]

Am 16.05.2018 um 17:10 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 16.05.2018 15:47, Kevin Wolf wrote:
> > Am 14.05.2018 um 12:09 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > 14.05.2018 09:41, Fam Zheng wrote:
> > > > On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
> > > > > Is it possible, that target will change the disk, and then we return control
> > > > > to the source? In this case bitmaps will be invalid. So, should not we drop
> > > > > all the bitmaps on inactivate?
> > > > Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
> > > > fails to start, and we want to resume VM at src, we could (optionally?) reload
> > > > the persistent bitmaps, I guess.
> > > Reload from where? We didn't store them.
> > Maybe this just means that it turns out that not storing them was a bad
> > idea?
> > 
> > What was the motivation for not storing the bitmap? The additional
> > downtime? Is it really that bad, though? Bitmaps should be fairly small
> > for the usual image sizes and writing them out should be quick.
> 
> What are usual ones? A bitmap of standard granularity of 64k for 16Tb disk
> is ~30mb. If we have several such bitmaps it may be significant downtime.

We could have an in-memory bitmap that tracks which parts of the
persistent bitmap are dirty so that you don't have to write out the
whole 30 MB during the migration downtime, but can already flush most of
the persistent bitmap before the VM is stopped.

Kevin

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Vladimir Sementsov-Ogievskiy]

16.05.2018 18:32, Kevin Wolf wrote:
> Am 16.05.2018 um 17:10 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 16.05.2018 15:47, Kevin Wolf wrote:
>>> Am 14.05.2018 um 12:09 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> 14.05.2018 09:41, Fam Zheng wrote:
>>>>> On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> Is it possible, that target will change the disk, and then we return control
>>>>>> to the source? In this case bitmaps will be invalid. So, should not we drop
>>>>>> all the bitmaps on inactivate?
>>>>> Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
>>>>> fails to start, and we want to resume VM at src, we could (optionally?) reload
>>>>> the persistent bitmaps, I guess.
>>>> Reload from where? We didn't store them.
>>> Maybe this just means that it turns out that not storing them was a bad
>>> idea?
>>>
>>> What was the motivation for not storing the bitmap? The additional
>>> downtime? Is it really that bad, though? Bitmaps should be fairly small
>>> for the usual image sizes and writing them out should be quick.
>> What are usual ones? A bitmap of standard granularity of 64k for 16Tb disk
>> is ~30mb. If we have several such bitmaps it may be significant downtime.
> We could have an in-memory bitmap that tracks which parts of the
> persistent bitmap are dirty so that you don't have to write out the
> whole 30 MB during the migration downtime, but can already flush most of
> the persistent bitmap before the VM is stopped.
>
> Kevin

Yes it looks possible. But how to control that downtime? Introduce 
migration state, with specific _pending function? However, it may be not 
necessary.

Anyway, I think we don't need to store it.

If we decided to resume source, bitmap is already in memory, why to 
reload it? If someone already killed source (which was in paused mode), 
it is inconsistent anyway and loss of dirty bitmap is not the worst 
possible problem.

So, finally, it looks safe enough, just to make bitmaps on source 
persistent again (or better, introduce another way to skip storing (may 
be with additional flag, so everybody will be happy), not dropping 
persistent flag). And, after source resume, we have one of the following 
situations:

1. disk was not changed during migration, so, all is ok and we have bitmaps
2. disk was changed. bitmaps are inconsistent. But not only bitmaps, the 
whole vm state is inconsistent with it's disk. This case is a bug in 
management layer and it should never happen. And possibly, we need some 
separate way, to catch such cases.

-- 
Best regards,
Vladimir

Re: [Qemu-devel] Restoring bitmaps after failed/cancelled migration

[Fam Zheng]

On Wed, 05/16 18:52, Vladimir Sementsov-Ogievskiy wrote:
> 16.05.2018 18:32, Kevin Wolf wrote:
> > Am 16.05.2018 um 17:10 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > 16.05.2018 15:47, Kevin Wolf wrote:
> > > > Am 14.05.2018 um 12:09 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > > > 14.05.2018 09:41, Fam Zheng wrote:
> > > > > > On Wed, 04/18 17:00, Vladimir Sementsov-Ogievskiy wrote:
> > > > > > > Is it possible, that target will change the disk, and then we return control
> > > > > > > to the source? In this case bitmaps will be invalid. So, should not we drop
> > > > > > > all the bitmaps on inactivate?
> > > > > > Yes, dropping all live bitmaps upon inactivate sounds reasonable. If the dst
> > > > > > fails to start, and we want to resume VM at src, we could (optionally?) reload
> > > > > > the persistent bitmaps, I guess.
> > > > > Reload from where? We didn't store them.
> > > > Maybe this just means that it turns out that not storing them was a bad
> > > > idea?
> > > > 
> > > > What was the motivation for not storing the bitmap? The additional
> > > > downtime? Is it really that bad, though? Bitmaps should be fairly small
> > > > for the usual image sizes and writing them out should be quick.
> > > What are usual ones? A bitmap of standard granularity of 64k for 16Tb disk
> > > is ~30mb. If we have several such bitmaps it may be significant downtime.
> > We could have an in-memory bitmap that tracks which parts of the
> > persistent bitmap are dirty so that you don't have to write out the
> > whole 30 MB during the migration downtime, but can already flush most of
> > the persistent bitmap before the VM is stopped.
> > 
> > Kevin
> 
> Yes it looks possible. But how to control that downtime? Introduce migration
> state, with specific _pending function? However, it may be not necessary.
> 
> Anyway, I think we don't need to store it.
> 
> If we decided to resume source, bitmap is already in memory, why to reload
> it? If someone already killed source (which was in paused mode), it is
> inconsistent anyway and loss of dirty bitmap is not the worst possible
> problem.
> 
> So, finally, it looks safe enough, just to make bitmaps on source persistent
> again (or better, introduce another way to skip storing (may be with
> additional flag, so everybody will be happy), not dropping persistent flag).

This makes some sense to me. We'll then use the current persistent flag to
indicate the bitmap "is" a persistent one, instead of "should it be persisted".
They are apparently two different properties in the case discussed in this
thread.

> And, after source resume, we have one of the following situations:
> 
> 1. disk was not changed during migration, so, all is ok and we have bitmaps
> 2. disk was changed. bitmaps are inconsistent. But not only bitmaps, the
> whole vm state is inconsistent with it's disk. This case is a bug in
> management layer and it should never happen. And possibly, we need some
> separate way, to catch such cases.

Fam