[Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Amit Shah]

We were announcing the dest host's IP as our new IP a bit too soon -- if
there were errors detected after this announcement was done, the
migration is failed and the VM could continue running on the src host --
causing problems later.

Move around the qemu_announce_self() call so it's done just before the
VM is runnable.

Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 migration/migration.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/migration/migration.c b/migration/migration.c
index b7de9b7..ca21ba8 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -295,7 +295,6 @@ static void process_incoming_migration_co(void *opaque)
         exit(EXIT_FAILURE);
     }
     migrate_generate_event(MIGRATION_STATUS_COMPLETED);
-    qemu_announce_self();
 
     /* Make sure all file formats flush their mutable metadata */
     bdrv_invalidate_cache_all(&local_err);
@@ -305,6 +304,12 @@ static void process_incoming_migration_co(void *opaque)
         exit(EXIT_FAILURE);
     }
 
+    /*
+     * This must happen after all error conditions are dealt with and
+     * we're sure the VM is going to be running on this host.
+     */
+    qemu_announce_self();
+
     /* If global state section was not received or we are in running
        state, we need to obey autostart. Any other state is set with
        runstate_set. */
-- 
2.4.3

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Michael S. Tsirkin]

On Wed, Oct 14, 2015 at 05:37:19PM +0530, Amit Shah wrote:
> We were announcing the dest host's IP as our new IP a bit too soon -- if
> there were errors detected after this announcement was done, the
> migration is failed and the VM could continue running on the src host --
> causing problems later.
> 
> Move around the qemu_announce_self() call so it's done just before the
> VM is runnable.
> 
> Signed-off-by: Amit Shah <amit.shah@redhat.com>

Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>


> ---
>  migration/migration.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index b7de9b7..ca21ba8 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -295,7 +295,6 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>      migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> -    qemu_announce_self();
>  
>      /* Make sure all file formats flush their mutable metadata */
>      bdrv_invalidate_cache_all(&local_err);
> @@ -305,6 +304,12 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>  
> +    /*
> +     * This must happen after all error conditions are dealt with and
> +     * we're sure the VM is going to be running on this host.
> +     */
> +    qemu_announce_self();
> +
>      /* If global state section was not received or we are in running
>         state, we need to obey autostart. Any other state is set with
>         runstate_set. */
> -- 
> 2.4.3

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Dr. David Alan Gilbert]

* Amit Shah (amit.shah@redhat.com) wrote:
> We were announcing the dest host's IP as our new IP a bit too soon -- if
> there were errors detected after this announcement was done, the
> migration is failed and the VM could continue running on the src host --
> causing problems later.
> 
> Move around the qemu_announce_self() call so it's done just before the
> VM is runnable.
> 
> Signed-off-by: Amit Shah <amit.shah@redhat.com>
> ---
>  migration/migration.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index b7de9b7..ca21ba8 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -295,7 +295,6 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>      migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> -    qemu_announce_self();
>  
>      /* Make sure all file formats flush their mutable metadata */
>      bdrv_invalidate_cache_all(&local_err);
> @@ -305,6 +304,12 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>  
> +    /*
> +     * This must happen after all error conditions are dealt with and
> +     * we're sure the VM is going to be running on this host.
> +     */
> +    qemu_announce_self();
> +

OK, so that's certainly better than it was before; but should be even
further down?  Should we do an announce-self at all if we are not
going to do the vm_start?

Dave

>      /* If global state section was not received or we are in running
>         state, we need to obey autostart. Any other state is set with
>         runstate_set. */
> -- 
> 2.4.3
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Amit Shah]

On (Wed) 14 Oct 2015 [13:38:30], Dr. David Alan Gilbert wrote:
> * Amit Shah (amit.shah@redhat.com) wrote:
> > We were announcing the dest host's IP as our new IP a bit too soon -- if
> > there were errors detected after this announcement was done, the
> > migration is failed and the VM could continue running on the src host --
> > causing problems later.
> > 
> > Move around the qemu_announce_self() call so it's done just before the
> > VM is runnable.
> > 
> > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > ---
> >  migration/migration.c | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/migration/migration.c b/migration/migration.c
> > index b7de9b7..ca21ba8 100644
> > --- a/migration/migration.c
> > +++ b/migration/migration.c
> > @@ -295,7 +295,6 @@ static void process_incoming_migration_co(void *opaque)
> >          exit(EXIT_FAILURE);
> >      }
> >      migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> > -    qemu_announce_self();
> >  
> >      /* Make sure all file formats flush their mutable metadata */
> >      bdrv_invalidate_cache_all(&local_err);
> > @@ -305,6 +304,12 @@ static void process_incoming_migration_co(void *opaque)
> >          exit(EXIT_FAILURE);
> >      }
> >  
> > +    /*
> > +     * This must happen after all error conditions are dealt with and
> > +     * we're sure the VM is going to be running on this host.
> > +     */
> > +    qemu_announce_self();
> > +
> 
> OK, so that's certainly better than it was before; but should be even
> further down?  Should we do an announce-self at all if we are not
> going to do the vm_start?

Assumption is that if migration has succeeded, the VM will be started
on this host.  The VM is paused only because it was in paused state
before the migration started -- due to an IO error, due to non-live
migration -- so IMO this actually is the right place for that.


		Amit

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Juan Quintela]

Amit Shah <amit.shah@redhat.com> wrote:
> We were announcing the dest host's IP as our new IP a bit too soon -- if
> there were errors detected after this announcement was done, the
> migration is failed and the VM could continue running on the src host --
> causing problems later.
>
> Move around the qemu_announce_self() call so it's done just before the
> VM is runnable.
>
> Signed-off-by: Amit Shah <amit.shah@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

applied.

I have the same question than Dave, but also agree that this is a
movement in the right direction.

Why it is not only needed when we do a vm_start()?

And why we can't delay it until that happens?

Later, Juan.

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Michael S. Tsirkin]

On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
> Amit Shah <amit.shah@redhat.com> wrote:
> > We were announcing the dest host's IP as our new IP a bit too soon -- if
> > there were errors detected after this announcement was done, the
> > migration is failed and the VM could continue running on the src host --
> > causing problems later.
> >
> > Move around the qemu_announce_self() call so it's done just before the
> > VM is runnable.
> >
> > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> 
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> 
> applied.
> 
> I have the same question than Dave, but also agree that this is a
> movement in the right direction.
> 
> Why it is not only needed when we do a vm_start()?

It's a complex question.  We don't want to do this on each
vmstop/vmcont. But maybe we want to, on the 1st vmstart after
qemu is started.

> And why we can't delay it until that happens?
> 
> Later, Juan.

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Jason Wang]

On 10/14/2015 08:07 PM, Amit Shah wrote:
> We were announcing the dest host's IP as our new IP a bit too soon -- if
> there were errors detected after this announcement was done, the
> migration is failed and the VM could continue running on the src host --
> causing problems later.
>
> Move around the qemu_announce_self() call so it's done just before the
> VM is runnable.
>
> Signed-off-by: Amit Shah <amit.shah@redhat.com>
> ---
>  migration/migration.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/migration/migration.c b/migration/migration.c
> index b7de9b7..ca21ba8 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -295,7 +295,6 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>      migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> -    qemu_announce_self();
>  
>      /* Make sure all file formats flush their mutable metadata */
>      bdrv_invalidate_cache_all(&local_err);
> @@ -305,6 +304,12 @@ static void process_incoming_migration_co(void *opaque)
>          exit(EXIT_FAILURE);
>      }
>  
> +    /*
> +     * This must happen after all error conditions are dealt with and
> +     * we're sure the VM is going to be running on this host.
> +     */
> +    qemu_announce_self();
> +
>      /* If global state section was not received or we are in running
>         state, we need to obey autostart. Any other state is set with
>         runstate_set. */

Acked-by: Jason Wang <jasowang@redhat.com>

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Jason Wang]

On 10/14/2015 10:12 PM, Michael S. Tsirkin wrote:
> On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
>> Amit Shah <amit.shah@redhat.com> wrote:
>>> We were announcing the dest host's IP as our new IP a bit too soon -- if
>>> there were errors detected after this announcement was done, the
>>> migration is failed and the VM could continue running on the src host --
>>> causing problems later.
>>>
>>> Move around the qemu_announce_self() call so it's done just before the
>>> VM is runnable.
>>>
>>> Signed-off-by: Amit Shah <amit.shah@redhat.com>
>> Reviewed-by: Juan Quintela <quintela@redhat.com>
>>
>> applied.
>>
>> I have the same question than Dave, but also agree that this is a
>> movement in the right direction.
>>
>> Why it is not only needed when we do a vm_start()?
> It's a complex question.  We don't want to do this on each
> vmstop/vmcont. But maybe we want to, on the 1st vmstart after
> qemu is started.

Yes, so may be better to do this when autostart is true?

>
>> And why we can't delay it until that happens?
>>
>> Later, Juan.

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Amit Shah]

On (Wed) 14 Oct 2015 [17:12:44], Michael S. Tsirkin wrote:
> On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
> > Amit Shah <amit.shah@redhat.com> wrote:
> > > We were announcing the dest host's IP as our new IP a bit too soon -- if
> > > there were errors detected after this announcement was done, the
> > > migration is failed and the VM could continue running on the src host --
> > > causing problems later.
> > >
> > > Move around the qemu_announce_self() call so it's done just before the
> > > VM is runnable.
> > >
> > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > 
> > Reviewed-by: Juan Quintela <quintela@redhat.com>
> > 
> > applied.
> > 
> > I have the same question than Dave, but also agree that this is a
> > movement in the right direction.
> > 
> > Why it is not only needed when we do a vm_start()?
> 
> It's a complex question.  We don't want to do this on each
> vmstop/vmcont. But maybe we want to, on the 1st vmstart after
> qemu is started.

Why?  When a guest starts, it will issue ARP requests and everything
will just work.  We need this announce_self only to tell the switches
that the MAC belonging to the guest's IP has changed..

		Amit

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Juan Quintela]

Amit Shah <amit.shah@redhat.com> wrote:
> On (Wed) 14 Oct 2015 [17:12:44], Michael S. Tsirkin wrote:
>> On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
>> > Amit Shah <amit.shah@redhat.com> wrote:
>> > > We were announcing the dest host's IP as our new IP a bit too soon -- if
>> > > there were errors detected after this announcement was done, the
>> > > migration is failed and the VM could continue running on the src host --
>> > > causing problems later.
>> > >
>> > > Move around the qemu_announce_self() call so it's done just before the
>> > > VM is runnable.
>> > >
>> > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
>> > 
>> > Reviewed-by: Juan Quintela <quintela@redhat.com>
>> > 
>> > applied.
>> > 
>> > I have the same question than Dave, but also agree that this is a
>> > movement in the right direction.
>> > 
>> > Why it is not only needed when we do a vm_start()?
>> 
>> It's a complex question.  We don't want to do this on each
>> vmstop/vmcont. But maybe we want to, on the 1st vmstart after
>> qemu is started.
>
> Why?  When a guest starts, it will issue ARP requests and everything
> will just work.  We need this announce_self only to tell the switches
> that the MAC belonging to the guest's IP has changed..

What happens if we stop a guest on one host and we start it on a
different host?

If the communication is started from a different place, packets will go
to old host, until some TCP timeout happens, right?

Later, Juan.

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Amit Shah]

On (Thu) 15 Oct 2015 [08:36:49], Juan Quintela wrote:
> Amit Shah <amit.shah@redhat.com> wrote:
> > On (Wed) 14 Oct 2015 [17:12:44], Michael S. Tsirkin wrote:
> >> On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
> >> > Amit Shah <amit.shah@redhat.com> wrote:
> >> > > We were announcing the dest host's IP as our new IP a bit too soon -- if
> >> > > there were errors detected after this announcement was done, the
> >> > > migration is failed and the VM could continue running on the src host --
> >> > > causing problems later.
> >> > >
> >> > > Move around the qemu_announce_self() call so it's done just before the
> >> > > VM is runnable.
> >> > >
> >> > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> >> > 
> >> > Reviewed-by: Juan Quintela <quintela@redhat.com>
> >> > 
> >> > applied.
> >> > 
> >> > I have the same question than Dave, but also agree that this is a
> >> > movement in the right direction.
> >> > 
> >> > Why it is not only needed when we do a vm_start()?
> >> 
> >> It's a complex question.  We don't want to do this on each
> >> vmstop/vmcont. But maybe we want to, on the 1st vmstart after
> >> qemu is started.
> >
> > Why?  When a guest starts, it will issue ARP requests and everything
> > will just work.  We need this announce_self only to tell the switches
> > that the MAC belonging to the guest's IP has changed..
> 
> What happens if we stop a guest on one host and we start it on a
> different host?

So the migration code doesn't get involved?  IOW, we don't call
qemu_announce_self() at all?  I'd say that's a corner case: we provide
live migration capability which sets up things fine; if people choose
to use something else, they have to do their own setup.

> If the communication is started from a different place, packets will go
> to old host, until some TCP timeout happens, right?

Yes, packets from remote will keep going to the old host.  If the old
host has since closed the qemu process, it will give tcp errors to the
remote, and the remote will in time shut down its sockets.  Also, when
the VM sends out any packets, switches could adjust their tables and
send remote packets to the new host.  Depends on how smart the
switches are.


		Amit

Re: [Qemu-devel] [PATCH 1/1] migration: announce VM's new home just before VM is runnable

[Michael S. Tsirkin]

On Thu, Oct 15, 2015 at 01:34:10PM +0530, Amit Shah wrote:
> On (Thu) 15 Oct 2015 [08:36:49], Juan Quintela wrote:
> > Amit Shah <amit.shah@redhat.com> wrote:
> > > On (Wed) 14 Oct 2015 [17:12:44], Michael S. Tsirkin wrote:
> > >> On Wed, Oct 14, 2015 at 03:21:15PM +0200, Juan Quintela wrote:
> > >> > Amit Shah <amit.shah@redhat.com> wrote:
> > >> > > We were announcing the dest host's IP as our new IP a bit too soon -- if
> > >> > > there were errors detected after this announcement was done, the
> > >> > > migration is failed and the VM could continue running on the src host --
> > >> > > causing problems later.
> > >> > >
> > >> > > Move around the qemu_announce_self() call so it's done just before the
> > >> > > VM is runnable.
> > >> > >
> > >> > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > >> > 
> > >> > Reviewed-by: Juan Quintela <quintela@redhat.com>
> > >> > 
> > >> > applied.
> > >> > 
> > >> > I have the same question than Dave, but also agree that this is a
> > >> > movement in the right direction.
> > >> > 
> > >> > Why it is not only needed when we do a vm_start()?
> > >> 
> > >> It's a complex question.  We don't want to do this on each
> > >> vmstop/vmcont. But maybe we want to, on the 1st vmstart after
> > >> qemu is started.
> > >
> > > Why?  When a guest starts, it will issue ARP requests and everything
> > > will just work.  We need this announce_self only to tell the switches
> > > that the MAC belonging to the guest's IP has changed..
> > 
> > What happens if we stop a guest on one host and we start it on a
> > different host?
> 
> So the migration code doesn't get involved?  IOW, we don't call
> qemu_announce_self() at all?  I'd say that's a corner case: we provide
> live migration capability which sets up things fine; if people choose
> to use something else, they have to do their own setup.

I agree it's not a big deal.
Still, if someone is inclined to always announce self on first vm start,
that would also be OK.

> > If the communication is started from a different place, packets will go
> > to old host, until some TCP timeout happens, right?
> 
> Yes, packets from remote will keep going to the old host.  If the old
> host has since closed the qemu process, it will give tcp errors to the
> remote, and the remote will in time shut down its sockets.  Also, when
> the VM sends out any packets, switches could adjust their tables and
> send remote packets to the new host.  Depends on how smart the
> switches are.
> 
> 
> 		Amit