From: Peter Xu <peterx@redhat.com>
To: qemu-devel@nongnu.org
Cc: Laurent Vivier <lvivier@redhat.com>,
Alexey Perevalov <a.perevalov@samsung.com>,
Juan Quintela <quintela@redhat.com>,
Andrea Arcangeli <aarcange@redhat.com>,
"Dr . David Alan Gilbert" <dgilbert@redhat.com>,
peterx@redhat.com
Subject: [Qemu-devel] [RFC 00/29] Migration: postcopy failure recovery
Date: Fri, 28 Jul 2017 16:06:09 +0800 [thread overview]
Message-ID: <1501229198-30588-1-git-send-email-peterx@redhat.com> (raw)
As we all know that postcopy migration has a potential risk to lost
the VM if the network is broken during the migration. This series
tries to solve the problem by allowing the migration to pause at the
failure point, and do recovery after the link is reconnected.
There was existing work on this issue from Md Haris Iqbal:
https://lists.nongnu.org/archive/html/qemu-devel/2016-08/msg03468.html
This series is a totally re-work of the issue, based on Alexey
Perevalov's recved bitmap v8 series:
https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg06401.html
Two new status are added to support the migration (used on both
sides):
MIGRATION_STATUS_POSTCOPY_PAUSED
MIGRATION_STATUS_POSTCOPY_RECOVER
The MIGRATION_STATUS_POSTCOPY_PAUSED state will be set when the
network failure is detected. It is a phase that we'll be in for a long
time as long as the failure is detected, and we'll be there until a
recovery is triggered. In this state, all the threads (on source:
send thread, return-path thread; destination: ram-load thread,
page-fault thread) will be halted.
The MIGRATION_STATUS_POSTCOPY_RECOVER state is short. If we triggered
a recovery, both source/destination VM will jump into this stage, do
whatever it needs to prepare the recovery (e.g., currently the most
important thing is to synchronize the dirty bitmap, please see commit
messages for more information). After the preparation is ready, the
source will do the final handshake with destination, then both sides
will switch back to MIGRATION_STATUS_POSTCOPY_ACTIVE again.
New commands/messages are defined as well to satisfy the need:
MIG_CMD_RECV_BITMAP & MIG_RP_MSG_RECV_BITMAP are introduced for
delivering received bitmaps
MIG_CMD_RESUME & MIG_RP_MSG_RESUME_ACK are introduced to do the final
handshake of postcopy recovery.
Here's some more details on how the whole failure/recovery routine is
happened:
- start migration
- ... (switch from precopy to postcopy)
- both sides are in "postcopy-active" state
- ... (failure happened, e.g., network unplugged)
- both sides switch to "postcopy-paused" state
- all the migration threads are stopped on both sides
- ... (both VMs hanged)
- ... (user triggers recovery using "migrate -r -d tcp:HOST:PORT" on
source side, "-r" means "recover")
- both sides switch to "postcopy-recover" state
- on source: send-thread, return-path-thread will be waked up
- on dest: ram-load-thread waked up, fault-thread still paused
- source calls new savevmhandler hook resume_prepare() (currently,
only ram is providing the hook):
- ram_resume_prepare(): for each ramblock, fetch recved bitmap by:
- src sends MIG_CMD_RECV_BITMAP to dst
- dst replies MIG_RP_MSG_RECV_BITMAP to src, with bitmap data
- src uses the recved bitmap to rebuild dirty bitmap
- source do final handshake with destination
- src sends MIG_CMD_RESUME to dst, telling "src is ready"
- when dst receives the command, fault thread will be waked up,
meanwhile, dst switch back to "postcopy-active"
- dst sends MIG_RP_MSG_RESUME_ACK to src, telling "dst is ready"
- when src receives the ack, state switch to "postcopy-active"
- postcopy migration continued
Testing:
As I said, it's still an extremely simple test. I used socat to create
a socket bridge:
socat tcp-listen:6666 tcp-connect:localhost:5555 &
Then do the migration via the bridge. I emulated the network failure
by killing the socat process (bridge down), then tries to recover the
migration using the other channel (default dst channel). It looks
like:
port:6666 +------------------+
+----------> | socat bridge [1] |-------+
| +------------------+ |
| (Original channel) |
| | port: 5555
+---------+ (Recovery channel) +--->+---------+
| src VM |------------------------------------>| dst VM |
+---------+ +---------+
Known issues/notes:
- currently destination listening port still cannot change. E.g., the
recovery should be using the same port on destination for
simplicity. (on source, we can specify new URL)
- the patch: "migration: let dst listen on port always" is still
hacky, it just kept the incoming accept open forever for now...
- some migration numbers might still be inaccurate, like total
migration time, etc. (But I don't really think that matters much
now)
- the patches are very lightly tested.
- Dave reported one problem that may hang destination main loop thread
(one vcpu thread holds the BQL) and the rest. I haven't encountered
it yet, but it does not mean this series can survive with it.
- other potential issues that I may have forgotten or unnoticed...
Anyway, the work is still in preliminary stage. Any suggestions and
comments are greatly welcomed. Thanks.
Peter Xu (29):
migration: fix incorrect postcopy recved_bitmap
migration: fix comment disorder in RAMState
io: fix qio_channel_socket_accept err handling
bitmap: introduce bitmap_invert()
bitmap: introduce bitmap_count_one()
migration: dump str in migrate_set_state trace
migration: better error handling with QEMUFile
migration: reuse mis->userfault_quit_fd
migration: provide postcopy_fault_thread_notify()
migration: new property "x-postcopy-fast"
migration: new postcopy-pause state
migration: allow dst vm pause on postcopy
migration: allow src return path to pause
migration: allow send_rq to fail
migration: allow fault thread to pause
qmp: hmp: add migrate "resume" option
migration: rebuild channel on source
migration: new state "postcopy-recover"
migration: let dst listen on port always
migration: wakeup dst ram-load-thread for recover
migration: new cmd MIG_CMD_RECV_BITMAP
migration: new message MIG_RP_MSG_RECV_BITMAP
migration: new cmd MIG_CMD_POSTCOPY_RESUME
migration: new message MIG_RP_MSG_RESUME_ACK
migration: introduce SaveVMHandlers.resume_prepare
migration: synchronize dirty bitmap for resume
migration: setup ramstate for resume
migration: final handshake for the resume
migration: reset migrate thread vars when resumed
hmp-commands.hx | 7 +-
hmp.c | 4 +-
include/migration/register.h | 2 +
include/qemu/bitmap.h | 20 ++
io/channel-socket.c | 1 +
migration/exec.c | 2 +-
migration/fd.c | 2 +-
migration/migration.c | 465 ++++++++++++++++++++++++++++++++++++++++---
migration/migration.h | 25 ++-
migration/postcopy-ram.c | 109 +++++++---
migration/postcopy-ram.h | 2 +
migration/ram.c | 209 ++++++++++++++++++-
migration/ram.h | 4 +
migration/savevm.c | 189 +++++++++++++++++-
migration/savevm.h | 3 +
migration/socket.c | 4 +-
migration/trace-events | 16 +-
qapi-schema.json | 12 +-
util/bitmap.c | 28 +++
19 files changed, 1024 insertions(+), 80 deletions(-)
--
2.7.4
next reply other threads:[~2017-07-28 8:06 UTC|newest]
Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-28 8:06 Peter Xu [this message]
2017-07-28 8:06 ` [Qemu-devel] [RFC 01/29] migration: fix incorrect postcopy recved_bitmap Peter Xu
2017-07-31 16:34 ` Dr. David Alan Gilbert
2017-08-01 2:11 ` Peter Xu
2017-08-01 5:48 ` Alexey Perevalov
2017-08-01 6:02 ` Peter Xu
2017-08-01 6:12 ` Alexey Perevalov
2017-07-28 8:06 ` [Qemu-devel] [RFC 02/29] migration: fix comment disorder in RAMState Peter Xu
2017-07-31 16:39 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 03/29] io: fix qio_channel_socket_accept err handling Peter Xu
2017-07-31 16:53 ` Dr. David Alan Gilbert
2017-08-01 2:25 ` Peter Xu
2017-08-01 8:32 ` Daniel P. Berrange
2017-08-01 8:55 ` Dr. David Alan Gilbert
2017-08-02 3:21 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 04/29] bitmap: introduce bitmap_invert() Peter Xu
2017-07-31 17:11 ` Dr. David Alan Gilbert
2017-08-01 2:43 ` Peter Xu
2017-08-01 8:40 ` Dr. David Alan Gilbert
2017-08-02 3:20 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 05/29] bitmap: introduce bitmap_count_one() Peter Xu
2017-07-31 17:58 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 06/29] migration: dump str in migrate_set_state trace Peter Xu
2017-07-31 18:27 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 07/29] migration: better error handling with QEMUFile Peter Xu
2017-07-31 18:39 ` Dr. David Alan Gilbert
2017-08-01 5:49 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 08/29] migration: reuse mis->userfault_quit_fd Peter Xu
2017-07-31 18:42 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 09/29] migration: provide postcopy_fault_thread_notify() Peter Xu
2017-07-31 18:45 ` Dr. David Alan Gilbert
2017-08-01 3:01 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 10/29] migration: new property "x-postcopy-fast" Peter Xu
2017-07-31 18:52 ` Dr. David Alan Gilbert
2017-08-01 3:13 ` Peter Xu
2017-08-01 8:50 ` Dr. David Alan Gilbert
2017-08-02 3:31 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 11/29] migration: new postcopy-pause state Peter Xu
2017-07-28 15:53 ` Eric Blake
2017-07-31 7:02 ` Peter Xu
2017-07-31 19:06 ` Dr. David Alan Gilbert
2017-08-01 6:28 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 12/29] migration: allow dst vm pause on postcopy Peter Xu
2017-08-01 9:47 ` Dr. David Alan Gilbert
2017-08-02 5:06 ` Peter Xu
2017-08-03 14:03 ` Dr. David Alan Gilbert
2017-08-04 3:43 ` Peter Xu
2017-08-04 9:33 ` Dr. David Alan Gilbert
2017-08-04 9:44 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 13/29] migration: allow src return path to pause Peter Xu
2017-08-01 10:01 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 14/29] migration: allow send_rq to fail Peter Xu
2017-08-01 10:30 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 15/29] migration: allow fault thread to pause Peter Xu
2017-08-01 10:41 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 16/29] qmp: hmp: add migrate "resume" option Peter Xu
2017-07-28 15:57 ` Eric Blake
2017-07-31 7:05 ` Peter Xu
2017-08-01 10:42 ` Dr. David Alan Gilbert
2017-08-01 11:03 ` Daniel P. Berrange
2017-08-02 5:56 ` Peter Xu
2017-08-02 9:28 ` Daniel P. Berrange
2017-07-28 8:06 ` [Qemu-devel] [RFC 17/29] migration: rebuild channel on source Peter Xu
2017-08-01 10:59 ` Dr. David Alan Gilbert
2017-08-02 6:14 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 18/29] migration: new state "postcopy-recover" Peter Xu
2017-08-01 11:36 ` Dr. David Alan Gilbert
2017-08-02 6:42 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 19/29] migration: let dst listen on port always Peter Xu
2017-08-01 10:56 ` Daniel P. Berrange
2017-08-02 7:02 ` Peter Xu
2017-08-02 9:26 ` Daniel P. Berrange
2017-08-02 11:02 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 20/29] migration: wakeup dst ram-load-thread for recover Peter Xu
2017-08-03 9:28 ` Dr. David Alan Gilbert
2017-08-04 5:46 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 21/29] migration: new cmd MIG_CMD_RECV_BITMAP Peter Xu
2017-08-03 9:49 ` Dr. David Alan Gilbert
2017-08-04 6:08 ` Peter Xu
2017-08-04 6:15 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 22/29] migration: new message MIG_RP_MSG_RECV_BITMAP Peter Xu
2017-08-03 10:50 ` Dr. David Alan Gilbert
2017-08-04 6:59 ` Peter Xu
2017-08-04 9:49 ` Dr. David Alan Gilbert
2017-08-07 6:11 ` Peter Xu
2017-08-07 9:04 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 23/29] migration: new cmd MIG_CMD_POSTCOPY_RESUME Peter Xu
2017-08-03 11:05 ` Dr. David Alan Gilbert
2017-08-04 7:04 ` Peter Xu
2017-08-04 7:09 ` Peter Xu
2017-08-04 8:30 ` Dr. David Alan Gilbert
2017-08-04 9:22 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 24/29] migration: new message MIG_RP_MSG_RESUME_ACK Peter Xu
2017-08-03 11:21 ` Dr. David Alan Gilbert
2017-08-04 7:23 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 25/29] migration: introduce SaveVMHandlers.resume_prepare Peter Xu
2017-08-03 11:38 ` Dr. David Alan Gilbert
2017-08-04 7:39 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 26/29] migration: synchronize dirty bitmap for resume Peter Xu
2017-08-03 11:56 ` Dr. David Alan Gilbert
2017-08-04 7:49 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 27/29] migration: setup ramstate " Peter Xu
2017-08-03 12:37 ` Dr. David Alan Gilbert
2017-08-04 8:39 ` Peter Xu
2017-07-28 8:06 ` [Qemu-devel] [RFC 28/29] migration: final handshake for the resume Peter Xu
2017-08-03 13:47 ` Dr. David Alan Gilbert
2017-08-04 9:05 ` Peter Xu
2017-08-04 9:53 ` Dr. David Alan Gilbert
2017-07-28 8:06 ` [Qemu-devel] [RFC 29/29] migration: reset migrate thread vars when resumed Peter Xu
2017-08-03 13:54 ` Dr. David Alan Gilbert
2017-08-04 8:52 ` Peter Xu
2017-08-04 9:52 ` Dr. David Alan Gilbert
2017-08-07 6:57 ` Peter Xu
2017-07-28 10:06 ` [Qemu-devel] [RFC 00/29] Migration: postcopy failure recovery Peter Xu
2017-08-03 15:57 ` Dr. David Alan Gilbert
2017-08-21 7:47 ` Peter Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1501229198-30588-1-git-send-email-peterx@redhat.com \
--to=peterx@redhat.com \
--cc=a.perevalov@samsung.com \
--cc=aarcange@redhat.com \
--cc=dgilbert@redhat.com \
--cc=lvivier@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.