All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v5 00/45] Postcopy implementation
@ 2015-02-25 16:51 Dr. David Alan Gilbert (git)
  2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works Dr. David Alan Gilbert (git)
                   ` (44 more replies)
  0 siblings, 45 replies; 181+ messages in thread
From: Dr. David Alan Gilbert (git) @ 2015-02-25 16:51 UTC (permalink / raw)
  To: qemu-devel
  Cc: aarcange, yamahata, quintela, amit.shah, pbonzini, yanghy, david

From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

Hi,
  This is the 5th cut of my version of postcopy; it is designed for use with
the Linux kernel additions posted by Andrea Arcangeli here:

git clone --reference linux -b userfault16 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git

(Note this is a different API from the last version)

This qemu series can be found at:

https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v5 tag.

I believe I've fixed most of the issues from the reviews, but there are
probably a few more to cover before the final version:
   * I've got a rare (1/1000 migration) bug I'm fighting
   * The kernel API is still being discussed
   
v5
  New kernel API
     Uses an atomic copy rather than remap to avoid IPIs
  Fix builds on older machines
  Fix seg if block migration enabled (thanks to Gary Hook for reporting)
  Fix for mlock (thanks to zhanghailiang for reporting)
  Send postcopy-notify before sending the 'begin' part of device state
  Disable ballooning during postcopy
  Don't allocate zero pages as they migrate
  Stop using sysconf to get pagesizes
    - using getpagesize() and qemu_host_page_size in target dependent parts
  Fix for worst case page size ratio (untested)
     - ARM is 64k host, 1k target; with 32bit longs
  Drop the assertion check on migration_dirty_pages
     I don't like killing src vms anyway, and I've not seen it trigger in 9months+
  Free up migration_incoming-state at end of postcopy
  Calculate downtime on source differently for postcopy
  Use trace rather than dprintf
  Paolo's comments: 
    Rename postcopy_ram_sensitise_area
    Tidy up of loadvm loop flags
    Remove one layer of socket_shutdown abstraction
    Remove _RAM_ from many of the postcopy symbols as appropriate
      - most messages/states could now be used for other postcopy
        state.
    Used a qemu_event to wait for the end of the main thread finishing rather
       than spinning on postcopy state
    Rename inward/outward cmd messages for consistency
    Fixup old-vm-running check for postcopy

  Dave Gibson's comments:
    Lots of cleanup

  Eric's comments
    80col in qapi-schema


Dr. David Alan Gilbert (45):
  Start documenting how postcopy works.
  Split header writing out of qemu_save_state_begin
  qemu_ram_foreach_block: pass up error value, and down the ramblock
    name
  Add qemu_get_counted_string to read a string prefixed by a count byte
  Create MigrationIncomingState
  Provide runtime Target page information
  Return path: Open a return path on QEMUFile for sockets
  Return path: socket_writev_buffer: Block even on non-blocking fd's
  Migration commands
  Return path: Control commands
  Return path: Send responses from destination to source
  Return path: Source handling of return path
  ram_debug_dump_bitmap: Dump a migration bitmap as text
  Move loadvm_handlers into MigrationIncomingState
  Rework loadvm path for subloops
  Add migration-capability boolean for postcopy-ram.
  Add wrappers and handlers for sending/receiving the postcopy-ram
    migration messages.
  MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
  migrate_init: Call from savevm
  Modify savevm handlers for postcopy
  Add Linux userfaultfd header
  postcopy: OS support test
  migrate_start_postcopy: Command to trigger transition to postcopy
  MIG_STATE_POSTCOPY_ACTIVE: Add new migration state
  qemu_savevm_state_complete: Postcopy changes
  Postcopy page-map-incoming (PMI) structure
  Postcopy: Maintain sentmap and calculate discard
  postcopy: Incoming initialisation
  postcopy: ram_enable_notify to switch on userfault
  Postcopy: Postcopy startup in migration thread
  Postcopy end in migration_thread
  Page request:  Add MIG_RP_CMD_REQ_PAGES reverse command
  Page request: Process incoming page request
  Page request: Consume pages off the post-copy queue
  postcopy_ram.c: place_page and helpers
  Postcopy: Use helpers to map pages during migration
  qemu_ram_block_from_host
  Don't sync dirty bitmaps in postcopy
  Host page!=target page: Cleanup bitmaps
  Postcopy; Handle userfault requests
  Start up a postcopy/listener thread ready for incoming page data
  postcopy: Wire up loadvm_postcopy_handle_{run,end} commands
  End of migration for postcopy
  Disable mlock around incoming postcopy
  Inhibit ballooning during postcopy

 arch_init.c                       |  908 +++++++++++++++++++++++++++++++--
 balloon.c                         |   11 +
 docs/migration.txt                |  189 +++++++
 exec.c                            |   76 ++-
 hmp-commands.hx                   |   15 +
 hmp.c                             |    7 +
 hmp.h                             |    1 +
 hw/virtio/virtio-balloon.c        |    4 +-
 include/exec/cpu-all.h            |    2 -
 include/exec/cpu-common.h         |    8 +-
 include/migration/migration.h     |  145 +++++-
 include/migration/postcopy-ram.h  |   99 ++++
 include/migration/qemu-file.h     |   10 +
 include/migration/vmstate.h       |   12 +-
 include/qemu/typedefs.h           |    6 +
 include/sysemu/balloon.h          |    2 +
 include/sysemu/sysemu.h           |   46 +-
 linux-headers/linux/userfaultfd.h |  150 ++++++
 migration/Makefile.objs           |    2 +-
 migration/block.c                 |    7 +-
 migration/migration.c             |  739 +++++++++++++++++++++++++--
 migration/postcopy-ram.c          | 1018 +++++++++++++++++++++++++++++++++++++
 migration/qemu-file-internal.h    |    2 +
 migration/qemu-file-unix.c        |   99 +++-
 migration/qemu-file.c             |   28 +
 migration/rdma.c                  |    4 +-
 qapi-schema.json                  |   15 +-
 qmp-commands.hx                   |   19 +
 savevm.c                          |  866 ++++++++++++++++++++++++++++---
 trace-events                      |   80 ++-
 30 files changed, 4381 insertions(+), 189 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 linux-headers/linux/userfaultfd.h
 create mode 100644 migration/postcopy-ram.c

-- 
2.1.0

^ permalink raw reply	[flat|nested] 181+ messages in thread

end of thread, other threads:[~2015-06-16 10:50 UTC | newest]

Thread overview: 181+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-02-25 16:51 [Qemu-devel] [PATCH v5 00/45] Postcopy implementation Dr. David Alan Gilbert (git)
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works Dr. David Alan Gilbert (git)
2015-03-05  3:21   ` David Gibson
2015-03-05  9:21     ` Dr. David Alan Gilbert
2015-03-10  1:04       ` David Gibson
2015-03-13 13:07         ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 02/45] Split header writing out of qemu_save_state_begin Dr. David Alan Gilbert (git)
2015-03-10  1:05   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 03/45] qemu_ram_foreach_block: pass up error value, and down the ramblock name Dr. David Alan Gilbert (git)
2015-03-10 15:30   ` Eric Blake
2015-03-10 16:21     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 04/45] Add qemu_get_counted_string to read a string prefixed by a count byte Dr. David Alan Gilbert (git)
2015-03-10  1:12   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 05/45] Create MigrationIncomingState Dr. David Alan Gilbert (git)
2015-03-10  2:37   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 06/45] Provide runtime Target page information Dr. David Alan Gilbert (git)
2015-03-10  2:38   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 07/45] Return path: Open a return path on QEMUFile for sockets Dr. David Alan Gilbert (git)
2015-03-10  2:49   ` David Gibson
2015-03-13 13:14     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's Dr. David Alan Gilbert (git)
2015-03-10  2:56   ` David Gibson
2015-03-10 13:35     ` Dr. David Alan Gilbert
2015-03-11  1:51       ` David Gibson
2015-03-28 15:30   ` Paolo Bonzini
2015-03-29  4:07     ` David Gibson
2015-03-29  9:03       ` Paolo Bonzini
2015-03-30 16:50         ` Dr. David Alan Gilbert
2015-03-30 18:22           ` Markus Armbruster
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 09/45] Migration commands Dr. David Alan Gilbert (git)
2015-03-10  4:58   ` David Gibson
2015-03-10 11:04     ` Dr. David Alan Gilbert
2015-03-10 11:06       ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 10/45] Return path: Control commands Dr. David Alan Gilbert (git)
2015-03-10  5:40   ` David Gibson
2015-03-28 15:32   ` Paolo Bonzini
2015-03-30 17:34     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source Dr. David Alan Gilbert (git)
2015-03-10  5:47   ` David Gibson
2015-03-10 14:34     ` Dr. David Alan Gilbert
2015-03-11  1:54       ` David Gibson
2015-03-25 18:47         ` Dr. David Alan Gilbert
2015-03-28 15:34     ` Paolo Bonzini
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path Dr. David Alan Gilbert (git)
2015-03-10  6:08   ` David Gibson
2015-03-20 18:17     ` Dr. David Alan Gilbert
2015-03-23  2:37       ` David Gibson
2015-04-01 15:14         ` Dr. David Alan Gilbert
2015-04-07  3:07           ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 13/45] ram_debug_dump_bitmap: Dump a migration bitmap as text Dr. David Alan Gilbert (git)
2015-03-10  6:11   ` David Gibson
2015-03-20 18:48     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 14/45] Move loadvm_handlers into MigrationIncomingState Dr. David Alan Gilbert (git)
2015-03-10  6:19   ` David Gibson
2015-03-10 10:12     ` Dr. David Alan Gilbert
2015-03-10 11:03       ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 15/45] Rework loadvm path for subloops Dr. David Alan Gilbert (git)
2015-03-12  6:11   ` David Gibson
2015-04-14 12:04     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 16/45] Add migration-capability boolean for postcopy-ram Dr. David Alan Gilbert (git)
2015-03-12  6:14   ` David Gibson
2015-03-13 12:58     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages Dr. David Alan Gilbert (git)
2015-03-12  9:30   ` David Gibson
2015-03-26 16:33     ` Dr. David Alan Gilbert
2015-03-27  4:13       ` David Gibson
2015-03-27 10:48         ` Dr. David Alan Gilbert
2015-03-28 16:00           ` Paolo Bonzini
2015-03-30  4:03           ` David Gibson
2015-03-28 15:58         ` Paolo Bonzini
2015-03-28 15:43   ` Paolo Bonzini
2015-03-30 17:46     ` Dr. David Alan Gilbert
2015-03-30 19:23       ` Paolo Bonzini
2015-03-31 11:05         ` Dr. David Alan Gilbert
2015-03-31 11:10           ` Paolo Bonzini
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 18/45] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Dr. David Alan Gilbert (git)
2015-03-13  0:55   ` David Gibson
2015-03-13 11:51     ` Dr. David Alan Gilbert
2015-03-16  6:16       ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 19/45] migrate_init: Call from savevm Dr. David Alan Gilbert (git)
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy Dr. David Alan Gilbert (git)
2015-03-13  1:00   ` David Gibson
2015-03-13 10:19     ` Dr. David Alan Gilbert
2015-03-16  6:18       ` David Gibson
2015-03-20 12:37         ` Dr. David Alan Gilbert
2015-03-23  2:25           ` David Gibson
2015-03-24 20:04             ` Dr. David Alan Gilbert
2015-03-24 22:32               ` David Gibson
2015-03-25 15:00                 ` Dr. David Alan Gilbert
2015-03-25 16:40                   ` Dr. David Alan Gilbert
2015-03-26  1:35                     ` David Gibson
2015-03-26 11:44                       ` Dr. David Alan Gilbert
2015-03-27  3:56                         ` David Gibson
2015-03-26  1:35                   ` David Gibson
2015-03-30  8:10                 ` Paolo Bonzini
2015-03-31  0:10                   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 21/45] Add Linux userfaultfd header Dr. David Alan Gilbert (git)
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test Dr. David Alan Gilbert (git)
2015-03-13  1:23   ` David Gibson
2015-03-13 10:41     ` Dr. David Alan Gilbert
2015-03-16  6:22       ` David Gibson
2015-03-30  8:14       ` Paolo Bonzini
2015-03-30 14:07         ` Dr. David Alan Gilbert
2015-03-30 14:09           ` Paolo Bonzini
2015-06-16 10:49             ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy Dr. David Alan Gilbert (git)
2015-03-13  1:26   ` David Gibson
2015-03-13 11:19     ` Dr. David Alan Gilbert
2015-03-16  6:23       ` David Gibson
2015-03-18 17:59         ` Dr. David Alan Gilbert
2015-03-19  4:18           ` David Gibson
2015-03-19  9:33             ` Dr. David Alan Gilbert
2015-03-23  2:20               ` David Gibson
2015-03-30  8:19                 ` Paolo Bonzini
2015-03-30 17:04                   ` Dr. David Alan Gilbert
2015-03-30 19:22                     ` Paolo Bonzini
2015-03-31 11:21                       ` Dr. David Alan Gilbert
2015-03-30  8:17               ` Paolo Bonzini
2015-03-31  2:23                 ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 24/45] MIG_STATE_POSTCOPY_ACTIVE: Add new migration state Dr. David Alan Gilbert (git)
2015-03-13  4:45   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 25/45] qemu_savevm_state_complete: Postcopy changes Dr. David Alan Gilbert (git)
2015-03-13  4:58   ` David Gibson
2015-03-13 12:25     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 26/45] Postcopy page-map-incoming (PMI) structure Dr. David Alan Gilbert (git)
2015-03-13  5:19   ` David Gibson
2015-03-13 13:47     ` Dr. David Alan Gilbert
2015-03-16  6:30       ` David Gibson
2015-03-18 17:58         ` Dr. David Alan Gilbert
2015-03-23  2:48           ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 27/45] Postcopy: Maintain sentmap and calculate discard Dr. David Alan Gilbert (git)
2015-03-23  3:30   ` David Gibson
2015-03-23 14:36     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 28/45] postcopy: Incoming initialisation Dr. David Alan Gilbert (git)
2015-03-23  3:41   ` David Gibson
2015-03-23 13:46     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 29/45] postcopy: ram_enable_notify to switch on userfault Dr. David Alan Gilbert (git)
2015-03-23  3:45   ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 30/45] Postcopy: Postcopy startup in migration thread Dr. David Alan Gilbert (git)
2015-03-23  4:20   ` David Gibson
2015-03-26 11:05     ` Dr. David Alan Gilbert
2015-03-30  8:31       ` Paolo Bonzini
2015-04-13 11:35         ` Dr. David Alan Gilbert
2015-04-13 13:26           ` Paolo Bonzini
2015-04-13 14:58             ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 31/45] Postcopy end in migration_thread Dr. David Alan Gilbert (git)
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 32/45] Page request: Add MIG_RP_CMD_REQ_PAGES reverse command Dr. David Alan Gilbert (git)
2015-03-23  5:00   ` David Gibson
2015-03-25 18:16     ` Dr. David Alan Gilbert
2015-03-26  1:28       ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 33/45] Page request: Process incoming page request Dr. David Alan Gilbert (git)
2015-03-24  1:53   ` David Gibson
2015-03-25 17:37     ` Dr. David Alan Gilbert
2015-03-26  1:31       ` David Gibson
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 34/45] Page request: Consume pages off the post-copy queue Dr. David Alan Gilbert (git)
2015-03-24  2:15   ` David Gibson
2015-06-16 10:48     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 35/45] postcopy_ram.c: place_page and helpers Dr. David Alan Gilbert (git)
2015-03-24  2:33   ` David Gibson
2015-03-25 17:46     ` Dr. David Alan Gilbert
2015-02-25 16:51 ` [Qemu-devel] [PATCH v5 36/45] Postcopy: Use helpers to map pages during migration Dr. David Alan Gilbert (git)
2015-03-24  4:51   ` David Gibson
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 37/45] qemu_ram_block_from_host Dr. David Alan Gilbert (git)
2015-03-24  4:55   ` David Gibson
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 38/45] Don't sync dirty bitmaps in postcopy Dr. David Alan Gilbert (git)
2015-03-24  4:58   ` David Gibson
2015-03-24  9:05     ` Dr. David Alan Gilbert
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 39/45] Host page!=target page: Cleanup bitmaps Dr. David Alan Gilbert (git)
2015-03-24  5:23   ` David Gibson
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 40/45] Postcopy; Handle userfault requests Dr. David Alan Gilbert (git)
2015-03-24  5:38   ` David Gibson
2015-03-26 11:59     ` Dr. David Alan Gilbert
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 41/45] Start up a postcopy/listener thread ready for incoming page data Dr. David Alan Gilbert (git)
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 42/45] postcopy: Wire up loadvm_postcopy_handle_{run, end} commands Dr. David Alan Gilbert (git)
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 43/45] End of migration for postcopy Dr. David Alan Gilbert (git)
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 44/45] Disable mlock around incoming postcopy Dr. David Alan Gilbert (git)
2015-03-23  4:33   ` David Gibson
2015-02-25 16:52 ` [Qemu-devel] [PATCH v5 45/45] Inhibit ballooning during postcopy Dr. David Alan Gilbert (git)
2015-03-23  4:32   ` David Gibson
2015-03-23 12:21     ` Dr. David Alan Gilbert
2015-03-24  1:25       ` David Gibson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.