From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60526) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X37Uw-0007HQ-89 for qemu-devel@nongnu.org; Fri, 04 Jul 2014 13:42:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1X37Ur-0008BS-CO for qemu-devel@nongnu.org; Fri, 04 Jul 2014 13:42:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43423) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X37Ur-0008BK-3Q for qemu-devel@nongnu.org; Fri, 04 Jul 2014 13:42:05 -0400 From: "Dr. David Alan Gilbert (git)" Date: Fri, 4 Jul 2014 18:41:11 +0100 Message-Id: <1404495717-4239-1-git-send-email-dgilbert@redhat.com> Subject: [Qemu-devel] [PATCH 00/46] Postcopy implementation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, lilei@linux.vnet.ibm.com, quintela@redhat.com From: "Dr. David Alan Gilbert" Hi, This is the 1st cut of my version of postcopy; it is designed for use with the Linux kernel additions recently posted by Andrea Arcangeli here: http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00525.html The current status is: 1) It works - I've done testing on large/busy VMs using google-stress app test, and it completes cleanly even over a 1Gbps link. 2) It's still rather rough around the corner cases and exit code, so it's probably not ready for review yet, but I thought I'd put it out for people working in the area and for people to see the general shape. I've taken some ideas from Isaku Yamahata's work, but the code is structured a bit differently, however I'm greatful for the understanding I got from Isaku's code of the problems postcopy has to deal with. Note, the last patch in the series is a chunk of documentation, which it's probably useful to read for detail. Some points: a) I've tried to keep the pieces I've added general rather than postcopy specific. b) Precopy is done using the main precopy code (with very few modifications) and it automatically switches to postcopy after a timeout. c) A 'command' section type is added that controls the progress of postcopy and can be used for anything else people need to control that's not actually guest state. d) There is a 'return path' mechanism for sending data from the destination back to the source, postcopy uses this for the page requests, but I think it's also useful for things like the failover implementations, and can be used to signal failure from the destination to the source. e) I've added a 'migration_set_parameter' command as somewhere to put integer parameters associated with migration. e.1) And I use that initially for the length of precopy to try. f) I've added a 'MigrationIncomingState' where postcopy keeps it's data, it seems a good idea to move other incoming state there as well. c & d together provide a 'reqack' that's basically a ping, quite useful for debugging Current TODO: 1) It's not bisectable yet 2) There are no testsuite additions (although I have a virt-test modification I've been using). 3) End-of-migration cleanup is unfinished, as is some of the error handling. 4) Not all the code is there for systems with hostpagesize!=qemupagesize 5) xbzrle needs disabling once in postcopy 6) RDMA needs some rework 7) I moved QEMUFile into a public header, it's a bad idea and I'll undo that. 8) My latency measurements on page requests are generally pretty good, but I'm seeing a few high spikes, I need to investigate. 9) Andrea has suggestions on ways to avoid some of the huge-page splitting that occurs during the discard phase after precopy. 10) I'd like to format the data on the return path in a more structured way (i.e. maybe using stuff from my BER world). I'm hoping to get that lot cleared up by 2.2 How to use: migrate_set_capability x-postcopy-ram on migrate_set_parameter x-postcopy-start-time 500 <--- that's in ms migrate -d tcp:whereever:port Other ideas: I think the returnpath+command could be used to send qmp commands to the destination to allow the src/destination to coordinate a hotplug or to set up the destination state. Dave Dr. David Alan Gilbert (46): qemu_ram_foreach_block: pass up error value, and down the ramblock name Move QEMUFile structure to qemu-file.h QEMUSizedBuffer/QEMUFile improve DPRINTF macros, add to savevm Add qemu_get_counted_string to read a string prefixed by a count byte Create MigrationIncomingState Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands Return path: Control commands Return path: Send responses from destination to source Return path: Source handling of return path qemu_loadvm debug ram_debug_dump_bitmap: Dump a migration bitmap as text Rework loadvm path for subloops Add migration-capability boolean for postcopy-ram. Add wrappers and handlers for sending/receiving the postcopy-ram migration messages. QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream migrate_init: Call from savevm Allow savevm handlers to state whether they could go into postcopy postcopy: OS support test Migration parameters: Add qmp/hmp commands for setting/viewing MIG_STATE_POSTCOPY_ACTIVE: Add new migration state qemu_savevm_state_complete: Postcopy changes Postcopy: Maintain sentmap during postcopy pre phase Postcopy page-map-incoming (PMI) structure postcopy: Add incoming_init/cleanup functions postcopy: Incoming initialisation postcopy: ram_enable_notify to switch on userfault Postcopy: postcopy_start Postcopy: Rework migration thread for postcopy mode mig fd_connect: open return path Postcopy: Create a fault handler thread before marking the ram as userfault Page request: Add MIG_RPCOMM_REQPAGES reverse command Page request: Process incoming page request Page request: Consume pages off the post-copy queue Add assertion to check migration_dirty_pages doesn't go -ve; have seen it happen once but not sure why postcopy_ram.c: place_page and helpers Postcopy: Use helpers to map pages during migration qemu_ram_block_from_host Handle userfault requests (although userfaultfd not done yet) Start up a postcopy/listener thread ready for incoming page data postcopy: Wire up loadvm_postcopy_ram_handle_{run,end} commands postcopy: Use userfaultfd End of migration for postcopy Start documenting how postcopy works. Makefile.objs | 2 +- arch_init.c | 477 ++++++++++++++++++++-- docs/migration.txt | 148 +++++++ exec.c | 60 ++- hmp-commands.hx | 17 + hmp.c | 54 +++ hmp.h | 4 + include/exec/cpu-common.h | 7 +- include/migration/migration.h | 115 ++++++ include/migration/postcopy-ram.h | 83 ++++ include/migration/qemu-file.h | 62 +++ include/migration/vmstate.h | 2 +- include/qemu/typedefs.h | 8 +- include/sysemu/sysemu.h | 41 +- migration-rdma.c | 4 +- migration.c | 695 ++++++++++++++++++++++++++++++-- monitor.c | 25 ++ postcopy-ram.c | 833 +++++++++++++++++++++++++++++++++++++++ qapi-schema.json | 56 ++- qemu-file.c | 578 +++++++++++++++++++++++++-- qmp-commands.hx | 23 ++ savevm.c | 822 +++++++++++++++++++++++++++++++++++--- 22 files changed, 3933 insertions(+), 183 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 postcopy-ram.c -- 1.9.3