From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34190) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Xsoiu-0002jC-78 for qemu-devel@nongnu.org; Mon, 24 Nov 2014 03:10:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Xsoip-0001nC-O6 for qemu-devel@nongnu.org; Mon, 24 Nov 2014 03:10:16 -0500 Received: from szxga02-in.huawei.com ([119.145.14.65]:49793) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Xsoin-0001XN-Dy for qemu-devel@nongnu.org; Mon, 24 Nov 2014 03:10:11 -0500 Message-ID: <5472E7DB.8080304@huawei.com> Date: Mon, 24 Nov 2014 16:10:03 +0800 From: zhanghailiang MIME-Version: 1.0 References: <1412358473-31398-1-git-send-email-dgilbert@redhat.com> <546EB5F3.6020600@huawei.com> <20141121101359.GD3165@work-vm> In-Reply-To: <20141121101359.GD3165@work-vm> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v4 00/47] Postcopy implementation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, cristian.klein@cs.umu.se, qemu-devel@nongnu.org, amit.shah@redhat.com, yanghy@cn.fujitsu.com On 2014/11/21 18:14, Dr. David Alan Gilbert wrote: > * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >> Hi David, >> >> When i migrated VM in postcopy way when configuring VM with '-realtime mlock=on' option, >> It failed, and reports "postcopy_ram_hosttest: remap_anon_pages not available: File exists" in destination, >> >> Is it a bug of userfaultfd API? > > Thanks. > >> cc: Andrea >> >> reproduce Steps: >> Source: >> qemu-postcopy/qemu # x86_64-softmmu/qemu-system-x86_64 -msg timestamp=on \ >> -machine pc-i440fx-2.2,accel=kvm -m 1024 -realtime mlock=on -smp 4 \ >> -hda /mnt/sdb/pure_IMG/redhat/redhat-6.4-httpd.img -vnc :11 -monitor stdio >> >> Destination: >> qemu-postcopy/qemu # x86_64-softmmu/qemu-system-x86_64 -msg timestamp=on \ >> -machine pc-i440fx-2.2,accel=kvm -m 1024 -realtime mlock=on -smp 4 \ >> -hda /mnt/sdb/pure_IMG/redhat/redhat-6.4-httpd.img -vnc :12 -monitor stdio \ >> -incoming unix:/mnt/migrate.sock >> (1) migrate_set_capability x-postcopy-ram on >> (2) migrate -d unix:/mnt/migrate.sock >> >> In Destination, it fails, reports: >> savevm@2040988668 qemu_loadvm_state_main QEMU_VM_COMMAND ret: 0 >> savevm@2040988668 qemu_loadvm_state loop: section_type=6 >> savevm@2040988668 loadvm_postcopy_ram_handle_advise >> postcopy_ram_hosttest: remap_anon_pages not available: File exists >> savevm@2040988668 qemu_loadvm_state_main QEMU_VM_COMMAND ret: -1 > > Yes, I think I need to chat to Andrea about how that's supposed to work with mlock. > I've added it to my list and we'll figure it out; I suspect on the destination > I need to avoid doing the mlockall until after postcopy completes. > >> And one more thing, i want to know: ;) >> Why we must start precopy first before start postcopy? >> Can we do postcopy at the beginning of migration? > > You can send migrate_start_postcopy immediately after you send the migrate > command, which is very close to no-precopy; the original API had a timeout > and if you set it to 0 then it would do exactly no-precopy, but the current API > was preferred by reviewers, and is simpler. > With testing, the best performance is from doing one full pass of precopy and > then starting postcopy; that way all of the kernel and other static stuff > has already moved to the destination, and there are much fewer page requests. > Got it, :) Thanks. > Thanks for the report, > > Dave > >> >> Thanks, >> zhanghailiang >> >> On 2014/10/4 1:47, Dr. David Alan Gilbert (git) wrote: >>> From: "Dr. David Alan Gilbert" >>> >>> Hi, >>> This is the 4th cut of my version of postcopy; it is designed for use with >>> the Linux kernel additions just posted by Andrea Arcangeli here: >>> >>> http://marc.info/?l=linux-kernel&m=141235633015100&w=2 >>> >>> (Note: This is a new version compared to my previous postcopy patchset; you'll >>> need to update the kernel to the new version.) >>> >>> Other than the new kernel ABI (which is only a small change to the userspace side); >>> the major changes are; >>> >>> a) Code for host page size != target page size >>> b) Support for migration over fd >>> From Cristian Klein; this is for libvirt support which Cristian recently >>> posted to the libvirt list. >>> c) It's now build bisectable and builds on 32bit >>> >>> Testing wise; I've now done many thousand of postcopy migrations without >>> failure (both of idle and busy guests); so it seems pretty solid. >>> >>> Must-TODO's: >>> 1) A partially repeatable migration_cancel failure >>> 2) virt_test's migrate.with_reboot test is failing >>> 3) The ACPI fix in 2.1 that allowed migrating RAMBlocks to be larger than >>> the source feels like it needs looking at for postcopy. >>> 4) Paolo's comments with respect to the wakeup_request/is_running code >>> in the migration thread >>> 5) xbzrle needs disabling once in postcopy >>> >>> Later-TODO's: >>> 1) Control the rate of background page transfers during postcopy to >>> reduce their impact on the latency of postcopy requests. >>> 2) Work with RDMA >>> 3) Could destination RP be made blocking (as per discussion with Paolo; >>> I'm still worried that that changes too many assumptions) >>> >>> >>> >>> V4: >>> Initial support for host page size != target page size >>> - tested heavily on hps==tps >>> - only partially tested on hps!=tps systems >>> - This involved quite a bit of rework around the discard code >>> Updated to new kernel userfault ABI >>> - It won't work with the previous version >>> Fix mis-optimisation of postcopy request for wrong RAMBlock >>> request for block A offset n >>> un-needed fault for block B/m (already received - no req sent) >>> request for block B/l - wrongly sent as request for A/l >>> Fix thinko in discard bitmap processing (missed last word of bitmap) >>> Symptom: remap failures near the top of RAM if postcopy started late >>> Fix bug that caused kernel page acknowledgments to be misaligned >>> May have meant the guest was paused for longer than required >>> Fix potential for crashing cleaning up failed RP >>> Fixes in docs (from Yang) >>> Handle migration by fd as sockets if they are sockets >>> Build tested on 32bit >>> Fully build bisectable (x86-64) >>> >>> >>> Dave >>> >>> Cristian Klein (1): >>> Handle bi-directional communication for fd migration >>> >>> Dr. David Alan Gilbert (46): >>> QEMUSizedBuffer based QEMUFile >>> Tests: QEMUSizedBuffer/QEMUBuffer >>> Start documenting how postcopy works. >>> qemu_ram_foreach_block: pass up error value, and down the ramblock >>> name >>> improve DPRINTF macros, add to savevm >>> Add qemu_get_counted_string to read a string prefixed by a count byte >>> Create MigrationIncomingState >>> socket shutdown >>> Provide runtime Target page information >>> Return path: Open a return path on QEMUFile for sockets >>> Return path: socket_writev_buffer: Block even on non-blocking fd's >>> Migration commands >>> Return path: Control commands >>> Return path: Send responses from destination to source >>> Return path: Source handling of return path >>> qemu_loadvm errors and debug >>> ram_debug_dump_bitmap: Dump a migration bitmap as text >>> Rework loadvm path for subloops >>> Add migration-capability boolean for postcopy-ram. >>> Add wrappers and handlers for sending/receiving the postcopy-ram >>> migration messages. >>> QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream >>> migrate_init: Call from savevm >>> Allow savevm handlers to state whether they could go into postcopy >>> postcopy: OS support test >>> migrate_start_postcopy: Command to trigger transition to postcopy >>> MIG_STATE_POSTCOPY_ACTIVE: Add new migration state >>> qemu_savevm_state_complete: Postcopy changes >>> Postcopy page-map-incoming (PMI) structure >>> Postcopy: Maintain sentmap and calculate discard >>> postcopy: Incoming initialisation >>> postcopy: ram_enable_notify to switch on userfault >>> Postcopy: Postcopy startup in migration thread >>> Postcopy: Create a fault handler thread before marking the ram as >>> userfault >>> Page request: Add MIG_RPCOMM_REQPAGES reverse command >>> Page request: Process incoming page request >>> Page request: Consume pages off the post-copy queue >>> Add assertion to check migration_dirty_pages >>> postcopy_ram.c: place_page and helpers >>> Postcopy: Use helpers to map pages during migration >>> qemu_ram_block_from_host >>> Don't sync dirty bitmaps in postcopy >>> Host page!=target page: Cleanup bitmaps >>> Postcopy; Handle userfault requests >>> Start up a postcopy/listener thread ready for incoming page data >>> postcopy: Wire up loadvm_postcopy_ram_handle_{run,end} commands >>> End of migration for postcopy >>> >>> Makefile.objs | 2 +- >>> arch_init.c | 739 +++++++++++++++++++++++++-- >>> docs/migration.txt | 189 +++++++ >>> exec.c | 76 ++- >>> hmp-commands.hx | 15 + >>> hmp.c | 7 + >>> hmp.h | 1 + >>> include/exec/cpu-common.h | 8 +- >>> include/migration/migration.h | 130 +++++ >>> include/migration/postcopy-ram.h | 106 ++++ >>> include/migration/qemu-file.h | 47 ++ >>> include/migration/vmstate.h | 2 +- >>> include/qemu/sockets.h | 1 + >>> include/qemu/typedefs.h | 9 +- >>> include/sysemu/sysemu.h | 43 +- >>> migration-fd.c | 24 +- >>> migration-rdma.c | 4 +- >>> migration.c | 693 +++++++++++++++++++++++++- >>> postcopy-ram.c | 1016 ++++++++++++++++++++++++++++++++++++++ >>> qapi-schema.json | 14 +- >>> qemu-file.c | 598 +++++++++++++++++++++- >>> qmp-commands.hx | 19 + >>> savevm.c | 881 +++++++++++++++++++++++++++++++-- >>> tests/Makefile | 2 +- >>> tests/test-vmstate.c | 74 +-- >>> util/qemu-sockets.c | 28 ++ >>> 26 files changed, 4550 insertions(+), 178 deletions(-) >>> create mode 100644 include/migration/postcopy-ram.h >>> create mode 100644 postcopy-ram.c >>> >> >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > . >