From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
To: David Hildenbrand <david@redhat.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Juan Quintela <quintela@redhat.com>,
Richard Henderson <richard.henderson@linaro.org>,
qemu-devel@nongnu.org, Peter Xu <peterx@redhat.com>,
Alex Williamson <alex.williamson@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating
Date: Tue, 11 May 2021 10:42:07 +0100 [thread overview]
Message-ID: <YJpRb5lrMYD5fLYq@work-vm> (raw)
In-Reply-To: <20210429112708.12291-1-david@redhat.com>
* David Hildenbrand (david@redhat.com) wrote:
> v4 has been floating around for a while. Let's see if we can find someone
> to merge this; or at least give some more feedback ... all patches have
> at least one RB.
>
>
> I realized that resizing RAM blocks while the guest is being migrated
> (precopy: resize while still running on the source, postcopy: resize
> while already running on the target) is buggy. In case of precopy, we
> can simply cancel migration. Postcopy handling is more involved. Resizing
> can currently happen during a guest reboot, triggered by ACPI rebuilds.
>
> Along with the fixes, some cleanups.
Queued
>
> --------------------------------------------------------------------------
>
> Example to highlight one part of the problem:
>
> 1. Start a paused VM (where a ramblock resize will trigger when booting):
> sudo build/qemu-system-x86_64 \
> --enable-kvm \
> -S \
> -machine q35,nvdimm=on \
> -smp 1 \
> -cpu host \
> -m size=20G,slots=8,maxmem=22G \
> -object memory-backend-file,id=mem0,mem-path=/tmp/nvdimm,size=256M \
> -device nvdimm,label-size=131072,memdev=mem0,id=nvdimm0,slot=1 \
> -nodefaults \
> -chardev stdio,nosignal,id=serial \
> -device isa-serial,chardev=serial \
> -chardev socket,id=monitor,path=/var/tmp/monitor,server,nowait \
> -mon chardev=monitor,mode=readline \
> -device vmgenid \
> -device intel-iommu \
> -nographic
>
> 2. Starting precopy and then starting the VM to trigger resizing during
> precopy:
> QEMU 5.2.95 monitor - type 'help' for more information
> (qemu) migrate -d "exec:gzip -c > STATEFILE.gz"
> QEMU 5.2.95 monitor - type 'help' for more information
> (qemu) cont
>
> 3a. Before this series, migration never completes:
> QEMU 5.2.95 monitor - type 'help' for more information
> (qemu) info migrate
> globals:
> store-global-state: on
> only-migratable: off
> send-configuration: on
> send-section-footer: on
> decompress-error-check: on
> clear-bitmap-shift: 18
> Migration status: active
> total time: 43826 ms
> expected downtime: 300 ms
> setup: 5 ms
> transferred ram: 65981 kbytes
> throughput: 8.27 mbps
> remaining ram: 18446744073709551612 kbytes
> total ram: 21234188 kbytes
> duplicate: 5308454 pages
> skipped: 0 pages
> normal: 93 pages
> normal bytes: 372 kbytes
> dirty sync count: 1
> page size: 4 kbytes
> multifd bytes: 0 kbytes
> pages-per-second: 0
>
> 4. With this change, migration is properly aborted:
> (qemu) info migrate
> globals:
> store-global-state: on
> only-migratable: off
> send-configuration: on
> send-section-footer: on
> decompress-error-check: on
> clear-bitmap-shift: 18
> Migration status: cancelled
> total time: 0 ms
>
> --------------------------------------------------------------------------
>
> Cc: Eduardo Habkost <ehabkost@redhat.com>
> Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Juan Quintela <quintela@redhat.com>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: Peter Xu <peterx@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
>
> v4 -> v5:
> - Rephrased some patch descriptions
> - Dropped some patches to reduce the footprint
> -- "stubs/ram-block: Remove stubs that are no longer needed"
> -- "migration/ram: Tolerate partially changed mappings in postcopy code"
> - Removed as already upstream now
> -- "migration/ram: Consolidate variable reset after placement in
> ram_load_postcopy()"
>
> v3 -> v4:
> - Rebased and retested
> - Added RBs
>
> v2 -> v3:
> - Rebased on current master
> - Added RBs
> - "migration/ram: Tolerate partially changed mappings in postcopy code"
> -- Extended the comment for the uffdio unregister part.
>
> v1 -> v2:
> - "util: vfio-helpers: Factor out and fix processing of existing ram
> blocks"
> -- Stringify error
> - "migraton/ram: Handle RAM block resizes during precopy"
> -- Simplified check if we're migrating on the source
> - "exec: Relax range check in ram_block_discard_range()"
> -- Added to make discard during resizes actually work
> - "migration/ram: Discard new RAM when growing RAM blocks after
> ram_postcopy_incoming_init()"
> -- Better checks if in the right postcopy mode.
> -- Better patch subject/description/comments
> - "migration/ram: Handle RAM block resizes during postcopy"
> -- Better comments
> -- Adapt to changed postcopy checks
> - "migrate/ram: Get rid of "place_source" in ram_load_postcopy()"
> -- Dropped, as broken
> - "migration/ram: Tolerate partially changed mappings in postcopy code"
> -- Better comment / description. Clarify that no implicit wakeup will
> happen
> -- Warn on EINVAL (older kernels)
> -- Wake up any waiter explicitly
>
> David Hildenbrand (10):
> util: vfio-helpers: Factor out and fix processing of existing ram
> blocks
> numa: Teach ram block notifiers about resizeable ram blocks
> numa: Make all callbacks of ram block notifiers optional
> migration/ram: Handle RAM block resizes during precopy
> exec: Relax range check in ram_block_discard_range()
> migration/ram: Discard RAM when growing RAM blocks after
> ram_postcopy_incoming_init()
> migration/ram: Simplify host page handling in ram_load_postcopy()
> migration/ram: Handle RAM block resizes during postcopy
> migration/multifd: Print used_length of memory block
> migration/ram: Use offset_in_ramblock() in range checks
>
> hw/core/numa.c | 41 +++++++++--
> hw/i386/xen/xen-mapcache.c | 7 +-
> include/exec/cpu-common.h | 1 +
> include/exec/memory.h | 10 +--
> include/exec/ramblock.h | 10 +++
> include/exec/ramlist.h | 13 ++--
> migration/migration.c | 9 ++-
> migration/migration.h | 1 +
> migration/multifd.c | 2 +-
> migration/postcopy-ram.c | 15 ++++-
> migration/ram.c | 135 +++++++++++++++++++++++++++++--------
> softmmu/physmem.c | 26 +++++--
> target/i386/hax/hax-mem.c | 5 +-
> target/i386/sev.c | 18 ++---
> util/vfio-helpers.c | 41 ++++-------
> 15 files changed, 241 insertions(+), 93 deletions(-)
>
> --
> 2.30.2
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
prev parent reply other threads:[~2021-05-11 9:48 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-29 11:26 [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating David Hildenbrand
2021-04-29 11:26 ` [PATCH v5 01/10] util: vfio-helpers: Factor out and fix processing of existing ram blocks David Hildenbrand
2021-04-29 13:13 ` Philippe Mathieu-Daudé
2021-04-29 11:27 ` [PATCH v5 02/10] numa: Teach ram block notifiers about resizeable " David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 03/10] numa: Make all callbacks of ram block notifiers optional David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 04/10] migration/ram: Handle RAM block resizes during precopy David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 05/10] exec: Relax range check in ram_block_discard_range() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 06/10] migration/ram: Discard RAM when growing RAM blocks after ram_postcopy_incoming_init() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 07/10] migration/ram: Simplify host page handling in ram_load_postcopy() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 08/10] migration/ram: Handle RAM block resizes during postcopy David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 09/10] migration/multifd: Print used_length of memory block David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 10/10] migration/ram: Use offset_in_ramblock() in range checks David Hildenbrand
2021-05-06 17:57 ` [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating Eduardo Habkost
2021-05-06 18:04 ` Dr. David Alan Gilbert
2021-05-11 9:42 ` Dr. David Alan Gilbert [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YJpRb5lrMYD5fLYq@work-vm \
--to=dgilbert@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=david@redhat.com \
--cc=ehabkost@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=richard.henderson@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).