From: David Hildenbrand <david@redhat.com>
To: qemu-devel@nongnu.org
Cc: Eduardo Habkost <ehabkost@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
David Hildenbrand <david@redhat.com>,
Richard Henderson <richard.henderson@linaro.org>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
Peter Xu <peterx@redhat.com>, Juan Quintela <quintela@redhat.com>,
Alex Williamson <alex.williamson@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>
Subject: [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating
Date: Thu, 29 Apr 2021 13:26:58 +0200 [thread overview]
Message-ID: <20210429112708.12291-1-david@redhat.com> (raw)
v4 has been floating around for a while. Let's see if we can find someone
to merge this; or at least give some more feedback ... all patches have
at least one RB.
I realized that resizing RAM blocks while the guest is being migrated
(precopy: resize while still running on the source, postcopy: resize
while already running on the target) is buggy. In case of precopy, we
can simply cancel migration. Postcopy handling is more involved. Resizing
can currently happen during a guest reboot, triggered by ACPI rebuilds.
Along with the fixes, some cleanups.
--------------------------------------------------------------------------
Example to highlight one part of the problem:
1. Start a paused VM (where a ramblock resize will trigger when booting):
sudo build/qemu-system-x86_64 \
--enable-kvm \
-S \
-machine q35,nvdimm=on \
-smp 1 \
-cpu host \
-m size=20G,slots=8,maxmem=22G \
-object memory-backend-file,id=mem0,mem-path=/tmp/nvdimm,size=256M \
-device nvdimm,label-size=131072,memdev=mem0,id=nvdimm0,slot=1 \
-nodefaults \
-chardev stdio,nosignal,id=serial \
-device isa-serial,chardev=serial \
-chardev socket,id=monitor,path=/var/tmp/monitor,server,nowait \
-mon chardev=monitor,mode=readline \
-device vmgenid \
-device intel-iommu \
-nographic
2. Starting precopy and then starting the VM to trigger resizing during
precopy:
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) migrate -d "exec:gzip -c > STATEFILE.gz"
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) cont
3a. Before this series, migration never completes:
QEMU 5.2.95 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 43826 ms
expected downtime: 300 ms
setup: 5 ms
transferred ram: 65981 kbytes
throughput: 8.27 mbps
remaining ram: 18446744073709551612 kbytes
total ram: 21234188 kbytes
duplicate: 5308454 pages
skipped: 0 pages
normal: 93 pages
normal bytes: 372 kbytes
dirty sync count: 1
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 0
4. With this change, migration is properly aborted:
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: cancelled
total time: 0 ms
--------------------------------------------------------------------------
Cc: Eduardo Habkost <ehabkost@redhat.com>
Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Juan Quintela <quintela@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
v4 -> v5:
- Rephrased some patch descriptions
- Dropped some patches to reduce the footprint
-- "stubs/ram-block: Remove stubs that are no longer needed"
-- "migration/ram: Tolerate partially changed mappings in postcopy code"
- Removed as already upstream now
-- "migration/ram: Consolidate variable reset after placement in
ram_load_postcopy()"
v3 -> v4:
- Rebased and retested
- Added RBs
v2 -> v3:
- Rebased on current master
- Added RBs
- "migration/ram: Tolerate partially changed mappings in postcopy code"
-- Extended the comment for the uffdio unregister part.
v1 -> v2:
- "util: vfio-helpers: Factor out and fix processing of existing ram
blocks"
-- Stringify error
- "migraton/ram: Handle RAM block resizes during precopy"
-- Simplified check if we're migrating on the source
- "exec: Relax range check in ram_block_discard_range()"
-- Added to make discard during resizes actually work
- "migration/ram: Discard new RAM when growing RAM blocks after
ram_postcopy_incoming_init()"
-- Better checks if in the right postcopy mode.
-- Better patch subject/description/comments
- "migration/ram: Handle RAM block resizes during postcopy"
-- Better comments
-- Adapt to changed postcopy checks
- "migrate/ram: Get rid of "place_source" in ram_load_postcopy()"
-- Dropped, as broken
- "migration/ram: Tolerate partially changed mappings in postcopy code"
-- Better comment / description. Clarify that no implicit wakeup will
happen
-- Warn on EINVAL (older kernels)
-- Wake up any waiter explicitly
David Hildenbrand (10):
util: vfio-helpers: Factor out and fix processing of existing ram
blocks
numa: Teach ram block notifiers about resizeable ram blocks
numa: Make all callbacks of ram block notifiers optional
migration/ram: Handle RAM block resizes during precopy
exec: Relax range check in ram_block_discard_range()
migration/ram: Discard RAM when growing RAM blocks after
ram_postcopy_incoming_init()
migration/ram: Simplify host page handling in ram_load_postcopy()
migration/ram: Handle RAM block resizes during postcopy
migration/multifd: Print used_length of memory block
migration/ram: Use offset_in_ramblock() in range checks
hw/core/numa.c | 41 +++++++++--
hw/i386/xen/xen-mapcache.c | 7 +-
include/exec/cpu-common.h | 1 +
include/exec/memory.h | 10 +--
include/exec/ramblock.h | 10 +++
include/exec/ramlist.h | 13 ++--
migration/migration.c | 9 ++-
migration/migration.h | 1 +
migration/multifd.c | 2 +-
migration/postcopy-ram.c | 15 ++++-
migration/ram.c | 135 +++++++++++++++++++++++++++++--------
softmmu/physmem.c | 26 +++++--
target/i386/hax/hax-mem.c | 5 +-
target/i386/sev.c | 18 ++---
util/vfio-helpers.c | 41 ++++-------
15 files changed, 241 insertions(+), 93 deletions(-)
--
2.30.2
next reply other threads:[~2021-04-29 11:28 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-29 11:26 David Hildenbrand [this message]
2021-04-29 11:26 ` [PATCH v5 01/10] util: vfio-helpers: Factor out and fix processing of existing ram blocks David Hildenbrand
2021-04-29 13:13 ` Philippe Mathieu-Daudé
2021-04-29 11:27 ` [PATCH v5 02/10] numa: Teach ram block notifiers about resizeable " David Hildenbrand
2021-04-29 11:27 ` David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 03/10] numa: Make all callbacks of ram block notifiers optional David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 04/10] migration/ram: Handle RAM block resizes during precopy David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 05/10] exec: Relax range check in ram_block_discard_range() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 06/10] migration/ram: Discard RAM when growing RAM blocks after ram_postcopy_incoming_init() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 07/10] migration/ram: Simplify host page handling in ram_load_postcopy() David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 08/10] migration/ram: Handle RAM block resizes during postcopy David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 09/10] migration/multifd: Print used_length of memory block David Hildenbrand
2021-04-29 11:27 ` [PATCH v5 10/10] migration/ram: Use offset_in_ramblock() in range checks David Hildenbrand
2021-05-06 17:57 ` [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating Eduardo Habkost
2021-05-06 18:04 ` Dr. David Alan Gilbert
2021-05-11 9:42 ` Dr. David Alan Gilbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210429112708.12291-1-david@redhat.com \
--to=david@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=dgilbert@redhat.com \
--cc=ehabkost@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=richard.henderson@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.