From: David Hildenbrand <david@redhat.com> To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson <rth@twiddle.net>, Paolo Bonzini <pbonzini@redhat.com>, "Dr . David Alan Gilbert" <dgilbert@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com> Subject: [PATCH v3 20/20] virtio-mem: Exclude unplugged memory during migration Date: Wed, 3 Jun 2020 16:49:14 +0200 [thread overview] Message-ID: <20200603144914.41645-21-david@redhat.com> (raw) In-Reply-To: <20200603144914.41645-1-david@redhat.com> The content of unplugged memory is undefined and should not be migrated, ever. Exclude all unplugged memory during precopy using the precopy notifier infrastructure introduced for free page hinting in virtio-balloon. Unplugged memory is marked as "not dirty", meaning it won't be considered for migration. Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Signed-off-by: David Hildenbrand <david@redhat.com> --- hw/virtio/virtio-mem.c | 54 +++++++++++++++++++++++++++++++++- include/hw/virtio/virtio-mem.h | 3 ++ 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index 4d0a2e78c0..e278b213e2 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -62,8 +62,14 @@ static bool virtio_mem_is_busy(void) /* * Postcopy cannot handle concurrent discards and we don't want to migrate * pages on-demand with stale content when plugging new blocks. + * + * For precopy, we don't want unplugged blocks in our migration stream, and + * when plugging new blocks, the page content might differ between source + * and destination (observable by the guest when not initializing pages + * after plugging them) until we're running on the destination (as we didn't + * migrate these blocks when they were unplugged). */ - return migration_in_incoming_postcopy(); + return migration_in_incoming_postcopy() || !migration_is_idle(); } static bool virtio_mem_test_bitmap(VirtIOMEM *vmem, uint64_t start_gpa, @@ -475,6 +481,7 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp) host_memory_backend_set_mapped(vmem->memdev, true); vmstate_register_ram(&vmem->memdev->mr, DEVICE(vmem)); qemu_register_reset(virtio_mem_system_reset, vmem); + precopy_add_notifier(&vmem->precopy_notifier); } static void virtio_mem_device_unrealize(DeviceState *dev) @@ -482,6 +489,7 @@ static void virtio_mem_device_unrealize(DeviceState *dev) VirtIODevice *vdev = VIRTIO_DEVICE(dev); VirtIOMEM *vmem = VIRTIO_MEM(dev); + precopy_remove_notifier(&vmem->precopy_notifier); qemu_unregister_reset(virtio_mem_system_reset, vmem); vmstate_unregister_ram(&vmem->memdev->mr, DEVICE(vmem)); host_memory_backend_set_mapped(vmem->memdev, false); @@ -760,12 +768,56 @@ static void virtio_mem_set_block_size(Object *obj, Visitor *v, const char *name, vmem->block_size = value; } +static void virtio_mem_precopy_exclude_unplugged(VirtIOMEM *vmem) +{ + void * const host = qemu_ram_get_host_addr(vmem->memdev->mr.ram_block); + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, length; + + /* + * Find consecutive unplugged blocks and exclude them from migration. + * + * Note: Blocks cannot get (un)plugged during precopy, no locking needed. + */ + first_zero_bit = find_first_zero_bit(vmem->bitmap, vmem->bitmap_size); + while (first_zero_bit < vmem->bitmap_size) { + offset = first_zero_bit * vmem->block_size; + last_zero_bit = find_next_bit(vmem->bitmap, vmem->bitmap_size, + first_zero_bit + 1) - 1; + length = (last_zero_bit - first_zero_bit + 1) * vmem->block_size; + + qemu_guest_free_page_hint(host + offset, length); + first_zero_bit = find_next_zero_bit(vmem->bitmap, vmem->bitmap_size, + last_zero_bit + 2); + } +} + +static int virtio_mem_precopy_notify(NotifierWithReturn *n, void *data) +{ + VirtIOMEM *vmem = container_of(n, VirtIOMEM, precopy_notifier); + PrecopyNotifyData *pnd = data; + + switch (pnd->reason) { + case PRECOPY_NOTIFY_SETUP: + precopy_enable_free_page_optimization(); + break; + case PRECOPY_NOTIFY_AFTER_BITMAP_SYNC: + virtio_mem_precopy_exclude_unplugged(vmem); + break; + default: + break; + } + + return 0; +} + static void virtio_mem_instance_init(Object *obj) { VirtIOMEM *vmem = VIRTIO_MEM(obj); vmem->block_size = VIRTIO_MEM_MIN_BLOCK_SIZE; notifier_list_init(&vmem->size_change_notifiers); + vmem->precopy_notifier.notify = virtio_mem_precopy_notify; object_property_add(obj, VIRTIO_MEM_SIZE_PROP, "size", virtio_mem_get_size, NULL, NULL, NULL); diff --git a/include/hw/virtio/virtio-mem.h b/include/hw/virtio/virtio-mem.h index 408a6ede50..ddb3822375 100644 --- a/include/hw/virtio/virtio-mem.h +++ b/include/hw/virtio/virtio-mem.h @@ -67,6 +67,9 @@ typedef struct VirtIOMEM { /* notifiers to notify when "size" changes */ NotifierList size_change_notifiers; + + /* don't migrate unplugged memory */ + NotifierWithReturn precopy_notifier; } VirtIOMEM; typedef struct VirtIOMEMClass { -- 2.25.4
WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com> To: qemu-devel@nongnu.org Cc: Eduardo Habkost <ehabkost@redhat.com>, kvm@vger.kernel.org, "Michael S . Tsirkin" <mst@redhat.com>, David Hildenbrand <david@redhat.com>, "Dr . David Alan Gilbert" <dgilbert@redhat.com>, qemu-s390x@nongnu.org, Paolo Bonzini <pbonzini@redhat.com>, Richard Henderson <rth@twiddle.net> Subject: [PATCH v3 20/20] virtio-mem: Exclude unplugged memory during migration Date: Wed, 3 Jun 2020 16:49:14 +0200 [thread overview] Message-ID: <20200603144914.41645-21-david@redhat.com> (raw) In-Reply-To: <20200603144914.41645-1-david@redhat.com> The content of unplugged memory is undefined and should not be migrated, ever. Exclude all unplugged memory during precopy using the precopy notifier infrastructure introduced for free page hinting in virtio-balloon. Unplugged memory is marked as "not dirty", meaning it won't be considered for migration. Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Signed-off-by: David Hildenbrand <david@redhat.com> --- hw/virtio/virtio-mem.c | 54 +++++++++++++++++++++++++++++++++- include/hw/virtio/virtio-mem.h | 3 ++ 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index 4d0a2e78c0..e278b213e2 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -62,8 +62,14 @@ static bool virtio_mem_is_busy(void) /* * Postcopy cannot handle concurrent discards and we don't want to migrate * pages on-demand with stale content when plugging new blocks. + * + * For precopy, we don't want unplugged blocks in our migration stream, and + * when plugging new blocks, the page content might differ between source + * and destination (observable by the guest when not initializing pages + * after plugging them) until we're running on the destination (as we didn't + * migrate these blocks when they were unplugged). */ - return migration_in_incoming_postcopy(); + return migration_in_incoming_postcopy() || !migration_is_idle(); } static bool virtio_mem_test_bitmap(VirtIOMEM *vmem, uint64_t start_gpa, @@ -475,6 +481,7 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp) host_memory_backend_set_mapped(vmem->memdev, true); vmstate_register_ram(&vmem->memdev->mr, DEVICE(vmem)); qemu_register_reset(virtio_mem_system_reset, vmem); + precopy_add_notifier(&vmem->precopy_notifier); } static void virtio_mem_device_unrealize(DeviceState *dev) @@ -482,6 +489,7 @@ static void virtio_mem_device_unrealize(DeviceState *dev) VirtIODevice *vdev = VIRTIO_DEVICE(dev); VirtIOMEM *vmem = VIRTIO_MEM(dev); + precopy_remove_notifier(&vmem->precopy_notifier); qemu_unregister_reset(virtio_mem_system_reset, vmem); vmstate_unregister_ram(&vmem->memdev->mr, DEVICE(vmem)); host_memory_backend_set_mapped(vmem->memdev, false); @@ -760,12 +768,56 @@ static void virtio_mem_set_block_size(Object *obj, Visitor *v, const char *name, vmem->block_size = value; } +static void virtio_mem_precopy_exclude_unplugged(VirtIOMEM *vmem) +{ + void * const host = qemu_ram_get_host_addr(vmem->memdev->mr.ram_block); + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, length; + + /* + * Find consecutive unplugged blocks and exclude them from migration. + * + * Note: Blocks cannot get (un)plugged during precopy, no locking needed. + */ + first_zero_bit = find_first_zero_bit(vmem->bitmap, vmem->bitmap_size); + while (first_zero_bit < vmem->bitmap_size) { + offset = first_zero_bit * vmem->block_size; + last_zero_bit = find_next_bit(vmem->bitmap, vmem->bitmap_size, + first_zero_bit + 1) - 1; + length = (last_zero_bit - first_zero_bit + 1) * vmem->block_size; + + qemu_guest_free_page_hint(host + offset, length); + first_zero_bit = find_next_zero_bit(vmem->bitmap, vmem->bitmap_size, + last_zero_bit + 2); + } +} + +static int virtio_mem_precopy_notify(NotifierWithReturn *n, void *data) +{ + VirtIOMEM *vmem = container_of(n, VirtIOMEM, precopy_notifier); + PrecopyNotifyData *pnd = data; + + switch (pnd->reason) { + case PRECOPY_NOTIFY_SETUP: + precopy_enable_free_page_optimization(); + break; + case PRECOPY_NOTIFY_AFTER_BITMAP_SYNC: + virtio_mem_precopy_exclude_unplugged(vmem); + break; + default: + break; + } + + return 0; +} + static void virtio_mem_instance_init(Object *obj) { VirtIOMEM *vmem = VIRTIO_MEM(obj); vmem->block_size = VIRTIO_MEM_MIN_BLOCK_SIZE; notifier_list_init(&vmem->size_change_notifiers); + vmem->precopy_notifier.notify = virtio_mem_precopy_notify; object_property_add(obj, VIRTIO_MEM_SIZE_PROP, "size", virtio_mem_get_size, NULL, NULL, NULL); diff --git a/include/hw/virtio/virtio-mem.h b/include/hw/virtio/virtio-mem.h index 408a6ede50..ddb3822375 100644 --- a/include/hw/virtio/virtio-mem.h +++ b/include/hw/virtio/virtio-mem.h @@ -67,6 +67,9 @@ typedef struct VirtIOMEM { /* notifiers to notify when "size" changes */ NotifierList size_change_notifiers; + + /* don't migrate unplugged memory */ + NotifierWithReturn precopy_notifier; } VirtIOMEM; typedef struct VirtIOMEMClass { -- 2.25.4
next prev parent reply other threads:[~2020-06-03 14:51 UTC|newest] Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-06-03 14:48 [PATCH v3 00/20] virtio-mem: Paravirtualized memory hot(un)plug David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:48 ` [PATCH v3 01/20] exec: Introduce ram_block_discard_(disable|require)() David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:48 ` [PATCH v3 02/20] vfio: Convert to ram_block_discard_disable() David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:48 ` [PATCH v3 03/20] accel/kvm: " David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:48 ` [PATCH v3 04/20] s390x/pv: " David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:48 ` [PATCH v3 05/20] virtio-balloon: Rip out qemu_balloon_inhibit() David Hildenbrand 2020-06-03 14:48 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 06/20] target/i386: sev: Use ram_block_discard_disable() David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 07/20] migration/rdma: " David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 08/20] migration/colo: " David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-06 19:34 ` Lukas Straub 2020-06-06 19:34 ` Lukas Straub 2020-06-03 14:49 ` [PATCH v3 09/20] linux-headers: update to contain virtio-mem David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 10/20] virtio-mem: Paravirtualized memory hot(un)plug David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 11/20] virtio-pci: Proxy for virtio-mem David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 12/20] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 13/20] hmp: Handle virtio-mem when printing memory device info David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 14/20] numa: Handle virtio-mem in NUMA stats David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 18:33 ` Pankaj Gupta 2020-06-03 18:33 ` Pankaj Gupta 2020-06-03 14:49 ` [PATCH v3 15/20] pc: Support for virtio-mem-pci David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 16/20] virtio-mem: Allow notifiers for size changes David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 17/20] virtio-pci: Send qapi events when the virtio-mem " David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 18/20] virtio-mem: Migration sanity checks David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` [PATCH v3 19/20] virtio-mem: Add trace events David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand 2020-06-03 14:49 ` David Hildenbrand [this message] 2020-06-03 14:49 ` [PATCH v3 20/20] virtio-mem: Exclude unplugged memory during migration David Hildenbrand 2020-06-03 15:15 ` [PATCH v3 00/20] virtio-mem: Paravirtualized memory hot(un)plug Eric Blake 2020-06-03 15:15 ` Eric Blake 2020-06-09 13:11 ` Michael S. Tsirkin 2020-06-09 13:11 ` Michael S. Tsirkin 2020-06-09 13:26 ` David Hildenbrand 2020-06-09 13:26 ` David Hildenbrand 2020-06-09 15:59 ` Michael S. Tsirkin 2020-06-09 15:59 ` Michael S. Tsirkin 2020-06-09 16:18 ` Eduardo Habkost 2020-06-09 16:18 ` Eduardo Habkost 2020-06-09 18:38 ` David Hildenbrand 2020-06-09 18:38 ` David Hildenbrand 2020-06-09 18:42 ` Michael S. Tsirkin 2020-06-09 18:42 ` Michael S. Tsirkin 2020-06-09 19:41 ` Sean Christopherson 2020-06-09 19:41 ` Sean Christopherson 2020-06-09 19:54 ` David Hildenbrand 2020-06-09 19:54 ` David Hildenbrand 2020-06-10 7:36 ` David Hildenbrand 2020-06-10 7:36 ` David Hildenbrand
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200603144914.41645-21-david@redhat.com \ --to=david@redhat.com \ --cc=dgilbert@redhat.com \ --cc=ehabkost@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=mst@redhat.com \ --cc=pbonzini@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=qemu-s390x@nongnu.org \ --cc=rth@twiddle.net \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.