Re: [PATCH v5 0/9] migration/ram: Optimize for virtio-mem via RamDiscardManager

[David Hildenbrand <david@redhat.com>]

On 04.09.21 18:09, David Hildenbrand wrote:
> virtio-mem exposes a dynamic amount of memory within RAMBlocks by
> coordinating with the VM. Memory within a RAMBlock can either get
> plugged and consequently used by the VM, or unplugged and consequently no
> longer used by the VM. Logical unplug is realized by discarding the
> physical memory backing for virtual memory ranges, similar to memory
> ballooning.
> 
> However, important difference to virtio-balloon are:
> 
> a) A virtio-mem device only operates on its assigned memory region /
>     RAMBlock ("device memory")
> b) Initially, all device memory is logically unplugged
> c) Virtual machines will never accidentally reuse memory that is currently
>     logically unplugged. The spec defines most accesses to unplugged memory
>     as "undefined behavior" -- except reading unplugged memory, which is
>     currently expected to work, but that will change in the future.
> d) The (un)plug granularity is in the range of megabytes -- "memory blocks"
> e) The state (plugged/unplugged) of a memory block is always known and
>     properly tracked.
> 
> Whenever memory blocks within the RAMBlock get (un)plugged, changes are
> communicated via the RamDiscardManager to other QEMU subsystems, most
> prominently vfio which updates the DMA mapping accordingly. "Unplugging"
> corresponds to "discarding" and "plugging" corresponds to "populating".
> 
> While migrating (precopy/postcopy) that state of such memory blocks cannot
> change, as virtio-mem will reject any guest requests that would change
> the state of blocks with "busy". We don't want to migrate such logically
> unplugged memory, because it can result in an unintended memory consumption
> both, on the source (when reading memory from some memory backends) and on
> the destination (when writing memory). Further, migration time can be
> heavily reduced when skipping logically unplugged blocks and we avoid
> populating unnecessary page tables in Linux.
> 
> Right now, virtio-mem reuses the free page hinting infrastructure during
> precopy to exclude all logically unplugged ("discarded") parts from the
> migration stream. However, there are some scenarios that are not handled
> properly and need fixing. Further, there are some ugly corner cases in
> postcopy code and background snapshotting code that similarly have to
> handle such special RAMBlocks.
> 
> Let's reuse the RamDiscardManager infrastructure to essentially handle
> precopy, postcopy and background snapshots cleanly, which means:
> 
> a) In precopy code, fixing up the initial dirty bitmaps (in the RAMBlock
>     and e.g., KVM) to exclude discarded ranges.
> b) In postcopy code, placing a zeropage when requested to handle a page
>     falling into a discarded range -- because the source will never send it.
>     Further, fix up the dirty bitmap when overwriting it in recovery mode.
> c) In background snapshot code, never populating discarded ranges, not even
>     with the shared zeropage, to avoid unintended memory consumption,
>     especially in the future with hugetlb and shmem.
> 
> Detail: When realizing a virtio-mem devices, it will register the RAM
>          for migration via vmstate_register_ram(). Further, it will
>          set itself as the RamDiscardManager for the corresponding memory
>          region of the RAMBlock via memory_region_set_ram_discard_manager().
>          Last but not least, memory device code will actually map the
>          memory region into guest physical address space. So migration
>          code can always properly identify such RAMBlocks.
> 
> Tested with precopy/postcopy on shmem, where even reading unpopulated
> memory ranges will populate actual memory and not the shared zeropage.
> Tested with background snapshots on anonymous memory, because other
> backends are not supported yet with upstream Linux.
> 

Gentle ping.


-- 
Thanks,

David / dhildenb