Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

From: Fabiano Rosas <farosas@suse.de>
To: David Hildenbrand <david@redhat.com>, qemu-devel@nongnu.org
Cc: "Claudio Fontana" <cfontana@suse.de>,
	jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com,
	"Daniel P . Berrangé" <berrange@redhat.com>,
	"Juan Quintela" <quintela@redhat.com>
Subject: Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram
Date: Mon, 03 Apr 2023 13:36:03 -0300	[thread overview]
Message-ID: <87v8icexe4.fsf@suse.de> (raw)
In-Reply-To: <39fb1702-1db3-4525-24f1-5f4f506a6d0b@redhat.com>

David Hildenbrand <david@redhat.com> writes:

> On 03.04.23 16:41, Fabiano Rosas wrote:
>> David Hildenbrand <david@redhat.com> writes:
>> 
>>> On 30.03.23 20:03, Fabiano Rosas wrote:
>>>> Hi folks,
>>>>
>>>> I'm continuing the work done last year to add a new format of
>>>> migration stream that can be used to migrate large guests to a single
>>>> file in a performant way.
>>>>
>>>> This is an early RFC with the previous code + my additions to support
>>>> multifd and direct IO. Let me know what you think!
>>>>
>>>> Here are the reference links for previous discussions:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg01813.html
>>>> https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg01338.html
>>>> https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg05536.html
>>>>
>>>> The series has 4 main parts:
>>>>
>>>> 1) File migration: A new "file:" migration URI. So "file:mig" does the
>>>>      same as "exec:cat > mig". Patches 1-4 implement this;
>>>>
>>>> 2) Fixed-ram format: A new format for the migration stream. Puts guest
>>>>      pages at their relative offsets in the migration file. This saves
>>>>      space on the worst case of RAM utilization because every page has a
>>>>      fixed offset in the migration file and (potentially) saves us time
>>>>      because we could write pages independently in parallel. It also
>>>>      gives alignment guarantees so we could use O_DIRECT. Patches 5-13
>>>>      implement this;
>>>>
>>>> With patches 1-13 these two^ can be used with:
>>>>
>>>> (qemu) migrate_set_capability fixed-ram on
>>>> (qemu) migrate[_incoming] file:mig
>>>
>>> There are some use cases (especially virtio-mem, but also virtio-balloon
>>> with free-page-hinting) where we end up having very sparse guest RAM. We
>>> don't want to have such "memory without meaning" in the migration stream
>>> nor restore it on the destination.
>>>
>> 
>> Is that what is currently defined by ramblock_page_is_discarded ->
>> virtio_mem_rdm_is_populated ?
>
> For virtio-mem, yes. For virtio-balloon we communicate that information 
> via qemu_guest_free_page_hint().
>
>> 
>>> Would that still be supported with the new format? For example, have a
>>> sparse VM savefile and remember which ranges actually contain reasonable
>>> data?
>> 
>> We do ignore zero pages, so I don't think it would be an issue to have
>> another criteria for ignoring pages. It seems if we do enable postcopy
>> load w/ fixed-ram that would be already handled in postcopy_request_page.
>
> Ok, good. Just to note that we do have migration of sparse RAM blocks 
> working and if fixed-ram would be incompatible we'd have to fence it.

Yep, thanks for the heads-up. I'll keep that in mind.