Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

From: Peter Xu <peterx@redhat.com>
To: Fabiano Rosas <farosas@suse.de>
Cc: qemu-devel@nongnu.org, "Claudio Fontana" <cfontana@suse.de>,
	jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com,
	"Daniel P . Berrangé" <berrange@redhat.com>,
	"Juan Quintela" <quintela@redhat.com>
Subject: Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram
Date: Thu, 30 Mar 2023 17:41:39 -0400	[thread overview]
Message-ID: <ZCYCE0llX9WANK18@x1n> (raw)
In-Reply-To: <20230330180336.2791-1-farosas@suse.de>

On Thu, Mar 30, 2023 at 03:03:10PM -0300, Fabiano Rosas wrote:
> Hi folks,

Hi,

> 
> I'm continuing the work done last year to add a new format of
> migration stream that can be used to migrate large guests to a single
> file in a performant way.
> 
> This is an early RFC with the previous code + my additions to support
> multifd and direct IO. Let me know what you think!
> 
> Here are the reference links for previous discussions:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg01813.html
> https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg01338.html
> https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg05536.html
> 
> The series has 4 main parts:
> 
> 1) File migration: A new "file:" migration URI. So "file:mig" does the
>    same as "exec:cat > mig". Patches 1-4 implement this;
> 
> 2) Fixed-ram format: A new format for the migration stream. Puts guest
>    pages at their relative offsets in the migration file. This saves
>    space on the worst case of RAM utilization because every page has a
>    fixed offset in the migration file and (potentially) saves us time
>    because we could write pages independently in parallel. It also
>    gives alignment guarantees so we could use O_DIRECT. Patches 5-13
>    implement this;
> 
> With patches 1-13 these two^ can be used with:
> 
> (qemu) migrate_set_capability fixed-ram on
> (qemu) migrate[_incoming] file:mig

Have you considered enabling the new fixed-ram format with postcopy when
loading?

Due to the linear offseting of pages, I think it can achieve super fast vm
loads due to O(1) lookup of pages and local page fault resolutions.

> 
> --> new in this series:
> 
> 3) MultiFD support: This is about making use of the parallelism
>    allowed by the new format. We just need the threading and page
>    queuing infrastructure that is already in place for
>    multifd. Patches 14-24 implement this;
> 
> (qemu) migrate_set_capability fixed-ram on
> (qemu) migrate_set_capability multifd on
> (qemu) migrate_set_parameter multifd-channels 4
> (qemu) migrate_set_parameter max-bandwith 0
> (qemu) migrate[_incoming] file:mig
> 
> 4) Add a new "direct_io" parameter and enable O_DIRECT for the
>    properly aligned segments of the migration (mostly ram). Patch 25.
> 
> (qemu) migrate_set_parameter direct-io on
> 
> Thanks! Some data below:
> =====
> 
> Outgoing migration to file. NVMe disk. XFS filesystem.
> 
> - Single migration runs of stopped 32G guest with ~90% RAM usage. Guest
>   running `stress-ng --vm 4 --vm-bytes 90% --vm-method all --verify -t
>   10m -v`:
> 
> migration type  | MB/s | pages/s |  ms
> ----------------+------+---------+------
> savevm io_uring |  434 |  102294 | 71473

So I assume this is the non-live migration scenario.  Could you explain
what does io_uring mean here?

> file:           | 3017 |  855862 | 10301
> fixed-ram       | 1982 |  330686 | 15637
> ----------------+------+---------+------
> fixed-ram + multifd + O_DIRECT
>          2 ch.  | 5565 | 1500882 |  5576
>          4 ch.  | 5735 | 1991549 |  5412
>          8 ch.  | 5650 | 1769650 |  5489
>         16 ch.  | 6071 | 1832407 |  5114
>         32 ch.  | 6147 | 1809588 |  5050
>         64 ch.  | 6344 | 1841728 |  4895
>        128 ch.  | 6120 | 1915669 |  5085
> ----------------+------+---------+------

Thanks,

-- 
Peter Xu