All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olaf Hering <olaf@aepfle.de>
To: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: <xen-devel@lists.xenproject.org>,
	Ian Jackson <iwj@xenproject.org>, Wei Liu <wl@xen.org>,
	Juergen Gross <jgross@suse.com>
Subject: Re: [PATCH v20210701 15/40] tools: prepare to allocate saverestore arrays once
Date: Mon, 5 Jul 2021 16:11:09 +0200	[thread overview]
Message-ID: <20210705161109.15d2d4e0.olaf@aepfle.de> (raw)
In-Reply-To: <6bd5fbb4-881f-e41c-46f5-d31c93a7ff7a@citrix.com>

[-- Attachment #1: Type: text/plain, Size: 2461 bytes --]

Am Mon, 5 Jul 2021 14:01:07 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> > The last one is always way faster because apparently map/unmap is less costly with a stopped guest.  
> That's suspicious.  If true, we've got some very wonky behaviour in the
> hypervisor...

At least the transfer rate this last iteration is consistent.
Since the only difference I can see is the fact that the domU is suspended, I suspect the mapping.
I did no investigation where the time is spent, I should probably do that one day to better understand this specific difference.

> > Right now the code may reach up to 15Gbit/s. The next step is to map the domU just once to reach wirespeed.  
> 
> We can in principle do that in 64bit toolstacks, for HVM guests.  But
> not usefully until we've fixed the fact that Xen has no idea what the
> guest physmap is supposed to look like.

Why would Xen care?
My attempt last year with a new save/restore code did just 'map' the memory on both sides. The 'unmap' was done in exit().

With this approach I got wirespeed in all iterations with a 10G link.

> At the moment, the current scheme is a little more resilient to bugs
> caused by the guest attempting to balloon during the live phase.

I did not specifically test how a domU behaves when it claims and releases pages while being migrated.
I think this series would handle at least parts of that:
If a page appears or disappears it will be recognized by getpageframeinfo.
If a page disappears between getpageframeinfo and MMAPBATCH I expect an error.
This error is fatal right now, perhaps the code could catch this and move on.
If a page disappears after MMAPBATCH it will be caught by later iterations.


> Another area to improve, which can be started now, is to avoid bounce
> buffering hypercall data.  Now that we have /dev/xen/hypercall which you
> can mmap() regular kernel pages from, what we want is a simple memory
> allocator which we can allocate permanent hypercall buffers from, rather
> than the internals of every xc_*() hypercall wrapper bouncing the data
> in (potentially) both directions.

That sounds like a good idea. Not sure how costly the current approach is.

> Oh - so the speedup might not be from reduced data handling?

At least not on the systems I have now.

Perhaps I should test how the numbers look like with the NIC and the toolstack in node#0, and the domU in node#1.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2021-07-05 14:11 UTC|newest]

Thread overview: 86+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-01  9:55 [PATCH v20210701 00/40] leftover from 2020 Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 01/40] hotplug/Linux: fix starting of xenstored with restarting systemd Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 02/40] tools: add API to work with sevaral bits at once Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 03/40] xl: fix description of migrate --debug Olaf Hering
2021-07-01 14:30   ` Anthony PERARD
2021-07-01 14:33   ` Andrew Cooper
2021-07-01 14:40     ` Olaf Hering
2021-07-01 14:41       ` Olaf Hering
2021-07-01 14:49         ` Andrew Cooper
2021-07-01 15:08           ` Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 04/40] tools: use integer division in convert-legacy-stream Olaf Hering
2021-07-02 15:10   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 05/40] tools: handle libxl__physmap_info.name properly " Olaf Hering
2021-07-02 15:35   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 06/40] tools: fix Python3.4 TypeError in format string Olaf Hering
2021-07-02 16:19   ` Marek Marczykowski-Górecki
2021-07-02 16:39     ` Andrew Cooper
2021-07-05  8:18       ` Olaf Hering
2021-07-05  9:47         ` Andrew Cooper
2021-07-05  8:07     ` Olaf Hering
2021-07-05 10:10       ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 07/40] tools: create libxensaverestore Olaf Hering
2021-07-09  9:20   ` Olaf Hering
2021-07-09  9:31     ` Julien Grall
2021-07-09  9:33       ` Olaf Hering
2021-07-09  9:35   ` Julien Grall
2021-07-01  9:56 ` [PATCH v20210701 08/40] MAINTAINERS: add myself as saverestore maintainer Olaf Hering
2021-07-01 10:39   ` Jan Beulich
2021-07-01 11:01     ` Olaf Hering
2021-07-01 11:40       ` Julien Grall
2021-07-01 12:00         ` Olaf Hering
2021-07-01 12:09           ` Julien Grall
2021-07-01  9:56 ` [PATCH v20210701 09/40] tools: add readv_exact to libxenctrl Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 10/40] tools: add xc_is_known_page_type " Olaf Hering
2021-07-02 19:20   ` Andrew Cooper
2021-07-05  8:22     ` Olaf Hering
2021-07-05  9:51       ` Andrew Cooper
2021-07-05 14:24         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 11/40] tools: use sr_is_known_page_type Olaf Hering
2021-07-02 19:27   ` Andrew Cooper
2021-07-05  8:25     ` Olaf Hering
2021-07-05  9:53       ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 12/40] tools: unify type checking for data pfns in migration stream Olaf Hering
2021-07-02 19:43   ` Andrew Cooper
2021-07-05  8:59     ` Olaf Hering
2021-07-05  9:53       ` Andrew Cooper
2021-07-05 13:10   ` Andrew Cooper
2021-07-05 13:53     ` Olaf Hering
2021-07-05 18:54       ` Andrew Cooper
2021-07-05 19:06         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 13/40] " Olaf Hering
2021-07-02 19:49   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 14/40] tools: show migration transfer rate in send_dirty_pages Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 15/40] tools: prepare to allocate saverestore arrays once Olaf Hering
2021-07-05 10:44   ` Andrew Cooper
2021-07-05 11:27     ` Olaf Hering
2021-07-05 13:01       ` Andrew Cooper
2021-07-05 14:11         ` Olaf Hering [this message]
2021-07-13 17:50         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 16/40] tools: save: move mfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 17/40] tools: save: move types array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 18/40] tools: save: move errors array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 19/40] tools: save: move iov array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 20/40] tools: save: move rec_pfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 21/40] tools: save: move guest_data array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 22/40] tools: save: move local_pages array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 23/40] tools: restore: move types array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 24/40] tools: restore: move mfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 25/40] tools: restore: move map_errs array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 26/40] tools: restore: move mfns array in populate_pfns Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 27/40] tools: restore: move pfns " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 28/40] tools: restore: split record processing Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 29/40] tools: restore: split handle_page_data Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 30/40] tools: restore: write data directly into guest Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 31/40] tools: recognize LIBXL_API_VERSION for 4.16 Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 32/40] tools: adjust libxl_domain_suspend to receive a struct props Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 33/40] tools: change struct precopy_stats to precopy_stats_t Olaf Hering
2021-07-01 16:45   ` Anthony PERARD
2021-07-01 17:08     ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 34/40] tools: add callback to libxl for precopy_policy and precopy_stats_t Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 35/40] tools: add --max_iters to libxl_domain_suspend Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 36/40] tools: add --min_remaining " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 37/40] tools: add --abort_if_busy " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 38/40] tools: add API for expandable bitmaps Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 39/40] tools: use xg_sr_bitmap for populated_pfns Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 40/40] tools/libxc: use superpages during restore of HVM guest Olaf Hering

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210705161109.15d2d4e0.olaf@aepfle.de \
    --to=olaf@aepfle.de \
    --cc=andrew.cooper3@citrix.com \
    --cc=iwj@xenproject.org \
    --cc=jgross@suse.com \
    --cc=wl@xen.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.