All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olaf Hering <olaf@aepfle.de>
To: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: <xen-devel@lists.xenproject.org>,
	Ian Jackson <iwj@xenproject.org>, Wei Liu <wl@xen.org>,
	Juergen Gross <jgross@suse.com>
Subject: Re: [PATCH v20210701 15/40] tools: prepare to allocate saverestore arrays once
Date: Mon, 5 Jul 2021 13:27:00 +0200	[thread overview]
Message-ID: <20210705132700.05d92744.olaf@aepfle.de> (raw)
In-Reply-To: <644a7a4c-4fab-07be-2e69-2637254de859@citrix.com>

[-- Attachment #1: Type: text/plain, Size: 4622 bytes --]

Am Mon, 5 Jul 2021 11:44:30 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> On 01/07/2021 10:56, Olaf Hering wrote:
> I agree that the repeated alloc/free of same-sized memory regions on
> each iteration is a waste.  However, if we are going to fix this by
> using one-off allocations, then we want to compensate with logic such as
> the call to VALGRIND_MAKE_MEM_UNDEFINED() in flush_batch(), and I think
> we still need individual allocations to let the tools work properly.

If this is a concern, lets just do a few individual arrays.

> > This patch is just prepartion, subsequent changes will populate the arrays.
> >
> > Once all changes are applied, migration of a busy HVM domU changes like that:
> >
> > Without this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3 xen_testing):
> > 2020-10-29 10:23:10.711+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 55.324905335 sec, 203 MiB/sec: Internal error
> > 2020-10-29 10:23:35.115+0000: xc: show_transfer_rate: 16829632 bytes + 2097552 pages in 24.401179720 sec, 335 MiB/sec: Internal error
> > 2020-10-29 10:23:59.436+0000: xc: show_transfer_rate: 16829032 bytes + 2097478 pages in 24.319025928 sec, 336 MiB/sec: Internal error
> > 2020-10-29 10:24:23.844+0000: xc: show_transfer_rate: 16829024 bytes + 2097477 pages in 24.406992500 sec, 335 MiB/sec: Internal error
> > 2020-10-29 10:24:48.292+0000: xc: show_transfer_rate: 16828912 bytes + 2097463 pages in 24.446489027 sec, 335 MiB/sec: Internal error
> > 2020-10-29 10:25:01.816+0000: xc: show_transfer_rate: 16836080 bytes + 2098356 pages in 13.447091818 sec, 609 MiB/sec: Internal error
> >
> > With this series, from sr650 to sr950 (xen-4.15.20201027T173911.16a20963b3 xen_unstable):
> > 2020-10-28 21:26:05.074+0000: xc: show_transfer_rate: 23663128 bytes + 2879563 pages in 52.564054368 sec, 213 MiB/sec: Internal error
> > 2020-10-28 21:26:23.527+0000: xc: show_transfer_rate: 16830040 bytes + 2097603 pages in 18.450592015 sec, 444 MiB/sec: Internal error
> > 2020-10-28 21:26:41.926+0000: xc: show_transfer_rate: 16830944 bytes + 2097717 pages in 18.397862306 sec, 445 MiB/sec: Internal error
> > 2020-10-28 21:27:00.339+0000: xc: show_transfer_rate: 16829176 bytes + 2097498 pages in 18.411973339 sec, 445 MiB/sec: Internal error
> > 2020-10-28 21:27:18.643+0000: xc: show_transfer_rate: 16828592 bytes + 2097425 pages in 18.303326695 sec, 447 MiB/sec: Internal error
> > 2020-10-28 21:27:26.289+0000: xc: show_transfer_rate: 16835952 bytes + 2098342 pages in 7.579846749 sec, 1081 MiB/sec: Internal error  
> 
> These are good numbers, and clearly show that there is some value here,
> but shouldn't they be in the series header?  They're not terribly
> relevant to this patch specifically.

The cover letter is unfortunately not under version control.
Perhaps there are ways with git notes, I never use it.

> Also, while I can believe that the first sample is slower than the later
> ones (in particular, during the first round, we've got to deal with the
> non-RAM regions too and therefore spend more time making hypercalls),
> I'm not sure I believe the final sample.  Given the byte/page count, the
> substantially smaller elapsed time looks suspicious.

The first one is slower because it has to wait for the receiver to allocate pages.
But maybe as you said there are other aspects as well.
The last one is always way faster because apparently map/unmap is less costly with a stopped guest.
Right now the code may reach up to 15Gbit/s. The next step is to map the domU just once to reach wirespeed.

> Are these observations with an otherwise idle dom0?

Yes. Idle dom0 and a domU busy with touching its memory.

Unfortunately, I'm not able to prove the reported gain with the systems I have today.
I'm waiting for preparation of different hardware, right now I have only a pair of CoyotePass and WilsonCity.

I'm sure there were NUMA effects involved. Last years libvirt was unable to properly pin vcpus. If I pin all the involved memory to node#0 there is some jitter in the logged numbers, but no obvious improvement. The fist iteration is slightly faster, but that is it.

Meanwhile I think this commit message needs to be redone.

> Even if CPU time in dom0 wasn't the bottlekneck with a 1G link, the
> reduction in CPU time you observe at higher link speeds will still be
> making a difference at 1G, and will probably be visible if you perform
> multiple concurrent migrations.

Yes, I will see what numbers I get with two or more migrations running in parallel.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2021-07-05 11:27 UTC|newest]

Thread overview: 86+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-01  9:55 [PATCH v20210701 00/40] leftover from 2020 Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 01/40] hotplug/Linux: fix starting of xenstored with restarting systemd Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 02/40] tools: add API to work with sevaral bits at once Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 03/40] xl: fix description of migrate --debug Olaf Hering
2021-07-01 14:30   ` Anthony PERARD
2021-07-01 14:33   ` Andrew Cooper
2021-07-01 14:40     ` Olaf Hering
2021-07-01 14:41       ` Olaf Hering
2021-07-01 14:49         ` Andrew Cooper
2021-07-01 15:08           ` Olaf Hering
2021-07-01  9:55 ` [PATCH v20210701 04/40] tools: use integer division in convert-legacy-stream Olaf Hering
2021-07-02 15:10   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 05/40] tools: handle libxl__physmap_info.name properly " Olaf Hering
2021-07-02 15:35   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 06/40] tools: fix Python3.4 TypeError in format string Olaf Hering
2021-07-02 16:19   ` Marek Marczykowski-Górecki
2021-07-02 16:39     ` Andrew Cooper
2021-07-05  8:18       ` Olaf Hering
2021-07-05  9:47         ` Andrew Cooper
2021-07-05  8:07     ` Olaf Hering
2021-07-05 10:10       ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 07/40] tools: create libxensaverestore Olaf Hering
2021-07-09  9:20   ` Olaf Hering
2021-07-09  9:31     ` Julien Grall
2021-07-09  9:33       ` Olaf Hering
2021-07-09  9:35   ` Julien Grall
2021-07-01  9:56 ` [PATCH v20210701 08/40] MAINTAINERS: add myself as saverestore maintainer Olaf Hering
2021-07-01 10:39   ` Jan Beulich
2021-07-01 11:01     ` Olaf Hering
2021-07-01 11:40       ` Julien Grall
2021-07-01 12:00         ` Olaf Hering
2021-07-01 12:09           ` Julien Grall
2021-07-01  9:56 ` [PATCH v20210701 09/40] tools: add readv_exact to libxenctrl Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 10/40] tools: add xc_is_known_page_type " Olaf Hering
2021-07-02 19:20   ` Andrew Cooper
2021-07-05  8:22     ` Olaf Hering
2021-07-05  9:51       ` Andrew Cooper
2021-07-05 14:24         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 11/40] tools: use sr_is_known_page_type Olaf Hering
2021-07-02 19:27   ` Andrew Cooper
2021-07-05  8:25     ` Olaf Hering
2021-07-05  9:53       ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 12/40] tools: unify type checking for data pfns in migration stream Olaf Hering
2021-07-02 19:43   ` Andrew Cooper
2021-07-05  8:59     ` Olaf Hering
2021-07-05  9:53       ` Andrew Cooper
2021-07-05 13:10   ` Andrew Cooper
2021-07-05 13:53     ` Olaf Hering
2021-07-05 18:54       ` Andrew Cooper
2021-07-05 19:06         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 13/40] " Olaf Hering
2021-07-02 19:49   ` Andrew Cooper
2021-07-01  9:56 ` [PATCH v20210701 14/40] tools: show migration transfer rate in send_dirty_pages Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 15/40] tools: prepare to allocate saverestore arrays once Olaf Hering
2021-07-05 10:44   ` Andrew Cooper
2021-07-05 11:27     ` Olaf Hering [this message]
2021-07-05 13:01       ` Andrew Cooper
2021-07-05 14:11         ` Olaf Hering
2021-07-13 17:50         ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 16/40] tools: save: move mfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 17/40] tools: save: move types array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 18/40] tools: save: move errors array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 19/40] tools: save: move iov array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 20/40] tools: save: move rec_pfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 21/40] tools: save: move guest_data array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 22/40] tools: save: move local_pages array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 23/40] tools: restore: move types array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 24/40] tools: restore: move mfns array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 25/40] tools: restore: move map_errs array Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 26/40] tools: restore: move mfns array in populate_pfns Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 27/40] tools: restore: move pfns " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 28/40] tools: restore: split record processing Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 29/40] tools: restore: split handle_page_data Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 30/40] tools: restore: write data directly into guest Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 31/40] tools: recognize LIBXL_API_VERSION for 4.16 Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 32/40] tools: adjust libxl_domain_suspend to receive a struct props Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 33/40] tools: change struct precopy_stats to precopy_stats_t Olaf Hering
2021-07-01 16:45   ` Anthony PERARD
2021-07-01 17:08     ` Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 34/40] tools: add callback to libxl for precopy_policy and precopy_stats_t Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 35/40] tools: add --max_iters to libxl_domain_suspend Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 36/40] tools: add --min_remaining " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 37/40] tools: add --abort_if_busy " Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 38/40] tools: add API for expandable bitmaps Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 39/40] tools: use xg_sr_bitmap for populated_pfns Olaf Hering
2021-07-01  9:56 ` [PATCH v20210701 40/40] tools/libxc: use superpages during restore of HVM guest Olaf Hering

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210705132700.05d92744.olaf@aepfle.de \
    --to=olaf@aepfle.de \
    --cc=andrew.cooper3@citrix.com \
    --cc=iwj@xenproject.org \
    --cc=jgross@suse.com \
    --cc=wl@xen.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.