All of lore.kernel.org
 help / color / mirror / Atom feed
From: Peter Xu <peterx@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Daniel P . Berrange" <berrange@redhat.com>,
	Leonardo Bras Soares Passos <lsoaresp@redhat.com>,
	Manish Mishra <manish.mishra@nutanix.com>,
	Juan Quintela <quintela@redhat.com>,
	"Dr . David Alan Gilbert" <dgilbert@redhat.com>,
	peterx@redhat.com
Subject: [PATCH RFC 00/13] migration: Postcopy Preempt-Full
Date: Mon, 29 Aug 2022 12:56:46 -0400	[thread overview]
Message-ID: <20220829165659.96046-1-peterx@redhat.com> (raw)

This is a RFC series.  Tree is here:

  https://github.com/xzpeter/qemu/tree/preempt-full

It's not complete because there're still something we need to do which will
be attached to the end of this cover letter, however this series can
already safely pass qtest and any of my test.

Comparing to the recently merged preempt mode I called it "preempt-full"
because it threadifies the postcopy channels so now urgent pages can be
fully handled separately outside of the ram save loop.  Sorry to have the
same name as the PREEMPT_FULL in the Linux RT world, it's just that we
needed a name for the capability and it was named as preempt already
anyway..

The existing preempt code has reduced ramdom page req latency over 10Gbps
network from ~12ms to ~500us which has already landed.

This preempt-full series can further reduces that ~500us to ~230us per my
initial test.  More to share below.

Note that no new capability is needed, IOW it's fully compatible with the
existing preempt mode.  So the naming is actually not important but just to
identify the difference on the binaries.  It's because this series only
reworks the sender side code and does not change the migration protocol, it
just runs faster.

IOW, old "preempt" QEMU can also migrate to "preempt-full" QEMU, vice versa.

  - When old "preempt" mode QEMU migrates to "preempt-full" QEMU, it'll be
    the same as running both old "preempt" QEMUs.

  - When "preempt-full" QEMU migrates to old "preempt" QEMU, it'll be the
    same as running both "preempt-full".

The logic of the series is quite simple too: simply moving the existing
preempt channel page sends to rp-return thread.  It can slow down rp-return
thread on receiving pages, but I don't really see a major issue with it so
far.

This latency number is getting close to the extreme of 4K page request
latency of any TCP roundtrip of the 10Gbps nic I have.  The 'extreme
number' is something I get from mig_mon tool which has a mode [1] to
emulate the extreme tcp roundtrips of page requests.

Performance
===========

Page request latencies has distributions as below, with a VM of 20G mem, 20
cores, 10Gbps nic, 18G fully random writes:

Postcopy Vanilla
----------------

Average: 12093 (us)
@delay_us:
[1]                    1 |                                                    |
[2, 4)                 0 |                                                    |
[4, 8)                 0 |                                                    |
[8, 16)                0 |                                                    |
[16, 32)               1 |                                                    |
[32, 64)               8 |                                                    |
[64, 128)             11 |                                                    |
[128, 256)            14 |                                                    |
[256, 512)            19 |                                                    |
[512, 1K)             14 |                                                    |
[1K, 2K)              35 |                                                    |
[2K, 4K)              18 |                                                    |
[4K, 8K)              87 |@                                                   |
[8K, 16K)           2397 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)             7 |                                                    |
[32K, 64K)             2 |                                                    |
[64K, 128K)           20 |                                                    |
[128K, 256K)           6 |                                                    |

Postcopy Preempt
----------------

Average: 496 (us)

@delay_us:
[32, 64)               2 |                                                    |
[64, 128)           2306 |@@@@                                                |
[128, 256)         25422 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256, 512)          8238 |@@@@@@@@@@@@@@@@                                    |
[512, 1K)           1066 |@@                                                  |
[1K, 2K)            2167 |@@@@                                                |
[2K, 4K)            3329 |@@@@@@                                              |
[4K, 8K)             109 |                                                    |
[8K, 16K)             48 |                                                    |

Postcopy Preempt-Full
---------------------

Average: 229 (us)

@delay_us:
[8, 16)                1 |                                                    |
[16, 32)               3 |                                                    |
[32, 64)               2 |                                                    |
[64, 128)          11956 |@@@@@@@@@@                                          |
[128, 256)         60403 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256, 512)         15047 |@@@@@@@@@@@@                                        |
[512, 1K)            846 |                                                    |
[1K, 2K)              25 |                                                    |
[2K, 4K)              41 |                                                    |
[4K, 8K)             131 |                                                    |
[8K, 16K)             72 |                                                    |
[16K, 32K)             2 |                                                    |
[32K, 64K)             8 |                                                    |
[64K, 128K)            6 |                                                    |

For fully sequential page access workloads, I have described in the
previous preempt-mode work that such workload may not benefit much from
preempt mode much, but surprisingly at least in my seq write test the
preempt-full mode can also benefit sequential access patterns at least when
I measured it:

Postcopy Vanilla
----------------

Average: 1487 (us)

@delay_us:
[0]                   93 |@                                                   |
[1]                 1920 |@@@@@@@@@@@@@@@@@@@@@@@                             |
[2, 4)               504 |@@@@@@                                              |
[4, 8)              2234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@                         |
[8, 16)             4199 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)            3782 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
[32, 64)            1016 |@@@@@@@@@@@@                                        |
[64, 128)             81 |@                                                   |
[128, 256)            14 |                                                    |
[256, 512)            26 |                                                    |
[512, 1K)             69 |                                                    |
[1K, 2K)             208 |@@                                                  |
[2K, 4K)             429 |@@@@@                                               |
[4K, 8K)            2779 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[8K, 16K)            792 |@@@@@@@@@                                           |
[16K, 32K)             9 |                                                    |

Postcopy Preempt-Full
---------------------

Average: 1582 (us)

@delay_us:
[0]                   45 |                                                    |
[1]                 1786 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                       |
[2, 4)               423 |@@@@@@@                                             |
[4, 8)              1903 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                     |
[8, 16)             2933 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    |
[16, 32)            3132 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[32, 64)             518 |@@@@@@@@                                            |
[64, 128)             30 |                                                    |
[128, 256)           218 |@@@                                                 |
[256, 512)           214 |@@@                                                 |
[512, 1K)            211 |@@@                                                 |
[1K, 2K)             131 |@@                                                  |
[2K, 4K)             336 |@@@@@                                               |
[4K, 8K)            3023 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  |
[8K, 16K)            479 |@@@@@@@                                             |

Postcopy Preempt-Full
---------------------

Average: 439 (us)

@delay_us:
[0]                    3 |                                                    |
[1]                 1058 |@                                                   |
[2, 4)               179 |                                                    |
[4, 8)              1079 |@                                                   |
[8, 16)             2251 |@@@                                                 |
[16, 32)            2345 |@@@@                                                |
[32, 64)             713 |@                                                   |
[64, 128)           5386 |@@@@@@@@@                                           |
[128, 256)         30252 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256, 512)         10789 |@@@@@@@@@@@@@@@@@@                                  |
[512, 1K)            367 |                                                    |
[1K, 2K)              26 |                                                    |
[2K, 4K)             256 |                                                    |
[4K, 8K)            1840 |@@@                                                 |
[8K, 16K)            300 |                                                    |

I always don't think seq access is important in migrations, because for any
not-small VM that has a migration challenge, any multiple seq accesses will
also be grown into a random access pattern.  But I'm anyway laying the data
around for good reference.

Comments welcomed, thanks.

TODO List
=========

- Make migration accountings atomic
- Drop rs->f?
- Disable xbzrle for preempt mode?  Is it already perhaps disabled for postcopy?
- If this series can be really accepted, we can logically drop some of the
  old (complcated) code with the old preempt series.
- Drop x-postcopy-preempt-break-huge parameter?
- More to come

[1] https://github.com/xzpeter/mig_mon#vm-live-migration-network-emulator

Peter Xu (13):
  migration: Use non-atomic ops for clear log bitmap
  migration: Add postcopy_preempt_active()
  migration: Yield bitmap_mutex properly when sending/sleeping
  migration: Cleanup xbzrle zero page cache update logic
  migration: Disallow postcopy preempt to be used with compress
  migration: Trivial cleanup save_page_header() on same block check
  migration: Remove RAMState.f references in compression code
  migration: Teach PSS about host page
  migration: Introduce pss_channel
  migration: Add pss_init()
  migration: Make PageSearchStatus part of RAMState
  migration: Move last_sent_block into PageSearchStatus
  migration: Send requested page directly in rp-return thread

 include/exec/ram_addr.h |  11 +-
 include/qemu/bitmap.h   |   1 +
 migration/migration.c   |  11 +
 migration/ram.c         | 496 +++++++++++++++++++++++++++++-----------
 util/bitmap.c           |  45 ++++
 5 files changed, 421 insertions(+), 143 deletions(-)

-- 
2.32.0



             reply	other threads:[~2022-08-29 17:00 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-29 16:56 Peter Xu [this message]
2022-08-29 16:56 ` [PATCH RFC 01/13] migration: Use non-atomic ops for clear log bitmap Peter Xu
2022-09-15 18:49   ` Dr. David Alan Gilbert
2022-09-15 19:44     ` Peter Xu
2022-08-29 16:56 ` [PATCH RFC 02/13] migration: Add postcopy_preempt_active() Peter Xu
2022-09-15 18:50   ` Dr. David Alan Gilbert
2022-08-29 16:56 ` [PATCH RFC 03/13] migration: Yield bitmap_mutex properly when sending/sleeping Peter Xu
2022-08-29 16:56 ` [PATCH RFC 04/13] migration: Cleanup xbzrle zero page cache update logic Peter Xu
2022-08-29 16:56 ` [PATCH RFC 05/13] migration: Disallow postcopy preempt to be used with compress Peter Xu
2022-08-29 16:56 ` [PATCH RFC 06/13] migration: Trivial cleanup save_page_header() on same block check Peter Xu
2022-08-29 16:56 ` [PATCH RFC 07/13] migration: Remove RAMState.f references in compression code Peter Xu
2022-08-29 16:56 ` [PATCH RFC 08/13] migration: Teach PSS about host page Peter Xu
2022-08-29 16:56 ` [PATCH RFC 09/13] migration: Introduce pss_channel Peter Xu
2022-08-29 16:56 ` [PATCH RFC 10/13] migration: Add pss_init() Peter Xu
2022-08-29 16:56 ` [PATCH RFC 11/13] migration: Make PageSearchStatus part of RAMState Peter Xu
2022-08-29 16:56 ` [PATCH RFC 12/13] migration: Move last_sent_block into PageSearchStatus Peter Xu
2022-08-29 16:56 ` [PATCH RFC 13/13] migration: Send requested page directly in rp-return thread Peter Xu
2022-08-29 17:39 ` [PATCH RFC 00/13] migration: Postcopy Preempt-Full Peter Xu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220829165659.96046-1-peterx@redhat.com \
    --to=peterx@redhat.com \
    --cc=berrange@redhat.com \
    --cc=dgilbert@redhat.com \
    --cc=lsoaresp@redhat.com \
    --cc=manish.mishra@nutanix.com \
    --cc=qemu-devel@nongnu.org \
    --cc=quintela@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.