From: "Wang, Wei W" <wei.w.wang@intel.com>
To: "Michael S. Tsirkin" <mst@redhat.com>,
David Hildenbrand <david@redhat.com>
Cc: "virtio-dev@lists.oasis-open.org"
<virtio-dev@lists.oasis-open.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"virtualization@lists.linux-foundation.org"
<virtualization@lists.linux-foundation.org>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"mhocko@kernel.org" <mhocko@kernel.org>,
"akpm@linux-foundation.org" <akpm@linux-foundation.org>,
"torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
"pbonzini@redhat.com" <pbonzini@redhat.com>,
"liliang.opensource@gmail.com" <liliang.opensource@gmail.com>,
"yang.zhang.wz@gmail.com" <yang.zhang.wz@gmail.com>,
"quan.xu0@gmail.com" <quan.xu0@gmail.com>,
"nilal@redhat.com" <nilal@redhat.com>,
"riel@redhat.com" <riel@redhat.com>,
"peterx@redhat.com" <peterx@redhat.com>
Subject: RE: [PATCH v34 0/4] Virtio-balloon: support free page reporting
Date: Fri, 29 Jun 2018 15:52:40 +0000 [thread overview]
Message-ID: <286AC319A985734F985F78AFA26841F7396C251E@shsmsx102.ccr.corp.intel.com> (raw)
In-Reply-To: <20180629172216-mutt-send-email-mst@kernel.org>
On Friday, June 29, 2018 10:46 PM, Michael S. Tsirkin wrote:
> To: David Hildenbrand <david@redhat.com>
> Cc: Wang, Wei W <wei.w.wang@intel.com>; virtio-dev@lists.oasis-open.org;
> linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org;
> kvm@vger.kernel.org; linux-mm@kvack.org; mhocko@kernel.org;
> akpm@linux-foundation.org; torvalds@linux-foundation.org;
> pbonzini@redhat.com; liliang.opensource@gmail.com;
> yang.zhang.wz@gmail.com; quan.xu0@gmail.com; nilal@redhat.com;
> riel@redhat.com; peterx@redhat.com
> Subject: Re: [PATCH v34 0/4] Virtio-balloon: support free page reporting
>
> On Wed, Jun 27, 2018 at 01:06:32PM +0200, David Hildenbrand wrote:
> > On 25.06.2018 14:05, Wei Wang wrote:
> > > This patch series is separated from the previous "Virtio-balloon
> > > Enhancement" series. The new feature,
> > > VIRTIO_BALLOON_F_FREE_PAGE_HINT, implemented by this series
> enables
> > > the virtio-balloon driver to report hints of guest free pages to the
> > > host. It can be used to accelerate live migration of VMs. Here is an
> introduction of this usage:
> > >
> > > Live migration needs to transfer the VM's memory from the source
> > > machine to the destination round by round. For the 1st round, all
> > > the VM's memory is transferred. From the 2nd round, only the pieces
> > > of memory that were written by the guest (after the 1st round) are
> > > transferred. One method that is popularly used by the hypervisor to
> > > track which part of memory is written is to write-protect all the guest
> memory.
> > >
> > > This feature enables the optimization by skipping the transfer of
> > > guest free pages during VM live migration. It is not concerned that
> > > the memory pages are used after they are given to the hypervisor as
> > > a hint of the free pages, because they will be tracked by the
> > > hypervisor and transferred in the subsequent round if they are used and
> written.
> > >
> > > * Tests
> > > - Test Environment
> > > Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> > > Guest: 8G RAM, 4 vCPU
> > > Migration setup: migrate_set_speed 100G, migrate_set_downtime 2
> > > second
> > >
> > > - Test Results
> > > - Idle Guest Live Migration Time (results are averaged over 10 runs):
> > > - Optimization v.s. Legacy = 284ms vs 1757ms --> ~84% reduction
> > > - Guest with Linux Compilation Workload (make bzImage -j4):
> > > - Live Migration Time (average)
> > > Optimization v.s. Legacy = 1402ms v.s. 2528ms --> ~44% reduction
> > > - Linux Compilation Time
> > > Optimization v.s. Legacy = 5min6s v.s. 5min12s
> > > --> no obvious difference
> > >
> >
> > Being in version 34 already, this whole thing still looks and feels
> > like a big hack to me. It might just be me, but especially if I read
> > about assumptions like "QEMU will not hotplug memory during
> > migration". This does not feel like a clean solution.
> >
> > I am still not sure if we really need this interface, especially as
> > real free page hinting might be on its way.
> >
> > a) we perform free page hinting by setting all free pages
> > (arch_free_page()) to zero. Migration will detect zero pages and
> > minimize #pages to migrate. I don't think this is a good idea but
> > Michel suggested to do a performance evaluation and Nitesh is looking
> > into that right now.
>
> Yes this test is needed I think. If we can get most of the benefit without PV
> interfaces, that's nice.
>
> Wei, I think you need this as part of your performance comparison
> too: set page poisoning value to 0 and enable KSM, compare with your
> patches.
Do you mean live migration with zero pages?
I can first share the amount of memory transferred during live migration I saw,
Legacy is around 380MB,
Optimization is around 340MB.
This proves that most pages have already been 0 and skipped during the legacy live migration. But the legacy time is still much larger because zero page checking is costly.
(It's late night here, I can get you that with my server probably tomorrow)
Best,
Wei
next prev parent reply other threads:[~2018-06-29 15:53 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-25 12:05 [PATCH v34 0/4] Virtio-balloon: support free page reporting Wei Wang
2018-06-25 12:05 ` [PATCH v34 1/4] mm: support to get hints of free page blocks Wei Wang
2018-06-25 12:05 ` [PATCH v34 2/4] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT Wei Wang
2018-06-26 1:37 ` Michael S. Tsirkin
2018-06-26 3:46 ` Wei Wang
2018-06-26 3:56 ` Michael S. Tsirkin
2018-06-26 12:27 ` Wei Wang
2018-06-26 13:34 ` Michael S. Tsirkin
2018-06-27 1:24 ` Wei Wang
2018-06-27 2:41 ` Michael S. Tsirkin
2018-06-27 3:00 ` Wei Wang
2018-06-27 3:58 ` Michael S. Tsirkin
2018-06-27 5:27 ` Wei Wang
2018-06-27 16:53 ` [virtio-dev] " Michael S. Tsirkin
2018-06-25 12:05 ` [PATCH v34 3/4] mm/page_poison: expose page_poisoning_enabled to kernel modules Wei Wang
2018-06-25 12:05 ` [PATCH v34 4/4] virtio-balloon: VIRTIO_BALLOON_F_PAGE_POISON Wei Wang
2018-06-27 11:06 ` [PATCH v34 0/4] Virtio-balloon: support free page reporting David Hildenbrand
2018-06-29 3:51 ` Wei Wang
2018-06-29 7:46 ` David Hildenbrand
2018-06-29 11:31 ` Wei Wang
2018-06-29 11:53 ` David Hildenbrand
2018-06-29 15:55 ` Wang, Wei W
2018-06-29 16:03 ` David Hildenbrand
2018-06-29 14:45 ` Michael S. Tsirkin
2018-06-29 15:28 ` David Hildenbrand
2018-06-29 15:52 ` Wang, Wei W [this message]
2018-06-29 16:32 ` Michael S. Tsirkin
2018-06-30 4:31 ` Wei Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=286AC319A985734F985F78AFA26841F7396C251E@shsmsx102.ccr.corp.intel.com \
--to=wei.w.wang@intel.com \
--cc=akpm@linux-foundation.org \
--cc=david@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=liliang.opensource@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mst@redhat.com \
--cc=nilal@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=quan.xu0@gmail.com \
--cc=riel@redhat.com \
--cc=torvalds@linux-foundation.org \
--cc=virtio-dev@lists.oasis-open.org \
--cc=virtualization@lists.linux-foundation.org \
--cc=yang.zhang.wz@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).