From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756245AbdEGVmD convert rfc822-to-8bit (ORCPT ); Sun, 7 May 2017 17:42:03 -0400 Received: from mga06.intel.com ([134.134.136.31]:57178 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756226AbdEGVlj (ORCPT ); Sun, 7 May 2017 17:41:39 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.38,302,1491289200"; d="scan'208";a="1165644932" From: "Wang, Wei W" To: "Michael S. Tsirkin" CC: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "david@redhat.com" , "Hansen, Dave" , "cornelia.huck@de.ibm.com" , "akpm@linux-foundation.org" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "liliang.opensource@gmail.com" Subject: RE: [PATCH v10 3/6] virtio-balloon: VIRTIO_BALLOON_F_PAGE_CHUNKS Thread-Topic: [PATCH v10 3/6] virtio-balloon: VIRTIO_BALLOON_F_PAGE_CHUNKS Thread-Index: AQHSxe8R2vL7EIsrz0GIzbzsKsMw9qHoM7RA Date: Sun, 7 May 2017 04:22:46 +0000 Message-ID: <286AC319A985734F985F78AFA26841F7391FFBE8@shsmsx102.ccr.corp.intel.com> References: <1493887815-6070-1-git-send-email-wei.w.wang@intel.com> <1493887815-6070-4-git-send-email-wei.w.wang@intel.com> <20170506012146-mutt-send-email-mst@kernel.org> In-Reply-To: <20170506012146-mutt-send-email-mst@kernel.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYWJiOTdmNjItNTQzMC00ZTUxLWE4ZTItNDkwN2M2Njg5ZWFhIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX1BVQkxJQyJ9XX1dfSwiU3ViamVjdExhYmVscyI6W10sIlRNQ1ZlcnNpb24iOiIxNS45LjYuNiIsIlRydXN0ZWRMYWJlbEhhc2giOiJpQTZDNXd5QThCWnBZWld3ZGtcL0dGQ1QzY2ZaRmhmRUtBeDNvVWJMd05EUT0ifQ== x-ctpclassification: CTP_PUBLIC dlp-product: dlpe-windows dlp-version: 10.0.102.7 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/06/2017 06:29 AM, Michael S. Tsirkin wrote: > On Thu, May 04, 2017 at 04:50:12PM +0800, Wei Wang wrote: > > Add a new feature, VIRTIO_BALLOON_F_PAGE_CHUNKS, which enables the > > transfer of the ballooned (i.e. inflated/deflated) pages in chunks to > > the host. > > > > The implementation of the previous virtio-balloon is not very > > efficient, because the ballooned pages are transferred to the host one > > by one. Here is the breakdown of the time in percentage spent on each > > step of the balloon inflating process (inflating 7GB of an 8GB idle > > guest). > > > > 1) allocating pages (6.5%) > > 2) sending PFNs to host (68.3%) > > 3) address translation (6.1%) > > 4) madvise (19%) > > > > It takes about 4126ms for the inflating process to complete. > > The above profiling shows that the bottlenecks are stage 2) and stage > > 4). > > > > This patch optimizes step 2) by transferring pages to the host in > > chunks. A chunk consists of guest physically continuous pages. > > When the pages are packed into a chunk, they are converted into > > balloon page size (4KB) pages. A chunk is offered to the host via a > > base PFN (i.e. the start PFN of those physically continuous > > pages) and the size (i.e. the total number of the 4KB balloon size > > pages). A chunk is formatted as below: > > -------------------------------------------------------- > > | Base (52 bit) | Rsvd (12 bit) | > > -------------------------------------------------------- > > -------------------------------------------------------- > > | Size (52 bit) | Rsvd (12 bit) | > > -------------------------------------------------------- > > > > By doing so, step 4) can also be optimized by doing address > > translation and madvise() in chunks rather than page by page. > > > > With this new feature, the above ballooning process takes ~590ms > > resulting in an improvement of ~85%. > > > > TODO: optimize stage 1) by allocating/freeing a chunk of pages instead > > of a single page each time. > > > > Signed-off-by: Wei Wang > > Signed-off-by: Liang Li > > Suggested-by: Michael S. Tsirkin > > > This is much cleaner, thanks. It might be even better to have wrappers that put > array and its size in a struct and manage that struct, but I won't require this for > submission. OK, thanks. Would this be your suggestion: struct virtio_balloon_page_struct { unsigned int page_bmap_num; unsigned long *page_bmap[VIRTIO_BALLOON_PAGE_BMAP_MAX_NUM]; } Best, Wei