From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932213AbcLGNfb convert rfc822-to-8bit (ORCPT ); Wed, 7 Dec 2016 08:35:31 -0500 Received: from mga04.intel.com ([192.55.52.120]:65144 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752651AbcLGNfa (ORCPT ); Wed, 7 Dec 2016 08:35:30 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,310,1477983600"; d="scan'208";a="909679808" From: "Li, Liang Z" To: David Hildenbrand , "kvm@vger.kernel.org" CC: "virtio-dev@lists.oasis-open.org" , "mhocko@suse.com" , "mst@redhat.com" , "Hansen, Dave" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "kirill.shutemov@linux.intel.com" , "pbonzini@redhat.com" , "akpm@linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "dgilbert@redhat.com" Subject: RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Thread-Topic: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Thread-Index: AQHSSufi4SQjg1m8CEK5jUmkErPNiaD6HPWAgAJj94A= Date: Wed, 7 Dec 2016 13:35:26 +0000 Message-ID: References: <1480495397-23225-1-git-send-email-liang.z.li@intel.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMGIyMjE1YjAtYjgyYS00ZmRkLTlmMDYtNmM4YzAyMTU0MjVmIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE1LjkuNi42IiwiVHJ1c3RlZExhYmVsSGFzaCI6IjF4Zk1jM2c4ZTROTVZObjVEVzNGRVBFZjgwSkFSck1QTFlJakh5ZDg0WjA9In0= x-ctpclassification: CTP_IC x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > Am 30.11.2016 um 09:43 schrieb Liang Li: > > This patch set contains two parts of changes to the virtio-balloon. > > > > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. > > Do you have some statistics/some rough feeling how many consecutive bits are > usually set in the bitmaps? Is it really just purely random or is there some > granularity that is usually consecutive? > I did something similar. Filled the balloon with 15GB for a 16GB idle guest, by using bitmap, the madvise count was reduced to 605. when using the PFNs, the madvise count was 3932160. It means there are quite a lot consecutive bits in the bitmap. I didn't test for a guest with heavy memory workload. > IOW in real examples, do we have really large consecutive areas or are all > pages just completely distributed over our memory? > The buddy system of Linux kernel memory management shows there should be quite a lot of consecutive pages as long as there are a portion of free memory in the guest. If all pages just completely distributed over our memory, it means the memory fragmentation is very serious, the kernel has the mechanism to avoid this happened. In the other hand, the inflating should not happen at this time because the guest is almost 'out of memory'. Liang > Thanks! > > -- > > David From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Li, Liang Z" Subject: RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Date: Wed, 7 Dec 2016 13:35:26 +0000 Message-ID: References: <1480495397-23225-1-git-send-email-liang.z.li@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "virtio-dev@lists.oasis-open.org" , "mhocko@suse.com" , "mst@redhat.com" , "Hansen, Dave" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "kirill.shutemov@linux.intel.com" , "pbonzini@redhat.com" , "akpm@linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "dgilbert@redhat.com" To: David Hildenbrand , "kvm@vger.kernel.org" Return-path: Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: In-Reply-To: Content-Language: en-US List-Id: kvm.vger.kernel.org > Am 30.11.2016 um 09:43 schrieb Liang Li: > > This patch set contains two parts of changes to the virtio-balloon. > > > > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. >=20 > Do you have some statistics/some rough feeling how many consecutive bits = are > usually set in the bitmaps? Is it really just purely random or is there s= ome > granularity that is usually consecutive? >=20 I did something similar. Filled the balloon with 15GB for a 16GB idle guest= , by using bitmap, the madvise count was reduced to 605. when using the PFNs, th= e madvise count was 3932160. It means there are quite a lot consecutive bits in the bitmap. I didn't test for a guest with heavy memory workload.=20 > IOW in real examples, do we have really large consecutive areas or are al= l > pages just completely distributed over our memory? >=20 The buddy system of Linux kernel memory management shows there should be qu= ite a lot of consecutive pages as long as there are a portion of free memory in the gue= st. If all pages just completely distributed over our memory, it means the memo= ry=20 fragmentation is very serious, the kernel has the mechanism to avoid this h= appened. In the other hand, the inflating should not happen at this time because the= guest is almost 'out of memory'. Liang > Thanks! >=20 > -- >=20 > David From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f198.google.com (mail-pf0-f198.google.com [209.85.192.198]) by kanga.kvack.org (Postfix) with ESMTP id BF8A06B0038 for ; Wed, 7 Dec 2016 08:35:30 -0500 (EST) Received: by mail-pf0-f198.google.com with SMTP id c4so601045449pfb.7 for ; Wed, 07 Dec 2016 05:35:30 -0800 (PST) Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id z21si24086972pgi.50.2016.12.07.05.35.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Dec 2016 05:35:29 -0800 (PST) From: "Li, Liang Z" Subject: RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Date: Wed, 7 Dec 2016 13:35:26 +0000 Message-ID: References: <1480495397-23225-1-git-send-email-liang.z.li@intel.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: David Hildenbrand , "kvm@vger.kernel.org" Cc: "virtio-dev@lists.oasis-open.org" , "mhocko@suse.com" , "mst@redhat.com" , "Hansen, Dave" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "kirill.shutemov@linux.intel.com" , "pbonzini@redhat.com" , "akpm@linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "dgilbert@redhat.com" > Am 30.11.2016 um 09:43 schrieb Liang Li: > > This patch set contains two parts of changes to the virtio-balloon. > > > > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. >=20 > Do you have some statistics/some rough feeling how many consecutive bits = are > usually set in the bitmaps? Is it really just purely random or is there s= ome > granularity that is usually consecutive? >=20 I did something similar. Filled the balloon with 15GB for a 16GB idle guest= , by using bitmap, the madvise count was reduced to 605. when using the PFNs, th= e madvise count was 3932160. It means there are quite a lot consecutive bits in the bitmap. I didn't test for a guest with heavy memory workload.=20 > IOW in real examples, do we have really large consecutive areas or are al= l > pages just completely distributed over our memory? >=20 The buddy system of Linux kernel memory management shows there should be qu= ite a lot of consecutive pages as long as there are a portion of free memory in the gue= st. If all pages just completely distributed over our memory, it means the memo= ry=20 fragmentation is very serious, the kernel has the mechanism to avoid this h= appened. In the other hand, the inflating should not happen at this time because the= guest is almost 'out of memory'. Liang > Thanks! >=20 > -- >=20 > David -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43561) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cEcNk-0008J5-Sf for qemu-devel@nongnu.org; Wed, 07 Dec 2016 08:35:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cEcNf-0005Gz-SX for qemu-devel@nongnu.org; Wed, 07 Dec 2016 08:35:36 -0500 Received: from mga14.intel.com ([192.55.52.115]:64643) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cEcNf-0005FR-Jq for qemu-devel@nongnu.org; Wed, 07 Dec 2016 08:35:31 -0500 From: "Li, Liang Z" Date: Wed, 7 Dec 2016 13:35:26 +0000 Message-ID: References: <1480495397-23225-1-git-send-email-liang.z.li@intel.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Hildenbrand , "kvm@vger.kernel.org" Cc: "virtio-dev@lists.oasis-open.org" , "mhocko@suse.com" , "mst@redhat.com" , "Hansen, Dave" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "kirill.shutemov@linux.intel.com" , "pbonzini@redhat.com" , "akpm@linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "dgilbert@redhat.com" > Am 30.11.2016 um 09:43 schrieb Liang Li: > > This patch set contains two parts of changes to the virtio-balloon. > > > > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. >=20 > Do you have some statistics/some rough feeling how many consecutive bits = are > usually set in the bitmaps? Is it really just purely random or is there s= ome > granularity that is usually consecutive? >=20 I did something similar. Filled the balloon with 15GB for a 16GB idle guest= , by using bitmap, the madvise count was reduced to 605. when using the PFNs, th= e madvise count was 3932160. It means there are quite a lot consecutive bits in the bitmap. I didn't test for a guest with heavy memory workload.=20 > IOW in real examples, do we have really large consecutive areas or are al= l > pages just completely distributed over our memory? >=20 The buddy system of Linux kernel memory management shows there should be qu= ite a lot of consecutive pages as long as there are a portion of free memory in the gue= st. If all pages just completely distributed over our memory, it means the memo= ry=20 fragmentation is very serious, the kernel has the mechanism to avoid this h= appened. In the other hand, the inflating should not happen at this time because the= guest is almost 'out of memory'. Liang > Thanks! >=20 > -- >=20 > David