From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752324AbcEYKLN (ORCPT ); Wed, 25 May 2016 06:11:13 -0400 Received: from mga09.intel.com ([134.134.136.24]:35747 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751604AbcEYKLL convert rfc822-to-8bit (ORCPT ); Wed, 25 May 2016 06:11:11 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.26,363,1459839600"; d="scan'208";a="974163790" From: "Li, Liang Z" To: "Michael S. Tsirkin" CC: "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "akpm@linux-foundation.org" , "pbonzini@redhat.com" , "dgilbert@redhat.com" , "amit.shah@redhat.com" , "kvm@vger.kernel.org" Subject: RE: [PATCH RFC kernel] balloon: speed up inflating/deflating process Thread-Topic: [PATCH RFC kernel] balloon: speed up inflating/deflating process Thread-Index: AQHRsn8KiIQymtzct0+t5K4Bv9fZh5/BM1gAgAaKRTD//5+jgIAB+Prg//+Fk4CAAIiU8P//g2AAABDf2cA= Date: Wed, 25 May 2016 10:10:47 +0000 Message-ID: References: <1463738386-30868-1-git-send-email-liang.z.li@intel.com> <20160520120038.GA28757@redhat.com> <20160524130041-mutt-send-email-mst@redhat.com> <20160525115419-mutt-send-email-mst@redhat.com> <20160525123546-mutt-send-email-mst@redhat.com> In-Reply-To: <20160525123546-mutt-send-email-mst@redhat.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNmY2M2IwYzktYTBjMi00YjhjLWJiY2QtZmYxOWFhYzUwMjc1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE1LjkuNi42IiwiVHJ1c3RlZExhYmVsSGFzaCI6ImdyRHRyYXZsR1NOSEpoWXh6K1hseEkwVGJtM1FzSkNLRTBPdUZZUkxBdEU9In0= x-ctpclassification: CTP_IC x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > > > > > > > > Hi MST, > > > > > > > > I have measured the performance when using a 32K page bitmap, > > > > > > Just to make sure. Do you mean a 32Kbyte bitmap? > > > Covering 1Gbyte of memory? > > Yes. > > > > > > > > > and inflate the balloon to 3GB > > > > of an idle guest with 4GB RAM. > > > > > > Should take 3 requests then, right? > > > > > > > No, we can't assign the PFN when allocating page in balloon driver, > > So the PFNs of pages allocated may be across a large range, we will > > tell the host once the pfn_max -pfn_min >= 0x40000(1GB range), so the > > requests count is most likely to be more than 3. > > > > > > Now: > > > > total inflating time: 338ms > > > > the count of virtio data transmission: 373 > > > > > > Why was this so high? I would expect 3 transmissions. > > > > I follow your suggestion: > > ---------------------------------------------------------------------- > > -------------- Suggestion to address all above comments: > > 1. allocate a bunch of pages and link them up, > > calculating the min and the max pfn. > > if max-min exceeds the allocated bitmap size, > > tell host. > > 2. limit allocated bitmap size to something reasonable. > > How about 32Kbytes? This is 256kilo bit in the map, which comes > > out to 1Giga bytes of memory in the balloon. > > ---------------------------------------------------------------------- > > --------------- Because the PFNs of the allocated pages are not linear > > increased, so 3 transmissions are impossible. > > > > > > Liang > > Interesting. How about instead of tell host, we do multiple scans, each time > ignoring pages out of range? > > for (pfn = min pfn; pfn < max pfn; pfn += 1G) { > foreach page > if page pfn < pfn || page pfn >= pfn + 1G > continue > set bit > tell host > } > That means we have to allocate/free all the requested pages first, and then tell the host. It works fine for inflating, but for deflating, because the page has been deleted from the vb-> vb_dev_info->pages, so, we have to use a struct to save the dequeued pages before calling release_pages_balloon(), I think a page bitmap is the best struct to save these pages, because it consumes less memory. And that bitmap should be large enough to save pfn 0 to max_pfn. If the above is true, then we are back to the square one. we really need a large page bitmap. Right? Liang From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Li, Liang Z" Subject: RE: [PATCH RFC kernel] balloon: speed up inflating/deflating process Date: Wed, 25 May 2016 10:10:47 +0000 Message-ID: References: <1463738386-30868-1-git-send-email-liang.z.li@intel.com> <20160520120038.GA28757@redhat.com> <20160524130041-mutt-send-email-mst@redhat.com> <20160525115419-mutt-send-email-mst@redhat.com> <20160525123546-mutt-send-email-mst@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , "virtualization@lists.linux-foundation.org" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "akpm@linux-foundation.org" , "dgilbert@redhat.com" To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20160525123546-mutt-send-email-mst@redhat.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: kvm.vger.kernel.org > > > > > > > > Hi MST, > > > > > > > > I have measured the performance when using a 32K page bitmap, > > > > > > Just to make sure. Do you mean a 32Kbyte bitmap? > > > Covering 1Gbyte of memory? > > Yes. > > > > > > > > > and inflate the balloon to 3GB > > > > of an idle guest with 4GB RAM. > > > > > > Should take 3 requests then, right? > > > > > > > No, we can't assign the PFN when allocating page in balloon driver, > > So the PFNs of pages allocated may be across a large range, we will > > tell the host once the pfn_max -pfn_min >= 0x40000(1GB range), so the > > requests count is most likely to be more than 3. > > > > > > Now: > > > > total inflating time: 338ms > > > > the count of virtio data transmission: 373 > > > > > > Why was this so high? I would expect 3 transmissions. > > > > I follow your suggestion: > > ---------------------------------------------------------------------- > > -------------- Suggestion to address all above comments: > > 1. allocate a bunch of pages and link them up, > > calculating the min and the max pfn. > > if max-min exceeds the allocated bitmap size, > > tell host. > > 2. limit allocated bitmap size to something reasonable. > > How about 32Kbytes? This is 256kilo bit in the map, which comes > > out to 1Giga bytes of memory in the balloon. > > ---------------------------------------------------------------------- > > --------------- Because the PFNs of the allocated pages are not linear > > increased, so 3 transmissions are impossible. > > > > > > Liang > > Interesting. How about instead of tell host, we do multiple scans, each time > ignoring pages out of range? > > for (pfn = min pfn; pfn < max pfn; pfn += 1G) { > foreach page > if page pfn < pfn || page pfn >= pfn + 1G > continue > set bit > tell host > } > That means we have to allocate/free all the requested pages first, and then tell the host. It works fine for inflating, but for deflating, because the page has been deleted from the vb-> vb_dev_info->pages, so, we have to use a struct to save the dequeued pages before calling release_pages_balloon(), I think a page bitmap is the best struct to save these pages, because it consumes less memory. And that bitmap should be large enough to save pfn 0 to max_pfn. If the above is true, then we are back to the square one. we really need a large page bitmap. Right? Liang From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34174) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b5VmJ-0003IX-8j for qemu-devel@nongnu.org; Wed, 25 May 2016 06:11:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b5VmD-00064x-4p for qemu-devel@nongnu.org; Wed, 25 May 2016 06:11:02 -0400 Received: from mga01.intel.com ([192.55.52.88]:23157) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b5VmC-00064n-QB for qemu-devel@nongnu.org; Wed, 25 May 2016 06:10:57 -0400 From: "Li, Liang Z" Date: Wed, 25 May 2016 10:10:47 +0000 Message-ID: References: <1463738386-30868-1-git-send-email-liang.z.li@intel.com> <20160520120038.GA28757@redhat.com> <20160524130041-mutt-send-email-mst@redhat.com> <20160525115419-mutt-send-email-mst@redhat.com> <20160525123546-mutt-send-email-mst@redhat.com> In-Reply-To: <20160525123546-mutt-send-email-mst@redhat.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH RFC kernel] balloon: speed up inflating/deflating process List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "akpm@linux-foundation.org" , "pbonzini@redhat.com" , "dgilbert@redhat.com" , "amit.shah@redhat.com" , "kvm@vger.kernel.org" > > > > > > > > Hi MST, > > > > > > > > I have measured the performance when using a 32K page bitmap, > > > > > > Just to make sure. Do you mean a 32Kbyte bitmap? > > > Covering 1Gbyte of memory? > > Yes. > > > > > > > > > and inflate the balloon to 3GB > > > > of an idle guest with 4GB RAM. > > > > > > Should take 3 requests then, right? > > > > > > > No, we can't assign the PFN when allocating page in balloon driver, > > So the PFNs of pages allocated may be across a large range, we will > > tell the host once the pfn_max -pfn_min >=3D 0x40000(1GB range), so the > > requests count is most likely to be more than 3. > > > > > > Now: > > > > total inflating time: 338ms > > > > the count of virtio data transmission: 373 > > > > > > Why was this so high? I would expect 3 transmissions. > > > > I follow your suggestion: > > ---------------------------------------------------------------------- > > -------------- Suggestion to address all above comments: > > 1. allocate a bunch of pages and link them up, > > calculating the min and the max pfn. > > if max-min exceeds the allocated bitmap size, > > tell host. > > 2. limit allocated bitmap size to something reasonable. > > How about 32Kbytes? This is 256kilo bit in the map, which comes > > out to 1Giga bytes of memory in the balloon. > > ---------------------------------------------------------------------- > > --------------- Because the PFNs of the allocated pages are not linear > > increased, so 3 transmissions are impossible. > > > > > > Liang >=20 > Interesting. How about instead of tell host, we do multiple scans, each t= ime > ignoring pages out of range? >=20 > for (pfn =3D min pfn; pfn < max pfn; pfn +=3D 1G) { > foreach page > if page pfn < pfn || page pfn >=3D pfn + 1G > continue > set bit > tell host > } >=20 That means we have to allocate/free all the requested pages first, and then= tell the host. It works fine for inflating, but for deflating, because the page has been d= eleted from the vb-> vb_dev_info->pages, so, we have to use a struct to save the dequeued pages before calling relea= se_pages_balloon(), I think a page bitmap is the best struct to save these pages, because it c= onsumes less memory. And that bitmap should be large enough to save pfn 0 to max_pfn. If the above is true, then we are back to the square one. we really need a = large page bitmap. Right? Liang