From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Li, Liang Z" Subject: Re: [RFC Design Doc]Speed up live migration by skipping free pages Date: Thu, 24 Mar 2016 14:50:56 +0000 Message-ID: References: <1458632629-4649-1-git-send-email-liang.z.li@intel.com> <20160322190530.GI2216@work-vm> <20160324012424.GB14956@linux-gk3p> <20160324090004.GA2230@work-vm> <20160324102354.GB2230@work-vm> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "rkagan@virtuozzo.com" , "linux-kernel@vger.kenel.org" , "ehabkost@redhat.com" , "kvm@vger.kernel.org" , "mst@redhat.com" , "simhan@hpe.com" , "quintela@redhat.com" , "qemu-devel@nongnu.org" , "jitendra.kolhe@hpe.com" , "mohan_parthasarathy@hpe.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , Wei Yang , "rth@twiddle.net" To: "Dr. David Alan Gilbert" Return-path: In-Reply-To: <20160324102354.GB2230@work-vm> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org > > > > >> Given the typical speed of networks; it wouldn't do too much > > > > >> harm to start sending assuming all pages are dirty and then > > > > >> when the guest finally gets around to finishing the bitmap then > > > > >> update, so it's asynchronous - and then if the guest never > > > > >> responds we don't really > > > care. > > > > > > > > > >Indeed, thanks! > > > > > > > > > > > > > This is interesting. By doing so, the threshold I mentioned in > > > > another mail is not necessary, since we can do it in parallel. > > > > > > Actually I just realised it's a little more complex; we can't sync > > > the dirty bitmap again from the guest until after we've received the > guests 'free' > > > bitmap; that's because we wouldn't know if a 'dirty' page reflected > > > that a page declared as 'free' had now been reused - so there is > > > still an ordering there. > > > > > > Dave > > > > Not very complex, we can implement like this: > > > > 1. Set all the bits in the migration_bitmap_rcu->bmap to 1 2. Clear > > all the bits in ram_list. dirty_memory[DIRTY_MEMORY_MIGRATION] > > 3. Send the get_free_page_bitmap request 4. Start to send pages to > > destination and check if the free_page_bitmap is ready > > if (is_ready) { > > filter out the free pages from migration_bitmap_rcu->bmap; > > migration_bitmap_sync(); > > } > > continue until live migration complete. > > > > > > Is that right? >=20 > The order I'm trying to understand is something like: >=20 > a) Send the get_free_page_bitmap request > b) Start sending pages > c) Reach the end of memory > [ is_ready is false - guest hasn't made free map yet ] > d) normal migration_bitmap_sync() at end of first pass > e) Carry on sending dirty pages > f) is_ready is true > f.1) filter out free pages? > f.2) migration_bitmap_sync() >=20 > It's f.1 I'm worried about. If the guest started generating the free bit= map > before (d), then a page marked as 'free' in f.1 might have become dirty > before (d) and so (f.2) doesn't set the dirty again, and so we can't filt= er out > pages in f.1. >=20 As you described, the order is incorrect. Liang > Dave >=20 > > > > Liang > > > > > > > > > > > >Liang > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56320) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aj6bI-0005GP-0i for qemu-devel@nongnu.org; Thu, 24 Mar 2016 10:51:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aj6bE-0004vI-Gj for qemu-devel@nongnu.org; Thu, 24 Mar 2016 10:51:03 -0400 Received: from mga14.intel.com ([192.55.52.115]:16092) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aj6bE-0004uY-7O for qemu-devel@nongnu.org; Thu, 24 Mar 2016 10:51:00 -0400 From: "Li, Liang Z" Date: Thu, 24 Mar 2016 14:50:56 +0000 Message-ID: References: <1458632629-4649-1-git-send-email-liang.z.li@intel.com> <20160322190530.GI2216@work-vm> <20160324012424.GB14956@linux-gk3p> <20160324090004.GA2230@work-vm> <20160324102354.GB2230@work-vm> In-Reply-To: <20160324102354.GB2230@work-vm> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [RFC Design Doc]Speed up live migration by skipping free pages List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: "rkagan@virtuozzo.com" , "linux-kernel@vger.kenel.org" , "ehabkost@redhat.com" , "kvm@vger.kernel.org" , "mst@redhat.com" , "simhan@hpe.com" , "quintela@redhat.com" , "qemu-devel@nongnu.org" , "jitendra.kolhe@hpe.com" , "mohan_parthasarathy@hpe.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , Wei Yang , "rth@twiddle.net" > > > > >> Given the typical speed of networks; it wouldn't do too much > > > > >> harm to start sending assuming all pages are dirty and then > > > > >> when the guest finally gets around to finishing the bitmap then > > > > >> update, so it's asynchronous - and then if the guest never > > > > >> responds we don't really > > > care. > > > > > > > > > >Indeed, thanks! > > > > > > > > > > > > > This is interesting. By doing so, the threshold I mentioned in > > > > another mail is not necessary, since we can do it in parallel. > > > > > > Actually I just realised it's a little more complex; we can't sync > > > the dirty bitmap again from the guest until after we've received the > guests 'free' > > > bitmap; that's because we wouldn't know if a 'dirty' page reflected > > > that a page declared as 'free' had now been reused - so there is > > > still an ordering there. > > > > > > Dave > > > > Not very complex, we can implement like this: > > > > 1. Set all the bits in the migration_bitmap_rcu->bmap to 1 2. Clear > > all the bits in ram_list. dirty_memory[DIRTY_MEMORY_MIGRATION] > > 3. Send the get_free_page_bitmap request 4. Start to send pages to > > destination and check if the free_page_bitmap is ready > > if (is_ready) { > > filter out the free pages from migration_bitmap_rcu->bmap; > > migration_bitmap_sync(); > > } > > continue until live migration complete. > > > > > > Is that right? >=20 > The order I'm trying to understand is something like: >=20 > a) Send the get_free_page_bitmap request > b) Start sending pages > c) Reach the end of memory > [ is_ready is false - guest hasn't made free map yet ] > d) normal migration_bitmap_sync() at end of first pass > e) Carry on sending dirty pages > f) is_ready is true > f.1) filter out free pages? > f.2) migration_bitmap_sync() >=20 > It's f.1 I'm worried about. If the guest started generating the free bit= map > before (d), then a page marked as 'free' in f.1 might have become dirty > before (d) and so (f.2) doesn't set the dirty again, and so we can't filt= er out > pages in f.1. >=20 As you described, the order is incorrect. Liang > Dave >=20 > > > > Liang > > > > > > > > > > > >Liang > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK