From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dr. David Alan Gilbert" Subject: Re: [RFC Design Doc]Speed up live migration by skipping free pages Date: Tue, 19 Apr 2016 20:05:42 +0100 Message-ID: <20160419190542.GH2255@work-vm> References: <20160324102354.GB2230@work-vm> <20160324165530-mutt-send-email-mst@redhat.com> <20160324175503-mutt-send-email-mst@redhat.com> <20160324181031-mutt-send-email-mst@redhat.com> <20160324174933.GA11662@work-vm> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "Michael S. Tsirkin" , Wei Yang , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "linux-kernel@vger.kenel.org" , "pbonzini@redhat.com" , "rth@twiddle.net" , "ehabkost@redhat.com" , "amit.shah@redhat.com" , "quintela@redhat.com" , "mohan_parthasarathy@hpe.com" , "jitendra.kolhe@hpe.com" , "simhan@hpe.com" , "rkagan@virtuozzo.com" , "riel@redhat.com" To: "Li, Liang Z" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:47836 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933162AbcDSTFt (ORCPT ); Tue, 19 Apr 2016 15:05:49 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: * Li, Liang Z (liang.z.li@intel.com) wrote: > Hi Dave, > > I am now working on how to benefit post-copy by skipping the free pages, > and I remember you have said we should let the destination know the info > of free pages so as to avoid request the free pages from the source. > > We have two solutions: > > a. send the migration dirty page bitmap to destination before post > copy start, so the destination can decide whether to request the pages or > place zero pages by checking the migration dirty page bitmap. The advantage > is that we can avoid sending the free pages. the disadvantage is that we have > to send extra data to destination. > > b. Check the page request on the source side, if it's not a dirty page, send a zero > page header to the destination. > > What's your opinion about them? (b) is certainly simpler - and requires no changes on the destination side or the protocol. If you then decided to add stuff to send the dirty page bit map later you could do. However, there are some other problems to figure out: 1) The source side quits when it thinks it's sent all pages; when is your source going to quit? If it quits while the destination still has unfulfilled pages then the destination will fail. 2) I sent a 'discard' bitmap of pages for the destination to unmap just at the change into postcopy; so I'm already sending one bitmap; this is for pages that have been changed since they were first sent but not yet resent. Be careful about how any changes you make interact with the generation of that bitmap. 3) It's potentially very slow if the destination has to keep requesting blank pages. Essentially what you're suggesting for (a) is a way to send a compressed set of 'page is zero' messages based on a bitmap, and you're worried about the time to send it - which I think is where we started the conversation about time to deal with zeros :-). Two ways to think of that are: 4) I already send one bitmap - so you're only doubling it in theory; I originally used a sparse bitmap but the suggestion was it was more complex than needed and it turned into more of a run-length encoding. 5) You're worried it would increase the downtime as you send the bitmap; however if you implement (b) as well as (a) then you can send the data for (a) after the destination is running and not increase the downtime. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49131) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1asayB-0004YK-Gk for qemu-devel@nongnu.org; Tue, 19 Apr 2016 15:05:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1asay6-00019v-Dm for qemu-devel@nongnu.org; Tue, 19 Apr 2016 15:05:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55669) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1asay5-00019r-NR for qemu-devel@nongnu.org; Tue, 19 Apr 2016 15:05:50 -0400 Date: Tue, 19 Apr 2016 20:05:42 +0100 From: "Dr. David Alan Gilbert" Message-ID: <20160419190542.GH2255@work-vm> References: <20160324102354.GB2230@work-vm> <20160324165530-mutt-send-email-mst@redhat.com> <20160324175503-mutt-send-email-mst@redhat.com> <20160324181031-mutt-send-email-mst@redhat.com> <20160324174933.GA11662@work-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] [RFC Design Doc]Speed up live migration by skipping free pages List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Li, Liang Z" Cc: "Michael S. Tsirkin" , Wei Yang , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "linux-kernel@vger.kenel.org" , "pbonzini@redhat.com" , "rth@twiddle.net" , "ehabkost@redhat.com" , "amit.shah@redhat.com" , "quintela@redhat.com" , "mohan_parthasarathy@hpe.com" , "jitendra.kolhe@hpe.com" , "simhan@hpe.com" , "rkagan@virtuozzo.com" , "riel@redhat.com" * Li, Liang Z (liang.z.li@intel.com) wrote: > Hi Dave, > > I am now working on how to benefit post-copy by skipping the free pages, > and I remember you have said we should let the destination know the info > of free pages so as to avoid request the free pages from the source. > > We have two solutions: > > a. send the migration dirty page bitmap to destination before post > copy start, so the destination can decide whether to request the pages or > place zero pages by checking the migration dirty page bitmap. The advantage > is that we can avoid sending the free pages. the disadvantage is that we have > to send extra data to destination. > > b. Check the page request on the source side, if it's not a dirty page, send a zero > page header to the destination. > > What's your opinion about them? (b) is certainly simpler - and requires no changes on the destination side or the protocol. If you then decided to add stuff to send the dirty page bit map later you could do. However, there are some other problems to figure out: 1) The source side quits when it thinks it's sent all pages; when is your source going to quit? If it quits while the destination still has unfulfilled pages then the destination will fail. 2) I sent a 'discard' bitmap of pages for the destination to unmap just at the change into postcopy; so I'm already sending one bitmap; this is for pages that have been changed since they were first sent but not yet resent. Be careful about how any changes you make interact with the generation of that bitmap. 3) It's potentially very slow if the destination has to keep requesting blank pages. Essentially what you're suggesting for (a) is a way to send a compressed set of 'page is zero' messages based on a bitmap, and you're worried about the time to send it - which I think is where we started the conversation about time to deal with zeros :-). Two ways to think of that are: 4) I already send one bitmap - so you're only doubling it in theory; I originally used a sparse bitmap but the suggestion was it was more complex than needed and it turned into more of a run-length encoding. 5) You're worried it would increase the downtime as you send the bitmap; however if you implement (b) as well as (a) then you can send the data for (a) after the destination is running and not increase the downtime. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK