From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54305) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z0pEV-0007r1-39 for qemu-devel@nongnu.org; Fri, 05 Jun 2015 06:52:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z0pER-0001cN-1X for qemu-devel@nongnu.org; Fri, 05 Jun 2015 06:52:15 -0400 Received: from mx2.parallels.com ([199.115.105.18]:60046) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z0pEQ-0001Zq-Ot for qemu-devel@nongnu.org; Fri, 05 Jun 2015 06:52:10 -0400 Message-ID: <55717F4E.2090301@odin.com> Date: Fri, 5 Jun 2015 13:51:58 +0300 From: "Denis V. Lunev" MIME-Version: 1.0 References: <1431531007-10269-1-git-send-email-vsementsov@parallels.com> <555DE2D7.2020200@parallels.com> <555DE431.60703@openvz.org> <555E0B5D.5060302@redhat.com> <556487BF.2000809@openvz.org> <55648887.7080804@odin.com> <55677603.8040506@redhat.com> <556780E8.6000700@odin.com> <556E2B87.5090302@redhat.com> In-Reply-To: <556E2B87.5090302@redhat.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: John Snow , dgilbert@redhat.com Cc: kwolf@redhat.com, peter.maydell@linaro.org, quintela@redhat.com, qemu-devel@nongnu.org, Vladimir Sementsov-Ogievskiy , stefanha@redhat.com, amit.shah@redhat.com, pbonzini@redhat.com On 03/06/15 01:17, John Snow wrote: > > On 05/28/2015 04:56 PM, Denis V. Lunev wrote: >> On 28/05/15 23:09, John Snow wrote: >>> On 05/26/2015 10:51 AM, Denis V. Lunev wrote: >>>> On 26/05/15 17:48, Denis V. Lunev wrote: >>>>> On 21/05/15 19:44, John Snow wrote: >>>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote: >>>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote: >>>>>>>> Hi all. >>>>>>>> >>>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) >>>>>>>> about >>>>>>>> how to drop meta bitmaps and make things easer. >>>>>>>> >>>>>>>> method: >>>>>>>> >>>>>>>>> start migration >>>>>>>> disk and memory are migrated, but not dirty bitmaps. >>>>>>>>> stop vm >>>>>>>> create all necessary bitmaps in destination vm (empty, but with same >>>>>>>> names and granularities and enabled flag) >>>>>>>>> start destination vm >>>>>>>> empty bitmaps are tracking now >>>>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps >>>>>>>> in destination >>>>>>>> while bitmaps are migrating, they should be in some kind of >>>>>>>> 'inconsistent' state. >>>>>>>> so, we can't start backup or other migration while bitmaps are >>>>>>>> migrating, but vm is already _running_ on destination. >>>>>>>> >>>>>>>> what do you think about it? >>>>>>>> >>>>>>> the description is a bit incorrect >>>>>>> >>>>>>> - start migration process, perform memory and disk migration >>>>>>> as usual. VM is still executed at source >>>>>>> - start VM on target. VM on source should be on pause as usual, >>>>>>> do not finish migration process. Running VM on target "writes" >>>>>>> normally setting dirty bits as usual >>>>>>> - copy active dirty bitmaps from source to target. This is safe >>>>>>> as VM on source is not running >>>>>>> - "OR" copied bitmaps with ones running on target >>>>>>> - finish migration process (stop source VM). >>>>>>> >>>>>>> Downtime will not be increased due to dirty bitmaps with this >>>>>>> approach, migration process is very simple - plain data copy. >>>>>>> >>>>>>> Regards, >>>>>>> Den >>>>>>> >>>>>> I was actually just discussing the live migration approach a little >>>>>> bit >>>>>> ago with Stefan, trying to decide on the "right" packet format (The >>>>>> only >>>>>> two patches I haven't ACKed yet are ones in which we need to choose a >>>>>> send size) and we decided that 1KiB chunk sends would be >>>>>> appropriate for >>>>>> live migration. >>>>>> >>>>>> I think I'm okay with that method, but obviously this approach >>>>>> outlined >>>>>> here would also work very well and would avoid meta bitmaps, chunk >>>>>> sizes, migration tuning, convergence questions, etc etc etc. >>>>>> >>>>>> You'd need to add a new status to the bitmap on the target (maybe >>>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a >>>>>> backup operation without preventing it from recording new writes. >>>>>> >>>>>> My only concern is how easy it will be to work this into the migration >>>>>> workflow. >>>>>> >>>>>> It would require some sort of "post-migration" ternary phase, I >>>>>> suppose, >>>>>> for devices/data that can be transferred after the VM starts -- and I >>>>>> suspect we'll be the only use of that phase for now. >>>>>> >>>>>> David, what are your thoughts, here? Would you prefer Vladimir and I >>>>>> push forward on the live migration approach, or add a new post-hoc >>>>>> phase? This approach might be simpler on the block layer, but I >>>>>> would be >>>>>> rather upset if he scrapped his entire series for the second time for >>>>>> another approach that also didn't get accepted. >>>>>> >>>>>> --js >>>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates. >>>>> There is not much interest at the moment. I think that we could >>>>> implement this later in 2.5 etc... >>>>> >>>>> Regards, >>>>> Den >>>> oops. I have written something strange. Anyway, I think that for >>>> now we should proceed with this patchset to fit QEMU 2.4 dates. >>>> The implementation with additional stage (my proposal) could be >>>> added later, f.e. in 2.5 as I do not see much interest from migration >>>> gurus. >>>> >>>> In this case the review will take a ... lot of time. >>>> >>>> Regards, >>>> Den >>>> >>> That sounds good to me. I think this solution is workable for 2.4, and >>> we can begin working on a post-migration phase for the future to help >>> simplify our cases a lot. >>> >>> I have been out sick much of this week, so apologies in my lack of >>> fervor getting this series upstream recently. >>> >>> --js >> no prob :) > Had a chat with Stefan about this approach and apparently that's what > the postcopy migration patches on-list are all about. > > Stefan brought up the point of post-hoc reliability: It's possible to > transfer control to the new VM and then lose your link, making migration > completion impossible. Adding a post-copy phase to our existing live > migration is a non-starter, because it introduces unfairly this > unreliability to the existing system. > > However, we can make this idea work for migrations started via the > post-copy mechanism, because the entire migration already carries that > known risk of completion failure. > > It seems like the likely outcome though is that migrations will be able > to be completed with either mechanism in the future: either up-front > migration or post-copy migration. In that light, it seems we won't be > able to fully rid ourselves of the meta_bitmap idea, making the > post-copy idea here not too useful in culling our complexity, since > we'll have to support the current standard live migration anyway. > > So I have reviewed the current set of patches under the assumption that > it seems like the right way to go for 2.4 and beyond. > > Thank you! > --js ok! seems reasonable. Sorry for late answer. Regards, Den