From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AE50C34031 for ; Thu, 20 Feb 2020 03:53:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D95F524656 for ; Thu, 20 Feb 2020 03:53:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=qnap.com header.i=@qnap.com header.b="Qc60ZWpK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D95F524656 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=qnap.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:35554 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4cta-0005jp-2B for qemu-devel@archiver.kernel.org; Wed, 19 Feb 2020 22:53:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43645) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4csh-0005I1-8c for qemu-devel@nongnu.org; Wed, 19 Feb 2020 22:52:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4csd-0008NG-On for qemu-devel@nongnu.org; Wed, 19 Feb 2020 22:52:07 -0500 Received: from mail-yb1-xb2a.google.com ([2607:f8b0:4864:20::b2a]:32838) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4csd-0008N2-Hg for qemu-devel@nongnu.org; Wed, 19 Feb 2020 22:52:03 -0500 Received: by mail-yb1-xb2a.google.com with SMTP id w190so1521201ybc.0 for ; Wed, 19 Feb 2020 19:52:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qnap.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Mu6xKXc8FPHMlvtLYLVO9PPXJlZjenX0u4VYVtnMibA=; b=Qc60ZWpKGy4HcO1uMJRLXfkbwwfpUVaezHr3nd9vDTIg8emmGoiG+BxDKVnEaxU+1E bfFx2taIS6NraSpGABWX5DWgQsd3JWnJZkERV/zDiqibgOQwoAFlRkvvkbC1KKK1Ocd1 wtZDiMCR8WO3Rg/L/sU2wMbpsNWbGFFXZr7zo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Mu6xKXc8FPHMlvtLYLVO9PPXJlZjenX0u4VYVtnMibA=; b=XMJ8LlCEB/VKSi3E+lrguD/ORQbISA9TPBLl/AL+O33kpzCi+PR+Q9e7Cb62fRqFG+ XoWnsLKZCXofMnYFZl103zrotWx/afDrF4jhJoDjjZ3cccxIERdR5HV8vpm26gqQEXZY 9uXMeH9kK2GnnkFMTdg4DYQPX3nB3cKu5niJ3nGyVfSl24InzMngY6iwkV8Vj9qHtoQx LzEzUkeHZb5vfGWqsVSAi3DmIiG9+FLHfySOuxpTNEKq8JrJZ9B9b9wQ3Uptu5JJ2cQl mKPB6AGFzte1gZorXeNSksQKJVor7EC/NA3czXbpCdRWO7LdSQdeCGeeT3Xp89cOcIFA Qwyw== X-Gm-Message-State: APjAAAXf6gnoUJ+1rubpe2lQvrXlhRFtvMtOf0GcaZm5DRaZoSFVtywR HvsEe6U/Bzo5pt3o518Uwmey1ApBWXXbQ3c/twjlrQ== X-Google-Smtp-Source: APXvYqwqlShPlD86X1srAn3YguryaRAJhV4TbYxuHNJv+AM1ATQZcXiJZ1SWIYrpHbv2t12mrM+XDzUx17kDLhCneec= X-Received: by 2002:a25:ba4a:: with SMTP id z10mr28122528ybj.48.1582170722706; Wed, 19 Feb 2020 19:52:02 -0800 (PST) MIME-Version: 1.0 References: <20200211174756.GA2798@work-vm> <8737854e2826400fa4d14dc408cfd947@huawei.com> <2b09c8650b944c908c0c95fefe6d759f@intel.com> <1bf96353e8e2490098a71d0d1182986a@huawei.com> <51f95ec9ed4a4cc682e63abf1414979b@intel.com> <20200213103752.GE2960@work-vm> <5d030380-76d6-67c6-39a1-82c197e320b4@intel.com> In-Reply-To: From: Daniel Cho Date: Thu, 20 Feb 2020 11:51:51 +0800 Message-ID: Subject: Re: The issues about architecture of the COLO checkpoint To: "Zhang, Chen" Content-Type: multipart/alternative; boundary="000000000000e6bf7e059ef9d3d4" X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::b2a X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "qemu-devel@nongnu.org" , Jason Wang , "Dr. David Alan Gilbert" , Zhanghailiang Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --000000000000e6bf7e059ef9d3d4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Hailiang, I have already patched the file to my branch, but there is a problem while doing migration. Here is the error message from SVM "qemu-system-x86_64: /root/download/qemu-4.1.0/memory.c:1079: memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()' failed." Do you have this problem? Best regards, Daniel Cho Daniel Cho =E6=96=BC 2020=E5=B9=B42=E6=9C=8820=E6=97= =A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=8811:49=E5=AF=AB=E9=81=93=EF=BC=9A > Hi Zhang, > > Thanks, I will configure on code for testing first. > However, if you have free time, could you please send the patch file to > us, Thanks. > > Best Regard, > Daniel Cho > > > Zhang, Chen =E6=96=BC 2020=E5=B9=B42=E6=9C=8820=E6= =97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=8811:07=E5=AF=AB=E9=81=93=EF=BC= =9A > >> >> On 2/18/2020 5:22 PM, Daniel Cho wrote: >> >> Hi Hailiang, >> Thanks for your help. If we have any problems we will contact you for >> your favor. >> >> >> Hi Zhang, >> >> " If colo-compare got a primary packet without related secondary packet >> in a certain time , it will automatically trigger checkpoint. " >> As you said, the colo-compare will trigger checkpoint, but does it need >> to limit checkpoint times? >> There is a problem about doing many checkpoints while we use fio to >> random write files. Then it will cause low throughput on PVM. >> Is this situation is normal on COLO? >> >> >> Hi Daniel, >> >> The checkpoint time is designed to be user adjustable based on user >> environment(workload/network status/business conditions...). >> >> In net/colo-compare.c >> >> /* TODO: Should be configurable */ >> #define REGULAR_PACKET_CHECK_MS 3000 >> >> If you need, I can send a patch for this issue. Make users can change th= e >> value by QMP and qemu monitor commands. >> >> Thanks >> >> Zhang Chen >> >> >> >> Best regards, >> Daniel Cho >> >> Zhang, Chen =E6=96=BC 2020=E5=B9=B42=E6=9C=8817= =E6=97=A5 =E9=80=B1=E4=B8=80 =E4=B8=8B=E5=8D=881:36=E5=AF=AB=E9=81=93=EF=BC= =9A >> >>> >>> On 2/15/2020 11:35 AM, Daniel Cho wrote: >>> >>> Hi Dave, >>> >>> Yes, I agree with you, it does need a timeout. >>> >>> >>> Hi Daniel and Dave, >>> >>> Current colo-compare already have the timeout mechanism. >>> >>> Named packet_check_timer, It will scan primary packet queue to make >>> sure all the primary packet not stay too long time. >>> >>> If colo-compare got a primary packet without related secondary packet i= n >>> a certain time , it will automatic trigger checkpoint. >>> >>> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847 >>> >>> >>> Thanks >>> >>> Zhang Chen >>> >>> >>> >>> Hi Hailiang, >>> >>> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a >>> lot of difference between your version and ours. >>> Could you give us a latest release version which is close your >>> developing code? >>> >>> Thanks. >>> >>> Regards >>> Daniel Cho >>> >>> Dr. David Alan Gilbert =E6=96=BC 2020=E5=B9=B42= =E6=9C=8813=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8B=E5=8D=886:38=E5=AF=AB=E9= =81=93=EF=BC=9A >>> >>>> * Daniel Cho (danielcho@qnap.com) wrote: >>>> > Hi Hailiang, >>>> > >>>> > 1. >>>> > OK, we will try the patch >>>> > =E2=80=9C0001-COLO-Optimize-memory-back-up-process.patch=E2=80=9D, >>>> > and thanks for your help. >>>> > >>>> > 2. >>>> > We understand the reason to compare PVM and SVM's packet. >>>> However, the >>>> > empty of SVM's packet queue might happened on setting COLO feature >>>> and SVM >>>> > broken. >>>> > >>>> > On situation 1 ( setting COLO feature ): >>>> > We could force do checkpoint after setting COLO feature finish, >>>> then it >>>> > will protect the state of PVM and SVM . As the Zhang Chen said. >>>> > >>>> > On situation 2 ( SVM broken ): >>>> > COLO will do failover for PVM, so it might not cause any wrong o= n >>>> PVM. >>>> > >>>> > However, those situations are our views, so there might be a big >>>> difference >>>> > between reality and our views. >>>> > If we have any wrong views and opinions, please let us know, and >>>> correct >>>> > us. >>>> >>>> It does need a timeout; the SVM being broken or being in a state where >>>> it never sends the corresponding packet (because of a state difference= ) >>>> can happen and COLO needs to timeout when the packet hasn't arrived >>>> after a while and trigger the checkpoint. >>>> >>>> Dave >>>> >>>> > Thanks. >>>> > >>>> > Best regards, >>>> > Daniel Cho >>>> > >>>> > Zhang, Chen =E6=96=BC 2020=E5=B9=B42=E6=9C=88= 13=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=8810:17=E5=AF=AB=E9=81=93=EF= =BC=9A >>>> > >>>> > > Add cc Jason Wang, he is a network expert. >>>> > > >>>> > > In case some network things goes wrong. >>>> > > >>>> > > >>>> > > >>>> > > Thanks >>>> > > >>>> > > Zhang Chen >>>> > > >>>> > > >>>> > > >>>> > > *From:* Zhang, Chen >>>> > > *Sent:* Thursday, February 13, 2020 10:10 AM >>>> > > *To:* 'Zhanghailiang' ; Daniel Cho >>>> < >>>> > > danielcho@qnap.com> >>>> > > *Cc:* Dr. David Alan Gilbert ; >>>> qemu-devel@nongnu.org >>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoin= t >>>> > > >>>> > > >>>> > > >>>> > > For the issue 2: >>>> > > >>>> > > >>>> > > >>>> > > COLO need use the network packets to confirm PVM and SVM in the >>>> same state, >>>> > > >>>> > > Generally speaking, we can=E2=80=99t send PVM packets without comp= ared with >>>> SVM >>>> > > packets. >>>> > > >>>> > > But to prevent jamming, I think COLO can do force checkpoint and >>>> send the >>>> > > PVM packets in this case. >>>> > > >>>> > > >>>> > > >>>> > > Thanks >>>> > > >>>> > > Zhang Chen >>>> > > >>>> > > >>>> > > >>>> > > *From:* Zhanghailiang >>>> > > *Sent:* Thursday, February 13, 2020 9:45 AM >>>> > > *To:* Daniel Cho >>>> > > *Cc:* Dr. David Alan Gilbert ; >>>> qemu-devel@nongnu.org; >>>> > > Zhang, Chen >>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoin= t >>>> > > >>>> > > >>>> > > >>>> > > Hi, >>>> > > >>>> > > >>>> > > >>>> > > 1. After re-walked through the codes, yes, you are right, >>>> actually, >>>> > > after the first migration, we will keep dirty log on in primary >>>> side, >>>> > > >>>> > > And only send the dirty pages in PVM to SVM. The ram cache in >>>> secondary >>>> > > side is always a backup of PVM, so we don=E2=80=99t have to >>>> > > >>>> > > Re-send the none-dirtied pages. >>>> > > >>>> > > The reason why the first checkpoint takes longer time is we have t= o >>>> backup >>>> > > the whole VM=E2=80=99s ram into ram cache, that is colo_init_ram_c= ache(). >>>> > > >>>> > > It is time consuming, but I have optimized in the second patch >>>> > > =E2=80=9C0001-COLO-Optimize-memory-back-up-process.patch=E2=80=9D = which you can >>>> find in my >>>> > > previous reply. >>>> > > >>>> > > >>>> > > >>>> > > Besides, I found that, In my previous reply =E2=80=9CWe can only c= opy the >>>> pages >>>> > > that dirtied by PVM and SVM in last checkpoint.=E2=80=9D, >>>> > > >>>> > > We have done this optimization in current upstream codes. >>>> > > >>>> > > >>>> > > >>>> > > 2=EF=BC=8EI don=E2=80=99t quite understand this question. For COLO= , we always need >>>> both >>>> > > network packets of PVM=E2=80=99s and SVM=E2=80=99s to compare befo= re send this >>>> packets to >>>> > > client. >>>> > > >>>> > > It depends on this to decide whether or not PVM and SVM are in sam= e >>>> state. >>>> > > >>>> > > >>>> > > >>>> > > Thanks, >>>> > > >>>> > > hailiang >>>> > > >>>> > > >>>> > > >>>> > > *From:* Daniel Cho [mailto:danielcho@qnap.com = ] >>>> > > *Sent:* Wednesday, February 12, 2020 4:37 PM >>>> > > *To:* Zhang, Chen >>>> > > *Cc:* Zhanghailiang ; Dr. David >>>> Alan >>>> > > Gilbert ; qemu-devel@nongnu.org >>>> > > *Subject:* Re: The issues about architecture of the COLO checkpoin= t >>>> > > >>>> > > >>>> > > >>>> > > Hi Hailiang, >>>> > > >>>> > > >>>> > > >>>> > > Thanks for your replaying and explain in detail. >>>> > > >>>> > > We will try to use the attachments to enhance memory copy. >>>> > > >>>> > > >>>> > > >>>> > > However, we have some questions for your replying. >>>> > > >>>> > > >>>> > > >>>> > > 1. As you said, "for each checkpoint, we have to send the whole >>>> PVM's >>>> > > pages To SVM", why the only first checkpoint will takes more pause >>>> time? >>>> > > >>>> > > In our observing, the first checkpoint will take more time for >>>> pausing, >>>> > > then other checkpoints will takes a few time for pausing. Does it >>>> means >>>> > > only the first checkpoint will send the whole pages to SVM, and th= e >>>> other >>>> > > checkpoints send the dirty pages to SVM for reloading? >>>> > > >>>> > > >>>> > > >>>> > > 2. We notice the COLO-COMPARE component will stuck the packet unti= l >>>> > > receive packets from PVM and SVM, as this rule, when we add the >>>> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it >>>> is an >>>> > > other issue to make PVM stuck while setting COLO feature. With thi= s >>>> issue, >>>> > > could we let colo-compare to pass the PVM's packet when the SVM's >>>> packet >>>> > > queue is empty? Then, the PVM's network won't stock, and "if PVM >>>> runs >>>> > > firstly, it still need to wait for The network packets from SVM to >>>> > > compare before send it to client side" won't happened either. >>>> > > >>>> > > >>>> > > >>>> > > Best regard, >>>> > > >>>> > > Daniel Cho >>>> > > >>>> > > >>>> > > >>>> > > Zhang, Chen =E6=96=BC 2020=E5=B9=B42=E6=9C= =8812=E6=97=A5 =E9=80=B1=E4=B8=89 =E4=B8=8B=E5=8D=881:45=E5=AF=AB=E9=81=93= =EF=BC=9A >>>> > > >>>> > > >>>> > > >>>> > > > -----Original Message----- >>>> > > > From: Zhanghailiang >>>> > > > Sent: Wednesday, February 12, 2020 11:18 AM >>>> > > > To: Dr. David Alan Gilbert ; Daniel Cho >>>> > > > ; Zhang, Chen >>>> > > > Cc: qemu-devel@nongnu.org >>>> > > > Subject: RE: The issues about architecture of the COLO checkpoin= t >>>> > > > >>>> > > > Hi, >>>> > > > >>>> > > > Thank you Dave, >>>> > > > >>>> > > > I'll reply here directly. >>>> > > > >>>> > > > -----Original Message----- >>>> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] >>>> > > > Sent: Wednesday, February 12, 2020 1:48 AM >>>> > > > To: Daniel Cho ; chen.zhang@intel.com; >>>> > > > Zhanghailiang >>>> > > > Cc: qemu-devel@nongnu.org >>>> > > > Subject: Re: The issues about architecture of the COLO checkpoin= t >>>> > > > >>>> > > > >>>> > > > cc'ing in COLO people: >>>> > > > >>>> > > > >>>> > > > * Daniel Cho (danielcho@qnap.com) wrote: >>>> > > > > Hi everyone, >>>> > > > > We have some issues about setting COLO feature. Hope >>>> somebody >>>> > > > > could give us some advice. >>>> > > > > >>>> > > > > Issue 1: >>>> > > > > We dynamic to set COLO feature for PVM(2 core, 16G >>>> memory), but >>>> > > > > the Primary VM will pause a long time(based on memory size) fo= r >>>> > > > > waiting SVM start. Does it have any idea to reduce the pause >>>> time? >>>> > > > > >>>> > > > >>>> > > > Yes, we do have some ideas to optimize this downtime. >>>> > > > >>>> > > > The main problem for current version is, for each checkpoint, we >>>> have to >>>> > > > send the whole PVM's pages >>>> > > > To SVM, and then copy the whole VM's state into SVM from ram >>>> cache, in >>>> > > > this process, we need both of them be paused. >>>> > > > Just as you said, the downtime is based on memory size. >>>> > > > >>>> > > > So firstly, we need to reduce the sending data while do >>>> checkpoint, >>>> > > actually, >>>> > > > we can migrate parts of PVM's dirty pages in background >>>> > > > While both of VMs are running. And then we load these pages into >>>> ram >>>> > > > cache (backup memory) in SVM temporarily. While do checkpoint, >>>> > > > We just send the last dirty pages of PVM to slave side and then >>>> copy the >>>> > > ram >>>> > > > cache into SVM. Further on, we don't have >>>> > > > To send the whole PVM's dirty pages, we can only send the pages >>>> that >>>> > > > dirtied by PVM or SVM during two checkpoints. (Because >>>> > > > If one page is not dirtied by both PVM and SVM, the data of this >>>> pages >>>> > > will >>>> > > > keep same in SVM, PVM, backup memory). This method can reduce >>>> > > > the time that consumed in sending data. >>>> > > > >>>> > > > For the second problem, we can reduce the memory copy by two >>>> methods, >>>> > > > first one, we don't have to copy the whole pages in ram cache, >>>> > > > We can only copy the pages that dirtied by PVM and SVM in last >>>> > > checkpoint. >>>> > > > Second, we can use userfault missing function to reduce the >>>> > > > Time consumed in memory copy. (For the second time, in theory, w= e >>>> can >>>> > > > reduce time consumed in memory into ms level). >>>> > > > >>>> > > > You can find the first optimization in attachment, it is based o= n >>>> an old >>>> > > qemu >>>> > > > version (qemu-2.6), it should not be difficult to rebase it >>>> > > > Into master or your version. And please feel free to send the ne= w >>>> > > version if >>>> > > > you want into community ;) >>>> > > > >>>> > > > >>>> > > >>>> > > Thanks Hailiang! >>>> > > By the way, Do you have time to push the patches to upstream? >>>> > > I think this is a better and faster option. >>>> > > >>>> > > Thanks >>>> > > Zhang Chen >>>> > > >>>> > > > > >>>> > > > > Issue 2: >>>> > > > > In >>>> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503= , >>>> > > > > could we move start_vm() before Line 488? Because at first >>>> checkpoint >>>> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while. >>>> > > > > >>>> > > > >>>> > > > No, that makes no sense, because if PVM runs firstly, it still >>>> need to >>>> > > wait for >>>> > > > The network packets from SVM to compare before send it to client >>>> side. >>>> > > > >>>> > > > >>>> > > > Thanks, >>>> > > > Hailiang >>>> > > > >>>> > > > > We set the COLO feature on running VM, so we hope the >>>> running VM >>>> > > > > could continuous service for users. >>>> > > > > Do you have any suggestions for those issues? >>>> > > > > >>>> > > > > Best regards, >>>> > > > > Daniel Cho >>>> > > > -- >>>> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> > > >>>> > > >>>> -- >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> >>>> --000000000000e6bf7e059ef9d3d4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Hailiang,=C2=A0

I have al= ready patched the file to my branch, but there is a problem while doing mig= ration.
Here is the error=C2=A0message from SVM=C2=A0
"qe= mu-system-x86_64: /root/download/qemu-4.1.0/memory.c:1079: memory_region_tr= ansaction_commit: Assertion `qemu_mutex_iothread_locked()' failed."= ;

Do you have this=C2=A0problem?

Best regards,
Daniel Cho

Daniel Cho <danielcho@qnap.com> =E6=96=BC 2020=E5=B9= =B42=E6=9C=8820=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=8811:49=E5=AF= =AB=E9=81=93=EF=BC=9A
Hi Zhang,

Thanks= , I will configure on code for testing first.=C2=A0=C2=A0
However= , if you have free time, could you please send the patch file to us, Thanks= .

Best Regard,
Daniel Cho

=

Zhang, Chen <chen.zhang@intel.com> =E6=96=BC 2020=E5=B9=B42=E6=9C=8820=E6=97= =A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=8811:07=E5=AF=AB=E9=81=93=EF=BC=9A
=20 =20 =20


On 2/18/2020 5:22 PM, Daniel Cho wrote:
=20
Hi Hailiang,=C2=A0
Thanks for your help. If we have any problems we will contact you for your favor.


Hi Zhang,=C2=A0

" If colo-compare got a primary packet without related secondary packet in a certain time , it will automatically trigger checkpoint.=C2=A0 "
As you said, the colo-compare will trigger checkpoint, but does it need to limit checkpoint=C2=A0times?
There is a problem about doing many checkpoints while we use fio to random write files. Then it will cause low throughput on PVM.
Is this situation is normal=C2=A0on=C2=A0COLO?


Hi Daniel,

The checkpoint time is designed to be user adjustable based on user environment(workload/network status/business conditions...).

In net/colo-compare.c

/* TODO: Should be configurable */
#define REGULAR_PACKET_CHECK_MS 3000

If you need, I can send a patch for this issue. Make users can change the value by QMP and qemu monitor commands.

Thanks

Zhang Chen



Best regards,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com>= =E6=96=BC 2020=E5=B9=B42=E6=9C=8817=E6=97=A5 =E9=80=B1=E4=B8=80 =E4=B8=8B= =E5=8D=881:36=E5=AF=AB=E9=81=93=EF=BC=9A


On 2/15/2020 11:35 AM, Daniel Cho wrote:
Hi Dave,=C2=A0

Yes, I agree with you, it does need a timeout.


Hi Daniel and Dave,

Current colo-compare already have the timeout mechanism.

Named packet_check_timer,=C2=A0 It will scan primary packe= t queue to make sure all the primary packet not stay too long time.

If colo-compare got a primary packet without related secondary packet in a certain time , it will automatic trigger checkpoint.

https://github.com/qemu/qemu/blob/mast= er/net/colo-compare.c#L847


Thanks

Zhang Chen



Hi Hailiang,=C2=A0

We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot of difference=C2=A0 between your version and ours.
Could you give us a latest release version which is close your developing code?

Thanks.=C2=A0

Regards
Daniel Cho

Dr. David Alan Gilbert <dgilbert@redhat.com> =E6=96=BC 2020=E5=B9=B42=E6=9C=8813=E6=97=A5 =E9=80=B1= =E5=9B=9B =E4=B8=8B=E5=8D=886:38=E5=AF=AB=E9=81=93=EF=BC=9A
* Daniel Cho (danielcho@qnap.com) wrote:
> Hi Hailiang,
>
> 1.
>=C2=A0 =C2=A0 =C2=A0OK, we will try the patch
> =E2=80=9C0001-COLO-Optimize-memory-back-up-process.patc= h=E2=80=9D,
> and thanks for your help.
>
> 2.
>=C2=A0 =C2=A0 =C2=A0We understand the reason to com= pare PVM and SVM's packet. However, the
> empty of SVM's packet queue might happened on setting COLO feature and SVM
> broken.
>
> On situation 1 ( setting COLO feature ):
>=C2=A0 =C2=A0 =C2=A0We could force do checkpoint af= ter setting COLO feature finish, then it
> will protect the state of PVM and SVM . As the Zhang Chen said.
>
> On situation 2 ( SVM broken ):
>=C2=A0 =C2=A0 =C2=A0COLO will do failover for PVM, = so it might not cause any wrong on PVM.
>
> However, those situations are our views, so there might be a big difference
> between reality and our views.
> If we have any wrong views and opinions, please let us know, and correct
> us.

It does need a timeout; the SVM being broken or being in a state where
it never sends the corresponding packet (because of a state difference)
can happen and COLO needs to timeout when the packet hasn't arrived
after a while and trigger the checkpoint.

Dave

> Thanks.
>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com> =E6=96=BC 2020=E5=B9=B42=E6=9C=8813=E6=97=A5 =E9=80=B1= =E5=9B=9B =E4=B8=8A=E5=8D=8810:17=E5=AF=AB=E9=81=93=EF=BC=9A
>
> > Add cc Jason Wang, he is a network expert. > >
> > In case some network things goes wrong.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhang, Chen
> > *Sent:* Thursday, February 13, 2020 10:10 AM
> > *To:* 'Zhanghailiang' <zhang.zhanghailian= g@huawei.com>; Daniel Cho <
> > danielcho@qnap.com>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > For the issue 2:
> >
> >
> >
> > COLO need use the network packets to confirm PVM and SVM in the same state,
> >
> > Generally speaking, we can=E2=80=99t send PVM packets without compared with SVM
> > packets.
> >
> > But to prevent jamming, I think COLO can do force checkpoint and send the
> > PVM packets in this case.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhanghailiang <zhang.zhanghailiang@huawei= .com>
> > *Sent:* Thursday, February 13, 2020 9:45 AM
> > *To:* Daniel Cho <danielcho@qnap.com>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org;
> > Zhang, Chen <chen.zhang@intel.com>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi,
> >
> >
> >
> > 1.=C2=A0 =C2=A0 =C2=A0 =C2=A0After re-walked = through the codes, yes, you are right, actually,
> > after the first migration, we will keep dirty log on in primary side,
> >
> > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > side is always a backup of PVM, so we don=E2=80=99t have to
> >
> > Re-send the none-dirtied pages.
> >
> > The reason why the first checkpoint takes longer time is we have to backup
> > the whole VM=E2=80=99s ram into ram cache, th= at is colo_init_ram_cache().
> >
> > It is time consuming, but I have optimized in the second patch
> > =E2=80=9C0001-COLO-Optimize-memory-back-up-process.patc= h=E2=80=9D which you can find in my
> > previous reply.
> >
> >
> >
> > Besides, I found that, In my previous reply =E2=80=9CWe can only copy the pages
> > that dirtied by PVM and SVM in last checkpoint.=E2=80=9D,
> >
> > We have done this optimization in current upstream codes.
> >
> >
> >
> > 2=EF=BC=8EI don=E2=80=99t quite understand th= is question. For COLO, we always need both
> > network packets of PVM=E2=80=99s and SVM=E2= =80=99s to compare before send this packets to
> > client.
> >
> > It depends on this to decide whether or not PVM and SVM are in same state.
> >
> >
> >
> > Thanks,
> >
> > hailiang
> >
> >
> >
> > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>] > > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > *To:* Zhang, Chen <chen.zhang@intel.com>
> > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.c= om>; Dr. David Alan
> > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > Thanks for your replaying and explain in detail.
> >
> > We will try to use the attachments to enhance memory copy.
> >
> >
> >
> > However, we have some questions for your replying.
> >
> >
> >
> > 1.=C2=A0 As you said, "for each checkpoi= nt, we have to send the whole PVM's
> > pages To SVM", why the only first checkpoint will takes more pause time?
> >
> > In our observing, the first checkpoint will take more time for pausing,
> > then other checkpoints will takes a few time for pausing. Does it means
> > only the first checkpoint will send the whole pages to SVM, and the other
> > checkpoints send the dirty pages to SVM for reloading?
> >
> >
> >
> > 2. We notice the COLO-COMPARE component will stuck the packet until
> > receive packets from PVM and SVM, as this rule, when we add the
> > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > other issue to make PVM stuck while setting COLO feature. With this issue,
> > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > firstly, it still need to wait for The network packets from SVM to
> > compare before send it to client side" won't happened either.
> >
> >
> >
> > Best regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> =E6=96=BC 2020=E5=B9=B42=E6=9C=8812=E6=97=A5 =E9=80=B1= =E4=B8=89 =E4=B8=8B=E5=8D=881:45=E5=AF=AB=E9=81=93=EF=BC=9A
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhanghailiang <zhang.zhanghailiang@hua= wei.com>
> > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > To: Dr. David Alan Gilbert <dgilbert@redhat.com&g= t;; Daniel Cho
> > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > > Cc: qemu-devel@nongnu.org
> > > Subject: RE: The issues about architecture of the COLO checkpoint
> > >
> > > Hi,
> > >
> > > Thank you Dave,
> > >
> > > I'll reply here directly.
> > >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > > Zhanghailiang <zhang.zhanghailiang@huawei.co= m>
> > > Cc: qemu-devel@nongnu.org
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > > cc'ing in COLO people:
> > >
> > >
> > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > Hi everyone,
> > > >=C2=A0 =C2=A0 =C2=A0 We have some is= sues about setting COLO feature. Hope somebody
> > > > could give us some advice.
> > > >
> > > > Issue 1:
> > > >=C2=A0 =C2=A0 =C2=A0 We dynamic to s= et COLO feature for PVM(2 core, 16G memory),=C2=A0 but
> > > > the Primary VM will pause a long time(based on memory size) for
> > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > >
> > >
> > > Yes, we do have some ideas to optimize this downtime.
> > >
> > > The main problem for current version is, for each checkpoint, we have to
> > > send the whole PVM's pages
> > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > this process, we need both of them be paused.
> > > Just as you said, the downtime is based on memory size.
> > >
> > > So firstly, we need to reduce the sending data while do checkpoint,
> > actually,
> > > we can migrate parts of PVM's dirty pages in background
> > > While both of VMs are running. And then we load these pages into ram
> > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > We just send the last dirty pages of PVM to slave side and then copy the
> > ram
> > > cache into SVM. Further on, we don't have
> > > To send the whole PVM's dirty pages, we can only send the pages that
> > > dirtied by PVM or SVM during two checkpoints. (Because
> > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > will
> > > keep same in SVM, PVM, backup memory). This method can reduce
> > > the time that consumed in sending data.
> > >
> > > For the second problem, we can reduce the memory copy by two methods,
> > > first one, we don't have to copy the whole pages in ram cache,
> > > We can only copy the pages that dirtied by PVM and SVM in last
> > checkpoint.
> > > Second, we can use userfault missing function to reduce the
> > > Time consumed in memory copy. (For the second time, in theory, we can
> > > reduce time consumed in memory into ms level).
> > >
> > > You can find the first optimization in attachment, it is based on an old
> > qemu
> > > version (qemu-2.6), it should not be difficult to rebase it
> > > Into master or your version. And please feel free to send the new
> > version if
> > > you want into community ;)
> > >
> > >
> >
> > Thanks Hailiang!
> > By the way, Do you have time to push the patches to upstream?
> > I think this is a better and faster option.
> >
> > Thanks
> > Zhang Chen
> >
> > > >
> > > > Issue 2:
> > > >=C2=A0 =C2=A0 =C2=A0 In
> > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > >
> > >
> > > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for
> > > The network packets from SVM to compare before send it to client side.
> > >
> > >
> > > Thanks,
> > > Hailiang
> > >
> > > >=C2=A0 =C2=A0 =C2=A0 We set the COLO= feature on running VM, so we hope the running VM
> > > > could continuous service for users.
> > > > Do you have any suggestions for those issues?
> > > >
> > > > Best regards,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--000000000000e6bf7e059ef9d3d4--