Re: The issues about architecture of the COLO checkpoint

["Zhang, Chen" <chen.zhang@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 13842 bytes --]


On 2/15/2020 11:35 AM, Daniel Cho wrote:
> Hi Dave,
>
> Yes, I agree with you, it does need a timeout.


Hi Daniel and Dave,

Current colo-compare already have the timeout mechanism.

Named packet_check_timer,  It will scan primary packet queue to make 
sure all the primary packet not stay too long time.

If colo-compare got a primary packet without related secondary packet in 
a certain time , it will automatic trigger checkpoint.

https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847


Thanks

Zhang Chen


>
> Hi Hailiang,
>
> We base on qemu-4.1.0 for using COLO feature, in your patch, we found 
> a lot of difference  between your version and ours.
> Could you give us a latest release version which is close your 
> developing code?
>
> Thanks.
>
> Regards
> Daniel Cho
>
> Dr. David Alan Gilbert <dgilbert@redhat.com 
> <mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
>
>     * Daniel Cho (danielcho@qnap.com <mailto:danielcho@qnap.com>) wrote:
>     > Hi Hailiang,
>     >
>     > 1.
>     >     OK, we will try the patch
>     > “0001-COLO-Optimize-memory-back-up-process.patch”,
>     > and thanks for your help.
>     >
>     > 2.
>     >     We understand the reason to compare PVM and SVM's packet.
>     However, the
>     > empty of SVM's packet queue might happened on setting COLO
>     feature and SVM
>     > broken.
>     >
>     > On situation 1 ( setting COLO feature ):
>     >     We could force do checkpoint after setting COLO feature
>     finish, then it
>     > will protect the state of PVM and SVM . As the Zhang Chen said.
>     >
>     > On situation 2 ( SVM broken ):
>     >     COLO will do failover for PVM, so it might not cause any
>     wrong on PVM.
>     >
>     > However, those situations are our views, so there might be a big
>     difference
>     > between reality and our views.
>     > If we have any wrong views and opinions, please let us know, and
>     correct
>     > us.
>
>     It does need a timeout; the SVM being broken or being in a state where
>     it never sends the corresponding packet (because of a state
>     difference)
>     can happen and COLO needs to timeout when the packet hasn't arrived
>     after a while and trigger the checkpoint.
>
>     Dave
>
>     > Thanks.
>     >
>     > Best regards,
>     > Daniel Cho
>     >
>     > Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     於 2020年2月13日 週四 上午10:17寫道：
>     >
>     > > Add cc Jason Wang, he is a network expert.
>     > >
>     > > In case some network things goes wrong.
>     > >
>     > >
>     > >
>     > > Thanks
>     > >
>     > > Zhang Chen
>     > >
>     > >
>     > >
>     > > *From:* Zhang, Chen
>     > > *Sent:* Thursday, February 13, 2020 10:10 AM
>     > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
>     > > danielcho@qnap.com <mailto:danielcho@qnap.com>>
>     > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>     <mailto:qemu-devel@nongnu.org>
>     > > *Subject:* RE: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > For the issue 2:
>     > >
>     > >
>     > >
>     > > COLO need use the network packets to confirm PVM and SVM in
>     the same state,
>     > >
>     > > Generally speaking, we can’t send PVM packets without compared
>     with SVM
>     > > packets.
>     > >
>     > > But to prevent jamming, I think COLO can do force checkpoint
>     and send the
>     > > PVM packets in this case.
>     > >
>     > >
>     > >
>     > > Thanks
>     > >
>     > > Zhang Chen
>     > >
>     > >
>     > >
>     > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > *Sent:* Thursday, February 13, 2020 9:45 AM
>     > > *To:* Daniel Cho <danielcho@qnap.com <mailto:danielcho@qnap.com>>
>     > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>     <mailto:qemu-devel@nongnu.org>;
>     > > Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     > > *Subject:* RE: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > Hi,
>     > >
>     > >
>     > >
>     > > 1.       After re-walked through the codes, yes, you are
>     right, actually,
>     > > after the first migration, we will keep dirty log on in
>     primary side,
>     > >
>     > > And only send the dirty pages in PVM to SVM. The ram cache in
>     secondary
>     > > side is always a backup of PVM, so we don’t have to
>     > >
>     > > Re-send the none-dirtied pages.
>     > >
>     > > The reason why the first checkpoint takes longer time is we
>     have to backup
>     > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>     > >
>     > > It is time consuming, but I have optimized in the second patch
>     > > “0001-COLO-Optimize-memory-back-up-process.patch” which you
>     can find in my
>     > > previous reply.
>     > >
>     > >
>     > >
>     > > Besides, I found that, In my previous reply “We can only copy
>     the pages
>     > > that dirtied by PVM and SVM in last checkpoint.”,
>     > >
>     > > We have done this optimization in current upstream codes.
>     > >
>     > >
>     > >
>     > > 2．I don’t quite understand this question. For COLO, we always
>     need both
>     > > network packets of PVM’s and SVM’s to compare before send this
>     packets to
>     > > client.
>     > >
>     > > It depends on this to decide whether or not PVM and SVM are in
>     same state.
>     > >
>     > >
>     > >
>     > > Thanks,
>     > >
>     > > hailiang
>     > >
>     > >
>     > >
>     > > *From:* Daniel Cho [mailto:danielcho@qnap.com
>     <mailto:danielcho@qnap.com> <danielcho@qnap.com
>     <mailto:danielcho@qnap.com>>]
>     > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>     > > *To:* Zhang, Chen <chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>>
>     > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
>     > > Gilbert <dgilbert@redhat.com <mailto:dgilbert@redhat.com>>;
>     qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > *Subject:* Re: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > Hi Hailiang,
>     > >
>     > >
>     > >
>     > > Thanks for your replaying and explain in detail.
>     > >
>     > > We will try to use the attachments to enhance memory copy.
>     > >
>     > >
>     > >
>     > > However, we have some questions for your replying.
>     > >
>     > >
>     > >
>     > > 1.  As you said, "for each checkpoint, we have to send the
>     whole PVM's
>     > > pages To SVM", why the only first checkpoint will takes more
>     pause time?
>     > >
>     > > In our observing, the first checkpoint will take more time for
>     pausing,
>     > > then other checkpoints will takes a few time for pausing. Does
>     it means
>     > > only the first checkpoint will send the whole pages to SVM,
>     and the other
>     > > checkpoints send the dirty pages to SVM for reloading?
>     > >
>     > >
>     > >
>     > > 2. We notice the COLO-COMPARE component will stuck the packet
>     until
>     > > receive packets from PVM and SVM, as this rule, when we add the
>     > > COLO-COMPARE to PVM, its network will stuck until SVM start.
>     So it is an
>     > > other issue to make PVM stuck while setting COLO feature. With
>     this issue,
>     > > could we let colo-compare to pass the PVM's packet when the
>     SVM's packet
>     > > queue is empty? Then, the PVM's network won't stock, and "if
>     PVM runs
>     > > firstly, it still need to wait for The network packets from SVM to
>     > > compare before send it to client side" won't happened either.
>     > >
>     > >
>     > >
>     > > Best regard,
>     > >
>     > > Daniel Cho
>     > >
>     > >
>     > >
>     > > Zhang, Chen <chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：
>     > >
>     > >
>     > >
>     > > > -----Original Message-----
>     > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > > Sent: Wednesday, February 12, 2020 11:18 AM
>     > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; Daniel Cho
>     > > > <danielcho@qnap.com <mailto:danielcho@qnap.com>>; Zhang,
>     Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > > Subject: RE: The issues about architecture of the COLO
>     checkpoint
>     > > >
>     > > > Hi,
>     > > >
>     > > > Thank you Dave,
>     > > >
>     > > > I'll reply here directly.
>     > > >
>     > > > -----Original Message-----
>     > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>]
>     > > > Sent: Wednesday, February 12, 2020 1:48 AM
>     > > > To: Daniel Cho <danielcho@qnap.com
>     <mailto:danielcho@qnap.com>>; chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>;
>     > > > Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > > Subject: Re: The issues about architecture of the COLO
>     checkpoint
>     > > >
>     > > >
>     > > > cc'ing in COLO people:
>     > > >
>     > > >
>     > > > * Daniel Cho (danielcho@qnap.com
>     <mailto:danielcho@qnap.com>) wrote:
>     > > > > Hi everyone,
>     > > > >      We have some issues about setting COLO feature. Hope
>     somebody
>     > > > > could give us some advice.
>     > > > >
>     > > > > Issue 1:
>     > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
>     memory),  but
>     > > > > the Primary VM will pause a long time(based on memory
>     size) for
>     > > > > waiting SVM start. Does it have any idea to reduce the
>     pause time?
>     > > > >
>     > > >
>     > > > Yes, we do have some ideas to optimize this downtime.
>     > > >
>     > > > The main problem for current version is, for each
>     checkpoint, we have to
>     > > > send the whole PVM's pages
>     > > > To SVM, and then copy the whole VM's state into SVM from ram
>     cache, in
>     > > > this process, we need both of them be paused.
>     > > > Just as you said, the downtime is based on memory size.
>     > > >
>     > > > So firstly, we need to reduce the sending data while do
>     checkpoint,
>     > > actually,
>     > > > we can migrate parts of PVM's dirty pages in background
>     > > > While both of VMs are running. And then we load these pages
>     into ram
>     > > > cache (backup memory) in SVM temporarily. While do checkpoint,
>     > > > We just send the last dirty pages of PVM to slave side and
>     then copy the
>     > > ram
>     > > > cache into SVM. Further on, we don't have
>     > > > To send the whole PVM's dirty pages, we can only send the
>     pages that
>     > > > dirtied by PVM or SVM during two checkpoints. (Because
>     > > > If one page is not dirtied by both PVM and SVM, the data of
>     this pages
>     > > will
>     > > > keep same in SVM, PVM, backup memory). This method can reduce
>     > > > the time that consumed in sending data.
>     > > >
>     > > > For the second problem, we can reduce the memory copy by two
>     methods,
>     > > > first one, we don't have to copy the whole pages in ram cache,
>     > > > We can only copy the pages that dirtied by PVM and SVM in last
>     > > checkpoint.
>     > > > Second, we can use userfault missing function to reduce the
>     > > > Time consumed in memory copy. (For the second time, in
>     theory, we can
>     > > > reduce time consumed in memory into ms level).
>     > > >
>     > > > You can find the first optimization in attachment, it is
>     based on an old
>     > > qemu
>     > > > version (qemu-2.6), it should not be difficult to rebase it
>     > > > Into master or your version. And please feel free to send
>     the new
>     > > version if
>     > > > you want into community ;)
>     > > >
>     > > >
>     > >
>     > > Thanks Hailiang!
>     > > By the way, Do you have time to push the patches to upstream?
>     > > I think this is a better and faster option.
>     > >
>     > > Thanks
>     > > Zhang Chen
>     > >
>     > > > >
>     > > > > Issue 2:
>     > > > >      In
>     > > > >
>     https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>     > > > > could we move start_vm() before Line 488? Because at first
>     checkpoint
>     > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
>     > > > >
>     > > >
>     > > > No, that makes no sense, because if PVM runs firstly, it
>     still need to
>     > > wait for
>     > > > The network packets from SVM to compare before send it to
>     client side.
>     > > >
>     > > >
>     > > > Thanks,
>     > > > Hailiang
>     > > >
>     > > > >      We set the COLO feature on running VM, so we hope the
>     running VM
>     > > > > could continuous service for users.
>     > > > > Do you have any suggestions for those issues?
>     > > > >
>     > > > > Best regards,
>     > > > > Daniel Cho
>     > > > --
>     > > > Dr. David Alan Gilbert / dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com> / Manchester, UK
>     > >
>     > >
>     --
>     Dr. David Alan Gilbert / dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com> / Manchester, UK
>

[-- Attachment #2: Type: text/html, Size: 22593 bytes --]