* [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
@ 2016-09-25 8:22 Chunguang Li
2016-09-26 11:23 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 21+ messages in thread
From: Chunguang Li @ 2016-09-25 8:22 UTC (permalink / raw)
To: qemu-devel; +Cc: quintela, amit.shah, pbonzini, stefanha
Hi all!
I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
Am I right about this consideration? If I am right, is there some advice to improve this?
Thanks,
Chunguang Li
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-25 8:22 [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent Chunguang Li
@ 2016-09-26 11:23 ` Dr. David Alan Gilbert
2016-09-26 14:55 ` Chunguang Li
0 siblings, 1 reply; 21+ messages in thread
From: Dr. David Alan Gilbert @ 2016-09-26 11:23 UTC (permalink / raw)
To: Chunguang Li; +Cc: qemu-devel, amit.shah, pbonzini, stefanha, quintela
* Chunguang Li (lichunguang@hust.edu.cn) wrote:
> Hi all!
> I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
>
> Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
>
>
> Am I right about this consideration? If I am right, is there some advice to improve this?
I think you're right that this can happen; to clarify I think the
case you're talking about is:
Iteration 1
sync bitmap
start sending pages
page 'n' is modified - but hasn't been sent yet
page 'n' gets sent
Iteration 2
sync bitmap
'page n is shown as modified'
send page 'n' again
So you're right that is wasteful; I guess it's more wasteful
on big VMs with slow networks where the length of each iteration
is large.
Fixing it is not easy, because you have to be really careful
never to miss a page modification, even if the page is sent
about the same time it's dirtied.
One way would be to sync the dirty log from the kernel
in smaller chunks.
Dave
>
> Thanks,
> Chunguang Li
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-26 11:23 ` Dr. David Alan Gilbert
@ 2016-09-26 14:55 ` Chunguang Li
2016-09-26 18:52 ` Dr. David Alan Gilbert
2016-09-30 5:46 ` Amit Shah
0 siblings, 2 replies; 21+ messages in thread
From: Chunguang Li @ 2016-09-26 14:55 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, amit.shah, pbonzini, stefanha, quintela
> -----原始邮件-----
> 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 发送时间: 2016年9月26日 星期一
> 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > Hi all!
> > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> >
> > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> >
> >
> > Am I right about this consideration? If I am right, is there some advice to improve this?
>
> I think you're right that this can happen; to clarify I think the
> case you're talking about is:
>
> Iteration 1
> sync bitmap
> start sending pages
> page 'n' is modified - but hasn't been sent yet
> page 'n' gets sent
> Iteration 2
> sync bitmap
> 'page n is shown as modified'
> send page 'n' again
>
Yes，this is right the case I am talking about.
> So you're right that is wasteful; I guess it's more wasteful
> on big VMs with slow networks where the length of each iteration
> is large.
I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
Thanks,
Chunguang
>
> Fixing it is not easy, because you have to be really careful
> never to miss a page modification, even if the page is sent
> about the same time it's dirtied.
>
> One way would be to sync the dirty log from the kernel
> in smaller chunks.
>
> Dave
>
>
> >
> > Thanks,
> > Chunguang Li
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-26 14:55 ` Chunguang Li
@ 2016-09-26 18:52 ` Dr. David Alan Gilbert
2016-09-27 12:28 ` Chunguang Li
2016-09-30 5:46 ` Amit Shah
1 sibling, 1 reply; 21+ messages in thread
From: Dr. David Alan Gilbert @ 2016-09-26 18:52 UTC (permalink / raw)
To: Chunguang Li; +Cc: qemu-devel, amit.shah, pbonzini, stefanha, quintela
* Chunguang Li (lichunguang@hust.edu.cn) wrote:
>
>
>
> > -----原始邮件-----
> > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 发送时间: 2016年9月26日 星期一
> > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> >
> > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > Hi all!
> > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > >
> > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > >
> > >
> > > Am I right about this consideration? If I am right, is there some advice to improve this?
> >
> > I think you're right that this can happen; to clarify I think the
> > case you're talking about is:
> >
> > Iteration 1
> > sync bitmap
> > start sending pages
> > page 'n' is modified - but hasn't been sent yet
> > page 'n' gets sent
> > Iteration 2
> > sync bitmap
> > 'page n is shown as modified'
> > send page 'n' again
> >
>
> Yes，this is right the case I am talking about.
>
> > So you're right that is wasteful; I guess it's more wasteful
> > on big VMs with slow networks where the length of each iteration
> > is large.
>
> I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
Yes, it's probably pretty bad; and we really need to do something like
split the sync into smaller chunks; there are other suggestions
for how to improve it (e.g. there's the page-modification-logging
changes).
However, I don't think you usually get really random writes, if you
do precopy rarely converges at all, because even without your
observation it changes lots and lots of pages.
Dave
> Thanks,
>
> Chunguang
>
> >
> > Fixing it is not easy, because you have to be really careful
> > never to miss a page modification, even if the page is sent
> > about the same time it's dirtied.
> >
> > One way would be to sync the dirty log from the kernel
> > in smaller chunks.
> >
> > Dave
> >
> >
> > >
> > > Thanks,
> > > Chunguang Li
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> --
> Chunguang Li, Ph.D. Candidate
> Wuhan National Laboratory for Optoelectronics (WNLO)
> Huazhong University of Science & Technology (HUST)
> Wuhan, Hubei Prov., China
>
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-26 18:52 ` Dr. David Alan Gilbert
@ 2016-09-27 12:28 ` Chunguang Li
0 siblings, 0 replies; 21+ messages in thread
From: Chunguang Li @ 2016-09-27 12:28 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, amit.shah, pbonzini, stefanha, quintela
> -----原始邮件-----
> 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 发送时间: 2016年9月27日 星期二
> 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> Yes, it's probably pretty bad; and we really need to do something like
> split the sync into smaller chunks; there are other suggestions
> for how to improve it (e.g. there's the page-modification-logging
> changes).
>
> However, I don't think you usually get really random writes, if you
> do precopy rarely converges at all, because even without your
> observation it changes lots and lots of pages.
>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
I have read a little about the page-modification-logging. I think
it is only a more efficient way for dirty logging with better performance,
compared with write protection, but will not solve the problem we are
talking about.
The only idea to handle this, which I have come up with so far,
is to split the sync into smaller chunks that you have mentioned.
Maybe I can start from this idea to try to fix it.
If you come up with some other idea or suggestion, please let me know.
Thank you~
Chunguang
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-26 14:55 ` Chunguang Li
2016-09-26 18:52 ` Dr. David Alan Gilbert
@ 2016-09-30 5:46 ` Amit Shah
2016-09-30 8:18 ` Chunguang Li
2016-10-08 7:55 ` Chunguang Li
1 sibling, 2 replies; 21+ messages in thread
From: Amit Shah @ 2016-09-30 5:46 UTC (permalink / raw)
To: Chunguang Li
Cc: Dr. David Alan Gilbert, qemu-devel, pbonzini, stefanha, quintela
On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
>
>
>
> > -----原始邮件-----
> > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 发送时间: 2016年9月26日 星期一
> > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> >
> > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > Hi all!
> > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > >
> > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > >
> > >
> > > Am I right about this consideration? If I am right, is there some advice to improve this?
> >
> > I think you're right that this can happen; to clarify I think the
> > case you're talking about is:
> >
> > Iteration 1
> > sync bitmap
> > start sending pages
> > page 'n' is modified - but hasn't been sent yet
> > page 'n' gets sent
> > Iteration 2
> > sync bitmap
> > 'page n is shown as modified'
> > send page 'n' again
> >
>
> Yes，this is right the case I am talking about.
>
> > So you're right that is wasteful; I guess it's more wasteful
> > on big VMs with slow networks where the length of each iteration
> > is large.
>
> I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
It makes sense, can you get some perf numbers to show what kinds of
workloads get impacted the most? That would also help us to figure
out what kinds of speed improvements we can expect.
Amit
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-30 5:46 ` Amit Shah
@ 2016-09-30 8:18 ` Chunguang Li
2016-10-08 7:55 ` Chunguang Li
1 sibling, 0 replies; 21+ messages in thread
From: Chunguang Li @ 2016-09-30 8:18 UTC (permalink / raw)
To: Amit Shah
Cc: Dr. David Alan Gilbert, qemu-devel, pbonzini, stefanha, quintela
> -----原始邮件-----
> 发件人: "Amit Shah" <amit.shah@redhat.com>
> 发送时间: 2016年9月30日 星期五
> 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> >
> >
> >
> > > -----原始邮件-----
> > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > 发送时间: 2016年9月26日 星期一
> > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > >
> > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > Hi all!
> > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > >
> > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > >
> > > >
> > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > >
> > > I think you're right that this can happen; to clarify I think the
> > > case you're talking about is:
> > >
> > > Iteration 1
> > > sync bitmap
> > > start sending pages
> > > page 'n' is modified - but hasn't been sent yet
> > > page 'n' gets sent
> > > Iteration 2
> > > sync bitmap
> > > 'page n is shown as modified'
> > > send page 'n' again
> > >
> >
> > Yes，this is right the case I am talking about.
> >
> > > So you're right that is wasteful; I guess it's more wasteful
> > > on big VMs with slow networks where the length of each iteration
> > > is large.
> >
> > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
>
> It makes sense, can you get some perf numbers to show what kinds of
> workloads get impacted the most? That would also help us to figure
> out what kinds of speed improvements we can expect.
>
>
> Amit
Yes, I can pick up some workloads to get some perf numbers.
However, I don't know how to get the quantity of non-dirty pages we are
resending in each iteration. Instead, I can get the numbers below:
1. The time consuming of each iteration;
2. The quantity of pages transferred during each iteration;
3. The quantity of dirty pages (including not-really-dirty pages) produced
during each iteration.
With these numbers, we can only estimate the quantity of not-really-dirty
pages to some extent. How do you think of this test plan? Any suggestions?
Chunguang
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-09-30 5:46 ` Amit Shah
2016-09-30 8:18 ` Chunguang Li
@ 2016-10-08 7:55 ` Chunguang Li
2016-10-14 11:15 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 21+ messages in thread
From: Chunguang Li @ 2016-10-08 7:55 UTC (permalink / raw)
To: Amit Shah
Cc: Dr. David Alan Gilbert, qemu-devel, pbonzini, stefanha, quintela
> -----原始邮件-----
> 发件人: "Amit Shah" <amit.shah@redhat.com>
> 发送时间: 2016年9月30日 星期五
> 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> >
> >
> >
> > > -----原始邮件-----
> > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > 发送时间: 2016年9月26日 星期一
> > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > >
> > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > Hi all!
> > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > >
> > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > >
> > > >
> > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > >
> > > I think you're right that this can happen; to clarify I think the
> > > case you're talking about is:
> > >
> > > Iteration 1
> > > sync bitmap
> > > start sending pages
> > > page 'n' is modified - but hasn't been sent yet
> > > page 'n' gets sent
> > > Iteration 2
> > > sync bitmap
> > > 'page n is shown as modified'
> > > send page 'n' again
> > >
> >
> > Yes，this is right the case I am talking about.
> >
> > > So you're right that is wasteful; I guess it's more wasteful
> > > on big VMs with slow networks where the length of each iteration
> > > is large.
> >
> > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
>
> It makes sense, can you get some perf numbers to show what kinds of
> workloads get impacted the most? That would also help us to figure
> out what kinds of speed improvements we can expect.
>
>
> Amit
I have picked up 6 workloads and got the following statistics numbers
of every iteration (except the last stop-copy one) during precopy.
These numbers are obtained with the basic precopy migration, without
the capabilities like xbzrle or compression, etc. The network for the
migration is exclusive, with a separate network for the workloads.
They are both gigabit ethernet. I use qemu-2.5.1.
Three (booting, idle, web server) of them converged to the stop-copy phase,
with the given bandwidth and default downtime (300ms), while the other
three (kernel compilation, zeusmp, memcached) did not.
One page is "not-really-dirty", if it is written first and is sent later
(and not written again after that) during one iteration. I guess this
would not happen so often during the other iterations as during the 1st
iteration. Because all the pages of the VM are sent to the dest node during
the 1st iteration, while during the others, only part of the pages are sent.
So I think the "not-really-dirty" pages should be produced mainly during
the 1st iteration , and maybe very little during the other iterations.
If we could avoid resending the "not-really-dirty" pages, intuitively, I
think the time spent on Iteration 2 would be halved. This is a chain reaction,
because the dirty pages produced during Iteration 2 is halved, which incurs
that the time spent on Iteration 3 is halved, then Iteration 4, 5...
So I think "booting" and "kernel compilation" should benefit a lot from this
improvement. The reason of "kernel compilation" would benefit is that some
iterations take around 600ms, and if they are halved into 300ms, then the precopy
may have the chance to step into stop and copy phase.
On the other hand, "idle" and "web server" would not benefit a lot, because
most of the time are spent on the 1st iteration and little on the others.
As to the "zeusmp" and "memcached", although the time spent on the other iterations
but the 1st one may be halved, they still could not converge to stop and copy
with the 300ms downtime.
--------------------1 vcpu, 1 GB ram, default bandwidth (32MB/s):------------------
1. booting : begin to migrate when the VM is booting
Iteration 1, duration: 6997 ms , transferred pages: 266450 (n: 57269, d: 209181 ) , new dirty pages: 56414 , remaining dirty pages: 56414
Iteration 2, duration: 6497 ms , transferred pages: 54008 (n: 52701, d: 1307 ) , new dirty pages: 48053 , remaining dirty pages: 50459
Iteration 3, duration: 5800 ms , transferred pages: 48232 (n: 47444, d: 788 ) , new dirty pages: 9129 , remaining dirty pages: 11356
Iteration 4, duration: 1100 ms , transferred pages: 9091 (n: 8998, d: 93 ) , new dirty pages: 165 , remaining dirty pages: 2430
Iteration 5, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2430
(note: When the workload does converge, the output of the last iteration is "fake". It just indicates that the precopy steps into stop-copy phase now.
"n" means "normal pages" and "d" means "duplicate (zero) pages".)
2. idle
Iteration 1, duration: 14496 ms , transferred pages: 266450 (n: 118980, d: 147470 ) , new dirty pages: 17398 , remaining dirty pages: 17398
Iteration 2, duration: 1896 ms , transferred pages: 14953 (n: 14854, d: 99 ) , new dirty pages: 1849 , remaining dirty pages: 4294
Iteration 3, duration: 300 ms , transferred pages: 2454 (n: 2454, d: 0 ) , new dirty pages: 9 , remaining dirty pages: 1849
Iteration 4, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 1849
3. kernel compilation (can not converge)
Iteration 1, duration: 20700 ms , transferred pages: 266450 (n: 169778, d: 96672 ) , new dirty pages: 40067 , remaining dirty pages: 40067
Iteration 2, duration: 4696 ms , transferred pages: 38401 (n: 37787, d: 614 ) , new dirty pages: 8852 , remaining dirty pages: 10518
Iteration 3, duration: 1000 ms , transferred pages: 8642 (n: 8180, d: 462 ) , new dirty pages: 6331 , remaining dirty pages: 8207
Iteration 4, duration: 700 ms , transferred pages: 6110 (n: 5726, d: 384 ) , new dirty pages: 5242 , remaining dirty pages: 7339
Iteration 5, duration: 600 ms , transferred pages: 5007 (n: 4908, d: 99 ) , new dirty pages: 4868 , remaining dirty pages: 7200
Iteration 6, duration: 600 ms , transferred pages: 5226 (n: 4908, d: 318 ) , new dirty pages: 6142 , remaining dirty pages: 8116
Iteration 7, duration: 700 ms , transferred pages: 5985 (n: 5726, d: 259 ) , new dirty pages: 5902 , remaining dirty pages: 8033
Iteration 8, duration: 701 ms , transferred pages: 5893 (n: 5726, d: 167 ) , new dirty pages: 7502 , remaining dirty pages: 9642
Iteration 9, duration: 900 ms , transferred pages: 7623 (n: 7362, d: 261 ) , new dirty pages: 6408 , remaining dirty pages: 8427
Iteration 10, duration: 700 ms , transferred pages: 6008 (n: 5726, d: 282 ) , new dirty pages: 8312 , remaining dirty pages: 10731
Iteration 11, duration: 1000 ms , transferred pages: 8353 (n: 8180, d: 173 ) , new dirty pages: 6874 , remaining dirty pages: 9252
Iteration 12, duration: 899 ms , transferred pages: 7477 (n: 7362, d: 115 ) , new dirty pages: 5573 , remaining dirty pages: 7348
Iteration 13, duration: 601 ms , transferred pages: 5099 (n: 4908, d: 191 ) , new dirty pages: 7671 , remaining dirty pages: 9920
Iteration 14, duration: 900 ms , transferred pages: 7586 (n: 7362, d: 224 ) , new dirty pages: 7359 , remaining dirty pages: 9693
Iteration 15, duration: 900 ms , transferred pages: 7682 (n: 7362, d: 320 ) , new dirty pages: 7371 , remaining dirty pages: 9382
4. cpu2006.zeusmp (can not converge)
Iteration 1, duration: 21603 ms , transferred pages: 266450 (n: 176660, d: 89790 ) , new dirty pages: 145625 , remaining dirty pages: 145625
Iteration 2, duration: 8696 ms , transferred pages: 144389 (n: 70862, d: 73527 ) , new dirty pages: 125124 , remaining dirty pages: 126360
Iteration 3, duration: 6301 ms , transferred pages: 124057 (n: 51379, d: 72678 ) , new dirty pages: 122528 , remaining dirty pages: 124831
Iteration 4, duration: 6400 ms , transferred pages: 124330 (n: 52196, d: 72134 ) , new dirty pages: 124267 , remaining dirty pages: 124768
Iteration 5, duration: 6703 ms , transferred pages: 124034 (n: 54656, d: 69378 ) , new dirty pages: 124151 , remaining dirty pages: 124885
Iteration 6, duration: 6703 ms , transferred pages: 124357 (n: 54658, d: 69699 ) , new dirty pages: 124106 , remaining dirty pages: 124634
Iteration 7, duration: 6602 ms , transferred pages: 124568 (n: 53838, d: 70730 ) , new dirty pages: 133828 , remaining dirty pages: 133894
Iteration 8, duration: 7600 ms , transferred pages: 133030 (n: 62021, d: 71009 ) , new dirty pages: 126612 , remaining dirty pages: 127476
Iteration 9, duration: 7299 ms , transferred pages: 126511 (n: 59569, d: 66942 ) , new dirty pages: 122727 , remaining dirty pages: 123692
Iteration 10, duration: 6609 ms , transferred pages: 123692 (n: 54539, d: 69153 ) , new dirty pages: 122727 , remaining dirty pages: 122727
Iteration 11, duration: 6995 ms , transferred pages: 120347 (n: 56423, d: 63924 ) , new dirty pages: 121430 , remaining dirty pages: 123810
Iteration 12, duration: 6703 ms , transferred pages: 123040 (n: 54657, d: 68383 ) , new dirty pages: 122043 , remaining dirty pages: 122813
Iteration 13, duration: 7006 ms , transferred pages: 122353 (n: 57121, d: 65232 ) , new dirty pages: 133869 , remaining dirty pages: 134329
Iteration 14, duration: 8209 ms , transferred pages: 132325 (n: 66932, d: 65393 ) , new dirty pages: 126914 , remaining dirty pages: 128918
Iteration 15, duration: 7802 ms , transferred pages: 126931 (n: 63671, d: 63260 ) , new dirty pages: 122351 , remaining dirty pages: 124338
5. web server : An apache web server. The client is configured with 50 concurrent connections.
Iteration 1, duration: 30697 ms , transferred pages: 266450 (n: 251215, d: 15235 ) , new dirty pages: 30628 , remaining dirty pages: 30628
Iteration 2, duration: 3496 ms , transferred pages: 28859 (n: 28513, d: 346 ) , new dirty pages: 5805 , remaining dirty pages: 7574
Iteration 3, duration: 701 ms , transferred pages: 5746 (n: 5726, d: 20 ) , new dirty pages: 3433 , remaining dirty pages: 5261
Iteration 4, duration: 400 ms , transferred pages: 3281 (n: 3272, d: 9 ) , new dirty pages: 1539 , remaining dirty pages: 3519
Iteration 5, duration: 199 ms , transferred pages: 1653 (n: 1636, d: 17 ) , new dirty pages: 301 , remaining dirty pages: 2167
Iteration 6, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2167
--------------------6 vcpu, 6 GB ram, max bandwidth (941.08 mbps):------------------
6. memcached : 4 GB cache, memaslap: all write, concurrency = 5 (can not converge)
Iteration 1, duration: 42486 ms , transferred pages: 1568087 (n: 1216079, d: 352008 ) , new dirty pages: 571940 , remaining dirty pages: 581023
Iteration 2, duration: 19774 ms , transferred pages: 571700 (n: 567416, d: 4284 ) , new dirty pages: 331690 , remaining dirty pages: 341013
Iteration 3, duration: 11589 ms , transferred pages: 332187 (n: 332095, d: 92 ) , new dirty pages: 222725 , remaining dirty pages: 231551
Iteration 4, duration: 7790 ms , transferred pages: 223571 (n: 223499, d: 72 ) , new dirty pages: 157658 , remaining dirty pages: 165638
Iteration 5, duration: 5518 ms , transferred pages: 158056 (n: 157998, d: 58 ) , new dirty pages: 128130 , remaining dirty pages: 135712
Iteration 6, duration: 4442 ms , transferred pages: 127764 (n: 127701, d: 63 ) , new dirty pages: 104839 , remaining dirty pages: 112787
Iteration 7, duration: 3649 ms , transferred pages: 104581 (n: 104523, d: 58 ) , new dirty pages: 100736 , remaining dirty pages: 108942
Iteration 8, duration: 3532 ms , transferred pages: 101379 (n: 101315, d: 64 ) , new dirty pages: 87869 , remaining dirty pages: 95432
Iteration 9, duration: 3030 ms , transferred pages: 86841 (n: 86786, d: 55 ) , new dirty pages: 77505 , remaining dirty pages: 86096
Iteration 10, duration: 2709 ms , transferred pages: 77875 (n: 77814, d: 61 ) , new dirty pages: 77197 , remaining dirty pages: 85418
Iteration 11, duration: 2696 ms , transferred pages: 77107 (n: 77044, d: 63 ) , new dirty pages: 65010 , remaining dirty pages: 73321
Iteration 12, duration: 2308 ms , transferred pages: 66540 (n: 66484, d: 56 ) , new dirty pages: 64388 , remaining dirty pages: 71169
Iteration 13, duration: 2198 ms , transferred pages: 62953 (n: 62897, d: 56 ) , new dirty pages: 62773 , remaining dirty pages: 70989
Iteration 14, duration: 2214 ms , transferred pages: 63466 (n: 63411, d: 55 ) , new dirty pages: 67538 , remaining dirty pages: 75061
Iteration 15, duration: 2329 ms , transferred pages: 66924 (n: 66875, d: 49 ) , new dirty pages: 63580 , remaining dirty pages: 71717
Iteration 16, duration: 2252 ms , transferred pages: 64554 (n: 64539, d: 15 ) , new dirty pages: 63094 , remaining dirty pages: 70257
Iteration 17, duration: 2188 ms , transferred pages: 62697 (n: 62641, d: 56 ) , new dirty pages: 63016 , remaining dirty pages: 70576
Iteration 18, duration: 2171 ms , transferred pages: 62377 (n: 62322, d: 55 ) , new dirty pages: 56764 , remaining dirty pages: 64963
Iteration 19, duration: 2003 ms , transferred pages: 57382 (n: 57324, d: 58 ) , new dirty pages: 65307 , remaining dirty pages: 72888
Iteration 20, duration: 2240 ms , transferred pages: 64426 (n: 64364, d: 62 ) , new dirty pages: 61585 , remaining dirty pages: 70047
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-10-08 7:55 ` Chunguang Li
@ 2016-10-14 11:15 ` Dr. David Alan Gilbert
2016-11-03 8:25 ` Chunguang Li
0 siblings, 1 reply; 21+ messages in thread
From: Dr. David Alan Gilbert @ 2016-10-14 11:15 UTC (permalink / raw)
To: Chunguang Li; +Cc: Amit Shah, qemu-devel, pbonzini, stefanha, quintela
* Chunguang Li (lichunguang@hust.edu.cn) wrote:
>
>
>
> > -----原始邮件-----
> > 发件人: "Amit Shah" <amit.shah@redhat.com>
> > 发送时间: 2016年9月30日 星期五
> > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> >
> > On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> > >
> > >
> > >
> > > > -----原始邮件-----
> > > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > 发送时间: 2016年9月26日 星期一
> > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > >
> > > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > > Hi all!
> > > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > > >
> > > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > > >
> > > > >
> > > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > > >
> > > > I think you're right that this can happen; to clarify I think the
> > > > case you're talking about is:
> > > >
> > > > Iteration 1
> > > > sync bitmap
> > > > start sending pages
> > > > page 'n' is modified - but hasn't been sent yet
> > > > page 'n' gets sent
> > > > Iteration 2
> > > > sync bitmap
> > > > 'page n is shown as modified'
> > > > send page 'n' again
> > > >
> > >
> > > Yes，this is right the case I am talking about.
> > >
> > > > So you're right that is wasteful; I guess it's more wasteful
> > > > on big VMs with slow networks where the length of each iteration
> > > > is large.
> > >
> > > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
> >
> > It makes sense, can you get some perf numbers to show what kinds of
> > workloads get impacted the most? That would also help us to figure
> > out what kinds of speed improvements we can expect.
> >
> >
> > Amit
>
> I have picked up 6 workloads and got the following statistics numbers
> of every iteration (except the last stop-copy one) during precopy.
> These numbers are obtained with the basic precopy migration, without
> the capabilities like xbzrle or compression, etc. The network for the
> migration is exclusive, with a separate network for the workloads.
> They are both gigabit ethernet. I use qemu-2.5.1.
>
> Three (booting, idle, web server) of them converged to the stop-copy phase,
> with the given bandwidth and default downtime (300ms), while the other
> three (kernel compilation, zeusmp, memcached) did not.
>
> One page is "not-really-dirty", if it is written first and is sent later
> (and not written again after that) during one iteration. I guess this
> would not happen so often during the other iterations as during the 1st
> iteration. Because all the pages of the VM are sent to the dest node during
> the 1st iteration, while during the others, only part of the pages are sent.
> So I think the "not-really-dirty" pages should be produced mainly during
> the 1st iteration , and maybe very little during the other iterations.
>
> If we could avoid resending the "not-really-dirty" pages, intuitively, I
> think the time spent on Iteration 2 would be halved. This is a chain reaction,
> because the dirty pages produced during Iteration 2 is halved, which incurs
> that the time spent on Iteration 3 is halved, then Iteration 4, 5...
Yes; these numbers don't show how many of them are false dirty though.
One problem is thinking about pages that have been redirtied, if the page is dirtied
after the sync but before the network write then it's the false-dirty that
you're describing.
However, if the page is being written a few times, and so it would have been written
after the network write then it isn't a false-dirty.
You might be able to figure that out with some kernel tracing of when the dirtying
happens, but it might be easier to write the fix!
Dave
> So I think "booting" and "kernel compilation" should benefit a lot from this
> improvement. The reason of "kernel compilation" would benefit is that some
> iterations take around 600ms, and if they are halved into 300ms, then the precopy
> may have the chance to step into stop and copy phase.
>
> On the other hand, "idle" and "web server" would not benefit a lot, because
> most of the time are spent on the 1st iteration and little on the others.
>
> As to the "zeusmp" and "memcached", although the time spent on the other iterations
> but the 1st one may be halved, they still could not converge to stop and copy
> with the 300ms downtime.
>
> --------------------1 vcpu, 1 GB ram, default bandwidth (32MB/s):------------------
>
> 1. booting : begin to migrate when the VM is booting
>
> Iteration 1, duration: 6997 ms , transferred pages: 266450 (n: 57269, d: 209181 ) , new dirty pages: 56414 , remaining dirty pages: 56414
> Iteration 2, duration: 6497 ms , transferred pages: 54008 (n: 52701, d: 1307 ) , new dirty pages: 48053 , remaining dirty pages: 50459
> Iteration 3, duration: 5800 ms , transferred pages: 48232 (n: 47444, d: 788 ) , new dirty pages: 9129 , remaining dirty pages: 11356
> Iteration 4, duration: 1100 ms , transferred pages: 9091 (n: 8998, d: 93 ) , new dirty pages: 165 , remaining dirty pages: 2430
> Iteration 5, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2430
> (note: When the workload does converge, the output of the last iteration is "fake". It just indicates that the precopy steps into stop-copy phase now.
> "n" means "normal pages" and "d" means "duplicate (zero) pages".)
>
> 2. idle
>
> Iteration 1, duration: 14496 ms , transferred pages: 266450 (n: 118980, d: 147470 ) , new dirty pages: 17398 , remaining dirty pages: 17398
> Iteration 2, duration: 1896 ms , transferred pages: 14953 (n: 14854, d: 99 ) , new dirty pages: 1849 , remaining dirty pages: 4294
> Iteration 3, duration: 300 ms , transferred pages: 2454 (n: 2454, d: 0 ) , new dirty pages: 9 , remaining dirty pages: 1849
> Iteration 4, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 1849
>
> 3. kernel compilation (can not converge)
>
> Iteration 1, duration: 20700 ms , transferred pages: 266450 (n: 169778, d: 96672 ) , new dirty pages: 40067 , remaining dirty pages: 40067
> Iteration 2, duration: 4696 ms , transferred pages: 38401 (n: 37787, d: 614 ) , new dirty pages: 8852 , remaining dirty pages: 10518
> Iteration 3, duration: 1000 ms , transferred pages: 8642 (n: 8180, d: 462 ) , new dirty pages: 6331 , remaining dirty pages: 8207
> Iteration 4, duration: 700 ms , transferred pages: 6110 (n: 5726, d: 384 ) , new dirty pages: 5242 , remaining dirty pages: 7339
> Iteration 5, duration: 600 ms , transferred pages: 5007 (n: 4908, d: 99 ) , new dirty pages: 4868 , remaining dirty pages: 7200
> Iteration 6, duration: 600 ms , transferred pages: 5226 (n: 4908, d: 318 ) , new dirty pages: 6142 , remaining dirty pages: 8116
> Iteration 7, duration: 700 ms , transferred pages: 5985 (n: 5726, d: 259 ) , new dirty pages: 5902 , remaining dirty pages: 8033
> Iteration 8, duration: 701 ms , transferred pages: 5893 (n: 5726, d: 167 ) , new dirty pages: 7502 , remaining dirty pages: 9642
> Iteration 9, duration: 900 ms , transferred pages: 7623 (n: 7362, d: 261 ) , new dirty pages: 6408 , remaining dirty pages: 8427
> Iteration 10, duration: 700 ms , transferred pages: 6008 (n: 5726, d: 282 ) , new dirty pages: 8312 , remaining dirty pages: 10731
> Iteration 11, duration: 1000 ms , transferred pages: 8353 (n: 8180, d: 173 ) , new dirty pages: 6874 , remaining dirty pages: 9252
> Iteration 12, duration: 899 ms , transferred pages: 7477 (n: 7362, d: 115 ) , new dirty pages: 5573 , remaining dirty pages: 7348
> Iteration 13, duration: 601 ms , transferred pages: 5099 (n: 4908, d: 191 ) , new dirty pages: 7671 , remaining dirty pages: 9920
> Iteration 14, duration: 900 ms , transferred pages: 7586 (n: 7362, d: 224 ) , new dirty pages: 7359 , remaining dirty pages: 9693
> Iteration 15, duration: 900 ms , transferred pages: 7682 (n: 7362, d: 320 ) , new dirty pages: 7371 , remaining dirty pages: 9382
>
> 4. cpu2006.zeusmp (can not converge)
>
> Iteration 1, duration: 21603 ms , transferred pages: 266450 (n: 176660, d: 89790 ) , new dirty pages: 145625 , remaining dirty pages: 145625
> Iteration 2, duration: 8696 ms , transferred pages: 144389 (n: 70862, d: 73527 ) , new dirty pages: 125124 , remaining dirty pages: 126360
> Iteration 3, duration: 6301 ms , transferred pages: 124057 (n: 51379, d: 72678 ) , new dirty pages: 122528 , remaining dirty pages: 124831
> Iteration 4, duration: 6400 ms , transferred pages: 124330 (n: 52196, d: 72134 ) , new dirty pages: 124267 , remaining dirty pages: 124768
> Iteration 5, duration: 6703 ms , transferred pages: 124034 (n: 54656, d: 69378 ) , new dirty pages: 124151 , remaining dirty pages: 124885
> Iteration 6, duration: 6703 ms , transferred pages: 124357 (n: 54658, d: 69699 ) , new dirty pages: 124106 , remaining dirty pages: 124634
> Iteration 7, duration: 6602 ms , transferred pages: 124568 (n: 53838, d: 70730 ) , new dirty pages: 133828 , remaining dirty pages: 133894
> Iteration 8, duration: 7600 ms , transferred pages: 133030 (n: 62021, d: 71009 ) , new dirty pages: 126612 , remaining dirty pages: 127476
> Iteration 9, duration: 7299 ms , transferred pages: 126511 (n: 59569, d: 66942 ) , new dirty pages: 122727 , remaining dirty pages: 123692
> Iteration 10, duration: 6609 ms , transferred pages: 123692 (n: 54539, d: 69153 ) , new dirty pages: 122727 , remaining dirty pages: 122727
> Iteration 11, duration: 6995 ms , transferred pages: 120347 (n: 56423, d: 63924 ) , new dirty pages: 121430 , remaining dirty pages: 123810
> Iteration 12, duration: 6703 ms , transferred pages: 123040 (n: 54657, d: 68383 ) , new dirty pages: 122043 , remaining dirty pages: 122813
> Iteration 13, duration: 7006 ms , transferred pages: 122353 (n: 57121, d: 65232 ) , new dirty pages: 133869 , remaining dirty pages: 134329
> Iteration 14, duration: 8209 ms , transferred pages: 132325 (n: 66932, d: 65393 ) , new dirty pages: 126914 , remaining dirty pages: 128918
> Iteration 15, duration: 7802 ms , transferred pages: 126931 (n: 63671, d: 63260 ) , new dirty pages: 122351 , remaining dirty pages: 124338
>
> 5. web server : An apache web server. The client is configured with 50 concurrent connections.
>
> Iteration 1, duration: 30697 ms , transferred pages: 266450 (n: 251215, d: 15235 ) , new dirty pages: 30628 , remaining dirty pages: 30628
> Iteration 2, duration: 3496 ms , transferred pages: 28859 (n: 28513, d: 346 ) , new dirty pages: 5805 , remaining dirty pages: 7574
> Iteration 3, duration: 701 ms , transferred pages: 5746 (n: 5726, d: 20 ) , new dirty pages: 3433 , remaining dirty pages: 5261
> Iteration 4, duration: 400 ms , transferred pages: 3281 (n: 3272, d: 9 ) , new dirty pages: 1539 , remaining dirty pages: 3519
> Iteration 5, duration: 199 ms , transferred pages: 1653 (n: 1636, d: 17 ) , new dirty pages: 301 , remaining dirty pages: 2167
> Iteration 6, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2167
>
> --------------------6 vcpu, 6 GB ram, max bandwidth (941.08 mbps):------------------
>
> 6. memcached : 4 GB cache, memaslap: all write, concurrency = 5 (can not converge)
>
> Iteration 1, duration: 42486 ms , transferred pages: 1568087 (n: 1216079, d: 352008 ) , new dirty pages: 571940 , remaining dirty pages: 581023
> Iteration 2, duration: 19774 ms , transferred pages: 571700 (n: 567416, d: 4284 ) , new dirty pages: 331690 , remaining dirty pages: 341013
> Iteration 3, duration: 11589 ms , transferred pages: 332187 (n: 332095, d: 92 ) , new dirty pages: 222725 , remaining dirty pages: 231551
> Iteration 4, duration: 7790 ms , transferred pages: 223571 (n: 223499, d: 72 ) , new dirty pages: 157658 , remaining dirty pages: 165638
> Iteration 5, duration: 5518 ms , transferred pages: 158056 (n: 157998, d: 58 ) , new dirty pages: 128130 , remaining dirty pages: 135712
> Iteration 6, duration: 4442 ms , transferred pages: 127764 (n: 127701, d: 63 ) , new dirty pages: 104839 , remaining dirty pages: 112787
> Iteration 7, duration: 3649 ms , transferred pages: 104581 (n: 104523, d: 58 ) , new dirty pages: 100736 , remaining dirty pages: 108942
> Iteration 8, duration: 3532 ms , transferred pages: 101379 (n: 101315, d: 64 ) , new dirty pages: 87869 , remaining dirty pages: 95432
> Iteration 9, duration: 3030 ms , transferred pages: 86841 (n: 86786, d: 55 ) , new dirty pages: 77505 , remaining dirty pages: 86096
> Iteration 10, duration: 2709 ms , transferred pages: 77875 (n: 77814, d: 61 ) , new dirty pages: 77197 , remaining dirty pages: 85418
> Iteration 11, duration: 2696 ms , transferred pages: 77107 (n: 77044, d: 63 ) , new dirty pages: 65010 , remaining dirty pages: 73321
> Iteration 12, duration: 2308 ms , transferred pages: 66540 (n: 66484, d: 56 ) , new dirty pages: 64388 , remaining dirty pages: 71169
> Iteration 13, duration: 2198 ms , transferred pages: 62953 (n: 62897, d: 56 ) , new dirty pages: 62773 , remaining dirty pages: 70989
> Iteration 14, duration: 2214 ms , transferred pages: 63466 (n: 63411, d: 55 ) , new dirty pages: 67538 , remaining dirty pages: 75061
> Iteration 15, duration: 2329 ms , transferred pages: 66924 (n: 66875, d: 49 ) , new dirty pages: 63580 , remaining dirty pages: 71717
> Iteration 16, duration: 2252 ms , transferred pages: 64554 (n: 64539, d: 15 ) , new dirty pages: 63094 , remaining dirty pages: 70257
> Iteration 17, duration: 2188 ms , transferred pages: 62697 (n: 62641, d: 56 ) , new dirty pages: 63016 , remaining dirty pages: 70576
> Iteration 18, duration: 2171 ms , transferred pages: 62377 (n: 62322, d: 55 ) , new dirty pages: 56764 , remaining dirty pages: 64963
> Iteration 19, duration: 2003 ms , transferred pages: 57382 (n: 57324, d: 58 ) , new dirty pages: 65307 , remaining dirty pages: 72888
> Iteration 20, duration: 2240 ms , transferred pages: 64426 (n: 64364, d: 62 ) , new dirty pages: 61585 , remaining dirty pages: 70047
>
>
> --
> Chunguang Li, Ph.D. Candidate
> Wuhan National Laboratory for Optoelectronics (WNLO)
> Huazhong University of Science & Technology (HUST)
> Wuhan, Hubei Prov., China
>
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-10-14 11:15 ` Dr. David Alan Gilbert
@ 2016-11-03 8:25 ` Chunguang Li
2016-11-03 9:59 ` Li, Liang Z
` (2 more replies)
0 siblings, 3 replies; 21+ messages in thread
From: Chunguang Li @ 2016-11-03 8:25 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Amit Shah, qemu-devel, pbonzini, stefanha, quintela
> -----Original Messages-----
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Sent Time: Friday, October 14, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>
> Cc: "Amit Shah" <amit.shah@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> >
> >
> >
> > > -----原始邮件-----
> > > 发件人: "Amit Shah" <amit.shah@redhat.com>
> > > 发送时间: 2016年9月30日 星期五
> > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > >
> > > On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> > > >
> > > >
> > > >
> > > > > -----原始邮件-----
> > > > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > > 发送时间: 2016年9月26日 星期一
> > > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > > >
> > > > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > > > Hi all!
> > > > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > > > >
> > > > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > > > >
> > > > > >
> > > > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > > > >
> > > > > I think you're right that this can happen; to clarify I think the
> > > > > case you're talking about is:
> > > > >
> > > > > Iteration 1
> > > > > sync bitmap
> > > > > start sending pages
> > > > > page 'n' is modified - but hasn't been sent yet
> > > > > page 'n' gets sent
> > > > > Iteration 2
> > > > > sync bitmap
> > > > > 'page n is shown as modified'
> > > > > send page 'n' again
> > > > >
> > > >
> > > > Yes，this is right the case I am talking about.
> > > >
> > > > > So you're right that is wasteful; I guess it's more wasteful
> > > > > on big VMs with slow networks where the length of each iteration
> > > > > is large.
> > > >
> > > > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
> > >
> > > It makes sense, can you get some perf numbers to show what kinds of
> > > workloads get impacted the most? That would also help us to figure
> > > out what kinds of speed improvements we can expect.
> > >
> > >
> > > Amit
> >
> > I have picked up 6 workloads and got the following statistics numbers
> > of every iteration (except the last stop-copy one) during precopy.
> > These numbers are obtained with the basic precopy migration, without
> > the capabilities like xbzrle or compression, etc. The network for the
> > migration is exclusive, with a separate network for the workloads.
> > They are both gigabit ethernet. I use qemu-2.5.1.
> >
> > Three (booting, idle, web server) of them converged to the stop-copy phase,
> > with the given bandwidth and default downtime (300ms), while the other
> > three (kernel compilation, zeusmp, memcached) did not.
> >
> > One page is "not-really-dirty", if it is written first and is sent later
> > (and not written again after that) during one iteration. I guess this
> > would not happen so often during the other iterations as during the 1st
> > iteration. Because all the pages of the VM are sent to the dest node during
> > the 1st iteration, while during the others, only part of the pages are sent.
> > So I think the "not-really-dirty" pages should be produced mainly during
> > the 1st iteration , and maybe very little during the other iterations.
> >
> > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > think the time spent on Iteration 2 would be halved. This is a chain reaction,
> > because the dirty pages produced during Iteration 2 is halved, which incurs
> > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
>
> Yes; these numbers don't show how many of them are false dirty though.
>
> One problem is thinking about pages that have been redirtied, if the page is dirtied
> after the sync but before the network write then it's the false-dirty that
> you're describing.
>
> However, if the page is being written a few times, and so it would have been written
> after the network write then it isn't a false-dirty.
>
> You might be able to figure that out with some kernel tracing of when the dirtying
> happens, but it might be easier to write the fix!
>
> Dave
Hi, I have made some new progress now.
To tell how many false dirty pages there are exactly in each iteration, I malloc a
buffer in memory as big as the size of the whole VM memory. When a page is
transferred to the dest node, it is copied to the buffer; During the next iteration,
if one page is transferred, it is compared to the old one in the buffer, and the
old one will be replaced for next comparison if it is really dirty. Thus, we are now
able to get the exact number of false dirty pages.
This time, I use 15 workloads to get the statistic number. They are:
1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all scientific
computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc. I pick
up these 11 benchmarks because compared to others, they have bigger memory
occupation and higher memory dirty rate. Thus most of them could not converge
to stop-and-copy using the default migration speed (32MB/s).
2. kernel compilation
3. idle VM
4. Apache web server which serves static content
(the above workloads are all running in VM with 1 vcpu and 1GB memory, and the
migration speed is the default 32MB/s)
5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used as the cache.
After filling up the 4GB cache, a client writes the cache at a constant speed
during migration. This time, migration speed has no limit, and is up to the
capability of 1Gbps Ethernet.
Summarize the results first: (and you can read the precise number below)
1. 4 of these 15 workloads have a big proportion (>60%, even >80% during some iterations)
of false dirty pages out of all the dirty pages since iteration 2 (and the big
proportion lasts during the following iterations). They are cpu2006.zeusmp,
cpu2006.bzip2, cpu2006.mcf, and memcached.
2. 2 workloads (idle, webserver) spend most of the migration time on iteration 1, even
though the proportion of false dirty pages is big since iteration 2, the space to
optimize is small.
3. 1 workload (kernel compilation) only have a big proportion during iteration 2, not
in the other iterations.
4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of false
dirty pages since iteration 2. So the spaces to optimize for them are small.
Now I want to talk a little more about the reasons why false dirty pages are produced.
The first reason is what we have discussed before---the mechanism to track the dirty
pages.
And then I come up with another reason. Here is the situation: a write operation to one
memory page happens, but it doesn't change any content of the page. So it's "write but
not dirty", and kernel still marks it as dirty. One guy in our lab has done some experiments
to figure out the proportion of "write but not dirty" operations, and he uses the cpu2006
benchmark suit. According to his results, general workloads has a little proportion (<10%)
of "write but not dirty" out of all the write operations, while few workloads have higher
proportion (one even as high as 50%). Now we are not sure why "write but not dirty" would
happen, it just happened.
So these two reasons contribute to the false dirty pages. To optimize, I compute and store
the SHA1 hash before transferring each page. Next time, if one page needs retransmission, its
SHA1 hash is computed again, and compared to the old hash. If the hash is the same, it's a
false dirty page, and we just skip this page; Otherwise, the page is transferred, and the new
hash replaces the old one for next comparison.
The reason to use SHA1 hash but not byte-by-byte comparison is the memory overheads. One SHA1
hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the whole VM memory, which
is relatively small.
As far as I know, SHA1 hash is widely used in the scenes of deduplication for backup systems.
They have proven that the probability of hash collision is far smaller than disk hardware fault,
so it's secure hash, that is, if the hashes of two chunks are the same, the content must be the
same. So I think the SHA1 hash could replace byte-to-byte comparison in the VM memory scenery.
Then I do the same migration experiments using the SHA1 hash. For the 4 workloads which have
big proportions of false dirty pages, the improvement is remarkable. Without optimization,
they either can not converge to stop-and-copy, or take a very long time to complete. With the
SHA1 hash method, all of them now complete in a relatively short time.
For the reason I have talked above, the other workloads don't get notable improvements from the
optimization. So below, I only show the exact number after optimization for the 4 workloads with
remarkable improvements.
Any comments or suggestions?
Below is the experiments data:
(
"dup" means zero page, this kind of pages takes very little migration time and network
resources, so they are always not regard as dirty pages in my numbers;
"rd" means really dirty pages;
"fd" means false dirty pages;
The numbers refer to the quantities of pages.
)
------------------The 4 workloads with remarkable improvements (both the results of original precopy and with optimization are shown)-------------------
1. memcached
----- original pre-copy (can not converge): -----
Iteration 1, duration: 42111 ms , transferred pages: 1568788 (dup: 416239, rd: 1152549, fd: 0) , new dirty pages: 499015 , remaining dirty pages: 507397
Iteration 2, duration: 17208 ms , transferred pages: 498946 (dup: 5456, rd: 160206, fd: 333284) , new dirty pages: 261237 , remaining dirty pages: 269688
Iteration 3, duration: 9134 ms , transferred pages: 262377 (dup: 519, rd: 111900, fd: 149958) , new dirty pages: 170281 , remaining dirty pages: 177592
Iteration 4, duration: 5920 ms , transferred pages: 169966 (dup: 87, rd: 82487, fd: 87392) , new dirty pages: 121154 , remaining dirty pages: 128780
Iteration 5, duration: 4239 ms , transferred pages: 121551 (dup: 81, rd: 64120, fd: 57350) , new dirty pages: 100976 , remaining dirty pages: 108205
Iteration 6, duration: 3495 ms , transferred pages: 100353 (dup: 90, rd: 56021, fd: 44242) , new dirty pages: 74547 , remaining dirty pages: 82399
Iteration 7, duration: 2583 ms , transferred pages: 74160 (dup: 56, rd: 38016, fd: 36088) , new dirty pages: 58209 , remaining dirty pages: 66448
Iteration 8, duration: 2039 ms , transferred pages: 58534 (dup: 81, rd: 26885, fd: 31568) , new dirty pages: 43511 , remaining dirty pages: 51425
Iteration 9, duration: 1513 ms , transferred pages: 43484 (dup: 55, rd: 26641, fd: 16788) , new dirty pages: 43722 , remaining dirty pages: 51663
Iteration 10, duration: 1521 ms , transferred pages: 43676 (dup: 62, rd: 26463, fd: 17151) , new dirty pages: 35347 , remaining dirty pages: 43334
Iteration 11, duration: 1230 ms , transferred pages: 35287 (dup: 0, rd: 21293, fd: 13994) , new dirty pages: 28851 , remaining dirty pages: 36898
Iteration 12, duration: 1031 ms , transferred pages: 29651 (dup: 82, rd: 18143, fd: 11426) , new dirty pages: 27062 , remaining dirty pages: 34309
Iteration 13, duration: 917 ms , transferred pages: 26385 (dup: 56, rd: 14149, fd: 12180) , new dirty pages: 22723 , remaining dirty pages: 30647
Iteration 14, duration: 762 ms , transferred pages: 21902 (dup: 55, rd: 16355, fd: 5492) , new dirty pages: 18208 , remaining dirty pages: 26953
Iteration 15, duration: 650 ms , transferred pages: 18636 (dup: 0, rd: 11943, fd: 6693) , new dirty pages: 16085 , remaining dirty pages: 24402
Iteration 16, duration: 554 ms , transferred pages: 15946 (dup: 56, rd: 9527, fd: 6363) , new dirty pages: 14766 , remaining dirty pages: 23222
Iteration 17, duration: 538 ms , transferred pages: 15434 (dup: 0, rd: 9779, fd: 5655) , new dirty pages: 13381 , remaining dirty pages: 21169
Iteration 18, duration: 487 ms , transferred pages: 14089 (dup: 81, rd: 7737, fd: 6271) , new dirty pages: 13325 , remaining dirty pages: 20405
Iteration 19, duration: 428 ms , transferred pages: 12232 (dup: 0, rd: 8488, fd: 3744) , new dirty pages: 10274 , remaining dirty pages: 18447
Iteration 20, duration: 377 ms , transferred pages: 10887 (dup: 56, rd: 6362, fd: 4469) , new dirty pages: 9708 , remaining dirty pages: 17268
Iteration 21, duration: 320 ms , transferred pages: 9222 (dup: 0, rd: 5789, fd: 3433) , new dirty pages: 8015 , remaining dirty pages: 16061
Iteration 22, duration: 268 ms , transferred pages: 7621 (dup: 0, rd: 6204, fd: 1417) , new dirty pages: 7227 , remaining dirty pages: 15667
Iteration 23, duration: 269 ms , transferred pages: 7813 (dup: 56, rd: 4410, fd: 3347) , new dirty pages: 7591 , remaining dirty pages: 15445
Iteration 24, duration: 271 ms , transferred pages: 7749 (dup: 0, rd: 4565, fd: 3184) , new dirty pages: 15126 , remaining dirty pages: 22822
Iteration 25, duration: 549 ms , transferred pages: 15818 (dup: 60, rd: 10545, fd: 5213) , new dirty pages: 14559 , remaining dirty pages: 21563
Iteration 26, duration: 499 ms , transferred pages: 14281 (dup: 3, rd: 8760, fd: 5518) , new dirty pages: 11947 , remaining dirty pages: 19229
Iteration 27, duration: 376 ms , transferred pages: 10823 (dup: 25, rd: 6550, fd: 4248) , new dirty pages: 8561 , remaining dirty pages: 16967
Iteration 28, duration: 324 ms , transferred pages: 9350 (dup: 31, rd: 5292, fd: 4027) , new dirty pages: 8655 , remaining dirty pages: 16272
Iteration 29, duration: 274 ms , transferred pages: 7813 (dup: 0, rd: 6088, fd: 1725) , new dirty pages: 6300 , remaining dirty pages: 14759
Iteration 30, duration: 218 ms , transferred pages: 6340 (dup: 45, rd: 3196, fd: 3099) , new dirty pages: 5143 , remaining dirty pages: 13562
----- after optimization: -----
Iteration 1, duration: 40664 ms , transferred pages: 1569037 (dup: 405940, rd: 1163097) , new dirty pages: 506846 , remaining dirty pages: 514979
Iteration 2, duration: 8032 ms , transferred pages: 161130 (dup: 4007, rd: 157123) , new dirty pages: 153479 , remaining dirty pages: 153479
Iteration 3, duration: 2620 ms , transferred pages: 65260 (dup: 20, rd: 65240) , new dirty pages: 64014 , remaining dirty pages: 67100
Iteration 4, duration: 1160 ms , transferred pages: 30227 (dup: 60, rd: 30167) , new dirty pages: 34031 , remaining dirty pages: 41414
Iteration 5, duration: 648 ms , transferred pages: 18700 (dup: 56, rd: 18644) , new dirty pages: 18375 , remaining dirty pages: 25536
Iteration 6, duration: 389 ms , transferred pages: 11399 (dup: 55, rd: 11344) , new dirty pages: 12536 , remaining dirty pages: 17516
Iteration 7, duration: 292 ms , transferred pages: 8197 (dup: 0, rd: 8197) , new dirty pages: 8387 , remaining dirty pages: 16802
Iteration 8, duration: 171 ms , transferred pages: 4931 (dup: 39, rd: 4892) , new dirty pages: 6182 , remaining dirty pages: 14060
Iteration 9, duration: 163 ms , transferred pages: 4355 (dup: 16, rd: 4339) , new dirty pages: 5530 , remaining dirty pages: 11973
Iteration 10, duration: 104 ms , transferred pages: 3266 (dup: 0, rd: 3266) , new dirty pages: 2893 , remaining dirty pages: 11014
Iteration 11, duration: 52 ms , transferred pages: 1153 (dup: 0, rd: 1153) , new dirty pages: 1586 , remaining dirty pages: 10516
Iteration 12, duration: 52 ms , transferred pages: 1921 (dup: 39, rd: 1882) , new dirty pages: 1619 , remaining dirty pages: 8842
Iteration 13, duration: 62 ms , transferred pages: 1537 (dup: 0, rd: 1537) , new dirty pages: 2052 , remaining dirty pages: 8871
Iteration 14, duration: 58 ms , transferred pages: 1665 (dup: 0, rd: 1665) , new dirty pages: 1947 , remaining dirty pages: 7989
Iteration 15, duration: 2 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 7989
total time: 54693 milliseconds
2. cpu2006.zeusmp
----- original pre-copy (can not converge): -----
Iteration 1, duration: 21112 ms , transferred pages: 266450 (dup: 93385, rd: 173065, fd: 0) , new dirty pages: 127866 , remaining dirty pages: 127866
Iteration 2, duration: 6192 ms , transferred pages: 125662 (dup: 75762, rd: 17389, fd: 32511) , new dirty pages: 131655 , remaining dirty pages: 133859
Iteration 3, duration: 6699 ms , transferred pages: 131937 (dup: 77298, rd: 20320, fd: 34319) , new dirty pages: 121027 , remaining dirty pages: 122949
Iteration 4, duration: 5999 ms , transferred pages: 122512 (dup: 73588, rd: 17236, fd: 31688) , new dirty pages: 122759 , remaining dirty pages: 123196
Iteration 5, duration: 5804 ms , transferred pages: 122717 (dup: 75436, rd: 19016, fd: 28265) , new dirty pages: 123697 , remaining dirty pages: 124176
Iteration 6, duration: 5698 ms , transferred pages: 123708 (dup: 77249, rd: 18022, fd: 28437) , new dirty pages: 121838 , remaining dirty pages: 122306
Iteration 7, duration: 5515 ms , transferred pages: 122306 (dup: 76727, rd: 14819, fd: 30760) , new dirty pages: 122382 , remaining dirty pages: 122382
Iteration 8, duration: 6086 ms , transferred pages: 120825 (dup: 71834, rd: 15987, fd: 33004) , new dirty pages: 121587 , remaining dirty pages: 123144
Iteration 9, duration: 5899 ms , transferred pages: 120964 (dup: 72860, rd: 18191, fd: 29913) , new dirty pages: 120391 , remaining dirty pages: 122571
Iteration 10, duration: 5801 ms , transferred pages: 121425 (dup: 74140, rd: 20722, fd: 26563) , new dirty pages: 122302 , remaining dirty pages: 123448
Iteration 11, duration: 5909 ms , transferred pages: 123448 (dup: 74735, rd: 19678, fd: 29035) , new dirty pages: 123258 , remaining dirty pages: 123258
Iteration 12, duration: 6293 ms , transferred pages: 121211 (dup: 70442, rd: 18128, fd: 32641) , new dirty pages: 123623 , remaining dirty pages: 125670
Iteration 13, duration: 6398 ms , transferred pages: 124897 (dup: 72701, rd: 21134, fd: 31062) , new dirty pages: 122355 , remaining dirty pages: 123128
Iteration 14, duration: 6301 ms , transferred pages: 121893 (dup: 70514, rd: 23470, fd: 27909) , new dirty pages: 120980 , remaining dirty pages: 122215
Iteration 15, duration: 6304 ms , transferred pages: 121389 (dup: 70005, rd: 21731, fd: 29653) , new dirty pages: 121628 , remaining dirty pages: 122454
Iteration 16, duration: 6398 ms , transferred pages: 122164 (dup: 69962, rd: 24376, fd: 27826) , new dirty pages: 122246 , remaining dirty pages: 122536
Iteration 17, duration: 6201 ms , transferred pages: 121548 (dup: 70984, rd: 23915, fd: 26649) , new dirty pages: 121460 , remaining dirty pages: 122448
Iteration 18, duration: 6401 ms , transferred pages: 122272 (dup: 70072, rd: 22261, fd: 29939) , new dirty pages: 123518 , remaining dirty pages: 123694
Iteration 19, duration: 7003 ms , transferred pages: 121873 (dup: 64754, rd: 27325, fd: 29794) , new dirty pages: 120568 , remaining dirty pages: 122389
Iteration 20, duration: 6400 ms , transferred pages: 121422 (dup: 69221, rd: 25300, fd: 26901) , new dirty pages: 121229 , remaining dirty pages: 122196
Iteration 21, duration: 6703 ms , transferred pages: 119895 (dup: 65232, rd: 25877, fd: 28786) , new dirty pages: 123284 , remaining dirty pages: 125585
Iteration 22, duration: 6902 ms , transferred pages: 123884 (dup: 67582, rd: 29020, fd: 27282) , new dirty pages: 122057 , remaining dirty pages: 123758
Iteration 23, duration: 6800 ms , transferred pages: 122010 (dup: 66529, rd: 30644, fd: 24837) , new dirty pages: 120916 , remaining dirty pages: 122664
Iteration 24, duration: 7202 ms , transferred pages: 121951 (dup: 63188, rd: 31105, fd: 27658) , new dirty pages: 122715 , remaining dirty pages: 123428
Iteration 25, duration: 7202 ms , transferred pages: 122919 (dup: 64161, rd: 32063, fd: 26695) , new dirty pages: 123180 , remaining dirty pages: 123689
Iteration 26, duration: 7404 ms , transferred pages: 123092 (dup: 62694, rd: 33459, fd: 26939) , new dirty pages: 122149 , remaining dirty pages: 122746
Iteration 27, duration: 7205 ms , transferred pages: 120427 (dup: 61664, rd: 34344, fd: 24419) , new dirty pages: 120299 , remaining dirty pages: 122618
Iteration 28, duration: 7100 ms , transferred pages: 121074 (dup: 63130, rd: 32403, fd: 25541) , new dirty pages: 122984 , remaining dirty pages: 124528
Iteration 29, duration: 7904 ms , transferred pages: 124060 (dup: 59564, rd: 35631, fd: 28865) , new dirty pages: 127080 , remaining dirty pages: 127548
Iteration 30, duration: 7906 ms , transferred pages: 127518 (dup: 63029, rd: 34416, fd: 30073) , new dirty pages: 125028 , remaining dirty pages: 125058
----- after optimization: -----
Iteration 1, duration: 21601 ms , transferred pages: 266450 (dup: 89731, rd: 176719) , new dirty pages: 139843 , remaining dirty pages: 139843
Iteration 2, duration: 1747 ms , transferred pages: 92077 (dup: 78364, rd: 13713) , new dirty pages: 90945 , remaining dirty pages: 90945
Iteration 3, duration: 1592 ms , transferred pages: 62253 (dup: 49435, rd: 12818) , new dirty pages: 76929 , remaining dirty pages: 76929
Iteration 4, duration: 992 ms , transferred pages: 44837 (dup: 37886, rd: 6951) , new dirty pages: 71331 , remaining dirty pages: 72916
Iteration 5, duration: 998 ms , transferred pages: 55229 (dup: 47150, rd: 8079) , new dirty pages: 21703 , remaining dirty pages: 23302
Iteration 6, duration: 211 ms , transferred pages: 20337 (dup: 18516, rd: 1821) , new dirty pages: 14500 , remaining dirty pages: 14500
Iteration 7, duration: 31 ms , transferred pages: 12933 (dup: 12627, rd: 306) , new dirty pages: 1520 , remaining dirty pages: 1520
Iteration 8, duration: 30 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 4 , remaining dirty pages: 1524
total time: 27225 milliseconds
3. cpu2006.bzip2
----- original pre-copy: -----
Iteration 1, duration: 18306 ms , transferred pages: 266450 (dup: 116569, rd: 149881, fd: 0) , new dirty pages: 106299 , remaining dirty pages: 106299
Iteration 2, duration: 10694 ms , transferred pages: 104611 (dup: 17550, rd: 10536, fd: 76525) , new dirty pages: 34394 , remaining dirty pages: 36082
Iteration 3, duration: 2998 ms , transferred pages: 34442 (dup: 9924, rd: 12254, fd: 12264) , new dirty pages: 6419 , remaining dirty pages: 8059
Iteration 4, duration: 699 ms , transferred pages: 5748 (dup: 22, rd: 2583, fd: 3143) , new dirty pages: 1226 , remaining dirty pages: 3537
Iteration 5, duration: 200 ms , transferred pages: 1636 (dup: 0, rd: 1194, fd: 442) , new dirty pages: 478 , remaining dirty pages: 2379
Iteration 6, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2379
----- after optimization: -----
Iteration 1, duration: 13995 ms , transferred pages: 266314 (dup: 152118, rd: 114196) , new dirty pages: 97009 , remaining dirty pages: 97145
Iteration 2, duration: 1215 ms , transferred pages: 33400 (dup: 26745, rd: 6655) , new dirty pages: 12866 , remaining dirty pages: 14017
Iteration 3, duration: 701 ms , transferred pages: 5774 (dup: 48, rd: 5726) , new dirty pages: 6342 , remaining dirty pages: 8761
Iteration 4, duration: 500 ms , transferred pages: 4111 (dup: 21, rd: 4090) , new dirty pages: 4311 , remaining dirty pages: 6485
Iteration 5, duration: 400 ms , transferred pages: 3273 (dup: 1, rd: 3272) , new dirty pages: 3034 , remaining dirty pages: 5431
Iteration 6, duration: 301 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2094 , remaining dirty pages: 4472
Iteration 7, duration: 299 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2066 , remaining dirty pages: 4082
Iteration 8, duration: 202 ms , transferred pages: 1636 (dup: 0, rd: 1636) , new dirty pages: 2881 , remaining dirty pages: 4648
Iteration 9, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 4775 , remaining dirty pages: 6778
Iteration 10, duration: 400 ms , transferred pages: 3281 (dup: 9, rd: 3272) , new dirty pages: 3757 , remaining dirty pages: 5576
Iteration 11, duration: 401 ms , transferred pages: 3279 (dup: 7, rd: 3272) , new dirty pages: 6980 , remaining dirty pages: 8906
Iteration 12, duration: 500 ms , transferred pages: 7118 (dup: 3035, rd: 4083) , new dirty pages: 10774 , remaining dirty pages: 11922
Iteration 13, duration: 116 ms , transferred pages: 11706 (dup: 10152, rd: 1554) , new dirty pages: 1326 , remaining dirty pages: 1326
Iteration 14, duration: 117 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 1326
total time: 19479 milliseconds
4. cpu2006.mcf
----- original pre-copy: -----
Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6925, rd: 259525, fd: 0) , new dirty pages: 244403 , remaining dirty pages: 244403
Iteration 2, duration: 29603 ms , transferred pages: 242275 (dup: 377, rd: 224001, fd: 17897) , new dirty pages: 227335 , remaining dirty pages: 229463
Iteration 3, duration: 27806 ms , transferred pages: 227573 (dup: 169, rd: 65681, fd: 161723) , new dirty pages: 195593 , remaining dirty pages: 197483
Iteration 4, duration: 23907 ms , transferred pages: 195543 (dup: 41, rd: 39838, fd: 155664) , new dirty pages: 215066 , remaining dirty pages: 217006
Iteration 5, duration: 26305 ms , transferred pages: 215289 (dup: 155, rd: 33082, fd: 182052) , new dirty pages: 111098 , remaining dirty pages: 112815
Iteration 6, duration: 13502 ms , transferred pages: 110452 (dup: 22, rd: 26793, fd: 83637) , new dirty pages: 161054 , remaining dirty pages: 163417
Iteration 7, duration: 19705 ms , transferred pages: 161266 (dup: 120, rd: 33818, fd: 127328) , new dirty pages: 220562 , remaining dirty pages: 222713
Iteration 8, duration: 27003 ms , transferred pages: 220881 (dup: 21, rd: 215721, fd: 5139) , new dirty pages: 219787 , remaining dirty pages: 221619
Iteration 9, duration: 26802 ms , transferred pages: 219248 (dup: 24, rd: 84648, fd: 134576) , new dirty pages: 207959 , remaining dirty pages: 210330
Iteration 10, duration: 25411 ms , transferred pages: 207916 (dup: 144, rd: 35842, fd: 171930) , new dirty pages: 144442 , remaining dirty pages: 146856
Iteration 11, duration: 17714 ms , transferred pages: 144804 (dup: 18, rd: 25414, fd: 119372) , new dirty pages: 205127 , remaining dirty pages: 207179
Iteration 12, duration: 25112 ms , transferred pages: 205446 (dup: 128, rd: 23197, fd: 182121) , new dirty pages: 167319 , remaining dirty pages: 169052
Iteration 13, duration: 20411 ms , transferred pages: 166886 (dup: 14, rd: 21960, fd: 144912) , new dirty pages: 221592 , remaining dirty pages: 223758
Iteration 14, duration: 27126 ms , transferred pages: 221800 (dup: 122, rd: 42368, fd: 179310) , new dirty pages: 233630 , remaining dirty pages: 235588
Iteration 15, duration: 28517 ms , transferred pages: 233321 (dup: 191, rd: 222528, fd: 10602) , new dirty pages: 224282 , remaining dirty pages: 226549
Iteration 16, duration: 27422 ms , transferred pages: 224187 (dup: 55, rd: 45773, fd: 178359) , new dirty pages: 209815 , remaining dirty pages: 212177
Iteration 17, duration: 25723 ms , transferred pages: 210260 (dup: 34, rd: 79405, fd: 130821) , new dirty pages: 220297 , remaining dirty pages: 222214
Iteration 18, duration: 26920 ms , transferred pages: 220056 (dup: 14, rd: 214128, fd: 5914) , new dirty pages: 192015 , remaining dirty pages: 194173
Iteration 19, duration: 23520 ms , transferred pages: 192239 (dup: 9, rd: 25140, fd: 167090) , new dirty pages: 96450 , remaining dirty pages: 98384
Iteration 20, duration: 11805 ms , transferred pages: 96538 (dup: 14, rd: 7424, fd: 89100) , new dirty pages: 6978 , remaining dirty pages: 8824
Iteration 21, duration: 799 ms , transferred pages: 6545 (dup: 1, rd: 1802, fd: 4742) , new dirty pages: 138 , remaining dirty pages: 2417
Iteration 22, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2417
----- after optimization: -----
Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6831, rd: 259619) , new dirty pages: 240209 , remaining dirty pages: 240209
Iteration 2, duration: 6250 ms , transferred pages: 51244 (dup: 211, rd: 51033) , new dirty pages: 226651 , remaining dirty pages: 228571
Iteration 3, duration: 4395 ms , transferred pages: 36008 (dup: 80, rd: 35928) , new dirty pages: 110719 , remaining dirty pages: 111478
Iteration 4, duration: 3390 ms , transferred pages: 28068 (dup: 28, rd: 28040) , new dirty pages: 185172 , remaining dirty pages: 185172
Iteration 5, duration: 2986 ms , transferred pages: 23780 (dup: 45, rd: 23735) , new dirty pages: 64357 , remaining dirty pages: 66305
Iteration 6, duration: 2727 ms , transferred pages: 22800 (dup: 12, rd: 22788) , new dirty pages: 61675 , remaining dirty pages: 61675
Iteration 7, duration: 2372 ms , transferred pages: 18943 (dup: 13, rd: 18930) , new dirty pages: 55144 , remaining dirty pages: 55265
Iteration 8, duration: 2100 ms , transferred pages: 17189 (dup: 11, rd: 17178) , new dirty pages: 55244 , remaining dirty pages: 55668
Iteration 9, duration: 2003 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 107058 , remaining dirty pages: 108014
Iteration 10, duration: 2132 ms , transferred pages: 17825 (dup: 24, rd: 17801) , new dirty pages: 126214 , remaining dirty pages: 126214
Iteration 11, duration: 2229 ms , transferred pages: 18156 (dup: 22, rd: 18134) , new dirty pages: 65725 , remaining dirty pages: 65725
Iteration 12, duration: 2315 ms , transferred pages: 18651 (dup: 21, rd: 18630) , new dirty pages: 52575 , remaining dirty pages: 53903
Iteration 13, duration: 2147 ms , transferred pages: 17435 (dup: 16, rd: 17419) , new dirty pages: 46652 , remaining dirty pages: 47260
Iteration 14, duration: 2000 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 42721 , remaining dirty pages: 43266
Iteration 15, duration: 1901 ms , transferred pages: 15552 (dup: 10, rd: 15542) , new dirty pages: 38593 , remaining dirty pages: 40792
Iteration 16, duration: 1801 ms , transferred pages: 14735 (dup: 11, rd: 14724) , new dirty pages: 54252 , remaining dirty pages: 55639
Iteration 17, duration: 1708 ms , transferred pages: 13860 (dup: 2, rd: 13858) , new dirty pages: 72379 , remaining dirty pages: 74170
Iteration 18, duration: 1923 ms , transferred pages: 15442 (dup: 12, rd: 15430) , new dirty pages: 101911 , remaining dirty pages: 103547
Iteration 19, duration: 2311 ms , transferred pages: 18823 (dup: 9, rd: 18814) , new dirty pages: 80534 , remaining dirty pages: 82521
Iteration 20, duration: 2081 ms , transferred pages: 17156 (dup: 34, rd: 17122) , new dirty pages: 36054 , remaining dirty pages: 36054
Iteration 21, duration: 1665 ms , transferred pages: 13777 (dup: 10, rd: 13767) , new dirty pages: 29624 , remaining dirty pages: 29624
Iteration 22, duration: 1657 ms , transferred pages: 13290 (dup: 7, rd: 13283) , new dirty pages: 25949 , remaining dirty pages: 28265
Iteration 23, duration: 1599 ms , transferred pages: 13088 (dup: 0, rd: 13088) , new dirty pages: 22356 , remaining dirty pages: 24813
Iteration 24, duration: 1500 ms , transferred pages: 12280 (dup: 10, rd: 12270) , new dirty pages: 21181 , remaining dirty pages: 22608
Iteration 25, duration: 1400 ms , transferred pages: 11457 (dup: 5, rd: 11452) , new dirty pages: 18657 , remaining dirty pages: 20311
Iteration 26, duration: 1200 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 15690 , remaining dirty pages: 17294
Iteration 27, duration: 1201 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 14810 , remaining dirty pages: 15936
Iteration 28, duration: 1000 ms , transferred pages: 8183 (dup: 3, rd: 8180) , new dirty pages: 15387 , remaining dirty pages: 16423
Iteration 29, duration: 900 ms , transferred pages: 7372 (dup: 10, rd: 7362) , new dirty pages: 13303 , remaining dirty pages: 15292
Iteration 30, duration: 1000 ms , transferred pages: 8181 (dup: 1, rd: 8180) , new dirty pages: 17879 , remaining dirty pages: 18457
Iteration 31, duration: 951 ms , transferred pages: 8140 (dup: 9, rd: 8131) , new dirty pages: 21738 , remaining dirty pages: 23304
Iteration 32, duration: 946 ms , transferred pages: 6946 (dup: 1, rd: 6945) , new dirty pages: 15815 , remaining dirty pages: 15815
Iteration 33, duration: 747 ms , transferred pages: 6192 (dup: 0, rd: 6192) , new dirty pages: 6249 , remaining dirty pages: 7670
Iteration 34, duration: 501 ms , transferred pages: 4090 (dup: 0, rd: 4090) , new dirty pages: 6163 , remaining dirty pages: 8422
Iteration 35, duration: 600 ms , transferred pages: 4910 (dup: 2, rd: 4908) , new dirty pages: 3673 , remaining dirty pages: 5222
Iteration 36, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2132 , remaining dirty pages: 4337
Iteration 37, duration: 200 ms , transferred pages: 1637 (dup: 1, rd: 1636) , new dirty pages: 544 , remaining dirty pages: 2251
Iteration 38, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 2251
total time: 97919 milliseconds
------------------The other 11 workloads without notable improvements (only the result of original precopy is shown)-------------------
5. idle
Iteration 1, duration: 14702 ms , transferred pages: 266450 (dup: 146393, rd: 120057, fd: 0) , new dirty pages: 14595 , remaining dirty pages: 14595
Iteration 2, duration: 1592 ms , transferred pages: 12412 (dup: 103, rd: 3280, fd: 9029) , new dirty pages: 218 , remaining dirty pages: 2401
Iteration 3, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2401
6. kernel compilation (can not converge)
Iteration 1, duration: 20607 ms , transferred pages: 266450 (dup: 97552, rd: 168898, fd: 0) , new dirty pages: 19293 , remaining dirty pages: 19293
Iteration 2, duration: 2092 ms , transferred pages: 17176 (dup: 597, rd: 8625, fd: 7954) , new dirty pages: 8318 , remaining dirty pages: 10435
Iteration 3, duration: 1000 ms , transferred pages: 8484 (dup: 304, rd: 6256, fd: 1924) , new dirty pages: 8736 , remaining dirty pages: 10687
Iteration 4, duration: 1000 ms , transferred pages: 8435 (dup: 255, rd: 7089, fd: 1091) , new dirty pages: 7627 , remaining dirty pages: 9879
Iteration 5, duration: 900 ms , transferred pages: 7553 (dup: 191, rd: 5602, fd: 1760) , new dirty pages: 7287 , remaining dirty pages: 9613
Iteration 6, duration: 900 ms , transferred pages: 7620 (dup: 258, rd: 5761, fd: 1601) , new dirty pages: 8958 , remaining dirty pages: 10951
Iteration 7, duration: 1099 ms , transferred pages: 9309 (dup: 311, rd: 8051, fd: 947) , new dirty pages: 7189 , remaining dirty pages: 8831
Iteration 8, duration: 800 ms , transferred pages: 6832 (dup: 288, rd: 5717, fd: 827) , new dirty pages: 5782 , remaining dirty pages: 7781
Iteration 9, duration: 701 ms , transferred pages: 5875 (dup: 149, rd: 4005, fd: 1721) , new dirty pages: 4587 , remaining dirty pages: 6493
Iteration 10, duration: 500 ms , transferred pages: 4234 (dup: 144, rd: 3057, fd: 1033) , new dirty pages: 7352 , remaining dirty pages: 9611
Iteration 11, duration: 900 ms , transferred pages: 7759 (dup: 397, rd: 6563, fd: 799) , new dirty pages: 6686 , remaining dirty pages: 8538
Iteration 12, duration: 800 ms , transferred pages: 6808 (dup: 264, rd: 6017, fd: 527) , new dirty pages: 6871 , remaining dirty pages: 8601
Iteration 13, duration: 800 ms , transferred pages: 6775 (dup: 231, rd: 5722, fd: 822) , new dirty pages: 7540 , remaining dirty pages: 9366
Iteration 14, duration: 900 ms , transferred pages: 7507 (dup: 145, rd: 5900, fd: 1462) , new dirty pages: 7581 , remaining dirty pages: 9440
Iteration 15, duration: 900 ms , transferred pages: 7630 (dup: 268, rd: 6211, fd: 1151) , new dirty pages: 7268 , remaining dirty pages: 9078
Iteration 16, duration: 800 ms , transferred pages: 6759 (dup: 215, rd: 5763, fd: 781) , new dirty pages: 6861 , remaining dirty pages: 9180
Iteration 17, duration: 800 ms , transferred pages: 6838 (dup: 294, rd: 6037, fd: 507) , new dirty pages: 6196 , remaining dirty pages: 8538
Iteration 18, duration: 800 ms , transferred pages: 6852 (dup: 308, rd: 4905, fd: 1639) , new dirty pages: 5947 , remaining dirty pages: 7633
Iteration 19, duration: 700 ms , transferred pages: 5919 (dup: 193, rd: 4853, fd: 873) , new dirty pages: 5861 , remaining dirty pages: 7575
Iteration 20, duration: 600 ms , transferred pages: 5284 (dup: 376, rd: 4408, fd: 500) , new dirty pages: 5206 , remaining dirty pages: 7497
Iteration 21, duration: 600 ms , transferred pages: 5147 (dup: 239, rd: 4308, fd: 600) , new dirty pages: 5031 , remaining dirty pages: 7381
Iteration 22, duration: 599 ms , transferred pages: 5064 (dup: 156, rd: 4026, fd: 882) , new dirty pages: 5601 , remaining dirty pages: 7918
Iteration 23, duration: 702 ms , transferred pages: 5965 (dup: 239, rd: 5028, fd: 698) , new dirty pages: 6079 , remaining dirty pages: 8032
Iteration 24, duration: 700 ms , transferred pages: 6175 (dup: 449, rd: 5146, fd: 580) , new dirty pages: 10932 , remaining dirty pages: 12789
Iteration 25, duration: 1300 ms , transferred pages: 10936 (dup: 302, rd: 6205, fd: 4429) , new dirty pages: 8713 , remaining dirty pages: 10566
Iteration 26, duration: 1000 ms , transferred pages: 8282 (dup: 102, rd: 5662, fd: 2518) , new dirty pages: 5119 , remaining dirty pages: 7403
Iteration 27, duration: 600 ms , transferred pages: 5007 (dup: 99, rd: 4099, fd: 809) , new dirty pages: 2226 , remaining dirty pages: 4622
Iteration 28, duration: 300 ms , transferred pages: 2491 (dup: 37, rd: 1794, fd: 660) , new dirty pages: 6746 , remaining dirty pages: 8877
Iteration 29, duration: 800 ms , transferred pages: 6757 (dup: 213, rd: 5532, fd: 1012) , new dirty pages: 6070 , remaining dirty pages: 8190
Iteration 30, duration: 700 ms , transferred pages: 6052 (dup: 326, rd: 5107, fd: 619) , new dirty pages: 5177 , remaining dirty pages: 7315
7. web server
Iteration 1, duration: 20902 ms , transferred pages: 266450 (dup: 95497, rd: 170953, fd: 0) , new dirty pages: 8528 , remaining dirty pages: 8528
Iteration 2, duration: 796 ms , transferred pages: 6472 (dup: 131, rd: 1885, fd: 4456) , new dirty pages: 650 , remaining dirty pages: 2706
Iteration 3, duration: 100 ms , transferred pages: 818 (dup: 0, rd: 383, fd: 435) , new dirty pages: 328 , remaining dirty pages: 2216
Iteration 4, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2216
8. cpu2006.bwaves (can not converge)
Iteration 1, duration: 31715 ms , transferred pages: 266450 (dup: 6766, rd: 259684, fd: 0) , new dirty pages: 242702 , remaining dirty pages: 242702
Iteration 2, duration: 29397 ms , transferred pages: 240508 (dup: 405, rd: 225588, fd: 14515) , new dirty pages: 230889 , remaining dirty pages: 233083
Iteration 3, duration: 28205 ms , transferred pages: 230858 (dup: 182, rd: 214596, fd: 16080) , new dirty pages: 226998 , remaining dirty pages: 229223
Iteration 4, duration: 27805 ms , transferred pages: 227574 (dup: 170, rd: 217045, fd: 10359) , new dirty pages: 227360 , remaining dirty pages: 229009
Iteration 5, duration: 27703 ms , transferred pages: 226786 (dup: 200, rd: 212130, fd: 14456) , new dirty pages: 225885 , remaining dirty pages: 228108
Iteration 6, duration: 27600 ms , transferred pages: 225923 (dup: 155, rd: 215503, fd: 10265) , new dirty pages: 223555 , remaining dirty pages: 225740
Iteration 7, duration: 27309 ms , transferred pages: 223574 (dup: 260, rd: 215641, fd: 7673) , new dirty pages: 231975 , remaining dirty pages: 234141
Iteration 8, duration: 28403 ms , transferred pages: 232397 (dup: 85, rd: 214086, fd: 18226) , new dirty pages: 222170 , remaining dirty pages: 223914
Iteration 9, duration: 27105 ms , transferred pages: 221809 (dup: 131, rd: 214988, fd: 6690) , new dirty pages: 230065 , remaining dirty pages: 232170
Iteration 10, duration: 28104 ms , transferred pages: 230201 (dup: 343, rd: 213531, fd: 16327) , new dirty pages: 227590 , remaining dirty pages: 229559
Iteration 11, duration: 27801 ms , transferred pages: 227717 (dup: 313, rd: 221408, fd: 5996) , new dirty pages: 228457 , remaining dirty pages: 230299
Iteration 12, duration: 27916 ms , transferred pages: 228560 (dup: 338, rd: 219660, fd: 8562) , new dirty pages: 238326 , remaining dirty pages: 240065
9. cpu2006.lbm (can not converge)
Iteration 1, duration: 31012 ms , transferred pages: 266450 (dup: 12253, rd: 254197, fd: 0) , new dirty pages: 108960 , remaining dirty pages: 108960
Iteration 2, duration: 13095 ms , transferred pages: 106522 (dup: 3, rd: 102045, fd: 4474) , new dirty pages: 129292 , remaining dirty pages: 131730
Iteration 3, duration: 15802 ms , transferred pages: 129688 (dup: 444, rd: 110860, fd: 18384) , new dirty pages: 116682 , remaining dirty pages: 118724
Iteration 4, duration: 14204 ms , transferred pages: 116316 (dup: 160, rd: 104951, fd: 11205) , new dirty pages: 107246 , remaining dirty pages: 109654
Iteration 5, duration: 13208 ms , transferred pages: 107977 (dup: 1, rd: 101834, fd: 6142) , new dirty pages: 105371 , remaining dirty pages: 107048
Iteration 6, duration: 12804 ms , transferred pages: 104705 (dup: 1, rd: 99629, fd: 5075) , new dirty pages: 103841 , remaining dirty pages: 106184
Iteration 7, duration: 12709 ms , transferred pages: 103891 (dup: 5, rd: 99212, fd: 4674) , new dirty pages: 106692 , remaining dirty pages: 108985
Iteration 8, duration: 13105 ms , transferred pages: 107169 (dup: 11, rd: 100125, fd: 7033) , new dirty pages: 103132 , remaining dirty pages: 104948
Iteration 9, duration: 12607 ms , transferred pages: 103068 (dup: 0, rd: 99460, fd: 3608) , new dirty pages: 102511 , remaining dirty pages: 104391
Iteration 10, duration: 12514 ms , transferred pages: 102250 (dup: 0, rd: 99094, fd: 3156) , new dirty pages: 102888 , remaining dirty pages: 105029
10. cpu2006.astar (can not converge)
Iteration 1, duration: 28402 ms , transferred pages: 266450 (dup: 33770, rd: 232680, fd: 0) , new dirty pages: 62078 , remaining dirty pages: 62078
Iteration 2, duration: 7393 ms , transferred pages: 60107 (dup: 10, rd: 51722, fd: 8375) , new dirty pages: 48854 , remaining dirty pages: 50825
Iteration 3, duration: 6001 ms , transferred pages: 49094 (dup: 14, rd: 46540, fd: 2540) , new dirty pages: 48137 , remaining dirty pages: 49868
Iteration 4, duration: 5800 ms , transferred pages: 47444 (dup: 0, rd: 45389, fd: 2055) , new dirty pages: 49147 , remaining dirty pages: 51571
Iteration 5, duration: 6102 ms , transferred pages: 49912 (dup: 14, rd: 46216, fd: 3682) , new dirty pages: 55606 , remaining dirty pages: 57265
Iteration 6, duration: 6699 ms , transferred pages: 54949 (dup: 143, rd: 20745, fd: 34061) , new dirty pages: 9166 , remaining dirty pages: 11482
Iteration 7, duration: 1200 ms , transferred pages: 9830 (dup: 14, rd: 7011, fd: 2805) , new dirty pages: 8294 , remaining dirty pages: 9946
Iteration 8, duration: 1000 ms , transferred pages: 8194 (dup: 14, rd: 7178, fd: 1002) , new dirty pages: 5475 , remaining dirty pages: 7227
Iteration 9, duration: 600 ms , transferred pages: 4908 (dup: 0, rd: 3470, fd: 1438) , new dirty pages: 4175 , remaining dirty pages: 6494
Iteration 10, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3856, fd: 234) , new dirty pages: 4095 , remaining dirty pages: 6499
Iteration 11, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3313, fd: 777) , new dirty pages: 3371 , remaining dirty pages: 5780
Iteration 12, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3823, fd: 267) , new dirty pages: 7518 , remaining dirty pages: 9208
Iteration 13, duration: 899 ms , transferred pages: 7376 (dup: 14, rd: 6028, fd: 1334) , new dirty pages: 3931 , remaining dirty pages: 5763
Iteration 14, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 4346 , remaining dirty pages: 6019
Iteration 15, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3817, fd: 273) , new dirty pages: 3054 , remaining dirty pages: 4983
Iteration 16, duration: 400 ms , transferred pages: 3272 (dup: 0, rd: 3138, fd: 134) , new dirty pages: 3874 , remaining dirty pages: 5585
Iteration 17, duration: 399 ms , transferred pages: 3272 (dup: 0, rd: 3248, fd: 24) , new dirty pages: 5285 , remaining dirty pages: 7598
Iteration 18, duration: 701 ms , transferred pages: 5726 (dup: 0, rd: 4385, fd: 1341) , new dirty pages: 8903 , remaining dirty pages: 10775
Iteration 19, duration: 1101 ms , transferred pages: 9010 (dup: 12, rd: 5597, fd: 3401) , new dirty pages: 4199 , remaining dirty pages: 5964
Iteration 20, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 3829 , remaining dirty pages: 5703
11. cpu2006.xalancbmk (can not converge)
Iteration 1, duration: 30407 ms , transferred pages: 266450 (dup: 17700, rd: 248750, fd: 0) , new dirty pages: 96169 , remaining dirty pages: 96169
Iteration 2, duration: 11495 ms , transferred pages: 94164 (dup: 205, rd: 67068, fd: 26891) , new dirty pages: 61766 , remaining dirty pages: 63771
Iteration 3, duration: 7501 ms , transferred pages: 61471 (dup: 121, rd: 53587, fd: 7763) , new dirty pages: 56569 , remaining dirty pages: 58869
Iteration 4, duration: 6902 ms , transferred pages: 56461 (dup: 19, rd: 50553, fd: 5889) , new dirty pages: 52181 , remaining dirty pages: 54589
Iteration 5, duration: 6402 ms , transferred pages: 52459 (dup: 107, rd: 46986, fd: 5366) , new dirty pages: 54051 , remaining dirty pages: 56181
Iteration 6, duration: 6601 ms , transferred pages: 54003 (dup: 15, rd: 47566, fd: 6422) , new dirty pages: 50844 , remaining dirty pages: 53022
Iteration 7, duration: 6202 ms , transferred pages: 50723 (dup: 7, rd: 47143, fd: 3573) , new dirty pages: 64880 , remaining dirty pages: 67179
Iteration 8, duration: 8001 ms , transferred pages: 65447 (dup: 7, rd: 61159, fd: 4281) , new dirty pages: 67854 , remaining dirty pages: 69586
Iteration 9, duration: 8202 ms , transferred pages: 67444 (dup: 368, rd: 56357, fd: 10719) , new dirty pages: 65178 , remaining dirty pages: 67320
Iteration 10, duration: 8000 ms , transferred pages: 65455 (dup: 15, rd: 60581, fd: 4859) , new dirty pages: 52421 , remaining dirty pages: 54286
12. cpu2006.milc (can not converge)
Iteration 1, duration: 31410 ms , transferred pages: 266450 (dup: 9454, rd: 256996, fd: 0) , new dirty pages: 158860 , remaining dirty pages: 158860
Iteration 2, duration: 19193 ms , transferred pages: 157048 (dup: 150, rd: 96807, fd: 60091) , new dirty pages: 102238 , remaining dirty pages: 104050
Iteration 3, duration: 12504 ms , transferred pages: 102271 (dup: 21, rd: 95107, fd: 7143) , new dirty pages: 97944 , remaining dirty pages: 99723
Iteration 4, duration: 11905 ms , transferred pages: 97360 (dup: 18, rd: 93610, fd: 3732) , new dirty pages: 99150 , remaining dirty pages: 101513
Iteration 5, duration: 12105 ms , transferred pages: 99094 (dup: 116, rd: 94125, fd: 4853) , new dirty pages: 98589 , remaining dirty pages: 101008
Iteration 6, duration: 12101 ms , transferred pages: 98995 (dup: 17, rd: 94069, fd: 4909) , new dirty pages: 147403 , remaining dirty pages: 149416
Iteration 7, duration: 18001 ms , transferred pages: 147284 (dup: 44, rd: 135691, fd: 11549) , new dirty pages: 136445 , remaining dirty pages: 138577
Iteration 8, duration: 16702 ms , transferred pages: 136636 (dup: 30, rd: 130805, fd: 5801) , new dirty pages: 145481 , remaining dirty pages: 147422
Iteration 9, duration: 17800 ms , transferred pages: 145734 (dup: 130, rd: 133239, fd: 12365) , new dirty pages: 98032 , remaining dirty pages: 99720
Iteration 10, duration: 11902 ms , transferred pages: 97364 (dup: 22, rd: 93096, fd: 4246) , new dirty pages: 95391 , remaining dirty pages: 97747
13. cpu2006.cactusADM (can not converge)
Iteration 1, duration: 23508 ms , transferred pages: 266450 (dup: 73568, rd: 192882, fd: 0) , new dirty pages: 123869 , remaining dirty pages: 123869
Iteration 2, duration: 13989 ms , transferred pages: 121594 (dup: 7874, rd: 81653, fd: 32067) , new dirty pages: 112960 , remaining dirty pages: 115235
Iteration 3, duration: 13605 ms , transferred pages: 113276 (dup: 2028, rd: 83783, fd: 27465) , new dirty pages: 112314 , remaining dirty pages: 114273
Iteration 4, duration: 13509 ms , transferred pages: 111935 (dup: 1505, rd: 83535, fd: 26895) , new dirty pages: 114078 , remaining dirty pages: 116416
Iteration 5, duration: 13810 ms , transferred pages: 114262 (dup: 1378, rd: 84039, fd: 28845) , new dirty pages: 112271 , remaining dirty pages: 114425
Iteration 6, duration: 13604 ms , transferred pages: 112664 (dup: 1416, rd: 84300, fd: 26948) , new dirty pages: 112903 , remaining dirty pages: 114664
Iteration 7, duration: 13604 ms , transferred pages: 112655 (dup: 1407, rd: 84027, fd: 27221) , new dirty pages: 110943 , remaining dirty pages: 112952
Iteration 8, duration: 13406 ms , transferred pages: 110720 (dup: 1108, rd: 84075, fd: 25537) , new dirty pages: 109321 , remaining dirty pages: 111553
Iteration 9, duration: 13306 ms , transferred pages: 109726 (dup: 932, rd: 83652, fd: 25142) , new dirty pages: 113446 , remaining dirty pages: 115273
Iteration 10, duration: 13705 ms , transferred pages: 113121 (dup: 1055, rd: 84671, fd: 27395) , new dirty pages: 108776 , remaining dirty pages: 110928
14. cpu2006.GmesFDTD (can not converge)
Iteration 1, duration: 13303 ms , transferred pages: 266450 (dup: 157809, rd: 108641, fd: 0) , new dirty pages: 226802 , remaining dirty pages: 226802
Iteration 2, duration: 10797 ms , transferred pages: 226507 (dup: 138637, rd: 61818, fd: 26052) , new dirty pages: 200769 , remaining dirty pages: 201064
Iteration 3, duration: 8900 ms , transferred pages: 199717 (dup: 127187, rd: 69340, fd: 3190) , new dirty pages: 203436 , remaining dirty pages: 204783
Iteration 4, duration: 10904 ms , transferred pages: 204127 (dup: 115211, rd: 85767, fd: 3149) , new dirty pages: 198407 , remaining dirty pages: 199063
Iteration 5, duration: 12109 ms , transferred pages: 198206 (dup: 99435, rd: 96956, fd: 1815) , new dirty pages: 213719 , remaining dirty pages: 214576
Iteration 6, duration: 16307 ms , transferred pages: 213595 (dup: 80422, rd: 116885, fd: 16288) , new dirty pages: 199637 , remaining dirty pages: 200618
Iteration 7, duration: 16915 ms , transferred pages: 198289 (dup: 60169, rd: 134208, fd: 3912) , new dirty pages: 199343 , remaining dirty pages: 201672
Iteration 8, duration: 19518 ms , transferred pages: 200452 (dup: 41014, rd: 156083, fd: 3355) , new dirty pages: 222927 , remaining dirty pages: 224147
15. cpu2006.wrf (can not converge)
Iteration 1, duration: 18499 ms , transferred pages: 266380 (dup: 115285, rd: 151095, fd: 0) , new dirty pages: 112322 , remaining dirty pages: 112392
Iteration 2, duration: 9802 ms , transferred pages: 110025 (dup: 29917, rd: 65782, fd: 14326) , new dirty pages: 88855 , remaining dirty pages: 91222
Iteration 3, duration: 8199 ms , transferred pages: 89761 (dup: 22728, rd: 57262, fd: 9771) , new dirty pages: 58431 , remaining dirty pages: 59892
Iteration 4, duration: 5603 ms , transferred pages: 58502 (dup: 12716, rd: 41809, fd: 3977) , new dirty pages: 80556 , remaining dirty pages: 81946
Iteration 5, duration: 7101 ms , transferred pages: 79778 (dup: 21738, rd: 50896, fd: 7144) , new dirty pages: 62592 , remaining dirty pages: 64760
Iteration 6, duration: 5702 ms , transferred pages: 63388 (dup: 16793, rd: 42726, fd: 3869) , new dirty pages: 80747 , remaining dirty pages: 82119
Iteration 7, duration: 7000 ms , transferred pages: 80868 (dup: 23652, rd: 52194, fd: 5022) , new dirty pages: 84593 , remaining dirty pages: 85844
Iteration 8, duration: 7099 ms , transferred pages: 83799 (dup: 25769, rd: 51772, fd: 6258) , new dirty pages: 67951 , remaining dirty pages: 69996
Iteration 9, duration: 6303 ms , transferred pages: 68478 (dup: 16979, rd: 36490, fd: 15009) , new dirty pages: 81181 , remaining dirty pages: 82699
Iteration 10, duration: 7000 ms , transferred pages: 80724 (dup: 23503, rd: 52826, fd: 4395) , new dirty pages: 47930 , remaining dirty pages: 49905
>
> > So I think "booting" and "kernel compilation" should benefit a lot from this
> > improvement. The reason of "kernel compilation" would benefit is that some
> > iterations take around 600ms, and if they are halved into 300ms, then the precopy
> > may have the chance to step into stop and copy phase.
> >
> > On the other hand, "idle" and "web server" would not benefit a lot, because
> > most of the time are spent on the 1st iteration and little on the others.
> >
> > As to the "zeusmp" and "memcached", although the time spent on the other iterations
> > but the 1st one may be halved, they still could not converge to stop and copy
> > with the 300ms downtime.
> >
> > --------------------1 vcpu, 1 GB ram, default bandwidth (32MB/s):------------------
> >
> > 1. booting : begin to migrate when the VM is booting
> >
> > Iteration 1, duration: 6997 ms , transferred pages: 266450 (n: 57269, d: 209181 ) , new dirty pages: 56414 , remaining dirty pages: 56414
> > Iteration 2, duration: 6497 ms , transferred pages: 54008 (n: 52701, d: 1307 ) , new dirty pages: 48053 , remaining dirty pages: 50459
> > Iteration 3, duration: 5800 ms , transferred pages: 48232 (n: 47444, d: 788 ) , new dirty pages: 9129 , remaining dirty pages: 11356
> > Iteration 4, duration: 1100 ms , transferred pages: 9091 (n: 8998, d: 93 ) , new dirty pages: 165 , remaining dirty pages: 2430
> > Iteration 5, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2430
> > (note: When the workload does converge, the output of the last iteration is "fake". It just indicates that the precopy steps into stop-copy phase now.
> > "n" means "normal pages" and "d" means "duplicate (zero) pages".)
> >
> > 2. idle
> >
> > Iteration 1, duration: 14496 ms , transferred pages: 266450 (n: 118980, d: 147470 ) , new dirty pages: 17398 , remaining dirty pages: 17398
> > Iteration 2, duration: 1896 ms , transferred pages: 14953 (n: 14854, d: 99 ) , new dirty pages: 1849 , remaining dirty pages: 4294
> > Iteration 3, duration: 300 ms , transferred pages: 2454 (n: 2454, d: 0 ) , new dirty pages: 9 , remaining dirty pages: 1849
> > Iteration 4, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 1849
> >
> > 3. kernel compilation (can not converge)
> >
> > Iteration 1, duration: 20700 ms , transferred pages: 266450 (n: 169778, d: 96672 ) , new dirty pages: 40067 , remaining dirty pages: 40067
> > Iteration 2, duration: 4696 ms , transferred pages: 38401 (n: 37787, d: 614 ) , new dirty pages: 8852 , remaining dirty pages: 10518
> > Iteration 3, duration: 1000 ms , transferred pages: 8642 (n: 8180, d: 462 ) , new dirty pages: 6331 , remaining dirty pages: 8207
> > Iteration 4, duration: 700 ms , transferred pages: 6110 (n: 5726, d: 384 ) , new dirty pages: 5242 , remaining dirty pages: 7339
> > Iteration 5, duration: 600 ms , transferred pages: 5007 (n: 4908, d: 99 ) , new dirty pages: 4868 , remaining dirty pages: 7200
> > Iteration 6, duration: 600 ms , transferred pages: 5226 (n: 4908, d: 318 ) , new dirty pages: 6142 , remaining dirty pages: 8116
> > Iteration 7, duration: 700 ms , transferred pages: 5985 (n: 5726, d: 259 ) , new dirty pages: 5902 , remaining dirty pages: 8033
> > Iteration 8, duration: 701 ms , transferred pages: 5893 (n: 5726, d: 167 ) , new dirty pages: 7502 , remaining dirty pages: 9642
> > Iteration 9, duration: 900 ms , transferred pages: 7623 (n: 7362, d: 261 ) , new dirty pages: 6408 , remaining dirty pages: 8427
> > Iteration 10, duration: 700 ms , transferred pages: 6008 (n: 5726, d: 282 ) , new dirty pages: 8312 , remaining dirty pages: 10731
> > Iteration 11, duration: 1000 ms , transferred pages: 8353 (n: 8180, d: 173 ) , new dirty pages: 6874 , remaining dirty pages: 9252
> > Iteration 12, duration: 899 ms , transferred pages: 7477 (n: 7362, d: 115 ) , new dirty pages: 5573 , remaining dirty pages: 7348
> > Iteration 13, duration: 601 ms , transferred pages: 5099 (n: 4908, d: 191 ) , new dirty pages: 7671 , remaining dirty pages: 9920
> > Iteration 14, duration: 900 ms , transferred pages: 7586 (n: 7362, d: 224 ) , new dirty pages: 7359 , remaining dirty pages: 9693
> > Iteration 15, duration: 900 ms , transferred pages: 7682 (n: 7362, d: 320 ) , new dirty pages: 7371 , remaining dirty pages: 9382
> >
> > 4. cpu2006.zeusmp (can not converge)
> >
> > Iteration 1, duration: 21603 ms , transferred pages: 266450 (n: 176660, d: 89790 ) , new dirty pages: 145625 , remaining dirty pages: 145625
> > Iteration 2, duration: 8696 ms , transferred pages: 144389 (n: 70862, d: 73527 ) , new dirty pages: 125124 , remaining dirty pages: 126360
> > Iteration 3, duration: 6301 ms , transferred pages: 124057 (n: 51379, d: 72678 ) , new dirty pages: 122528 , remaining dirty pages: 124831
> > Iteration 4, duration: 6400 ms , transferred pages: 124330 (n: 52196, d: 72134 ) , new dirty pages: 124267 , remaining dirty pages: 124768
> > Iteration 5, duration: 6703 ms , transferred pages: 124034 (n: 54656, d: 69378 ) , new dirty pages: 124151 , remaining dirty pages: 124885
> > Iteration 6, duration: 6703 ms , transferred pages: 124357 (n: 54658, d: 69699 ) , new dirty pages: 124106 , remaining dirty pages: 124634
> > Iteration 7, duration: 6602 ms , transferred pages: 124568 (n: 53838, d: 70730 ) , new dirty pages: 133828 , remaining dirty pages: 133894
> > Iteration 8, duration: 7600 ms , transferred pages: 133030 (n: 62021, d: 71009 ) , new dirty pages: 126612 , remaining dirty pages: 127476
> > Iteration 9, duration: 7299 ms , transferred pages: 126511 (n: 59569, d: 66942 ) , new dirty pages: 122727 , remaining dirty pages: 123692
> > Iteration 10, duration: 6609 ms , transferred pages: 123692 (n: 54539, d: 69153 ) , new dirty pages: 122727 , remaining dirty pages: 122727
> > Iteration 11, duration: 6995 ms , transferred pages: 120347 (n: 56423, d: 63924 ) , new dirty pages: 121430 , remaining dirty pages: 123810
> > Iteration 12, duration: 6703 ms , transferred pages: 123040 (n: 54657, d: 68383 ) , new dirty pages: 122043 , remaining dirty pages: 122813
> > Iteration 13, duration: 7006 ms , transferred pages: 122353 (n: 57121, d: 65232 ) , new dirty pages: 133869 , remaining dirty pages: 134329
> > Iteration 14, duration: 8209 ms , transferred pages: 132325 (n: 66932, d: 65393 ) , new dirty pages: 126914 , remaining dirty pages: 128918
> > Iteration 15, duration: 7802 ms , transferred pages: 126931 (n: 63671, d: 63260 ) , new dirty pages: 122351 , remaining dirty pages: 124338
> >
> > 5. web server : An apache web server. The client is configured with 50 concurrent connections.
> >
> > Iteration 1, duration: 30697 ms , transferred pages: 266450 (n: 251215, d: 15235 ) , new dirty pages: 30628 , remaining dirty pages: 30628
> > Iteration 2, duration: 3496 ms , transferred pages: 28859 (n: 28513, d: 346 ) , new dirty pages: 5805 , remaining dirty pages: 7574
> > Iteration 3, duration: 701 ms , transferred pages: 5746 (n: 5726, d: 20 ) , new dirty pages: 3433 , remaining dirty pages: 5261
> > Iteration 4, duration: 400 ms , transferred pages: 3281 (n: 3272, d: 9 ) , new dirty pages: 1539 , remaining dirty pages: 3519
> > Iteration 5, duration: 199 ms , transferred pages: 1653 (n: 1636, d: 17 ) , new dirty pages: 301 , remaining dirty pages: 2167
> > Iteration 6, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2167
> >
> > --------------------6 vcpu, 6 GB ram, max bandwidth (941.08 mbps):------------------
> >
> > 6. memcached : 4 GB cache, memaslap: all write, concurrency = 5 (can not converge)
> >
> > Iteration 1, duration: 42486 ms , transferred pages: 1568087 (n: 1216079, d: 352008 ) , new dirty pages: 571940 , remaining dirty pages: 581023
> > Iteration 2, duration: 19774 ms , transferred pages: 571700 (n: 567416, d: 4284 ) , new dirty pages: 331690 , remaining dirty pages: 341013
> > Iteration 3, duration: 11589 ms , transferred pages: 332187 (n: 332095, d: 92 ) , new dirty pages: 222725 , remaining dirty pages: 231551
> > Iteration 4, duration: 7790 ms , transferred pages: 223571 (n: 223499, d: 72 ) , new dirty pages: 157658 , remaining dirty pages: 165638
> > Iteration 5, duration: 5518 ms , transferred pages: 158056 (n: 157998, d: 58 ) , new dirty pages: 128130 , remaining dirty pages: 135712
> > Iteration 6, duration: 4442 ms , transferred pages: 127764 (n: 127701, d: 63 ) , new dirty pages: 104839 , remaining dirty pages: 112787
> > Iteration 7, duration: 3649 ms , transferred pages: 104581 (n: 104523, d: 58 ) , new dirty pages: 100736 , remaining dirty pages: 108942
> > Iteration 8, duration: 3532 ms , transferred pages: 101379 (n: 101315, d: 64 ) , new dirty pages: 87869 , remaining dirty pages: 95432
> > Iteration 9, duration: 3030 ms , transferred pages: 86841 (n: 86786, d: 55 ) , new dirty pages: 77505 , remaining dirty pages: 86096
> > Iteration 10, duration: 2709 ms , transferred pages: 77875 (n: 77814, d: 61 ) , new dirty pages: 77197 , remaining dirty pages: 85418
> > Iteration 11, duration: 2696 ms , transferred pages: 77107 (n: 77044, d: 63 ) , new dirty pages: 65010 , remaining dirty pages: 73321
> > Iteration 12, duration: 2308 ms , transferred pages: 66540 (n: 66484, d: 56 ) , new dirty pages: 64388 , remaining dirty pages: 71169
> > Iteration 13, duration: 2198 ms , transferred pages: 62953 (n: 62897, d: 56 ) , new dirty pages: 62773 , remaining dirty pages: 70989
> > Iteration 14, duration: 2214 ms , transferred pages: 63466 (n: 63411, d: 55 ) , new dirty pages: 67538 , remaining dirty pages: 75061
> > Iteration 15, duration: 2329 ms , transferred pages: 66924 (n: 66875, d: 49 ) , new dirty pages: 63580 , remaining dirty pages: 71717
> > Iteration 16, duration: 2252 ms , transferred pages: 64554 (n: 64539, d: 15 ) , new dirty pages: 63094 , remaining dirty pages: 70257
> > Iteration 17, duration: 2188 ms , transferred pages: 62697 (n: 62641, d: 56 ) , new dirty pages: 63016 , remaining dirty pages: 70576
> > Iteration 18, duration: 2171 ms , transferred pages: 62377 (n: 62322, d: 55 ) , new dirty pages: 56764 , remaining dirty pages: 64963
> > Iteration 19, duration: 2003 ms , transferred pages: 57382 (n: 57324, d: 58 ) , new dirty pages: 65307 , remaining dirty pages: 72888
> > Iteration 20, duration: 2240 ms , transferred pages: 64426 (n: 64364, d: 62 ) , new dirty pages: 61585 , remaining dirty pages: 70047
> >
> >
> > --
> > Chunguang Li, Ph.D. Candidate
> > Wuhan National Laboratory for Optoelectronics (WNLO)
> > Huazhong University of Science & Technology (HUST)
> > Wuhan, Hubei Prov., China
> >
> >
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-03 8:25 ` Chunguang Li
@ 2016-11-03 9:59 ` Li, Liang Z
2016-11-03 10:13 ` Li, Liang Z
2016-11-08 11:05 ` Dr. David Alan Gilbert
2 siblings, 0 replies; 21+ messages in thread
From: Li, Liang Z @ 2016-11-03 9:59 UTC (permalink / raw)
To: Chunguang Li, Dr. David Alan Gilbert
Cc: Amit Shah, pbonzini, qemu-devel, stefanha, quintela
> pages will be sent. Before that during the migration setup, the
> ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce
> the dirty bitmap from this moment. When the pages "that haven't been
> sent" are written, the kernel space marks them as dirty. However I don't
> think this is correct, because these pages will be sent during this and the next
> iterations with the same content (if they are not written again after they are
> sent). It only makes sense to mark the pages which have already been sent
> during one iteration as dirty when they are written.
> > > > > > >
> > > > > > >
> > > > > > > Am I right about this consideration? If I am right, is there some
> advice to improve this?
> > > > > >
> > > > > > I think you're right that this can happen; to clarify I think the
> > > > > > case you're talking about is:
> > > > > >
> > > > > > Iteration 1
> > > > > > sync bitmap
> > > > > > start sending pages
> > > > > > page 'n' is modified - but hasn't been sent yet
> > > > > > page 'n' gets sent
> > > > > > Iteration 2
> > > > > > sync bitmap
> > > > > > 'page n is shown as modified'
> > > > > > send page 'n' again
> > > > > >
> > > > >
> > > > > Yes，this is right the case I am talking about.
> > > > >
> > > > > > So you're right that is wasteful; I guess it's more wasteful
> > > > > > on big VMs with slow networks where the length of each iteration
> > > > > > is large.
> > > > >
> > > > > I think this is "very" wasteful. Assume the workload writes the pages
> dirty randomly within the guest address space, and the transfer speed is
> constant. Intuitively, I think nearly half of the dirty pages produced in
> Iteration 1 is not really dirty. This means the time of Iteration 2 is double of
> that to send only really dirty pages.
> > > >
> > > > It makes sense, can you get some perf numbers to show what kinds of
> > > > workloads get impacted the most? That would also help us to figure
> > > > out what kinds of speed improvements we can expect.
> > > >
> > > >
> > > > Amit
> > >
> > > I have picked up 6 workloads and got the following statistics numbers
> > > of every iteration (except the last stop-copy one) during precopy.
> > > These numbers are obtained with the basic precopy migration, without
> > > the capabilities like xbzrle or compression, etc. The network for the
> > > migration is exclusive, with a separate network for the workloads.
> > > They are both gigabit ethernet. I use qemu-2.5.1.
> > >
> > > Three (booting, idle, web server) of them converged to the stop-copy
> phase,
> > > with the given bandwidth and default downtime (300ms), while the other
> > > three (kernel compilation, zeusmp, memcached) did not.
> > >
> > > One page is "not-really-dirty", if it is written first and is sent later
> > > (and not written again after that) during one iteration. I guess this
> > > would not happen so often during the other iterations as during the 1st
> > > iteration. Because all the pages of the VM are sent to the dest node
> during
> > > the 1st iteration, while during the others, only part of the pages are sent.
> > > So I think the "not-really-dirty" pages should be produced mainly during
> > > the 1st iteration , and maybe very little during the other iterations.
> > >
> > > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > > think the time spent on Iteration 2 would be halved. This is a chain
> reaction,
> > > because the dirty pages produced during Iteration 2 is halved, which
> incurs
> > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> >
> > Yes; these numbers don't show how many of them are false dirty though.
> >
> > One problem is thinking about pages that have been redirtied, if the page is
> dirtied
> > after the sync but before the network write then it's the false-dirty that
> > you're describing.
> >
> > However, if the page is being written a few times, and so it would have
> been written
> > after the network write then it isn't a false-dirty.
> >
> > You might be able to figure that out with some kernel tracing of when the
> dirtying
> > happens, but it might be easier to write the fix!
> >
> > Dave
>
> Hi, I have made some new progress now.
>
> To tell how many false dirty pages there are exactly in each iteration, I malloc
> a
> buffer in memory as big as the size of the whole VM memory. When a page
> is
> transferred to the dest node, it is copied to the buffer; During the next
> iteration,
> if one page is transferred, it is compared to the old one in the buffer, and the
> old one will be replaced for next comparison if it is really dirty. Thus, we are
> now
> able to get the exact number of false dirty pages.
>
> This time, I use 15 workloads to get the statistic number. They are:
>
> 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all
> scientific
> computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc.
> I pick
> up these 11 benchmarks because compared to others, they have bigger
> memory
> occupation and higher memory dirty rate. Thus most of them could not
> converge
> to stop-and-copy using the default migration speed (32MB/s).
> 2. kernel compilation
> 3. idle VM
> 4. Apache web server which serves static content
>
> (the above workloads are all running in VM with 1 vcpu and 1GB memory,
> and the
> migration speed is the default 32MB/s)
>
> 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used
> as the cache.
> After filling up the 4GB cache, a client writes the cache at a constant speed
> during migration. This time, migration speed has no limit, and is up to the
> capability of 1Gbps Ethernet.
>
> Summarize the results first: (and you can read the precise number below)
>
> 1. 4 of these 15 workloads have a big proportion (>60%, even >80% during
> some iterations)
> of false dirty pages out of all the dirty pages since iteration 2 (and the big
> proportion lasts during the following iterations). They are cpu2006.zeusmp,
> cpu2006.bzip2, cpu2006.mcf, and memcached.
> 2. 2 workloads (idle, webserver) spend most of the migration time on
> iteration 1, even
> though the proportion of false dirty pages is big since iteration 2, the space
> to
> optimize is small.
> 3. 1 workload (kernel compilation) only have a big proportion during
> iteration 2, not
> in the other iterations.
> 4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of
> false
> dirty pages since iteration 2. So the spaces to optimize for them are small.
>
> Now I want to talk a little more about the reasons why false dirty pages are
> produced.
> The first reason is what we have discussed before---the mechanism to track
> the dirty
> pages.
> And then I come up with another reason. Here is the situation: a write
> operation to one
> memory page happens, but it doesn't change any content of the page. So it's
> "write but
> not dirty", and kernel still marks it as dirty. One guy in our lab has done some
> experiments
> to figure out the proportion of "write but not dirty" operations, and he uses
> the cpu2006
> benchmark suit. According to his results, general workloads has a little
> proportion (<10%)
> of "write but not dirty" out of all the write operations, while few workloads
> have higher
> proportion (one even as high as 50%). Now we are not sure why "write but
> not dirty" would
> happen, it just happened.
>
> So these two reasons contribute to the false dirty pages. To optimize, I
> compute and store
> the SHA1 hash before transferring each page. Next time, if one page needs
> retransmission, its
> SHA1 hash is computed again, and compared to the old hash. If the hash is
> the same, it's a
> false dirty page, and we just skip this page; Otherwise, the page is
> transferred, and the new
> hash replaces the old one for next comparison.
> The reason to use SHA1 hash but not byte-by-byte comparison is the
> memory overheads. One SHA1
> hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the
> whole VM memory, which
> is relatively small.
> As far as I know, SHA1 hash is widely used in the scenes of deduplication for
> backup systems.
> They have proven that the probability of hash collision is far smaller than disk
> hardware fault,
> so it's secure hash, that is, if the hashes of two chunks are the same, the
> content must be the
> same. So I think the SHA1 hash could replace byte-to-byte comparison in the
> VM memory scenery.
>
> Then I do the same migration experiments using the SHA1 hash. For the 4
> workloads which have
> big proportions of false dirty pages, the improvement is remarkable. Without
> optimization,
> they either can not converge to stop-and-copy, or take a very long time to
> complete. With the
> SHA1 hash method, all of them now complete in a relatively short time.
> For the reason I have talked above, the other workloads don't get notable
> improvements from the
> optimization. So below, I only show the exact number after optimization for
> the 4 workloads with
> remarkable improvements.
>
> Any comments or suggestions?
>
It seems the current XBZRLE feature can be used to solve false dirty issue, no?
Liang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-03 8:25 ` Chunguang Li
2016-11-03 9:59 ` Li, Liang Z
@ 2016-11-03 10:13 ` Li, Liang Z
2016-11-04 3:07 ` Chunguang Li
2016-11-08 11:05 ` Dr. David Alan Gilbert
2 siblings, 1 reply; 21+ messages in thread
From: Li, Liang Z @ 2016-11-03 10:13 UTC (permalink / raw)
To: Chunguang Li, Dr. David Alan Gilbert
Cc: Amit Shah, pbonzini, qemu-devel, stefanha, quintela
> > > > > I think this is "very" wasteful. Assume the workload writes the pages
> dirty randomly within the guest address space, and the transfer speed is
> constant. Intuitively, I think nearly half of the dirty pages produced in
> Iteration 1 is not really dirty. This means the time of Iteration 2 is double of
> that to send only really dirty pages.
> > > >
> > > > It makes sense, can you get some perf numbers to show what kinds of
> > > > workloads get impacted the most? That would also help us to figure
> > > > out what kinds of speed improvements we can expect.
> > > >
> > > >
> > > > Amit
> > >
> > > I have picked up 6 workloads and got the following statistics numbers
> > > of every iteration (except the last stop-copy one) during precopy.
> > > These numbers are obtained with the basic precopy migration, without
> > > the capabilities like xbzrle or compression, etc. The network for the
> > > migration is exclusive, with a separate network for the workloads.
> > > They are both gigabit ethernet. I use qemu-2.5.1.
> > >
> > > Three (booting, idle, web server) of them converged to the stop-copy
> phase,
> > > with the given bandwidth and default downtime (300ms), while the other
> > > three (kernel compilation, zeusmp, memcached) did not.
> > >
> > > One page is "not-really-dirty", if it is written first and is sent later
> > > (and not written again after that) during one iteration. I guess this
> > > would not happen so often during the other iterations as during the 1st
> > > iteration. Because all the pages of the VM are sent to the dest node
> during
> > > the 1st iteration, while during the others, only part of the pages are sent.
> > > So I think the "not-really-dirty" pages should be produced mainly during
> > > the 1st iteration , and maybe very little during the other iterations.
> > >
> > > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > > think the time spent on Iteration 2 would be halved. This is a chain
> reaction,
> > > because the dirty pages produced during Iteration 2 is halved, which
> incurs
> > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> >
> > Yes; these numbers don't show how many of them are false dirty though.
> >
> > One problem is thinking about pages that have been redirtied, if the page is
> dirtied
> > after the sync but before the network write then it's the false-dirty that
> > you're describing.
> >
> > However, if the page is being written a few times, and so it would have
> been written
> > after the network write then it isn't a false-dirty.
> >
> > You might be able to figure that out with some kernel tracing of when the
> dirtying
> > happens, but it might be easier to write the fix!
> >
> > Dave
>
> Hi, I have made some new progress now.
>
> To tell how many false dirty pages there are exactly in each iteration, I malloc
> a
> buffer in memory as big as the size of the whole VM memory. When a page
> is
> transferred to the dest node, it is copied to the buffer; During the next
> iteration,
> if one page is transferred, it is compared to the old one in the buffer, and the
> old one will be replaced for next comparison if it is really dirty. Thus, we are
> now
> able to get the exact number of false dirty pages.
>
> This time, I use 15 workloads to get the statistic number. They are:
>
> 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all
> scientific
> computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc.
> I pick
> up these 11 benchmarks because compared to others, they have bigger
> memory
> occupation and higher memory dirty rate. Thus most of them could not
> converge
> to stop-and-copy using the default migration speed (32MB/s).
> 2. kernel compilation
> 3. idle VM
> 4. Apache web server which serves static content
>
> (the above workloads are all running in VM with 1 vcpu and 1GB memory,
> and the
> migration speed is the default 32MB/s)
>
> 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used
> as the cache.
> After filling up the 4GB cache, a client writes the cache at a constant speed
> during migration. This time, migration speed has no limit, and is up to the
> capability of 1Gbps Ethernet.
>
> Summarize the results first: (and you can read the precise number below)
>
> 1. 4 of these 15 workloads have a big proportion (>60%, even >80% during
> some iterations)
> of false dirty pages out of all the dirty pages since iteration 2 (and the big
> proportion lasts during the following iterations). They are cpu2006.zeusmp,
> cpu2006.bzip2, cpu2006.mcf, and memcached.
> 2. 2 workloads (idle, webserver) spend most of the migration time on
> iteration 1, even
> though the proportion of false dirty pages is big since iteration 2, the space
> to
> optimize is small.
> 3. 1 workload (kernel compilation) only have a big proportion during
> iteration 2, not
> in the other iterations.
> 4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of
> false
> dirty pages since iteration 2. So the spaces to optimize for them are small.
>
> Now I want to talk a little more about the reasons why false dirty pages are
> produced.
> The first reason is what we have discussed before---the mechanism to track
> the dirty
> pages.
> And then I come up with another reason. Here is the situation: a write
> operation to one
> memory page happens, but it doesn't change any content of the page. So it's
> "write but
> not dirty", and kernel still marks it as dirty. One guy in our lab has done some
> experiments
> to figure out the proportion of "write but not dirty" operations, and he uses
> the cpu2006
> benchmark suit. According to his results, general workloads has a little
> proportion (<10%)
> of "write but not dirty" out of all the write operations, while few workloads
> have higher
> proportion (one even as high as 50%). Now we are not sure why "write but
> not dirty" would
> happen, it just happened.
>
> So these two reasons contribute to the false dirty pages. To optimize, I
> compute and store
> the SHA1 hash before transferring each page. Next time, if one page needs
> retransmission, its
> SHA1 hash is computed again, and compared to the old hash. If the hash is
> the same, it's a
> false dirty page, and we just skip this page; Otherwise, the page is
> transferred, and the new
> hash replaces the old one for next comparison.
> The reason to use SHA1 hash but not byte-by-byte comparison is the
> memory overheads. One SHA1
> hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the
> whole VM memory, which
> is relatively small.
> As far as I know, SHA1 hash is widely used in the scenes of deduplication for
> backup systems.
> They have proven that the probability of hash collision is far smaller than disk
> hardware fault,
> so it's secure hash, that is, if the hashes of two chunks are the same, the
> content must be the
> same. So I think the SHA1 hash could replace byte-to-byte comparison in the
> VM memory scenery.
>
> Then I do the same migration experiments using the SHA1 hash. For the 4
> workloads which have
> big proportions of false dirty pages, the improvement is remarkable. Without
> optimization,
> they either can not converge to stop-and-copy, or take a very long time to
> complete. With the
> SHA1 hash method, all of them now complete in a relatively short time.
> For the reason I have talked above, the other workloads don't get notable
> improvements from the
> optimization. So below, I only show the exact number after optimization for
> the 4 workloads with
> remarkable improvements.
>
> Any comments or suggestions?
Maybe you can compare the performance of your solution as that of XBZRLE to see which one is better.
The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, and need less buffer.
How about the overhead of calculating the SHA1? Is it faster than copying a page?
Liang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-03 10:13 ` Li, Liang Z
@ 2016-11-04 3:07 ` Chunguang Li
2016-11-04 4:50 ` Li, Liang Z
0 siblings, 1 reply; 21+ messages in thread
From: Chunguang Li @ 2016-11-04 3:07 UTC (permalink / raw)
To: Li, Liang Z
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> -----Original Messages-----
> From: "Li, Liang Z" <liang.z.li@intel.com>
> Sent Time: Thursday, November 3, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: "Amit Shah" <amit.shah@redhat.com>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "stefanha@redhat.com" <stefanha@redhat.com>, "quintela@redhat.com" <quintela@redhat.com>
> Subject: RE: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> > > > > > I think this is "very" wasteful. Assume the workload writes the pages
> > dirty randomly within the guest address space, and the transfer speed is
> > constant. Intuitively, I think nearly half of the dirty pages produced in
> > Iteration 1 is not really dirty. This means the time of Iteration 2 is double of
> > that to send only really dirty pages.
> > > > >
> > > > > It makes sense, can you get some perf numbers to show what kinds of
> > > > > workloads get impacted the most? That would also help us to figure
> > > > > out what kinds of speed improvements we can expect.
> > > > >
> > > > >
> > > > > Amit
> > > >
> > > > I have picked up 6 workloads and got the following statistics numbers
> > > > of every iteration (except the last stop-copy one) during precopy.
> > > > These numbers are obtained with the basic precopy migration, without
> > > > the capabilities like xbzrle or compression, etc. The network for the
> > > > migration is exclusive, with a separate network for the workloads.
> > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > >
> > > > Three (booting, idle, web server) of them converged to the stop-copy
> > phase,
> > > > with the given bandwidth and default downtime (300ms), while the other
> > > > three (kernel compilation, zeusmp, memcached) did not.
> > > >
> > > > One page is "not-really-dirty", if it is written first and is sent later
> > > > (and not written again after that) during one iteration. I guess this
> > > > would not happen so often during the other iterations as during the 1st
> > > > iteration. Because all the pages of the VM are sent to the dest node
> > during
> > > > the 1st iteration, while during the others, only part of the pages are sent.
> > > > So I think the "not-really-dirty" pages should be produced mainly during
> > > > the 1st iteration , and maybe very little during the other iterations.
> > > >
> > > > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > > > think the time spent on Iteration 2 would be halved. This is a chain
> > reaction,
> > > > because the dirty pages produced during Iteration 2 is halved, which
> > incurs
> > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > >
> > > Yes; these numbers don't show how many of them are false dirty though.
> > >
> > > One problem is thinking about pages that have been redirtied, if the page is
> > dirtied
> > > after the sync but before the network write then it's the false-dirty that
> > > you're describing.
> > >
> > > However, if the page is being written a few times, and so it would have
> > been written
> > > after the network write then it isn't a false-dirty.
> > >
> > > You might be able to figure that out with some kernel tracing of when the
> > dirtying
> > > happens, but it might be easier to write the fix!
> > >
> > > Dave
> >
> > Hi, I have made some new progress now.
> >
> > To tell how many false dirty pages there are exactly in each iteration, I malloc
> > a
> > buffer in memory as big as the size of the whole VM memory. When a page
> > is
> > transferred to the dest node, it is copied to the buffer; During the next
> > iteration,
> > if one page is transferred, it is compared to the old one in the buffer, and the
> > old one will be replaced for next comparison if it is really dirty. Thus, we are
> > now
> > able to get the exact number of false dirty pages.
> >
> > This time, I use 15 workloads to get the statistic number. They are:
> >
> > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all
> > scientific
> > computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc.
> > I pick
> > up these 11 benchmarks because compared to others, they have bigger
> > memory
> > occupation and higher memory dirty rate. Thus most of them could not
> > converge
> > to stop-and-copy using the default migration speed (32MB/s).
> > 2. kernel compilation
> > 3. idle VM
> > 4. Apache web server which serves static content
> >
> > (the above workloads are all running in VM with 1 vcpu and 1GB memory,
> > and the
> > migration speed is the default 32MB/s)
> >
> > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used
> > as the cache.
> > After filling up the 4GB cache, a client writes the cache at a constant speed
> > during migration. This time, migration speed has no limit, and is up to the
> > capability of 1Gbps Ethernet.
> >
> > Summarize the results first: (and you can read the precise number below)
> >
> > 1. 4 of these 15 workloads have a big proportion (>60%, even >80% during
> > some iterations)
> > of false dirty pages out of all the dirty pages since iteration 2 (and the big
> > proportion lasts during the following iterations). They are cpu2006.zeusmp,
> > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > 2. 2 workloads (idle, webserver) spend most of the migration time on
> > iteration 1, even
> > though the proportion of false dirty pages is big since iteration 2, the space
> > to
> > optimize is small.
> > 3. 1 workload (kernel compilation) only have a big proportion during
> > iteration 2, not
> > in the other iterations.
> > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of
> > false
> > dirty pages since iteration 2. So the spaces to optimize for them are small.
> >
> > Now I want to talk a little more about the reasons why false dirty pages are
> > produced.
> > The first reason is what we have discussed before---the mechanism to track
> > the dirty
> > pages.
> > And then I come up with another reason. Here is the situation: a write
> > operation to one
> > memory page happens, but it doesn't change any content of the page. So it's
> > "write but
> > not dirty", and kernel still marks it as dirty. One guy in our lab has done some
> > experiments
> > to figure out the proportion of "write but not dirty" operations, and he uses
> > the cpu2006
> > benchmark suit. According to his results, general workloads has a little
> > proportion (<10%)
> > of "write but not dirty" out of all the write operations, while few workloads
> > have higher
> > proportion (one even as high as 50%). Now we are not sure why "write but
> > not dirty" would
> > happen, it just happened.
> >
> > So these two reasons contribute to the false dirty pages. To optimize, I
> > compute and store
> > the SHA1 hash before transferring each page. Next time, if one page needs
> > retransmission, its
> > SHA1 hash is computed again, and compared to the old hash. If the hash is
> > the same, it's a
> > false dirty page, and we just skip this page; Otherwise, the page is
> > transferred, and the new
> > hash replaces the old one for next comparison.
> > The reason to use SHA1 hash but not byte-by-byte comparison is the
> > memory overheads. One SHA1
> > hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the
> > whole VM memory, which
> > is relatively small.
> > As far as I know, SHA1 hash is widely used in the scenes of deduplication for
> > backup systems.
> > They have proven that the probability of hash collision is far smaller than disk
> > hardware fault,
> > so it's secure hash, that is, if the hashes of two chunks are the same, the
> > content must be the
> > same. So I think the SHA1 hash could replace byte-to-byte comparison in the
> > VM memory scenery.
> >
> > Then I do the same migration experiments using the SHA1 hash. For the 4
> > workloads which have
> > big proportions of false dirty pages, the improvement is remarkable. Without
> > optimization,
> > they either can not converge to stop-and-copy, or take a very long time to
> > complete. With the
> > SHA1 hash method, all of them now complete in a relatively short time.
> > For the reason I have talked above, the other workloads don't get notable
> > improvements from the
> > optimization. So below, I only show the exact number after optimization for
> > the 4 workloads with
> > remarkable improvements.
> >
> > Any comments or suggestions?
>
> Maybe you can compare the performance of your solution as that of XBZRLE to see which one is better.
> The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, and need less buffer.
> How about the overhead of calculating the SHA1? Is it faster than copying a page?
>
> Liang
>
>
Yes, XBZRLE is able to handle the false dirty pages. However, if we want to avoid
transferring all of the false dirty pages using XBZRLE, we need a buffer as big as
the whole VM memory, while SHA1 needs a much small buffer. Of course, if we
have a buffer as big as the whole VM memory using XBZRLE, we could transfer less data
on network than SHA1, because XBZRLE is able to compress similar pages. In a word, yes,
the merit of using SHA1 is that it needs much less buffer, and leads to nice improvement
if there are many false dirty pages.
In terms of the overhead of calculating the SHA1 compared with transferring a page,
it's related to the CPU and network performance. In my test environment(Intel Xeon
E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra computing overhead
caused by calculating the SHA1, because the throughput of network (got by "info migrate")
remains almost the same.
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-04 3:07 ` Chunguang Li
@ 2016-11-04 4:50 ` Li, Liang Z
2016-11-04 7:03 ` Chunguang Li
2016-11-07 13:52 ` Chunguang Li
0 siblings, 2 replies; 21+ messages in thread
From: Li, Liang Z @ 2016-11-04 4:50 UTC (permalink / raw)
To: Chunguang Li
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> > > > > > > I think this is "very" wasteful. Assume the workload writes
> > > > > > > the pages
> > > dirty randomly within the guest address space, and the transfer
> > > speed is constant. Intuitively, I think nearly half of the dirty
> > > pages produced in Iteration 1 is not really dirty. This means the
> > > time of Iteration 2 is double of that to send only really dirty pages.
> > > > > >
> > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > kinds of workloads get impacted the most? That would also
> > > > > > help us to figure out what kinds of speed improvements we can
> expect.
> > > > > >
> > > > > >
> > > > > > Amit
> > > > >
> > > > > I have picked up 6 workloads and got the following statistics
> > > > > numbers of every iteration (except the last stop-copy one) during
> precopy.
> > > > > These numbers are obtained with the basic precopy migration,
> > > > > without the capabilities like xbzrle or compression, etc. The
> > > > > network for the migration is exclusive, with a separate network for
> the workloads.
> > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > >
> > > > > Three (booting, idle, web server) of them converged to the
> > > > > stop-copy
> > > phase,
> > > > > with the given bandwidth and default downtime (300ms), while the
> > > > > other three (kernel compilation, zeusmp, memcached) did not.
> > > > >
> > > > > One page is "not-really-dirty", if it is written first and is
> > > > > sent later (and not written again after that) during one
> > > > > iteration. I guess this would not happen so often during the
> > > > > other iterations as during the 1st iteration. Because all the
> > > > > pages of the VM are sent to the dest node
> > > during
> > > > > the 1st iteration, while during the others, only part of the pages are
> sent.
> > > > > So I think the "not-really-dirty" pages should be produced
> > > > > mainly during the 1st iteration , and maybe very little during the other
> iterations.
> > > > >
> > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > halved. This is a chain
> > > reaction,
> > > > > because the dirty pages produced during Iteration 2 is halved,
> > > > > which
> > > incurs
> > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > >
> > > > Yes; these numbers don't show how many of them are false dirty
> though.
> > > >
> > > > One problem is thinking about pages that have been redirtied, if
> > > > the page is
> > > dirtied
> > > > after the sync but before the network write then it's the
> > > > false-dirty that you're describing.
> > > >
> > > > However, if the page is being written a few times, and so it would
> > > > have
> > > been written
> > > > after the network write then it isn't a false-dirty.
> > > >
> > > > You might be able to figure that out with some kernel tracing of
> > > > when the
> > > dirtying
> > > > happens, but it might be easier to write the fix!
> > > >
> > > > Dave
> > >
> > > Hi, I have made some new progress now.
> > >
> > > To tell how many false dirty pages there are exactly in each
> > > iteration, I malloc a buffer in memory as big as the size of the
> > > whole VM memory. When a page is transferred to the dest node, it is
> > > copied to the buffer; During the next iteration, if one page is
> > > transferred, it is compared to the old one in the buffer, and the
> > > old one will be replaced for next comparison if it is really dirty.
> > > Thus, we are now able to get the exact number of false dirty pages.
> > >
> > > This time, I use 15 workloads to get the statistic number. They are:
> > >
> > > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are
> > > all scientific
> > > computing workloads like Quantum Chromodynamics, Fluid Dynamics,
> etc.
> > > I pick
> > > up these 11 benchmarks because compared to others, they have
> > > bigger memory
> > > occupation and higher memory dirty rate. Thus most of them
> > > could not converge
> > > to stop-and-copy using the default migration speed (32MB/s).
> > > 2. kernel compilation
> > > 3. idle VM
> > > 4. Apache web server which serves static content
> > >
> > > (the above workloads are all running in VM with 1 vcpu and 1GB
> > > memory, and the
> > > migration speed is the default 32MB/s)
> > >
> > > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are
> > > used as the cache.
> > > After filling up the 4GB cache, a client writes the cache at a constant
> speed
> > > during migration. This time, migration speed has no limit, and is up to
> the
> > > capability of 1Gbps Ethernet.
> > >
> > > Summarize the results first: (and you can read the precise number
> > > below)
> > >
> > > 1. 4 of these 15 workloads have a big proportion (>60%, even >80%
> > > during some iterations)
> > > of false dirty pages out of all the dirty pages since iteration 2 (and the
> big
> > > proportion lasts during the following iterations). They are
> cpu2006.zeusmp,
> > > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > 2. 2 workloads (idle, webserver) spend most of the migration time
> > > on iteration 1, even
> > > though the proportion of false dirty pages is big since
> > > iteration 2, the space to
> > > optimize is small.
> > > 3. 1 workload (kernel compilation) only have a big proportion
> > > during iteration 2, not
> > > in the other iterations.
> > > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > proportion of false
> > > dirty pages since iteration 2. So the spaces to optimize for them are
> small.
> > >
> > > Now I want to talk a little more about the reasons why false dirty
> > > pages are produced.
> > > The first reason is what we have discussed before---the mechanism to
> > > track the dirty pages.
> > > And then I come up with another reason. Here is the situation: a
> > > write operation to one memory page happens, but it doesn't change
> > > any content of the page. So it's "write but not dirty", and kernel
> > > still marks it as dirty. One guy in our lab has done some
> > > experiments to figure out the proportion of "write but not dirty"
> > > operations, and he uses the cpu2006 benchmark suit. According to his
> > > results, general workloads has a little proportion (<10%) of "write
> > > but not dirty" out of all the write operations, while few workloads
> > > have higher proportion (one even as high as 50%). Now we are not
> > > sure why "write but not dirty" would happen, it just happened.
> > >
> > > So these two reasons contribute to the false dirty pages. To
> > > optimize, I compute and store the SHA1 hash before transferring each
> > > page. Next time, if one page needs retransmission, its
> > > SHA1 hash is computed again, and compared to the old hash. If the
> > > hash is the same, it's a false dirty page, and we just skip this
> > > page; Otherwise, the page is transferred, and the new hash replaces
> > > the old one for next comparison.
> > > The reason to use SHA1 hash but not byte-by-byte comparison is the
> > > memory overheads. One SHA1 hash is 20 bytes. So we need extra
> > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > relatively small.
> > > As far as I know, SHA1 hash is widely used in the scenes of
> > > deduplication for backup systems.
> > > They have proven that the probability of hash collision is far
> > > smaller than disk hardware fault, so it's secure hash, that is, if
> > > the hashes of two chunks are the same, the content must be the same.
> > > So I think the SHA1 hash could replace byte-to-byte comparison in
> > > the VM memory scenery.
> > >
> > > Then I do the same migration experiments using the SHA1 hash. For
> > > the 4 workloads which have big proportions of false dirty pages, the
> > > improvement is remarkable. Without optimization, they either can not
> > > converge to stop-and-copy, or take a very long time to complete.
> > > With the
> > > SHA1 hash method, all of them now complete in a relatively short time.
> > > For the reason I have talked above, the other workloads don't get
> > > notable improvements from the optimization. So below, I only show
> > > the exact number after optimization for the 4 workloads with
> > > remarkable improvements.
> > >
> > > Any comments or suggestions?
> >
> > Maybe you can compare the performance of your solution as that of
> XBZRLE to see which one is better.
> > The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, and
> need less buffer.
> > How about the overhead of calculating the SHA1? Is it faster than copying a
> page?
> >
> > Liang
> >
> >
>
> Yes, XBZRLE is able to handle the false dirty pages. However, if we want to
> avoid transferring all of the false dirty pages using XBZRLE, we need a buffer
> as big as the whole VM memory, while SHA1 needs a much small buffer. Of
> course, if we have a buffer as big as the whole VM memory using XBZRLE, we
> could transfer less data on network than SHA1, because XBZRLE is able to
> compress similar pages. In a word, yes, the merit of using SHA1 is that it
> needs much less buffer, and leads to nice improvement if there are many
> false dirty pages.
>
The current implementation of XBZRLE begins to buffer page from the second iteration,
Maybe it's worth to make it start to work from the first iteration based on your finding.
> In terms of the overhead of calculating the SHA1 compared with transferring
> a page, it's related to the CPU and network performance. In my test
> environment(Intel Xeon
> E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra computing
> overhead caused by calculating the SHA1, because the throughput of
> network (got by "info migrate") remains almost the same.
You can check the CPU usage, or to measure the time spend on a local live migration
which use SHA1/ XBZRLE.
Liang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-04 4:50 ` Li, Liang Z
@ 2016-11-04 7:03 ` Chunguang Li
2016-11-07 13:52 ` Chunguang Li
1 sibling, 0 replies; 21+ messages in thread
From: Chunguang Li @ 2016-11-04 7:03 UTC (permalink / raw)
To: Li, Liang Z
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> -----Original Messages-----
> From: "Li, Liang Z" <liang.z.li@intel.com>
> Sent Time: Friday, November 4, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Amit Shah" <amit.shah@redhat.com>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "stefanha@redhat.com" <stefanha@redhat.com>, "quintela@redhat.com" <quintela@redhat.com>
> Subject: RE: RE: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> > > > > > > > I think this is "very" wasteful. Assume the workload writes
> > > > > > > > the pages
> > > > dirty randomly within the guest address space, and the transfer
> > > > speed is constant. Intuitively, I think nearly half of the dirty
> > > > pages produced in Iteration 1 is not really dirty. This means the
> > > > time of Iteration 2 is double of that to send only really dirty pages.
> > > > > > >
> > > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > > kinds of workloads get impacted the most? That would also
> > > > > > > help us to figure out what kinds of speed improvements we can
> > expect.
> > > > > > >
> > > > > > >
> > > > > > > Amit
> > > > > >
> > > > > > I have picked up 6 workloads and got the following statistics
> > > > > > numbers of every iteration (except the last stop-copy one) during
> > precopy.
> > > > > > These numbers are obtained with the basic precopy migration,
> > > > > > without the capabilities like xbzrle or compression, etc. The
> > > > > > network for the migration is exclusive, with a separate network for
> > the workloads.
> > > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > > >
> > > > > > Three (booting, idle, web server) of them converged to the
> > > > > > stop-copy
> > > > phase,
> > > > > > with the given bandwidth and default downtime (300ms), while the
> > > > > > other three (kernel compilation, zeusmp, memcached) did not.
> > > > > >
> > > > > > One page is "not-really-dirty", if it is written first and is
> > > > > > sent later (and not written again after that) during one
> > > > > > iteration. I guess this would not happen so often during the
> > > > > > other iterations as during the 1st iteration. Because all the
> > > > > > pages of the VM are sent to the dest node
> > > > during
> > > > > > the 1st iteration, while during the others, only part of the pages are
> > sent.
> > > > > > So I think the "not-really-dirty" pages should be produced
> > > > > > mainly during the 1st iteration , and maybe very little during the other
> > iterations.
> > > > > >
> > > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > > halved. This is a chain
> > > > reaction,
> > > > > > because the dirty pages produced during Iteration 2 is halved,
> > > > > > which
> > > > incurs
> > > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > > >
> > > > > Yes; these numbers don't show how many of them are false dirty
> > though.
> > > > >
> > > > > One problem is thinking about pages that have been redirtied, if
> > > > > the page is
> > > > dirtied
> > > > > after the sync but before the network write then it's the
> > > > > false-dirty that you're describing.
> > > > >
> > > > > However, if the page is being written a few times, and so it would
> > > > > have
> > > > been written
> > > > > after the network write then it isn't a false-dirty.
> > > > >
> > > > > You might be able to figure that out with some kernel tracing of
> > > > > when the
> > > > dirtying
> > > > > happens, but it might be easier to write the fix!
> > > > >
> > > > > Dave
> > > >
> > > > Hi, I have made some new progress now.
> > > >
> > > > To tell how many false dirty pages there are exactly in each
> > > > iteration, I malloc a buffer in memory as big as the size of the
> > > > whole VM memory. When a page is transferred to the dest node, it is
> > > > copied to the buffer; During the next iteration, if one page is
> > > > transferred, it is compared to the old one in the buffer, and the
> > > > old one will be replaced for next comparison if it is really dirty.
> > > > Thus, we are now able to get the exact number of false dirty pages.
> > > >
> > > > This time, I use 15 workloads to get the statistic number. They are:
> > > >
> > > > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are
> > > > all scientific
> > > > computing workloads like Quantum Chromodynamics, Fluid Dynamics,
> > etc.
> > > > I pick
> > > > up these 11 benchmarks because compared to others, they have
> > > > bigger memory
> > > > occupation and higher memory dirty rate. Thus most of them
> > > > could not converge
> > > > to stop-and-copy using the default migration speed (32MB/s).
> > > > 2. kernel compilation
> > > > 3. idle VM
> > > > 4. Apache web server which serves static content
> > > >
> > > > (the above workloads are all running in VM with 1 vcpu and 1GB
> > > > memory, and the
> > > > migration speed is the default 32MB/s)
> > > >
> > > > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are
> > > > used as the cache.
> > > > After filling up the 4GB cache, a client writes the cache at a constant
> > speed
> > > > during migration. This time, migration speed has no limit, and is up to
> > the
> > > > capability of 1Gbps Ethernet.
> > > >
> > > > Summarize the results first: (and you can read the precise number
> > > > below)
> > > >
> > > > 1. 4 of these 15 workloads have a big proportion (>60%, even >80%
> > > > during some iterations)
> > > > of false dirty pages out of all the dirty pages since iteration 2 (and the
> > big
> > > > proportion lasts during the following iterations). They are
> > cpu2006.zeusmp,
> > > > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > > 2. 2 workloads (idle, webserver) spend most of the migration time
> > > > on iteration 1, even
> > > > though the proportion of false dirty pages is big since
> > > > iteration 2, the space to
> > > > optimize is small.
> > > > 3. 1 workload (kernel compilation) only have a big proportion
> > > > during iteration 2, not
> > > > in the other iterations.
> > > > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > > proportion of false
> > > > dirty pages since iteration 2. So the spaces to optimize for them are
> > small.
> > > >
> > > > Now I want to talk a little more about the reasons why false dirty
> > > > pages are produced.
> > > > The first reason is what we have discussed before---the mechanism to
> > > > track the dirty pages.
> > > > And then I come up with another reason. Here is the situation: a
> > > > write operation to one memory page happens, but it doesn't change
> > > > any content of the page. So it's "write but not dirty", and kernel
> > > > still marks it as dirty. One guy in our lab has done some
> > > > experiments to figure out the proportion of "write but not dirty"
> > > > operations, and he uses the cpu2006 benchmark suit. According to his
> > > > results, general workloads has a little proportion (<10%) of "write
> > > > but not dirty" out of all the write operations, while few workloads
> > > > have higher proportion (one even as high as 50%). Now we are not
> > > > sure why "write but not dirty" would happen, it just happened.
> > > >
> > > > So these two reasons contribute to the false dirty pages. To
> > > > optimize, I compute and store the SHA1 hash before transferring each
> > > > page. Next time, if one page needs retransmission, its
> > > > SHA1 hash is computed again, and compared to the old hash. If the
> > > > hash is the same, it's a false dirty page, and we just skip this
> > > > page; Otherwise, the page is transferred, and the new hash replaces
> > > > the old one for next comparison.
> > > > The reason to use SHA1 hash but not byte-by-byte comparison is the
> > > > memory overheads. One SHA1 hash is 20 bytes. So we need extra
> > > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > > relatively small.
> > > > As far as I know, SHA1 hash is widely used in the scenes of
> > > > deduplication for backup systems.
> > > > They have proven that the probability of hash collision is far
> > > > smaller than disk hardware fault, so it's secure hash, that is, if
> > > > the hashes of two chunks are the same, the content must be the same.
> > > > So I think the SHA1 hash could replace byte-to-byte comparison in
> > > > the VM memory scenery.
> > > >
> > > > Then I do the same migration experiments using the SHA1 hash. For
> > > > the 4 workloads which have big proportions of false dirty pages, the
> > > > improvement is remarkable. Without optimization, they either can not
> > > > converge to stop-and-copy, or take a very long time to complete.
> > > > With the
> > > > SHA1 hash method, all of them now complete in a relatively short time.
> > > > For the reason I have talked above, the other workloads don't get
> > > > notable improvements from the optimization. So below, I only show
> > > > the exact number after optimization for the 4 workloads with
> > > > remarkable improvements.
> > > >
> > > > Any comments or suggestions?
> > >
> > > Maybe you can compare the performance of your solution as that of
> > XBZRLE to see which one is better.
> > > The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, and
> > need less buffer.
> > > How about the overhead of calculating the SHA1? Is it faster than copying a
> > page?
> > >
> > > Liang
> > >
> > >
> >
> > Yes, XBZRLE is able to handle the false dirty pages. However, if we want to
> > avoid transferring all of the false dirty pages using XBZRLE, we need a buffer
> > as big as the whole VM memory, while SHA1 needs a much small buffer. Of
> > course, if we have a buffer as big as the whole VM memory using XBZRLE, we
> > could transfer less data on network than SHA1, because XBZRLE is able to
> > compress similar pages. In a word, yes, the merit of using SHA1 is that it
> > needs much less buffer, and leads to nice improvement if there are many
> > false dirty pages.
> >
>
> The current implementation of XBZRLE begins to buffer page from the second iteration,
> Maybe it's worth to make it start to work from the first iteration based on your finding.
Yes, I noticed that. If we make it start to work from the first iteration, I think the
buffer should be large enough to obtain obvious effect.
>
> > In terms of the overhead of calculating the SHA1 compared with transferring
> > a page, it's related to the CPU and network performance. In my test
> > environment(Intel Xeon
> > E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra computing
> > overhead caused by calculating the SHA1, because the throughput of
> > network (got by "info migrate") remains almost the same.
>
> You can check the CPU usage, or to measure the time spend on a local live migration
> which use SHA1/ XBZRLE.
Yes, I can compare SHA1 with XBZRLE. Maybe I will post the results later.
Chunguang
>
> Liang
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-04 4:50 ` Li, Liang Z
2016-11-04 7:03 ` Chunguang Li
@ 2016-11-07 13:52 ` Chunguang Li
2016-11-07 14:17 ` Li, Liang Z
2016-11-07 14:44 ` Li, Liang Z
1 sibling, 2 replies; 21+ messages in thread
From: Chunguang Li @ 2016-11-07 13:52 UTC (permalink / raw)
To: Li, Liang Z
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> -----Original Messages-----
> From: "Li, Liang Z" <liang.z.li@intel.com>
> Sent Time: Friday, November 4, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Amit Shah" <amit.shah@redhat.com>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "stefanha@redhat.com" <stefanha@redhat.com>, "quintela@redhat.com" <quintela@redhat.com>
> Subject: RE: RE: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> > > > > > > > I think this is "very" wasteful. Assume the workload writes
> > > > > > > > the pages
> > > > dirty randomly within the guest address space, and the transfer
> > > > speed is constant. Intuitively, I think nearly half of the dirty
> > > > pages produced in Iteration 1 is not really dirty. This means the
> > > > time of Iteration 2 is double of that to send only really dirty pages.
> > > > > > >
> > > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > > kinds of workloads get impacted the most? That would also
> > > > > > > help us to figure out what kinds of speed improvements we can
> > expect.
> > > > > > >
> > > > > > >
> > > > > > > Amit
> > > > > >
> > > > > > I have picked up 6 workloads and got the following statistics
> > > > > > numbers of every iteration (except the last stop-copy one) during
> > precopy.
> > > > > > These numbers are obtained with the basic precopy migration,
> > > > > > without the capabilities like xbzrle or compression, etc. The
> > > > > > network for the migration is exclusive, with a separate network for
> > the workloads.
> > > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > > >
> > > > > > Three (booting, idle, web server) of them converged to the
> > > > > > stop-copy
> > > > phase,
> > > > > > with the given bandwidth and default downtime (300ms), while the
> > > > > > other three (kernel compilation, zeusmp, memcached) did not.
> > > > > >
> > > > > > One page is "not-really-dirty", if it is written first and is
> > > > > > sent later (and not written again after that) during one
> > > > > > iteration. I guess this would not happen so often during the
> > > > > > other iterations as during the 1st iteration. Because all the
> > > > > > pages of the VM are sent to the dest node
> > > > during
> > > > > > the 1st iteration, while during the others, only part of the pages are
> > sent.
> > > > > > So I think the "not-really-dirty" pages should be produced
> > > > > > mainly during the 1st iteration , and maybe very little during the other
> > iterations.
> > > > > >
> > > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > > halved. This is a chain
> > > > reaction,
> > > > > > because the dirty pages produced during Iteration 2 is halved,
> > > > > > which
> > > > incurs
> > > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > > >
> > > > > Yes; these numbers don't show how many of them are false dirty
> > though.
> > > > >
> > > > > One problem is thinking about pages that have been redirtied, if
> > > > > the page is
> > > > dirtied
> > > > > after the sync but before the network write then it's the
> > > > > false-dirty that you're describing.
> > > > >
> > > > > However, if the page is being written a few times, and so it would
> > > > > have
> > > > been written
> > > > > after the network write then it isn't a false-dirty.
> > > > >
> > > > > You might be able to figure that out with some kernel tracing of
> > > > > when the
> > > > dirtying
> > > > > happens, but it might be easier to write the fix!
> > > > >
> > > > > Dave
> > > >
> > > > Hi, I have made some new progress now.
> > > >
> > > > To tell how many false dirty pages there are exactly in each
> > > > iteration, I malloc a buffer in memory as big as the size of the
> > > > whole VM memory. When a page is transferred to the dest node, it is
> > > > copied to the buffer; During the next iteration, if one page is
> > > > transferred, it is compared to the old one in the buffer, and the
> > > > old one will be replaced for next comparison if it is really dirty.
> > > > Thus, we are now able to get the exact number of false dirty pages.
> > > >
> > > > This time, I use 15 workloads to get the statistic number. They are:
> > > >
> > > > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are
> > > > all scientific
> > > > computing workloads like Quantum Chromodynamics, Fluid Dynamics,
> > etc.
> > > > I pick
> > > > up these 11 benchmarks because compared to others, they have
> > > > bigger memory
> > > > occupation and higher memory dirty rate. Thus most of them
> > > > could not converge
> > > > to stop-and-copy using the default migration speed (32MB/s).
> > > > 2. kernel compilation
> > > > 3. idle VM
> > > > 4. Apache web server which serves static content
> > > >
> > > > (the above workloads are all running in VM with 1 vcpu and 1GB
> > > > memory, and the
> > > > migration speed is the default 32MB/s)
> > > >
> > > > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are
> > > > used as the cache.
> > > > After filling up the 4GB cache, a client writes the cache at a constant
> > speed
> > > > during migration. This time, migration speed has no limit, and is up to
> > the
> > > > capability of 1Gbps Ethernet.
> > > >
> > > > Summarize the results first: (and you can read the precise number
> > > > below)
> > > >
> > > > 1. 4 of these 15 workloads have a big proportion (>60%, even >80%
> > > > during some iterations)
> > > > of false dirty pages out of all the dirty pages since iteration 2 (and the
> > big
> > > > proportion lasts during the following iterations). They are
> > cpu2006.zeusmp,
> > > > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > > 2. 2 workloads (idle, webserver) spend most of the migration time
> > > > on iteration 1, even
> > > > though the proportion of false dirty pages is big since
> > > > iteration 2, the space to
> > > > optimize is small.
> > > > 3. 1 workload (kernel compilation) only have a big proportion
> > > > during iteration 2, not
> > > > in the other iterations.
> > > > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > > proportion of false
> > > > dirty pages since iteration 2. So the spaces to optimize for them are
> > small.
> > > >
> > > > Now I want to talk a little more about the reasons why false dirty
> > > > pages are produced.
> > > > The first reason is what we have discussed before---the mechanism to
> > > > track the dirty pages.
> > > > And then I come up with another reason. Here is the situation: a
> > > > write operation to one memory page happens, but it doesn't change
> > > > any content of the page. So it's "write but not dirty", and kernel
> > > > still marks it as dirty. One guy in our lab has done some
> > > > experiments to figure out the proportion of "write but not dirty"
> > > > operations, and he uses the cpu2006 benchmark suit. According to his
> > > > results, general workloads has a little proportion (<10%) of "write
> > > > but not dirty" out of all the write operations, while few workloads
> > > > have higher proportion (one even as high as 50%). Now we are not
> > > > sure why "write but not dirty" would happen, it just happened.
> > > >
> > > > So these two reasons contribute to the false dirty pages. To
> > > > optimize, I compute and store the SHA1 hash before transferring each
> > > > page. Next time, if one page needs retransmission, its
> > > > SHA1 hash is computed again, and compared to the old hash. If the
> > > > hash is the same, it's a false dirty page, and we just skip this
> > > > page; Otherwise, the page is transferred, and the new hash replaces
> > > > the old one for next comparison.
> > > > The reason to use SHA1 hash but not byte-by-byte comparison is the
> > > > memory overheads. One SHA1 hash is 20 bytes. So we need extra
> > > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > > relatively small.
> > > > As far as I know, SHA1 hash is widely used in the scenes of
> > > > deduplication for backup systems.
> > > > They have proven that the probability of hash collision is far
> > > > smaller than disk hardware fault, so it's secure hash, that is, if
> > > > the hashes of two chunks are the same, the content must be the same.
> > > > So I think the SHA1 hash could replace byte-to-byte comparison in
> > > > the VM memory scenery.
> > > >
> > > > Then I do the same migration experiments using the SHA1 hash. For
> > > > the 4 workloads which have big proportions of false dirty pages, the
> > > > improvement is remarkable. Without optimization, they either can not
> > > > converge to stop-and-copy, or take a very long time to complete.
> > > > With the
> > > > SHA1 hash method, all of them now complete in a relatively short time.
> > > > For the reason I have talked above, the other workloads don't get
> > > > notable improvements from the optimization. So below, I only show
> > > > the exact number after optimization for the 4 workloads with
> > > > remarkable improvements.
> > > >
> > > > Any comments or suggestions?
> > >
> > > Maybe you can compare the performance of your solution as that of
> > XBZRLE to see which one is better.
> > > The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, and
> > need less buffer.
> > > How about the overhead of calculating the SHA1? Is it faster than copying a
> > page?
> > >
> > > Liang
> > >
> > >
> >
> > Yes, XBZRLE is able to handle the false dirty pages. However, if we want to
> > avoid transferring all of the false dirty pages using XBZRLE, we need a buffer
> > as big as the whole VM memory, while SHA1 needs a much small buffer. Of
> > course, if we have a buffer as big as the whole VM memory using XBZRLE, we
> > could transfer less data on network than SHA1, because XBZRLE is able to
> > compress similar pages. In a word, yes, the merit of using SHA1 is that it
> > needs much less buffer, and leads to nice improvement if there are many
> > false dirty pages.
> >
>
> The current implementation of XBZRLE begins to buffer page from the second iteration,
> Maybe it's worth to make it start to work from the first iteration based on your finding.
>
> > In terms of the overhead of calculating the SHA1 compared with transferring
> > a page, it's related to the CPU and network performance. In my test
> > environment(Intel Xeon
> > E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra computing
> > overhead caused by calculating the SHA1, because the throughput of
> > network (got by "info migrate") remains almost the same.
>
> You can check the CPU usage, or to measure the time spend on a local live migration
> which use SHA1/ XBZRLE.
>
> Liang
>
>
I compare SHA1 with XBZRLE. I use XBZRLE in two ways:
1. Begins to buffer pages from iteration 1;
2. As current implementation, begins to buffer pages from iteration 2.
I post the results of three workloads: cpu2006.zeusmp, cpu2006.mcf, memcached.
I set the cache size as 256MB for zeusmp & mcf (they run in VM with 1GB ram),
and set the cache size as 1GB for memcached (it run in VM with 6GB ram, and
memcached takes 4GB as cache).
As you can read from the data below, beginning to buffer pages from iteration 1
is better than the current implementation(from iteration 2), because the total
migration time is shorter.
SHA1 is better than the XBZRLE with the cache size I choose, because it leads to shorter
migration time, and consumes far less memory overhead (<1/200 of the total VM memory).
1. zeusmp
(1) XBZRLE 256MB cache, begins to buffer pages from iteration 1
Iteration 1, duration: 21402 ms , transferred pages: 266450 (dup: 91456, n: 174994, x: 0) , new dirty pages: 129225 , remaining dirty pages: 129225
Iteration 2, duration: 2295 ms , transferred pages: 101471 (dup: 77921, n: 16665, x: 6885) , new dirty pages: 76125 , remaining dirty pages: 77424
Iteration 3, duration: 2000 ms , transferred pages: 56092 (dup: 36345, n: 11249, x: 8498) , new dirty pages: 111498 , remaining dirty pages: 112836
Iteration 4, duration: 1604 ms , transferred pages: 87335 (dup: 69441, n: 10018, x: 7876) , new dirty pages: 19982 , remaining dirty pages: 19982
Iteration 5, duration: 302 ms , transferred pages: 19850 (dup: 16718, n: 2547, x: 585) , new dirty pages: 14084 , remaining dirty pages: 14084
Iteration 6, duration: 194 ms , transferred pages: 13403 (dup: 12338, n: 846, x: 219) , new dirty pages: 3900 , remaining dirty pages: 4243
Iteration 7, duration: 8 ms , transferred pages: 3938 (dup: 3425, n: 239, x: 274) , new dirty pages: 372 , remaining dirty pages: 372
Iteration 8, duration: 71 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 0 , remaining dirty pages: 372
total time: 27891 milliseconds
(2) XBZRLE 256MB cache, begins to buffer pages from iteration 2
can not converge
Iteration 1, duration: 21698 ms , transferred pages: 266331 (dup: 89009, n: 177322, x: 0) , new dirty pages: 125990 , remaining dirty pages: 126109
Iteration 2, duration: 5909 ms , transferred pages: 126109 (dup: 77248, n: 48861, x: 0) , new dirty pages: 124870 , remaining dirty pages: 124870
Iteration 3, duration: 3197 ms , transferred pages: 110583 (dup: 75471, n: 23129, x: 11983) , new dirty pages: 118035 , remaining dirty pages: 118035
Iteration 4, duration: 3195 ms , transferred pages: 102787 (dup: 72708, n: 22158, x: 7921) , new dirty pages: 86576 , remaining dirty pages: 86773
Iteration 5, duration: 3111 ms , transferred pages: 79563 (dup: 52073, n: 21289, x: 6201) , new dirty pages: 97402 , remaining dirty pages: 97402
Iteration 6, duration: 2407 ms , transferred pages: 79567 (dup: 56415, n: 16013, x: 7139) , new dirty pages: 101193 , remaining dirty pages: 101193
Iteration 7, duration: 2896 ms , transferred pages: 83278 (dup: 55778, n: 20652, x: 6848) , new dirty pages: 90683 , remaining dirty pages: 92977
Iteration 8, duration: 2701 ms , transferred pages: 89112 (dup: 62579, n: 18699, x: 7834) , new dirty pages: 109827 , remaining dirty pages: 110008
Iteration 9, duration: 3602 ms , transferred pages: 95866 (dup: 61631, n: 25632, x: 8603) , new dirty pages: 94551 , remaining dirty pages: 96227
Iteration 10, duration: 3802 ms , transferred pages: 83693 (dup: 50558, n: 26427, x: 6708) , new dirty pages: 123537 , remaining dirty pages: 124170
Iteration 11, duration: 3399 ms , transferred pages: 108770 (dup: 75144, n: 23952, x: 9674) , new dirty pages: 103934 , remaining dirty pages: 104981
Iteration 12, duration: 2700 ms , transferred pages: 91080 (dup: 62981, n: 16600, x: 11499) , new dirty pages: 88314 , remaining dirty pages: 88948
Iteration 13, duration: 3102 ms , transferred pages: 78406 (dup: 50165, n: 21409, x: 6832) , new dirty pages: 73586 , remaining dirty pages: 74025
Iteration 14, duration: 806 ms , transferred pages: 66530 (dup: 51013, n: 3973, x: 11544) , new dirty pages: 67941 , remaining dirty pages: 67941
Iteration 15, duration: 2398 ms , transferred pages: 53117 (dup: 33312, n: 18436, x: 1369) , new dirty pages: 116502 , remaining dirty pages: 118956
Iteration 16, duration: 3200 ms , transferred pages: 103009 (dup: 71642, n: 21378, x: 9989) , new dirty pages: 81777 , remaining dirty pages: 83724
Iteration 17, duration: 3005 ms , transferred pages: 73096 (dup: 45738, n: 19016, x: 8342) , new dirty pages: 116671 , remaining dirty pages: 118397
Iteration 18, duration: 3302 ms , transferred pages: 101507 (dup: 67290, n: 22721, x: 11496) , new dirty pages: 104163 , remaining dirty pages: 105921
Iteration 19, duration: 3705 ms , transferred pages: 90516 (dup: 56932, n: 26394, x: 7190) , new dirty pages: 118139 , remaining dirty pages: 120170
Iteration 20, duration: 3903 ms , transferred pages: 102710 (dup: 67623, n: 25811, x: 9276) , new dirty pages: 103608 , remaining dirty pages: 105496
(3) SHA1
Iteration 1, duration: 21601 ms , transferred pages: 266450 (dup: 89731, rd: 176719) , new dirty pages: 139843 , remaining dirty pages: 139843
Iteration 2, duration: 1747 ms , transferred pages: 92077 (dup: 78364, rd: 13713) , new dirty pages: 90945 , remaining dirty pages: 90945
Iteration 3, duration: 1592 ms , transferred pages: 62253 (dup: 49435, rd: 12818) , new dirty pages: 76929 , remaining dirty pages: 76929
Iteration 4, duration: 992 ms , transferred pages: 44837 (dup: 37886, rd: 6951) , new dirty pages: 71331 , remaining dirty pages: 72916
Iteration 5, duration: 998 ms , transferred pages: 55229 (dup: 47150, rd: 8079) , new dirty pages: 21703 , remaining dirty pages: 23302
Iteration 6, duration: 211 ms , transferred pages: 20337 (dup: 18516, rd: 1821) , new dirty pages: 14500 , remaining dirty pages: 14500
Iteration 7, duration: 31 ms , transferred pages: 12933 (dup: 12627, rd: 306) , new dirty pages: 1520 , remaining dirty pages: 1520
Iteration 8, duration: 30 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 4 , remaining dirty pages: 1524
total time: 27225 milliseconds
2. mcf
(1) XBZRLE 256MB cache, begins to buffer pages from iteration 1
Iteration 1, duration: 31706 ms , transferred pages: 266325 (dup: 7032, n: 259293, x: 0) , new dirty pages: 238215 , remaining dirty pages: 238340
Iteration 2, duration: 21807 ms , transferred pages: 186619 (dup: 335, n: 176826, x: 9458) , new dirty pages: 226886 , remaining dirty pages: 228857
Iteration 3, duration: 21300 ms , transferred pages: 181925 (dup: 201, n: 172974, x: 8750) , new dirty pages: 202288 , remaining dirty pages: 204100
Iteration 4, duration: 17300 ms , transferred pages: 148972 (dup: 38, n: 141113, x: 7821) , new dirty pages: 136220 , remaining dirty pages: 137992
Iteration 5, duration: 13699 ms , transferred pages: 118247 (dup: 38, n: 112030, x: 6179) , new dirty pages: 48397 , remaining dirty pages: 50466
Iteration 6, duration: 4499 ms , transferred pages: 41719 (dup: 24, n: 36790, x: 4905) , new dirty pages: 3753 , remaining dirty pages: 5690
Iteration 7, duration: 399 ms , transferred pages: 3826 (dup: 4, n: 3265, x: 557) , new dirty pages: 1261 , remaining dirty pages: 2437
Iteration 8, duration: 72 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 0 , remaining dirty pages: 2437
total time: 110812 milliseconds
(2) XBZRLE 256MB cache, begins to buffer pages from iteration 2
Iteration 1, duration: 31606 ms , transferred pages: 266450 (dup: 7267, n: 259183, x: 0) , new dirty pages: 233582 , remaining dirty pages: 233582
Iteration 2, duration: 28413 ms , transferred pages: 231693 (dup: 89, n: 231604, x: 0) , new dirty pages: 216962 , remaining dirty pages: 218851
Iteration 3, duration: 18618 ms , transferred pages: 159936 (dup: 3, n: 151579, x: 8354) , new dirty pages: 216400 , remaining dirty pages: 218790
Iteration 4, duration: 18621 ms , transferred pages: 159665 (dup: 0, n: 152102, x: 7563) , new dirty pages: 209860 , remaining dirty pages: 211611
Iteration 5, duration: 17709 ms , transferred pages: 151672 (dup: 4, n: 144493, x: 7175) , new dirty pages: 146273 , remaining dirty pages: 148006
Iteration 6, duration: 9911 ms , transferred pages: 86971 (dup: 2, n: 80842, x: 6127) , new dirty pages: 118364 , remaining dirty pages: 120396
Iteration 7, duration: 14212 ms , transferred pages: 117460 (dup: 0, n: 116149, x: 1311) , new dirty pages: 213993 , remaining dirty pages: 216107
Iteration 8, duration: 22913 ms , transferred pages: 213698 (dup: 4, n: 161520, x: 52174) , new dirty pages: 217947 , remaining dirty pages: 219955
Iteration 9, duration: 23808 ms , transferred pages: 217375 (dup: 3, n: 152315, x: 65057) , new dirty pages: 172615 , remaining dirty pages: 174859
Iteration 10, duration: 15099 ms , transferred pages: 131265 (dup: 0, n: 123463, x: 7802) , new dirty pages: 113946 , remaining dirty pages: 116026
Iteration 11, duration: 10002 ms , transferred pages: 88477 (dup: 8, n: 81753, x: 6716) , new dirty pages: 97006 , remaining dirty pages: 99110
Iteration 12, duration: 6898 ms , transferred pages: 62861 (dup: 4, n: 56392, x: 6465) , new dirty pages: 45164 , remaining dirty pages: 47297
Iteration 13, duration: 3601 ms , transferred pages: 35360 (dup: 0, n: 29390, x: 5970) , new dirty pages: 24581 , remaining dirty pages: 26779
Iteration 14, duration: 1902 ms , transferred pages: 19794 (dup: 0, n: 15475, x: 4319) , new dirty pages: 66153 , remaining dirty pages: 67850
Iteration 15, duration: 5504 ms , transferred pages: 50369 (dup: 0, n: 44902, x: 5467) , new dirty pages: 49279 , remaining dirty pages: 51198
Iteration 16, duration: 3699 ms , transferred pages: 36519 (dup: 2, n: 30184, x: 6333) , new dirty pages: 23672 , remaining dirty pages: 25914
Iteration 17, duration: 1601 ms , transferred pages: 17628 (dup: 0, n: 12972, x: 4656) , new dirty pages: 8685 , remaining dirty pages: 10646
Iteration 18, duration: 599 ms , transferred pages: 7835 (dup: 0, n: 4825, x: 3010) , new dirty pages: 6167 , remaining dirty pages: 6266
Iteration 19, duration: 200 ms , transferred pages: 3590 (dup: 0, n: 1576, x: 2014) , new dirty pages: 3873 , remaining dirty pages: 5709
Iteration 20, duration: 200 ms , transferred pages: 4134 (dup: 0, n: 1574, x: 2560) , new dirty pages: 3609 , remaining dirty pages: 4099
Iteration 21, duration: 100 ms , transferred pages: 2785 (dup: 0, n: 787, x: 1998) , new dirty pages: 1900 , remaining dirty pages: 2585
Iteration 22, duration: 9 ms , transferred pages: 2191 (dup: 0, n: 539, x: 1652) , new dirty pages: 596 , remaining dirty pages: 596
Iteration 23, duration: 41 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 0 , remaining dirty pages: 596
total time: 235286 milliseconds
(3) SHA1
Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6831, rd: 259619) , new dirty pages: 240209 , remaining dirty pages: 240209
Iteration 2, duration: 6250 ms , transferred pages: 51244 (dup: 211, rd: 51033) , new dirty pages: 226651 , remaining dirty pages: 228571
Iteration 3, duration: 4395 ms , transferred pages: 36008 (dup: 80, rd: 35928) , new dirty pages: 110719 , remaining dirty pages: 111478
Iteration 4, duration: 3390 ms , transferred pages: 28068 (dup: 28, rd: 28040) , new dirty pages: 185172 , remaining dirty pages: 185172
Iteration 5, duration: 2986 ms , transferred pages: 23780 (dup: 45, rd: 23735) , new dirty pages: 64357 , remaining dirty pages: 66305
Iteration 6, duration: 2727 ms , transferred pages: 22800 (dup: 12, rd: 22788) , new dirty pages: 61675 , remaining dirty pages: 61675
Iteration 7, duration: 2372 ms , transferred pages: 18943 (dup: 13, rd: 18930) , new dirty pages: 55144 , remaining dirty pages: 55265
Iteration 8, duration: 2100 ms , transferred pages: 17189 (dup: 11, rd: 17178) , new dirty pages: 55244 , remaining dirty pages: 55668
Iteration 9, duration: 2003 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 107058 , remaining dirty pages: 108014
Iteration 10, duration: 2132 ms , transferred pages: 17825 (dup: 24, rd: 17801) , new dirty pages: 126214 , remaining dirty pages: 126214
Iteration 11, duration: 2229 ms , transferred pages: 18156 (dup: 22, rd: 18134) , new dirty pages: 65725 , remaining dirty pages: 65725
Iteration 12, duration: 2315 ms , transferred pages: 18651 (dup: 21, rd: 18630) , new dirty pages: 52575 , remaining dirty pages: 53903
Iteration 13, duration: 2147 ms , transferred pages: 17435 (dup: 16, rd: 17419) , new dirty pages: 46652 , remaining dirty pages: 47260
Iteration 14, duration: 2000 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 42721 , remaining dirty pages: 43266
Iteration 15, duration: 1901 ms , transferred pages: 15552 (dup: 10, rd: 15542) , new dirty pages: 38593 , remaining dirty pages: 40792
Iteration 16, duration: 1801 ms , transferred pages: 14735 (dup: 11, rd: 14724) , new dirty pages: 54252 , remaining dirty pages: 55639
Iteration 17, duration: 1708 ms , transferred pages: 13860 (dup: 2, rd: 13858) , new dirty pages: 72379 , remaining dirty pages: 74170
Iteration 18, duration: 1923 ms , transferred pages: 15442 (dup: 12, rd: 15430) , new dirty pages: 101911 , remaining dirty pages: 103547
Iteration 19, duration: 2311 ms , transferred pages: 18823 (dup: 9, rd: 18814) , new dirty pages: 80534 , remaining dirty pages: 82521
Iteration 20, duration: 2081 ms , transferred pages: 17156 (dup: 34, rd: 17122) , new dirty pages: 36054 , remaining dirty pages: 36054
Iteration 21, duration: 1665 ms , transferred pages: 13777 (dup: 10, rd: 13767) , new dirty pages: 29624 , remaining dirty pages: 29624
Iteration 22, duration: 1657 ms , transferred pages: 13290 (dup: 7, rd: 13283) , new dirty pages: 25949 , remaining dirty pages: 28265
Iteration 23, duration: 1599 ms , transferred pages: 13088 (dup: 0, rd: 13088) , new dirty pages: 22356 , remaining dirty pages: 24813
Iteration 24, duration: 1500 ms , transferred pages: 12280 (dup: 10, rd: 12270) , new dirty pages: 21181 , remaining dirty pages: 22608
Iteration 25, duration: 1400 ms , transferred pages: 11457 (dup: 5, rd: 11452) , new dirty pages: 18657 , remaining dirty pages: 20311
Iteration 26, duration: 1200 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 15690 , remaining dirty pages: 17294
Iteration 27, duration: 1201 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 14810 , remaining dirty pages: 15936
Iteration 28, duration: 1000 ms , transferred pages: 8183 (dup: 3, rd: 8180) , new dirty pages: 15387 , remaining dirty pages: 16423
Iteration 29, duration: 900 ms , transferred pages: 7372 (dup: 10, rd: 7362) , new dirty pages: 13303 , remaining dirty pages: 15292
Iteration 30, duration: 1000 ms , transferred pages: 8181 (dup: 1, rd: 8180) , new dirty pages: 17879 , remaining dirty pages: 18457
Iteration 31, duration: 951 ms , transferred pages: 8140 (dup: 9, rd: 8131) , new dirty pages: 21738 , remaining dirty pages: 23304
Iteration 32, duration: 946 ms , transferred pages: 6946 (dup: 1, rd: 6945) , new dirty pages: 15815 , remaining dirty pages: 15815
Iteration 33, duration: 747 ms , transferred pages: 6192 (dup: 0, rd: 6192) , new dirty pages: 6249 , remaining dirty pages: 7670
Iteration 34, duration: 501 ms , transferred pages: 4090 (dup: 0, rd: 4090) , new dirty pages: 6163 , remaining dirty pages: 8422
Iteration 35, duration: 600 ms , transferred pages: 4910 (dup: 2, rd: 4908) , new dirty pages: 3673 , remaining dirty pages: 5222
Iteration 36, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2132 , remaining dirty pages: 4337
Iteration 37, duration: 200 ms , transferred pages: 1637 (dup: 1, rd: 1636) , new dirty pages: 544 , remaining dirty pages: 2251
Iteration 38, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 2251
total time: 97919 milliseconds
3. memcached
(1) XBZRLE 1024MB cache, begins to buffer pages from iteration 1
Iteration 1, duration: 40763 ms , transferred pages: 1570149 (dup: 404139, n: 1166010, x: 0) , new dirty pages: 526462 , remaining dirty pages: 533483
Iteration 2, duration: 15741 ms , transferred pages: 461867 (dup: 4070, n: 437704, x: 20093) , new dirty pages: 256841 , remaining dirty pages: 265501
Iteration 3, duration: 7874 ms , transferred pages: 231950 (dup: 280, n: 207569, x: 24101) , new dirty pages: 153526 , remaining dirty pages: 160865
Iteration 4, duration: 4260 ms , transferred pages: 135181 (dup: 135, n: 116768, x: 18278) , new dirty pages: 100298 , remaining dirty pages: 107278
Iteration 5, duration: 2506 ms , transferred pages: 87596 (dup: 180, n: 67600, x: 19816) , new dirty pages: 63685 , remaining dirty pages: 71790
Iteration 6, duration: 1373 ms , transferred pages: 51800 (dup: 128, n: 37336, x: 14336) , new dirty pages: 38785 , remaining dirty pages: 46064
Iteration 7, duration: 872 ms , transferred pages: 32015 (dup: 56, n: 23414, x: 8545) , new dirty pages: 23580 , remaining dirty pages: 31629
Iteration 8, duration: 527 ms , transferred pages: 21833 (dup: 40, n: 14372, x: 7421) , new dirty pages: 16624 , remaining dirty pages: 23482
Iteration 9, duration: 291 ms , transferred pages: 14917 (dup: 16, n: 6572, x: 8329) , new dirty pages: 10039 , remaining dirty pages: 16753
Iteration 10, duration: 113 ms , transferred pages: 6082 (dup: 111, n: 3300, x: 2671) , new dirty pages: 4081 , remaining dirty pages: 12703
Iteration 11, duration: 119 ms , transferred pages: 3970 (dup: 16, n: 2953, x: 1001) , new dirty pages: 3824 , remaining dirty pages: 11936
Iteration 12, duration: 51 ms , transferred pages: 3585 (dup: 0, n: 1154, x: 2431) , new dirty pages: 1711 , remaining dirty pages: 9900
Iteration 13, duration: 62 ms , transferred pages: 2945 (dup: 0, n: 1589, x: 1356) , new dirty pages: 1909 , remaining dirty pages: 8503
Iteration 14, duration: 2 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 1 , remaining dirty pages: 8504
total time: 74738 milliseconds
(2) XBZRLE 1024MB cache, begins to buffer pages from iteration 2
Iteration 1, duration: 40375 ms , transferred pages: 1570347 (dup: 415923, n: 1154424, x: 0) , new dirty pages: 511859 , remaining dirty pages: 518682
Iteration 2, duration: 17580 ms , transferred pages: 510145 (dup: 5970, n: 504175, x: 0) , new dirty pages: 291686 , remaining dirty pages: 300223
Iteration 3, duration: 8259 ms , transferred pages: 253656 (dup: 929, n: 230020, x: 22707) , new dirty pages: 166721 , remaining dirty pages: 174231
Iteration 4, duration: 4733 ms , transferred pages: 147925 (dup: 257, n: 132454, x: 15214) , new dirty pages: 103965 , remaining dirty pages: 111436
Iteration 5, duration: 2587 ms , transferred pages: 90734 (dup: 251, n: 70008, x: 20475) , new dirty pages: 61266 , remaining dirty pages: 69202
Iteration 6, duration: 1377 ms , transferred pages: 51416 (dup: 55, n: 37776, x: 13585) , new dirty pages: 45236 , remaining dirty pages: 52106
Iteration 7, duration: 1126 ms , transferred pages: 40020 (dup: 259, n: 30064, x: 9697) , new dirty pages: 28433 , remaining dirty pages: 35358
Iteration 8, duration: 574 ms , transferred pages: 23754 (dup: 40, n: 16066, x: 7648) , new dirty pages: 18067 , remaining dirty pages: 26353
Iteration 9, duration: 395 ms , transferred pages: 17607 (dup: 16, n: 9463, x: 8128) , new dirty pages: 11507 , remaining dirty pages: 18488
Iteration 10, duration: 171 ms , transferred pages: 8195 (dup: 40, n: 4726, x: 3429) , new dirty pages: 5482 , remaining dirty pages: 13898
Iteration 11, duration: 116 ms , transferred pages: 6594 (dup: 16, n: 2679, x: 3899) , new dirty pages: 3884 , remaining dirty pages: 9581
Iteration 12, duration: 54 ms , transferred pages: 1793 (dup: 0, n: 1634, x: 159) , new dirty pages: 1515 , remaining dirty pages: 9189
Iteration 13, duration: 62 ms , transferred pages: 1793 (dup: 0, n: 1643, x: 150) , new dirty pages: 1657 , remaining dirty pages: 8871
Iteration 14, duration: 3 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 1 , remaining dirty pages: 8872
total time: 77578 milliseconds
(3) SHA1
Iteration 1, duration: 40664 ms , transferred pages: 1569037 (dup: 405940, rd: 1163097) , new dirty pages: 506846 , remaining dirty pages: 514979
Iteration 2, duration: 8032 ms , transferred pages: 161130 (dup: 4007, rd: 157123) , new dirty pages: 153479 , remaining dirty pages: 153479
Iteration 3, duration: 2620 ms , transferred pages: 65260 (dup: 20, rd: 65240) , new dirty pages: 64014 , remaining dirty pages: 67100
Iteration 4, duration: 1160 ms , transferred pages: 30227 (dup: 60, rd: 30167) , new dirty pages: 34031 , remaining dirty pages: 41414
Iteration 5, duration: 648 ms , transferred pages: 18700 (dup: 56, rd: 18644) , new dirty pages: 18375 , remaining dirty pages: 25536
Iteration 6, duration: 389 ms , transferred pages: 11399 (dup: 55, rd: 11344) , new dirty pages: 12536 , remaining dirty pages: 17516
Iteration 7, duration: 292 ms , transferred pages: 8197 (dup: 0, rd: 8197) , new dirty pages: 8387 , remaining dirty pages: 16802
Iteration 8, duration: 171 ms , transferred pages: 4931 (dup: 39, rd: 4892) , new dirty pages: 6182 , remaining dirty pages: 14060
Iteration 9, duration: 163 ms , transferred pages: 4355 (dup: 16, rd: 4339) , new dirty pages: 5530 , remaining dirty pages: 11973
Iteration 10, duration: 104 ms , transferred pages: 3266 (dup: 0, rd: 3266) , new dirty pages: 2893 , remaining dirty pages: 11014
Iteration 11, duration: 52 ms , transferred pages: 1153 (dup: 0, rd: 1153) , new dirty pages: 1586 , remaining dirty pages: 10516
Iteration 12, duration: 52 ms , transferred pages: 1921 (dup: 39, rd: 1882) , new dirty pages: 1619 , remaining dirty pages: 8842
Iteration 13, duration: 62 ms , transferred pages: 1537 (dup: 0, rd: 1537) , new dirty pages: 2052 , remaining dirty pages: 8871
Iteration 14, duration: 58 ms , transferred pages: 1665 (dup: 0, rd: 1665) , new dirty pages: 1947 , remaining dirty pages: 7989
Iteration 15, duration: 2 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 7989
total time: 54693 milliseconds
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-07 13:52 ` Chunguang Li
@ 2016-11-07 14:17 ` Li, Liang Z
2016-11-08 5:27 ` Chunguang Li
2016-11-07 14:44 ` Li, Liang Z
1 sibling, 1 reply; 21+ messages in thread
From: Li, Liang Z @ 2016-11-07 14:17 UTC (permalink / raw)
To: Chunguang Li
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> > > > > > > > > I think this is "very" wasteful. Assume the workload
> > > > > > > > > writes the pages
> > > > > dirty randomly within the guest address space, and the transfer
> > > > > speed is constant. Intuitively, I think nearly half of the dirty
> > > > > pages produced in Iteration 1 is not really dirty. This means
> > > > > the time of Iteration 2 is double of that to send only really dirty pages.
> > > > > > > >
> > > > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > > > kinds of workloads get impacted the most? That would also
> > > > > > > > help us to figure out what kinds of speed improvements we
> > > > > > > > can
> > > expect.
> > > > > > > >
> > > > > > > >
> > > > > > > > Amit
> > > > > > >
> > > > > > > I have picked up 6 workloads and got the following
> > > > > > > statistics numbers of every iteration (except the last
> > > > > > > stop-copy one) during
> > > precopy.
> > > > > > > These numbers are obtained with the basic precopy migration,
> > > > > > > without the capabilities like xbzrle or compression, etc.
> > > > > > > The network for the migration is exclusive, with a separate
> > > > > > > network for
> > > the workloads.
> > > > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > > > >
> > > > > > > Three (booting, idle, web server) of them converged to the
> > > > > > > stop-copy
> > > > > phase,
> > > > > > > with the given bandwidth and default downtime (300ms), while
> > > > > > > the other three (kernel compilation, zeusmp, memcached) did not.
> > > > > > >
> > > > > > > One page is "not-really-dirty", if it is written first and
> > > > > > > is sent later (and not written again after that) during one
> > > > > > > iteration. I guess this would not happen so often during the
> > > > > > > other iterations as during the 1st iteration. Because all
> > > > > > > the pages of the VM are sent to the dest node
> > > > > during
> > > > > > > the 1st iteration, while during the others, only part of the
> > > > > > > pages are
> > > sent.
> > > > > > > So I think the "not-really-dirty" pages should be produced
> > > > > > > mainly during the 1st iteration , and maybe very little
> > > > > > > during the other
> > > iterations.
> > > > > > >
> > > > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > > > halved. This is a chain
> > > > > reaction,
> > > > > > > because the dirty pages produced during Iteration 2 is
> > > > > > > halved, which
> > > > > incurs
> > > > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > > > >
> > > > > > Yes; these numbers don't show how many of them are false dirty
> > > though.
> > > > > >
> > > > > > One problem is thinking about pages that have been redirtied,
> > > > > > if the page is
> > > > > dirtied
> > > > > > after the sync but before the network write then it's the
> > > > > > false-dirty that you're describing.
> > > > > >
> > > > > > However, if the page is being written a few times, and so it
> > > > > > would have
> > > > > been written
> > > > > > after the network write then it isn't a false-dirty.
> > > > > >
> > > > > > You might be able to figure that out with some kernel tracing
> > > > > > of when the
> > > > > dirtying
> > > > > > happens, but it might be easier to write the fix!
> > > > > >
> > > > > > Dave
> > > > >
> > > > > Hi, I have made some new progress now.
> > > > >
> > > > > To tell how many false dirty pages there are exactly in each
> > > > > iteration, I malloc a buffer in memory as big as the size of the
> > > > > whole VM memory. When a page is transferred to the dest node, it
> > > > > is copied to the buffer; During the next iteration, if one page
> > > > > is transferred, it is compared to the old one in the buffer, and
> > > > > the old one will be replaced for next comparison if it is really dirty.
> > > > > Thus, we are now able to get the exact number of false dirty pages.
> > > > >
> > > > > This time, I use 15 workloads to get the statistic number. They are:
> > > > >
> > > > > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They
> > > > > are all scientific
> > > > > computing workloads like Quantum Chromodynamics, Fluid
> > > > > Dynamics,
> > > etc.
> > > > > I pick
> > > > > up these 11 benchmarks because compared to others, they
> > > > > have bigger memory
> > > > > occupation and higher memory dirty rate. Thus most of them
> > > > > could not converge
> > > > > to stop-and-copy using the default migration speed (32MB/s).
> > > > > 2. kernel compilation
> > > > > 3. idle VM
> > > > > 4. Apache web server which serves static content
> > > > >
> > > > > (the above workloads are all running in VM with 1 vcpu and 1GB
> > > > > memory, and the
> > > > > migration speed is the default 32MB/s)
> > > > >
> > > > > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB
> > > > > are used as the cache.
> > > > > After filling up the 4GB cache, a client writes the cache
> > > > > at a constant
> > > speed
> > > > > during migration. This time, migration speed has no limit,
> > > > > and is up to
> > > the
> > > > > capability of 1Gbps Ethernet.
> > > > >
> > > > > Summarize the results first: (and you can read the precise
> > > > > number
> > > > > below)
> > > > >
> > > > > 1. 4 of these 15 workloads have a big proportion (>60%, even
> > > > > >80% during some iterations)
> > > > > of false dirty pages out of all the dirty pages since
> > > > > iteration 2 (and the
> > > big
> > > > > proportion lasts during the following iterations). They are
> > > cpu2006.zeusmp,
> > > > > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > > > 2. 2 workloads (idle, webserver) spend most of the migration
> > > > > time on iteration 1, even
> > > > > though the proportion of false dirty pages is big since
> > > > > iteration 2, the space to
> > > > > optimize is small.
> > > > > 3. 1 workload (kernel compilation) only have a big proportion
> > > > > during iteration 2, not
> > > > > in the other iterations.
> > > > > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > > > proportion of false
> > > > > dirty pages since iteration 2. So the spaces to optimize
> > > > > for them are
> > > small.
> > > > >
> > > > > Now I want to talk a little more about the reasons why false
> > > > > dirty pages are produced.
> > > > > The first reason is what we have discussed before---the
> > > > > mechanism to track the dirty pages.
> > > > > And then I come up with another reason. Here is the situation: a
> > > > > write operation to one memory page happens, but it doesn't
> > > > > change any content of the page. So it's "write but not dirty",
> > > > > and kernel still marks it as dirty. One guy in our lab has done
> > > > > some experiments to figure out the proportion of "write but not
> dirty"
> > > > > operations, and he uses the cpu2006 benchmark suit. According to
> > > > > his results, general workloads has a little proportion (<10%) of
> > > > > "write but not dirty" out of all the write operations, while few
> > > > > workloads have higher proportion (one even as high as 50%). Now
> > > > > we are not sure why "write but not dirty" would happen, it just
> happened.
> > > > >
> > > > > So these two reasons contribute to the false dirty pages. To
> > > > > optimize, I compute and store the SHA1 hash before transferring
> > > > > each page. Next time, if one page needs retransmission, its
> > > > > SHA1 hash is computed again, and compared to the old hash. If
> > > > > the hash is the same, it's a false dirty page, and we just skip
> > > > > this page; Otherwise, the page is transferred, and the new hash
> > > > > replaces the old one for next comparison.
> > > > > The reason to use SHA1 hash but not byte-by-byte comparison is
> > > > > the memory overheads. One SHA1 hash is 20 bytes. So we need
> > > > > extra
> > > > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > > > relatively small.
> > > > > As far as I know, SHA1 hash is widely used in the scenes of
> > > > > deduplication for backup systems.
> > > > > They have proven that the probability of hash collision is far
> > > > > smaller than disk hardware fault, so it's secure hash, that is,
> > > > > if the hashes of two chunks are the same, the content must be the
> same.
> > > > > So I think the SHA1 hash could replace byte-to-byte comparison
> > > > > in the VM memory scenery.
> > > > >
> > > > > Then I do the same migration experiments using the SHA1 hash.
> > > > > For the 4 workloads which have big proportions of false dirty
> > > > > pages, the improvement is remarkable. Without optimization, they
> > > > > either can not converge to stop-and-copy, or take a very long time to
> complete.
> > > > > With the
> > > > > SHA1 hash method, all of them now complete in a relatively short
> time.
> > > > > For the reason I have talked above, the other workloads don't
> > > > > get notable improvements from the optimization. So below, I only
> > > > > show the exact number after optimization for the 4 workloads
> > > > > with remarkable improvements.
> > > > >
> > > > > Any comments or suggestions?
> > > >
> > > > Maybe you can compare the performance of your solution as that of
> > > XBZRLE to see which one is better.
> > > > The merit of using SHA1 is that it can avoid data copy as that in
> > > > XBZRLE, and
> > > need less buffer.
> > > > How about the overhead of calculating the SHA1? Is it faster than
> > > > copying a
> > > page?
> > > >
> > > > Liang
> > > >
> > > >
> > >
> > > Yes, XBZRLE is able to handle the false dirty pages. However, if we
> > > want to avoid transferring all of the false dirty pages using
> > > XBZRLE, we need a buffer as big as the whole VM memory, while SHA1
> > > needs a much small buffer. Of course, if we have a buffer as big as
> > > the whole VM memory using XBZRLE, we could transfer less data on
> > > network than SHA1, because XBZRLE is able to compress similar pages.
> > > In a word, yes, the merit of using SHA1 is that it needs much less
> > > buffer, and leads to nice improvement if there are many false dirty pages.
> > >
> >
> > The current implementation of XBZRLE begins to buffer page from the
> > second iteration, Maybe it's worth to make it start to work from the first
> iteration based on your finding.
> >
> > > In terms of the overhead of calculating the SHA1 compared with
> > > transferring a page, it's related to the CPU and network
> > > performance. In my test environment(Intel Xeon
> > > E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra
> > > computing overhead caused by calculating the SHA1, because the
> > > throughput of network (got by "info migrate") remains almost the same.
> >
> > You can check the CPU usage, or to measure the time spend on a local
> > live migration which use SHA1/ XBZRLE.
> >
> > Liang
> >
> >
>
> I compare SHA1 with XBZRLE. I use XBZRLE in two ways:
> 1. Begins to buffer pages from iteration 1; 2. As current implementation,
> begins to buffer pages from iteration 2.
>
> I post the results of three workloads: cpu2006.zeusmp, cpu2006.mcf,
> memcached.
> I set the cache size as 256MB for zeusmp & mcf (they run in VM with 1GB
> ram), and set the cache size as 1GB for memcached (it run in VM with 6GB
> ram, and memcached takes 4GB as cache).
>
> As you can read from the data below, beginning to buffer pages from
> iteration 1 is better than the current implementation(from iteration 2),
> because the total migration time is shorter.
>
> SHA1 is better than the XBZRLE with the cache size I choose, because it leads
> to shorter migration time, and consumes far less memory overhead (<1/200
> of the total VM memory).
>
Hi Chunguang,
Have you tried to use a large XBZRLE cache size which equals to the guest's RAM size?
Is SHA1 faster in that case?
Thanks!
Liang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-07 13:52 ` Chunguang Li
2016-11-07 14:17 ` Li, Liang Z
@ 2016-11-07 14:44 ` Li, Liang Z
1 sibling, 0 replies; 21+ messages in thread
From: Li, Liang Z @ 2016-11-07 14:44 UTC (permalink / raw)
To: Chunguang Li
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> > I compare SHA1 with XBZRLE. I use XBZRLE in two ways:
> > 1. Begins to buffer pages from iteration 1; 2. As current
> > implementation, begins to buffer pages from iteration 2.
> >
> > I post the results of three workloads: cpu2006.zeusmp, cpu2006.mcf,
> > memcached.
> > I set the cache size as 256MB for zeusmp & mcf (they run in VM with
> > 1GB ram), and set the cache size as 1GB for memcached (it run in VM
> > with 6GB ram, and memcached takes 4GB as cache).
> >
> > As you can read from the data below, beginning to buffer pages from
> > iteration 1 is better than the current implementation(from iteration
> > 2), because the total migration time is shorter.
> >
> > SHA1 is better than the XBZRLE with the cache size I choose, because
> > it leads to shorter migration time, and consumes far less memory
> > overhead (<1/200 of the total VM memory).
> >
>
> Hi Chunguang,
>
> Have you tried to use a large XBZRLE cache size which equals to the guest's
> RAM size?
> Is SHA1 faster in that case?
>
> Thanks!
> Liang
Intel's future chipset will contain hardware engines which supports SHA-x and MD5,
We can make use these engines to offload the overhead from CPU for SHA/MD5 calculation.
Liang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-07 14:17 ` Li, Liang Z
@ 2016-11-08 5:27 ` Chunguang Li
0 siblings, 0 replies; 21+ messages in thread
From: Chunguang Li @ 2016-11-08 5:27 UTC (permalink / raw)
To: Li, Liang Z
Cc: Dr. David Alan Gilbert, Amit Shah, pbonzini, qemu-devel,
stefanha, quintela
> -----Original Messages-----
> From: "Li, Liang Z" <liang.z.li@intel.com>
> Sent Time: Monday, November 7, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Amit Shah" <amit.shah@redhat.com>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "stefanha@redhat.com" <stefanha@redhat.com>, "quintela@redhat.com" <quintela@redhat.com>
> Subject: RE: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> > > > > > > > > > I think this is "very" wasteful. Assume the workload
> > > > > > > > > > writes the pages
> > > > > > dirty randomly within the guest address space, and the transfer
> > > > > > speed is constant. Intuitively, I think nearly half of the dirty
> > > > > > pages produced in Iteration 1 is not really dirty. This means
> > > > > > the time of Iteration 2 is double of that to send only really dirty pages.
> > > > > > > > >
> > > > > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > > > > kinds of workloads get impacted the most? That would also
> > > > > > > > > help us to figure out what kinds of speed improvements we
> > > > > > > > > can
> > > > expect.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amit
> > > > > > > >
> > > > > > > > I have picked up 6 workloads and got the following
> > > > > > > > statistics numbers of every iteration (except the last
> > > > > > > > stop-copy one) during
> > > > precopy.
> > > > > > > > These numbers are obtained with the basic precopy migration,
> > > > > > > > without the capabilities like xbzrle or compression, etc.
> > > > > > > > The network for the migration is exclusive, with a separate
> > > > > > > > network for
> > > > the workloads.
> > > > > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > > > > >
> > > > > > > > Three (booting, idle, web server) of them converged to the
> > > > > > > > stop-copy
> > > > > > phase,
> > > > > > > > with the given bandwidth and default downtime (300ms), while
> > > > > > > > the other three (kernel compilation, zeusmp, memcached) did not.
> > > > > > > >
> > > > > > > > One page is "not-really-dirty", if it is written first and
> > > > > > > > is sent later (and not written again after that) during one
> > > > > > > > iteration. I guess this would not happen so often during the
> > > > > > > > other iterations as during the 1st iteration. Because all
> > > > > > > > the pages of the VM are sent to the dest node
> > > > > > during
> > > > > > > > the 1st iteration, while during the others, only part of the
> > > > > > > > pages are
> > > > sent.
> > > > > > > > So I think the "not-really-dirty" pages should be produced
> > > > > > > > mainly during the 1st iteration , and maybe very little
> > > > > > > > during the other
> > > > iterations.
> > > > > > > >
> > > > > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > > > > halved. This is a chain
> > > > > > reaction,
> > > > > > > > because the dirty pages produced during Iteration 2 is
> > > > > > > > halved, which
> > > > > > incurs
> > > > > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > > > > >
> > > > > > > Yes; these numbers don't show how many of them are false dirty
> > > > though.
> > > > > > >
> > > > > > > One problem is thinking about pages that have been redirtied,
> > > > > > > if the page is
> > > > > > dirtied
> > > > > > > after the sync but before the network write then it's the
> > > > > > > false-dirty that you're describing.
> > > > > > >
> > > > > > > However, if the page is being written a few times, and so it
> > > > > > > would have
> > > > > > been written
> > > > > > > after the network write then it isn't a false-dirty.
> > > > > > >
> > > > > > > You might be able to figure that out with some kernel tracing
> > > > > > > of when the
> > > > > > dirtying
> > > > > > > happens, but it might be easier to write the fix!
> > > > > > >
> > > > > > > Dave
> > > > > >
> > > > > > Hi, I have made some new progress now.
> > > > > >
> > > > > > To tell how many false dirty pages there are exactly in each
> > > > > > iteration, I malloc a buffer in memory as big as the size of the
> > > > > > whole VM memory. When a page is transferred to the dest node, it
> > > > > > is copied to the buffer; During the next iteration, if one page
> > > > > > is transferred, it is compared to the old one in the buffer, and
> > > > > > the old one will be replaced for next comparison if it is really dirty.
> > > > > > Thus, we are now able to get the exact number of false dirty pages.
> > > > > >
> > > > > > This time, I use 15 workloads to get the statistic number. They are:
> > > > > >
> > > > > > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They
> > > > > > are all scientific
> > > > > > computing workloads like Quantum Chromodynamics, Fluid
> > > > > > Dynamics,
> > > > etc.
> > > > > > I pick
> > > > > > up these 11 benchmarks because compared to others, they
> > > > > > have bigger memory
> > > > > > occupation and higher memory dirty rate. Thus most of them
> > > > > > could not converge
> > > > > > to stop-and-copy using the default migration speed (32MB/s).
> > > > > > 2. kernel compilation
> > > > > > 3. idle VM
> > > > > > 4. Apache web server which serves static content
> > > > > >
> > > > > > (the above workloads are all running in VM with 1 vcpu and 1GB
> > > > > > memory, and the
> > > > > > migration speed is the default 32MB/s)
> > > > > >
> > > > > > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB
> > > > > > are used as the cache.
> > > > > > After filling up the 4GB cache, a client writes the cache
> > > > > > at a constant
> > > > speed
> > > > > > during migration. This time, migration speed has no limit,
> > > > > > and is up to
> > > > the
> > > > > > capability of 1Gbps Ethernet.
> > > > > >
> > > > > > Summarize the results first: (and you can read the precise
> > > > > > number
> > > > > > below)
> > > > > >
> > > > > > 1. 4 of these 15 workloads have a big proportion (>60%, even
> > > > > > >80% during some iterations)
> > > > > > of false dirty pages out of all the dirty pages since
> > > > > > iteration 2 (and the
> > > > big
> > > > > > proportion lasts during the following iterations). They are
> > > > cpu2006.zeusmp,
> > > > > > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > > > > 2. 2 workloads (idle, webserver) spend most of the migration
> > > > > > time on iteration 1, even
> > > > > > though the proportion of false dirty pages is big since
> > > > > > iteration 2, the space to
> > > > > > optimize is small.
> > > > > > 3. 1 workload (kernel compilation) only have a big proportion
> > > > > > during iteration 2, not
> > > > > > in the other iterations.
> > > > > > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > > > > proportion of false
> > > > > > dirty pages since iteration 2. So the spaces to optimize
> > > > > > for them are
> > > > small.
> > > > > >
> > > > > > Now I want to talk a little more about the reasons why false
> > > > > > dirty pages are produced.
> > > > > > The first reason is what we have discussed before---the
> > > > > > mechanism to track the dirty pages.
> > > > > > And then I come up with another reason. Here is the situation: a
> > > > > > write operation to one memory page happens, but it doesn't
> > > > > > change any content of the page. So it's "write but not dirty",
> > > > > > and kernel still marks it as dirty. One guy in our lab has done
> > > > > > some experiments to figure out the proportion of "write but not
> > dirty"
> > > > > > operations, and he uses the cpu2006 benchmark suit. According to
> > > > > > his results, general workloads has a little proportion (<10%) of
> > > > > > "write but not dirty" out of all the write operations, while few
> > > > > > workloads have higher proportion (one even as high as 50%). Now
> > > > > > we are not sure why "write but not dirty" would happen, it just
> > happened.
> > > > > >
> > > > > > So these two reasons contribute to the false dirty pages. To
> > > > > > optimize, I compute and store the SHA1 hash before transferring
> > > > > > each page. Next time, if one page needs retransmission, its
> > > > > > SHA1 hash is computed again, and compared to the old hash. If
> > > > > > the hash is the same, it's a false dirty page, and we just skip
> > > > > > this page; Otherwise, the page is transferred, and the new hash
> > > > > > replaces the old one for next comparison.
> > > > > > The reason to use SHA1 hash but not byte-by-byte comparison is
> > > > > > the memory overheads. One SHA1 hash is 20 bytes. So we need
> > > > > > extra
> > > > > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > > > > relatively small.
> > > > > > As far as I know, SHA1 hash is widely used in the scenes of
> > > > > > deduplication for backup systems.
> > > > > > They have proven that the probability of hash collision is far
> > > > > > smaller than disk hardware fault, so it's secure hash, that is,
> > > > > > if the hashes of two chunks are the same, the content must be the
> > same.
> > > > > > So I think the SHA1 hash could replace byte-to-byte comparison
> > > > > > in the VM memory scenery.
> > > > > >
> > > > > > Then I do the same migration experiments using the SHA1 hash.
> > > > > > For the 4 workloads which have big proportions of false dirty
> > > > > > pages, the improvement is remarkable. Without optimization, they
> > > > > > either can not converge to stop-and-copy, or take a very long time to
> > complete.
> > > > > > With the
> > > > > > SHA1 hash method, all of them now complete in a relatively short
> > time.
> > > > > > For the reason I have talked above, the other workloads don't
> > > > > > get notable improvements from the optimization. So below, I only
> > > > > > show the exact number after optimization for the 4 workloads
> > > > > > with remarkable improvements.
> > > > > >
> > > > > > Any comments or suggestions?
> > > > >
> > > > > Maybe you can compare the performance of your solution as that of
> > > > XBZRLE to see which one is better.
> > > > > The merit of using SHA1 is that it can avoid data copy as that in
> > > > > XBZRLE, and
> > > > need less buffer.
> > > > > How about the overhead of calculating the SHA1? Is it faster than
> > > > > copying a
> > > > page?
> > > > >
> > > > > Liang
> > > > >
> > > > >
> > > >
> > > > Yes, XBZRLE is able to handle the false dirty pages. However, if we
> > > > want to avoid transferring all of the false dirty pages using
> > > > XBZRLE, we need a buffer as big as the whole VM memory, while SHA1
> > > > needs a much small buffer. Of course, if we have a buffer as big as
> > > > the whole VM memory using XBZRLE, we could transfer less data on
> > > > network than SHA1, because XBZRLE is able to compress similar pages.
> > > > In a word, yes, the merit of using SHA1 is that it needs much less
> > > > buffer, and leads to nice improvement if there are many false dirty pages.
> > > >
> > >
> > > The current implementation of XBZRLE begins to buffer page from the
> > > second iteration, Maybe it's worth to make it start to work from the first
> > iteration based on your finding.
> > >
> > > > In terms of the overhead of calculating the SHA1 compared with
> > > > transferring a page, it's related to the CPU and network
> > > > performance. In my test environment(Intel Xeon
> > > > E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra
> > > > computing overhead caused by calculating the SHA1, because the
> > > > throughput of network (got by "info migrate") remains almost the same.
> > >
> > > You can check the CPU usage, or to measure the time spend on a local
> > > live migration which use SHA1/ XBZRLE.
> > >
> > > Liang
> > >
> > >
> >
> > I compare SHA1 with XBZRLE. I use XBZRLE in two ways:
> > 1. Begins to buffer pages from iteration 1; 2. As current implementation,
> > begins to buffer pages from iteration 2.
> >
> > I post the results of three workloads: cpu2006.zeusmp, cpu2006.mcf,
> > memcached.
> > I set the cache size as 256MB for zeusmp & mcf (they run in VM with 1GB
> > ram), and set the cache size as 1GB for memcached (it run in VM with 6GB
> > ram, and memcached takes 4GB as cache).
> >
> > As you can read from the data below, beginning to buffer pages from
> > iteration 1 is better than the current implementation(from iteration 2),
> > because the total migration time is shorter.
> >
> > SHA1 is better than the XBZRLE with the cache size I choose, because it leads
> > to shorter migration time, and consumes far less memory overhead (<1/200
> > of the total VM memory).
> >
>
> Hi Chunguang,
>
> Have you tried to use a large XBZRLE cache size which equals to the guest's RAM size?
> Is SHA1 faster in that case?
>
> Thanks!
> Liang
You can check the data below. For zeusmp and mcf when the XBZRLE cache size equals to
the guest's RAM size (in fact, the 1024 cache size is a little smaller than the RAM
size, because the guest's RAM has a little extra ram space besides the 1GB we set),
XBZRLE is faster than SHA1.
For the memcached, I am not able to set the cache size as the 6GB RAM size, because the
cache size has to be a power of 2; And I am not able to set it larger than RAM size, because
the current implementation doesn't allow that. So I set the cache size as 4GB, and XBZRLE
with this cache size is almost the same as SHA1 in terms of migration time.
Note that XBZRLE begins to buffer pages from iteration 1.
zeusmp 1024MB cache
Iteration 1, duration: 21604 ms , transferred pages: 266450 (dup: 89509, n: 176941, x: 0) , new dirty pages: 129647 , remaining dirty pages: 129647
Iteration 2, duration: 652 ms , transferred pages: 89270 (dup: 78176, n: 1085, x: 10009) , new dirty pages: 46438 , remaining dirty pages: 46438
Iteration 3, duration: 400 ms , transferred pages: 35789 (dup: 30536, n: 0, x: 5253) , new dirty pages: 33569 , remaining dirty pages: 33569
Iteration 4, duration: 470 ms , transferred pages: 19106 (dup: 10317, n: 75, x: 8714) , new dirty pages: 39307 , remaining dirty pages: 39307
Iteration 5, duration: 72 ms , transferred pages: 17853 (dup: 15904, n: 0, x: 1949) , new dirty pages: 4078 , remaining dirty pages: 4078
Iteration 6, duration: 10 ms , transferred pages: 3280 (dup: 2910, n: 0, x: 370) , new dirty pages: 521 , remaining dirty pages: 521
Iteration 7, duration: 254 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 0 , remaining dirty pages: 521
total time: 23481 milliseconds (v.s. 27225 milliseconds for SHA1)
mcf 1024MB cache
Iteration 1, duration: 31704 ms , transferred pages: 266450 (dup: 6794, n: 259656, x: 0) , new dirty pages: 233250 , remaining dirty pages: 233250
Iteration 2, duration: 544 ms , transferred pages: 34186 (dup: 182, n: 423, x: 33581) , new dirty pages: 32757 , remaining dirty pages: 32757
Iteration 3, duration: 67 ms , transferred pages: 8536 (dup: 0, n: 0, x: 8536) , new dirty pages: 5305 , remaining dirty pages: 5305
Iteration 4, duration: 13 ms , transferred pages: 2125 (dup: 0, n: 0, x: 2125) , new dirty pages: 1632 , remaining dirty pages: 1632
Iteration 5, duration: 9 ms , transferred pages: 1038 (dup: 0, n: 0, x: 1038) , new dirty pages: 1095 , remaining dirty pages: 1095
Iteration 6, duration: 3 ms , transferred pages: 592 (dup: 0, n: 0, x: 592) , new dirty pages: 1148 , remaining dirty pages: 1148
Iteration 7, duration: 2 ms , transferred pages: 136 (dup: 0, n: 0, x: 136) , new dirty pages: 1123 , remaining dirty pages: 1123
Iteration 8, duration: 2 ms , transferred pages: 2 (dup: 0, n: 0, x: 2) , new dirty pages: 985 , remaining dirty pages: 985
Iteration 9, duration: 2 ms , transferred pages: 14 (dup: 0, n: 0, x: 14) , new dirty pages: 640 , remaining dirty pages: 640
Iteration 10, duration: 2 ms , transferred pages: 16 (dup: 0, n: 0, x: 16) , new dirty pages: 622 , remaining dirty pages: 622
Iteration 11, duration: 1 ms , transferred pages: 1 (dup: 0, n: 0, x: 1) , new dirty pages: 693 , remaining dirty pages: 693
Iteration 12, duration: 1 ms , transferred pages: 122 (dup: 0, n: 0, x: 122) , new dirty pages: 639 , remaining dirty pages: 639
Iteration 13, duration: 2 ms , transferred pages: 475 (dup: 0, n: 0, x: 475) , new dirty pages: 522 , remaining dirty pages: 522
Iteration 14, duration: 22 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 27 , remaining dirty pages: 549
total time: 32393 milliseconds (v.s. 97919 milliseconds for SHA1)
memcached 4096MB cache
Iteration 1, duration: 41025 ms , transferred pages: 1569059 (dup: 395085, n: 1173974, x: 0) , new dirty pages: 560788 , remaining dirty pages: 568899
Iteration 2, duration: 8218 ms , transferred pages: 300889 (dup: 3963, n: 142928, x: 153998) , new dirty pages: 158832 , remaining dirty pages: 167022
Iteration 3, duration: 2408 ms , transferred pages: 98923 (dup: 285, n: 33854, x: 64784) , new dirty pages: 68647 , remaining dirty pages: 77338
Iteration 4, duration: 869 ms , transferred pages: 43408 (dup: 64, n: 17911, x: 25433) , new dirty pages: 26087 , remaining dirty pages: 33845
Iteration 5, duration: 455 ms , transferred pages: 23048 (dup: 55, n: 10156, x: 12837) , new dirty pages: 15275 , remaining dirty pages: 16636
Iteration 6, duration: 162 ms , transferred pages: 7939 (dup: 55, n: 2425, x: 5459) , new dirty pages: 6009 , remaining dirty pages: 10051
Iteration 7, duration: 52 ms , transferred pages: 5761 (dup: 212, n: 707, x: 4842) , new dirty pages: 2204 , remaining dirty pages: 4027
Iteration 8, duration: 1 ms , transferred pages: 0 (dup: 0, n: 0, x: 0) , new dirty pages: 0 , remaining dirty pages: 4027
total time: 53255 milliseconds (v.s. 54693 milliseconds for SHA1)
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-03 8:25 ` Chunguang Li
2016-11-03 9:59 ` Li, Liang Z
2016-11-03 10:13 ` Li, Liang Z
@ 2016-11-08 11:05 ` Dr. David Alan Gilbert
2016-11-08 13:40 ` Chunguang Li
2 siblings, 1 reply; 21+ messages in thread
From: Dr. David Alan Gilbert @ 2016-11-08 11:05 UTC (permalink / raw)
To: Chunguang Li; +Cc: Amit Shah, qemu-devel, pbonzini, stefanha, quintela
* Chunguang Li (lichunguang@hust.edu.cn) wrote:
>
>
>
> > -----Original Messages-----
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > Sent Time: Friday, October 14, 2016
> > To: "Chunguang Li" <lichunguang@hust.edu.cn>
> > Cc: "Amit Shah" <amit.shah@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> >
> > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > >
> > >
> > >
> > > > -----原始邮件-----
> > > > 发件人: "Amit Shah" <amit.shah@redhat.com>
> > > > 发送时间: 2016年9月30日 星期五
> > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > >
> > > > On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> > > > >
> > > > >
> > > > >
> > > > > > -----原始邮件-----
> > > > > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > > > 发送时间: 2016年9月26日 星期一
> > > > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > > > >
> > > > > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > > > > Hi all!
> > > > > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > > > > >
> > > > > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > > > > >
> > > > > > >
> > > > > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > > > > >
> > > > > > I think you're right that this can happen; to clarify I think the
> > > > > > case you're talking about is:
> > > > > >
> > > > > > Iteration 1
> > > > > > sync bitmap
> > > > > > start sending pages
> > > > > > page 'n' is modified - but hasn't been sent yet
> > > > > > page 'n' gets sent
> > > > > > Iteration 2
> > > > > > sync bitmap
> > > > > > 'page n is shown as modified'
> > > > > > send page 'n' again
> > > > > >
> > > > >
> > > > > Yes，this is right the case I am talking about.
> > > > >
> > > > > > So you're right that is wasteful; I guess it's more wasteful
> > > > > > on big VMs with slow networks where the length of each iteration
> > > > > > is large.
> > > > >
> > > > > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
> > > >
> > > > It makes sense, can you get some perf numbers to show what kinds of
> > > > workloads get impacted the most? That would also help us to figure
> > > > out what kinds of speed improvements we can expect.
> > > >
> > > >
> > > > Amit
> > >
> > > I have picked up 6 workloads and got the following statistics numbers
> > > of every iteration (except the last stop-copy one) during precopy.
> > > These numbers are obtained with the basic precopy migration, without
> > > the capabilities like xbzrle or compression, etc. The network for the
> > > migration is exclusive, with a separate network for the workloads.
> > > They are both gigabit ethernet. I use qemu-2.5.1.
> > >
> > > Three (booting, idle, web server) of them converged to the stop-copy phase,
> > > with the given bandwidth and default downtime (300ms), while the other
> > > three (kernel compilation, zeusmp, memcached) did not.
> > >
> > > One page is "not-really-dirty", if it is written first and is sent later
> > > (and not written again after that) during one iteration. I guess this
> > > would not happen so often during the other iterations as during the 1st
> > > iteration. Because all the pages of the VM are sent to the dest node during
> > > the 1st iteration, while during the others, only part of the pages are sent.
> > > So I think the "not-really-dirty" pages should be produced mainly during
> > > the 1st iteration , and maybe very little during the other iterations.
> > >
> > > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > > think the time spent on Iteration 2 would be halved. This is a chain reaction,
> > > because the dirty pages produced during Iteration 2 is halved, which incurs
> > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> >
> > Yes; these numbers don't show how many of them are false dirty though.
> >
> > One problem is thinking about pages that have been redirtied, if the page is dirtied
> > after the sync but before the network write then it's the false-dirty that
> > you're describing.
> >
> > However, if the page is being written a few times, and so it would have been written
> > after the network write then it isn't a false-dirty.
> >
> > You might be able to figure that out with some kernel tracing of when the dirtying
> > happens, but it might be easier to write the fix!
> >
> > Dave
>
> Hi, I have made some new progress now.
>
> To tell how many false dirty pages there are exactly in each iteration, I malloc a
> buffer in memory as big as the size of the whole VM memory. When a page is
> transferred to the dest node, it is copied to the buffer; During the next iteration,
> if one page is transferred, it is compared to the old one in the buffer, and the
> old one will be replaced for next comparison if it is really dirty. Thus, we are now
> able to get the exact number of false dirty pages.
>
> This time, I use 15 workloads to get the statistic number. They are:
>
> 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all scientific
> computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc. I pick
> up these 11 benchmarks because compared to others, they have bigger memory
> occupation and higher memory dirty rate. Thus most of them could not converge
> to stop-and-copy using the default migration speed (32MB/s).
> 2. kernel compilation
> 3. idle VM
> 4. Apache web server which serves static content
>
> (the above workloads are all running in VM with 1 vcpu and 1GB memory, and the
> migration speed is the default 32MB/s)
>
> 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used as the cache.
> After filling up the 4GB cache, a client writes the cache at a constant speed
> during migration. This time, migration speed has no limit, and is up to the
> capability of 1Gbps Ethernet.
>
> Summarize the results first: (and you can read the precise number below)
>
> 1. 4 of these 15 workloads have a big proportion (>60%, even >80% during some iterations)
> of false dirty pages out of all the dirty pages since iteration 2 (and the big
> proportion lasts during the following iterations). They are cpu2006.zeusmp,
> cpu2006.bzip2, cpu2006.mcf, and memcached.
> 2. 2 workloads (idle, webserver) spend most of the migration time on iteration 1, even
> though the proportion of false dirty pages is big since iteration 2, the space to
> optimize is small.
> 3. 1 workload (kernel compilation) only have a big proportion during iteration 2, not
> in the other iterations.
> 4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of false
> dirty pages since iteration 2. So the spaces to optimize for them are small.
>
> Now I want to talk a little more about the reasons why false dirty pages are produced.
> The first reason is what we have discussed before---the mechanism to track the dirty
> pages.
> And then I come up with another reason. Here is the situation: a write operation to one
> memory page happens, but it doesn't change any content of the page. So it's "write but
> not dirty", and kernel still marks it as dirty. One guy in our lab has done some experiments
> to figure out the proportion of "write but not dirty" operations, and he uses the cpu2006
> benchmark suit. According to his results, general workloads has a little proportion (<10%)
> of "write but not dirty" out of all the write operations, while few workloads have higher
> proportion (one even as high as 50%). Now we are not sure why "write but not dirty" would
> happen, it just happened.
I think there are a few different reasons I can think of:
a) You have a flag or mutex that's set and cleared; so it gets set (marked
dirty) and cleared around some operation. By the time we come to migrate
it then it's back to cleared again.
Similarly with other temporary data structures.
b) Some system operation causes the page to be moved - e.g. swap or the kernel
reorganising memory.
However, it's a shame I don't think you can tell in your experiment which of the
two cases we're hitting? I'd like to know if it's worth working on
making the page sync mechanism better or if it's nore important to deal
with the second reason you show.
> So these two reasons contribute to the false dirty pages. To optimize, I compute and store
> the SHA1 hash before transferring each page. Next time, if one page needs retransmission, its
> SHA1 hash is computed again, and compared to the old hash. If the hash is the same, it's a
> false dirty page, and we just skip this page; Otherwise, the page is transferred, and the new
> hash replaces the old one for next comparison.
> The reason to use SHA1 hash but not byte-by-byte comparison is the memory overheads. One SHA1
> hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the whole VM memory, which
> is relatively small.
> As far as I know, SHA1 hash is widely used in the scenes of deduplication for backup systems.
> They have proven that the probability of hash collision is far smaller than disk hardware fault,
> so it's secure hash, that is, if the hashes of two chunks are the same, the content must be the
> same. So I think the SHA1 hash could replace byte-to-byte comparison in the VM memory scenery.
There was a proposal ( https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg05331.html )
to do a migration system where
a copy of the migration RAM is stored on disc on the destination for cases where similar VMs
are migrated, and it used a checksum for each page to find the matching page
in the cache; that originally used a smaller hash, I think in the end they used a SHA-256.
(Hash based checks still make me nervous for intentional collisions but that's probably
me being paranoid?)
> Then I do the same migration experiments using the SHA1 hash. For the 4 workloads which have
> big proportions of false dirty pages, the improvement is remarkable. Without optimization,
> they either can not converge to stop-and-copy, or take a very long time to complete. With the
> SHA1 hash method, all of them now complete in a relatively short time.
> For the reason I have talked above, the other workloads don't get notable improvements from the
> optimization. So below, I only show the exact number after optimization for the 4 workloads with
> remarkable improvements.
>
> Any comments or suggestions?
You might be able to save some of the CPU time; we've
got a test that checks if a page is all-zero; if you're doing
the SHA calculation you could avoid doing the all-zero check
and replace it by comparing hte output of the SHA.
>
> Below is the experiments data:
> (
> "dup" means zero page, this kind of pages takes very little migration time and network
> resources, so they are always not regard as dirty pages in my numbers;
> "rd" means really dirty pages;
> "fd" means false dirty pages;
> The numbers refer to the quantities of pages.
> )
>
> ------------------The 4 workloads with remarkable improvements (both the results of original precopy and with optimization are shown)-------------------
>
> 1. memcached
>
> ----- original pre-copy (can not converge): -----
> Iteration 1, duration: 42111 ms , transferred pages: 1568788 (dup: 416239, rd: 1152549, fd: 0) , new dirty pages: 499015 , remaining dirty pages: 507397
> Iteration 2, duration: 17208 ms , transferred pages: 498946 (dup: 5456, rd: 160206, fd: 333284) , new dirty pages: 261237 , remaining dirty pages: 269688
> Iteration 3, duration: 9134 ms , transferred pages: 262377 (dup: 519, rd: 111900, fd: 149958) , new dirty pages: 170281 , remaining dirty pages: 177592
> Iteration 4, duration: 5920 ms , transferred pages: 169966 (dup: 87, rd: 82487, fd: 87392) , new dirty pages: 121154 , remaining dirty pages: 128780
> Iteration 5, duration: 4239 ms , transferred pages: 121551 (dup: 81, rd: 64120, fd: 57350) , new dirty pages: 100976 , remaining dirty pages: 108205
> Iteration 6, duration: 3495 ms , transferred pages: 100353 (dup: 90, rd: 56021, fd: 44242) , new dirty pages: 74547 , remaining dirty pages: 82399
> Iteration 7, duration: 2583 ms , transferred pages: 74160 (dup: 56, rd: 38016, fd: 36088) , new dirty pages: 58209 , remaining dirty pages: 66448
> Iteration 8, duration: 2039 ms , transferred pages: 58534 (dup: 81, rd: 26885, fd: 31568) , new dirty pages: 43511 , remaining dirty pages: 51425
> Iteration 9, duration: 1513 ms , transferred pages: 43484 (dup: 55, rd: 26641, fd: 16788) , new dirty pages: 43722 , remaining dirty pages: 51663
> Iteration 10, duration: 1521 ms , transferred pages: 43676 (dup: 62, rd: 26463, fd: 17151) , new dirty pages: 35347 , remaining dirty pages: 43334
> Iteration 11, duration: 1230 ms , transferred pages: 35287 (dup: 0, rd: 21293, fd: 13994) , new dirty pages: 28851 , remaining dirty pages: 36898
> Iteration 12, duration: 1031 ms , transferred pages: 29651 (dup: 82, rd: 18143, fd: 11426) , new dirty pages: 27062 , remaining dirty pages: 34309
> Iteration 13, duration: 917 ms , transferred pages: 26385 (dup: 56, rd: 14149, fd: 12180) , new dirty pages: 22723 , remaining dirty pages: 30647
> Iteration 14, duration: 762 ms , transferred pages: 21902 (dup: 55, rd: 16355, fd: 5492) , new dirty pages: 18208 , remaining dirty pages: 26953
> Iteration 15, duration: 650 ms , transferred pages: 18636 (dup: 0, rd: 11943, fd: 6693) , new dirty pages: 16085 , remaining dirty pages: 24402
> Iteration 16, duration: 554 ms , transferred pages: 15946 (dup: 56, rd: 9527, fd: 6363) , new dirty pages: 14766 , remaining dirty pages: 23222
> Iteration 17, duration: 538 ms , transferred pages: 15434 (dup: 0, rd: 9779, fd: 5655) , new dirty pages: 13381 , remaining dirty pages: 21169
> Iteration 18, duration: 487 ms , transferred pages: 14089 (dup: 81, rd: 7737, fd: 6271) , new dirty pages: 13325 , remaining dirty pages: 20405
> Iteration 19, duration: 428 ms , transferred pages: 12232 (dup: 0, rd: 8488, fd: 3744) , new dirty pages: 10274 , remaining dirty pages: 18447
> Iteration 20, duration: 377 ms , transferred pages: 10887 (dup: 56, rd: 6362, fd: 4469) , new dirty pages: 9708 , remaining dirty pages: 17268
> Iteration 21, duration: 320 ms , transferred pages: 9222 (dup: 0, rd: 5789, fd: 3433) , new dirty pages: 8015 , remaining dirty pages: 16061
> Iteration 22, duration: 268 ms , transferred pages: 7621 (dup: 0, rd: 6204, fd: 1417) , new dirty pages: 7227 , remaining dirty pages: 15667
> Iteration 23, duration: 269 ms , transferred pages: 7813 (dup: 56, rd: 4410, fd: 3347) , new dirty pages: 7591 , remaining dirty pages: 15445
> Iteration 24, duration: 271 ms , transferred pages: 7749 (dup: 0, rd: 4565, fd: 3184) , new dirty pages: 15126 , remaining dirty pages: 22822
> Iteration 25, duration: 549 ms , transferred pages: 15818 (dup: 60, rd: 10545, fd: 5213) , new dirty pages: 14559 , remaining dirty pages: 21563
> Iteration 26, duration: 499 ms , transferred pages: 14281 (dup: 3, rd: 8760, fd: 5518) , new dirty pages: 11947 , remaining dirty pages: 19229
> Iteration 27, duration: 376 ms , transferred pages: 10823 (dup: 25, rd: 6550, fd: 4248) , new dirty pages: 8561 , remaining dirty pages: 16967
> Iteration 28, duration: 324 ms , transferred pages: 9350 (dup: 31, rd: 5292, fd: 4027) , new dirty pages: 8655 , remaining dirty pages: 16272
> Iteration 29, duration: 274 ms , transferred pages: 7813 (dup: 0, rd: 6088, fd: 1725) , new dirty pages: 6300 , remaining dirty pages: 14759
> Iteration 30, duration: 218 ms , transferred pages: 6340 (dup: 45, rd: 3196, fd: 3099) , new dirty pages: 5143 , remaining dirty pages: 13562
>
> ----- after optimization: -----
> Iteration 1, duration: 40664 ms , transferred pages: 1569037 (dup: 405940, rd: 1163097) , new dirty pages: 506846 , remaining dirty pages: 514979
> Iteration 2, duration: 8032 ms , transferred pages: 161130 (dup: 4007, rd: 157123) , new dirty pages: 153479 , remaining dirty pages: 153479
Big difference.
> Iteration 3, duration: 2620 ms , transferred pages: 65260 (dup: 20, rd: 65240) , new dirty pages: 64014 , remaining dirty pages: 67100
> Iteration 4, duration: 1160 ms , transferred pages: 30227 (dup: 60, rd: 30167) , new dirty pages: 34031 , remaining dirty pages: 41414
> Iteration 5, duration: 648 ms , transferred pages: 18700 (dup: 56, rd: 18644) , new dirty pages: 18375 , remaining dirty pages: 25536
> Iteration 6, duration: 389 ms , transferred pages: 11399 (dup: 55, rd: 11344) , new dirty pages: 12536 , remaining dirty pages: 17516
> Iteration 7, duration: 292 ms , transferred pages: 8197 (dup: 0, rd: 8197) , new dirty pages: 8387 , remaining dirty pages: 16802
> Iteration 8, duration: 171 ms , transferred pages: 4931 (dup: 39, rd: 4892) , new dirty pages: 6182 , remaining dirty pages: 14060
> Iteration 9, duration: 163 ms , transferred pages: 4355 (dup: 16, rd: 4339) , new dirty pages: 5530 , remaining dirty pages: 11973
> Iteration 10, duration: 104 ms , transferred pages: 3266 (dup: 0, rd: 3266) , new dirty pages: 2893 , remaining dirty pages: 11014
> Iteration 11, duration: 52 ms , transferred pages: 1153 (dup: 0, rd: 1153) , new dirty pages: 1586 , remaining dirty pages: 10516
> Iteration 12, duration: 52 ms , transferred pages: 1921 (dup: 39, rd: 1882) , new dirty pages: 1619 , remaining dirty pages: 8842
> Iteration 13, duration: 62 ms , transferred pages: 1537 (dup: 0, rd: 1537) , new dirty pages: 2052 , remaining dirty pages: 8871
> Iteration 14, duration: 58 ms , transferred pages: 1665 (dup: 0, rd: 1665) , new dirty pages: 1947 , remaining dirty pages: 7989
> Iteration 15, duration: 2 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 7989
> total time: 54693 milliseconds
Very nice.
Dave
> 2. cpu2006.zeusmp
>
> ----- original pre-copy (can not converge): -----
> Iteration 1, duration: 21112 ms , transferred pages: 266450 (dup: 93385, rd: 173065, fd: 0) , new dirty pages: 127866 , remaining dirty pages: 127866
> Iteration 2, duration: 6192 ms , transferred pages: 125662 (dup: 75762, rd: 17389, fd: 32511) , new dirty pages: 131655 , remaining dirty pages: 133859
> Iteration 3, duration: 6699 ms , transferred pages: 131937 (dup: 77298, rd: 20320, fd: 34319) , new dirty pages: 121027 , remaining dirty pages: 122949
> Iteration 4, duration: 5999 ms , transferred pages: 122512 (dup: 73588, rd: 17236, fd: 31688) , new dirty pages: 122759 , remaining dirty pages: 123196
> Iteration 5, duration: 5804 ms , transferred pages: 122717 (dup: 75436, rd: 19016, fd: 28265) , new dirty pages: 123697 , remaining dirty pages: 124176
> Iteration 6, duration: 5698 ms , transferred pages: 123708 (dup: 77249, rd: 18022, fd: 28437) , new dirty pages: 121838 , remaining dirty pages: 122306
> Iteration 7, duration: 5515 ms , transferred pages: 122306 (dup: 76727, rd: 14819, fd: 30760) , new dirty pages: 122382 , remaining dirty pages: 122382
> Iteration 8, duration: 6086 ms , transferred pages: 120825 (dup: 71834, rd: 15987, fd: 33004) , new dirty pages: 121587 , remaining dirty pages: 123144
> Iteration 9, duration: 5899 ms , transferred pages: 120964 (dup: 72860, rd: 18191, fd: 29913) , new dirty pages: 120391 , remaining dirty pages: 122571
> Iteration 10, duration: 5801 ms , transferred pages: 121425 (dup: 74140, rd: 20722, fd: 26563) , new dirty pages: 122302 , remaining dirty pages: 123448
> Iteration 11, duration: 5909 ms , transferred pages: 123448 (dup: 74735, rd: 19678, fd: 29035) , new dirty pages: 123258 , remaining dirty pages: 123258
> Iteration 12, duration: 6293 ms , transferred pages: 121211 (dup: 70442, rd: 18128, fd: 32641) , new dirty pages: 123623 , remaining dirty pages: 125670
> Iteration 13, duration: 6398 ms , transferred pages: 124897 (dup: 72701, rd: 21134, fd: 31062) , new dirty pages: 122355 , remaining dirty pages: 123128
> Iteration 14, duration: 6301 ms , transferred pages: 121893 (dup: 70514, rd: 23470, fd: 27909) , new dirty pages: 120980 , remaining dirty pages: 122215
> Iteration 15, duration: 6304 ms , transferred pages: 121389 (dup: 70005, rd: 21731, fd: 29653) , new dirty pages: 121628 , remaining dirty pages: 122454
> Iteration 16, duration: 6398 ms , transferred pages: 122164 (dup: 69962, rd: 24376, fd: 27826) , new dirty pages: 122246 , remaining dirty pages: 122536
> Iteration 17, duration: 6201 ms , transferred pages: 121548 (dup: 70984, rd: 23915, fd: 26649) , new dirty pages: 121460 , remaining dirty pages: 122448
> Iteration 18, duration: 6401 ms , transferred pages: 122272 (dup: 70072, rd: 22261, fd: 29939) , new dirty pages: 123518 , remaining dirty pages: 123694
> Iteration 19, duration: 7003 ms , transferred pages: 121873 (dup: 64754, rd: 27325, fd: 29794) , new dirty pages: 120568 , remaining dirty pages: 122389
> Iteration 20, duration: 6400 ms , transferred pages: 121422 (dup: 69221, rd: 25300, fd: 26901) , new dirty pages: 121229 , remaining dirty pages: 122196
> Iteration 21, duration: 6703 ms , transferred pages: 119895 (dup: 65232, rd: 25877, fd: 28786) , new dirty pages: 123284 , remaining dirty pages: 125585
> Iteration 22, duration: 6902 ms , transferred pages: 123884 (dup: 67582, rd: 29020, fd: 27282) , new dirty pages: 122057 , remaining dirty pages: 123758
> Iteration 23, duration: 6800 ms , transferred pages: 122010 (dup: 66529, rd: 30644, fd: 24837) , new dirty pages: 120916 , remaining dirty pages: 122664
> Iteration 24, duration: 7202 ms , transferred pages: 121951 (dup: 63188, rd: 31105, fd: 27658) , new dirty pages: 122715 , remaining dirty pages: 123428
> Iteration 25, duration: 7202 ms , transferred pages: 122919 (dup: 64161, rd: 32063, fd: 26695) , new dirty pages: 123180 , remaining dirty pages: 123689
> Iteration 26, duration: 7404 ms , transferred pages: 123092 (dup: 62694, rd: 33459, fd: 26939) , new dirty pages: 122149 , remaining dirty pages: 122746
> Iteration 27, duration: 7205 ms , transferred pages: 120427 (dup: 61664, rd: 34344, fd: 24419) , new dirty pages: 120299 , remaining dirty pages: 122618
> Iteration 28, duration: 7100 ms , transferred pages: 121074 (dup: 63130, rd: 32403, fd: 25541) , new dirty pages: 122984 , remaining dirty pages: 124528
> Iteration 29, duration: 7904 ms , transferred pages: 124060 (dup: 59564, rd: 35631, fd: 28865) , new dirty pages: 127080 , remaining dirty pages: 127548
> Iteration 30, duration: 7906 ms , transferred pages: 127518 (dup: 63029, rd: 34416, fd: 30073) , new dirty pages: 125028 , remaining dirty pages: 125058
>
> ----- after optimization: -----
> Iteration 1, duration: 21601 ms , transferred pages: 266450 (dup: 89731, rd: 176719) , new dirty pages: 139843 , remaining dirty pages: 139843
> Iteration 2, duration: 1747 ms , transferred pages: 92077 (dup: 78364, rd: 13713) , new dirty pages: 90945 , remaining dirty pages: 90945
> Iteration 3, duration: 1592 ms , transferred pages: 62253 (dup: 49435, rd: 12818) , new dirty pages: 76929 , remaining dirty pages: 76929
> Iteration 4, duration: 992 ms , transferred pages: 44837 (dup: 37886, rd: 6951) , new dirty pages: 71331 , remaining dirty pages: 72916
> Iteration 5, duration: 998 ms , transferred pages: 55229 (dup: 47150, rd: 8079) , new dirty pages: 21703 , remaining dirty pages: 23302
> Iteration 6, duration: 211 ms , transferred pages: 20337 (dup: 18516, rd: 1821) , new dirty pages: 14500 , remaining dirty pages: 14500
> Iteration 7, duration: 31 ms , transferred pages: 12933 (dup: 12627, rd: 306) , new dirty pages: 1520 , remaining dirty pages: 1520
> Iteration 8, duration: 30 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 4 , remaining dirty pages: 1524
> total time: 27225 milliseconds
>
> 3. cpu2006.bzip2
>
> ----- original pre-copy: -----
> Iteration 1, duration: 18306 ms , transferred pages: 266450 (dup: 116569, rd: 149881, fd: 0) , new dirty pages: 106299 , remaining dirty pages: 106299
> Iteration 2, duration: 10694 ms , transferred pages: 104611 (dup: 17550, rd: 10536, fd: 76525) , new dirty pages: 34394 , remaining dirty pages: 36082
> Iteration 3, duration: 2998 ms , transferred pages: 34442 (dup: 9924, rd: 12254, fd: 12264) , new dirty pages: 6419 , remaining dirty pages: 8059
> Iteration 4, duration: 699 ms , transferred pages: 5748 (dup: 22, rd: 2583, fd: 3143) , new dirty pages: 1226 , remaining dirty pages: 3537
> Iteration 5, duration: 200 ms , transferred pages: 1636 (dup: 0, rd: 1194, fd: 442) , new dirty pages: 478 , remaining dirty pages: 2379
> Iteration 6, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2379
>
> ----- after optimization: -----
> Iteration 1, duration: 13995 ms , transferred pages: 266314 (dup: 152118, rd: 114196) , new dirty pages: 97009 , remaining dirty pages: 97145
> Iteration 2, duration: 1215 ms , transferred pages: 33400 (dup: 26745, rd: 6655) , new dirty pages: 12866 , remaining dirty pages: 14017
> Iteration 3, duration: 701 ms , transferred pages: 5774 (dup: 48, rd: 5726) , new dirty pages: 6342 , remaining dirty pages: 8761
> Iteration 4, duration: 500 ms , transferred pages: 4111 (dup: 21, rd: 4090) , new dirty pages: 4311 , remaining dirty pages: 6485
> Iteration 5, duration: 400 ms , transferred pages: 3273 (dup: 1, rd: 3272) , new dirty pages: 3034 , remaining dirty pages: 5431
> Iteration 6, duration: 301 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2094 , remaining dirty pages: 4472
> Iteration 7, duration: 299 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2066 , remaining dirty pages: 4082
> Iteration 8, duration: 202 ms , transferred pages: 1636 (dup: 0, rd: 1636) , new dirty pages: 2881 , remaining dirty pages: 4648
> Iteration 9, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 4775 , remaining dirty pages: 6778
> Iteration 10, duration: 400 ms , transferred pages: 3281 (dup: 9, rd: 3272) , new dirty pages: 3757 , remaining dirty pages: 5576
> Iteration 11, duration: 401 ms , transferred pages: 3279 (dup: 7, rd: 3272) , new dirty pages: 6980 , remaining dirty pages: 8906
> Iteration 12, duration: 500 ms , transferred pages: 7118 (dup: 3035, rd: 4083) , new dirty pages: 10774 , remaining dirty pages: 11922
> Iteration 13, duration: 116 ms , transferred pages: 11706 (dup: 10152, rd: 1554) , new dirty pages: 1326 , remaining dirty pages: 1326
> Iteration 14, duration: 117 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 1326
> total time: 19479 milliseconds
>
> 4. cpu2006.mcf
>
> ----- original pre-copy: -----
> Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6925, rd: 259525, fd: 0) , new dirty pages: 244403 , remaining dirty pages: 244403
> Iteration 2, duration: 29603 ms , transferred pages: 242275 (dup: 377, rd: 224001, fd: 17897) , new dirty pages: 227335 , remaining dirty pages: 229463
> Iteration 3, duration: 27806 ms , transferred pages: 227573 (dup: 169, rd: 65681, fd: 161723) , new dirty pages: 195593 , remaining dirty pages: 197483
> Iteration 4, duration: 23907 ms , transferred pages: 195543 (dup: 41, rd: 39838, fd: 155664) , new dirty pages: 215066 , remaining dirty pages: 217006
> Iteration 5, duration: 26305 ms , transferred pages: 215289 (dup: 155, rd: 33082, fd: 182052) , new dirty pages: 111098 , remaining dirty pages: 112815
> Iteration 6, duration: 13502 ms , transferred pages: 110452 (dup: 22, rd: 26793, fd: 83637) , new dirty pages: 161054 , remaining dirty pages: 163417
> Iteration 7, duration: 19705 ms , transferred pages: 161266 (dup: 120, rd: 33818, fd: 127328) , new dirty pages: 220562 , remaining dirty pages: 222713
> Iteration 8, duration: 27003 ms , transferred pages: 220881 (dup: 21, rd: 215721, fd: 5139) , new dirty pages: 219787 , remaining dirty pages: 221619
> Iteration 9, duration: 26802 ms , transferred pages: 219248 (dup: 24, rd: 84648, fd: 134576) , new dirty pages: 207959 , remaining dirty pages: 210330
> Iteration 10, duration: 25411 ms , transferred pages: 207916 (dup: 144, rd: 35842, fd: 171930) , new dirty pages: 144442 , remaining dirty pages: 146856
> Iteration 11, duration: 17714 ms , transferred pages: 144804 (dup: 18, rd: 25414, fd: 119372) , new dirty pages: 205127 , remaining dirty pages: 207179
> Iteration 12, duration: 25112 ms , transferred pages: 205446 (dup: 128, rd: 23197, fd: 182121) , new dirty pages: 167319 , remaining dirty pages: 169052
> Iteration 13, duration: 20411 ms , transferred pages: 166886 (dup: 14, rd: 21960, fd: 144912) , new dirty pages: 221592 , remaining dirty pages: 223758
> Iteration 14, duration: 27126 ms , transferred pages: 221800 (dup: 122, rd: 42368, fd: 179310) , new dirty pages: 233630 , remaining dirty pages: 235588
> Iteration 15, duration: 28517 ms , transferred pages: 233321 (dup: 191, rd: 222528, fd: 10602) , new dirty pages: 224282 , remaining dirty pages: 226549
> Iteration 16, duration: 27422 ms , transferred pages: 224187 (dup: 55, rd: 45773, fd: 178359) , new dirty pages: 209815 , remaining dirty pages: 212177
> Iteration 17, duration: 25723 ms , transferred pages: 210260 (dup: 34, rd: 79405, fd: 130821) , new dirty pages: 220297 , remaining dirty pages: 222214
> Iteration 18, duration: 26920 ms , transferred pages: 220056 (dup: 14, rd: 214128, fd: 5914) , new dirty pages: 192015 , remaining dirty pages: 194173
> Iteration 19, duration: 23520 ms , transferred pages: 192239 (dup: 9, rd: 25140, fd: 167090) , new dirty pages: 96450 , remaining dirty pages: 98384
> Iteration 20, duration: 11805 ms , transferred pages: 96538 (dup: 14, rd: 7424, fd: 89100) , new dirty pages: 6978 , remaining dirty pages: 8824
> Iteration 21, duration: 799 ms , transferred pages: 6545 (dup: 1, rd: 1802, fd: 4742) , new dirty pages: 138 , remaining dirty pages: 2417
> Iteration 22, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2417
>
> ----- after optimization: -----
> Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6831, rd: 259619) , new dirty pages: 240209 , remaining dirty pages: 240209
> Iteration 2, duration: 6250 ms , transferred pages: 51244 (dup: 211, rd: 51033) , new dirty pages: 226651 , remaining dirty pages: 228571
> Iteration 3, duration: 4395 ms , transferred pages: 36008 (dup: 80, rd: 35928) , new dirty pages: 110719 , remaining dirty pages: 111478
> Iteration 4, duration: 3390 ms , transferred pages: 28068 (dup: 28, rd: 28040) , new dirty pages: 185172 , remaining dirty pages: 185172
> Iteration 5, duration: 2986 ms , transferred pages: 23780 (dup: 45, rd: 23735) , new dirty pages: 64357 , remaining dirty pages: 66305
> Iteration 6, duration: 2727 ms , transferred pages: 22800 (dup: 12, rd: 22788) , new dirty pages: 61675 , remaining dirty pages: 61675
> Iteration 7, duration: 2372 ms , transferred pages: 18943 (dup: 13, rd: 18930) , new dirty pages: 55144 , remaining dirty pages: 55265
> Iteration 8, duration: 2100 ms , transferred pages: 17189 (dup: 11, rd: 17178) , new dirty pages: 55244 , remaining dirty pages: 55668
> Iteration 9, duration: 2003 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 107058 , remaining dirty pages: 108014
> Iteration 10, duration: 2132 ms , transferred pages: 17825 (dup: 24, rd: 17801) , new dirty pages: 126214 , remaining dirty pages: 126214
> Iteration 11, duration: 2229 ms , transferred pages: 18156 (dup: 22, rd: 18134) , new dirty pages: 65725 , remaining dirty pages: 65725
> Iteration 12, duration: 2315 ms , transferred pages: 18651 (dup: 21, rd: 18630) , new dirty pages: 52575 , remaining dirty pages: 53903
> Iteration 13, duration: 2147 ms , transferred pages: 17435 (dup: 16, rd: 17419) , new dirty pages: 46652 , remaining dirty pages: 47260
> Iteration 14, duration: 2000 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 42721 , remaining dirty pages: 43266
> Iteration 15, duration: 1901 ms , transferred pages: 15552 (dup: 10, rd: 15542) , new dirty pages: 38593 , remaining dirty pages: 40792
> Iteration 16, duration: 1801 ms , transferred pages: 14735 (dup: 11, rd: 14724) , new dirty pages: 54252 , remaining dirty pages: 55639
> Iteration 17, duration: 1708 ms , transferred pages: 13860 (dup: 2, rd: 13858) , new dirty pages: 72379 , remaining dirty pages: 74170
> Iteration 18, duration: 1923 ms , transferred pages: 15442 (dup: 12, rd: 15430) , new dirty pages: 101911 , remaining dirty pages: 103547
> Iteration 19, duration: 2311 ms , transferred pages: 18823 (dup: 9, rd: 18814) , new dirty pages: 80534 , remaining dirty pages: 82521
> Iteration 20, duration: 2081 ms , transferred pages: 17156 (dup: 34, rd: 17122) , new dirty pages: 36054 , remaining dirty pages: 36054
> Iteration 21, duration: 1665 ms , transferred pages: 13777 (dup: 10, rd: 13767) , new dirty pages: 29624 , remaining dirty pages: 29624
> Iteration 22, duration: 1657 ms , transferred pages: 13290 (dup: 7, rd: 13283) , new dirty pages: 25949 , remaining dirty pages: 28265
> Iteration 23, duration: 1599 ms , transferred pages: 13088 (dup: 0, rd: 13088) , new dirty pages: 22356 , remaining dirty pages: 24813
> Iteration 24, duration: 1500 ms , transferred pages: 12280 (dup: 10, rd: 12270) , new dirty pages: 21181 , remaining dirty pages: 22608
> Iteration 25, duration: 1400 ms , transferred pages: 11457 (dup: 5, rd: 11452) , new dirty pages: 18657 , remaining dirty pages: 20311
> Iteration 26, duration: 1200 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 15690 , remaining dirty pages: 17294
> Iteration 27, duration: 1201 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 14810 , remaining dirty pages: 15936
> Iteration 28, duration: 1000 ms , transferred pages: 8183 (dup: 3, rd: 8180) , new dirty pages: 15387 , remaining dirty pages: 16423
> Iteration 29, duration: 900 ms , transferred pages: 7372 (dup: 10, rd: 7362) , new dirty pages: 13303 , remaining dirty pages: 15292
> Iteration 30, duration: 1000 ms , transferred pages: 8181 (dup: 1, rd: 8180) , new dirty pages: 17879 , remaining dirty pages: 18457
> Iteration 31, duration: 951 ms , transferred pages: 8140 (dup: 9, rd: 8131) , new dirty pages: 21738 , remaining dirty pages: 23304
> Iteration 32, duration: 946 ms , transferred pages: 6946 (dup: 1, rd: 6945) , new dirty pages: 15815 , remaining dirty pages: 15815
> Iteration 33, duration: 747 ms , transferred pages: 6192 (dup: 0, rd: 6192) , new dirty pages: 6249 , remaining dirty pages: 7670
> Iteration 34, duration: 501 ms , transferred pages: 4090 (dup: 0, rd: 4090) , new dirty pages: 6163 , remaining dirty pages: 8422
> Iteration 35, duration: 600 ms , transferred pages: 4910 (dup: 2, rd: 4908) , new dirty pages: 3673 , remaining dirty pages: 5222
> Iteration 36, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2132 , remaining dirty pages: 4337
> Iteration 37, duration: 200 ms , transferred pages: 1637 (dup: 1, rd: 1636) , new dirty pages: 544 , remaining dirty pages: 2251
> Iteration 38, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 2251
> total time: 97919 milliseconds
>
> ------------------The other 11 workloads without notable improvements (only the result of original precopy is shown)-------------------
>
> 5. idle
>
> Iteration 1, duration: 14702 ms , transferred pages: 266450 (dup: 146393, rd: 120057, fd: 0) , new dirty pages: 14595 , remaining dirty pages: 14595
> Iteration 2, duration: 1592 ms , transferred pages: 12412 (dup: 103, rd: 3280, fd: 9029) , new dirty pages: 218 , remaining dirty pages: 2401
> Iteration 3, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2401
>
> 6. kernel compilation (can not converge)
>
> Iteration 1, duration: 20607 ms , transferred pages: 266450 (dup: 97552, rd: 168898, fd: 0) , new dirty pages: 19293 , remaining dirty pages: 19293
> Iteration 2, duration: 2092 ms , transferred pages: 17176 (dup: 597, rd: 8625, fd: 7954) , new dirty pages: 8318 , remaining dirty pages: 10435
> Iteration 3, duration: 1000 ms , transferred pages: 8484 (dup: 304, rd: 6256, fd: 1924) , new dirty pages: 8736 , remaining dirty pages: 10687
> Iteration 4, duration: 1000 ms , transferred pages: 8435 (dup: 255, rd: 7089, fd: 1091) , new dirty pages: 7627 , remaining dirty pages: 9879
> Iteration 5, duration: 900 ms , transferred pages: 7553 (dup: 191, rd: 5602, fd: 1760) , new dirty pages: 7287 , remaining dirty pages: 9613
> Iteration 6, duration: 900 ms , transferred pages: 7620 (dup: 258, rd: 5761, fd: 1601) , new dirty pages: 8958 , remaining dirty pages: 10951
> Iteration 7, duration: 1099 ms , transferred pages: 9309 (dup: 311, rd: 8051, fd: 947) , new dirty pages: 7189 , remaining dirty pages: 8831
> Iteration 8, duration: 800 ms , transferred pages: 6832 (dup: 288, rd: 5717, fd: 827) , new dirty pages: 5782 , remaining dirty pages: 7781
> Iteration 9, duration: 701 ms , transferred pages: 5875 (dup: 149, rd: 4005, fd: 1721) , new dirty pages: 4587 , remaining dirty pages: 6493
> Iteration 10, duration: 500 ms , transferred pages: 4234 (dup: 144, rd: 3057, fd: 1033) , new dirty pages: 7352 , remaining dirty pages: 9611
> Iteration 11, duration: 900 ms , transferred pages: 7759 (dup: 397, rd: 6563, fd: 799) , new dirty pages: 6686 , remaining dirty pages: 8538
> Iteration 12, duration: 800 ms , transferred pages: 6808 (dup: 264, rd: 6017, fd: 527) , new dirty pages: 6871 , remaining dirty pages: 8601
> Iteration 13, duration: 800 ms , transferred pages: 6775 (dup: 231, rd: 5722, fd: 822) , new dirty pages: 7540 , remaining dirty pages: 9366
> Iteration 14, duration: 900 ms , transferred pages: 7507 (dup: 145, rd: 5900, fd: 1462) , new dirty pages: 7581 , remaining dirty pages: 9440
> Iteration 15, duration: 900 ms , transferred pages: 7630 (dup: 268, rd: 6211, fd: 1151) , new dirty pages: 7268 , remaining dirty pages: 9078
> Iteration 16, duration: 800 ms , transferred pages: 6759 (dup: 215, rd: 5763, fd: 781) , new dirty pages: 6861 , remaining dirty pages: 9180
> Iteration 17, duration: 800 ms , transferred pages: 6838 (dup: 294, rd: 6037, fd: 507) , new dirty pages: 6196 , remaining dirty pages: 8538
> Iteration 18, duration: 800 ms , transferred pages: 6852 (dup: 308, rd: 4905, fd: 1639) , new dirty pages: 5947 , remaining dirty pages: 7633
> Iteration 19, duration: 700 ms , transferred pages: 5919 (dup: 193, rd: 4853, fd: 873) , new dirty pages: 5861 , remaining dirty pages: 7575
> Iteration 20, duration: 600 ms , transferred pages: 5284 (dup: 376, rd: 4408, fd: 500) , new dirty pages: 5206 , remaining dirty pages: 7497
> Iteration 21, duration: 600 ms , transferred pages: 5147 (dup: 239, rd: 4308, fd: 600) , new dirty pages: 5031 , remaining dirty pages: 7381
> Iteration 22, duration: 599 ms , transferred pages: 5064 (dup: 156, rd: 4026, fd: 882) , new dirty pages: 5601 , remaining dirty pages: 7918
> Iteration 23, duration: 702 ms , transferred pages: 5965 (dup: 239, rd: 5028, fd: 698) , new dirty pages: 6079 , remaining dirty pages: 8032
> Iteration 24, duration: 700 ms , transferred pages: 6175 (dup: 449, rd: 5146, fd: 580) , new dirty pages: 10932 , remaining dirty pages: 12789
> Iteration 25, duration: 1300 ms , transferred pages: 10936 (dup: 302, rd: 6205, fd: 4429) , new dirty pages: 8713 , remaining dirty pages: 10566
> Iteration 26, duration: 1000 ms , transferred pages: 8282 (dup: 102, rd: 5662, fd: 2518) , new dirty pages: 5119 , remaining dirty pages: 7403
> Iteration 27, duration: 600 ms , transferred pages: 5007 (dup: 99, rd: 4099, fd: 809) , new dirty pages: 2226 , remaining dirty pages: 4622
> Iteration 28, duration: 300 ms , transferred pages: 2491 (dup: 37, rd: 1794, fd: 660) , new dirty pages: 6746 , remaining dirty pages: 8877
> Iteration 29, duration: 800 ms , transferred pages: 6757 (dup: 213, rd: 5532, fd: 1012) , new dirty pages: 6070 , remaining dirty pages: 8190
> Iteration 30, duration: 700 ms , transferred pages: 6052 (dup: 326, rd: 5107, fd: 619) , new dirty pages: 5177 , remaining dirty pages: 7315
>
> 7. web server
>
> Iteration 1, duration: 20902 ms , transferred pages: 266450 (dup: 95497, rd: 170953, fd: 0) , new dirty pages: 8528 , remaining dirty pages: 8528
> Iteration 2, duration: 796 ms , transferred pages: 6472 (dup: 131, rd: 1885, fd: 4456) , new dirty pages: 650 , remaining dirty pages: 2706
> Iteration 3, duration: 100 ms , transferred pages: 818 (dup: 0, rd: 383, fd: 435) , new dirty pages: 328 , remaining dirty pages: 2216
> Iteration 4, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2216
>
>
> 8. cpu2006.bwaves (can not converge)
>
> Iteration 1, duration: 31715 ms , transferred pages: 266450 (dup: 6766, rd: 259684, fd: 0) , new dirty pages: 242702 , remaining dirty pages: 242702
> Iteration 2, duration: 29397 ms , transferred pages: 240508 (dup: 405, rd: 225588, fd: 14515) , new dirty pages: 230889 , remaining dirty pages: 233083
> Iteration 3, duration: 28205 ms , transferred pages: 230858 (dup: 182, rd: 214596, fd: 16080) , new dirty pages: 226998 , remaining dirty pages: 229223
> Iteration 4, duration: 27805 ms , transferred pages: 227574 (dup: 170, rd: 217045, fd: 10359) , new dirty pages: 227360 , remaining dirty pages: 229009
> Iteration 5, duration: 27703 ms , transferred pages: 226786 (dup: 200, rd: 212130, fd: 14456) , new dirty pages: 225885 , remaining dirty pages: 228108
> Iteration 6, duration: 27600 ms , transferred pages: 225923 (dup: 155, rd: 215503, fd: 10265) , new dirty pages: 223555 , remaining dirty pages: 225740
> Iteration 7, duration: 27309 ms , transferred pages: 223574 (dup: 260, rd: 215641, fd: 7673) , new dirty pages: 231975 , remaining dirty pages: 234141
> Iteration 8, duration: 28403 ms , transferred pages: 232397 (dup: 85, rd: 214086, fd: 18226) , new dirty pages: 222170 , remaining dirty pages: 223914
> Iteration 9, duration: 27105 ms , transferred pages: 221809 (dup: 131, rd: 214988, fd: 6690) , new dirty pages: 230065 , remaining dirty pages: 232170
> Iteration 10, duration: 28104 ms , transferred pages: 230201 (dup: 343, rd: 213531, fd: 16327) , new dirty pages: 227590 , remaining dirty pages: 229559
> Iteration 11, duration: 27801 ms , transferred pages: 227717 (dup: 313, rd: 221408, fd: 5996) , new dirty pages: 228457 , remaining dirty pages: 230299
> Iteration 12, duration: 27916 ms , transferred pages: 228560 (dup: 338, rd: 219660, fd: 8562) , new dirty pages: 238326 , remaining dirty pages: 240065
>
> 9. cpu2006.lbm (can not converge)
> Iteration 1, duration: 31012 ms , transferred pages: 266450 (dup: 12253, rd: 254197, fd: 0) , new dirty pages: 108960 , remaining dirty pages: 108960
> Iteration 2, duration: 13095 ms , transferred pages: 106522 (dup: 3, rd: 102045, fd: 4474) , new dirty pages: 129292 , remaining dirty pages: 131730
> Iteration 3, duration: 15802 ms , transferred pages: 129688 (dup: 444, rd: 110860, fd: 18384) , new dirty pages: 116682 , remaining dirty pages: 118724
> Iteration 4, duration: 14204 ms , transferred pages: 116316 (dup: 160, rd: 104951, fd: 11205) , new dirty pages: 107246 , remaining dirty pages: 109654
> Iteration 5, duration: 13208 ms , transferred pages: 107977 (dup: 1, rd: 101834, fd: 6142) , new dirty pages: 105371 , remaining dirty pages: 107048
> Iteration 6, duration: 12804 ms , transferred pages: 104705 (dup: 1, rd: 99629, fd: 5075) , new dirty pages: 103841 , remaining dirty pages: 106184
> Iteration 7, duration: 12709 ms , transferred pages: 103891 (dup: 5, rd: 99212, fd: 4674) , new dirty pages: 106692 , remaining dirty pages: 108985
> Iteration 8, duration: 13105 ms , transferred pages: 107169 (dup: 11, rd: 100125, fd: 7033) , new dirty pages: 103132 , remaining dirty pages: 104948
> Iteration 9, duration: 12607 ms , transferred pages: 103068 (dup: 0, rd: 99460, fd: 3608) , new dirty pages: 102511 , remaining dirty pages: 104391
> Iteration 10, duration: 12514 ms , transferred pages: 102250 (dup: 0, rd: 99094, fd: 3156) , new dirty pages: 102888 , remaining dirty pages: 105029
>
> 10. cpu2006.astar (can not converge)
>
> Iteration 1, duration: 28402 ms , transferred pages: 266450 (dup: 33770, rd: 232680, fd: 0) , new dirty pages: 62078 , remaining dirty pages: 62078
> Iteration 2, duration: 7393 ms , transferred pages: 60107 (dup: 10, rd: 51722, fd: 8375) , new dirty pages: 48854 , remaining dirty pages: 50825
> Iteration 3, duration: 6001 ms , transferred pages: 49094 (dup: 14, rd: 46540, fd: 2540) , new dirty pages: 48137 , remaining dirty pages: 49868
> Iteration 4, duration: 5800 ms , transferred pages: 47444 (dup: 0, rd: 45389, fd: 2055) , new dirty pages: 49147 , remaining dirty pages: 51571
> Iteration 5, duration: 6102 ms , transferred pages: 49912 (dup: 14, rd: 46216, fd: 3682) , new dirty pages: 55606 , remaining dirty pages: 57265
> Iteration 6, duration: 6699 ms , transferred pages: 54949 (dup: 143, rd: 20745, fd: 34061) , new dirty pages: 9166 , remaining dirty pages: 11482
> Iteration 7, duration: 1200 ms , transferred pages: 9830 (dup: 14, rd: 7011, fd: 2805) , new dirty pages: 8294 , remaining dirty pages: 9946
> Iteration 8, duration: 1000 ms , transferred pages: 8194 (dup: 14, rd: 7178, fd: 1002) , new dirty pages: 5475 , remaining dirty pages: 7227
> Iteration 9, duration: 600 ms , transferred pages: 4908 (dup: 0, rd: 3470, fd: 1438) , new dirty pages: 4175 , remaining dirty pages: 6494
> Iteration 10, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3856, fd: 234) , new dirty pages: 4095 , remaining dirty pages: 6499
> Iteration 11, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3313, fd: 777) , new dirty pages: 3371 , remaining dirty pages: 5780
> Iteration 12, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3823, fd: 267) , new dirty pages: 7518 , remaining dirty pages: 9208
> Iteration 13, duration: 899 ms , transferred pages: 7376 (dup: 14, rd: 6028, fd: 1334) , new dirty pages: 3931 , remaining dirty pages: 5763
> Iteration 14, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 4346 , remaining dirty pages: 6019
> Iteration 15, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3817, fd: 273) , new dirty pages: 3054 , remaining dirty pages: 4983
> Iteration 16, duration: 400 ms , transferred pages: 3272 (dup: 0, rd: 3138, fd: 134) , new dirty pages: 3874 , remaining dirty pages: 5585
> Iteration 17, duration: 399 ms , transferred pages: 3272 (dup: 0, rd: 3248, fd: 24) , new dirty pages: 5285 , remaining dirty pages: 7598
> Iteration 18, duration: 701 ms , transferred pages: 5726 (dup: 0, rd: 4385, fd: 1341) , new dirty pages: 8903 , remaining dirty pages: 10775
> Iteration 19, duration: 1101 ms , transferred pages: 9010 (dup: 12, rd: 5597, fd: 3401) , new dirty pages: 4199 , remaining dirty pages: 5964
> Iteration 20, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 3829 , remaining dirty pages: 5703
>
> 11. cpu2006.xalancbmk (can not converge)
>
> Iteration 1, duration: 30407 ms , transferred pages: 266450 (dup: 17700, rd: 248750, fd: 0) , new dirty pages: 96169 , remaining dirty pages: 96169
> Iteration 2, duration: 11495 ms , transferred pages: 94164 (dup: 205, rd: 67068, fd: 26891) , new dirty pages: 61766 , remaining dirty pages: 63771
> Iteration 3, duration: 7501 ms , transferred pages: 61471 (dup: 121, rd: 53587, fd: 7763) , new dirty pages: 56569 , remaining dirty pages: 58869
> Iteration 4, duration: 6902 ms , transferred pages: 56461 (dup: 19, rd: 50553, fd: 5889) , new dirty pages: 52181 , remaining dirty pages: 54589
> Iteration 5, duration: 6402 ms , transferred pages: 52459 (dup: 107, rd: 46986, fd: 5366) , new dirty pages: 54051 , remaining dirty pages: 56181
> Iteration 6, duration: 6601 ms , transferred pages: 54003 (dup: 15, rd: 47566, fd: 6422) , new dirty pages: 50844 , remaining dirty pages: 53022
> Iteration 7, duration: 6202 ms , transferred pages: 50723 (dup: 7, rd: 47143, fd: 3573) , new dirty pages: 64880 , remaining dirty pages: 67179
> Iteration 8, duration: 8001 ms , transferred pages: 65447 (dup: 7, rd: 61159, fd: 4281) , new dirty pages: 67854 , remaining dirty pages: 69586
> Iteration 9, duration: 8202 ms , transferred pages: 67444 (dup: 368, rd: 56357, fd: 10719) , new dirty pages: 65178 , remaining dirty pages: 67320
> Iteration 10, duration: 8000 ms , transferred pages: 65455 (dup: 15, rd: 60581, fd: 4859) , new dirty pages: 52421 , remaining dirty pages: 54286
>
> 12. cpu2006.milc (can not converge)
>
> Iteration 1, duration: 31410 ms , transferred pages: 266450 (dup: 9454, rd: 256996, fd: 0) , new dirty pages: 158860 , remaining dirty pages: 158860
> Iteration 2, duration: 19193 ms , transferred pages: 157048 (dup: 150, rd: 96807, fd: 60091) , new dirty pages: 102238 , remaining dirty pages: 104050
> Iteration 3, duration: 12504 ms , transferred pages: 102271 (dup: 21, rd: 95107, fd: 7143) , new dirty pages: 97944 , remaining dirty pages: 99723
> Iteration 4, duration: 11905 ms , transferred pages: 97360 (dup: 18, rd: 93610, fd: 3732) , new dirty pages: 99150 , remaining dirty pages: 101513
> Iteration 5, duration: 12105 ms , transferred pages: 99094 (dup: 116, rd: 94125, fd: 4853) , new dirty pages: 98589 , remaining dirty pages: 101008
> Iteration 6, duration: 12101 ms , transferred pages: 98995 (dup: 17, rd: 94069, fd: 4909) , new dirty pages: 147403 , remaining dirty pages: 149416
> Iteration 7, duration: 18001 ms , transferred pages: 147284 (dup: 44, rd: 135691, fd: 11549) , new dirty pages: 136445 , remaining dirty pages: 138577
> Iteration 8, duration: 16702 ms , transferred pages: 136636 (dup: 30, rd: 130805, fd: 5801) , new dirty pages: 145481 , remaining dirty pages: 147422
> Iteration 9, duration: 17800 ms , transferred pages: 145734 (dup: 130, rd: 133239, fd: 12365) , new dirty pages: 98032 , remaining dirty pages: 99720
> Iteration 10, duration: 11902 ms , transferred pages: 97364 (dup: 22, rd: 93096, fd: 4246) , new dirty pages: 95391 , remaining dirty pages: 97747
>
> 13. cpu2006.cactusADM (can not converge)
>
> Iteration 1, duration: 23508 ms , transferred pages: 266450 (dup: 73568, rd: 192882, fd: 0) , new dirty pages: 123869 , remaining dirty pages: 123869
> Iteration 2, duration: 13989 ms , transferred pages: 121594 (dup: 7874, rd: 81653, fd: 32067) , new dirty pages: 112960 , remaining dirty pages: 115235
> Iteration 3, duration: 13605 ms , transferred pages: 113276 (dup: 2028, rd: 83783, fd: 27465) , new dirty pages: 112314 , remaining dirty pages: 114273
> Iteration 4, duration: 13509 ms , transferred pages: 111935 (dup: 1505, rd: 83535, fd: 26895) , new dirty pages: 114078 , remaining dirty pages: 116416
> Iteration 5, duration: 13810 ms , transferred pages: 114262 (dup: 1378, rd: 84039, fd: 28845) , new dirty pages: 112271 , remaining dirty pages: 114425
> Iteration 6, duration: 13604 ms , transferred pages: 112664 (dup: 1416, rd: 84300, fd: 26948) , new dirty pages: 112903 , remaining dirty pages: 114664
> Iteration 7, duration: 13604 ms , transferred pages: 112655 (dup: 1407, rd: 84027, fd: 27221) , new dirty pages: 110943 , remaining dirty pages: 112952
> Iteration 8, duration: 13406 ms , transferred pages: 110720 (dup: 1108, rd: 84075, fd: 25537) , new dirty pages: 109321 , remaining dirty pages: 111553
> Iteration 9, duration: 13306 ms , transferred pages: 109726 (dup: 932, rd: 83652, fd: 25142) , new dirty pages: 113446 , remaining dirty pages: 115273
> Iteration 10, duration: 13705 ms , transferred pages: 113121 (dup: 1055, rd: 84671, fd: 27395) , new dirty pages: 108776 , remaining dirty pages: 110928
>
> 14. cpu2006.GmesFDTD (can not converge)
>
> Iteration 1, duration: 13303 ms , transferred pages: 266450 (dup: 157809, rd: 108641, fd: 0) , new dirty pages: 226802 , remaining dirty pages: 226802
> Iteration 2, duration: 10797 ms , transferred pages: 226507 (dup: 138637, rd: 61818, fd: 26052) , new dirty pages: 200769 , remaining dirty pages: 201064
> Iteration 3, duration: 8900 ms , transferred pages: 199717 (dup: 127187, rd: 69340, fd: 3190) , new dirty pages: 203436 , remaining dirty pages: 204783
> Iteration 4, duration: 10904 ms , transferred pages: 204127 (dup: 115211, rd: 85767, fd: 3149) , new dirty pages: 198407 , remaining dirty pages: 199063
> Iteration 5, duration: 12109 ms , transferred pages: 198206 (dup: 99435, rd: 96956, fd: 1815) , new dirty pages: 213719 , remaining dirty pages: 214576
> Iteration 6, duration: 16307 ms , transferred pages: 213595 (dup: 80422, rd: 116885, fd: 16288) , new dirty pages: 199637 , remaining dirty pages: 200618
> Iteration 7, duration: 16915 ms , transferred pages: 198289 (dup: 60169, rd: 134208, fd: 3912) , new dirty pages: 199343 , remaining dirty pages: 201672
> Iteration 8, duration: 19518 ms , transferred pages: 200452 (dup: 41014, rd: 156083, fd: 3355) , new dirty pages: 222927 , remaining dirty pages: 224147
>
> 15. cpu2006.wrf (can not converge)
>
> Iteration 1, duration: 18499 ms , transferred pages: 266380 (dup: 115285, rd: 151095, fd: 0) , new dirty pages: 112322 , remaining dirty pages: 112392
> Iteration 2, duration: 9802 ms , transferred pages: 110025 (dup: 29917, rd: 65782, fd: 14326) , new dirty pages: 88855 , remaining dirty pages: 91222
> Iteration 3, duration: 8199 ms , transferred pages: 89761 (dup: 22728, rd: 57262, fd: 9771) , new dirty pages: 58431 , remaining dirty pages: 59892
> Iteration 4, duration: 5603 ms , transferred pages: 58502 (dup: 12716, rd: 41809, fd: 3977) , new dirty pages: 80556 , remaining dirty pages: 81946
> Iteration 5, duration: 7101 ms , transferred pages: 79778 (dup: 21738, rd: 50896, fd: 7144) , new dirty pages: 62592 , remaining dirty pages: 64760
> Iteration 6, duration: 5702 ms , transferred pages: 63388 (dup: 16793, rd: 42726, fd: 3869) , new dirty pages: 80747 , remaining dirty pages: 82119
> Iteration 7, duration: 7000 ms , transferred pages: 80868 (dup: 23652, rd: 52194, fd: 5022) , new dirty pages: 84593 , remaining dirty pages: 85844
> Iteration 8, duration: 7099 ms , transferred pages: 83799 (dup: 25769, rd: 51772, fd: 6258) , new dirty pages: 67951 , remaining dirty pages: 69996
> Iteration 9, duration: 6303 ms , transferred pages: 68478 (dup: 16979, rd: 36490, fd: 15009) , new dirty pages: 81181 , remaining dirty pages: 82699
> Iteration 10, duration: 7000 ms , transferred pages: 80724 (dup: 23503, rd: 52826, fd: 4395) , new dirty pages: 47930 , remaining dirty pages: 49905
>
>
>
> >
> > > So I think "booting" and "kernel compilation" should benefit a lot from this
> > > improvement. The reason of "kernel compilation" would benefit is that some
> > > iterations take around 600ms, and if they are halved into 300ms, then the precopy
> > > may have the chance to step into stop and copy phase.
> > >
> > > On the other hand, "idle" and "web server" would not benefit a lot, because
> > > most of the time are spent on the 1st iteration and little on the others.
> > >
> > > As to the "zeusmp" and "memcached", although the time spent on the other iterations
> > > but the 1st one may be halved, they still could not converge to stop and copy
> > > with the 300ms downtime.
> > >
> > > --------------------1 vcpu, 1 GB ram, default bandwidth (32MB/s):------------------
> > >
> > > 1. booting : begin to migrate when the VM is booting
> > >
> > > Iteration 1, duration: 6997 ms , transferred pages: 266450 (n: 57269, d: 209181 ) , new dirty pages: 56414 , remaining dirty pages: 56414
> > > Iteration 2, duration: 6497 ms , transferred pages: 54008 (n: 52701, d: 1307 ) , new dirty pages: 48053 , remaining dirty pages: 50459
> > > Iteration 3, duration: 5800 ms , transferred pages: 48232 (n: 47444, d: 788 ) , new dirty pages: 9129 , remaining dirty pages: 11356
> > > Iteration 4, duration: 1100 ms , transferred pages: 9091 (n: 8998, d: 93 ) , new dirty pages: 165 , remaining dirty pages: 2430
> > > Iteration 5, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2430
> > > (note: When the workload does converge, the output of the last iteration is "fake". It just indicates that the precopy steps into stop-copy phase now.
> > > "n" means "normal pages" and "d" means "duplicate (zero) pages".)
> > >
> > > 2. idle
> > >
> > > Iteration 1, duration: 14496 ms , transferred pages: 266450 (n: 118980, d: 147470 ) , new dirty pages: 17398 , remaining dirty pages: 17398
> > > Iteration 2, duration: 1896 ms , transferred pages: 14953 (n: 14854, d: 99 ) , new dirty pages: 1849 , remaining dirty pages: 4294
> > > Iteration 3, duration: 300 ms , transferred pages: 2454 (n: 2454, d: 0 ) , new dirty pages: 9 , remaining dirty pages: 1849
> > > Iteration 4, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 1849
> > >
> > > 3. kernel compilation (can not converge)
> > >
> > > Iteration 1, duration: 20700 ms , transferred pages: 266450 (n: 169778, d: 96672 ) , new dirty pages: 40067 , remaining dirty pages: 40067
> > > Iteration 2, duration: 4696 ms , transferred pages: 38401 (n: 37787, d: 614 ) , new dirty pages: 8852 , remaining dirty pages: 10518
> > > Iteration 3, duration: 1000 ms , transferred pages: 8642 (n: 8180, d: 462 ) , new dirty pages: 6331 , remaining dirty pages: 8207
> > > Iteration 4, duration: 700 ms , transferred pages: 6110 (n: 5726, d: 384 ) , new dirty pages: 5242 , remaining dirty pages: 7339
> > > Iteration 5, duration: 600 ms , transferred pages: 5007 (n: 4908, d: 99 ) , new dirty pages: 4868 , remaining dirty pages: 7200
> > > Iteration 6, duration: 600 ms , transferred pages: 5226 (n: 4908, d: 318 ) , new dirty pages: 6142 , remaining dirty pages: 8116
> > > Iteration 7, duration: 700 ms , transferred pages: 5985 (n: 5726, d: 259 ) , new dirty pages: 5902 , remaining dirty pages: 8033
> > > Iteration 8, duration: 701 ms , transferred pages: 5893 (n: 5726, d: 167 ) , new dirty pages: 7502 , remaining dirty pages: 9642
> > > Iteration 9, duration: 900 ms , transferred pages: 7623 (n: 7362, d: 261 ) , new dirty pages: 6408 , remaining dirty pages: 8427
> > > Iteration 10, duration: 700 ms , transferred pages: 6008 (n: 5726, d: 282 ) , new dirty pages: 8312 , remaining dirty pages: 10731
> > > Iteration 11, duration: 1000 ms , transferred pages: 8353 (n: 8180, d: 173 ) , new dirty pages: 6874 , remaining dirty pages: 9252
> > > Iteration 12, duration: 899 ms , transferred pages: 7477 (n: 7362, d: 115 ) , new dirty pages: 5573 , remaining dirty pages: 7348
> > > Iteration 13, duration: 601 ms , transferred pages: 5099 (n: 4908, d: 191 ) , new dirty pages: 7671 , remaining dirty pages: 9920
> > > Iteration 14, duration: 900 ms , transferred pages: 7586 (n: 7362, d: 224 ) , new dirty pages: 7359 , remaining dirty pages: 9693
> > > Iteration 15, duration: 900 ms , transferred pages: 7682 (n: 7362, d: 320 ) , new dirty pages: 7371 , remaining dirty pages: 9382
> > >
> > > 4. cpu2006.zeusmp (can not converge)
> > >
> > > Iteration 1, duration: 21603 ms , transferred pages: 266450 (n: 176660, d: 89790 ) , new dirty pages: 145625 , remaining dirty pages: 145625
> > > Iteration 2, duration: 8696 ms , transferred pages: 144389 (n: 70862, d: 73527 ) , new dirty pages: 125124 , remaining dirty pages: 126360
> > > Iteration 3, duration: 6301 ms , transferred pages: 124057 (n: 51379, d: 72678 ) , new dirty pages: 122528 , remaining dirty pages: 124831
> > > Iteration 4, duration: 6400 ms , transferred pages: 124330 (n: 52196, d: 72134 ) , new dirty pages: 124267 , remaining dirty pages: 124768
> > > Iteration 5, duration: 6703 ms , transferred pages: 124034 (n: 54656, d: 69378 ) , new dirty pages: 124151 , remaining dirty pages: 124885
> > > Iteration 6, duration: 6703 ms , transferred pages: 124357 (n: 54658, d: 69699 ) , new dirty pages: 124106 , remaining dirty pages: 124634
> > > Iteration 7, duration: 6602 ms , transferred pages: 124568 (n: 53838, d: 70730 ) , new dirty pages: 133828 , remaining dirty pages: 133894
> > > Iteration 8, duration: 7600 ms , transferred pages: 133030 (n: 62021, d: 71009 ) , new dirty pages: 126612 , remaining dirty pages: 127476
> > > Iteration 9, duration: 7299 ms , transferred pages: 126511 (n: 59569, d: 66942 ) , new dirty pages: 122727 , remaining dirty pages: 123692
> > > Iteration 10, duration: 6609 ms , transferred pages: 123692 (n: 54539, d: 69153 ) , new dirty pages: 122727 , remaining dirty pages: 122727
> > > Iteration 11, duration: 6995 ms , transferred pages: 120347 (n: 56423, d: 63924 ) , new dirty pages: 121430 , remaining dirty pages: 123810
> > > Iteration 12, duration: 6703 ms , transferred pages: 123040 (n: 54657, d: 68383 ) , new dirty pages: 122043 , remaining dirty pages: 122813
> > > Iteration 13, duration: 7006 ms , transferred pages: 122353 (n: 57121, d: 65232 ) , new dirty pages: 133869 , remaining dirty pages: 134329
> > > Iteration 14, duration: 8209 ms , transferred pages: 132325 (n: 66932, d: 65393 ) , new dirty pages: 126914 , remaining dirty pages: 128918
> > > Iteration 15, duration: 7802 ms , transferred pages: 126931 (n: 63671, d: 63260 ) , new dirty pages: 122351 , remaining dirty pages: 124338
> > >
> > > 5. web server : An apache web server. The client is configured with 50 concurrent connections.
> > >
> > > Iteration 1, duration: 30697 ms , transferred pages: 266450 (n: 251215, d: 15235 ) , new dirty pages: 30628 , remaining dirty pages: 30628
> > > Iteration 2, duration: 3496 ms , transferred pages: 28859 (n: 28513, d: 346 ) , new dirty pages: 5805 , remaining dirty pages: 7574
> > > Iteration 3, duration: 701 ms , transferred pages: 5746 (n: 5726, d: 20 ) , new dirty pages: 3433 , remaining dirty pages: 5261
> > > Iteration 4, duration: 400 ms , transferred pages: 3281 (n: 3272, d: 9 ) , new dirty pages: 1539 , remaining dirty pages: 3519
> > > Iteration 5, duration: 199 ms , transferred pages: 1653 (n: 1636, d: 17 ) , new dirty pages: 301 , remaining dirty pages: 2167
> > > Iteration 6, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2167
> > >
> > > --------------------6 vcpu, 6 GB ram, max bandwidth (941.08 mbps):------------------
> > >
> > > 6. memcached : 4 GB cache, memaslap: all write, concurrency = 5 (can not converge)
> > >
> > > Iteration 1, duration: 42486 ms , transferred pages: 1568087 (n: 1216079, d: 352008 ) , new dirty pages: 571940 , remaining dirty pages: 581023
> > > Iteration 2, duration: 19774 ms , transferred pages: 571700 (n: 567416, d: 4284 ) , new dirty pages: 331690 , remaining dirty pages: 341013
> > > Iteration 3, duration: 11589 ms , transferred pages: 332187 (n: 332095, d: 92 ) , new dirty pages: 222725 , remaining dirty pages: 231551
> > > Iteration 4, duration: 7790 ms , transferred pages: 223571 (n: 223499, d: 72 ) , new dirty pages: 157658 , remaining dirty pages: 165638
> > > Iteration 5, duration: 5518 ms , transferred pages: 158056 (n: 157998, d: 58 ) , new dirty pages: 128130 , remaining dirty pages: 135712
> > > Iteration 6, duration: 4442 ms , transferred pages: 127764 (n: 127701, d: 63 ) , new dirty pages: 104839 , remaining dirty pages: 112787
> > > Iteration 7, duration: 3649 ms , transferred pages: 104581 (n: 104523, d: 58 ) , new dirty pages: 100736 , remaining dirty pages: 108942
> > > Iteration 8, duration: 3532 ms , transferred pages: 101379 (n: 101315, d: 64 ) , new dirty pages: 87869 , remaining dirty pages: 95432
> > > Iteration 9, duration: 3030 ms , transferred pages: 86841 (n: 86786, d: 55 ) , new dirty pages: 77505 , remaining dirty pages: 86096
> > > Iteration 10, duration: 2709 ms , transferred pages: 77875 (n: 77814, d: 61 ) , new dirty pages: 77197 , remaining dirty pages: 85418
> > > Iteration 11, duration: 2696 ms , transferred pages: 77107 (n: 77044, d: 63 ) , new dirty pages: 65010 , remaining dirty pages: 73321
> > > Iteration 12, duration: 2308 ms , transferred pages: 66540 (n: 66484, d: 56 ) , new dirty pages: 64388 , remaining dirty pages: 71169
> > > Iteration 13, duration: 2198 ms , transferred pages: 62953 (n: 62897, d: 56 ) , new dirty pages: 62773 , remaining dirty pages: 70989
> > > Iteration 14, duration: 2214 ms , transferred pages: 63466 (n: 63411, d: 55 ) , new dirty pages: 67538 , remaining dirty pages: 75061
> > > Iteration 15, duration: 2329 ms , transferred pages: 66924 (n: 66875, d: 49 ) , new dirty pages: 63580 , remaining dirty pages: 71717
> > > Iteration 16, duration: 2252 ms , transferred pages: 64554 (n: 64539, d: 15 ) , new dirty pages: 63094 , remaining dirty pages: 70257
> > > Iteration 17, duration: 2188 ms , transferred pages: 62697 (n: 62641, d: 56 ) , new dirty pages: 63016 , remaining dirty pages: 70576
> > > Iteration 18, duration: 2171 ms , transferred pages: 62377 (n: 62322, d: 55 ) , new dirty pages: 56764 , remaining dirty pages: 64963
> > > Iteration 19, duration: 2003 ms , transferred pages: 57382 (n: 57324, d: 58 ) , new dirty pages: 65307 , remaining dirty pages: 72888
> > > Iteration 20, duration: 2240 ms , transferred pages: 64426 (n: 64364, d: 62 ) , new dirty pages: 61585 , remaining dirty pages: 70047
> > >
> > >
> > > --
> > > Chunguang Li, Ph.D. Candidate
> > > Wuhan National Laboratory for Optoelectronics (WNLO)
> > > Huazhong University of Science & Technology (HUST)
> > > Wuhan, Hubei Prov., China
> > >
> > >
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> --
> Chunguang Li, Ph.D. Candidate
> Wuhan National Laboratory for Optoelectronics (WNLO)
> Huazhong University of Science & Technology (HUST)
> Wuhan, Hubei Prov., China
>
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
2016-11-08 11:05 ` Dr. David Alan Gilbert
@ 2016-11-08 13:40 ` Chunguang Li
0 siblings, 0 replies; 21+ messages in thread
From: Chunguang Li @ 2016-11-08 13:40 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Amit Shah, qemu-devel, pbonzini, stefanha, quintela
> -----Original Messages-----
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Sent Time: Tuesday, November 8, 2016
> To: "Chunguang Li" <lichunguang@hust.edu.cn>
> Cc: "Amit Shah" <amit.shah@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
>
> * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> >
> >
> >
> > > -----Original Messages-----
> > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > Sent Time: Friday, October 14, 2016
> > > To: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > Cc: "Amit Shah" <amit.shah@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > >
> > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > >
> > > >
> > > >
> > > > > -----原始邮件-----
> > > > > 发件人: "Amit Shah" <amit.shah@redhat.com>
> > > > > 发送时间: 2016年9月30日 星期五
> > > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > > 抄送: "Dr. David Alan Gilbert" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > > 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > > >
> > > > > On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > > -----原始邮件-----
> > > > > > > 发件人: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > > > > 发送时间: 2016年9月26日 星期一
> > > > > > > 收件人: "Chunguang Li" <lichunguang@hust.edu.cn>
> > > > > > > 抄送: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com
> > > > > > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent
> > > > > > >
> > > > > > > * Chunguang Li (lichunguang@hust.edu.cn) wrote:
> > > > > > > > Hi all!
> > > > > > > > I have some confusion about the dirty bitmap during migration. I have digged into the code. I figure out that every now and then during migration, the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I think this mechanism leads to resendness of some NON-dirty pages.
> > > > > > > >
> > > > > > > > Take the first iteration of precopy for instance, during which all the pages will be sent. Before that during the migration setup, the ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bitmap from this moment. When the pages "that haven't been sent" are written, the kernel space marks them as dirty. However I don't think this is correct, because these pages will be sent during this and the next iterations with the same content (if they are not written again after they are sent). It only makes sense to mark the pages which have already been sent during one iteration as dirty when they are written.
> > > > > > > >
> > > > > > > >
> > > > > > > > Am I right about this consideration? If I am right, is there some advice to improve this?
> > > > > > >
> > > > > > > I think you're right that this can happen; to clarify I think the
> > > > > > > case you're talking about is:
> > > > > > >
> > > > > > > Iteration 1
> > > > > > > sync bitmap
> > > > > > > start sending pages
> > > > > > > page 'n' is modified - but hasn't been sent yet
> > > > > > > page 'n' gets sent
> > > > > > > Iteration 2
> > > > > > > sync bitmap
> > > > > > > 'page n is shown as modified'
> > > > > > > send page 'n' again
> > > > > > >
> > > > > >
> > > > > > Yes，this is right the case I am talking about.
> > > > > >
> > > > > > > So you're right that is wasteful; I guess it's more wasteful
> > > > > > > on big VMs with slow networks where the length of each iteration
> > > > > > > is large.
> > > > > >
> > > > > > I think this is "very" wasteful. Assume the workload writes the pages dirty randomly within the guest address space, and the transfer speed is constant. Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is not really dirty. This means the time of Iteration 2 is double of that to send only really dirty pages.
> > > > >
> > > > > It makes sense, can you get some perf numbers to show what kinds of
> > > > > workloads get impacted the most? That would also help us to figure
> > > > > out what kinds of speed improvements we can expect.
> > > > >
> > > > >
> > > > > Amit
> > > >
> > > > I have picked up 6 workloads and got the following statistics numbers
> > > > of every iteration (except the last stop-copy one) during precopy.
> > > > These numbers are obtained with the basic precopy migration, without
> > > > the capabilities like xbzrle or compression, etc. The network for the
> > > > migration is exclusive, with a separate network for the workloads.
> > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > >
> > > > Three (booting, idle, web server) of them converged to the stop-copy phase,
> > > > with the given bandwidth and default downtime (300ms), while the other
> > > > three (kernel compilation, zeusmp, memcached) did not.
> > > >
> > > > One page is "not-really-dirty", if it is written first and is sent later
> > > > (and not written again after that) during one iteration. I guess this
> > > > would not happen so often during the other iterations as during the 1st
> > > > iteration. Because all the pages of the VM are sent to the dest node during
> > > > the 1st iteration, while during the others, only part of the pages are sent.
> > > > So I think the "not-really-dirty" pages should be produced mainly during
> > > > the 1st iteration , and maybe very little during the other iterations.
> > > >
> > > > If we could avoid resending the "not-really-dirty" pages, intuitively, I
> > > > think the time spent on Iteration 2 would be halved. This is a chain reaction,
> > > > because the dirty pages produced during Iteration 2 is halved, which incurs
> > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > >
> > > Yes; these numbers don't show how many of them are false dirty though.
> > >
> > > One problem is thinking about pages that have been redirtied, if the page is dirtied
> > > after the sync but before the network write then it's the false-dirty that
> > > you're describing.
> > >
> > > However, if the page is being written a few times, and so it would have been written
> > > after the network write then it isn't a false-dirty.
> > >
> > > You might be able to figure that out with some kernel tracing of when the dirtying
> > > happens, but it might be easier to write the fix!
> > >
> > > Dave
> >
> > Hi, I have made some new progress now.
> >
> > To tell how many false dirty pages there are exactly in each iteration, I malloc a
> > buffer in memory as big as the size of the whole VM memory. When a page is
> > transferred to the dest node, it is copied to the buffer; During the next iteration,
> > if one page is transferred, it is compared to the old one in the buffer, and the
> > old one will be replaced for next comparison if it is really dirty. Thus, we are now
> > able to get the exact number of false dirty pages.
> >
> > This time, I use 15 workloads to get the statistic number. They are:
> >
> > 1. 11 benchmarks picked up from cpu2006 benchmark suit. They are all scientific
> > computing workloads like Quantum Chromodynamics, Fluid Dynamics, etc. I pick
> > up these 11 benchmarks because compared to others, they have bigger memory
> > occupation and higher memory dirty rate. Thus most of them could not converge
> > to stop-and-copy using the default migration speed (32MB/s).
> > 2. kernel compilation
> > 3. idle VM
> > 4. Apache web server which serves static content
> >
> > (the above workloads are all running in VM with 1 vcpu and 1GB memory, and the
> > migration speed is the default 32MB/s)
> >
> > 5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are used as the cache.
> > After filling up the 4GB cache, a client writes the cache at a constant speed
> > during migration. This time, migration speed has no limit, and is up to the
> > capability of 1Gbps Ethernet.
> >
> > Summarize the results first: (and you can read the precise number below)
> >
> > 1. 4 of these 15 workloads have a big proportion (>60%, even >80% during some iterations)
> > of false dirty pages out of all the dirty pages since iteration 2 (and the big
> > proportion lasts during the following iterations). They are cpu2006.zeusmp,
> > cpu2006.bzip2, cpu2006.mcf, and memcached.
> > 2. 2 workloads (idle, webserver) spend most of the migration time on iteration 1, even
> > though the proportion of false dirty pages is big since iteration 2, the space to
> > optimize is small.
> > 3. 1 workload (kernel compilation) only have a big proportion during iteration 2, not
> > in the other iterations.
> > 4. 8 workloads (the other 8 benchmarks of cpu2006) have little proportion of false
> > dirty pages since iteration 2. So the spaces to optimize for them are small.
> >
> > Now I want to talk a little more about the reasons why false dirty pages are produced.
> > The first reason is what we have discussed before---the mechanism to track the dirty
> > pages.
> > And then I come up with another reason. Here is the situation: a write operation to one
> > memory page happens, but it doesn't change any content of the page. So it's "write but
> > not dirty", and kernel still marks it as dirty. One guy in our lab has done some experiments
> > to figure out the proportion of "write but not dirty" operations, and he uses the cpu2006
> > benchmark suit. According to his results, general workloads has a little proportion (<10%)
> > of "write but not dirty" out of all the write operations, while few workloads have higher
> > proportion (one even as high as 50%). Now we are not sure why "write but not dirty" would
> > happen, it just happened.
>
> I think there are a few different reasons I can think of:
> a) You have a flag or mutex that's set and cleared; so it gets set (marked
> dirty) and cleared around some operation. By the time we come to migrate
> it then it's back to cleared again.
> Similarly with other temporary data structures.
> b) Some system operation causes the page to be moved - e.g. swap or the kernel
> reorganising memory.
Sorry, I don't quite understand reason (b). Take swap as example, do you mean a page
is swapped out and swapped in to the old address again, so the content remains unchanged?
>
> However, it's a shame I don't think you can tell in your experiment which of the
> two cases we're hitting? I'd like to know if it's worth working on
> making the page sync mechanism better or if it's nore important to deal
> with the second reason you show.
Yes, you are right, it's hard to tell which case we're hitting (including the cases you
think of). However, as I use the SHA1 method, now I don't have to tell them. Because it
just handle all the cases we have thought of.
>
> > So these two reasons contribute to the false dirty pages. To optimize, I compute and store
> > the SHA1 hash before transferring each page. Next time, if one page needs retransmission, its
> > SHA1 hash is computed again, and compared to the old hash. If the hash is the same, it's a
> > false dirty page, and we just skip this page; Otherwise, the page is transferred, and the new
> > hash replaces the old one for next comparison.
> > The reason to use SHA1 hash but not byte-by-byte comparison is the memory overheads. One SHA1
> > hash is 20 bytes. So we need extra 20/4096 (<1/200) memory space of the whole VM memory, which
> > is relatively small.
> > As far as I know, SHA1 hash is widely used in the scenes of deduplication for backup systems.
> > They have proven that the probability of hash collision is far smaller than disk hardware fault,
> > so it's secure hash, that is, if the hashes of two chunks are the same, the content must be the
> > same. So I think the SHA1 hash could replace byte-to-byte comparison in the VM memory scenery.
>
> There was a proposal ( https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg05331.html )
> to do a migration system where
> a copy of the migration RAM is stored on disc on the destination for cases where similar VMs
> are migrated, and it used a checksum for each page to find the matching page
> in the cache; that originally used a smaller hash, I think in the end they used a SHA-256.
> (Hash based checks still make me nervous for intentional collisions but that's probably
> me being paranoid?)
Em... I don't know if most people would accept the hash based checks.
Maybe it needs some more mathematical proving like they have done in the
field of deduplication for backup systems.
>
> > Then I do the same migration experiments using the SHA1 hash. For the 4 workloads which have
> > big proportions of false dirty pages, the improvement is remarkable. Without optimization,
> > they either can not converge to stop-and-copy, or take a very long time to complete. With the
> > SHA1 hash method, all of them now complete in a relatively short time.
> > For the reason I have talked above, the other workloads don't get notable improvements from the
> > optimization. So below, I only show the exact number after optimization for the 4 workloads with
> > remarkable improvements.
> >
> > Any comments or suggestions?
>
> You might be able to save some of the CPU time; we've
> got a test that checks if a page is all-zero; if you're doing
> the SHA calculation you could avoid doing the all-zero check
> and replace it by comparing hte output of the SHA.
Yes, this is one way. However now I'm doing the opposite. I first
calculate the SHA1 of the all-zero page and remember that. Then next time,
if I recognize an all-zero page after the check, I just store the
SHA1 I have got earlier, avoiding calculating the SHA1 of the all-zero
page again. I think this is better, because I think the current implementation
to check all-zero pages is faster than calculating SHA1.
Thanks,
Chunguang
>
> >
> > Below is the experiments data:
> > (
> > "dup" means zero page, this kind of pages takes very little migration time and network
> > resources, so they are always not regard as dirty pages in my numbers;
> > "rd" means really dirty pages;
> > "fd" means false dirty pages;
> > The numbers refer to the quantities of pages.
> > )
> >
> > ------------------The 4 workloads with remarkable improvements (both the results of original precopy and with optimization are shown)-------------------
> >
> > 1. memcached
> >
> > ----- original pre-copy (can not converge): -----
> > Iteration 1, duration: 42111 ms , transferred pages: 1568788 (dup: 416239, rd: 1152549, fd: 0) , new dirty pages: 499015 , remaining dirty pages: 507397
> > Iteration 2, duration: 17208 ms , transferred pages: 498946 (dup: 5456, rd: 160206, fd: 333284) , new dirty pages: 261237 , remaining dirty pages: 269688
> > Iteration 3, duration: 9134 ms , transferred pages: 262377 (dup: 519, rd: 111900, fd: 149958) , new dirty pages: 170281 , remaining dirty pages: 177592
> > Iteration 4, duration: 5920 ms , transferred pages: 169966 (dup: 87, rd: 82487, fd: 87392) , new dirty pages: 121154 , remaining dirty pages: 128780
> > Iteration 5, duration: 4239 ms , transferred pages: 121551 (dup: 81, rd: 64120, fd: 57350) , new dirty pages: 100976 , remaining dirty pages: 108205
> > Iteration 6, duration: 3495 ms , transferred pages: 100353 (dup: 90, rd: 56021, fd: 44242) , new dirty pages: 74547 , remaining dirty pages: 82399
> > Iteration 7, duration: 2583 ms , transferred pages: 74160 (dup: 56, rd: 38016, fd: 36088) , new dirty pages: 58209 , remaining dirty pages: 66448
> > Iteration 8, duration: 2039 ms , transferred pages: 58534 (dup: 81, rd: 26885, fd: 31568) , new dirty pages: 43511 , remaining dirty pages: 51425
> > Iteration 9, duration: 1513 ms , transferred pages: 43484 (dup: 55, rd: 26641, fd: 16788) , new dirty pages: 43722 , remaining dirty pages: 51663
> > Iteration 10, duration: 1521 ms , transferred pages: 43676 (dup: 62, rd: 26463, fd: 17151) , new dirty pages: 35347 , remaining dirty pages: 43334
> > Iteration 11, duration: 1230 ms , transferred pages: 35287 (dup: 0, rd: 21293, fd: 13994) , new dirty pages: 28851 , remaining dirty pages: 36898
> > Iteration 12, duration: 1031 ms , transferred pages: 29651 (dup: 82, rd: 18143, fd: 11426) , new dirty pages: 27062 , remaining dirty pages: 34309
> > Iteration 13, duration: 917 ms , transferred pages: 26385 (dup: 56, rd: 14149, fd: 12180) , new dirty pages: 22723 , remaining dirty pages: 30647
> > Iteration 14, duration: 762 ms , transferred pages: 21902 (dup: 55, rd: 16355, fd: 5492) , new dirty pages: 18208 , remaining dirty pages: 26953
> > Iteration 15, duration: 650 ms , transferred pages: 18636 (dup: 0, rd: 11943, fd: 6693) , new dirty pages: 16085 , remaining dirty pages: 24402
> > Iteration 16, duration: 554 ms , transferred pages: 15946 (dup: 56, rd: 9527, fd: 6363) , new dirty pages: 14766 , remaining dirty pages: 23222
> > Iteration 17, duration: 538 ms , transferred pages: 15434 (dup: 0, rd: 9779, fd: 5655) , new dirty pages: 13381 , remaining dirty pages: 21169
> > Iteration 18, duration: 487 ms , transferred pages: 14089 (dup: 81, rd: 7737, fd: 6271) , new dirty pages: 13325 , remaining dirty pages: 20405
> > Iteration 19, duration: 428 ms , transferred pages: 12232 (dup: 0, rd: 8488, fd: 3744) , new dirty pages: 10274 , remaining dirty pages: 18447
> > Iteration 20, duration: 377 ms , transferred pages: 10887 (dup: 56, rd: 6362, fd: 4469) , new dirty pages: 9708 , remaining dirty pages: 17268
> > Iteration 21, duration: 320 ms , transferred pages: 9222 (dup: 0, rd: 5789, fd: 3433) , new dirty pages: 8015 , remaining dirty pages: 16061
> > Iteration 22, duration: 268 ms , transferred pages: 7621 (dup: 0, rd: 6204, fd: 1417) , new dirty pages: 7227 , remaining dirty pages: 15667
> > Iteration 23, duration: 269 ms , transferred pages: 7813 (dup: 56, rd: 4410, fd: 3347) , new dirty pages: 7591 , remaining dirty pages: 15445
> > Iteration 24, duration: 271 ms , transferred pages: 7749 (dup: 0, rd: 4565, fd: 3184) , new dirty pages: 15126 , remaining dirty pages: 22822
> > Iteration 25, duration: 549 ms , transferred pages: 15818 (dup: 60, rd: 10545, fd: 5213) , new dirty pages: 14559 , remaining dirty pages: 21563
> > Iteration 26, duration: 499 ms , transferred pages: 14281 (dup: 3, rd: 8760, fd: 5518) , new dirty pages: 11947 , remaining dirty pages: 19229
> > Iteration 27, duration: 376 ms , transferred pages: 10823 (dup: 25, rd: 6550, fd: 4248) , new dirty pages: 8561 , remaining dirty pages: 16967
> > Iteration 28, duration: 324 ms , transferred pages: 9350 (dup: 31, rd: 5292, fd: 4027) , new dirty pages: 8655 , remaining dirty pages: 16272
> > Iteration 29, duration: 274 ms , transferred pages: 7813 (dup: 0, rd: 6088, fd: 1725) , new dirty pages: 6300 , remaining dirty pages: 14759
> > Iteration 30, duration: 218 ms , transferred pages: 6340 (dup: 45, rd: 3196, fd: 3099) , new dirty pages: 5143 , remaining dirty pages: 13562
> >
> > ----- after optimization: -----
> > Iteration 1, duration: 40664 ms , transferred pages: 1569037 (dup: 405940, rd: 1163097) , new dirty pages: 506846 , remaining dirty pages: 514979
> > Iteration 2, duration: 8032 ms , transferred pages: 161130 (dup: 4007, rd: 157123) , new dirty pages: 153479 , remaining dirty pages: 153479
>
> Big difference.
>
> > Iteration 3, duration: 2620 ms , transferred pages: 65260 (dup: 20, rd: 65240) , new dirty pages: 64014 , remaining dirty pages: 67100
> > Iteration 4, duration: 1160 ms , transferred pages: 30227 (dup: 60, rd: 30167) , new dirty pages: 34031 , remaining dirty pages: 41414
> > Iteration 5, duration: 648 ms , transferred pages: 18700 (dup: 56, rd: 18644) , new dirty pages: 18375 , remaining dirty pages: 25536
> > Iteration 6, duration: 389 ms , transferred pages: 11399 (dup: 55, rd: 11344) , new dirty pages: 12536 , remaining dirty pages: 17516
> > Iteration 7, duration: 292 ms , transferred pages: 8197 (dup: 0, rd: 8197) , new dirty pages: 8387 , remaining dirty pages: 16802
> > Iteration 8, duration: 171 ms , transferred pages: 4931 (dup: 39, rd: 4892) , new dirty pages: 6182 , remaining dirty pages: 14060
> > Iteration 9, duration: 163 ms , transferred pages: 4355 (dup: 16, rd: 4339) , new dirty pages: 5530 , remaining dirty pages: 11973
> > Iteration 10, duration: 104 ms , transferred pages: 3266 (dup: 0, rd: 3266) , new dirty pages: 2893 , remaining dirty pages: 11014
> > Iteration 11, duration: 52 ms , transferred pages: 1153 (dup: 0, rd: 1153) , new dirty pages: 1586 , remaining dirty pages: 10516
> > Iteration 12, duration: 52 ms , transferred pages: 1921 (dup: 39, rd: 1882) , new dirty pages: 1619 , remaining dirty pages: 8842
> > Iteration 13, duration: 62 ms , transferred pages: 1537 (dup: 0, rd: 1537) , new dirty pages: 2052 , remaining dirty pages: 8871
> > Iteration 14, duration: 58 ms , transferred pages: 1665 (dup: 0, rd: 1665) , new dirty pages: 1947 , remaining dirty pages: 7989
> > Iteration 15, duration: 2 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 7989
> > total time: 54693 milliseconds
>
> Very nice.
>
> Dave
>
> > 2. cpu2006.zeusmp
> >
> > ----- original pre-copy (can not converge): -----
> > Iteration 1, duration: 21112 ms , transferred pages: 266450 (dup: 93385, rd: 173065, fd: 0) , new dirty pages: 127866 , remaining dirty pages: 127866
> > Iteration 2, duration: 6192 ms , transferred pages: 125662 (dup: 75762, rd: 17389, fd: 32511) , new dirty pages: 131655 , remaining dirty pages: 133859
> > Iteration 3, duration: 6699 ms , transferred pages: 131937 (dup: 77298, rd: 20320, fd: 34319) , new dirty pages: 121027 , remaining dirty pages: 122949
> > Iteration 4, duration: 5999 ms , transferred pages: 122512 (dup: 73588, rd: 17236, fd: 31688) , new dirty pages: 122759 , remaining dirty pages: 123196
> > Iteration 5, duration: 5804 ms , transferred pages: 122717 (dup: 75436, rd: 19016, fd: 28265) , new dirty pages: 123697 , remaining dirty pages: 124176
> > Iteration 6, duration: 5698 ms , transferred pages: 123708 (dup: 77249, rd: 18022, fd: 28437) , new dirty pages: 121838 , remaining dirty pages: 122306
> > Iteration 7, duration: 5515 ms , transferred pages: 122306 (dup: 76727, rd: 14819, fd: 30760) , new dirty pages: 122382 , remaining dirty pages: 122382
> > Iteration 8, duration: 6086 ms , transferred pages: 120825 (dup: 71834, rd: 15987, fd: 33004) , new dirty pages: 121587 , remaining dirty pages: 123144
> > Iteration 9, duration: 5899 ms , transferred pages: 120964 (dup: 72860, rd: 18191, fd: 29913) , new dirty pages: 120391 , remaining dirty pages: 122571
> > Iteration 10, duration: 5801 ms , transferred pages: 121425 (dup: 74140, rd: 20722, fd: 26563) , new dirty pages: 122302 , remaining dirty pages: 123448
> > Iteration 11, duration: 5909 ms , transferred pages: 123448 (dup: 74735, rd: 19678, fd: 29035) , new dirty pages: 123258 , remaining dirty pages: 123258
> > Iteration 12, duration: 6293 ms , transferred pages: 121211 (dup: 70442, rd: 18128, fd: 32641) , new dirty pages: 123623 , remaining dirty pages: 125670
> > Iteration 13, duration: 6398 ms , transferred pages: 124897 (dup: 72701, rd: 21134, fd: 31062) , new dirty pages: 122355 , remaining dirty pages: 123128
> > Iteration 14, duration: 6301 ms , transferred pages: 121893 (dup: 70514, rd: 23470, fd: 27909) , new dirty pages: 120980 , remaining dirty pages: 122215
> > Iteration 15, duration: 6304 ms , transferred pages: 121389 (dup: 70005, rd: 21731, fd: 29653) , new dirty pages: 121628 , remaining dirty pages: 122454
> > Iteration 16, duration: 6398 ms , transferred pages: 122164 (dup: 69962, rd: 24376, fd: 27826) , new dirty pages: 122246 , remaining dirty pages: 122536
> > Iteration 17, duration: 6201 ms , transferred pages: 121548 (dup: 70984, rd: 23915, fd: 26649) , new dirty pages: 121460 , remaining dirty pages: 122448
> > Iteration 18, duration: 6401 ms , transferred pages: 122272 (dup: 70072, rd: 22261, fd: 29939) , new dirty pages: 123518 , remaining dirty pages: 123694
> > Iteration 19, duration: 7003 ms , transferred pages: 121873 (dup: 64754, rd: 27325, fd: 29794) , new dirty pages: 120568 , remaining dirty pages: 122389
> > Iteration 20, duration: 6400 ms , transferred pages: 121422 (dup: 69221, rd: 25300, fd: 26901) , new dirty pages: 121229 , remaining dirty pages: 122196
> > Iteration 21, duration: 6703 ms , transferred pages: 119895 (dup: 65232, rd: 25877, fd: 28786) , new dirty pages: 123284 , remaining dirty pages: 125585
> > Iteration 22, duration: 6902 ms , transferred pages: 123884 (dup: 67582, rd: 29020, fd: 27282) , new dirty pages: 122057 , remaining dirty pages: 123758
> > Iteration 23, duration: 6800 ms , transferred pages: 122010 (dup: 66529, rd: 30644, fd: 24837) , new dirty pages: 120916 , remaining dirty pages: 122664
> > Iteration 24, duration: 7202 ms , transferred pages: 121951 (dup: 63188, rd: 31105, fd: 27658) , new dirty pages: 122715 , remaining dirty pages: 123428
> > Iteration 25, duration: 7202 ms , transferred pages: 122919 (dup: 64161, rd: 32063, fd: 26695) , new dirty pages: 123180 , remaining dirty pages: 123689
> > Iteration 26, duration: 7404 ms , transferred pages: 123092 (dup: 62694, rd: 33459, fd: 26939) , new dirty pages: 122149 , remaining dirty pages: 122746
> > Iteration 27, duration: 7205 ms , transferred pages: 120427 (dup: 61664, rd: 34344, fd: 24419) , new dirty pages: 120299 , remaining dirty pages: 122618
> > Iteration 28, duration: 7100 ms , transferred pages: 121074 (dup: 63130, rd: 32403, fd: 25541) , new dirty pages: 122984 , remaining dirty pages: 124528
> > Iteration 29, duration: 7904 ms , transferred pages: 124060 (dup: 59564, rd: 35631, fd: 28865) , new dirty pages: 127080 , remaining dirty pages: 127548
> > Iteration 30, duration: 7906 ms , transferred pages: 127518 (dup: 63029, rd: 34416, fd: 30073) , new dirty pages: 125028 , remaining dirty pages: 125058
> >
> > ----- after optimization: -----
> > Iteration 1, duration: 21601 ms , transferred pages: 266450 (dup: 89731, rd: 176719) , new dirty pages: 139843 , remaining dirty pages: 139843
> > Iteration 2, duration: 1747 ms , transferred pages: 92077 (dup: 78364, rd: 13713) , new dirty pages: 90945 , remaining dirty pages: 90945
> > Iteration 3, duration: 1592 ms , transferred pages: 62253 (dup: 49435, rd: 12818) , new dirty pages: 76929 , remaining dirty pages: 76929
> > Iteration 4, duration: 992 ms , transferred pages: 44837 (dup: 37886, rd: 6951) , new dirty pages: 71331 , remaining dirty pages: 72916
> > Iteration 5, duration: 998 ms , transferred pages: 55229 (dup: 47150, rd: 8079) , new dirty pages: 21703 , remaining dirty pages: 23302
> > Iteration 6, duration: 211 ms , transferred pages: 20337 (dup: 18516, rd: 1821) , new dirty pages: 14500 , remaining dirty pages: 14500
> > Iteration 7, duration: 31 ms , transferred pages: 12933 (dup: 12627, rd: 306) , new dirty pages: 1520 , remaining dirty pages: 1520
> > Iteration 8, duration: 30 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 4 , remaining dirty pages: 1524
> > total time: 27225 milliseconds
> >
> > 3. cpu2006.bzip2
> >
> > ----- original pre-copy: -----
> > Iteration 1, duration: 18306 ms , transferred pages: 266450 (dup: 116569, rd: 149881, fd: 0) , new dirty pages: 106299 , remaining dirty pages: 106299
> > Iteration 2, duration: 10694 ms , transferred pages: 104611 (dup: 17550, rd: 10536, fd: 76525) , new dirty pages: 34394 , remaining dirty pages: 36082
> > Iteration 3, duration: 2998 ms , transferred pages: 34442 (dup: 9924, rd: 12254, fd: 12264) , new dirty pages: 6419 , remaining dirty pages: 8059
> > Iteration 4, duration: 699 ms , transferred pages: 5748 (dup: 22, rd: 2583, fd: 3143) , new dirty pages: 1226 , remaining dirty pages: 3537
> > Iteration 5, duration: 200 ms , transferred pages: 1636 (dup: 0, rd: 1194, fd: 442) , new dirty pages: 478 , remaining dirty pages: 2379
> > Iteration 6, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2379
> >
> > ----- after optimization: -----
> > Iteration 1, duration: 13995 ms , transferred pages: 266314 (dup: 152118, rd: 114196) , new dirty pages: 97009 , remaining dirty pages: 97145
> > Iteration 2, duration: 1215 ms , transferred pages: 33400 (dup: 26745, rd: 6655) , new dirty pages: 12866 , remaining dirty pages: 14017
> > Iteration 3, duration: 701 ms , transferred pages: 5774 (dup: 48, rd: 5726) , new dirty pages: 6342 , remaining dirty pages: 8761
> > Iteration 4, duration: 500 ms , transferred pages: 4111 (dup: 21, rd: 4090) , new dirty pages: 4311 , remaining dirty pages: 6485
> > Iteration 5, duration: 400 ms , transferred pages: 3273 (dup: 1, rd: 3272) , new dirty pages: 3034 , remaining dirty pages: 5431
> > Iteration 6, duration: 301 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2094 , remaining dirty pages: 4472
> > Iteration 7, duration: 299 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2066 , remaining dirty pages: 4082
> > Iteration 8, duration: 202 ms , transferred pages: 1636 (dup: 0, rd: 1636) , new dirty pages: 2881 , remaining dirty pages: 4648
> > Iteration 9, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 4775 , remaining dirty pages: 6778
> > Iteration 10, duration: 400 ms , transferred pages: 3281 (dup: 9, rd: 3272) , new dirty pages: 3757 , remaining dirty pages: 5576
> > Iteration 11, duration: 401 ms , transferred pages: 3279 (dup: 7, rd: 3272) , new dirty pages: 6980 , remaining dirty pages: 8906
> > Iteration 12, duration: 500 ms , transferred pages: 7118 (dup: 3035, rd: 4083) , new dirty pages: 10774 , remaining dirty pages: 11922
> > Iteration 13, duration: 116 ms , transferred pages: 11706 (dup: 10152, rd: 1554) , new dirty pages: 1326 , remaining dirty pages: 1326
> > Iteration 14, duration: 117 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 1326
> > total time: 19479 milliseconds
> >
> > 4. cpu2006.mcf
> >
> > ----- original pre-copy: -----
> > Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6925, rd: 259525, fd: 0) , new dirty pages: 244403 , remaining dirty pages: 244403
> > Iteration 2, duration: 29603 ms , transferred pages: 242275 (dup: 377, rd: 224001, fd: 17897) , new dirty pages: 227335 , remaining dirty pages: 229463
> > Iteration 3, duration: 27806 ms , transferred pages: 227573 (dup: 169, rd: 65681, fd: 161723) , new dirty pages: 195593 , remaining dirty pages: 197483
> > Iteration 4, duration: 23907 ms , transferred pages: 195543 (dup: 41, rd: 39838, fd: 155664) , new dirty pages: 215066 , remaining dirty pages: 217006
> > Iteration 5, duration: 26305 ms , transferred pages: 215289 (dup: 155, rd: 33082, fd: 182052) , new dirty pages: 111098 , remaining dirty pages: 112815
> > Iteration 6, duration: 13502 ms , transferred pages: 110452 (dup: 22, rd: 26793, fd: 83637) , new dirty pages: 161054 , remaining dirty pages: 163417
> > Iteration 7, duration: 19705 ms , transferred pages: 161266 (dup: 120, rd: 33818, fd: 127328) , new dirty pages: 220562 , remaining dirty pages: 222713
> > Iteration 8, duration: 27003 ms , transferred pages: 220881 (dup: 21, rd: 215721, fd: 5139) , new dirty pages: 219787 , remaining dirty pages: 221619
> > Iteration 9, duration: 26802 ms , transferred pages: 219248 (dup: 24, rd: 84648, fd: 134576) , new dirty pages: 207959 , remaining dirty pages: 210330
> > Iteration 10, duration: 25411 ms , transferred pages: 207916 (dup: 144, rd: 35842, fd: 171930) , new dirty pages: 144442 , remaining dirty pages: 146856
> > Iteration 11, duration: 17714 ms , transferred pages: 144804 (dup: 18, rd: 25414, fd: 119372) , new dirty pages: 205127 , remaining dirty pages: 207179
> > Iteration 12, duration: 25112 ms , transferred pages: 205446 (dup: 128, rd: 23197, fd: 182121) , new dirty pages: 167319 , remaining dirty pages: 169052
> > Iteration 13, duration: 20411 ms , transferred pages: 166886 (dup: 14, rd: 21960, fd: 144912) , new dirty pages: 221592 , remaining dirty pages: 223758
> > Iteration 14, duration: 27126 ms , transferred pages: 221800 (dup: 122, rd: 42368, fd: 179310) , new dirty pages: 233630 , remaining dirty pages: 235588
> > Iteration 15, duration: 28517 ms , transferred pages: 233321 (dup: 191, rd: 222528, fd: 10602) , new dirty pages: 224282 , remaining dirty pages: 226549
> > Iteration 16, duration: 27422 ms , transferred pages: 224187 (dup: 55, rd: 45773, fd: 178359) , new dirty pages: 209815 , remaining dirty pages: 212177
> > Iteration 17, duration: 25723 ms , transferred pages: 210260 (dup: 34, rd: 79405, fd: 130821) , new dirty pages: 220297 , remaining dirty pages: 222214
> > Iteration 18, duration: 26920 ms , transferred pages: 220056 (dup: 14, rd: 214128, fd: 5914) , new dirty pages: 192015 , remaining dirty pages: 194173
> > Iteration 19, duration: 23520 ms , transferred pages: 192239 (dup: 9, rd: 25140, fd: 167090) , new dirty pages: 96450 , remaining dirty pages: 98384
> > Iteration 20, duration: 11805 ms , transferred pages: 96538 (dup: 14, rd: 7424, fd: 89100) , new dirty pages: 6978 , remaining dirty pages: 8824
> > Iteration 21, duration: 799 ms , transferred pages: 6545 (dup: 1, rd: 1802, fd: 4742) , new dirty pages: 138 , remaining dirty pages: 2417
> > Iteration 22, duration: 1 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2417
> >
> > ----- after optimization: -----
> > Iteration 1, duration: 31711 ms , transferred pages: 266450 (dup: 6831, rd: 259619) , new dirty pages: 240209 , remaining dirty pages: 240209
> > Iteration 2, duration: 6250 ms , transferred pages: 51244 (dup: 211, rd: 51033) , new dirty pages: 226651 , remaining dirty pages: 228571
> > Iteration 3, duration: 4395 ms , transferred pages: 36008 (dup: 80, rd: 35928) , new dirty pages: 110719 , remaining dirty pages: 111478
> > Iteration 4, duration: 3390 ms , transferred pages: 28068 (dup: 28, rd: 28040) , new dirty pages: 185172 , remaining dirty pages: 185172
> > Iteration 5, duration: 2986 ms , transferred pages: 23780 (dup: 45, rd: 23735) , new dirty pages: 64357 , remaining dirty pages: 66305
> > Iteration 6, duration: 2727 ms , transferred pages: 22800 (dup: 12, rd: 22788) , new dirty pages: 61675 , remaining dirty pages: 61675
> > Iteration 7, duration: 2372 ms , transferred pages: 18943 (dup: 13, rd: 18930) , new dirty pages: 55144 , remaining dirty pages: 55265
> > Iteration 8, duration: 2100 ms , transferred pages: 17189 (dup: 11, rd: 17178) , new dirty pages: 55244 , remaining dirty pages: 55668
> > Iteration 9, duration: 2003 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 107058 , remaining dirty pages: 108014
> > Iteration 10, duration: 2132 ms , transferred pages: 17825 (dup: 24, rd: 17801) , new dirty pages: 126214 , remaining dirty pages: 126214
> > Iteration 11, duration: 2229 ms , transferred pages: 18156 (dup: 22, rd: 18134) , new dirty pages: 65725 , remaining dirty pages: 65725
> > Iteration 12, duration: 2315 ms , transferred pages: 18651 (dup: 21, rd: 18630) , new dirty pages: 52575 , remaining dirty pages: 53903
> > Iteration 13, duration: 2147 ms , transferred pages: 17435 (dup: 16, rd: 17419) , new dirty pages: 46652 , remaining dirty pages: 47260
> > Iteration 14, duration: 2000 ms , transferred pages: 16371 (dup: 11, rd: 16360) , new dirty pages: 42721 , remaining dirty pages: 43266
> > Iteration 15, duration: 1901 ms , transferred pages: 15552 (dup: 10, rd: 15542) , new dirty pages: 38593 , remaining dirty pages: 40792
> > Iteration 16, duration: 1801 ms , transferred pages: 14735 (dup: 11, rd: 14724) , new dirty pages: 54252 , remaining dirty pages: 55639
> > Iteration 17, duration: 1708 ms , transferred pages: 13860 (dup: 2, rd: 13858) , new dirty pages: 72379 , remaining dirty pages: 74170
> > Iteration 18, duration: 1923 ms , transferred pages: 15442 (dup: 12, rd: 15430) , new dirty pages: 101911 , remaining dirty pages: 103547
> > Iteration 19, duration: 2311 ms , transferred pages: 18823 (dup: 9, rd: 18814) , new dirty pages: 80534 , remaining dirty pages: 82521
> > Iteration 20, duration: 2081 ms , transferred pages: 17156 (dup: 34, rd: 17122) , new dirty pages: 36054 , remaining dirty pages: 36054
> > Iteration 21, duration: 1665 ms , transferred pages: 13777 (dup: 10, rd: 13767) , new dirty pages: 29624 , remaining dirty pages: 29624
> > Iteration 22, duration: 1657 ms , transferred pages: 13290 (dup: 7, rd: 13283) , new dirty pages: 25949 , remaining dirty pages: 28265
> > Iteration 23, duration: 1599 ms , transferred pages: 13088 (dup: 0, rd: 13088) , new dirty pages: 22356 , remaining dirty pages: 24813
> > Iteration 24, duration: 1500 ms , transferred pages: 12280 (dup: 10, rd: 12270) , new dirty pages: 21181 , remaining dirty pages: 22608
> > Iteration 25, duration: 1400 ms , transferred pages: 11457 (dup: 5, rd: 11452) , new dirty pages: 18657 , remaining dirty pages: 20311
> > Iteration 26, duration: 1200 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 15690 , remaining dirty pages: 17294
> > Iteration 27, duration: 1201 ms , transferred pages: 9822 (dup: 6, rd: 9816) , new dirty pages: 14810 , remaining dirty pages: 15936
> > Iteration 28, duration: 1000 ms , transferred pages: 8183 (dup: 3, rd: 8180) , new dirty pages: 15387 , remaining dirty pages: 16423
> > Iteration 29, duration: 900 ms , transferred pages: 7372 (dup: 10, rd: 7362) , new dirty pages: 13303 , remaining dirty pages: 15292
> > Iteration 30, duration: 1000 ms , transferred pages: 8181 (dup: 1, rd: 8180) , new dirty pages: 17879 , remaining dirty pages: 18457
> > Iteration 31, duration: 951 ms , transferred pages: 8140 (dup: 9, rd: 8131) , new dirty pages: 21738 , remaining dirty pages: 23304
> > Iteration 32, duration: 946 ms , transferred pages: 6946 (dup: 1, rd: 6945) , new dirty pages: 15815 , remaining dirty pages: 15815
> > Iteration 33, duration: 747 ms , transferred pages: 6192 (dup: 0, rd: 6192) , new dirty pages: 6249 , remaining dirty pages: 7670
> > Iteration 34, duration: 501 ms , transferred pages: 4090 (dup: 0, rd: 4090) , new dirty pages: 6163 , remaining dirty pages: 8422
> > Iteration 35, duration: 600 ms , transferred pages: 4910 (dup: 2, rd: 4908) , new dirty pages: 3673 , remaining dirty pages: 5222
> > Iteration 36, duration: 300 ms , transferred pages: 2454 (dup: 0, rd: 2454) , new dirty pages: 2132 , remaining dirty pages: 4337
> > Iteration 37, duration: 200 ms , transferred pages: 1637 (dup: 1, rd: 1636) , new dirty pages: 544 , remaining dirty pages: 2251
> > Iteration 38, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0) , new dirty pages: 0 , remaining dirty pages: 2251
> > total time: 97919 milliseconds
> >
> > ------------------The other 11 workloads without notable improvements (only the result of original precopy is shown)-------------------
> >
> > 5. idle
> >
> > Iteration 1, duration: 14702 ms , transferred pages: 266450 (dup: 146393, rd: 120057, fd: 0) , new dirty pages: 14595 , remaining dirty pages: 14595
> > Iteration 2, duration: 1592 ms , transferred pages: 12412 (dup: 103, rd: 3280, fd: 9029) , new dirty pages: 218 , remaining dirty pages: 2401
> > Iteration 3, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2401
> >
> > 6. kernel compilation (can not converge)
> >
> > Iteration 1, duration: 20607 ms , transferred pages: 266450 (dup: 97552, rd: 168898, fd: 0) , new dirty pages: 19293 , remaining dirty pages: 19293
> > Iteration 2, duration: 2092 ms , transferred pages: 17176 (dup: 597, rd: 8625, fd: 7954) , new dirty pages: 8318 , remaining dirty pages: 10435
> > Iteration 3, duration: 1000 ms , transferred pages: 8484 (dup: 304, rd: 6256, fd: 1924) , new dirty pages: 8736 , remaining dirty pages: 10687
> > Iteration 4, duration: 1000 ms , transferred pages: 8435 (dup: 255, rd: 7089, fd: 1091) , new dirty pages: 7627 , remaining dirty pages: 9879
> > Iteration 5, duration: 900 ms , transferred pages: 7553 (dup: 191, rd: 5602, fd: 1760) , new dirty pages: 7287 , remaining dirty pages: 9613
> > Iteration 6, duration: 900 ms , transferred pages: 7620 (dup: 258, rd: 5761, fd: 1601) , new dirty pages: 8958 , remaining dirty pages: 10951
> > Iteration 7, duration: 1099 ms , transferred pages: 9309 (dup: 311, rd: 8051, fd: 947) , new dirty pages: 7189 , remaining dirty pages: 8831
> > Iteration 8, duration: 800 ms , transferred pages: 6832 (dup: 288, rd: 5717, fd: 827) , new dirty pages: 5782 , remaining dirty pages: 7781
> > Iteration 9, duration: 701 ms , transferred pages: 5875 (dup: 149, rd: 4005, fd: 1721) , new dirty pages: 4587 , remaining dirty pages: 6493
> > Iteration 10, duration: 500 ms , transferred pages: 4234 (dup: 144, rd: 3057, fd: 1033) , new dirty pages: 7352 , remaining dirty pages: 9611
> > Iteration 11, duration: 900 ms , transferred pages: 7759 (dup: 397, rd: 6563, fd: 799) , new dirty pages: 6686 , remaining dirty pages: 8538
> > Iteration 12, duration: 800 ms , transferred pages: 6808 (dup: 264, rd: 6017, fd: 527) , new dirty pages: 6871 , remaining dirty pages: 8601
> > Iteration 13, duration: 800 ms , transferred pages: 6775 (dup: 231, rd: 5722, fd: 822) , new dirty pages: 7540 , remaining dirty pages: 9366
> > Iteration 14, duration: 900 ms , transferred pages: 7507 (dup: 145, rd: 5900, fd: 1462) , new dirty pages: 7581 , remaining dirty pages: 9440
> > Iteration 15, duration: 900 ms , transferred pages: 7630 (dup: 268, rd: 6211, fd: 1151) , new dirty pages: 7268 , remaining dirty pages: 9078
> > Iteration 16, duration: 800 ms , transferred pages: 6759 (dup: 215, rd: 5763, fd: 781) , new dirty pages: 6861 , remaining dirty pages: 9180
> > Iteration 17, duration: 800 ms , transferred pages: 6838 (dup: 294, rd: 6037, fd: 507) , new dirty pages: 6196 , remaining dirty pages: 8538
> > Iteration 18, duration: 800 ms , transferred pages: 6852 (dup: 308, rd: 4905, fd: 1639) , new dirty pages: 5947 , remaining dirty pages: 7633
> > Iteration 19, duration: 700 ms , transferred pages: 5919 (dup: 193, rd: 4853, fd: 873) , new dirty pages: 5861 , remaining dirty pages: 7575
> > Iteration 20, duration: 600 ms , transferred pages: 5284 (dup: 376, rd: 4408, fd: 500) , new dirty pages: 5206 , remaining dirty pages: 7497
> > Iteration 21, duration: 600 ms , transferred pages: 5147 (dup: 239, rd: 4308, fd: 600) , new dirty pages: 5031 , remaining dirty pages: 7381
> > Iteration 22, duration: 599 ms , transferred pages: 5064 (dup: 156, rd: 4026, fd: 882) , new dirty pages: 5601 , remaining dirty pages: 7918
> > Iteration 23, duration: 702 ms , transferred pages: 5965 (dup: 239, rd: 5028, fd: 698) , new dirty pages: 6079 , remaining dirty pages: 8032
> > Iteration 24, duration: 700 ms , transferred pages: 6175 (dup: 449, rd: 5146, fd: 580) , new dirty pages: 10932 , remaining dirty pages: 12789
> > Iteration 25, duration: 1300 ms , transferred pages: 10936 (dup: 302, rd: 6205, fd: 4429) , new dirty pages: 8713 , remaining dirty pages: 10566
> > Iteration 26, duration: 1000 ms , transferred pages: 8282 (dup: 102, rd: 5662, fd: 2518) , new dirty pages: 5119 , remaining dirty pages: 7403
> > Iteration 27, duration: 600 ms , transferred pages: 5007 (dup: 99, rd: 4099, fd: 809) , new dirty pages: 2226 , remaining dirty pages: 4622
> > Iteration 28, duration: 300 ms , transferred pages: 2491 (dup: 37, rd: 1794, fd: 660) , new dirty pages: 6746 , remaining dirty pages: 8877
> > Iteration 29, duration: 800 ms , transferred pages: 6757 (dup: 213, rd: 5532, fd: 1012) , new dirty pages: 6070 , remaining dirty pages: 8190
> > Iteration 30, duration: 700 ms , transferred pages: 6052 (dup: 326, rd: 5107, fd: 619) , new dirty pages: 5177 , remaining dirty pages: 7315
> >
> > 7. web server
> >
> > Iteration 1, duration: 20902 ms , transferred pages: 266450 (dup: 95497, rd: 170953, fd: 0) , new dirty pages: 8528 , remaining dirty pages: 8528
> > Iteration 2, duration: 796 ms , transferred pages: 6472 (dup: 131, rd: 1885, fd: 4456) , new dirty pages: 650 , remaining dirty pages: 2706
> > Iteration 3, duration: 100 ms , transferred pages: 818 (dup: 0, rd: 383, fd: 435) , new dirty pages: 328 , remaining dirty pages: 2216
> > Iteration 4, duration: 0 ms , transferred pages: 0 (dup: 0, rd: 0, fd: 0) , new dirty pages: 0 , remaining dirty pages: 2216
> >
> >
> > 8. cpu2006.bwaves (can not converge)
> >
> > Iteration 1, duration: 31715 ms , transferred pages: 266450 (dup: 6766, rd: 259684, fd: 0) , new dirty pages: 242702 , remaining dirty pages: 242702
> > Iteration 2, duration: 29397 ms , transferred pages: 240508 (dup: 405, rd: 225588, fd: 14515) , new dirty pages: 230889 , remaining dirty pages: 233083
> > Iteration 3, duration: 28205 ms , transferred pages: 230858 (dup: 182, rd: 214596, fd: 16080) , new dirty pages: 226998 , remaining dirty pages: 229223
> > Iteration 4, duration: 27805 ms , transferred pages: 227574 (dup: 170, rd: 217045, fd: 10359) , new dirty pages: 227360 , remaining dirty pages: 229009
> > Iteration 5, duration: 27703 ms , transferred pages: 226786 (dup: 200, rd: 212130, fd: 14456) , new dirty pages: 225885 , remaining dirty pages: 228108
> > Iteration 6, duration: 27600 ms , transferred pages: 225923 (dup: 155, rd: 215503, fd: 10265) , new dirty pages: 223555 , remaining dirty pages: 225740
> > Iteration 7, duration: 27309 ms , transferred pages: 223574 (dup: 260, rd: 215641, fd: 7673) , new dirty pages: 231975 , remaining dirty pages: 234141
> > Iteration 8, duration: 28403 ms , transferred pages: 232397 (dup: 85, rd: 214086, fd: 18226) , new dirty pages: 222170 , remaining dirty pages: 223914
> > Iteration 9, duration: 27105 ms , transferred pages: 221809 (dup: 131, rd: 214988, fd: 6690) , new dirty pages: 230065 , remaining dirty pages: 232170
> > Iteration 10, duration: 28104 ms , transferred pages: 230201 (dup: 343, rd: 213531, fd: 16327) , new dirty pages: 227590 , remaining dirty pages: 229559
> > Iteration 11, duration: 27801 ms , transferred pages: 227717 (dup: 313, rd: 221408, fd: 5996) , new dirty pages: 228457 , remaining dirty pages: 230299
> > Iteration 12, duration: 27916 ms , transferred pages: 228560 (dup: 338, rd: 219660, fd: 8562) , new dirty pages: 238326 , remaining dirty pages: 240065
> >
> > 9. cpu2006.lbm (can not converge)
> > Iteration 1, duration: 31012 ms , transferred pages: 266450 (dup: 12253, rd: 254197, fd: 0) , new dirty pages: 108960 , remaining dirty pages: 108960
> > Iteration 2, duration: 13095 ms , transferred pages: 106522 (dup: 3, rd: 102045, fd: 4474) , new dirty pages: 129292 , remaining dirty pages: 131730
> > Iteration 3, duration: 15802 ms , transferred pages: 129688 (dup: 444, rd: 110860, fd: 18384) , new dirty pages: 116682 , remaining dirty pages: 118724
> > Iteration 4, duration: 14204 ms , transferred pages: 116316 (dup: 160, rd: 104951, fd: 11205) , new dirty pages: 107246 , remaining dirty pages: 109654
> > Iteration 5, duration: 13208 ms , transferred pages: 107977 (dup: 1, rd: 101834, fd: 6142) , new dirty pages: 105371 , remaining dirty pages: 107048
> > Iteration 6, duration: 12804 ms , transferred pages: 104705 (dup: 1, rd: 99629, fd: 5075) , new dirty pages: 103841 , remaining dirty pages: 106184
> > Iteration 7, duration: 12709 ms , transferred pages: 103891 (dup: 5, rd: 99212, fd: 4674) , new dirty pages: 106692 , remaining dirty pages: 108985
> > Iteration 8, duration: 13105 ms , transferred pages: 107169 (dup: 11, rd: 100125, fd: 7033) , new dirty pages: 103132 , remaining dirty pages: 104948
> > Iteration 9, duration: 12607 ms , transferred pages: 103068 (dup: 0, rd: 99460, fd: 3608) , new dirty pages: 102511 , remaining dirty pages: 104391
> > Iteration 10, duration: 12514 ms , transferred pages: 102250 (dup: 0, rd: 99094, fd: 3156) , new dirty pages: 102888 , remaining dirty pages: 105029
> >
> > 10. cpu2006.astar (can not converge)
> >
> > Iteration 1, duration: 28402 ms , transferred pages: 266450 (dup: 33770, rd: 232680, fd: 0) , new dirty pages: 62078 , remaining dirty pages: 62078
> > Iteration 2, duration: 7393 ms , transferred pages: 60107 (dup: 10, rd: 51722, fd: 8375) , new dirty pages: 48854 , remaining dirty pages: 50825
> > Iteration 3, duration: 6001 ms , transferred pages: 49094 (dup: 14, rd: 46540, fd: 2540) , new dirty pages: 48137 , remaining dirty pages: 49868
> > Iteration 4, duration: 5800 ms , transferred pages: 47444 (dup: 0, rd: 45389, fd: 2055) , new dirty pages: 49147 , remaining dirty pages: 51571
> > Iteration 5, duration: 6102 ms , transferred pages: 49912 (dup: 14, rd: 46216, fd: 3682) , new dirty pages: 55606 , remaining dirty pages: 57265
> > Iteration 6, duration: 6699 ms , transferred pages: 54949 (dup: 143, rd: 20745, fd: 34061) , new dirty pages: 9166 , remaining dirty pages: 11482
> > Iteration 7, duration: 1200 ms , transferred pages: 9830 (dup: 14, rd: 7011, fd: 2805) , new dirty pages: 8294 , remaining dirty pages: 9946
> > Iteration 8, duration: 1000 ms , transferred pages: 8194 (dup: 14, rd: 7178, fd: 1002) , new dirty pages: 5475 , remaining dirty pages: 7227
> > Iteration 9, duration: 600 ms , transferred pages: 4908 (dup: 0, rd: 3470, fd: 1438) , new dirty pages: 4175 , remaining dirty pages: 6494
> > Iteration 10, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3856, fd: 234) , new dirty pages: 4095 , remaining dirty pages: 6499
> > Iteration 11, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 3313, fd: 777) , new dirty pages: 3371 , remaining dirty pages: 5780
> > Iteration 12, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3823, fd: 267) , new dirty pages: 7518 , remaining dirty pages: 9208
> > Iteration 13, duration: 899 ms , transferred pages: 7376 (dup: 14, rd: 6028, fd: 1334) , new dirty pages: 3931 , remaining dirty pages: 5763
> > Iteration 14, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 4346 , remaining dirty pages: 6019
> > Iteration 15, duration: 502 ms , transferred pages: 4090 (dup: 0, rd: 3817, fd: 273) , new dirty pages: 3054 , remaining dirty pages: 4983
> > Iteration 16, duration: 400 ms , transferred pages: 3272 (dup: 0, rd: 3138, fd: 134) , new dirty pages: 3874 , remaining dirty pages: 5585
> > Iteration 17, duration: 399 ms , transferred pages: 3272 (dup: 0, rd: 3248, fd: 24) , new dirty pages: 5285 , remaining dirty pages: 7598
> > Iteration 18, duration: 701 ms , transferred pages: 5726 (dup: 0, rd: 4385, fd: 1341) , new dirty pages: 8903 , remaining dirty pages: 10775
> > Iteration 19, duration: 1101 ms , transferred pages: 9010 (dup: 12, rd: 5597, fd: 3401) , new dirty pages: 4199 , remaining dirty pages: 5964
> > Iteration 20, duration: 500 ms , transferred pages: 4090 (dup: 0, rd: 4078, fd: 12) , new dirty pages: 3829 , remaining dirty pages: 5703
> >
> > 11. cpu2006.xalancbmk (can not converge)
> >
> > Iteration 1, duration: 30407 ms , transferred pages: 266450 (dup: 17700, rd: 248750, fd: 0) , new dirty pages: 96169 , remaining dirty pages: 96169
> > Iteration 2, duration: 11495 ms , transferred pages: 94164 (dup: 205, rd: 67068, fd: 26891) , new dirty pages: 61766 , remaining dirty pages: 63771
> > Iteration 3, duration: 7501 ms , transferred pages: 61471 (dup: 121, rd: 53587, fd: 7763) , new dirty pages: 56569 , remaining dirty pages: 58869
> > Iteration 4, duration: 6902 ms , transferred pages: 56461 (dup: 19, rd: 50553, fd: 5889) , new dirty pages: 52181 , remaining dirty pages: 54589
> > Iteration 5, duration: 6402 ms , transferred pages: 52459 (dup: 107, rd: 46986, fd: 5366) , new dirty pages: 54051 , remaining dirty pages: 56181
> > Iteration 6, duration: 6601 ms , transferred pages: 54003 (dup: 15, rd: 47566, fd: 6422) , new dirty pages: 50844 , remaining dirty pages: 53022
> > Iteration 7, duration: 6202 ms , transferred pages: 50723 (dup: 7, rd: 47143, fd: 3573) , new dirty pages: 64880 , remaining dirty pages: 67179
> > Iteration 8, duration: 8001 ms , transferred pages: 65447 (dup: 7, rd: 61159, fd: 4281) , new dirty pages: 67854 , remaining dirty pages: 69586
> > Iteration 9, duration: 8202 ms , transferred pages: 67444 (dup: 368, rd: 56357, fd: 10719) , new dirty pages: 65178 , remaining dirty pages: 67320
> > Iteration 10, duration: 8000 ms , transferred pages: 65455 (dup: 15, rd: 60581, fd: 4859) , new dirty pages: 52421 , remaining dirty pages: 54286
> >
> > 12. cpu2006.milc (can not converge)
> >
> > Iteration 1, duration: 31410 ms , transferred pages: 266450 (dup: 9454, rd: 256996, fd: 0) , new dirty pages: 158860 , remaining dirty pages: 158860
> > Iteration 2, duration: 19193 ms , transferred pages: 157048 (dup: 150, rd: 96807, fd: 60091) , new dirty pages: 102238 , remaining dirty pages: 104050
> > Iteration 3, duration: 12504 ms , transferred pages: 102271 (dup: 21, rd: 95107, fd: 7143) , new dirty pages: 97944 , remaining dirty pages: 99723
> > Iteration 4, duration: 11905 ms , transferred pages: 97360 (dup: 18, rd: 93610, fd: 3732) , new dirty pages: 99150 , remaining dirty pages: 101513
> > Iteration 5, duration: 12105 ms , transferred pages: 99094 (dup: 116, rd: 94125, fd: 4853) , new dirty pages: 98589 , remaining dirty pages: 101008
> > Iteration 6, duration: 12101 ms , transferred pages: 98995 (dup: 17, rd: 94069, fd: 4909) , new dirty pages: 147403 , remaining dirty pages: 149416
> > Iteration 7, duration: 18001 ms , transferred pages: 147284 (dup: 44, rd: 135691, fd: 11549) , new dirty pages: 136445 , remaining dirty pages: 138577
> > Iteration 8, duration: 16702 ms , transferred pages: 136636 (dup: 30, rd: 130805, fd: 5801) , new dirty pages: 145481 , remaining dirty pages: 147422
> > Iteration 9, duration: 17800 ms , transferred pages: 145734 (dup: 130, rd: 133239, fd: 12365) , new dirty pages: 98032 , remaining dirty pages: 99720
> > Iteration 10, duration: 11902 ms , transferred pages: 97364 (dup: 22, rd: 93096, fd: 4246) , new dirty pages: 95391 , remaining dirty pages: 97747
> >
> > 13. cpu2006.cactusADM (can not converge)
> >
> > Iteration 1, duration: 23508 ms , transferred pages: 266450 (dup: 73568, rd: 192882, fd: 0) , new dirty pages: 123869 , remaining dirty pages: 123869
> > Iteration 2, duration: 13989 ms , transferred pages: 121594 (dup: 7874, rd: 81653, fd: 32067) , new dirty pages: 112960 , remaining dirty pages: 115235
> > Iteration 3, duration: 13605 ms , transferred pages: 113276 (dup: 2028, rd: 83783, fd: 27465) , new dirty pages: 112314 , remaining dirty pages: 114273
> > Iteration 4, duration: 13509 ms , transferred pages: 111935 (dup: 1505, rd: 83535, fd: 26895) , new dirty pages: 114078 , remaining dirty pages: 116416
> > Iteration 5, duration: 13810 ms , transferred pages: 114262 (dup: 1378, rd: 84039, fd: 28845) , new dirty pages: 112271 , remaining dirty pages: 114425
> > Iteration 6, duration: 13604 ms , transferred pages: 112664 (dup: 1416, rd: 84300, fd: 26948) , new dirty pages: 112903 , remaining dirty pages: 114664
> > Iteration 7, duration: 13604 ms , transferred pages: 112655 (dup: 1407, rd: 84027, fd: 27221) , new dirty pages: 110943 , remaining dirty pages: 112952
> > Iteration 8, duration: 13406 ms , transferred pages: 110720 (dup: 1108, rd: 84075, fd: 25537) , new dirty pages: 109321 , remaining dirty pages: 111553
> > Iteration 9, duration: 13306 ms , transferred pages: 109726 (dup: 932, rd: 83652, fd: 25142) , new dirty pages: 113446 , remaining dirty pages: 115273
> > Iteration 10, duration: 13705 ms , transferred pages: 113121 (dup: 1055, rd: 84671, fd: 27395) , new dirty pages: 108776 , remaining dirty pages: 110928
> >
> > 14. cpu2006.GmesFDTD (can not converge)
> >
> > Iteration 1, duration: 13303 ms , transferred pages: 266450 (dup: 157809, rd: 108641, fd: 0) , new dirty pages: 226802 , remaining dirty pages: 226802
> > Iteration 2, duration: 10797 ms , transferred pages: 226507 (dup: 138637, rd: 61818, fd: 26052) , new dirty pages: 200769 , remaining dirty pages: 201064
> > Iteration 3, duration: 8900 ms , transferred pages: 199717 (dup: 127187, rd: 69340, fd: 3190) , new dirty pages: 203436 , remaining dirty pages: 204783
> > Iteration 4, duration: 10904 ms , transferred pages: 204127 (dup: 115211, rd: 85767, fd: 3149) , new dirty pages: 198407 , remaining dirty pages: 199063
> > Iteration 5, duration: 12109 ms , transferred pages: 198206 (dup: 99435, rd: 96956, fd: 1815) , new dirty pages: 213719 , remaining dirty pages: 214576
> > Iteration 6, duration: 16307 ms , transferred pages: 213595 (dup: 80422, rd: 116885, fd: 16288) , new dirty pages: 199637 , remaining dirty pages: 200618
> > Iteration 7, duration: 16915 ms , transferred pages: 198289 (dup: 60169, rd: 134208, fd: 3912) , new dirty pages: 199343 , remaining dirty pages: 201672
> > Iteration 8, duration: 19518 ms , transferred pages: 200452 (dup: 41014, rd: 156083, fd: 3355) , new dirty pages: 222927 , remaining dirty pages: 224147
> >
> > 15. cpu2006.wrf (can not converge)
> >
> > Iteration 1, duration: 18499 ms , transferred pages: 266380 (dup: 115285, rd: 151095, fd: 0) , new dirty pages: 112322 , remaining dirty pages: 112392
> > Iteration 2, duration: 9802 ms , transferred pages: 110025 (dup: 29917, rd: 65782, fd: 14326) , new dirty pages: 88855 , remaining dirty pages: 91222
> > Iteration 3, duration: 8199 ms , transferred pages: 89761 (dup: 22728, rd: 57262, fd: 9771) , new dirty pages: 58431 , remaining dirty pages: 59892
> > Iteration 4, duration: 5603 ms , transferred pages: 58502 (dup: 12716, rd: 41809, fd: 3977) , new dirty pages: 80556 , remaining dirty pages: 81946
> > Iteration 5, duration: 7101 ms , transferred pages: 79778 (dup: 21738, rd: 50896, fd: 7144) , new dirty pages: 62592 , remaining dirty pages: 64760
> > Iteration 6, duration: 5702 ms , transferred pages: 63388 (dup: 16793, rd: 42726, fd: 3869) , new dirty pages: 80747 , remaining dirty pages: 82119
> > Iteration 7, duration: 7000 ms , transferred pages: 80868 (dup: 23652, rd: 52194, fd: 5022) , new dirty pages: 84593 , remaining dirty pages: 85844
> > Iteration 8, duration: 7099 ms , transferred pages: 83799 (dup: 25769, rd: 51772, fd: 6258) , new dirty pages: 67951 , remaining dirty pages: 69996
> > Iteration 9, duration: 6303 ms , transferred pages: 68478 (dup: 16979, rd: 36490, fd: 15009) , new dirty pages: 81181 , remaining dirty pages: 82699
> > Iteration 10, duration: 7000 ms , transferred pages: 80724 (dup: 23503, rd: 52826, fd: 4395) , new dirty pages: 47930 , remaining dirty pages: 49905
> >
> >
> >
> > >
> > > > So I think "booting" and "kernel compilation" should benefit a lot from this
> > > > improvement. The reason of "kernel compilation" would benefit is that some
> > > > iterations take around 600ms, and if they are halved into 300ms, then the precopy
> > > > may have the chance to step into stop and copy phase.
> > > >
> > > > On the other hand, "idle" and "web server" would not benefit a lot, because
> > > > most of the time are spent on the 1st iteration and little on the others.
> > > >
> > > > As to the "zeusmp" and "memcached", although the time spent on the other iterations
> > > > but the 1st one may be halved, they still could not converge to stop and copy
> > > > with the 300ms downtime.
> > > >
> > > > --------------------1 vcpu, 1 GB ram, default bandwidth (32MB/s):------------------
> > > >
> > > > 1. booting : begin to migrate when the VM is booting
> > > >
> > > > Iteration 1, duration: 6997 ms , transferred pages: 266450 (n: 57269, d: 209181 ) , new dirty pages: 56414 , remaining dirty pages: 56414
> > > > Iteration 2, duration: 6497 ms , transferred pages: 54008 (n: 52701, d: 1307 ) , new dirty pages: 48053 , remaining dirty pages: 50459
> > > > Iteration 3, duration: 5800 ms , transferred pages: 48232 (n: 47444, d: 788 ) , new dirty pages: 9129 , remaining dirty pages: 11356
> > > > Iteration 4, duration: 1100 ms , transferred pages: 9091 (n: 8998, d: 93 ) , new dirty pages: 165 , remaining dirty pages: 2430
> > > > Iteration 5, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2430
> > > > (note: When the workload does converge, the output of the last iteration is "fake". It just indicates that the precopy steps into stop-copy phase now.
> > > > "n" means "normal pages" and "d" means "duplicate (zero) pages".)
> > > >
> > > > 2. idle
> > > >
> > > > Iteration 1, duration: 14496 ms , transferred pages: 266450 (n: 118980, d: 147470 ) , new dirty pages: 17398 , remaining dirty pages: 17398
> > > > Iteration 2, duration: 1896 ms , transferred pages: 14953 (n: 14854, d: 99 ) , new dirty pages: 1849 , remaining dirty pages: 4294
> > > > Iteration 3, duration: 300 ms , transferred pages: 2454 (n: 2454, d: 0 ) , new dirty pages: 9 , remaining dirty pages: 1849
> > > > Iteration 4, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 1849
> > > >
> > > > 3. kernel compilation (can not converge)
> > > >
> > > > Iteration 1, duration: 20700 ms , transferred pages: 266450 (n: 169778, d: 96672 ) , new dirty pages: 40067 , remaining dirty pages: 40067
> > > > Iteration 2, duration: 4696 ms , transferred pages: 38401 (n: 37787, d: 614 ) , new dirty pages: 8852 , remaining dirty pages: 10518
> > > > Iteration 3, duration: 1000 ms , transferred pages: 8642 (n: 8180, d: 462 ) , new dirty pages: 6331 , remaining dirty pages: 8207
> > > > Iteration 4, duration: 700 ms , transferred pages: 6110 (n: 5726, d: 384 ) , new dirty pages: 5242 , remaining dirty pages: 7339
> > > > Iteration 5, duration: 600 ms , transferred pages: 5007 (n: 4908, d: 99 ) , new dirty pages: 4868 , remaining dirty pages: 7200
> > > > Iteration 6, duration: 600 ms , transferred pages: 5226 (n: 4908, d: 318 ) , new dirty pages: 6142 , remaining dirty pages: 8116
> > > > Iteration 7, duration: 700 ms , transferred pages: 5985 (n: 5726, d: 259 ) , new dirty pages: 5902 , remaining dirty pages: 8033
> > > > Iteration 8, duration: 701 ms , transferred pages: 5893 (n: 5726, d: 167 ) , new dirty pages: 7502 , remaining dirty pages: 9642
> > > > Iteration 9, duration: 900 ms , transferred pages: 7623 (n: 7362, d: 261 ) , new dirty pages: 6408 , remaining dirty pages: 8427
> > > > Iteration 10, duration: 700 ms , transferred pages: 6008 (n: 5726, d: 282 ) , new dirty pages: 8312 , remaining dirty pages: 10731
> > > > Iteration 11, duration: 1000 ms , transferred pages: 8353 (n: 8180, d: 173 ) , new dirty pages: 6874 , remaining dirty pages: 9252
> > > > Iteration 12, duration: 899 ms , transferred pages: 7477 (n: 7362, d: 115 ) , new dirty pages: 5573 , remaining dirty pages: 7348
> > > > Iteration 13, duration: 601 ms , transferred pages: 5099 (n: 4908, d: 191 ) , new dirty pages: 7671 , remaining dirty pages: 9920
> > > > Iteration 14, duration: 900 ms , transferred pages: 7586 (n: 7362, d: 224 ) , new dirty pages: 7359 , remaining dirty pages: 9693
> > > > Iteration 15, duration: 900 ms , transferred pages: 7682 (n: 7362, d: 320 ) , new dirty pages: 7371 , remaining dirty pages: 9382
> > > >
> > > > 4. cpu2006.zeusmp (can not converge)
> > > >
> > > > Iteration 1, duration: 21603 ms , transferred pages: 266450 (n: 176660, d: 89790 ) , new dirty pages: 145625 , remaining dirty pages: 145625
> > > > Iteration 2, duration: 8696 ms , transferred pages: 144389 (n: 70862, d: 73527 ) , new dirty pages: 125124 , remaining dirty pages: 126360
> > > > Iteration 3, duration: 6301 ms , transferred pages: 124057 (n: 51379, d: 72678 ) , new dirty pages: 122528 , remaining dirty pages: 124831
> > > > Iteration 4, duration: 6400 ms , transferred pages: 124330 (n: 52196, d: 72134 ) , new dirty pages: 124267 , remaining dirty pages: 124768
> > > > Iteration 5, duration: 6703 ms , transferred pages: 124034 (n: 54656, d: 69378 ) , new dirty pages: 124151 , remaining dirty pages: 124885
> > > > Iteration 6, duration: 6703 ms , transferred pages: 124357 (n: 54658, d: 69699 ) , new dirty pages: 124106 , remaining dirty pages: 124634
> > > > Iteration 7, duration: 6602 ms , transferred pages: 124568 (n: 53838, d: 70730 ) , new dirty pages: 133828 , remaining dirty pages: 133894
> > > > Iteration 8, duration: 7600 ms , transferred pages: 133030 (n: 62021, d: 71009 ) , new dirty pages: 126612 , remaining dirty pages: 127476
> > > > Iteration 9, duration: 7299 ms , transferred pages: 126511 (n: 59569, d: 66942 ) , new dirty pages: 122727 , remaining dirty pages: 123692
> > > > Iteration 10, duration: 6609 ms , transferred pages: 123692 (n: 54539, d: 69153 ) , new dirty pages: 122727 , remaining dirty pages: 122727
> > > > Iteration 11, duration: 6995 ms , transferred pages: 120347 (n: 56423, d: 63924 ) , new dirty pages: 121430 , remaining dirty pages: 123810
> > > > Iteration 12, duration: 6703 ms , transferred pages: 123040 (n: 54657, d: 68383 ) , new dirty pages: 122043 , remaining dirty pages: 122813
> > > > Iteration 13, duration: 7006 ms , transferred pages: 122353 (n: 57121, d: 65232 ) , new dirty pages: 133869 , remaining dirty pages: 134329
> > > > Iteration 14, duration: 8209 ms , transferred pages: 132325 (n: 66932, d: 65393 ) , new dirty pages: 126914 , remaining dirty pages: 128918
> > > > Iteration 15, duration: 7802 ms , transferred pages: 126931 (n: 63671, d: 63260 ) , new dirty pages: 122351 , remaining dirty pages: 124338
> > > >
> > > > 5. web server : An apache web server. The client is configured with 50 concurrent connections.
> > > >
> > > > Iteration 1, duration: 30697 ms , transferred pages: 266450 (n: 251215, d: 15235 ) , new dirty pages: 30628 , remaining dirty pages: 30628
> > > > Iteration 2, duration: 3496 ms , transferred pages: 28859 (n: 28513, d: 346 ) , new dirty pages: 5805 , remaining dirty pages: 7574
> > > > Iteration 3, duration: 701 ms , transferred pages: 5746 (n: 5726, d: 20 ) , new dirty pages: 3433 , remaining dirty pages: 5261
> > > > Iteration 4, duration: 400 ms , transferred pages: 3281 (n: 3272, d: 9 ) , new dirty pages: 1539 , remaining dirty pages: 3519
> > > > Iteration 5, duration: 199 ms , transferred pages: 1653 (n: 1636, d: 17 ) , new dirty pages: 301 , remaining dirty pages: 2167
> > > > Iteration 6, duration: 1 ms , transferred pages: 0 (n: 0, d: 0 ) , new dirty pages: 0 , remaining dirty pages: 2167
> > > >
> > > > --------------------6 vcpu, 6 GB ram, max bandwidth (941.08 mbps):------------------
> > > >
> > > > 6. memcached : 4 GB cache, memaslap: all write, concurrency = 5 (can not converge)
> > > >
> > > > Iteration 1, duration: 42486 ms , transferred pages: 1568087 (n: 1216079, d: 352008 ) , new dirty pages: 571940 , remaining dirty pages: 581023
> > > > Iteration 2, duration: 19774 ms , transferred pages: 571700 (n: 567416, d: 4284 ) , new dirty pages: 331690 , remaining dirty pages: 341013
> > > > Iteration 3, duration: 11589 ms , transferred pages: 332187 (n: 332095, d: 92 ) , new dirty pages: 222725 , remaining dirty pages: 231551
> > > > Iteration 4, duration: 7790 ms , transferred pages: 223571 (n: 223499, d: 72 ) , new dirty pages: 157658 , remaining dirty pages: 165638
> > > > Iteration 5, duration: 5518 ms , transferred pages: 158056 (n: 157998, d: 58 ) , new dirty pages: 128130 , remaining dirty pages: 135712
> > > > Iteration 6, duration: 4442 ms , transferred pages: 127764 (n: 127701, d: 63 ) , new dirty pages: 104839 , remaining dirty pages: 112787
> > > > Iteration 7, duration: 3649 ms , transferred pages: 104581 (n: 104523, d: 58 ) , new dirty pages: 100736 , remaining dirty pages: 108942
> > > > Iteration 8, duration: 3532 ms , transferred pages: 101379 (n: 101315, d: 64 ) , new dirty pages: 87869 , remaining dirty pages: 95432
> > > > Iteration 9, duration: 3030 ms , transferred pages: 86841 (n: 86786, d: 55 ) , new dirty pages: 77505 , remaining dirty pages: 86096
> > > > Iteration 10, duration: 2709 ms , transferred pages: 77875 (n: 77814, d: 61 ) , new dirty pages: 77197 , remaining dirty pages: 85418
> > > > Iteration 11, duration: 2696 ms , transferred pages: 77107 (n: 77044, d: 63 ) , new dirty pages: 65010 , remaining dirty pages: 73321
> > > > Iteration 12, duration: 2308 ms , transferred pages: 66540 (n: 66484, d: 56 ) , new dirty pages: 64388 , remaining dirty pages: 71169
> > > > Iteration 13, duration: 2198 ms , transferred pages: 62953 (n: 62897, d: 56 ) , new dirty pages: 62773 , remaining dirty pages: 70989
> > > > Iteration 14, duration: 2214 ms , transferred pages: 63466 (n: 63411, d: 55 ) , new dirty pages: 67538 , remaining dirty pages: 75061
> > > > Iteration 15, duration: 2329 ms , transferred pages: 66924 (n: 66875, d: 49 ) , new dirty pages: 63580 , remaining dirty pages: 71717
> > > > Iteration 16, duration: 2252 ms , transferred pages: 64554 (n: 64539, d: 15 ) , new dirty pages: 63094 , remaining dirty pages: 70257
> > > > Iteration 17, duration: 2188 ms , transferred pages: 62697 (n: 62641, d: 56 ) , new dirty pages: 63016 , remaining dirty pages: 70576
> > > > Iteration 18, duration: 2171 ms , transferred pages: 62377 (n: 62322, d: 55 ) , new dirty pages: 56764 , remaining dirty pages: 64963
> > > > Iteration 19, duration: 2003 ms , transferred pages: 57382 (n: 57324, d: 58 ) , new dirty pages: 65307 , remaining dirty pages: 72888
> > > > Iteration 20, duration: 2240 ms , transferred pages: 64426 (n: 64364, d: 62 ) , new dirty pages: 61585 , remaining dirty pages: 70047
> > > >
> > > >
> > > > --
> > > > Chunguang Li, Ph.D. Candidate
> > > > Wuhan National Laboratory for Optoelectronics (WNLO)
> > > > Huazhong University of Science & Technology (HUST)
> > > > Wuhan, Hubei Prov., China
> > > >
> > > >
> > > >
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >
> > --
> > Chunguang Li, Ph.D. Candidate
> > Wuhan National Laboratory for Optoelectronics (WNLO)
> > Huazhong University of Science & Technology (HUST)
> > Wuhan, Hubei Prov., China
> >
> >
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2016-11-08 13:32 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-25 8:22 [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent Chunguang Li
2016-09-26 11:23 ` Dr. David Alan Gilbert
2016-09-26 14:55 ` Chunguang Li
2016-09-26 18:52 ` Dr. David Alan Gilbert
2016-09-27 12:28 ` Chunguang Li
2016-09-30 5:46 ` Amit Shah
2016-09-30 8:18 ` Chunguang Li
2016-10-08 7:55 ` Chunguang Li
2016-10-14 11:15 ` Dr. David Alan Gilbert
2016-11-03 8:25 ` Chunguang Li
2016-11-03 9:59 ` Li, Liang Z
2016-11-03 10:13 ` Li, Liang Z
2016-11-04 3:07 ` Chunguang Li
2016-11-04 4:50 ` Li, Liang Z
2016-11-04 7:03 ` Chunguang Li
2016-11-07 13:52 ` Chunguang Li
2016-11-07 14:17 ` Li, Liang Z
2016-11-08 5:27 ` Chunguang Li
2016-11-07 14:44 ` Li, Liang Z
2016-11-08 11:05 ` Dr. David Alan Gilbert
2016-11-08 13:40 ` Chunguang Li
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.