[Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[FENG, Jiasheng]

Dear QEMU Development Team,


It is my honor to contact with you.



I am a postgraduate student from University of Hong Kong. Currently I am
working on a project related to QEMU MicroCheckpointing and I have
encountered a performance issue during checkpoint pause & resume.



Please kindly refer to migration/checkpoint.c file, in function
capture_checkpoint, I proceeded a test to see the time consumption between
vm_stop_force_state and vm_start. I found out that even if the system is
idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
Moreover, latency will be increased while more cpus equipped by my virtual
machine. I have done some research on that and I realized that it is
related to the Memory Barrier in KVM kernel. Each cpu will proceed a
smp_wmb() request during pause & resume and it takes about  3-5ms to finish
the request ( mem=2G, vCPU=4 ).



Therefore, I would like to ask 3 questions regarding on the above issue:


1. What is your consideration with calling smp_wmb() in checkpoint period;

2. Is it any other solution to minimize the latency to improve the
performance in checkpoint period;

3. Is smp_wmb() able to be safely disabled during the checkpoint period



Really appreciate your help with my problems and hope to receive your
feedback soon.



Thanks again for your contribution to QEMU and it is such a masterpiece.



Thanks and best regards,

Niko Jiasheng Feng

University of Hong Kong

-- 
*Niko Jiasheng *
*Feng **Computer Science(General Stream), Faculty of Engineering, The
University of Hong Kong*
Contact:  （852）97908620
Address: Pokfulam Road, The University of Hong Kong
Email:      nikofeng@hku.hk / niko_jiasheng@163.com

Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[Dr. David Alan Gilbert]

* FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> Dear QEMU Development Team,
> 
> 
> It is my honor to contact with you.
> 
> 
> 
> I am a postgraduate student from University of Hong Kong. Currently I am
> working on a project related to QEMU MicroCheckpointing and I have
> encountered a performance issue during checkpoint pause & resume.

The microcheckpointing code hasn't been maintained for a long time;
most of the current checkpointing work is based on the COLO work which is
still under development.

> Please kindly refer to migration/checkpoint.c file, in function
> capture_checkpoint, I proceeded a test to see the time consumption between
> vm_stop_force_state and vm_start. I found out that even if the system is
> idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
> Moreover, latency will be increased while more cpus equipped by my virtual
> machine. I have done some research on that and I realized that it is
> related to the Memory Barrier in KVM kernel. Each cpu will proceed a
> smp_wmb() request during pause & resume and it takes about  3-5ms to finish
> the request ( mem=2G, vCPU=4 ).
> 
> 
> 
> Therefore, I would like to ask 3 questions regarding on the above issue:
> 
> 
> 1. What is your consideration with calling smp_wmb() in checkpoint period;
> 
> 2. Is it any other solution to minimize the latency to improve the
> performance in checkpoint period;
> 
> 3. Is smp_wmb() able to be safely disabled during the checkpoint period

Well you'd have to understand where it's used; but for example, when taking
a checkpoint you'd want to be sure that the checkpoint data contained
a consistent copy of the last write data from all of the vCPUs; so I think
a wmb would be needed to make sure it's consistent.

I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
time, and that it's quite so long.  
Are the vCPUs idle or are they busy - does it make difference?

Dave

> Really appreciate your help with my problems and hope to receive your
> feedback soon.
> 
> 
> Thanks again for your contribution to QEMU and it is such a masterpiece.

Dave

> 
> 
> 
> Thanks and best regards,
> 
> Niko Jiasheng Feng
> 
> University of Hong Kong
> 
> -- 
> *Niko Jiasheng *
> *Feng **Computer Science(General Stream), Faculty of Engineering, The
> University of Hong Kong*
> Contact:  （852）97908620
> Address: Pokfulam Road, The University of Hong Kong
> Email:      nikofeng@hku.hk / niko_jiasheng@163.com
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[FENG, Jiasheng]

Dear David,

Really appreciate your feedback.

I have proceeded the experiments in both conditions, and no matter the
vCPUs are in idle or busy situation, there is no difference that smp_wmb()
will consume a lot of time to proceed its work.

In your opinion, may I know that what is the alternative way to minimize
the time consumption of smp_wmb() or any other system setting could speed
up smp_wmb()?

Thanks in advance for your assistance and hope to receive your feedback soon

Thanks and best regards,
Niko Jiasheng Feng



On Thu, Mar 9, 2017 at 11:19 PM, Dr. David Alan Gilbert <dgilbert@redhat.com
> wrote:

> * FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> > Dear QEMU Development Team,
> >
> >
> > It is my honor to contact with you.
> >
> >
> >
> > I am a postgraduate student from University of Hong Kong. Currently I am
> > working on a project related to QEMU MicroCheckpointing and I have
> > encountered a performance issue during checkpoint pause & resume.
>
> The microcheckpointing code hasn't been maintained for a long time;
> most of the current checkpointing work is based on the COLO work which is
> still under development.
>
> > Please kindly refer to migration/checkpoint.c file, in function
> > capture_checkpoint, I proceeded a test to see the time consumption
> between
> > vm_stop_force_state and vm_start. I found out that even if the system is
> > idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
> > Moreover, latency will be increased while more cpus equipped by my
> virtual
> > machine. I have done some research on that and I realized that it is
> > related to the Memory Barrier in KVM kernel. Each cpu will proceed a
> > smp_wmb() request during pause & resume and it takes about  3-5ms to
> finish
> > the request ( mem=2G, vCPU=4 ).
> >
> >
> >
> > Therefore, I would like to ask 3 questions regarding on the above issue:
> >
> >
> > 1. What is your consideration with calling smp_wmb() in checkpoint
> period;
> >
> > 2. Is it any other solution to minimize the latency to improve the
> > performance in checkpoint period;
> >
> > 3. Is smp_wmb() able to be safely disabled during the checkpoint period
>
> Well you'd have to understand where it's used; but for example, when taking
> a checkpoint you'd want to be sure that the checkpoint data contained
> a consistent copy of the last write data from all of the vCPUs; so I think
> a wmb would be needed to make sure it's consistent.
>
> I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
> time, and that it's quite so long.
> Are the vCPUs idle or are they busy - does it make difference?
>
> Dave
>
> > Really appreciate your help with my problems and hope to receive your
> > feedback soon.
> >
> >
> > Thanks again for your contribution to QEMU and it is such a masterpiece.
>
> Dave
>
> >
> >
> >
> > Thanks and best regards,
> >
> > Niko Jiasheng Feng
> >
> > University of Hong Kong
> >
> > --
> > *Niko Jiasheng *
> > *Feng **Computer Science(General Stream), Faculty of Engineering, The
> > University of Hong Kong*
> > Contact:  （852）97908620
> > Address: Pokfulam Road, The University of Hong Kong
> > Email:      nikofeng@hku.hk / niko_jiasheng@163.com
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>



-- 
*Niko Jiasheng *
*Feng **Computer Science(General Stream), Faculty of Engineering, The
University of Hong Kong*
Contact:  （852）97908620
Address: Pokfulam Road, The University of Hong Kong
Email:      nikofeng@hku.hk / niko_jiasheng@163.com

Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[Dr. David Alan Gilbert]

(cc'ing in Paolo since he knows our barrier code)

* FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> Dear David,
> 
> Really appreciate your feedback.
> 
> I have proceeded the experiments in both conditions, and no matter the
> vCPUs are in idle or busy situation, there is no difference that smp_wmb()
> will consume a lot of time to proceed its work.
> 
> In your opinion, may I know that what is the alternative way to minimize
> the time consumption of smp_wmb() or any other system setting could speed
> up smp_wmb()?
> 
> Thanks in advance for your assistance and hope to receive your feedback soon

Just checking, is this on a normal x86 PC?
Your numbers of 3-5ms just seem quite high to me but I've not tried timing that
code.

Dave

> 
> Thanks and best regards,
> Niko Jiasheng Feng
> 
> 
> 
> On Thu, Mar 9, 2017 at 11:19 PM, Dr. David Alan Gilbert <dgilbert@redhat.com
> > wrote:
> 
> > * FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> > > Dear QEMU Development Team,
> > >
> > >
> > > It is my honor to contact with you.
> > >
> > >
> > >
> > > I am a postgraduate student from University of Hong Kong. Currently I am
> > > working on a project related to QEMU MicroCheckpointing and I have
> > > encountered a performance issue during checkpoint pause & resume.
> >
> > The microcheckpointing code hasn't been maintained for a long time;
> > most of the current checkpointing work is based on the COLO work which is
> > still under development.
> >
> > > Please kindly refer to migration/checkpoint.c file, in function
> > > capture_checkpoint, I proceeded a test to see the time consumption
> > between
> > > vm_stop_force_state and vm_start. I found out that even if the system is
> > > idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
> > > Moreover, latency will be increased while more cpus equipped by my
> > virtual
> > > machine. I have done some research on that and I realized that it is
> > > related to the Memory Barrier in KVM kernel. Each cpu will proceed a
> > > smp_wmb() request during pause & resume and it takes about  3-5ms to
> > finish
> > > the request ( mem=2G, vCPU=4 ).
> > >
> > >
> > >
> > > Therefore, I would like to ask 3 questions regarding on the above issue:
> > >
> > >
> > > 1. What is your consideration with calling smp_wmb() in checkpoint
> > period;
> > >
> > > 2. Is it any other solution to minimize the latency to improve the
> > > performance in checkpoint period;
> > >
> > > 3. Is smp_wmb() able to be safely disabled during the checkpoint period
> >
> > Well you'd have to understand where it's used; but for example, when taking
> > a checkpoint you'd want to be sure that the checkpoint data contained
> > a consistent copy of the last write data from all of the vCPUs; so I think
> > a wmb would be needed to make sure it's consistent.
> >
> > I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
> > time, and that it's quite so long.
> > Are the vCPUs idle or are they busy - does it make difference?
> >
> > Dave
> >
> > > Really appreciate your help with my problems and hope to receive your
> > > feedback soon.
> > >
> > >
> > > Thanks again for your contribution to QEMU and it is such a masterpiece.
> >
> > Dave
> >
> > >
> > >
> > >
> > > Thanks and best regards,
> > >
> > > Niko Jiasheng Feng
> > >
> > > University of Hong Kong
> > >
> > > --
> > > *Niko Jiasheng *
> > > *Feng **Computer Science(General Stream), Faculty of Engineering, The
> > > University of Hong Kong*
> > > Contact:  （852）97908620
> > > Address: Pokfulam Road, The University of Hong Kong
> > > Email:      nikofeng@hku.hk / niko_jiasheng@163.com
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> 
> 
> 
> -- 
> *Niko Jiasheng *
> *Feng **Computer Science(General Stream), Faculty of Engineering, The
> University of Hong Kong*
> Contact:  （852）97908620
> Address: Pokfulam Road, The University of Hong Kong
> Email:      nikofeng@hku.hk / niko_jiasheng@163.com
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[nikofeng]

Dear David,

Yes, it is a normal x86 PC server.

Thanks so much for your help and hope to receive your following feedback.

Best Regards,
Niko Jiasheng Feng

Sent from Mail for Windows 10

From: Dr. David Alan Gilbert
Sent: Friday, March 10, 2017 1:06 AM
To: FENG, Jiasheng
Cc: qemu-devel@nongnu.org; Wang Cheng; YE, Chen; CHEN, XUSHENG; Heming Cui; pbonzini@redhat.com
Subject: Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

(cc'ing in Paolo since he knows our barrier code)

* FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> Dear David,
> 
> Really appreciate your feedback.
> 
> I have proceeded the experiments in both conditions, and no matter the
> vCPUs are in idle or busy situation, there is no difference that smp_wmb()
> will consume a lot of time to proceed its work.
> 
> In your opinion, may I know that what is the alternative way to minimize
> the time consumption of smp_wmb() or any other system setting could speed
> up smp_wmb()?
> 
> Thanks in advance for your assistance and hope to receive your feedback soon

Just checking, is this on a normal x86 PC?
Your numbers of 3-5ms just seem quite high to me but I've not tried timing that
code.

Dave

> 
> Thanks and best regards,
> Niko Jiasheng Feng
> 
> 
> 
> On Thu, Mar 9, 2017 at 11:19 PM, Dr. David Alan Gilbert <dgilbert@redhat.com
> > wrote:
> 
> > * FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
> > > Dear QEMU Development Team,
> > >
> > >
> > > It is my honor to contact with you.
> > >
> > >
> > >
> > > I am a postgraduate student from University of Hong Kong. Currently I am
> > > working on a project related to QEMU MicroCheckpointing and I have
> > > encountered a performance issue during checkpoint pause & resume.
> >
> > The microcheckpointing code hasn't been maintained for a long time;
> > most of the current checkpointing work is based on the COLO work which is
> > still under development.
> >
> > > Please kindly refer to migration/checkpoint.c file, in function
> > > capture_checkpoint, I proceeded a test to see the time consumption
> > between
> > > vm_stop_force_state and vm_start. I found out that even if the system is
> > > idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
> > > Moreover, latency will be increased while more cpus equipped by my
> > virtual
> > > machine. I have done some research on that and I realized that it is
> > > related to the Memory Barrier in KVM kernel. Each cpu will proceed a
> > > smp_wmb() request during pause & resume and it takes about  3-5ms to
> > finish
> > > the request ( mem=2G, vCPU=4 ).
> > >
> > >
> > >
> > > Therefore, I would like to ask 3 questions regarding on the above issue:
> > >
> > >
> > > 1. What is your consideration with calling smp_wmb() in checkpoint
> > period;
> > >
> > > 2. Is it any other solution to minimize the latency to improve the
> > > performance in checkpoint period;
> > >
> > > 3. Is smp_wmb() able to be safely disabled during the checkpoint period
> >
> > Well you'd have to understand where it's used; but for example, when taking
> > a checkpoint you'd want to be sure that the checkpoint data contained
> > a consistent copy of the last write data from all of the vCPUs; so I think
> > a wmb would be needed to make sure it's consistent.
> >
> > I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
> > time, and that it's quite so long.
> > Are the vCPUs idle or are they busy - does it make difference?
> >
> > Dave
> >
> > > Really appreciate your help with my problems and hope to receive your
> > > feedback soon.
> > >
> > >
> > > Thanks again for your contribution to QEMU and it is such a masterpiece.
> >
> > Dave
> >
> > >
> > >
> > >
> > > Thanks and best regards,
> > >
> > > Niko Jiasheng Feng
> > >
> > > University of Hong Kong
> > >
> > > --
> > > *Niko Jiasheng *
> > > *Feng **Computer Science(General Stream), Faculty of Engineering, The
> > > University of Hong Kong*
> > > Contact:  （852）97908620
> > > Address: Pokfulam Road, The University of Hong Kong
> > > Email:      nikofeng@hku.hk / niko_jiasheng@163.com
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> 
> 
> 
> -- 
> *Niko Jiasheng *
> *Feng **Computer Science(General Stream), Faculty of Engineering, The
> University of Hong Kong*
> Contact:  （852）97908620
> Address: Pokfulam Road, The University of Hong Kong
> Email:      nikofeng@hku.hk / niko_jiasheng@163.com
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[Paolo Bonzini]

On 09/03/2017 18:06, Dr. David Alan Gilbert wrote:
> (cc'ing in Paolo since he knows our barrier code)
> 
> * FENG, Jiasheng (nikofeng@connect.hku.hk) wrote:
>> Dear David,
>>
>> Really appreciate your feedback.
>>
>> I have proceeded the experiments in both conditions, and no matter the
>> vCPUs are in idle or busy situation, there is no difference that smp_wmb()
>> will consume a lot of time to proceed its work.
>>
>> In your opinion, may I know that what is the alternative way to minimize
>> the time consumption of smp_wmb() or any other system setting could speed
>> up smp_wmb()?
>>
>> Thanks in advance for your assistance and hope to receive your feedback soon
> 
> Just checking, is this on a normal x86 PC?
> Your numbers of 3-5ms just seem quite high to me but I've not tried timing that
> code.

smp_wmb does not produce a single machine instruction, so this is
probably a fluke in the profiling tool.

The most expensive part of vm_stop_force_state is going to be
bdrv_drain_all/bdrv_flush_all.  bdrv_flush_all is definitely not needed
for checkpointing purposes.

Paolo

>>>
>>>> Please kindly refer to migration/checkpoint.c file, in function
>>>> capture_checkpoint, I proceeded a test to see the time consumption
>>> between
>>>> vm_stop_force_state and vm_start. I found out that even if the system is
>>>> idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
>>>> Moreover, latency will be increased while more cpus equipped by my
>>> virtual
>>>> machine. I have done some research on that and I realized that it is
>>>> related to the Memory Barrier in KVM kernel. Each cpu will proceed a
>>>> smp_wmb() request during pause & resume and it takes about  3-5ms to
>>> finish
>>>> the request ( mem=2G, vCPU=4 ).
>>>>
>>>>
>>>>
>>>> Therefore, I would like to ask 3 questions regarding on the above issue:
>>>>
>>>>
>>>> 1. What is your consideration with calling smp_wmb() in checkpoint
>>> period;
>>>>
>>>> 2. Is it any other solution to minimize the latency to improve the
>>>> performance in checkpoint period;
>>>>
>>>> 3. Is smp_wmb() able to be safely disabled during the checkpoint period
>>>
>>> Well you'd have to understand where it's used; but for example, when taking
>>> a checkpoint you'd want to be sure that the checkpoint data contained
>>> a consistent copy of the last write data from all of the vCPUs; so I think
>>> a wmb would be needed to make sure it's consistent.
>>>
>>> I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
>>> time, and that it's quite so long.
>>> Are the vCPUs idle or are they busy - does it make difference?
>>>
>>> Dave
>>>
>>>> Really appreciate your help with my problems and hope to receive your
>>>> feedback soon.
>>>>
>>>>
>>>> Thanks again for your contribution to QEMU and it is such a masterpiece.
>>>
>>> Dave
>>>
>>>>
>>>>
>>>>
>>>> Thanks and best regards,
>>>>
>>>> Niko Jiasheng Feng
>>>>
>>>> University of Hong Kong
>>>>
>>>> --
>>>> *Niko Jiasheng *
>>>> *Feng **Computer Science(General Stream), Faculty of Engineering, The
>>>> University of Hong Kong*
>>>> Contact:  （852）97908620
>>>> Address: Pokfulam Road, The University of Hong Kong
>>>> Email:      nikofeng@hku.hk / niko_jiasheng@163.com
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>
>>
>>
>> -- 
>> *Niko Jiasheng *
>> *Feng **Computer Science(General Stream), Faculty of Engineering, The
>> University of Hong Kong*
>> Contact:  （852）97908620
>> Address: Pokfulam Road, The University of Hong Kong
>> Email:      nikofeng@hku.hk / niko_jiasheng@163.com
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>