* [PATCH v2] sched/cputime: add steal clock warp handling
@ 2016-06-03 5:21 Wanpeng Li
2016-06-03 13:10 ` Rik van Riel
0 siblings, 1 reply; 5+ messages in thread
From: Wanpeng Li @ 2016-06-03 5:21 UTC (permalink / raw)
To: linux-kernel, kvm
Cc: Wanpeng Li, Ingo Molnar, Peter Zijlstra (Intel),
Rik van Riel, Thomas Gleixner, Frederic Weisbecker,
Paolo Bonzini, Radim
From: Wanpeng Li <wanpeng.li@hotmail.com>
I observed that sometimes st is 100% instantaneous, then idle is 100%
even if there is a cpu hog on the guest cpu after the cpu hotplug comes
back(N.B. this can not always be readily reproduced). I add trace to
capture it as below:
cpuhp/1-12 [001] d.h1 167.461657: account_process_tick: steal = 1291385514, prev_steal_time = 0
cpuhp/1-12 [001] d.h1 167.461659: account_process_tick: steal_jiffies = 1291
<idle>-0 [001] d.h1 167.462663: account_process_tick: steal = 18732255, prev_steal_time = 1291000000
<idle>-0 [001] d.h1 167.462664: account_process_tick: steal_jiffies = 18446744072437
The steal clock warp and then steal_jiffies overflow.
Rik also pointed out to me:
| I have seen stuff like that with live migration too, in the past
This patch adds steal clock warp handling by a safe threshold to only
apply steal times that are positive and smaller than one second (as
long as nohz_full has the one second timer tick left), ignoring intervals
that are negative or longer than a second, and using those to sync up
the guest with the host.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim <rkrcmar@redhat.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
v1 -> v2:
* update patch subject, description and comments
* deal with the case where steal time suddenly increases by a ludicrous amount
kernel/sched/cputime.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c
index f51c98c..751798a 100644
--- a/kernel/sched/cputime.c
+++ b/kernel/sched/cputime.c
@@ -262,17 +262,28 @@ static __always_inline unsigned long steal_account_process_tick(void)
#ifdef CONFIG_PARAVIRT
if (static_key_false(¶virt_steal_enabled)) {
u64 steal;
+ s64 delta;
unsigned long steal_jiffies;
steal = paravirt_steal_clock(smp_processor_id());
- steal -= this_rq()->prev_steal_time;
+ delta = steal - this_rq()->prev_steal_time;
+ /*
+ * Ignore this steal time difference if the guest and the host got
+ * out of sync. This can happen due to events like live migration,
+ * or CPU hotplug. The upper threshold is set to one second to match
+ * the one second timer tick with nohz_full.
+ */
+ if (unlikely(delta < 0 || delta > NSEC_PER_SEC)) {
+ this_rq()->prev_steal_time = steal;
+ return 0;
+ }
/*
* steal is in nsecs but our caller is expecting steal
* time in jiffies. Lets cast the result to jiffies
* granularity and account the rest on the next rounds.
*/
- steal_jiffies = nsecs_to_jiffies(steal);
+ steal_jiffies = nsecs_to_jiffies(delta);
this_rq()->prev_steal_time += jiffies_to_nsecs(steal_jiffies);
account_steal_time(jiffies_to_cputime(steal_jiffies));
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v2] sched/cputime: add steal clock warp handling
2016-06-03 5:21 [PATCH v2] sched/cputime: add steal clock warp handling Wanpeng Li
@ 2016-06-03 13:10 ` Rik van Riel
2016-06-06 13:44 ` Paolo Bonzini
0 siblings, 1 reply; 5+ messages in thread
From: Rik van Riel @ 2016-06-03 13:10 UTC (permalink / raw)
To: Wanpeng Li, linux-kernel, kvm
Cc: Wanpeng Li, Ingo Molnar, Peter Zijlstra (Intel),
Thomas Gleixner, Frederic Weisbecker, Paolo Bonzini, Radim
[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]
On Fri, 2016-06-03 at 13:21 +0800, Wanpeng Li wrote:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
>
> I observed that sometimes st is 100% instantaneous, then idle is
> 100%
> even if there is a cpu hog on the guest cpu after the cpu hotplug
> comes
> back(N.B. this can not always be readily reproduced). I add trace to
> capture it as below:
>
> cpuhp/1-12 [001] d.h1 167.461657: account_process_tick: steal =
> 1291385514, prev_steal_time = 0
> cpuhp/1-12 [001] d.h1 167.461659: account_process_tick:
> steal_jiffies = 1291
> <idle>-0 [001] d.h1 167.462663: account_process_tick: steal =
> 18732255, prev_steal_time = 1291000000
> <idle>-0 [001] d.h1 167.462664: account_process_tick:
> steal_jiffies = 18446744072437
>
> The steal clock warp and then steal_jiffies overflow.
>
> Rik also pointed out to me:
>
> >
> > I have seen stuff like that with live migration too, in the past
> This patch adds steal clock warp handling by a safe threshold to
> only
> apply steal times that are positive and smaller than one second (as
> long as nohz_full has the one second timer tick left), ignoring
> intervals
> that are negative or longer than a second, and using those to sync
> up
> the guest with the host.
>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Radim <rkrcmar@redhat.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>
Reviewed-by: Rik van Riel <riel@redhat.com>
--
All Rights Reversed.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] sched/cputime: add steal clock warp handling
2016-06-03 13:10 ` Rik van Riel
@ 2016-06-06 13:44 ` Paolo Bonzini
2016-06-06 14:22 ` Rik van Riel
0 siblings, 1 reply; 5+ messages in thread
From: Paolo Bonzini @ 2016-06-06 13:44 UTC (permalink / raw)
To: Rik van Riel, Wanpeng Li, linux-kernel, kvm
Cc: Wanpeng Li, Ingo Molnar, Peter Zijlstra (Intel),
Thomas Gleixner, Frederic Weisbecker, Radim
On 03/06/2016 15:10, Rik van Riel wrote:
> On Fri, 2016-06-03 at 13:21 +0800, Wanpeng Li wrote:
>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>
>> I observed that sometimes st is 100% instantaneous, then idle is
>> 100%
>> even if there is a cpu hog on the guest cpu after the cpu hotplug
>> comes
>> back(N.B. this can not always be readily reproduced). I add trace to
>> capture it as below:
>>
>> cpuhp/1-12 [001] d.h1 167.461657: account_process_tick: steal =
>> 1291385514, prev_steal_time = 0
>> cpuhp/1-12 [001] d.h1 167.461659: account_process_tick:
>> steal_jiffies = 1291
>> <idle>-0 [001] d.h1 167.462663: account_process_tick: steal =
>> 18732255, prev_steal_time = 1291000000
>> <idle>-0 [001] d.h1 167.462664: account_process_tick:
>> steal_jiffies = 18446744072437
>>
>> The steal clock warp and then steal_jiffies overflow.
>>
>> Rik also pointed out to me:
>>
>>>
>>> I have seen stuff like that with live migration too, in the past
>> This patch adds steal clock warp handling by a safe threshold to
>> only
>> apply steal times that are positive and smaller than one second (as
>> long as nohz_full has the one second timer tick left), ignoring
>> intervals
>> that are negative or longer than a second, and using those to sync
>> up
>> the guest with the host.
>>
>> Cc: Ingo Molnar <mingo@kernel.org>
>> Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
>> Cc: Rik van Riel <riel@redhat.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Frederic Weisbecker <fweisbec@gmail.com>
>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>> Cc: Radim <rkrcmar@redhat.com>
>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>
> Reviewed-by: Rik van Riel <riel@redhat.com>
Sorry for being late---again, I'd like to give a shot to a fix in KVM
guest code.
Paolo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] sched/cputime: add steal clock warp handling
2016-06-06 13:44 ` Paolo Bonzini
@ 2016-06-06 14:22 ` Rik van Riel
2016-06-06 14:34 ` Paolo Bonzini
0 siblings, 1 reply; 5+ messages in thread
From: Rik van Riel @ 2016-06-06 14:22 UTC (permalink / raw)
To: Paolo Bonzini, Wanpeng Li, linux-kernel, kvm
Cc: Wanpeng Li, Ingo Molnar, Peter Zijlstra (Intel),
Thomas Gleixner, Frederic Weisbecker, Radim
[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]
On Mon, 2016-06-06 at 15:44 +0200, Paolo Bonzini wrote:
>
> On 03/06/2016 15:10, Rik van Riel wrote:
> >
> > On Fri, 2016-06-03 at 13:21 +0800, Wanpeng Li wrote:
> > >
> > > From: Wanpeng Li <wanpeng.li@hotmail.com>
> > >
> > > I observed that sometimes st is 100% instantaneous, then idle is
> > > 100%
> > > even if there is a cpu hog on the guest cpu after the cpu hotplug
> > > comes
> > > back(N.B. this can not always be readily reproduced). I add trace
> > > to
> > > capture it as below:
> > >
> > > cpuhp/1-12 [001] d.h1 167.461657: account_process_tick:
> > > steal =
> > > 1291385514, prev_steal_time = 0
> > > cpuhp/1-12 [001] d.h1 167.461659: account_process_tick:
> > > steal_jiffies = 1291
> > > <idle>-0 [001] d.h1 167.462663: account_process_tick: steal
> > > =
> > > 18732255, prev_steal_time = 1291000000
> > > <idle>-0 [001] d.h1 167.462664: account_process_tick:
> > > steal_jiffies = 18446744072437
> > >
> > > The steal clock warp and then steal_jiffies overflow.
> > >
> > > Rik also pointed out to me:
> > >
> > > >
> > > >
> > > > I have seen stuff like that with live migration too, in the
> > > > past
> > > This patch adds steal clock warp handling by a safe threshold to
> > > only
> > > apply steal times that are positive and smaller than one second
> > > (as
> > > long as nohz_full has the one second timer tick left), ignoring
> > > intervals
> > > that are negative or longer than a second, and using those to
> > > sync
> > > up
> > > the guest with the host.
> > >
> > > Cc: Ingo Molnar <mingo@kernel.org>
> > > Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > Cc: Rik van Riel <riel@redhat.com>
> > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > > Cc: Paolo Bonzini <pbonzini@redhat.com>
> > > Cc: Radim <rkrcmar@redhat.com>
> > > Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> > Reviewed-by: Rik van Riel <riel@redhat.com>
> Sorry for being late---again, I'd like to give a shot to a fix in KVM
> guest code.
This is in the guest code.
Do you mean a fix on the host side?
--
All Rights Reversed.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] sched/cputime: add steal clock warp handling
2016-06-06 14:22 ` Rik van Riel
@ 2016-06-06 14:34 ` Paolo Bonzini
0 siblings, 0 replies; 5+ messages in thread
From: Paolo Bonzini @ 2016-06-06 14:34 UTC (permalink / raw)
To: Rik van Riel, Wanpeng Li, linux-kernel, kvm
Cc: Wanpeng Li, Ingo Molnar, Peter Zijlstra (Intel),
Thomas Gleixner, Frederic Weisbecker, Radim
On 06/06/2016 16:22, Rik van Riel wrote:
>> Sorry for being late---again, I'd like to give a shot to a fix in KVM
>> guest code.
>
> This is in the guest code.
The key word is "KVM" guest code, :) i.e. not in kernel/sched/cputime.c
but in arch/x86/kernel/kvm.c.
Thanks,
Paolo
> Do you mean a fix on the host side?
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-06-06 14:34 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-03 5:21 [PATCH v2] sched/cputime: add steal clock warp handling Wanpeng Li
2016-06-03 13:10 ` Rik van Riel
2016-06-06 13:44 ` Paolo Bonzini
2016-06-06 14:22 ` Rik van Riel
2016-06-06 14:34 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).