guest gettimeofday behavior

guest gettimeofday behavior

[Eran Rom]

Hi All,
Am a newbie (to kvm, linux kernel, git, etc.) so apologize in advance for
missing/inaccurate info.
I am experiencing inconsistent behavior of guest gettimeofday, described below.
I have seen prior reference to the problem, however, it was not clear whether
the issue was solved or not and where.

Below is a description of my setup and the behavior I see.
Another question is whether oprofile with timer mode would suffer from the same
problem?

Otherwise I saw that kvm-85 has "generic performance counter msr handling", does
this mean that oprofile can be executed without the timer mode?

Thanks very much,
Eran

Setup:
Guest 32 bit ubuntu with 2.6.27 kernel
Host 64 bit Suse 10.2 with 2.6.27 kernel
1 quadcore Intel Xeon CPU
I am not sure about the kvm userspace code version.
Used git to create a repository, and then did
git checkout kvm-updates-2.6.27 (is that fixed in time?)

Behavior:
Running a code doing:
t1 = gettimeofday
t2 = gettimeofday
while t2-t1 < 5 minutes {
    sleep(1)
    t2 = gettimeofday
}

Ran it 10 times, each time in a 'newly launched' VM, halting it after the test.
8 out of 10 times the wall clock showed 5 minutes
1 time 4 minutes and 40 seconds
1 time 0 seconds

Re: guest gettimeofday behavior

[Dor Laor]

On 06/25/2009 04:25 PM, Eran Rom wrote:
> Hi All,
> Am a newbie (to kvm, linux kernel, git, etc.) so apologize in advance for
> missing/inaccurate info.
> I am experiencing inconsistent behavior of guest gettimeofday, described below.
> I have seen prior reference to the problem, however, it was not clear whether
> the issue was solved or not and where.
>
> Below is a description of my setup and the behavior I see.
> Another question is whether oprofile with timer mode would suffer from the same
> problem?
>
> Otherwise I saw that kvm-85 has "generic performance counter msr handling", does
> this mean that oprofile can be executed without the timer mode?
>
> Thanks very much,
> Eran
>
> Setup:
> Guest 32 bit ubuntu with 2.6.27 kernel
> Host 64 bit Suse 10.2 with 2.6.27 kernel
> 1 quadcore Intel Xeon CPU
> I am not sure about the kvm userspace code version.
> Used git to create a repository, and then did
> git checkout kvm-updates-2.6.27 (is that fixed in time?)
>
> Behavior:
> Running a code doing:
> t1 = gettimeofday
> t2 = gettimeofday
> while t2-t1<  5 minutes {
>      sleep(1)
>      t2 = gettimeofday
> }
>
> Ran it 10 times, each time in a 'newly launched' VM, halting it after the test.
> 8 out of 10 times the wall clock showed 5 minutes
> 1 time 4 minutes and 40 seconds
> 1 time 0 seconds
>    

I'm not sure what's your 'wall clock' value, there is not printf in your 
script.
Nevertheless, the tsc clock is not reliable, the host can scale it, or 
go into
deep sleep state.

So either use newer kernel with kvmclock (pv) or change the clock source
into rtc/pit


> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: guest gettimeofday behavior

[Avi Kivity]

On 06/25/2009 04:25 PM, Eran Rom wrote:
> Hi All,
> Am a newbie (to kvm, linux kernel, git, etc.) so apologize in advance for
> missing/inaccurate info.
> I am experiencing inconsistent behavior of guest gettimeofday, described below.
> I have seen prior reference to the problem, however, it was not clear whether
> the issue was solved or not and where.
>
> Below is a description of my setup and the behavior I see.
> Another question is whether oprofile with timer mode would suffer from the same
> problem?
>    

Probably, yes.

> Otherwise I saw that kvm-85 has "generic performance counter msr handling", does
> this mean that oprofile can be executed without the timer mode?
>    

No, we don't really emulate the performance counters, only when they're 
disabled.

> Setup:
> Guest 32 bit ubuntu with 2.6.27 kernel
> Host 64 bit Suse 10.2 with 2.6.27 kernel
>    

What's the guest clocksource? (/sys/.../current_clocksource)?

> 1 quadcore Intel Xeon CPU
> I am not sure about the kvm userspace code version.
> Used git to create a repository, and then did
> git checkout kvm-updates-2.6.27 (is that fixed in time?)
>    


That's really old.  Just use qemu-kvm.git.

-- 
error compiling committee.c: too many arguments to function

Re: guest gettimeofday behavior

[Eran Rom]

Dor Laor <dlaor <at> redhat.com> writes:

> 
> On 06/25/2009 04:25 PM, Eran Rom wrote:
> > Behavior:
> > Running a code doing:
> > t1 = gettimeofday
> > t2 = gettimeofday
> > while t2-t1<  5 minutes {
> >      sleep(1)
> >      t2 = gettimeofday
> > }
> >
> > Ran it 10 times, each time in a 'newly launched' VM, 
> > halting it after the test.
> > 8 out of 10 times the wall clock showed 5 minutes
> > 1 time 4 minutes and 40 seconds
> > 1 time 0 seconds
> >    
> 
> I'm not sure what's your 'wall clock' value, 
> there is not printf in your 
> script.
the 'wall clock' is host time.
Here is what I am doing form the host:
#! /bin/bash
date
ssh $GuestIP run_gettimeofday_binary
date

> Nevertheless, the tsc clock is not reliable, 
> the host can scale it, or 
> go into
> deep sleep state.
indeed my current clock source is tsc

> 
> So either use newer kernel with kvmclock (pv) 
> or change the clock source
> into rtc/pit
>
No rtc/pit in my available clock sources, however, 
I assume its a question of kernel compilation parameters, 
is that right?
What is the advantage of rtc/pit/kvmclock (pv)over tsc?
Also, is kvmclock (pv) better than the other two options. 
the thing is that I am 'confined' to use 2.6.27, 
and if I decide on kvmclock, i will need to add it
'manually'
 
Thanks,
Eran
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo <at> vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >    
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: guest gettimeofday behavior

[Avi Kivity]

On 06/29/2009 10:11 PM, Eran Rom wrote:
>> Nevertheless, the tsc clock is not reliable,
>> the host can scale it, or
>> go into
>> deep sleep state.
>>      
> indeed my current clock source is tsc
>
>    

You won't get accurate timing with tsc.

>> So either use newer kernel with kvmclock (pv)
>> or change the clock source
>> into rtc/pit
>>
>>      
> No rtc/pit in my available clock sources, however,
> I assume its a question of kernel compilation parameters,
> is that right?
> What is the advantage of rtc/pit/kvmclock (pv)over tsc?
>    

They are accurate.

> Also, is kvmclock (pv) better than the other two options.
>    

Yes.

> the thing is that I am 'confined' to use 2.6.27,
> and if I decide on kvmclock, i will need to add it
> 'manually'
>    

2.6.27 has kvmclock.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: guest gettimeofday behavior

[Eran Rom]

Avi Kivity <avi <at> redhat.com> writes:

> 2.6.27 has kvmclock.
> 
Thanks a lot!
Will try the kvmclock
Eran 

Re: guest gettimeofday behavior

[Eran Rom]

Eran Rom <eranr <at> il.ibm.com> writes:

> 
> Avi Kivity <avi <at> redhat.com> writes:
> 
> > 2.6.27 has kvmclock.
> > 
> Thanks a lot!
> Will try the kvmclock
> Eran 
> 
Still getting a misbehaving clock:
guest uses kvmclock with kernel 2.6.27
host kernel is 2.6.27 with kvm-qemu-87 & kvm-kmod-87

doing:
now = gettimeofday
goal = now + 10 minutes
while (now < goal) {
    sleep(10)
    now = gettimeofday
    print(now)
}

getting:
...
now     =       2098031895
now     =       2108035868
now     =       2118039818
now     =       2128043797
now     =       2138048010
now     =       18446744071562636328

Thanks very much,
Eran

Re: guest gettimeofday behavior

[Eran Rom]

Eran Rom <eranr <at> il.ibm.com> writes:

> 
> Eran Rom <eranr <at> il.ibm.com> writes:
> 
> Still getting a misbehaving clock:
> guest uses kvmclock with kernel 2.6.27
> host kernel is 2.6.27 with kvm-qemu-87 & kvm-kmod-87
> 
> getting:
> ...
> now     =       2128043797
> now     =       2138048010
> now     =       18446744071562636328
> 
Below is the exact code used.
Also, I use a script in the host to run this code in a loop for 5 minutes, each
time in a newly launched guest. This happens on average once every 10 executions.

Thanks very much,
Eran

#include <sys/time.h>
#include <time.h>

int main(int argc, char **argv)
{
        int seconds;
        unsigned long long now, goal;
        struct timeval time1, time2;

        if (argc != 2)
                return;

        seconds = atoi(argv[1]);

        gettimeofday(&time1, NULL);
        now = ((time1.tv_sec) * 1000000) + time1.tv_usec;
        goal = now + (seconds * 1000000);
        while (now < goal) {
                sleep(10);
                gettimeofday(&time2, NULL);
                now = (time2.tv_sec * 1000000) + time2.tv_usec;
        }
}

Re: guest gettimeofday behavior

[Marcelo Tosatti]

On Tue, Jul 07, 2009 at 09:58:46AM +0000, Eran Rom wrote:
> Eran Rom <eranr <at> il.ibm.com> writes:
> 
> > 
> > Eran Rom <eranr <at> il.ibm.com> writes:
> > 
> > Still getting a misbehaving clock:
> > guest uses kvmclock with kernel 2.6.27
> > host kernel is 2.6.27 with kvm-qemu-87 & kvm-kmod-87
> > 
> > getting:
> > ...
> > now     =       2128043797
> > now     =       2138048010
> > now     =       18446744071562636328
> > 

Eran,

This was happening with kvmclock back then when it did not handle
cpu frequency changes. What are the details of the host hw?

Glauber, any clues?

> Below is the exact code used.
> Also, I use a script in the host to run this code in a loop for 5 minutes, each
> time in a newly launched guest. This happens on average once every 10 executions.
> 
> Thanks very much,
> Eran
> 
> #include <sys/time.h>
> #include <time.h>
> 
> int main(int argc, char **argv)
> {
>         int seconds;
>         unsigned long long now, goal;
>         struct timeval time1, time2;
> 
>         if (argc != 2)
>                 return;
> 
>         seconds = atoi(argv[1]);
> 
>         gettimeofday(&time1, NULL);
>         now = ((time1.tv_sec) * 1000000) + time1.tv_usec;
>         goal = now + (seconds * 1000000);
>         while (now < goal) {
>                 sleep(10);
>                 gettimeofday(&time2, NULL);
>                 now = (time2.tv_sec * 1000000) + time2.tv_usec;
>         }
> }

Re: guest gettimeofday behavior

[Glauber Costa]

On Tue, Jul 07, 2009 at 05:12:16PM -0300, Marcelo Tosatti wrote:
> On Tue, Jul 07, 2009 at 09:58:46AM +0000, Eran Rom wrote:
> > Eran Rom <eranr <at> il.ibm.com> writes:
> > 
> > > 
> > > Eran Rom <eranr <at> il.ibm.com> writes:
> > > 
> > > Still getting a misbehaving clock:
> > > guest uses kvmclock with kernel 2.6.27
> > > host kernel is 2.6.27 with kvm-qemu-87 & kvm-kmod-87
> > > 
> > > getting:
> > > ...
> > > now     =       2128043797
> > > now     =       2138048010
> > > now     =       18446744071562636328
> > > 
> 
> Eran,
> 
> This was happening with kvmclock back then when it did not handle
> cpu frequency changes. What are the details of the host hw?
> 
> Glauber, any clues?
maybe it is exactly that. 2.6.27 is a little bit old, and probably does not
have the fixes.

Re: guest gettimeofday behavior

[Eran Rom]

Glauber Costa <glommer <at> redhat.com> writes:

> > This was happening with kvmclock back then when it did not handle
> > cpu frequency changes. What are the details of the host hw?
> > 
> > Glauber, any clues?
> maybe it is exactly that. 2.6.27 is a little bit old, and probably does not
> have the fixes.
> 
Will test with a newer kernel in the coming days and update.
Thanks very much,
Eran

Re: guest gettimeofday behavior

[Eran Rom]

Glauber Costa <glommer <at> redhat.com> writes:

> 
> > This was happening with kvmclock back then when it did not handle
> > cpu frequency changes. What are the details of the host hw?
> > 
> > Glauber, any clues?
> maybe it is exactly that. 2.6.27 is a little bit old, and probably does not
> have the fixes.
>

Tested with 2.6.30 and the problem indeed disappeared.
When Host and Guest ran 2.6.27 with kvm-87 (both qemu-kvm and kvm-kmod), the
problem persisted. Thus, I am looking for a kernel fix that is not part of KVM,
any lead? Am confined to use 2.6.27

Thanks very much,
Eran

Re: guest gettimeofday behavior

[Eran Rom]

Eran Rom <eranr <at> il.ibm.com> writes:

> > When Host and Guest ran 2.6.27 with kvm-87 (both qemu-kvm and kvm-kmod), the
> > problem persisted. Thus, I am looking for a kernel fix that is not
> part of KVM,
> > any lead? Am confined to use 2.6.27

> Marcelo Tosatti <mtosatti@redhat.com> wrote on 16/07/2009 01:19:40: 
> Is it an AMD host with cpufreq?

Intel(R) Xeon(TM) CPU 3.00GHz, CONFIG_CPU_FREQ=y
Guest clock source is kvmclock

Thanks very much,
Eran