Re: strange guest slowness after some time

[Avi Kivity <avi@redhat.com>]

Tomasz Chmielewski wrote:
> I have a strange slowness which affects some guests after they are 
> running for some time. "Slowness" can happen a few hours after guest 
> start, or, a couple of days after guest start.
>
> What do I mean by "slowness"?
>
> This is how long it takes to log in via SSH to an unaffected guest - 
> below a second:
>
> $ time ssh backupuser@normal_guest exit
> 0.02user 0.01system 0:00.67elapsed 4%CPU (0avgtext+0avgdata 0maxresident)
>
> Now, let's try to log in to the affected guest running on the same 
> host - more than 12 seconds:
>
> $ time ssh backupuser@slow_guest exit
> 0.02user 0.01system 0:12.56elapsed 0%CPU (0avgtext+0avgdata 0maxresident)
>
> If I log in via SSH to the affected guest, any key presses lag a 
> second or two.
>
>
> This is actually weird - if I run something IO intensive on the guest, 
> the login is much faster (running CPU-intensive tasks makes no 
> difference):
>
> guest# dd if=/dev/vda of=/dev/null
>
> $ time ssh backupuser@slow_guest exit
> 0.02user 0.00system 0:00.70elapsed 2%CPU (0avgtext+0avgdata 0maxresident)
>
> Also, running "ping -f <slow_guest>" helps a lot and SSH logins are fast.
>
>
> Look at the difference here - 7470ms vs 139183ms (and packet losses):
>
> # ping -f -c 10000 normal_guest
>
> 10000 packets transmitted, 10000 received, 0% packet loss, time 7470ms
> rtt min/avg/max/mdev = 0.443/0.709/6.487/0.112 ms, ipg/ewma 
> 0.747/0.716 ms
>
> # ping -f -c 10000 slow_guest
>
> 10000 packets transmitted, 9934 received, 0% packet loss, time 139183ms
> rtt min/avg/max/mdev = 0.470/14.337/50.455/5.409 ms, pipe 4, ipg/ewma 
> 13.919/14.788 ms
>
>
> CPU-intensive tasks are as fast as on unaffected guests.
> Reading from /dev/vda is as fast as on unaffected guests.
>
> So the only thing broken seems to be the network.
>
>
> Rebooting the guest does not help - it is still slow.
> The only thing that helps is stopping the guest and starting it again 
> (i.e., stopping kvm process and starting a new one).
>
>
> Is there an explanation to this phenomenon? Looks like a problem with 
> virtio drivers somewhere, or?
>
>
>
> The host is running kvm-83.
> Affected guests are running 2.6.27.14 kernels and use virtio drivers.
> The problem happens only _sometimes_. Out of 9 guests I have running 
> on this host, I saw this problem only on 3 guests. I never saw this 
> happening on more than one guest at a time.
> All three have 512 MB memory assigned, other guests have less memory.
>

I'm guessing there's a problem with timers or timer interrupts.

What is the host cpu?

Does the problem occur if you pin a guest to a cpu with taskset?

-- 
error compiling committee.c: too many arguments to function