Re: Strange CPU usage pattern in SMP guest

[Avi Kivity <avi@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On 03/21/2010 02:13 AM, Sebastian Hetze wrote:
> Hi *,
>
> in an 6 CPU SMP guest running on an host with 2 quad core
> Intel Xeon E5520 with hyperthrading enabled
> we see one or more guest CPUs working in a very strange
> pattern. It looks like all or nothing. We can easily identify
> the effected CPU with xosview. Here is the mpstat output
> compared to one regular working CPU:
>
>
> mpstat -P 4 1
> Linux 2.6.31-16-generic-pae (guest) 	21.03.2010 	_i686_	(6 CPU)
> 00:45:19     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
> 00:45:20       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
> 00:45:21       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
> 00:45:22       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
> 00:45:23       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
> 00:45:24       4    0,00   66,67    0,00    0,00    0,00   33,33    0,00    0,00    0,00
> 00:45:25       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
> 00:45:26       4    0,00  100,00    0,00    0,00    0,00    0,00    0,00    0,00    0,00
>    

Looks like the guest is only receiving 3-4 timer interrupts per second, 
so time becomes quantized.

Please run the attached irqtop in the affected guest and report the results.

Is the host overly busy?  What host kernel, kvm, and qemu are you 
running?  Is the guest running an I/O workload? if so, how are the disks 
configured?

-- 
error compiling committee.c: too many arguments to function


[-- Attachment #2: irqtop --]
[-- Type: text/plain, Size: 1756 bytes --]

#!/usr/bin/python

import curses
import sys, os, time, optparse

def read_interrupts():
    irq = {}
    proc = file('/proc/interrupts')
    nrcpu = len(proc.readline().split())
    for line in proc.readlines():
        vec, data = line.strip().split(':', 1)
        if vec in ('ERR', 'MIS'):
            continue
        counts = data.split(None, nrcpu)
        counts, rest = (counts[:-1], counts[-1])
        count = sum([int(x) for x in counts])
        try:
            v = int(vec)
            name = rest.split(None, 1)[1]
        except:
            name = rest
        irq[name] = count
    return irq

def delta_interrupts():
    old = read_interrupts()
    while True:
        irq = read_interrupts()
        delta = {}
        for key in irq.keys():
            delta[key] = irq[key] - old[key]
        yield delta
        old = irq

label_width = 30
number_width = 10

def tui(screen):
    curses.use_default_colors()
    curses.noecho()
    def getcount(x):
        return x[1]
    def refresh(irq):
        screen.erase()
        screen.addstr(0, 0, 'irqtop')
        row = 2
        for name, count in sorted(irq.items(), key = getcount, reverse = True):
            if row >= screen.getmaxyx()[0]:
                break
            col = 1
            screen.addstr(row, col, name)
            col += label_width
            screen.addstr(row, col, '%10d' % (count,))
            row += 1
        screen.refresh()

    for irqs in delta_interrupts():
        refresh(irqs)
        curses.halfdelay(10)
        try:
            c = screen.getkey()
            if c == 'q':
                break
        except KeyboardInterrupt:
            break
        except curses.error:
            continue

import curses.wrapper
curses.wrapper(tui)