From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: Strange CPU usage pattern in SMP guest Date: Sun, 21 Mar 2010 12:09:00 +0200 Message-ID: <4BA5F03C.1020900@redhat.com> References: <20100321001304.B8EAF30301DA@mail.linux-ag.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------080908010802000209000203" Cc: kvm@vger.kernel.org To: Sebastian Hetze Return-path: Received: from mx1.redhat.com ([209.132.183.28]:33789 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752332Ab0CUKJF (ORCPT ); Sun, 21 Mar 2010 06:09:05 -0400 In-Reply-To: <20100321001304.B8EAF30301DA@mail.linux-ag.de> Sender: kvm-owner@vger.kernel.org List-ID: This is a multi-part message in MIME format. --------------080908010802000209000203 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 03/21/2010 02:13 AM, Sebastian Hetze wrote: > Hi *, > > in an 6 CPU SMP guest running on an host with 2 quad core > Intel Xeon E5520 with hyperthrading enabled > we see one or more guest CPUs working in a very strange > pattern. It looks like all or nothing. We can easily identify > the effected CPU with xosview. Here is the mpstat output > compared to one regular working CPU: > > > mpstat -P 4 1 > Linux 2.6.31-16-generic-pae (guest) 21.03.2010 _i686_ (6 CPU) > 00:45:19 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 00:45:20 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > 00:45:21 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > 00:45:22 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > 00:45:23 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > 00:45:24 4 0,00 66,67 0,00 0,00 0,00 33,33 0,00 0,00 0,00 > 00:45:25 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > 00:45:26 4 0,00 100,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > Looks like the guest is only receiving 3-4 timer interrupts per second, so time becomes quantized. Please run the attached irqtop in the affected guest and report the results. Is the host overly busy? What host kernel, kvm, and qemu are you running? Is the guest running an I/O workload? if so, how are the disks configured? -- error compiling committee.c: too many arguments to function --------------080908010802000209000203 Content-Type: text/plain; name="irqtop" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="irqtop" #!/usr/bin/python import curses import sys, os, time, optparse def read_interrupts(): irq = {} proc = file('/proc/interrupts') nrcpu = len(proc.readline().split()) for line in proc.readlines(): vec, data = line.strip().split(':', 1) if vec in ('ERR', 'MIS'): continue counts = data.split(None, nrcpu) counts, rest = (counts[:-1], counts[-1]) count = sum([int(x) for x in counts]) try: v = int(vec) name = rest.split(None, 1)[1] except: name = rest irq[name] = count return irq def delta_interrupts(): old = read_interrupts() while True: irq = read_interrupts() delta = {} for key in irq.keys(): delta[key] = irq[key] - old[key] yield delta old = irq label_width = 30 number_width = 10 def tui(screen): curses.use_default_colors() curses.noecho() def getcount(x): return x[1] def refresh(irq): screen.erase() screen.addstr(0, 0, 'irqtop') row = 2 for name, count in sorted(irq.items(), key = getcount, reverse = True): if row >= screen.getmaxyx()[0]: break col = 1 screen.addstr(row, col, name) col += label_width screen.addstr(row, col, '%10d' % (count,)) row += 1 screen.refresh() for irqs in delta_interrupts(): refresh(irqs) curses.halfdelay(10) try: c = screen.getkey() if c == 'q': break except KeyboardInterrupt: break except curses.error: continue import curses.wrapper curses.wrapper(tui) --------------080908010802000209000203--