From: Sebastian Hetze <s.hetze@linux-ag.com>
To: Avi Kivity <avi@redhat.com>
Cc: Sebastian Hetze <s.hetze@linux-ag.com>,
kvm@vger.kernel.org, Marcelo Tosatti <mtosatti@redhat.com>
Subject: Re: Strange CPU usage pattern in SMP guest
Date: Sun, 21 Mar 2010 16:47:53 +0100 [thread overview]
Message-ID: <20100321154753.EC5DCA0017@mail.linux-ag.de> (raw)
In-Reply-To: <4BA63892.6090006@redhat.com>
On Sun, Mar 21, 2010 at 05:17:38PM +0200, Avi Kivity wrote:
> On 03/21/2010 04:55 PM, Sebastian Hetze wrote:
>> On Sun, Mar 21, 2010 at 02:19:40PM +0200, Avi Kivity wrote:
>>
>>> On 03/21/2010 02:02 PM, Sebastian Hetze wrote:
>>>
>>>> 12:46:02 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 12:46:03 all 0,20 11,35 10,96 8,96 0,40 2,99 0,00 0,00 65,14
>>>> 12:46:03 0 1,00 11,00 7,00 15,00 0,00 1,00 0,00 0,00 65,00
>>>> 12:46:03 1 0,00 7,14 2,04 6,12 1,02 11,22 0,00 0,00 72,45
>>>> 12:46:03 2 0,00 15,00 1,00 12,00 0,00 1,00 0,00 0,00 71,00
>>>> 12:46:03 3 0,00 11,00 23,00 8,00 0,00 0,00 0,00 0,00 58,00
>>>> 12:46:03 4 0,00 0,00 50,00 0,00 0,00 0,00 0,00 0,00 50,00
>>>> 12:46:03 5 0,00 13,00 20,00 4,00 0,00 1,00 0,00 0,00 62,00
>>>>
>>>> So it is only CPU4 that is showing this strange behaviour.
>>>>
>>>>
>>> Can you adjust irqtop to only count cpu4? or even just post a few 'cat
>>> /proc/interrupts' from that guest.
>>>
>>> Most likely the timer interrupt for cpu4 died.
>>>
>> I've added two keys +/- to your irqtop to focus up and down
>> in the row of available CPUs.
>> The irqtop for CPU4 shows a constant number of 6 local timer interrupts
>> per update, while the other CPUs show various higher values:
>>
>> irqtop for cpu 4
>>
>> eth0 188
>> Rescheduling interrupts 162
>> Local timer interrupts 6
>> ata_piix 3
>> TLB shootdowns 1
>> Spurious interrupts 0
>> Machine check exceptions 0
>>
>>
>> irqtop for cpu 5
>>
>> eth0 257
>> Local timer interrupts 251
>> Rescheduling interrupts 237
>> Spurious interrupts 0
>> Machine check exceptions 0
>>
>> So the timer interrupt for cpu4 is not completely dead but somehow
>> broken.
>
> That is incredibly weird.
>
>> What can cause this problem? Any way to speed it up again?
>>
>
> The host has 8 cpus and is only running this 6 vcpu guest, yes?
The host is an dual quad core E5520 with hyperthrading enabled, so we
see 2x4x2=16 CPUs on the host. The guest is started with 6 CPUs.
> Can you confirm the other vcpus are ticking at 250 Hz?
The irqtop shows different numbers for local timer interrupts on the
other CPUs. The total number (summed up over all CPUs) varies between
something like 700 and 1400. Any CPU can be down to 10 and next update
up to 260. Only CPU4 stays at the 6 local timer interrupts.
>
> What does 'top' show running on cpu 4? Pressing 'f' 'j' will add a
> last-used-cpu field in the display.
The processes are not bound to a particular CPU, so the picture varies.
Here are two shots:
take1:
15 root RT -5 0 0 0 S 0 0.0 0:01.70 4 migration/4
16 root 15 -5 0 0 0 S 0 0.0 0:00.08 4 ksoftirqd/4
17 root RT -5 0 0 0 S 0 0.0 0:00.00 4 watchdog/4
25 root 15 -5 0 0 0 S 0 0.0 0:00.01 4 events/4
35 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kintegrityd/4
41 root 15 -5 0 0 0 S 0 0.0 0:00.03 4 kblockd/4
50 root 15 -5 0 0 0 S 0 0.0 0:00.90 4 ata/4
55 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kseriod
66 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 aio/4
73 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 crypto/4
80 root 15 -5 0 0 0 S 0 0.0 2:11.71 4 scsi_eh_1
87 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kmpathd/4
95 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kondemand/4
101 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kconservative/4
103 root 10 -10 0 0 0 S 0 0.0 0:00.00 4 krfcommd
681 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
686 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
691 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
737 root 15 -5 0 0 0 S 0 0.0 0:00.71 4 kjournald
826 root 16 -4 2100 452 312 S 0 0.0 0:00.14 4 udevd
1350 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kpsmoused
1444 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kgameportd
1718 root 15 -5 0 0 0 S 0 0.0 0:14.62 4 kjournald
2108 statd 20 0 2252 1152 760 S 0 0.0 0:02.66 4 rpc.statd
2117 root 15 -5 0 0 0 S 0 0.0 0:00.36 4 rpciod/4
2123 root 15 -5 0 0 0 S 0 0.0 0:06.61 4 nfsiod
2259 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty
2265 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty
2584 root 20 0 1860 656 604 S 0 0.0 0:01.92 4 automount
2712 mysql 20 0 146m 4580 3464 S 0 0.1 0:59.71 4 mysqld
2999 root 20 0 28944 12m 1704 S 0 0.2 1:23.04 4 console-kit-dae
3834 root 20 0 13476 2640 1572 S 0 0.0 0:00.03 4 smbd
4015 root 15 -5 0 0 0 S 0 0.0 7:22.33 4 nfsd
4017 root 15 -5 0 0 0 S 0 0.0 10:46.87 4 nfsd
4030 root 20 0 2644 1180 876 S 0 0.0 0:04.32 4 rpc.mountd
4134 root 20 0 6164 1672 1344 S 0 0.0 0:00.37 4 master
4311 root 20 0 5068 1796 1580 S 0 0.0 1:25.80 4 hald-addon-stor
4560 root 20 0 2092 728 572 S 0 0.0 0:01.78 4 cron
5053 www-data 20 0 33312 4092 1004 S 0 0.0 0:00.03 4 apache2
take 2:
15 root RT -5 0 0 0 S 0 0.0 0:01.70 4 migration/4
16 root 15 -5 0 0 0 S 0 0.0 0:00.08 4 ksoftirqd/4
17 root RT -5 0 0 0 S 0 0.0 0:00.00 4 watchdog/4
25 root 15 -5 0 0 0 S 0 0.0 0:00.01 4 events/4
35 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kintegrityd/4
41 root 15 -5 0 0 0 S 0 0.0 0:00.03 4 kblockd/4
50 root 15 -5 0 0 0 S 0 0.0 0:00.90 4 ata/4
55 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kseriod
66 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 aio/4
73 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 crypto/4
87 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kmpathd/4
95 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kondemand/4
101 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kconservative/4
103 root 10 -10 0 0 0 S 0 0.0 0:00.00 4 krfcommd
681 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
686 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
691 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush
737 root 15 -5 0 0 0 S 0 0.0 0:00.71 4 kjournald
826 root 16 -4 2100 452 312 S 0 0.0 0:00.14 4 udevd
1350 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kpsmoused
1444 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kgameportd
2091 daemon 20 0 1820 552 464 S 0 0.0 0:44.31 4 portmap
2108 statd 20 0 2252 1152 760 S 0 0.0 0:02.66 4 rpc.statd
2117 root 15 -5 0 0 0 S 0 0.0 0:00.36 4 rpciod/4
2123 root 15 -5 0 0 0 S 0 0.0 0:06.61 4 nfsiod
2259 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty
2265 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty
2712 mysql 20 0 146m 4580 3464 S 0 0.1 0:59.72 4 mysqld
3834 root 20 0 13476 2640 1572 S 0 0.0 0:00.03 4 smbd
4017 root 15 -5 0 0 0 S 0 0.0 10:46.87 4 nfsd
4134 root 20 0 6164 1672 1344 S 0 0.0 0:00.37 4 master
4311 root 20 0 5068 1796 1580 S 0 0.0 1:25.80 4 hald-addon-stor
4560 root 20 0 2092 728 572 S 0 0.0 0:01.78 4 cron
5053 www-data 20 0 33312 4092 1004 S 0 0.0 0:00.03 4 apache2
next prev parent reply other threads:[~2010-03-21 15:47 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-03-21 0:13 Strange CPU usage pattern in SMP guest Sebastian Hetze
2010-03-21 10:09 ` Avi Kivity
2010-03-21 12:02 ` Sebastian Hetze
[not found] ` <20100321120236.55228A0015@mail.linux-ag.de>
2010-03-21 12:19 ` Avi Kivity
2010-03-21 14:55 ` Sebastian Hetze
[not found] ` <20100321145548.CC027A0015@mail.linux-ag.de>
2010-03-21 15:17 ` Avi Kivity
2010-03-21 15:47 ` Sebastian Hetze [this message]
2010-03-22 12:51 ` Sebastian Hetze
2010-03-23 21:18 ` Marcelo Tosatti
2010-03-30 8:27 ` Sebastian Hetze
[not found] ` <20100330082743.49A113030135@mail.linux-ag.de>
2010-04-05 21:24 ` Sebastian Hetze
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100321154753.EC5DCA0017@mail.linux-ag.de \
--to=s.hetze@linux-ag.com \
--cc=avi@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=mtosatti@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.