* Q: Stopped VM still using host cpu CPU ?
@ 2009-10-01 10:32 Daniel Schwager
2009-10-01 11:47 ` Daniel Schwager
[not found] ` <4AC8B6D9.7070003@redhat.com>
0 siblings, 2 replies; 9+ messages in thread
From: Daniel Schwager @ 2009-10-01 10:32 UTC (permalink / raw)
To: kvm; +Cc: Daniel Schwager
Hi,
we are running some stopped (sending "stop" via kvm-monitor socket)
vm's on our system. My intention was to pause (stop) the vm's and
unpause (cont) them on demand (very fast, without time delay, within 2
seconds ..).
After 'stop'ing, the vm's still using CPU-load, like the "top" will
tell:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25983 root 20 0 495m 407m 1876 R 8.9 2.5 228:09.15
qemu-system-x86
25523 root 20 0 495m 2040 1868 S 7.9 0.0 2700:16
qemu-system-x86
19738 root 20 0 1139m 40m 1876 R 7.3 0.3 2501:57
qemu-system-x86
32521 root 20 0 495m 407m 1876 S 6.9 2.5 136:23.23
qemu-system-x86
21773 root 20 0 495m 407m 1876 S 5.6 2.5 575:01.44
qemu-system-x86
6720 root 20 0 2168m 2.0g 1876 R 4.6 12.9 524:55.70
qemu-system-x86
6819 root 20 0 2168m 2.0g 1876 S 4.6 12.9 489:24.24
qemu-system-x86
12752 root 20 0 495m 407m 1876 S 4.6 2.5 0:44.57
qemu-system-x86
22803 root 20 0 1139m 211m 1868 S 4.0 1.3 1217:21
qemu-system-x86
27083 root 20 0 1139m 1.0g 1868 S 4.0 6.5 121:58.05
qemu-system-x86
21970 root 20 0 495m 407m 1876 S 3.6 2.5 434:48.36
qemu-system-x86
3649 root 20 0 496m 394m 1908 S 3.3 2.5 39:41.61
qemu-system-x86
19919 root 20 0 559m 402m 1880 S 3.3 2.5 469:57.14
qemu-system-x86
24461 root 20 0 495m 2048 1876 S 3.3 0.0 1081:41
qemu-system-x86
24619 root 20 0 688m 536m 1880 S 3.0 3.3 293:15.38
qemu-system-x86
24776 root 20 0 495m 407m 1856 S 3.0 2.5 92:08.95
qemu-system-x86
...
So, how can I prevent the paused/stopped VM's to use my CPU from
the hostsystem ? Is there a way to handle this ?
Sure, stop+migrating the vm to filesystem will solve my problem - but
restart
(migrate back from filesystem to RAM) takes a lot (>30 seconds) of time.
best regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ?
2009-10-01 10:32 Q: Stopped VM still using host cpu CPU ? Daniel Schwager
@ 2009-10-01 11:47 ` Daniel Schwager
[not found] ` <4AC8B6D9.7070003@redhat.com>
1 sibling, 0 replies; 9+ messages in thread
From: Daniel Schwager @ 2009-10-01 11:47 UTC (permalink / raw)
To: kvm
One more,
> So, how can I prevent the paused/stopped VM's to use my CPU from
> the hostsystem ? Is there a way to handle this ?
If i send a signal STOP/CONT (kill -STOP <pid> or kill -CONT <pid>)
to the KVM-process, it looks like the kvm does not (sure ;-) use
any host CPU usage.
- Are there some side effects using this approach ?
(e.g. with networking, ...)
- And, why is there a way to send STOP/CONT via socket to KVM-process ?
Why
not using the sending-signal-apporch ?
best regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ?
[not found] ` <4AC8C018.6050302@redhat.com>
@ 2009-10-16 21:35 ` Daniel Schwager
2009-10-18 6:59 ` Avi Kivity
0 siblings, 1 reply; 9+ messages in thread
From: Daniel Schwager @ 2009-10-16 21:35 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm
> On 10/04/2009 05:21 PM, Daniel Schwager wrote:
> >
> >> How long is this after the 'stop'?
> >>
> > 30 seconds or 2 days ... the process takes CPU all the time
> >
>
> Can you take an oprofile run to see where it's spending its time?
I'm not that familiar with oprofile ...
so, setup with
opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start
gave me the following result:
kvm03:~# opreport
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
TIMER:0|
samples| %|
------------------
613327 95.3150 no-vmlinux
12416 1.9295 qemu-system-x86_64
TIMER:0|
samples| %|
------------------
10888 87.6933 qemu-system-x86_64
209 1.6833 [vdso] (tgid:2164
range:0x7ffff37ff000-0x7ffff3800000)
171 1.3773 [vdso] (tgid:28126
range:0x7fffe7dff000-0x7fffe7e00000)
168 1.3531 [vdso] (tgid:11172
range:0x7ffff67fe000-0x7ffff67ff000)
147 1.1840 [vdso] (tgid:1511
range:0x7fff741fe000-0x7fff741ff000)
143 1.1517 [vdso] (tgid:28327
range:0x7fffdcbfe000-0x7fffdcbff000)
141 1.1356 [vdso] (tgid:17010
range:0x7fff2a1fe000-0x7fff2a1ff000)
127 1.0229 [vdso] (tgid:27527
range:0x7fff2cffe000-0x7fff2cfff000)
111 0.8940 [vdso] (tgid:27644
range:0x7fffec7fe000-0x7fffec7ff000)
92 0.7410 [vdso] (tgid:11565
range:0x7fff247fe000-0x7fff247ff000)
81 0.6524 [vdso] (tgid:27438
range:0x7fff7c3fe000-0x7fff7c3ff000)
76 0.6121 [vdso] (tgid:26857
range:0x7ffffa7fe000-0x7ffffa7ff000)
62 0.4994 [vdso] (tgid:1676
range:0x7fff2f5ff000-0x7fff2f600000)
11488 1.7853 libpthread-2.8.so
....
Try to start without timer=1 hung up my physical machine after daemon is
started:
kvm03:/opt/kvm-86-vnc-patch# opcontrol --deinit; modprobe oprofile ;
opcontrol --start
Stopping profiling.
Killing daemon.
Unloading oprofile module
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Could I give you more information ? Should I setup oprofile in another
way ?
regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ?
2009-10-16 21:35 ` Daniel Schwager
@ 2009-10-18 6:59 ` Avi Kivity
2009-10-18 18:29 ` Daniel Schwager
0 siblings, 1 reply; 9+ messages in thread
From: Avi Kivity @ 2009-10-18 6:59 UTC (permalink / raw)
To: Daniel Schwager; +Cc: kvm
On 10/17/2009 06:35 AM, Daniel Schwager wrote:
>
> I'm not that familiar with oprofile ...
>
> so, setup with
> opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start
>
> gave me the following result:
>
> kvm03:~# opreport
>
Use 'opreport -l'. Make sure your qemu isn't stripped.
> Try to start without timer=1 hung up my physical machine after daemon is
> started:
> kvm03:/opt/kvm-86-vnc-patch# opcontrol --deinit; modprobe oprofile ;
> opcontrol --start
> Stopping profiling.
> Killing daemon.
> Unloading oprofile module
> Using default event: CPU_CLK_UNHALTED:100000:0:1:1
> Using 2.6+ OProfile kernel interface.
> Using log file /var/lib/oprofile/samples/oprofiled.log
> Daemon started.
>
>
> Could I give you more information ? Should I setup oprofile in another
> way ?
>
>
Is there an oops in dmesg, or is the machine dead? What's the host
kernel version?
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ?
2009-10-18 6:59 ` Avi Kivity
@ 2009-10-18 18:29 ` Daniel Schwager
2009-10-18 18:47 ` Daniel Schwager
0 siblings, 1 reply; 9+ messages in thread
From: Daniel Schwager @ 2009-10-18 18:29 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm
> > so, setup with
> > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start
> >
> Use 'opreport -l'. Make sure your qemu isn't stripped.
ok ... will do
>
> > Try to start without timer=1 hung up my physical machine after daemon is
> > started:
>
> Is there an oops in dmesg, or is the machine dead? What's the host
> kernel version?
´
Machine was dead. no infos inside dmesg / messages.
We use FC9, Linux kvm03.dev.softwaredemo.de 2.6.27.21-78.2.41.fc9.x86_64 #1 SMP Mon Mar 23 23:22:15 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ?
2009-10-18 18:29 ` Daniel Schwager
@ 2009-10-18 18:47 ` Daniel Schwager
2009-10-18 21:57 ` Avi Kivity
2009-10-21 20:29 ` Daniel Schwager
0 siblings, 2 replies; 9+ messages in thread
From: Daniel Schwager @ 2009-10-18 18:47 UTC (permalink / raw)
To: Daniel Schwager, Avi Kivity; +Cc: kvm
> > > so, setup with
> > > opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start
> > >
>
> > Use 'opreport -l'. Make sure your qemu isn't stripped.
Build kvm-86 with "./configure ... --disable-strip"
and paused vm, now I got:
opreport -l
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % image name app name
symbol name
112184 99.1919 no-vmlinux no-vmlinux (no
symbols)
266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no
symbols)
94 0.0831 libpthread-2.8.so libpthread-2.8.so
__read_nocancel
72 0.0637 opreport opreport (no
symbols)
63 0.0557 libc-2.8.so libc-2.8.so
__select_nocancel
55 0.0486 libpthread-2.8.so libpthread-2.8.so
__write_nocancel
37 0.0327 libpthread-2.8.so libpthread-2.8.so
__pthread_enable_asynccancel
29 0.0256 libpthread-2.8.so libpthread-2.8.so
__pthread_disable_asynccancel
28 0.0248 libpthread-2.8.so libpthread-2.8.so
__libc_sigaction
28 0.0248 libpthread-2.8.so libpthread-2.8.so
pthread_mutex_lock
21 0.0186 librt-2.8.so librt-2.8.so
clock_gettime
20 0.0177 bash bash (no
symbols)
19 0.0168 [vdso] (tgid:4783 range:0x7ffffb3ff000-0x7ffffb400000)
qemu-system-x86_64 (no symbols)
19 0.0168 libc-2.8.so libc-2.8.so
__libc_disable_asynccancel
17 0.0150 librt-2.8.so librt-2.8.so
timer_settime@@GLIBC_2.3.3
15 0.0133 libpthread-2.8.so libpthread-2.8.so
__pthread_mutex_unlock_usercnt
15 0.0133 librt-2.8.so librt-2.8.so
timer_gettime@@GLIBC_2.3.3
13 0.0115 oprofiled oprofiled (no
symbols)
12 0.0106 libc-2.8.so libc-2.8.so
__libc_enable_asynccancel
12 0.0106 libstdc++.so.6.0.10 libstdc++.so.6.0.10 (no
symbols)
10 0.0088 libc-2.8.so libc-2.8.so
_int_malloc
6 ...
And top tell me
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4783 root 20 0 631m 536m 1956 S 7.0 6.7 0:22.94
qemu-system-x86
But the VM is paused - definitely ..
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ?
2009-10-18 18:47 ` Daniel Schwager
@ 2009-10-18 21:57 ` Avi Kivity
2009-10-21 20:29 ` Daniel Schwager
1 sibling, 0 replies; 9+ messages in thread
From: Avi Kivity @ 2009-10-18 21:57 UTC (permalink / raw)
To: Daniel Schwager; +Cc: kvm
On 10/19/2009 03:47 AM, Daniel Schwager wrote:
> Build kvm-86 with "./configure ... --disable-strip"
> and paused vm, now I got:
>
> opreport -l
>
> CPU: CPU with timer interrupt, speed 0 MHz (estimated)
> Profiling through timer interrupt
> samples % image name app name
> symbol name
> 112184 99.1919 no-vmlinux no-vmlinux (no
> symbols)
> 266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no
> symbols)
>
Are you using the right qemu? Note 'no symbols'.
Also, most samples come from the kernel, install the kernel-debuginfo
package as well.
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Q: Stopped VM still using host cpu CPU ?
2009-10-18 18:47 ` Daniel Schwager
2009-10-18 21:57 ` Avi Kivity
@ 2009-10-21 20:29 ` Daniel Schwager
2009-10-22 13:29 ` Avi Kivity
1 sibling, 1 reply; 9+ messages in thread
From: Daniel Schwager @ 2009-10-21 20:29 UTC (permalink / raw)
To: kvm; +Cc: avi
Hi Avi,
> > > > so, setup with
> > > > opcontrol --deinit; modprobe oprofile timer=1; opcontrol
--start
> > > >
> >
> > > Use 'opreport -l'. Make sure your qemu isn't stripped.
All VM's are in paused state:
top - 22:08:15 up 2 days, 12:18, 8 users, load average: 0.12, 0.19,
0.14
Tasks: 185 total, 1 running, 182 sleeping, 2 stopped, 0 zombie
Cpu(s): 0.5%us, 1.7%sy, 0.0%ni, 97.8%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 8196468k total, 4166468k used, 4030000k free, 76044k buffers
Swap: 39780320k total, 70232k used, 39710088k free, 475876k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9587 root 20 0 1145m 1.0g 1944 S 6.6 13.1 0:28.78
qemu-system-x86
9525 root 20 0 1143m 1.0g 1900 S 6.3 13.1 0:27.26
qemu-system-x86
9305 root 20 0 1143m 1.0g 1900 S 5.0 13.1 0:22.86
qemu-system-x86
**** opreport -l --symbols | less
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % image name app name
symbol name
418814 98.5250 no-vmlinux no-vmlinux (no
symbols)
1228 0.2889 qemu-system-x86_64 qemu-system-x86_64
main_loop_wait
888 0.2089 libpthread-2.8.so libpthread-2.8.so
__read_nocancel
...
**** opreport -l
/opt/kvm-86-vnc-patch.oprofile/usr/bin/qemu-system-x86_64 --symbols |
less
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % image name symbol name
1036 52.2441 qemu-system-x86_64 main_loop_wait
155 7.8164 qemu-system-x86_64 dynticks_rearm_timer
85 4.2864 [vdso] (tgid:9525 range:0x7fff565fe000-0x7fff565ff000)
(no symbols)
77 3.8830 qemu-system-x86_64 qemu_get_clock
63 3.1770 qemu-system-x86_64 .plt
59 2.9753 qemu-system-x86_64 host_alarm_handler
57 2.8744 qemu-system-x86_64 qemu_shutdown_requested
51 2.5719 [vdso] (tgid:9305 range:0x7fff3d9ff000-0x7fff3da00000)
(no symbols)
49 2.4710 qemu-system-x86_64 sigfd_handler
47 2.3701 qemu-system-x86_64 tap_can_send
41 2.0676 [vdso] (tgid:9587 range:0x7ffffabfe000-0x7ffffabff000)
(no symbols)
38 1.9163 qemu-system-x86_64 kvm_mutex_lock
36 1.8154 qemu-system-x86_64 kvm_main_loop
31 1.5633 qemu-system-x86_64 get_clock
29 1.4624 qemu-system-x86_64 qemu_bh_poll
24 1.2103 qemu-system-x86_64 io_thread_wakeup
23 1.1599 qemu-system-x86_64 qemu_event_read
19 0.9581 qemu-system-x86_64 qemu_powerdown_requested
18 0.9077 qemu-system-x86_64 kvm_mutex_unlock
12 0.6051 qemu-system-x86_64 qemu_kvm_notify_work
12 0.6051 qemu-system-x86_64 qemu_reset_requested
11 0.5547 qemu-system-x86_64 e1000_can_receive
6 0.3026 qemu-system-x86_64 slirp_is_inited
3 0.1513 qemu-system-x86_64 qemu_notify_event
1 0.0504 qemu-system-x86_64 qemu_set_fd_handler2
I do not understand, why only ~1036 kvm-samples produces a load of (in
sum) 18% ...
I'm not able to convert vmlinuz to vmlinux (for profiling the kernel
...) on my old FC9 - sorry.
Because I cannot update my old fc9, I will move to CentOS later - and
hopefully
get the kernel profiling running.
May these oprofile-information will help you to track down the problem -
if not,
I try to came back soon with kernel-profiling information on CentOS.
regards
Danny
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Q: Stopped VM still using host cpu CPU ?
2009-10-21 20:29 ` Daniel Schwager
@ 2009-10-22 13:29 ` Avi Kivity
0 siblings, 0 replies; 9+ messages in thread
From: Avi Kivity @ 2009-10-22 13:29 UTC (permalink / raw)
To: Daniel Schwager; +Cc: kvm
On 10/21/2009 10:29 PM, Daniel Schwager wrote:
>
> **** opreport -l --symbols | less
> CPU: CPU with timer interrupt, speed 0 MHz (estimated)
> Profiling through timer interrupt
> samples % image name app name
> symbol name
> 418814 98.5250 no-vmlinux no-vmlinux (no
> symbols)
> 1228 0.2889 qemu-system-x86_64 qemu-system-x86_64
> main_loop_wait
> 888 0.2089 libpthread-2.8.so libpthread-2.8.so
> __read_nocancel
> ...
>
>
Sorry, I gave you the wrong instructions. Since most of the time is
spent in vmlinux, we want to trace vmlinux, not qemu.
Three ways to see what is happening:
- set up vmlinux (you'll need to install the correct kernel-debuginfo
package to get this).
- strace -p `pgrep qemu` and see what syscalls take place
- run 'top' and look at the wchan field ('f' 'y' will show it if it is
hidden)
Probably easiest in reverse order as listed.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-10-22 13:29 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-10-01 10:32 Q: Stopped VM still using host cpu CPU ? Daniel Schwager
2009-10-01 11:47 ` Daniel Schwager
[not found] ` <4AC8B6D9.7070003@redhat.com>
[not found] ` <EB31672367A401439CD5A4A10889D57B02118226@exchange.dtnet.de>
[not found] ` <4AC8C018.6050302@redhat.com>
2009-10-16 21:35 ` Daniel Schwager
2009-10-18 6:59 ` Avi Kivity
2009-10-18 18:29 ` Daniel Schwager
2009-10-18 18:47 ` Daniel Schwager
2009-10-18 21:57 ` Avi Kivity
2009-10-21 20:29 ` Daniel Schwager
2009-10-22 13:29 ` Avi Kivity
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.