frequent guest lockup (and mouse/keyboard problems)

frequent guest lockup (and mouse/keyboard problems)

[Leonard Norrgård]

The guest OS is 32-bit Debian Etch (KDE), the host AMD64 X2 5200+.
Kvm runs in SDL, command line: kvm -m 80 -hda disk.qcow

All of this happens with kvm trunk.

Quite often, once or more per hour, the guest hangs/freezes, consuming
nearly 100% kernel cpu (user cpu usage is 0,00%) on the host. I use
the guest for browsing the net and irc, so it is used many hours/day
for real work.

The qemu monitor in the guest still works nicely, thoughm such as
stopping and starting the cpu, inspecting registers etc. Looking at
guest $eip, it seems to be in a loop at about 0xc011ac70 - 0xc011b508.
One can also release the mouse grab and enter the grab again. I
haven't found a way to make the guest exit the cpu loop.

Output from kvm_stat when the guest freezes is:

kvm statistics

 exits               0       0
 halt_exits   20050310       0
 invlpg              0       0
 io_exits    170429877       0
 irq_exits           0       0
 irq_window          0       0
 mmio_exits    9859649       0
 pf_fixed    285553491       0
 pf_guest     14202518       0
 request_irq    489317       0
 signal_exit   3584242     249
 tlb_flush    22082315       0


The keyboard/mouse problems occure much more often in full screen mode.

Very often (nearly every minute), the guest mouse will do something
random without warning, like:

- move instantly to some other location on the screen

- click a mouse button (any of the three buttons). (If mouse happens
  to be over a web browser window and button 2 is pressed, firefox
  will try to treat the clipboard text as an url and go to that page,
  so this sometimes manifests itself in the web browser doing
  seemingly random stuff).

Current keyboard problems are:

- keys sometimes repeat on a normal keypress and don't stop until
  another key is pressed (nearly every minute, sometimes more often)

- any of the Ctrl, Shift and Alt seem to get automagically pressed and
  stuck (several times per hour)


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Dor Laor]

>
>The guest OS is 32-bit Debian Etch (KDE), the host AMD64 X2 5200+.
>Kvm runs in SDL, command line: kvm -m 80 -hda disk.qcow
>
>All of this happens with kvm trunk.


I'm trying to replicate the problem. Running your Guest on AMD operton.
So far every thing works ;) Let's see if it last the night.
If so I'll decrease the memory from 384 to your 80M.


>
>Quite often, once or more per hour, the guest hangs/freezes, consuming
>nearly 100% kernel cpu (user cpu usage is 0,00%) on the host. I use
>the guest for browsing the net and irc, so it is used many hours/day
>for real work.
>
>The qemu monitor in the guest still works nicely, thoughm such as
>stopping and starting the cpu, inspecting registers etc. Looking at
>guest $eip, it seems to be in a loop at about 0xc011ac70 - 0xc011b508.
>One can also release the mouse grab and enter the grab again. I
>haven't found a way to make the guest exit the cpu loop.
>
>Output from kvm_stat when the guest freezes is:
>
>kvm statistics
>
> exits               0       0
> halt_exits   20050310       0
> invlpg              0       0
> io_exits    170429877       0
> irq_exits           0       0
> irq_window          0       0
> mmio_exits    9859649       0
> pf_fixed    285553491       0
> pf_guest     14202518       0
> request_irq    489317       0
> signal_exit   3584242     249
> tlb_flush    22082315       0
>
>
>The keyboard/mouse problems occure much more often in full screen mode.
>
>Very often (nearly every minute), the guest mouse will do something
>random without warning, like:
>
>- move instantly to some other location on the screen
>
>- click a mouse button (any of the three buttons). (If mouse happens
>  to be over a web browser window and button 2 is pressed, firefox
>  will try to treat the clipboard text as an url and go to that page,
>  so this sometimes manifests itself in the web browser doing
>  seemingly random stuff).
>
>Current keyboard problems are:
>
>- keys sometimes repeat on a normal keypress and don't stop until
>  another key is pressed (nearly every minute, sometimes more often)
>
>- any of the Ctrl, Shift and Alt seem to get automagically pressed and
>  stuck (several times per hour)
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job
>easier.
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>kvm-devel mailing list
>kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Leonard Norrgard]

Dor Laor wrote:
>> The guest OS is 32-bit Debian Etch (KDE), the host AMD64 X2 5200+.
>> Kvm runs in SDL, command line: kvm -m 80 -hda disk.qcow
>>
>> All of this happens with kvm trunk.
>>     
>
>
> I'm trying to replicate the problem. Running your Guest on AMD operton.
> So far every thing works ;) Let's see if it last the night.
> If so I'll decrease the memory from 384 to your 80M.
>   
It doesn't seem to be related to the amount of memory - I've tried
various amounts.

The fastest way I know to replicate this currently, is to install the
sun-java5-jre and sun-java5-plugin packages (in non-free) in the guest.
Then:

* Switch kvm to full-screen mode
* Open firefox and go to the, uh, operating system virtualization test
page at http://uk.games.yahoo.com/d/zpbook.html :)
* Once the java applet has loaded, click Play, select one of the images,
click start.
* Now solve the puzzle (rotate with right mouse button, drag with left).

Almost for every piece, I have a mouse hickup of one sort or other.

I just verified that this is still the case with the released 2.6.20
kernel (but kvm disabled in the kernel config and the modules instead
installed from kvm svn) and kvm svn r4384 (less than an hour old).

-- Leonard


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Leonard Norrgard]

Seeing the message from Matthew Hall on "KVM, AMD & interrupts" (25th
Jan.), I tried rebooting the host kernel with "noapic", "nolapic" and
"noapic nolapic".

- For noapic, no change in behaviour was seen. Mouse/keyboard problem
still frequent.

- For nolapic, the network didn't come up (aborted test there, but I
guess I could try that a bit more and testing locally only), and usb was
experiencing trouble.

- For "noapic nolapic" the kernel refused to continue after an unhandled
interrupt so testing was impossible.

selected information from lshal(1):

system.product = 'MS-7280 0A' (string)
system.vendor = 'MSI' (string)
smbios.system.version = '0A' (string)
smbios.system.product = 'MS-7280' (string)
smbios.system.manufacturer = 'MSI' (string)
smbios.bios.release_date = '07/17/2006' (string)
smbios.bios.version = '080013' (string)
smbios.bios.vendor = 'American Megatrends Inc.' (string)

(The motherboard is an MSI K9A Platinum).

-- Leonard


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Dor Laor]

>> I'm trying to replicate the problem. Running your Guest on AMD
operton.
>> So far every thing works ;) Let's see if it last the night.
>> If so I'll decrease the memory from 384 to your 80M.
>>
>It doesn't seem to be related to the amount of memory - I've tried
>various amounts.
>
>The fastest way I know to replicate this currently, is to install the
>sun-java5-jre and sun-java5-plugin packages (in non-free) in the guest.
>Then:
>
>* Switch kvm to full-screen mode
>* Open firefox and go to the, uh, operating system virtualization test
>page at http://uk.games.yahoo.com/d/zpbook.html :)
>* Once the java applet has loaded, click Play, select one of the
images,
>click start.
>* Now solve the puzzle (rotate with right mouse button, drag with
left).
>
>Almost for every piece, I have a mouse hickup of one sort or other.
>
>I just verified that this is still the case with the released 2.6.20
>kernel (but kvm disabled in the kernel config and the modules instead
>installed from kvm svn) and kvm svn r4384 (less than an hour old).

Finally I manage to reproduce the problem. I did manage to play the
puzzle without problems, but I discovered that under load there are
problems with the mouse.
I didn't have any problems with 100% cpu consumption (Operton cpu).

I believe the mouse problems are caused by using the relative mouse
coordinates. So when the host is loaded there is latency between the
host reading and the guest irq injection.

Can you try using usbtablet mouse? (Note that you'll need to install
evtoucher driver).


>
>-- Leonard


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Leonard Norrgard]

Dor Laor wrote:
> Finally I manage to reproduce the problem. I did manage to play the
> puzzle without problems, but I discovered that under load there are
> problems with the mouse.
> I didn't have any problems with 100% cpu consumption (Operton cpu).
>   
I'll try to find a good way to reproduce the 100% cpu problem. I have a
feeling it may be network related (could be a qemu 0.8.2 bug, even).
> I believe the mouse problems are caused by using the relative mouse
> coordinates. So when the host is loaded there is latency between the
> host reading and the guest irq injection.
>   
Perhaps, but his wouldn't explain why the mouse buttons are randomly
clicked. It seems that this often happens at the very same time as the
pointer jumps somewhere.
> Can you try using usbtablet mouse? (Note that you'll need to install
> evtoucher driver).
>   
No trouble at all with -usbdevice tablet (Btw, new debian supporting
package coming up to enable this for linux guests, did the basic work on
it in order to do this test), except that the pointer doesn't move
smoothly when the guest is run in kvm. Using qemu 0.9.0 + kqemu (GPL:d
now!), the pointer moves very smoothly with -usbdevice tablet - it's in
fact difficult to distinguish from native. (There are also no mouse
problems when running the guest with kvm -no-kvm).

The keyboard problems are very easy to reproduce here. Just open a text
editor in the guest and try to edit something. Move around in the text,
make edits. A key will get stuck very quickly, causing auto-repeat.

In fact, keys get stuck so often in the guest, that I've installed the
Debian KDE package named kbstate. After installation, right click on the
bottom panel and select "Add applet to panel...", then add the applet
named "Keyboard Status Applet". This gives a visual indicator as to
which key is stuck right now.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Dor Laor]

>> Finally I manage to reproduce the problem. I did manage to play the
>> puzzle without problems, but I discovered that under load there are
>> problems with the mouse.
>> I didn't have any problems with 100% cpu consumption (Operton cpu).
>>
>I'll try to find a good way to reproduce the 100% cpu problem. I have a
>feeling it may be network related (could be a qemu 0.8.2 bug, even).
>> I believe the mouse problems are caused by using the relative mouse
>> coordinates. So when the host is loaded there is latency between the
>> host reading and the guest irq injection.
>>
>Perhaps, but his wouldn't explain why the mouse buttons are randomly
>clicked. It seems that this often happens at the very same time as the
>pointer jumps somewhere.
>> Can you try using usbtablet mouse? (Note that you'll need to install
>> evtoucher driver).
>>
>No trouble at all with -usbdevice tablet (Btw, new debian supporting
>package coming up to enable this for linux guests, did the basic work
on
>it in order to do this test), except that the pointer doesn't move
>smoothly when the guest is run in kvm. Using qemu 0.9.0 + kqemu (GPL:d
>now!), the pointer moves very smoothly with -usbdevice tablet - it's in
>fact difficult to distinguish from native. (There are also no mouse
>problems when running the guest with kvm -no-kvm).
>
>The keyboard problems are very easy to reproduce here. Just open a text
>editor in the guest and try to edit something. Move around in the text,
>make edits. A key will get stuck very quickly, causing auto-repeat.
>
>In fact, keys get stuck so often in the guest, that I've installed the
>Debian KDE package named kbstate. After installation, right click on
the
>bottom panel and select "Add applet to panel...", then add the applet
>named "Keyboard Status Applet". This gives a visual indicator as to
>which key is stuck right now.


Well I didn't know you were experiencing keyboard problems as well.
Also if the mouse does not move smoothly it's an interrupt problem.
I'll now look deeper into it, I was using sdl over vnc to the host
because it's remote, and maybe because of that I got fewer problems.

It might take me a day to clear my desk from other issues thought ;)


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: frequent guest lockup (and mouse/keyboard problems)

[Dor Laor]

[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]

Hi Leonard,

Can you try the attached patch and tell me what's going on?
Sorry for the delay response, I was a little busy.

The patch zero's the svm interrupt vector after vmexit, right after we
checked for pending irq's that were not injected.
Although the IRQ_MASK is zerod I suspect it might get injected again.

Dor.


>-----Original Message-----
>From: Leonard Norrgard [mailto:leonard.norrgard-g2GXA8XeJSExHbG02/KK1g@public.gmane.org]
>Sent: Tuesday, February 06, 2007 10:19 AM
>To: Dor Laor
>Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>Subject: Re: [kvm-devel] frequent guest lockup (and mouse/keyboard
>problems)
>
>Dor Laor wrote:
>> Finally I manage to reproduce the problem. I did manage to play the
>> puzzle without problems, but I discovered that under load there are
>> problems with the mouse.
>> I didn't have any problems with 100% cpu consumption (Operton cpu).
>>
>I'll try to find a good way to reproduce the 100% cpu problem. I have a
>feeling it may be network related (could be a qemu 0.8.2 bug, even).
>> I believe the mouse problems are caused by using the relative mouse
>> coordinates. So when the host is loaded there is latency between the
>> host reading and the guest irq injection.
>>
>Perhaps, but his wouldn't explain why the mouse buttons are randomly
>clicked. It seems that this often happens at the very same time as the
>pointer jumps somewhere.
>> Can you try using usbtablet mouse? (Note that you'll need to install
>> evtoucher driver).
>>
>No trouble at all with -usbdevice tablet (Btw, new debian supporting
>package coming up to enable this for linux guests, did the basic work
on
>it in order to do this test), except that the pointer doesn't move
>smoothly when the guest is run in kvm. Using qemu 0.9.0 + kqemu (GPL:d
>now!), the pointer moves very smoothly with -usbdevice tablet - it's in
>fact difficult to distinguish from native. (There are also no mouse
>problems when running the guest with kvm -no-kvm).
>
>The keyboard problems are very easy to reproduce here. Just open a text
>editor in the guest and try to edit something. Move around in the text,
>make edits. A key will get stuck very quickly, causing auto-repeat.
>
>In fact, keys get stuck so often in the guest, that I've installed the
>Debian KDE package named kbstate. After installation, right click on
the
>bottom panel and select "Add applet to panel...", then add the applet
>named "Keyboard Status Applet". This gives a visual indicator as to
>which key is stuck right now.


[-- Attachment #2: irq.svm.patch --]
[-- Type: application/octet-stream, Size: 349 bytes --]

Index: svm.c
===================================================================
--- svm.c	(revision 4390)
+++ svm.c	(working copy)
@@ -1360,6 +1360,7 @@
 		control->int_ctl &= ~V_IRQ_MASK;
 		push_irq(vcpu, control->int_vector);
 	}
+	control->int_vector = 0;
 
 	vcpu->interrupt_window_open =
 		!(control->int_state & SVM_INTERRUPT_SHADOW_MASK);

[-- Attachment #3: Type: text/plain, Size: 374 bytes --]

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: frequent guest lockup (and mouse/keyboard problems)

[Leonard Norrgard]

Dor Laor wrote:
> Hi Leonard,
>
> Can you try the attached patch and tell me what's going on?
> Sorry for the delay response, I was a little busy.
>
> The patch zero's the svm interrupt vector after vmexit, right after we
> checked for pending irq's that were not injected.
> Although the IRQ_MASK is zerod I suspect it might get injected again.
>   
With that patch applied, the 32-bit guest freezes before it has
completed booting, so it ends up in the same state  I've describes as a
guest freeze in this thread before: it ends up doing about 250
signal_exit/s but nothing else.

Kills 64-bit guests as well.
> Dor.
>   
-- Leonard

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642