WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Hi all,

I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 
release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.

I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, 
however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the 
login screen, the WinXP guest goes BSOD with the following error: 
DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.

I've confirmed by switching between the two installations several times 
that the error consistently occurs with qemu-kvm-0.12.1.2 but not 
qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 Debian 
Lenny host with a 2.6.32.2 kernel on Intel.


Many thanks,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/05/2010 06:09 PM, Mark Cave-Ayland wrote:
> Hi all,
>
> I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 
> release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
>
> I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, 
> however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the 
> login screen, the WinXP guest goes BSOD with the following error: 
> DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
>
> I've confirmed by switching between the two installations several 
> times that the error consistently occurs with qemu-kvm-0.12.1.2 but 
> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 
> Debian Lenny host with a 2.6.32.2 kernel on Intel.

It's not a known issue.  What's your command line?  What's your host cpu 
type?

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> I've confirmed by switching between the two installations several 
>> times that the error consistently occurs with qemu-kvm-0.12.1.2 but 
>> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 
>> Debian Lenny host with a 2.6.32.2 kernel on Intel.
> 
> It's not a known issue.  What's your command line?  What's your host cpu 
> type?

Hi Avi,

The command line is created using libvirt 0.7.5 (compiled by me) and 
looks like this:

/usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 512 -smp 1 -name winxp -uuid 
f99c79b6-2706-dff0-af5c-cea53559f89c -chardev 
socket,id=monitor,path=/usr/local/var/lib/libvirt/qemu/winxp.monitor,server,nowait 
-monitor chardev:monitor -boot c -drive 
file=/home/images/winxp.raw,if=ide,index=0,boot=on -net 
nic,macaddr=52:54:00:c4:90:94,vlan=0,model=rtl8139,name=rtl8139.0 -net 
tap,fd=41,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc :5 
-vga cirrus

While the host /proc/cpuinfo contents look like this:

demo-laptop:/home/images# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T6670  @ 2.20GHz
stepping        : 10
cpu MHz         : 1200.000
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni 
dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave 
lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4388.31
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:


I may also be able to arrange for remote SSH access into the box - mail 
me off-list if you are interested.


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2 
>> release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
>>
>> I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1, 
>> however under qemu-kvm-0.12.1.2 a couple of seconds after reaching the 
>> login screen, the WinXP guest goes BSOD with the following error: 
>> DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
>>
>> I've confirmed by switching between the two installations several 
>> times that the error consistently occurs with qemu-kvm-0.12.1.2 but 
>> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64 
>> Debian Lenny host with a 2.6.32.2 kernel on Intel.
> 
> It's not a known issue.  What's your command line?  What's your host cpu 
> type?

Hi Avi,

Good news - I downloaded the userspace git repository and managed to 
identify the offending commit between 0.12.1.1 and 0.12.1.2 using git 
bisect:


4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit
commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
Author: Avi Kivity <avi@redhat.com>
Date:   Mon Dec 28 10:48:00 2009 +0200

     Reinstate cpuid vendor override when kvm is enabled

     Due to upstream qemu changes we no longer expose the host cpu vendor id
     to the guest.  This leads to failures when the syscall/sysenter 
instructions
     are used in compatibility mode.

     Change the default to override when kvm is enabled.

     Signed-off-by: Avi Kivity <avi@redhat.com>
     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Marcelo Tosatti]

On Wed, Jan 06, 2010 at 11:25:14AM +0000, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>>> I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2  
>>> release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
>>>
>>> I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1,  
>>> however under qemu-kvm-0.12.1.2 a couple of seconds after reaching 
>>> the login screen, the WinXP guest goes BSOD with the following error: 
>>> DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
>>>
>>> I've confirmed by switching between the two installations several  
>>> times that the error consistently occurs with qemu-kvm-0.12.1.2 but  
>>> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64  
>>> Debian Lenny host with a 2.6.32.2 kernel on Intel.
>>
>> It's not a known issue.  What's your command line?  What's your host 
>> cpu type?
>
> Hi Avi,
>
> Good news - I downloaded the userspace git repository and managed to  
> identify the offending commit between 0.12.1.1 and 0.12.1.2 using git  
> bisect:
>
>
> 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit
> commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
> Author: Avi Kivity <avi@redhat.com>
> Date:   Mon Dec 28 10:48:00 2009 +0200
>
>     Reinstate cpuid vendor override when kvm is enabled
>
>     Due to upstream qemu changes we no longer expose the host cpu vendor id
>     to the guest.  This leads to failures when the syscall/sysenter  
> instructions
>     are used in compatibility mode.
>
>     Change the default to override when kvm is enabled.
>
>     Signed-off-by: Avi Kivity <avi@redhat.com>
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
>
> HTH,
>
> Mark.

Mark,

Thanks for tracking it down. Is there any difference with "-cpu host"
option?

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Gleb Natapov]

On Wed, Jan 06, 2010 at 10:29:18AM -0200, Marcelo Tosatti wrote:
> On Wed, Jan 06, 2010 at 11:25:14AM +0000, Mark Cave-Ayland wrote:
> > Avi Kivity wrote:
> >
> >>> I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2  
> >>> release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
> >>>
> >>> I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1,  
> >>> however under qemu-kvm-0.12.1.2 a couple of seconds after reaching 
> >>> the login screen, the WinXP guest goes BSOD with the following error: 
> >>> DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
> >>>
> >>> I've confirmed by switching between the two installations several  
> >>> times that the error consistently occurs with qemu-kvm-0.12.1.2 but  
> >>> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64  
> >>> Debian Lenny host with a 2.6.32.2 kernel on Intel.
> >>
> >> It's not a known issue.  What's your command line?  What's your host 
> >> cpu type?
> >
> > Hi Avi,
> >
> > Good news - I downloaded the userspace git repository and managed to  
> > identify the offending commit between 0.12.1.1 and 0.12.1.2 using git  
> > bisect:
> >
> >
> > 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit
> > commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
> > Author: Avi Kivity <avi@redhat.com>
> > Date:   Mon Dec 28 10:48:00 2009 +0200
> >
> >     Reinstate cpuid vendor override when kvm is enabled
> >
> >     Due to upstream qemu changes we no longer expose the host cpu vendor id
> >     to the guest.  This leads to failures when the syscall/sysenter  
> > instructions
> >     are used in compatibility mode.
> >
> >     Change the default to override when kvm is enabled.
> >
> >     Signed-off-by: Avi Kivity <avi@redhat.com>
> >     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >
> >
> > HTH,
> >
> > Mark.
> 
> Mark,
> 
> Thanks for tracking it down. Is there any difference with "-cpu host"
> option?
> 
And what output of "cat /proc/cpuinfo" on the host looks like?

--
			Gleb.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Gleb Natapov]

On Wed, Jan 06, 2010 at 02:33:36PM +0200, Gleb Natapov wrote:
> On Wed, Jan 06, 2010 at 10:29:18AM -0200, Marcelo Tosatti wrote:
> > On Wed, Jan 06, 2010 at 11:25:14AM +0000, Mark Cave-Ayland wrote:
> > > Avi Kivity wrote:
> > >
> > >>> I think I'm experiencing a regression with the new qemu-kvm-0.12.1.2  
> > >>> release compared to qemu-kvm-0.12.1.1 with a WinXP guest on Linux.
> > >>>
> > >>> I can boot my WinXP guest without a problem under qemu-kvm-0.12.1.1,  
> > >>> however under qemu-kvm-0.12.1.2 a couple of seconds after reaching 
> > >>> the login screen, the WinXP guest goes BSOD with the following error: 
> > >>> DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS.
> > >>>
> > >>> I've confirmed by switching between the two installations several  
> > >>> times that the error consistently occurs with qemu-kvm-0.12.1.2 but  
> > >>> not qemu-kvm-0.12.1.1. Is this a known issue? This is on an x86_64  
> > >>> Debian Lenny host with a 2.6.32.2 kernel on Intel.
> > >>
> > >> It's not a known issue.  What's your command line?  What's your host 
> > >> cpu type?
> > >
> > > Hi Avi,
> > >
> > > Good news - I downloaded the userspace git repository and managed to  
> > > identify the offending commit between 0.12.1.1 and 0.12.1.2 using git  
> > > bisect:
> > >
> > >
> > > 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit
> > > commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
> > > Author: Avi Kivity <avi@redhat.com>
> > > Date:   Mon Dec 28 10:48:00 2009 +0200
> > >
> > >     Reinstate cpuid vendor override when kvm is enabled
> > >
> > >     Due to upstream qemu changes we no longer expose the host cpu vendor id
> > >     to the guest.  This leads to failures when the syscall/sysenter  
> > > instructions
> > >     are used in compatibility mode.
> > >
> > >     Change the default to override when kvm is enabled.
> > >
> > >     Signed-off-by: Avi Kivity <avi@redhat.com>
> > >     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> > >
> > >
> > > HTH,
> > >
> > > Mark.
> > 
> > Mark,
> > 
> > Thanks for tracking it down. Is there any difference with "-cpu host"
> > option?
> > 
> And what output of "cat /proc/cpuinfo" on the host looks like?
> 
Ah sorry, you already did that.

--
			Gleb.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Marcelo Tosatti wrote:

 > Mark,
 >
 > Thanks for tracking it down. Is there any difference with "-cpu host"
 > option?

Yeah; if I launch 0.12.1.1 which normally works with an additional "-cpu 
host" option then kvm crashes in exactly the same way.


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/06/2010 01:25 PM, Mark Cave-Ayland wrote:
> Good news - I downloaded the userspace git repository and managed to 
> identify the offending commit between 0.12.1.1 and 0.12.1.2 using git 
> bisect:
>
>
> 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1 is first bad commit
> commit 4dad7ff32aa6dcf18cef0c606d8fb43ff0b939a1
> Author: Avi Kivity <avi@redhat.com>
> Date:   Mon Dec 28 10:48:00 2009 +0200
>
>     Reinstate cpuid vendor override when kvm is enabled
>
>     Due to upstream qemu changes we no longer expose the host cpu 
> vendor id
>     to the guest.  This leads to failures when the syscall/sysenter 
> instructions
>     are used in compatibility mode.
>
>     Change the default to override when kvm is enabled.
>
>     Signed-off-by: Avi Kivity <avi@redhat.com>
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>

Did you install using 0.12.1.1 and then run using 0.12.1.2?  If so, it 
seems Windows XP is not able to move from AMD to Intel.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> Did you install using 0.12.1.1 and then run using 0.12.1.2?  If so, it 
> seems Windows XP is not able to move from AMD to Intel.

No, although the original VM was built in VirtualBox if that makes a 
difference? I had to manually implement the "merge ide" fix here 
(http://support.microsoft.com/kb/314082) in order to bring up the VM 
under KVM but it has worked fine until the upgrade to 0.12.1.2.


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Gleb Natapov]

On Wed, Jan 06, 2010 at 12:50:29PM +0000, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
> 
> >Did you install using 0.12.1.1 and then run using 0.12.1.2?  If
> >so, it seems Windows XP is not able to move from AMD to Intel.
> 
> No, although the original VM was built in VirtualBox if that makes a
> difference? I had to manually implement the "merge ide" fix here
> (http://support.microsoft.com/kb/314082) in order to bring up the VM
> under KVM but it has worked fine until the upgrade to 0.12.1.2.
> 
> 
Can you start is in safe mode and see which driver fails?

--
			Gleb.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Gleb Natapov wrote:

> Can you start is in safe mode and see which driver fails?

Hmmm. Booting in Safe Mode seems to work fine, although the next attempt 
to boot into Normal Mode returns a BSOD with "INTERNAL_POWER_ERROR" 
which is new to me. Subsequent reboots then return to the 
"DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD.


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Yaniv Kaul]

On 1/6/2010 3:14 PM, Mark Cave-Ayland wrote:
> Gleb Natapov wrote:
>
>> Can you start is in safe mode and see which driver fails?
>
> Hmmm. Booting in Safe Mode seems to work fine, although the next 
> attempt to boot into Normal Mode returns a BSOD with 
> "INTERNAL_POWER_ERROR" which is new to me. Subsequent reboots then 
> return to the "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" 
> BSOD.
>
>
> ATB,
>
> Mark.
>

Try logging the boot (possible via the F8 menu as well).
Y.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Gleb Natapov]

On Wed, Jan 06, 2010 at 12:50:29PM +0000, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
> 
> >Did you install using 0.12.1.1 and then run using 0.12.1.2?  If
> >so, it seems Windows XP is not able to move from AMD to Intel.
> 
> No, although the original VM was built in VirtualBox if that makes a
> difference? I had to manually implement the "merge ide" fix here
> (http://support.microsoft.com/kb/314082) in order to bring up the VM
> under KVM but it has worked fine until the upgrade to 0.12.1.2.
> 
> 
Which version of kvm you've used before 0.12.1.1? Does this VM work on
0.11?

--
			Gleb.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/06/2010 02:50 PM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>> Did you install using 0.12.1.1 and then run using 0.12.1.2?  If so, 
>> it seems Windows XP is not able to move from AMD to Intel.
>
> No, although the original VM was built in VirtualBox if that makes a 
> difference? I had to manually implement the "merge ide" fix here 
> (http://support.microsoft.com/kb/314082) in order to bring up the VM 
> under KVM but it has worked fine until the upgrade to 0.12.1.2.
>

It probably did make some kind of difference.  Please try a clean install.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> It probably did make some kind of difference.  Please try a clean install.

After several hours of testing, I've finally found out what the problem is.

I tried a clean WinXP guest install and that worked, so it was obviously 
a driver issue. After disabling various drivers in the WinXP guest, I 
didn't get anywhere so I decided to take a break and test Marcelo's VNC 
patch. With this applied, I could actually see all of the information in 
the BSOD which showed the error was in intelppm.sys.

A quick search took me to this page here: 
http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx 
which explains the issue in more detail. I first tried disabling the 
intelppm driver and rebooting, but that didn't make a difference; 
however disabling the Processor driver worked and my guest VM booted in 
Normal Mode :)

I think the issue is probably similar to that explained in the article 
above; with a new processor reported to the guest, the internal 
processor driver tries to upload some kind of microcode to the new 
device which fails and causes the guest to fall over. Can we teach KVM 
to silently discard these kinds of updates?


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Mark Cave-Ayland wrote:

> A quick search took me to this page here: 
> http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx 
> which explains the issue in more detail. I first tried disabling the 
> intelppm driver and rebooting, but that didn't make a difference; 
> however disabling the Processor driver worked and my guest VM booted in 
> Normal Mode :)

I've just re-created the KVM image fresh from the VDI image once again 
and can confirm that disabling just the Processor driver is enough to 
allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the 
default for "-cpu host" should not be changed in a micro release as 
there is a risk of breaking existing VMs?


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/06/2010 07:08 PM, Mark Cave-Ayland wrote:
> Mark Cave-Ayland wrote:
>
>> A quick search took me to this page here: 
>> http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx 
>> which explains the issue in more detail. I first tried disabling the 
>> intelppm driver and rebooting, but that didn't make a difference; 
>> however disabling the Processor driver worked and my guest VM booted 
>> in Normal Mode :)
>
> I've just re-created the KVM image fresh from the VDI image once again 
> and can confirm that disabling just the Processor driver is enough to 
> allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the 
> default for "-cpu host" should not be changed in a micro release as 
> there is a risk of breaking existing VMs?
>

That was actually a fix for a regression relative to 0.11.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> I've just re-created the KVM image fresh from the VDI image once again 
>> and can confirm that disabling just the Processor driver is enough to 
>> allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. Perhaps the 
>> default for "-cpu host" should not be changed in a micro release as 
>> there is a risk of breaking existing VMs?
>>
> 
> That was actually a fix for a regression relative to 0.11.

Really? Damn :(  Any pointers towards the relevant bug in the bug tracker?


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/07/2010 11:57 AM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>>> I've just re-created the KVM image fresh from the VDI image once 
>>> again and can confirm that disabling just the Processor driver is 
>>> enough to allow the guest WinXP VM to function in qemu-kvm-0.12.1.2. 
>>> Perhaps the default for "-cpu host" should not be changed in a micro 
>>> release as there is a risk of breaking existing VMs?
>>>
>>
>> That was actually a fix for a regression relative to 0.11.
>
> Really? Damn :(  Any pointers towards the relevant bug in the bug 
> tracker?
>
>

No, it was reported on list.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/06/2010 05:45 PM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>> It probably did make some kind of difference.  Please try a clean 
>> install.
>
> After several hours of testing, I've finally found out what the 
> problem is.
>
> I tried a clean WinXP guest install and that worked, so it was 
> obviously a driver issue. After disabling various drivers in the WinXP 
> guest, I didn't get anywhere so I decided to take a break and test 
> Marcelo's VNC patch. With this applied, I could actually see all of 
> the information in the BSOD which showed the error was in intelppm.sys.
>
> A quick search took me to this page here: 
> http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/24/484461.aspx 
> which explains the issue in more detail. I first tried disabling the 
> intelppm driver and rebooting, but that didn't make a difference; 
> however disabling the Processor driver worked and my guest VM booted 
> in Normal Mode :)
>
> I think the issue is probably similar to that explained in the article 
> above; with a new processor reported to the guest, the internal 
> processor driver tries to upload some kind of microcode to the new 
> device which fails and causes the guest to fall over. Can we teach KVM 
> to silently discard these kinds of updates?
>

Can you try loading kvm.ko with the ignore_msrs module parameter set?

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> I think the issue is probably similar to that explained in the article 
>> above; with a new processor reported to the guest, the internal 
>> processor driver tries to upload some kind of microcode to the new 
>> device which fails and causes the guest to fall over. Can we teach KVM 
>> to silently discard these kinds of updates?
>>
> 
> Can you try loading kvm.ko with the ignore_msrs module parameter set?

Hi Avi,

I've just done a quick test re-enabling processor.sys on my WinXP guest 
and then did the following:

virsh stop winxp
rmmod kvm_intel
rmmod kvm
modprobe kvm ignore_msrs=1
modprobe kvm_intel
virsh start winxp

Unfortunately it still crashes with the same 
"DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/07/2010 11:55 AM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>>> I think the issue is probably similar to that explained in the 
>>> article above; with a new processor reported to the guest, the 
>>> internal processor driver tries to upload some kind of microcode to 
>>> the new device which fails and causes the guest to fall over. Can we 
>>> teach KVM to silently discard these kinds of updates?
>>>
>>
>> Can you try loading kvm.ko with the ignore_msrs module parameter set?
>
> Hi Avi,
>
> I've just done a quick test re-enabling processor.sys on my WinXP 
> guest and then did the following:
>
> virsh stop winxp
> rmmod kvm_intel
> rmmod kvm
> modprobe kvm ignore_msrs=1
> modprobe kvm_intel
> virsh start winxp
>
> Unfortunately it still crashes with the same 
> "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(

Well, don't do that then.  Is there any specific functionality in 
processor.sys that you're missing?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> Hi Avi,
>>
>> I've just done a quick test re-enabling processor.sys on my WinXP 
>> guest and then did the following:
>>
>> virsh stop winxp
>> rmmod kvm_intel
>> rmmod kvm
>> modprobe kvm ignore_msrs=1
>> modprobe kvm_intel
>> virsh start winxp
>>
>> Unfortunately it still crashes with the same 
>> "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(
> 
> Well, don't do that then.  Is there any specific functionality in 
> processor.sys that you're missing?

No, not at all. My only concern was that the VM had been running 
absolutely fine under older KVM and VirtualBox until the upgrade from 
0.12.1.1 to 0.12.1.2 which made me think there had been a regression 
somewhere along the line.

I appreciate from tracking both qemu and kvm mailing lists that there is 
currently a lot of rapid development occuring across both QEMU and KVM, 
and hence sometimes things can break. It would be interesting to find 
out exactly *why* this doesn't work in KVM and so I can provide 
debugging assistance if you can point me in the right direction.

At the moment, I'm just happy that I can run the VM under KVM even with 
the processor.sys driver disabled. At least by bringing up the problem 
and solution on this mailing list thread then the solution is documented 
for other people who find themselves in the same situation.


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/25/2010 05:15 PM, Mark Cave-Ayland wrote:
>>> Unfortunately it still crashes with the same 
>>> "DRIVER_UNLOADED_WITHOUT_CANCELING_PENDING_OPERATIONS" BSOD :(
>>
>> Well, don't do that then.  Is there any specific functionality in 
>> processor.sys that you're missing?
>
>
> No, not at all. My only concern was that the VM had been running 
> absolutely fine under older KVM and VirtualBox until the upgrade from 
> 0.12.1.1 to 0.12.1.2 which made me think there had been a regression 
> somewhere along the line.

Well, there was a regression, but it was in 0.12.1.1.

There were two bugs involved, a serious one (that caused the cpuid to 
show up as AMD) hiding the less serious one (that causes processor.sys 
to BSOD).

>
> I appreciate from tracking both qemu and kvm mailing lists that there 
> is currently a lot of rapid development occuring across both QEMU and 
> KVM, and hence sometimes things can break. It would be interesting to 
> find out exactly *why* this doesn't work in KVM and so I can provide 
> debugging assistance if you can point me in the right direction.
>
> At the moment, I'm just happy that I can run the VM under KVM even 
> with the processor.sys driver disabled. At least by bringing up the 
> problem and solution on this mailing list thread then the solution is 
> documented for other people who find themselves in the same situation.

I'd like to find out why processor.sys fails, but the .1->.2 change 
isn't any help unfortunately.  It looks like here too there are two bugs 
involved: one in kvm which doesn't act like processor.sys expects it, 
and one in processor.sys which causes it to 
UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS.  You might try 
running with kvm trace enabled and look at msr and cpuid accesses just 
prior to the crash.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> Well, there was a regression, but it was in 0.12.1.1.
> 
> There were two bugs involved, a serious one (that caused the cpuid to 
> show up as AMD) hiding the less serious one (that causes processor.sys 
> to BSOD).

Okay, understood. I must have just got lucky when upgrading from older 
KVM to 0.12.1.1.

> I'd like to find out why processor.sys fails, but the .1->.2 change 
> isn't any help unfortunately.  It looks like here too there are two bugs 
> involved: one in kvm which doesn't act like processor.sys expects it, 
> and one in processor.sys which causes it to 
> UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS.  You might try 
> running with kvm trace enabled and look at msr and cpuid accesses just 
> prior to the crash.

I've had a quick look at this, and I can't work out how to enable kvm 
trace in Kconfig. Is there any documentation on how to build with kvm 
trace enabled for debugging?


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/25/2010 06:06 PM, Mark Cave-Ayland wrote:
>
>> I'd like to find out why processor.sys fails, but the .1->.2 change 
>> isn't any help unfortunately.  It looks like here too there are two 
>> bugs involved: one in kvm which doesn't act like processor.sys 
>> expects it, and one in processor.sys which causes it to 
>> UNLOAD_ITSELF_WITHOUT_CANCELLING_PENDING_OPERATIONS.  You might try 
>> running with kvm trace enabled and look at msr and cpuid accesses 
>> just prior to the crash.
>
> I've had a quick look at this, and I can't work out how to enable kvm 
> trace in Kconfig. Is there any documentation on how to build with kvm 
> trace enabled for debugging?
>

CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think.  2.6.32 
or later IIRC.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

>> I've had a quick look at this, and I can't work out how to enable kvm 
>> trace in Kconfig. Is there any documentation on how to build with kvm 
>> trace enabled for debugging?
> 
> CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think.  2.6.32 
> or later IIRC.

Hmmm these already look as if they have been enabled. Does the output go 
to syslog or does it appear on a device somewhere in /proc?


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/25/2010 06:18 PM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>>> I've had a quick look at this, and I can't work out how to enable 
>>> kvm trace in Kconfig. Is there any documentation on how to build 
>>> with kvm trace enabled for debugging?
>>
>> CONFIG_FTRACE, CONFIG_TRACEPOINTS should be sufficient, I think.  
>> 2.6.32 or later IIRC.
>
> Hmmm these already look as if they have been enabled. Does the output 
> go to syslog or does it appear on a device somewhere in /proc?
>

echo kvm > /sys/kernel/debug/tracing/set_event
(you can also enable just the msr and cpuid events, see 
/sys/kernel/debug/tracing/events)

cat /sys/kernel/debug/trace

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> echo kvm > /sys/kernel/debug/tracing/set_event
> (you can also enable just the msr and cpuid events, see 
> /sys/kernel/debug/tracing/events)
> 
> cat /sys/kernel/debug/trace

(goes on a kernel debugging crash course)

Okay I think I've got the information you need for msr and cpuid:

http://pastebin.com/m39e26e6e

This is from a fresh start of the VM up to just after the point where it 
reaches the BSOD.


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/25/2010 06:54 PM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>> echo kvm > /sys/kernel/debug/tracing/set_event
>> (you can also enable just the msr and cpuid events, see 
>> /sys/kernel/debug/tracing/events)
>>
>> cat /sys/kernel/debug/trace
>
> (goes on a kernel debugging crash course)
>
> Okay I think I've got the information you need for msr and cpuid:
>
> http://pastebin.com/m39e26e6e
>
> This is from a fresh start of the VM up to just after the point where 
> it reaches the BSOD.
>
>

Unfortunately msr tracing fails to record some important information.  I 
just posted a patch to fix this.  Can you rerun from kvm.git branch 
msr-trace?  That contains the fix.

-- 
error compiling committee.c: too many arguments to function

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> Unfortunately msr tracing fails to record some important information.  I 
> just posted a patch to fix this.  Can you rerun from kvm.git branch 
> msr-trace?  That contains the fix.

Done.

http://pastebin.com/m209b1f13

Hopefully this trace should give a better indication of when the problem 
occurs since I also found the <on_reboot> option in libvirt - hence on 
this trace the VM shuts down immediately instead of rebooting like the 
earlier trace.


HTH,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Avi Kivity]

On 01/26/2010 12:25 AM, Mark Cave-Ayland wrote:
> Avi Kivity wrote:
>
>> Unfortunately msr tracing fails to record some important 
>> information.  I just posted a patch to fix this.  Can you rerun from 
>> kvm.git branch msr-trace?  That contains the fix.
>
> Done.
>
> http://pastebin.com/m209b1f13
>
> Hopefully this trace should give a better indication of when the 
> problem occurs since I also found the <on_reboot> option in libvirt - 
> hence on this trace the VM shuts down immediately instead of rebooting 
> like the earlier trace.
>
>

Unfortunately, no such luck.  Apparently this is not msr/cpuid related - 
perhaps power management.  Can you enable the kvm_mmio and kvm_pio 
events?  Perhaps they will provide a clue.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: WinXP virtual crashes on 0.12.1.2 but not 0.12.1.1

[Mark Cave-Ayland]

Avi Kivity wrote:

> Unfortunately, no such luck.  Apparently this is not msr/cpuid related - 
> perhaps power management.  Can you enable the kvm_mmio and kvm_pio 
> events?  Perhaps they will provide a clue.

Yeah, I did take a quick glance at the trace and figured that there 
would be some "(#GP)" strings in there if this were the cause. At least 
that probably explains why the ignore_msrs=1 parameter didn't have an 
effect.

I've just created the kvm_mmio and kvm_pio trace here for you to look 
at: http://www.siriusit.co.uk/tmp/kvm1.trace.


ATB,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs