[BUG] kvm: guest can not startup due to triple fault

[BUG] kvm: guest can not startup due to triple fault

[Dave Young]

linux guest can not startup normally, qemu keep rebooting.

host & guest both use linux-2.6.36-rc8+ kernel

Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
[ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+ #62
[ 3924.503045] Call Trace:
[ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26 [kvm_intel]
[ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474 [kvm_intel]
[ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43 [kvm]
[ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
[ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
[ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
[ 3924.503117]  [<ffffffff810bee76>] ? fire_user_return_notifiers+0x2c/0x36
[ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
[ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
[ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
[ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
[ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
[ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b

Anyone can help on this?

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Sun, Oct 17, 2010 at 4:54 PM, Dave Young <hidave.darkstar@gmail.com> wrote:
> linux guest can not startup normally, qemu keep rebooting.
>
> host & guest both use linux-2.6.36-rc8+ kernel

BTW,  qemu cmdline with -kernel will more likely trigger this bug

>
> Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+ #62
> [ 3924.503045] Call Trace:
> [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26 [kvm_intel]
> [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474 [kvm_intel]
> [ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43 [kvm]
> [ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
> [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
> [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> [ 3924.503117]  [<ffffffff810bee76>] ? fire_user_return_notifiers+0x2c/0x36
> [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
>
> Anyone can help on this?
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/17/2010 10:54 AM, Dave Young wrote:
> linux guest can not startup normally, qemu keep rebooting.
>
> host&  guest both use linux-2.6.36-rc8+ kernel
>
> Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+ #62
> [ 3924.503045] Call Trace:
> [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26 [kvm_intel]
> [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474 [kvm_intel]
> [ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43 [kvm]
> [ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
> [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
> [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> [ 3924.503117]  [<ffffffff810bee76>] ? fire_user_return_notifiers+0x2c/0x36
> [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
>
> Anyone can help on this?

Please provide a trace (trace-cmd record -e kvm) leading to the failure.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/17/2010 10:54 AM, Dave Young wrote:
>>
>> linux guest can not startup normally, qemu keep rebooting.
>>
>> host&  guest both use linux-2.6.36-rc8+ kernel
>>
>> Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
>> [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+
>> #62
>> [ 3924.503045] Call Trace:
>> [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
>> [kvm_intel]
>> [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
>> [kvm_intel]
>> [ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43
>> [kvm]
>> [ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
>> [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
>> [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
>> [ 3924.503117]  [<ffffffff810bee76>] ?
>> fire_user_return_notifiers+0x2c/0x36
>> [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
>> [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
>> [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
>> [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
>> [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
>> [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
>>
>> Anyone can help on this?
>
> Please provide a trace (trace-cmd record -e kvm) leading to the failure.
>

Hi, please see:
http://userweb.kernel.org/~davey/tmp/trace.dat

> --
> error compiling committee.c: too many arguments to function
>
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/17/2010 12:14 PM, Dave Young wrote:
> On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>  wrote:
> >    On 10/17/2010 10:54 AM, Dave Young wrote:
> >>
> >>  linux guest can not startup normally, qemu keep rebooting.
> >>
> >>  host&    guest both use linux-2.6.36-rc8+ kernel
> >>
> >>  Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> >>  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted 2.6.36-rc8+
> >>  #62
> >>  [ 3924.503045] Call Trace:
> >>  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
> >>  [kvm_intel]
> >>  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
> >>  [kvm_intel]
> >>  [ 3924.503080]  [<ffffffffa021f203>] kvm_arch_vcpu_ioctl_run+0x95f/0xc43
> >>  [kvm]
> >>  [ 3924.503087]  [<ffffffff81503e54>] ? __mutex_unlock_slowpath+0x11c/0x131
> >>  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6 [kvm]
> >>  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> >>  [ 3924.503117]  [<ffffffff810bee76>] ?
> >>  fire_user_return_notifiers+0x2c/0x36
> >>  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> >>  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> >>  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> >>  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> >>  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> >>  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
> >>
> >>  Anyone can help on this?
> >
> >  Please provide a trace (trace-cmd record -e kvm) leading to the failure.
> >
>
> Hi, please see:
> http://userweb.kernel.org/~davey/tmp/trace.dat
>

Was this done with an old version of trace-cmd?  Please use the latest 
git.  Make sure you have udis86 and udis86-devel installed when you 
build it.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/17/2010 12:14 PM, Dave Young wrote:
>>
>> On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 10/17/2010 10:54 AM, Dave Young wrote:
>> >>
>> >>  linux guest can not startup normally, qemu keep rebooting.
>> >>
>> >>  host&    guest both use linux-2.6.36-rc8+ kernel
>> >>
>> >>  Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
>> >>  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
>> >> 2.6.36-rc8+
>> >>  #62
>> >>  [ 3924.503045] Call Trace:
>> >>  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
>> >>  [kvm_intel]
>> >>  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
>> >>  [kvm_intel]
>> >>  [ 3924.503080]  [<ffffffffa021f203>]
>> >> kvm_arch_vcpu_ioctl_run+0x95f/0xc43
>> >>  [kvm]
>> >>  [ 3924.503087]  [<ffffffff81503e54>] ?
>> >> __mutex_unlock_slowpath+0x11c/0x131
>> >>  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
>> >> [kvm]
>> >>  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
>> >>  [ 3924.503117]  [<ffffffff810bee76>] ?
>> >>  fire_user_return_notifiers+0x2c/0x36
>> >>  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
>> >>  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
>> >>  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
>> >>  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
>> >>  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
>> >>  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
>> >>
>> >>  Anyone can help on this?
>> >
>> >  Please provide a trace (trace-cmd record -e kvm) leading to the
>> > failure.
>> >
>>
>> Hi, please see:
>> http://userweb.kernel.org/~davey/tmp/trace.dat
>>
>
> Was this done with an old version of trace-cmd?  Please use the latest git.
>  Make sure you have udis86 and udis86-devel installed when you build it.

Hi, the trace-cmd was build from git version, but without udis86.

I updated trace.dat on the same url, is it a correct one?

>
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2682 bytes --]

Am 18.10.2010 06:43, Dave Young wrote:
> On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity <avi@redhat.com> wrote:
>>  On 10/17/2010 12:14 PM, Dave Young wrote:
>>>
>>> On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>  wrote:
>>>>    On 10/17/2010 10:54 AM, Dave Young wrote:
>>>>>
>>>>>  linux guest can not startup normally, qemu keep rebooting.
>>>>>
>>>>>  host&    guest both use linux-2.6.36-rc8+ kernel
>>>>>
>>>>>  Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
>>>>>  [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
>>>>> 2.6.36-rc8+
>>>>>  #62
>>>>>  [ 3924.503045] Call Trace:
>>>>>  [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
>>>>>  [kvm_intel]
>>>>>  [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
>>>>>  [kvm_intel]
>>>>>  [ 3924.503080]  [<ffffffffa021f203>]
>>>>> kvm_arch_vcpu_ioctl_run+0x95f/0xc43
>>>>>  [kvm]
>>>>>  [ 3924.503087]  [<ffffffff81503e54>] ?
>>>>> __mutex_unlock_slowpath+0x11c/0x131
>>>>>  [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
>>>>> [kvm]
>>>>>  [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
>>>>>  [ 3924.503117]  [<ffffffff810bee76>] ?
>>>>>  fire_user_return_notifiers+0x2c/0x36
>>>>>  [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
>>>>>  [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
>>>>>  [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
>>>>>  [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
>>>>>  [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
>>>>>  [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
>>>>>
>>>>>  Anyone can help on this?
>>>>
>>>>  Please provide a trace (trace-cmd record -e kvm) leading to the
>>>> failure.
>>>>
>>>
>>> Hi, please see:
>>> http://userweb.kernel.org/~davey/tmp/trace.dat
>>>
>>
>> Was this done with an old version of trace-cmd?  Please use the latest git.
>>  Make sure you have udis86 and udis86-devel installed when you build it.
> 
> Hi, the trace-cmd was build from git version, but without udis86.
> 
> I updated trace.dat on the same url, is it a correct one?
> 

IIUC, plugin_kvm is only involved in reporting, not recording, so udis86
shouldn't matter.

I had a look out of curiosity (just managed to make my trace tools work
smoothly), and the data is properly parsed by kernelshark here. There is
a strange NMI storm going on right before the triple fault (on some MSR
write) hits the fan.

Jan


>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
> 
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 09:14 AM, Jan Kiszka wrote:
> Am 18.10.2010 06:43, Dave Young wrote:
> >  On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity<avi@redhat.com>  wrote:
> >>   On 10/17/2010 12:14 PM, Dave Young wrote:
> >>>
> >>>  On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>   wrote:
> >>>>     On 10/17/2010 10:54 AM, Dave Young wrote:
> >>>>>
> >>>>>   linux guest can not startup normally, qemu keep rebooting.
> >>>>>
> >>>>>   host&     guest both use linux-2.6.36-rc8+ kernel
> >>>>>
> >>>>>   Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> >>>>>   [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
> >>>>>  2.6.36-rc8+
> >>>>>   #62
> >>>>>   [ 3924.503045] Call Trace:
> >>>>>   [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
> >>>>>   [kvm_intel]
> >>>>>   [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
> >>>>>   [kvm_intel]
> >>>>>   [ 3924.503080]  [<ffffffffa021f203>]
> >>>>>  kvm_arch_vcpu_ioctl_run+0x95f/0xc43
> >>>>>   [kvm]
> >>>>>   [ 3924.503087]  [<ffffffff81503e54>] ?
> >>>>>  __mutex_unlock_slowpath+0x11c/0x131
> >>>>>   [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
> >>>>>  [kvm]
> >>>>>   [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> >>>>>   [ 3924.503117]  [<ffffffff810bee76>] ?
> >>>>>   fire_user_return_notifiers+0x2c/0x36
> >>>>>   [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> >>>>>   [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> >>>>>   [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> >>>>>   [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> >>>>>   [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> >>>>>   [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
> >>>>>
> >>>>>   Anyone can help on this?
> >>>>
> >>>>   Please provide a trace (trace-cmd record -e kvm) leading to the
> >>>>  failure.
> >>>>
> >>>
> >>>  Hi, please see:
> >>>  http://userweb.kernel.org/~davey/tmp/trace.dat
> >>>
> >>
> >>  Was this done with an old version of trace-cmd?  Please use the latest git.
> >>   Make sure you have udis86 and udis86-devel installed when you build it.
> >
> >  Hi, the trace-cmd was build from git version, but without udis86.
> >
> >  I updated trace.dat on the same url, is it a correct one?
> >
>
> IIUC, plugin_kvm is only involved in reporting, not recording, so udis86
> shouldn't matter.
>
> I had a look out of curiosity (just managed to make my trace tools work
> smoothly), and the data is properly parsed by kernelshark here. There is
> a strange NMI storm going on right before the triple fault (on some MSR
> write) hits the fan.

Doesn't parse here.  I get a bus error.

Steven, is trace.dat meant to be transportable?  What does it depend on?

Jan, can you upload your parsed data somewhere?  I want to have a look 
as well.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 10:55 AM, Avi Kivity wrote:
>  On 10/18/2010 09:14 AM, Jan Kiszka wrote:
>> Am 18.10.2010 06:43, Dave Young wrote:
>> >  On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >>   On 10/17/2010 12:14 PM, Dave Young wrote:
>> >>>
>> >>>  On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>   
>> wrote:
>> >>>>     On 10/17/2010 10:54 AM, Dave Young wrote:
>> >>>>>
>> >>>>>   linux guest can not startup normally, qemu keep rebooting.
>> >>>>>
>> >>>>>   host&     guest both use linux-2.6.36-rc8+ kernel
>> >>>>>
>> >>>>>   Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
>> >>>>>   [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
>> >>>>>  2.6.36-rc8+
>> >>>>>   #62
>> >>>>>   [ 3924.503045] Call Trace:
>> >>>>>   [ 3924.503055]  [<ffffffffa0176047>] 
>> handle_triple_fault+0x11/0x26
>> >>>>>   [kvm_intel]
>> >>>>>   [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
>> >>>>>   [kvm_intel]
>> >>>>>   [ 3924.503080]  [<ffffffffa021f203>]
>> >>>>>  kvm_arch_vcpu_ioctl_run+0x95f/0xc43
>> >>>>>   [kvm]
>> >>>>>   [ 3924.503087]  [<ffffffff81503e54>] ?
>> >>>>>  __mutex_unlock_slowpath+0x11c/0x131
>> >>>>>   [ 3924.503101]  [<ffffffffa021e1f9>] ? 
>> kvm_arch_vcpu_load+0x90/0xe6
>> >>>>>  [kvm]
>> >>>>>   [ 3924.503112]  [<ffffffffa0210faf>] 
>> kvm_vcpu_ioctl+0x109/0x54a [kvm]
>> >>>>>   [ 3924.503117]  [<ffffffff810bee76>] ?
>> >>>>>   fire_user_return_notifiers+0x2c/0x36
>> >>>>>   [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
>> >>>>>   [ 3924.503127]  [<ffffffff81002789>] ? 
>> do_notify_resume+0x717/0x73d
>> >>>>>   [ 3924.503131]  [<ffffffff8103994e>] ? 
>> finish_task_switch+0x69/0xda
>> >>>>>   [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
>> >>>>>   [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
>> >>>>>   [ 3924.503142]  [<ffffffff81002c02>] 
>> system_call_fastpath+0x16/0x1b
>> >>>>>
>> >>>>>   Anyone can help on this?
>> >>>>
>> >>>>   Please provide a trace (trace-cmd record -e kvm) leading to the
>> >>>>  failure.
>> >>>>
>> >>>
>> >>>  Hi, please see:
>> >>>  http://userweb.kernel.org/~davey/tmp/trace.dat
>> >>>
>> >>
>> >>  Was this done with an old version of trace-cmd?  Please use the 
>> latest git.
>> >>   Make sure you have udis86 and udis86-devel installed when you 
>> build it.
>> >
>> >  Hi, the trace-cmd was build from git version, but without udis86.
>> >
>> >  I updated trace.dat on the same url, is it a correct one?
>> >
>>
>> IIUC, plugin_kvm is only involved in reporting, not recording, so udis86
>> shouldn't matter.
>>
>> I had a look out of curiosity (just managed to make my trace tools work
>> smoothly), and the data is properly parsed by kernelshark here. There is
>> a strange NMI storm going on right before the triple fault (on some MSR
>> write) hits the fan.
>
> Doesn't parse here.  I get a bus error.
>
> Steven, is trace.dat meant to be transportable?  What does it depend on?
>
> Jan, can you upload your parsed data somewhere?  I want to have a look 
> as well.
>

Never mind, user error here.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 09:14 AM, Jan Kiszka wrote:
> Am 18.10.2010 06:43, Dave Young wrote:
> >  On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity<avi@redhat.com>  wrote:
> >>   On 10/17/2010 12:14 PM, Dave Young wrote:
> >>>
> >>>  On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>   wrote:
> >>>>     On 10/17/2010 10:54 AM, Dave Young wrote:
> >>>>>
> >>>>>   linux guest can not startup normally, qemu keep rebooting.
> >>>>>
> >>>>>   host&     guest both use linux-2.6.36-rc8+ kernel
> >>>>>
> >>>>>   Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> >>>>>   [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
> >>>>>  2.6.36-rc8+
> >>>>>   #62
> >>>>>   [ 3924.503045] Call Trace:
> >>>>>   [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
> >>>>>   [kvm_intel]
> >>>>>   [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
> >>>>>   [kvm_intel]
> >>>>>   [ 3924.503080]  [<ffffffffa021f203>]
> >>>>>  kvm_arch_vcpu_ioctl_run+0x95f/0xc43
> >>>>>   [kvm]
> >>>>>   [ 3924.503087]  [<ffffffff81503e54>] ?
> >>>>>  __mutex_unlock_slowpath+0x11c/0x131
> >>>>>   [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
> >>>>>  [kvm]
> >>>>>   [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> >>>>>   [ 3924.503117]  [<ffffffff810bee76>] ?
> >>>>>   fire_user_return_notifiers+0x2c/0x36
> >>>>>   [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> >>>>>   [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> >>>>>   [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> >>>>>   [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> >>>>>   [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> >>>>>   [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
> >>>>>
> >>>>>   Anyone can help on this?
> >>>>
> >>>>   Please provide a trace (trace-cmd record -e kvm) leading to the
> >>>>  failure.
> >>>>
> >>>
> >>>  Hi, please see:
> >>>  http://userweb.kernel.org/~davey/tmp/trace.dat
> >>>
> >>
> >>  Was this done with an old version of trace-cmd?  Please use the latest git.
> >>   Make sure you have udis86 and udis86-devel installed when you build it.
> >
> >  Hi, the trace-cmd was build from git version, but without udis86.
> >
> >  I updated trace.dat on the same url, is it a correct one?
> >
>
> IIUC, plugin_kvm is only involved in reporting, not recording, so udis86
> shouldn't matter.
>
> I had a look out of curiosity (just managed to make my trace tools work
> smoothly), and the data is properly parsed by kernelshark here. There is
> a strange NMI storm going on right before the triple fault (on some MSR
> write) hits the fan.
>

It's secondary_startup_64 enabling efer.sce and efer.nx.

Dave, please post your /proc/cpuinfo.  Is your host running with NX 
disabled?

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Steven Rostedt]

On Mon, 2010-10-18 at 10:58 +0200, Avi Kivity wrote:

> >
> > Doesn't parse here.  I get a bus error.
> >
> > Steven, is trace.dat meant to be transportable?  What does it depend on?
> >
> > Jan, can you upload your parsed data somewhere?  I want to have a look 
> > as well.
> >
> 
> Never mind, user error here.
> 

Everything all set then?

-- Steve

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 02:26 PM, Steven Rostedt wrote:
> On Mon, 2010-10-18 at 10:58 +0200, Avi Kivity wrote:
>
> >  >
> >  >  Doesn't parse here.  I get a bus error.
> >  >
> >  >  Steven, is trace.dat meant to be transportable?  What does it depend on?
> >  >
> >  >  Jan, can you upload your parsed data somewhere?  I want to have a look
> >  >  as well.
> >  >
> >
> >  Never mind, user error here.
> >
>
> Everything all set then?

Yes, sorry for the noise.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 11:15:51AM +0200, Avi Kivity wrote:
>  On 10/18/2010 09:14 AM, Jan Kiszka wrote:
> >Am 18.10.2010 06:43, Dave Young wrote:
> >>  On Sun, Oct 17, 2010 at 8:15 PM, Avi Kivity<avi@redhat.com>  wrote:
> >>>   On 10/17/2010 12:14 PM, Dave Young wrote:
> >>>>
> >>>>  On Sun, Oct 17, 2010 at 5:34 PM, Avi Kivity<avi@redhat.com>   wrote:
> >>>>>     On 10/17/2010 10:54 AM, Dave Young wrote:
> >>>>>>
> >>>>>>   linux guest can not startup normally, qemu keep rebooting.
> >>>>>>
> >>>>>>   host&     guest both use linux-2.6.36-rc8+ kernel
> >>>>>>
> >>>>>>   Add a dump_stack in x86.c around EXIT_SHUTDOWN code, I got:
> >>>>>>   [ 3924.503041] Pid: 1685, comm: qemu-system-x86 Not tainted
> >>>>>>  2.6.36-rc8+
> >>>>>>   #62
> >>>>>>   [ 3924.503045] Call Trace:
> >>>>>>   [ 3924.503055]  [<ffffffffa0176047>] handle_triple_fault+0x11/0x26
> >>>>>>   [kvm_intel]
> >>>>>>   [ 3924.503060]  [<ffffffffa01772d2>] vmx_handle_exit+0x3df/0x474
> >>>>>>   [kvm_intel]
> >>>>>>   [ 3924.503080]  [<ffffffffa021f203>]
> >>>>>>  kvm_arch_vcpu_ioctl_run+0x95f/0xc43
> >>>>>>   [kvm]
> >>>>>>   [ 3924.503087]  [<ffffffff81503e54>] ?
> >>>>>>  __mutex_unlock_slowpath+0x11c/0x131
> >>>>>>   [ 3924.503101]  [<ffffffffa021e1f9>] ? kvm_arch_vcpu_load+0x90/0xe6
> >>>>>>  [kvm]
> >>>>>>   [ 3924.503112]  [<ffffffffa0210faf>] kvm_vcpu_ioctl+0x109/0x54a [kvm]
> >>>>>>   [ 3924.503117]  [<ffffffff810bee76>] ?
> >>>>>>   fire_user_return_notifiers+0x2c/0x36
> >>>>>>   [ 3924.503122]  [<ffffffff8106be93>] ? trace_hardirqs_on+0xd/0xf
> >>>>>>   [ 3924.503127]  [<ffffffff81002789>] ? do_notify_resume+0x717/0x73d
> >>>>>>   [ 3924.503131]  [<ffffffff8103994e>] ? finish_task_switch+0x69/0xda
> >>>>>>   [ 3924.503135]  [<ffffffff8110628c>] do_vfs_ioctl+0x4c3/0x50d
> >>>>>>   [ 3924.503139]  [<ffffffff8110632b>] sys_ioctl+0x55/0x77
> >>>>>>   [ 3924.503142]  [<ffffffff81002c02>] system_call_fastpath+0x16/0x1b
> >>>>>>
> >>>>>>   Anyone can help on this?
> >>>>>
> >>>>>   Please provide a trace (trace-cmd record -e kvm) leading to the
> >>>>>  failure.
> >>>>>
> >>>>
> >>>>  Hi, please see:
> >>>>  http://userweb.kernel.org/~davey/tmp/trace.dat
> >>>>
> >>>
> >>>  Was this done with an old version of trace-cmd?  Please use the latest git.
> >>>   Make sure you have udis86 and udis86-devel installed when you build it.
> >>
> >>  Hi, the trace-cmd was build from git version, but without udis86.
> >>
> >>  I updated trace.dat on the same url, is it a correct one?
> >>
> >
> >IIUC, plugin_kvm is only involved in reporting, not recording, so udis86
> >shouldn't matter.
> >
> >I had a look out of curiosity (just managed to make my trace tools work
> >smoothly), and the data is properly parsed by kernelshark here. There is
> >a strange NMI storm going on right before the triple fault (on some MSR
> >write) hits the fan.
> >
> 
> It's secondary_startup_64 enabling efer.sce and efer.nx.
> 
> Dave, please post your /proc/cpuinfo.  Is your host running with NX
> disabled?

I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7250  @ 2.00GHz
stepping	: 13
cpu MHz		: 800.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips	: 3991.42
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7250  @ 2.00GHz
stepping	: 13
cpu MHz		: 2001.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
bogomips	: 3991.36
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 02:58 PM, Dave Young wrote:
> >
> >  It's secondary_startup_64 enabling efer.sce and efer.nx.
> >
> >  Dave, please post your /proc/cpuinfo.  Is your host running with NX
> >  disabled?
>
> I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.

Strange.

Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a global), 
efer (an argument) and old_efer (a local), and post the results.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 03:26:30PM +0200, Avi Kivity wrote:
>  On 10/18/2010 02:58 PM, Dave Young wrote:
> >>
> >>  It's secondary_startup_64 enabling efer.sce and efer.nx.
> >>
> >>  Dave, please post your /proc/cpuinfo.  Is your host running with NX
> >>  disabled?
> >
> >I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.
> 
> Strange.
> 
> Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a
> global), efer (an argument) and old_efer (a local), and post the
> results.

I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:

[ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe
[ 3228.468328] efer = 0x100
[ 3228.468330] old_efer = 0x0

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 03:45 PM, Dave Young wrote:
> On Mon, Oct 18, 2010 at 03:26:30PM +0200, Avi Kivity wrote:
> >   On 10/18/2010 02:58 PM, Dave Young wrote:
> >  >>
> >  >>   It's secondary_startup_64 enabling efer.sce and efer.nx.
> >  >>
> >  >>   Dave, please post your /proc/cpuinfo.  Is your host running with NX
> >  >>   disabled?
> >  >
> >  >I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.
> >
> >  Strange.
> >
> >  Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a
> >  global), efer (an argument) and old_efer (a local), and post the
> >  results.
>
> I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:
>

Add it at the beginning please, otherwise we'll miss the important one 
due to an early return.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 03:46 PM, Avi Kivity wrote:
>  On 10/18/2010 03:45 PM, Dave Young wrote:
>> On Mon, Oct 18, 2010 at 03:26:30PM +0200, Avi Kivity wrote:
>> >   On 10/18/2010 02:58 PM, Dave Young wrote:
>> > >>
>> > >>   It's secondary_startup_64 enabling efer.sce and efer.nx.
>> > >>
>> > >>   Dave, please post your /proc/cpuinfo.  Is your host running 
>> with NX
>> > >>   disabled?
>> > >
>> > >I see nx bit in /proc/cpuinfo, Actually I don't know how to 
>> disable it.
>> >
>> >  Strange.
>> >
>> >  Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a
>> >  global), efer (an argument) and old_efer (a local), and post the
>> >  results.
>>
>> I add printk before kvm_mmu_reset_context(vcpu), correct? get 
>> following value:
>>
>
> Add it at the beginning please, otherwise we'll miss the important one 
> due to an early return.
>

No need actually:

> [ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe


Bit 11 is reserved, so the guest can't enable nx.

 From arch/x86/kvm/vmx.c:
> static __init int hardware_setup(void)
> {
>     if (setup_vmcs_config(&vmcs_config) < 0)
>         return -EIO;
>
>     if (boot_cpu_has(X86_FEATURE_NX))
>         kvm_enable_efer_bits(EFER_NX);
>

Please patch hardware_setup() to show the value of 
boot_cpu_has(X86_FEATURE_NX).

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 03:46:17PM +0200, Avi Kivity wrote:
>  On 10/18/2010 03:45 PM, Dave Young wrote:
> >On Mon, Oct 18, 2010 at 03:26:30PM +0200, Avi Kivity wrote:
> >>   On 10/18/2010 02:58 PM, Dave Young wrote:
> >>  >>
> >>  >>   It's secondary_startup_64 enabling efer.sce and efer.nx.
> >>  >>
> >>  >>   Dave, please post your /proc/cpuinfo.  Is your host running with NX
> >>  >>   disabled?
> >>  >
> >>  >I see nx bit in /proc/cpuinfo, Actually I don't know how to disable it.
> >>
> >>  Strange.
> >>
> >>  Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a
> >>  global), efer (an argument) and old_efer (a local), and post the
> >>  results.
> >
> >I add printk before kvm_mmu_reset_context(vcpu), correct? get following value:
> >
> 
> Add it at the beginning please, otherwise we'll miss the important
> one due to an early return.

here it is:

[ 3679.616500] efer_reserved_bits = 0xfffffffffffffafe
[ 3679.616504] efer = 0xd01
[ 3679.616506] old_efer = 0x500

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 03:50:15PM +0200, Avi Kivity wrote:
>  On 10/18/2010 03:46 PM, Avi Kivity wrote:
> > On 10/18/2010 03:45 PM, Dave Young wrote:
> >>On Mon, Oct 18, 2010 at 03:26:30PM +0200, Avi Kivity wrote:
> >>>   On 10/18/2010 02:58 PM, Dave Young wrote:
> >>> >>
> >>> >>   It's secondary_startup_64 enabling efer.sce and efer.nx.
> >>> >>
> >>> >>   Dave, please post your /proc/cpuinfo.  Is your host
> >>running with NX
> >>> >>   disabled?
> >>> >
> >>> >I see nx bit in /proc/cpuinfo, Actually I don't know how to
> >>disable it.
> >>>
> >>>  Strange.
> >>>
> >>>  Please patch arch/x86/kvm/x86.c to printk efer_reserved_bits (a
> >>>  global), efer (an argument) and old_efer (a local), and post the
> >>>  results.
> >>
> >>I add printk before kvm_mmu_reset_context(vcpu), correct? get
> >>following value:
> >>
> >
> >Add it at the beginning please, otherwise we'll miss the important
> >one due to an early return.
> >
> 
> No need actually:
> 
> >[ 3228.468325] efer_reserved_bits = 0xfffffffffffffafe
> 
> 
> Bit 11 is reserved, so the guest can't enable nx.
> 
> From arch/x86/kvm/vmx.c:
> >static __init int hardware_setup(void)
> >{
> >    if (setup_vmcs_config(&vmcs_config) < 0)
> >        return -EIO;
> >
> >    if (boot_cpu_has(X86_FEATURE_NX))
> >        kvm_enable_efer_bits(EFER_NX);
> >
> 
> Please patch hardware_setup() to show the value of
> boot_cpu_has(X86_FEATURE_NX).

It's 0

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 03:58 PM, Dave Young wrote:
> >
> >  Please patch hardware_setup() to show the value of
> >  boot_cpu_has(X86_FEATURE_NX).
>
> It's 0

Well, that's ridiculous.  Looking back at your /proc/cpuinfo:

> processor	: 0
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>
> processor	: 1
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority

processor 0 doesn't have nx.

Ingo, Peter, any ideas?  Where did the missing feature go?

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/18/2010 03:58 PM, Dave Young wrote:
>>
>> >
>> >  Please patch hardware_setup() to show the value of
>> >  boot_cpu_has(X86_FEATURE_NX).
>>
>> It's 0

Please wait, I reply too quick, actually it boots two times fine then
one time fail.

The printk only happen once. So to make sure the result I retested
several times, can not get the print as well.

>
> Well, that's ridiculous.  Looking back at your /proc/cpuinfo:
>
>> processor       : 0
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm
>> constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor
>> ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi
>> flexpriority
>>
>> processor       : 1
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm
>> constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor
>> ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi
>> flexpriority
>
> processor 0 doesn't have nx.
>
> Ingo, Peter, any ideas?  Where did the missing feature go?
>
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 04:11 PM, Dave Young wrote:
> On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<avi@redhat.com>  wrote:
> >    On 10/18/2010 03:58 PM, Dave Young wrote:
> >>
> >>  >
> >>  >    Please patch hardware_setup() to show the value of
> >>  >    boot_cpu_has(X86_FEATURE_NX).
> >>
> >>  It's 0
>
> Please wait, I reply too quick, actually it boots two times fine then
> one time fail.

What boots fine?  The guest?

> The printk only happen once. So to make sure the result I retested
> several times, can not get the print as well.

boot_cpu_has() is called only when the kvm module is inserted.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 10:13 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/18/2010 04:11 PM, Dave Young wrote:
>>
>> On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 10/18/2010 03:58 PM, Dave Young wrote:
>> >>
>> >>  >
>> >>  >    Please patch hardware_setup() to show the value of
>> >>  >    boot_cpu_has(X86_FEATURE_NX).
>> >>
>> >>  It's 0
>>
>> Please wait, I reply too quick, actually it boots two times fine then
>> one time fail.
>
> What boots fine?  The guest?

Yes the guest

>
>> The printk only happen once. So to make sure the result I retested
>> several times, can not get the print as well.
>
> boot_cpu_has() is called only when the kvm module is inserted.

Oh, then I need to test again.

>
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 10:15 PM, Dave Young <hidave.darkstar@gmail.com> wrote:
> On Mon, Oct 18, 2010 at 10:13 PM, Avi Kivity <avi@redhat.com> wrote:
>>  On 10/18/2010 04:11 PM, Dave Young wrote:
>>>
>>> On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<avi@redhat.com>  wrote:
>>> >    On 10/18/2010 03:58 PM, Dave Young wrote:
>>> >>
>>> >>  >
>>> >>  >    Please patch hardware_setup() to show the value of
>>> >>  >    boot_cpu_has(X86_FEATURE_NX).
>>> >>
>>> >>  It's 0
>>>
>>> Please wait, I reply too quick, actually it boots two times fine then
>>> one time fail.
>>
>> What boots fine?  The guest?
>
> Yes the guest
>
>>
>>> The printk only happen once. So to make sure the result I retested
>>> several times, can not get the print as well.
>>
>> boot_cpu_has() is called only when the kvm module is inserted.
>
> Oh, then I need to test again.

I'm lucky, this time I can make sure has_nx = 0

after insmod, guest boot failed at the first test
>
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
>
>
>
> --
> Regards
> dave
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 04:15 PM, Dave Young wrote:
> On Mon, Oct 18, 2010 at 10:13 PM, Avi Kivity<avi@redhat.com>  wrote:
> >    On 10/18/2010 04:11 PM, Dave Young wrote:
> >>
> >>  On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<avi@redhat.com>    wrote:
> >>  >      On 10/18/2010 03:58 PM, Dave Young wrote:
> >>  >>
> >>  >>    >
> >>  >>    >      Please patch hardware_setup() to show the value of
> >>  >>    >      boot_cpu_has(X86_FEATURE_NX).
> >>  >>
> >>  >>    It's 0
> >>
> >>  Please wait, I reply too quick, actually it boots two times fine then
> >>  one time fail.
> >
> >  What boots fine?  The guest?
>
> Yes the guest
>

That makes some sort of wierd sense.  First kvm samples X86_FEATURE_NX, 
sees that it has the feature, reports it to qemu, qemu runs the guest, 
NX gets disabled in between, the guest tries to enable NX, kvm kills it.

Second time round, kvm reports that NX is not available, qemu does not 
enable it, neither does the guest, and everyone is happy.

Please confirm that when the guest does boot, it boots without NX in 
/proc/cpuinfo.

-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Mon, Oct 18, 2010 at 10:19 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/18/2010 04:15 PM, Dave Young wrote:
>>
>> On Mon, Oct 18, 2010 at 10:13 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 10/18/2010 04:11 PM, Dave Young wrote:
>> >>
>> >>  On Mon, Oct 18, 2010 at 10:06 PM, Avi Kivity<avi@redhat.com>    wrote:
>> >>  >      On 10/18/2010 03:58 PM, Dave Young wrote:
>> >>  >>
>> >>  >>    >
>> >>  >>    >      Please patch hardware_setup() to show the value of
>> >>  >>    >      boot_cpu_has(X86_FEATURE_NX).
>> >>  >>
>> >>  >>    It's 0
>> >>
>> >>  Please wait, I reply too quick, actually it boots two times fine then
>> >>  one time fail.
>> >
>> >  What boots fine?  The guest?
>>
>> Yes the guest
>>
>
> That makes some sort of wierd sense.  First kvm samples X86_FEATURE_NX, sees
> that it has the feature, reports it to qemu, qemu runs the guest, NX gets
> disabled in between, the guest tries to enable NX, kvm kills it.
>
> Second time round, kvm reports that NX is not available, qemu does not
> enable it, neither does the guest, and everyone is happy.
>
> Please confirm that when the guest does boot, it boots without NX in
> /proc/cpuinfo.
>

Yes, there's no nx in guest cpuinfo

hardware_setup is run only when insmod so the nx is of host,  thus the
first printk result is right value?
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/18/2010 04:25 PM, Dave Young wrote:
> >
> >  That makes some sort of wierd sense.  First kvm samples X86_FEATURE_NX, sees
> >  that it has the feature, reports it to qemu, qemu runs the guest, NX gets
> >  disabled in between, the guest tries to enable NX, kvm kills it.
> >
> >  Second time round, kvm reports that NX is not available, qemu does not
> >  enable it, neither does the guest, and everyone is happy.
> >
> >  Please confirm that when the guest does boot, it boots without NX in
> >  /proc/cpuinfo.
> >
>
> Yes, there's no nx in guest cpuinfo
>
> hardware_setup is run only when insmod so the nx is of host,  thus the
> first printk result is right value?

There's no right value since the two cpus have different values for nx.


-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[H. Peter Anvin]

On 10/18/2010 07:06 AM, Avi Kivity wrote:
>   On 10/18/2010 03:58 PM, Dave Young wrote:
>>>
>>>  Please patch hardware_setup() to show the value of
>>>  boot_cpu_has(X86_FEATURE_NX).
>>
>> It's 0
> 
> Well, that's ridiculous.  Looking back at your /proc/cpuinfo:
> 
>> processor	: 0
>> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>
>> processor	: 1
>> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
> 
> processor 0 doesn't have nx.
> 
> Ingo, Peter, any ideas?  Where did the missing feature go?
> 

I don't see the original report -- in particular I can't see the kernel
version -- but I suspect this *might* be an instance of the same bug
fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055
(for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also
needed; these both should be in -stable or on the way there.)

	-hpa

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Tue, Oct 19, 2010 at 2:47 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 10/18/2010 07:06 AM, Avi Kivity wrote:
>>   On 10/18/2010 03:58 PM, Dave Young wrote:
>>>>
>>>>  Please patch hardware_setup() to show the value of
>>>>  boot_cpu_has(X86_FEATURE_NX).
>>>
>>> It's 0
>>
>> Well, that's ridiculous.  Looking back at your /proc/cpuinfo:
>>
>>> processor    : 0
>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>>
>>> processor    : 1
>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>
>> processor 0 doesn't have nx.
>>
>> Ingo, Peter, any ideas?  Where did the missing feature go?
>>
>
> I don't see the original report -- in particular I can't see the kernel
> version -- but I suspect this *might* be an instance of the same bug
> fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055
> (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also
> needed; these both should be in -stable or on the way there.)

The kernel is git version after 2.6.36-rc8, so I think the commits
don't fix this problem.
-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[H. Peter Anvin]

On 10/18/2010 06:19 PM, Dave Young wrote:
> On Tue, Oct 19, 2010 at 2:47 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>> On 10/18/2010 07:06 AM, Avi Kivity wrote:
>>>   On 10/18/2010 03:58 PM, Dave Young wrote:
>>>>>
>>>>>  Please patch hardware_setup() to show the value of
>>>>>  boot_cpu_has(X86_FEATURE_NX).
>>>>
>>>> It's 0
>>>
>>> Well, that's ridiculous.  Looking back at your /proc/cpuinfo:
>>>
>>>> processor    : 0
>>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>>>
>>>> processor    : 1
>>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>>
>>> processor 0 doesn't have nx.
>>>
>>> Ingo, Peter, any ideas?  Where did the missing feature go?
>>>
>>
>> I don't see the original report -- in particular I can't see the kernel
>> version -- but I suspect this *might* be an instance of the same bug
>> fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055
>> (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also
>> needed; these both should be in -stable or on the way there.)
> 
> The kernel is git version after 2.6.36-rc8, so I think the commits
> don't fix this problem.

OK.  Please install msr-tools on your system, and give us the outputs of:

rdmsr -p 0 -xc 0x1a0
rdmsr -p 1 -xc 0x1a0

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Tue, Oct 19, 2010 at 10:59 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 10/18/2010 06:19 PM, Dave Young wrote:
>> On Tue, Oct 19, 2010 at 2:47 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>> On 10/18/2010 07:06 AM, Avi Kivity wrote:
>>>>   On 10/18/2010 03:58 PM, Dave Young wrote:
>>>>>>
>>>>>>  Please patch hardware_setup() to show the value of
>>>>>>  boot_cpu_has(X86_FEATURE_NX).
>>>>>
>>>>> It's 0
>>>>
>>>> Well, that's ridiculous.  Looking back at your /proc/cpuinfo:
>>>>
>>>>> processor    : 0
>>>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>>>>
>>>>> processor    : 1
>>>>> flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dts tpr_shadow vnmi flexpriority
>>>>
>>>> processor 0 doesn't have nx.
>>>>
>>>> Ingo, Peter, any ideas?  Where did the missing feature go?
>>>>
>>>
>>> I don't see the original report -- in particular I can't see the kernel
>>> version -- but I suspect this *might* be an instance of the same bug
>>> fixed in mainline by checkin d900329e20f4476db6461752accebcf7935a8055
>>> (for backporting 1dedefd1a066a795a87afca9c0236e1a94de9bf6 is also
>>> needed; these both should be in -stable or on the way there.)
>>
>> The kernel is git version after 2.6.36-rc8, so I think the commits
>> don't fix this problem.
>
> OK.  Please install msr-tools on your system, and give us the outputs of:
>
> rdmsr -p 0 -xc 0x1a0
0x1764952489

> rdmsr -p 1 -xc 0x1a0
0x1364952489

-- 
Regards
dave

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/19/2010 08:55 AM, Dave Young wrote:
> >  OK.  Please install msr-tools on your system, and give us the outputs of:
> >
> >  rdmsr -p 0 -xc 0x1a0
> 0x1764952489
>
> >  rdmsr -p 1 -xc 0x1a0
> 0x1364952489
>

That's MSR_IA32_MISC_ENABLE_XD_DISABLE.  Looks like the kernel never 
plays with it.

Suggest:
- updating your BIOS
- playing with the NX feature in your BIOS setup screens


-- 
error compiling committee.c: too many arguments to function

Re: [BUG] kvm: guest can not startup due to triple fault

[Dave Young]

On Tue, Oct 19, 2010 at 10:57:51AM +0200, Avi Kivity wrote:
>  On 10/19/2010 08:55 AM, Dave Young wrote:
> >>  OK.  Please install msr-tools on your system, and give us the outputs of:
> >>
> >>  rdmsr -p 0 -xc 0x1a0
> >0x1764952489
> >
> >>  rdmsr -p 1 -xc 0x1a0
> >0x1364952489
> >
> 
> That's MSR_IA32_MISC_ENABLE_XD_DISABLE.  Looks like the kernel never
> plays with it.

Hope someone can fix this. IMO we should not depends on bios setting

> 
> Suggest:
> - updating your BIOS
> - playing with the NX feature in your BIOS setup screens

Enable the XD in bios resolved the problem. Thanks you all for the help.

> 
> 
> -- 
> error compiling committee.c: too many arguments to function
> 

Re: [BUG] kvm: guest can not startup due to triple fault

[H. Peter Anvin]

Right... note in particular that the BIOS sets them up inconsistently.  The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.

"Avi Kivity" <avi@redhat.com> wrote:

>  On 10/19/2010 08:55 AM, Dave Young wrote:
>> >  OK.  Please install msr-tools on your system, and give us the outputs of:
>> >
>> >  rdmsr -p 0 -xc 0x1a0
>> 0x1764952489
>>
>> >  rdmsr -p 1 -xc 0x1a0
>> 0x1364952489
>>
>
>That's MSR_IA32_MISC_ENABLE_XD_DISABLE.  Looks like the kernel never 
>plays with it.
>
>Suggest:
>- updating your BIOS
>- playing with the NX feature in your BIOS setup screens
>
>
>-- 
>error compiling committee.c: too many arguments to function
>

-- 
Sent from my mobile phone.  Please pardon any lack of formatting.

Re: [BUG] kvm: guest can not startup due to triple fault

[Avi Kivity]

  On 10/19/2010 04:56 PM, H. Peter Anvin wrote:
> Right... note in particular that the BIOS sets them up inconsistently.  The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.

I guess kvm should be modified to mask GET_SUPPORTED_CPUID results with 
the boot_cpu features.  This will make kvm take advantage of the core 
kernel doing the right thing.

Re: [BUG] kvm: guest can not startup due to triple fault

[H. Peter Anvin]

That would probably be a good idea.  The kernel also sets and clears bits for various errata or other special issues.

"Avi Kivity" <avi@redhat.com> wrote:

>  On 10/19/2010 04:56 PM, H. Peter Anvin wrote:
>> Right... note in particular that the BIOS sets them up inconsistently.  The kernel is doing the rufht thing: since NX isn't available on all processors it is considered unavailable.
>
>I guess kvm should be modified to mask GET_SUPPORTED_CPUID results with 
>the boot_cpu features.  This will make kvm take advantage of the core 
>kernel doing the right thing.
>

-- 
Sent from my mobile phone.  Please pardon any lack of formatting.