* E5-2620v2 - emulation stop error
@ 2015-03-05 22:14 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-05 22:14 UTC (permalink / raw)
To: qemu-devel; +Cc: kvm
Hello,
recently I`ve got a couple of shiny new Intel 2620v2s for future
replacement of the E5-2620v1, but I experienced relatively many events
with emulation errors, all traces looks simular to the one below. I am
running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
can switch to some other versions if necessary. Most of crashes
happened during reboot cycle or at the end of ACPI-based shutdown
action, if this can help. I have zero clues of what can introduce such
a mess inside same processor family using identical software, as
2620v1 has no simular problem ever. Please let me know if there can be
some side measures for making entire story more clear.
Thanks!
KVM internal error. Suberror: 2
extra data[0]: 800000d1
extra data[1]: 80000b0d
EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6e98 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
b8 00 e0 00 00 8e
^ permalink raw reply [flat|nested] 157+ messages in thread
* [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-05 22:14 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-05 22:14 UTC (permalink / raw)
To: qemu-devel; +Cc: kvm
Hello,
recently I`ve got a couple of shiny new Intel 2620v2s for future
replacement of the E5-2620v1, but I experienced relatively many events
with emulation errors, all traces looks simular to the one below. I am
running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
can switch to some other versions if necessary. Most of crashes
happened during reboot cycle or at the end of ACPI-based shutdown
action, if this can help. I have zero clues of what can introduce such
a mess inside same processor family using identical software, as
2620v1 has no simular problem ever. Please let me know if there can be
some side measures for making entire story more clear.
Thanks!
KVM internal error. Suberror: 2
extra data[0]: 800000d1
extra data[1]: 80000b0d
EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6e98 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
b8 00 e0 00 00 8e
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-05 22:14 ` [Qemu-devel] " Andrey Korolyov
@ 2015-03-05 23:44 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-05 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: kvm
On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> Hello,
>
> recently I`ve got a couple of shiny new Intel 2620v2s for future
> replacement of the E5-2620v1, but I experienced relatively many events
> with emulation errors, all traces looks simular to the one below. I am
> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
> can switch to some other versions if necessary. Most of crashes
> happened during reboot cycle or at the end of ACPI-based shutdown
> action, if this can help. I have zero clues of what can introduce such
> a mess inside same processor family using identical software, as
> 2620v1 has no simular problem ever. Please let me know if there can be
> some side measures for making entire story more clear.
>
> Thanks!
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000d1
> extra data[1]: 80000b0d
> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6e98 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> b8 00 e0 00 00 8e
It turns out that those errors are introduced by APICv, which gets
enabled due to different feature set. If anyone is interested in
reproducing/fixing this exactly on 3.10, it takes about one hundred of
migrations/power state changes for an issue to appear, guest OS can be
Linux or Win.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-05 23:44 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-05 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: kvm
On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> Hello,
>
> recently I`ve got a couple of shiny new Intel 2620v2s for future
> replacement of the E5-2620v1, but I experienced relatively many events
> with emulation errors, all traces looks simular to the one below. I am
> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
> can switch to some other versions if necessary. Most of crashes
> happened during reboot cycle or at the end of ACPI-based shutdown
> action, if this can help. I have zero clues of what can introduce such
> a mess inside same processor family using identical software, as
> 2620v1 has no simular problem ever. Please let me know if there can be
> some side measures for making entire story more clear.
>
> Thanks!
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000d1
> extra data[1]: 80000b0d
> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6e98 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> b8 00 e0 00 00 8e
It turns out that those errors are introduced by APICv, which gets
enabled due to different feature set. If anyone is interested in
reproducing/fixing this exactly on 3.10, it takes about one hundred of
migrations/power state changes for an issue to appear, guest OS can be
Linux or Win.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-05 23:44 ` [Qemu-devel] " Andrey Korolyov
@ 2015-03-06 16:57 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-06 16:57 UTC (permalink / raw)
To: Andrey Korolyov; +Cc: qemu-devel, kvm
Andrey Korolyov <andrey@xdel.ru> writes:
> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> Hello,
>>
>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>> replacement of the E5-2620v1, but I experienced relatively many events
>> with emulation errors, all traces looks simular to the one below. I am
>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>> can switch to some other versions if necessary. Most of crashes
>> happened during reboot cycle or at the end of ACPI-based shutdown
>> action, if this can help. I have zero clues of what can introduce such
>> a mess inside same processor family using identical software, as
>> 2620v1 has no simular problem ever. Please let me know if there can be
>> some side measures for making entire story more clear.
>>
>> Thanks!
>>
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000d1
>> extra data[1]: 80000b0d
>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT= 000f6e98 00000037
>> IDT= 00000000 000003ff
>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> b8 00 e0 00 00 8e
>
>
> It turns out that those errors are introduced by APICv, which gets
> enabled due to different feature set. If anyone is interested in
> reproducing/fixing this exactly on 3.10, it takes about one hundred of
> migrations/power state changes for an issue to appear, guest OS can be
> Linux or Win.
Are you able to reproduce this on a more recent upstream kernel as well ?
Bandan
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-06 16:57 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-06 16:57 UTC (permalink / raw)
To: Andrey Korolyov; +Cc: qemu-devel, kvm
Andrey Korolyov <andrey@xdel.ru> writes:
> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> Hello,
>>
>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>> replacement of the E5-2620v1, but I experienced relatively many events
>> with emulation errors, all traces looks simular to the one below. I am
>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>> can switch to some other versions if necessary. Most of crashes
>> happened during reboot cycle or at the end of ACPI-based shutdown
>> action, if this can help. I have zero clues of what can introduce such
>> a mess inside same processor family using identical software, as
>> 2620v1 has no simular problem ever. Please let me know if there can be
>> some side measures for making entire story more clear.
>>
>> Thanks!
>>
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000d1
>> extra data[1]: 80000b0d
>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT= 000f6e98 00000037
>> IDT= 00000000 000003ff
>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> b8 00 e0 00 00 8e
>
>
> It turns out that those errors are introduced by APICv, which gets
> enabled due to different feature set. If anyone is interested in
> reproducing/fixing this exactly on 3.10, it takes about one hundred of
> migrations/power state changes for an issue to appear, guest OS can be
> Linux or Win.
Are you able to reproduce this on a more recent upstream kernel as well ?
Bandan
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-06 16:57 ` Bandan Das
(?)
@ 2015-03-07 0:00 ` Andrey Korolyov
2015-03-10 14:24 ` Andrey Korolyov
-1 siblings, 1 reply; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-07 0:00 UTC (permalink / raw)
To: Bandan Das; +Cc: qemu-devel, kvm
On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
> Andrey Korolyov <andrey@xdel.ru> writes:
>
>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>> Hello,
>>>
>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>>> replacement of the E5-2620v1, but I experienced relatively many events
>>> with emulation errors, all traces looks simular to the one below. I am
>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>>> can switch to some other versions if necessary. Most of crashes
>>> happened during reboot cycle or at the end of ACPI-based shutdown
>>> action, if this can help. I have zero clues of what can introduce such
>>> a mess inside same processor family using identical software, as
>>> 2620v1 has no simular problem ever. Please let me know if there can be
>>> some side measures for making entire story more clear.
>>>
>>> Thanks!
>>>
>>> KVM internal error. Suberror: 2
>>> extra data[0]: 800000d1
>>> extra data[1]: 80000b0d
>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>> ES =0000 00000000 0000ffff 00009300
>>> CS =f000 000f0000 0000ffff 00009b00
>>> SS =0000 00000000 0000ffff 00009300
>>> DS =0000 00000000 0000ffff 00009300
>>> FS =0000 00000000 0000ffff 00009300
>>> GS =0000 00000000 0000ffff 00009300
>>> LDT=0000 00000000 0000ffff 00008200
>>> TR =0000 00000000 0000ffff 00008b00
>>> GDT= 000f6e98 00000037
>>> IDT= 00000000 000003ff
>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>> DR3=0000000000000000
>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>> EFER=0000000000000000
>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>>> b8 00 e0 00 00 8e
>>
>>
>> It turns out that those errors are introduced by APICv, which gets
>> enabled due to different feature set. If anyone is interested in
>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
>> migrations/power state changes for an issue to appear, guest OS can be
>> Linux or Win.
>
> Are you able to reproduce this on a more recent upstream kernel as well ?
>
> Bandan
I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
follow up with any reproduceable results.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-07 0:00 ` Andrey Korolyov
@ 2015-03-10 14:24 ` Andrey Korolyov
2015-03-10 16:57 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-10 14:24 UTC (permalink / raw)
To: Bandan Das; +Cc: qemu-devel, kvm
On Sat, Mar 7, 2015 at 3:00 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
>> Andrey Korolyov <andrey@xdel.ru> writes:
>>
>>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>>> Hello,
>>>>
>>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>>>> replacement of the E5-2620v1, but I experienced relatively many events
>>>> with emulation errors, all traces looks simular to the one below. I am
>>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>>>> can switch to some other versions if necessary. Most of crashes
>>>> happened during reboot cycle or at the end of ACPI-based shutdown
>>>> action, if this can help. I have zero clues of what can introduce such
>>>> a mess inside same processor family using identical software, as
>>>> 2620v1 has no simular problem ever. Please let me know if there can be
>>>> some side measures for making entire story more clear.
>>>>
>>>> Thanks!
>>>>
>>>> KVM internal error. Suberror: 2
>>>> extra data[0]: 800000d1
>>>> extra data[1]: 80000b0d
>>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>> ES =0000 00000000 0000ffff 00009300
>>>> CS =f000 000f0000 0000ffff 00009b00
>>>> SS =0000 00000000 0000ffff 00009300
>>>> DS =0000 00000000 0000ffff 00009300
>>>> FS =0000 00000000 0000ffff 00009300
>>>> GS =0000 00000000 0000ffff 00009300
>>>> LDT=0000 00000000 0000ffff 00008200
>>>> TR =0000 00000000 0000ffff 00008b00
>>>> GDT= 000f6e98 00000037
>>>> IDT= 00000000 000003ff
>>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>> DR3=0000000000000000
>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>> EFER=0000000000000000
>>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>>>> b8 00 e0 00 00 8e
>>>
>>>
>>> It turns out that those errors are introduced by APICv, which gets
>>> enabled due to different feature set. If anyone is interested in
>>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
>>> migrations/power state changes for an issue to appear, guest OS can be
>>> Linux or Win.
>>
>> Are you able to reproduce this on a more recent upstream kernel as well ?
>>
>> Bandan
>
> I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
> follow up with any reproduceable results.
Heh.. issue is not triggered on 2603v2 at all, at least I am not able
to hit this. The only difference with 2620v2 except lower frequency is
an Intel Dynamic Acceleration feature. I`d appreciate any testing with
higher CPU models with same or richer feature set. The testing itself
can be done on both generic 3.10 or RH7 kernels, as both of them are
experiencing this issue. I conducted all tests with disabled cstates
so I advise to do the same for a first reproduction step.
Thanks!
model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
stepping : 4
microcode : 0x416
cpu MHz : 2100.039
cache size : 15360 KB
siblings : 12
apicid : 43
initial apicid : 43
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi
flexpriority ept vpid fsgsbase smep erms
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 14:24 ` Andrey Korolyov
@ 2015-03-10 16:57 ` Dr. David Alan Gilbert
2015-03-10 18:08 ` Andrey Korolyov
2015-03-10 18:10 ` Paolo Bonzini
0 siblings, 2 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-10 16:57 UTC (permalink / raw)
To: Andrey Korolyov; +Cc: Bandan Das, qemu-devel, kvm
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Sat, Mar 7, 2015 at 3:00 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
> >> Andrey Korolyov <andrey@xdel.ru> writes:
> >>
> >>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> >>>> Hello,
> >>>>
> >>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
> >>>> replacement of the E5-2620v1, but I experienced relatively many events
> >>>> with emulation errors, all traces looks simular to the one below. I am
> >>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
> >>>> can switch to some other versions if necessary. Most of crashes
> >>>> happened during reboot cycle or at the end of ACPI-based shutdown
> >>>> action, if this can help. I have zero clues of what can introduce such
> >>>> a mess inside same processor family using identical software, as
> >>>> 2620v1 has no simular problem ever. Please let me know if there can be
> >>>> some side measures for making entire story more clear.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> KVM internal error. Suberror: 2
> >>>> extra data[0]: 800000d1
> >>>> extra data[1]: 80000b0d
> >>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
> >>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
> >>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>>> ES =0000 00000000 0000ffff 00009300
> >>>> CS =f000 000f0000 0000ffff 00009b00
> >>>> SS =0000 00000000 0000ffff 00009300
> >>>> DS =0000 00000000 0000ffff 00009300
> >>>> FS =0000 00000000 0000ffff 00009300
> >>>> GS =0000 00000000 0000ffff 00009300
> >>>> LDT=0000 00000000 0000ffff 00008200
> >>>> TR =0000 00000000 0000ffff 00008b00
> >>>> GDT= 000f6e98 00000037
> >>>> IDT= 00000000 000003ff
> >>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> >>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >>>> DR3=0000000000000000
> >>>> DR6=00000000ffff0ff0 DR7=0000000000000400
> >>>> EFER=0000000000000000
> >>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> >>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> >>>> b8 00 e0 00 00 8e
> >>>
> >>>
> >>> It turns out that those errors are introduced by APICv, which gets
> >>> enabled due to different feature set. If anyone is interested in
> >>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
> >>> migrations/power state changes for an issue to appear, guest OS can be
> >>> Linux or Win.
> >>
> >> Are you able to reproduce this on a more recent upstream kernel as well ?
> >>
> >> Bandan
> >
> > I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
> > follow up with any reproduceable results.
>
> Heh.. issue is not triggered on 2603v2 at all, at least I am not able
> to hit this. The only difference with 2620v2 except lower frequency is
> an Intel Dynamic Acceleration feature. I`d appreciate any testing with
> higher CPU models with same or richer feature set. The testing itself
> can be done on both generic 3.10 or RH7 kernels, as both of them are
> experiencing this issue. I conducted all tests with disabled cstates
> so I advise to do the same for a first reproduction step.
>
> Thanks!
>
> model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
> stepping : 4
> microcode : 0x416
> cpu MHz : 2100.039
> cache size : 15360 KB
> siblings : 12
> apicid : 43
> initial apicid : 43
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
> sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
> rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi
> flexpriority ept vpid fsgsbase smep erms
I'm seeing something similar; it's very intermittent and generally
happening right at boot of the guest; I'm running this on qemu
head+my postcopy world (but it's happening right at boot before postcopy
gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
but hey maybe I'm seeing a different bug.
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 16:57 ` Dr. David Alan Gilbert
@ 2015-03-10 18:08 ` Andrey Korolyov
2015-03-10 18:16 ` Dr. David Alan Gilbert
2015-03-10 18:10 ` Paolo Bonzini
1 sibling, 1 reply; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-10 18:08 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: Bandan Das, qemu-devel, kvm
On Tue, Mar 10, 2015 at 7:57 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Sat, Mar 7, 2015 at 3:00 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
>> >> Andrey Korolyov <andrey@xdel.ru> writes:
>> >>
>> >>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> >>>> Hello,
>> >>>>
>> >>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>> >>>> replacement of the E5-2620v1, but I experienced relatively many events
>> >>>> with emulation errors, all traces looks simular to the one below. I am
>> >>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>> >>>> can switch to some other versions if necessary. Most of crashes
>> >>>> happened during reboot cycle or at the end of ACPI-based shutdown
>> >>>> action, if this can help. I have zero clues of what can introduce such
>> >>>> a mess inside same processor family using identical software, as
>> >>>> 2620v1 has no simular problem ever. Please let me know if there can be
>> >>>> some side measures for making entire story more clear.
>> >>>>
>> >>>> Thanks!
>> >>>>
>> >>>> KVM internal error. Suberror: 2
>> >>>> extra data[0]: 800000d1
>> >>>> extra data[1]: 80000b0d
>> >>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>> >>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>> >>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> >>>> ES =0000 00000000 0000ffff 00009300
>> >>>> CS =f000 000f0000 0000ffff 00009b00
>> >>>> SS =0000 00000000 0000ffff 00009300
>> >>>> DS =0000 00000000 0000ffff 00009300
>> >>>> FS =0000 00000000 0000ffff 00009300
>> >>>> GS =0000 00000000 0000ffff 00009300
>> >>>> LDT=0000 00000000 0000ffff 00008200
>> >>>> TR =0000 00000000 0000ffff 00008b00
>> >>>> GDT= 000f6e98 00000037
>> >>>> IDT= 00000000 000003ff
>> >>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> >>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> >>>> DR3=0000000000000000
>> >>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> >>>> EFER=0000000000000000
>> >>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> >>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> >>>> b8 00 e0 00 00 8e
>> >>>
>> >>>
>> >>> It turns out that those errors are introduced by APICv, which gets
>> >>> enabled due to different feature set. If anyone is interested in
>> >>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
>> >>> migrations/power state changes for an issue to appear, guest OS can be
>> >>> Linux or Win.
>> >>
>> >> Are you able to reproduce this on a more recent upstream kernel as well ?
>> >>
>> >> Bandan
>> >
>> > I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
>> > follow up with any reproduceable results.
>>
>> Heh.. issue is not triggered on 2603v2 at all, at least I am not able
>> to hit this. The only difference with 2620v2 except lower frequency is
>> an Intel Dynamic Acceleration feature. I`d appreciate any testing with
>> higher CPU models with same or richer feature set. The testing itself
>> can be done on both generic 3.10 or RH7 kernels, as both of them are
>> experiencing this issue. I conducted all tests with disabled cstates
>> so I advise to do the same for a first reproduction step.
>>
>> Thanks!
>>
>> model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
>> stepping : 4
>> microcode : 0x416
>> cpu MHz : 2100.039
>> cache size : 15360 KB
>> siblings : 12
>> apicid : 43
>> initial apicid : 43
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 13
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
>> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
>> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
>> sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
>> rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi
>> flexpriority ept vpid fsgsbase smep erms
>
> I'm seeing something similar; it's very intermittent and generally
> happening right at boot of the guest; I'm running this on qemu
> head+my postcopy world (but it's happening right at boot before postcopy
> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> but hey maybe I'm seeing a different bug.
>
> Dave
Yep, looks like we are hitting same bug - two thirds of mine failure
events shot during boot/reboot cycle and approx. one third of events
happened in the middle of runtime. What CPU, v0 or v2 are you using
(in other words, is APICv enabled)?
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 16:57 ` Dr. David Alan Gilbert
2015-03-10 18:08 ` Andrey Korolyov
@ 2015-03-10 18:10 ` Paolo Bonzini
2015-03-10 18:21 ` Bandan Das
1 sibling, 1 reply; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-10 18:10 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Andrey Korolyov; +Cc: Bandan Das, qemu-devel, kvm
On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> I'm seeing something similar; it's very intermittent and generally
> happening right at boot of the guest; I'm running this on qemu
> head+my postcopy world (but it's happening right at boot before postcopy
> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> but hey maybe I'm seeing a different bug.
Same here on 3.16 + Xeon E5 v3 kernel.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 18:08 ` Andrey Korolyov
@ 2015-03-10 18:16 ` Dr. David Alan Gilbert
2015-03-10 18:21 ` Andrey Korolyov
2015-03-10 19:30 ` Paolo Bonzini
0 siblings, 2 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-10 18:16 UTC (permalink / raw)
To: Andrey Korolyov; +Cc: Bandan Das, qemu-devel, kvm
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Tue, Mar 10, 2015 at 7:57 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Andrey Korolyov (andrey@xdel.ru) wrote:
> >> On Sat, Mar 7, 2015 at 3:00 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> >> > On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
> >> >> Andrey Korolyov <andrey@xdel.ru> writes:
> >> >>
> >> >>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> >> >>>> Hello,
> >> >>>>
> >> >>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
> >> >>>> replacement of the E5-2620v1, but I experienced relatively many events
> >> >>>> with emulation errors, all traces looks simular to the one below. I am
> >> >>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
> >> >>>> can switch to some other versions if necessary. Most of crashes
> >> >>>> happened during reboot cycle or at the end of ACPI-based shutdown
> >> >>>> action, if this can help. I have zero clues of what can introduce such
> >> >>>> a mess inside same processor family using identical software, as
> >> >>>> 2620v1 has no simular problem ever. Please let me know if there can be
> >> >>>> some side measures for making entire story more clear.
> >> >>>>
> >> >>>> Thanks!
> >> >>>>
> >> >>>> KVM internal error. Suberror: 2
> >> >>>> extra data[0]: 800000d1
> >> >>>> extra data[1]: 80000b0d
> >> >>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
> >> >>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
> >> >>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >> >>>> ES =0000 00000000 0000ffff 00009300
> >> >>>> CS =f000 000f0000 0000ffff 00009b00
> >> >>>> SS =0000 00000000 0000ffff 00009300
> >> >>>> DS =0000 00000000 0000ffff 00009300
> >> >>>> FS =0000 00000000 0000ffff 00009300
> >> >>>> GS =0000 00000000 0000ffff 00009300
> >> >>>> LDT=0000 00000000 0000ffff 00008200
> >> >>>> TR =0000 00000000 0000ffff 00008b00
> >> >>>> GDT= 000f6e98 00000037
> >> >>>> IDT= 00000000 000003ff
> >> >>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> >> >>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >> >>>> DR3=0000000000000000
> >> >>>> DR6=00000000ffff0ff0 DR7=0000000000000400
> >> >>>> EFER=0000000000000000
> >> >>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> >> >>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> >> >>>> b8 00 e0 00 00 8e
> >> >>>
> >> >>>
> >> >>> It turns out that those errors are introduced by APICv, which gets
> >> >>> enabled due to different feature set. If anyone is interested in
> >> >>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
> >> >>> migrations/power state changes for an issue to appear, guest OS can be
> >> >>> Linux or Win.
> >> >>
> >> >> Are you able to reproduce this on a more recent upstream kernel as well ?
> >> >>
> >> >> Bandan
> >> >
> >> > I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
> >> > follow up with any reproduceable results.
> >>
> >> Heh.. issue is not triggered on 2603v2 at all, at least I am not able
> >> to hit this. The only difference with 2620v2 except lower frequency is
> >> an Intel Dynamic Acceleration feature. I`d appreciate any testing with
> >> higher CPU models with same or richer feature set. The testing itself
> >> can be done on both generic 3.10 or RH7 kernels, as both of them are
> >> experiencing this issue. I conducted all tests with disabled cstates
> >> so I advise to do the same for a first reproduction step.
> >>
> >> Thanks!
> >>
> >> model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
> >> stepping : 4
> >> microcode : 0x416
> >> cpu MHz : 2100.039
> >> cache size : 15360 KB
> >> siblings : 12
> >> apicid : 43
> >> initial apicid : 43
> >> fpu : yes
> >> fpu_exception : yes
> >> cpuid level : 13
> >> wp : yes
> >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> >> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> >> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> >> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
> >> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
> >> sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
> >> rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi
> >> flexpriority ept vpid fsgsbase smep erms
> >
> > I'm seeing something similar; it's very intermittent and generally
> > happening right at boot of the guest; I'm running this on qemu
> > head+my postcopy world (but it's happening right at boot before postcopy
> > gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> > but hey maybe I'm seeing a different bug.
> >
> > Dave
>
> Yep, looks like we are hitting same bug - two thirds of mine failure
> events shot during boot/reboot cycle and approx. one third of events
> happened in the middle of runtime. What CPU, v0 or v2 are you using
> (in other words, is APICv enabled)?
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2200.000
cache size : 10240 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 38
initial apicid : 38
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt
bugs :
bogomips : 4409.23
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
It's really random as well; I had two within half an hour yesterday, and then
it survived overnight with no change.
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2bc
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=000fd2c5 EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT= 000f6a80 00000037
IDT= 000f6abe 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 ba bc d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb bf 00 <72> f3 8b 25 00 ff fb bf e8 44 66 ff ff c7 05 04 ff
fb bf 00 00 00 00 f4 eb fd fa fc 66 b8
KVM internal error. Suberror: 1
emulation failure
and
11:37:49 INFO | [qemu output] KVM internal error. Suberror: 1
11:37:49 INFO | [qemu output] emulation failure
11:37:49 INFO | [qemu output] EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2bc
11:37:49 INFO | [qemu output] ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
11:37:49 INFO | [qemu output] EIP=000fd2bc EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
11:37:49 INFO | [qemu output] ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
11:37:49 INFO | [qemu output] CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
11:37:49 INFO | [qemu output] SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
11:37:49 INFO | [qemu output] DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
11:37:49 INFO | [qemu output] FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
11:37:49 INFO | [qemu output] GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
11:37:49 INFO | [qemu output] LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
11:37:49 INFO | [qemu output] TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
11:37:49 INFO | [qemu output] GDT= 000f6a80 00000037
11:37:49 INFO | [qemu output] IDT= 000f6abe 00000000
11:37:49 INFO | [qemu output] CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
11:37:49 INFO | [qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
11:37:49 INFO | [qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400
11:37:49 INFO | [qemu output] EFER=0000000000000000
11:37:49 INFO | [qemu output] Code=0a 00 e8 a0 64 ff ff 0f aa 66 ba bc d2 0f 00 e9 a2 fe f3 90 <f0> 0f ba 2d 04 ff fb 3f 00 72 f3 8b 25 00 ff fb 3f e8 44 66 ff ff c7 05 04 ff fb 3f 00 00
note the code in that second one is in the middle of the bios,
but the code has a few bytes different from what an objdump gets,
so I'm not quite sure if something is stamping on the bios or
if that's separate.
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 18:16 ` Dr. David Alan Gilbert
@ 2015-03-10 18:21 ` Andrey Korolyov
2015-03-10 19:30 ` Paolo Bonzini
1 sibling, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-10 18:21 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: Bandan Das, qemu-devel, kvm
On Tue, Mar 10, 2015 at 9:16 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Tue, Mar 10, 2015 at 7:57 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Andrey Korolyov (andrey@xdel.ru) wrote:
>> >> On Sat, Mar 7, 2015 at 3:00 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> >> > On Fri, Mar 6, 2015 at 7:57 PM, Bandan Das <bsd@redhat.com> wrote:
>> >> >> Andrey Korolyov <andrey@xdel.ru> writes:
>> >> >>
>> >> >>> On Fri, Mar 6, 2015 at 1:14 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> >> >>>> Hello,
>> >> >>>>
>> >> >>>> recently I`ve got a couple of shiny new Intel 2620v2s for future
>> >> >>>> replacement of the E5-2620v1, but I experienced relatively many events
>> >> >>>> with emulation errors, all traces looks simular to the one below. I am
>> >> >>>> running qemu-2.1 on x86 on top of 3.10 branch for testing purposes but
>> >> >>>> can switch to some other versions if necessary. Most of crashes
>> >> >>>> happened during reboot cycle or at the end of ACPI-based shutdown
>> >> >>>> action, if this can help. I have zero clues of what can introduce such
>> >> >>>> a mess inside same processor family using identical software, as
>> >> >>>> 2620v1 has no simular problem ever. Please let me know if there can be
>> >> >>>> some side measures for making entire story more clear.
>> >> >>>>
>> >> >>>> Thanks!
>> >> >>>>
>> >> >>>> KVM internal error. Suberror: 2
>> >> >>>> extra data[0]: 800000d1
>> >> >>>> extra data[1]: 80000b0d
>> >> >>>> EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
>> >> >>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
>> >> >>>> EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> >> >>>> ES =0000 00000000 0000ffff 00009300
>> >> >>>> CS =f000 000f0000 0000ffff 00009b00
>> >> >>>> SS =0000 00000000 0000ffff 00009300
>> >> >>>> DS =0000 00000000 0000ffff 00009300
>> >> >>>> FS =0000 00000000 0000ffff 00009300
>> >> >>>> GS =0000 00000000 0000ffff 00009300
>> >> >>>> LDT=0000 00000000 0000ffff 00008200
>> >> >>>> TR =0000 00000000 0000ffff 00008b00
>> >> >>>> GDT= 000f6e98 00000037
>> >> >>>> IDT= 00000000 000003ff
>> >> >>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> >> >>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> >> >>>> DR3=0000000000000000
>> >> >>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> >> >>>> EFER=0000000000000000
>> >> >>>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> >> >>>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> >> >>>> b8 00 e0 00 00 8e
>> >> >>>
>> >> >>>
>> >> >>> It turns out that those errors are introduced by APICv, which gets
>> >> >>> enabled due to different feature set. If anyone is interested in
>> >> >>> reproducing/fixing this exactly on 3.10, it takes about one hundred of
>> >> >>> migrations/power state changes for an issue to appear, guest OS can be
>> >> >>> Linux or Win.
>> >> >>
>> >> >> Are you able to reproduce this on a more recent upstream kernel as well ?
>> >> >>
>> >> >> Bandan
>> >> >
>> >> > I`ll go through test cycle with 3.18 and 2603v2 around tomorrow and
>> >> > follow up with any reproduceable results.
>> >>
>> >> Heh.. issue is not triggered on 2603v2 at all, at least I am not able
>> >> to hit this. The only difference with 2620v2 except lower frequency is
>> >> an Intel Dynamic Acceleration feature. I`d appreciate any testing with
>> >> higher CPU models with same or richer feature set. The testing itself
>> >> can be done on both generic 3.10 or RH7 kernels, as both of them are
>> >> experiencing this issue. I conducted all tests with disabled cstates
>> >> so I advise to do the same for a first reproduction step.
>> >>
>> >> Thanks!
>> >>
>> >> model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
>> >> stepping : 4
>> >> microcode : 0x416
>> >> cpu MHz : 2100.039
>> >> cache size : 15360 KB
>> >> siblings : 12
>> >> apicid : 43
>> >> initial apicid : 43
>> >> fpu : yes
>> >> fpu_exception : yes
>> >> cpuid level : 13
>> >> wp : yes
>> >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> >> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> >> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
>> >> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
>> >> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
>> >> sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
>> >> rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi
>> >> flexpriority ept vpid fsgsbase smep erms
>> >
>> > I'm seeing something similar; it's very intermittent and generally
>> > happening right at boot of the guest; I'm running this on qemu
>> > head+my postcopy world (but it's happening right at boot before postcopy
>> > gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> > but hey maybe I'm seeing a different bug.
>> >
>> > Dave
>>
>> Yep, looks like we are hitting same bug - two thirds of mine failure
>> events shot during boot/reboot cycle and approx. one third of events
>> happened in the middle of runtime. What CPU, v0 or v2 are you using
>> (in other words, is APICv enabled)?
>
> processor : 7
> vendor_id : GenuineIntel
> cpu family : 6
> model : 45
> model name : Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz
> stepping : 7
> microcode : 0x70d
> cpu MHz : 2200.000
> cache size : 10240 KB
> physical id : 1
> siblings : 4
> core id : 3
> cpu cores : 4
> apicid : 38
> initial apicid : 38
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt
> bugs :
> bogomips : 4409.23
> clflush size : 64
> cache_alignment : 64
> address sizes : 46 bits physical, 48 bits virtual
> power management:
>
> It's really random as well; I had two within half an hour yesterday, and then
> it survived overnight with no change.
>
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2bc
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=000fd2c5 EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> GDT= 000f6a80 00000037
> IDT= 000f6abe 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 ba bc d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb bf 00 <72> f3 8b 25 00 ff fb bf e8 44 66 ff ff c7 05 04 ff
> fb bf 00 00 00 00 f4 eb fd fa fc 66 b8
> KVM internal error. Suberror: 1
> emulation failure
>
> and
>
> 11:37:49 INFO | [qemu output] KVM internal error. Suberror: 1
> 11:37:49 INFO | [qemu output] emulation failure
> 11:37:49 INFO | [qemu output] EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2bc
> 11:37:49 INFO | [qemu output] ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> 11:37:49 INFO | [qemu output] EIP=000fd2bc EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
> 11:37:49 INFO | [qemu output] ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> 11:37:49 INFO | [qemu output] CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> 11:37:49 INFO | [qemu output] SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> 11:37:49 INFO | [qemu output] DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> 11:37:49 INFO | [qemu output] FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> 11:37:49 INFO | [qemu output] GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> 11:37:49 INFO | [qemu output] LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> 11:37:49 INFO | [qemu output] TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> 11:37:49 INFO | [qemu output] GDT= 000f6a80 00000037
> 11:37:49 INFO | [qemu output] IDT= 000f6abe 00000000
> 11:37:49 INFO | [qemu output] CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> 11:37:49 INFO | [qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> 11:37:49 INFO | [qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400
> 11:37:49 INFO | [qemu output] EFER=0000000000000000
> 11:37:49 INFO | [qemu output] Code=0a 00 e8 a0 64 ff ff 0f aa 66 ba bc d2 0f 00 e9 a2 fe f3 90 <f0> 0f ba 2d 04 ff fb 3f 00 72 f3 8b 25 00 ff fb 3f e8 44 66 ff ff c7 05 04 ff fb 3f 00 00
>
> note the code in that second one is in the middle of the bios,
> but the code has a few bytes different from what an objdump gets,
> so I'm not quite sure if something is stamping on the bios or
> if that's separate.
>
> Dave
Thanks, AFAIU suberror 1 and suberror 2 are completely different by a
nature so this is a different bug. What is interesting that you`ve got
same reproduction pattern as in mine case, it may point to a single
userspace issue triggering two independent KVM bugs...
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 18:10 ` Paolo Bonzini
@ 2015-03-10 18:21 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-10 18:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Dr. David Alan Gilbert, Andrey Korolyov, qemu-devel, kvm
Paolo Bonzini <pbonzini@redhat.com> writes:
> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>> I'm seeing something similar; it's very intermittent and generally
>> happening right at boot of the guest; I'm running this on qemu
>> head+my postcopy world (but it's happening right at boot before postcopy
>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> but hey maybe I'm seeing a different bug.
Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
>From a cursory look and my limited understanding, it seems his failure is #GP
when executing video bios.
> Same here on 3.16 + Xeon E5 v3 kernel.
I will try to reproduce this on a v2.
Bandan
> Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-10 18:21 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-10 18:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Andrey Korolyov, Dr. David Alan Gilbert, kvm, qemu-devel
Paolo Bonzini <pbonzini@redhat.com> writes:
> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>> I'm seeing something similar; it's very intermittent and generally
>> happening right at boot of the guest; I'm running this on qemu
>> head+my postcopy world (but it's happening right at boot before postcopy
>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> but hey maybe I'm seeing a different bug.
Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
>From a cursory look and my limited understanding, it seems his failure is #GP
when executing video bios.
> Same here on 3.16 + Xeon E5 v3 kernel.
I will try to reproduce this on a v2.
Bandan
> Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 18:21 ` Bandan Das
@ 2015-03-10 19:25 ` Paolo Bonzini
-1 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-10 19:25 UTC (permalink / raw)
To: Bandan Das; +Cc: Dr. David Alan Gilbert, Andrey Korolyov, qemu-devel, kvm
On 10/03/2015 19:21, Bandan Das wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
>
>> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>>> I'm seeing something similar; it's very intermittent and generally
>>> happening right at boot of the guest; I'm running this on qemu
>>> head+my postcopy world (but it's happening right at boot before postcopy
>>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>>> but hey maybe I'm seeing a different bug.
>
> Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> From a cursory look and my limited understanding, it seems his failure is #GP
> when executing video bios.
>
>> Same here on 3.16 + Xeon E5 v3 kernel.
>
> I will try to reproduce this on a v2.
I see several failures, usually mine have suberror 1. With a 32-VCPU
guest I can reproduce it roughly half of the time.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-10 19:25 ` Paolo Bonzini
0 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-10 19:25 UTC (permalink / raw)
To: Bandan Das; +Cc: Andrey Korolyov, Dr. David Alan Gilbert, kvm, qemu-devel
On 10/03/2015 19:21, Bandan Das wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
>
>> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>>> I'm seeing something similar; it's very intermittent and generally
>>> happening right at boot of the guest; I'm running this on qemu
>>> head+my postcopy world (but it's happening right at boot before postcopy
>>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>>> but hey maybe I'm seeing a different bug.
>
> Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> From a cursory look and my limited understanding, it seems his failure is #GP
> when executing video bios.
>
>> Same here on 3.16 + Xeon E5 v3 kernel.
>
> I will try to reproduce this on a v2.
I see several failures, usually mine have suberror 1. With a 32-VCPU
guest I can reproduce it roughly half of the time.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 18:16 ` Dr. David Alan Gilbert
2015-03-10 18:21 ` Andrey Korolyov
@ 2015-03-10 19:30 ` Paolo Bonzini
1 sibling, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-10 19:30 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Andrey Korolyov; +Cc: Bandan Das, qemu-devel, kvm
On 10/03/2015 19:16, Dr. David Alan Gilbert wrote:
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2bc
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=000fd2c5 EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> GDT= 000f6a80 00000037
> IDT= 000f6abe 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 ba bc d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb bf 00 <72> f3 8b 25 00 ff fb bf e8 44 66 ff ff c7 05 04 ff
> fb bf 00 00 00 00 f4 eb fd fa fc 66 b8
> KVM internal error. Suberror: 1
> emulation failure
This is exactly the same as mine. I'm using upstream QEMU right now,
but I can try to reproduce using RHEL7's too.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 19:25 ` Paolo Bonzini
(?)
@ 2015-03-10 19:37 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-10 19:37 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Bandan Das, Andrey Korolyov, qemu-devel, kvm
* Paolo Bonzini (pbonzini@redhat.com) wrote:
>
>
> On 10/03/2015 19:21, Bandan Das wrote:
> > Paolo Bonzini <pbonzini@redhat.com> writes:
> >
> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> >>> I'm seeing something similar; it's very intermittent and generally
> >>> happening right at boot of the guest; I'm running this on qemu
> >>> head+my postcopy world (but it's happening right at boot before postcopy
> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> >>> but hey maybe I'm seeing a different bug.
> >
> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> > From a cursory look and my limited understanding, it seems his failure is #GP
> > when executing video bios.
> >
> >> Same here on 3.16 + Xeon E5 v3 kernel.
> >
> > I will try to reproduce this on a v2.
>
> I see several failures, usually mine have suberror 1. With a 32-VCPU
> guest I can reproduce it roughly half of the time.
Oh yes, that helps:
while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.3,accel=kvm -m 8192 -smp 20 -nographic -device sga; done
gets it under a minute on my other box; Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
(Running a 3.18)
Dave
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 19:25 ` Paolo Bonzini
@ 2015-03-10 20:29 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-10 20:29 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Bandan Das, kraxel, Andrey Korolyov, qemu-devel, kvm
* Paolo Bonzini (pbonzini@redhat.com) wrote:
>
>
> On 10/03/2015 19:21, Bandan Das wrote:
> > Paolo Bonzini <pbonzini@redhat.com> writes:
> >
> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> >>> I'm seeing something similar; it's very intermittent and generally
> >>> happening right at boot of the guest; I'm running this on qemu
> >>> head+my postcopy world (but it's happening right at boot before postcopy
> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> >>> but hey maybe I'm seeing a different bug.
> >
> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> > From a cursory look and my limited understanding, it seems his failure is #GP
> > when executing video bios.
> >
> >> Same here on 3.16 + Xeon E5 v3 kernel.
> >
> > I will try to reproduce this on a v2.
>
> I see several failures, usually mine have suberror 1. With a 32-VCPU
> guest I can reproduce it roughly half of the time.
>
> Paolo
while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
(and leave about 2mins of runs before declaring good)
bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
bad: 21f5826a04d38e19488f917e1eef22751490c769
good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
good: c3edd62851098e6417786193ed9e9341781fcf57
good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
good: 73104fd399c6778112f64fe0d439319f24508d9a
good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
good: 4478aa768ccefcc5b234c23d035435fd71b932f6
good: 2.2.0
[root@virtlab413 qemu-world3]# git bisect bad
21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
commit 21f5826a04d38e19488f917e1eef22751490c769
Author: Gerd Hoffmann <kraxel@redhat.com>
Date: Thu Feb 19 09:33:03 2015 +0100
seabios: update to 1.8.0 release
'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
David Woodhouse (4):
Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
build: use -m16 where available instead of asm(".code16gcc")
romlayout: Use .code16 not .code16gcc
vgabios: Use .code16 not .code16gcc
Gerd Hoffmann (2):
add scripts/tarball.sh
build: set LC_ALL=C
Hannes Reinecke (1):
megasas: read addional PCI I/O bar
Ian Campbell (1):
romlayout: Use "rep ; nop" not "rep nop".
Kevin O'Connor (139):
vgabios: Return from handle_1011() if handler found.
edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
edd: Use sectors==-1 to detect removable media.
edd: Separate out ATA and virtio specific parts of fill_edd().
cdemu: store internal cdemu fields in standard "el-torito" spec format.
Move cdemu call interface and disk_ret helper code to disk.c.
smm: Replace SMI assembler code with C code.
smm: Use a C struct to define the layout of the SMM area.
smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
Don't enable thread preemption during S3 resume vga option rom execution.
Remove old Bochs bios fixed address string at 0xfff00.
Move most of the VAR16FIXED() defs to misc.c.
build: Avoid absolute paths during "whole-program" compiling.
Make sure handle_smi() and handle_smp() are compiled out if not enabled.
Remove the TODO file.
Abstract reset call (and possible 16bit mode switch) into reset() function.
build: Remove unused function getSectionsStart() from layoutrom.py.
build: Extract section visiting logic in layoutrom.py.
build: Refactor layoutrom.py gc() function.
build: Use customized entry point for each type of build.
build: Refactor findInit() function.
build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
build: Keep segmented sections separate until final link step.
build: Use fileid instead of category to write sections in layoutrom.py.
build: Only export needed fields in LayoutInfo in layoutrom.py.
build: Get fixed address variables from 32bit compile pass (not 16bit)
build: Minor - fix comments referring to old tools/ directory.
xhci: Update the times for usb command timeouts.
ehci: Update usb command timeouts to use usb_xfer_time()
uhci: Update usb command timeouts to use usb_xfer_time()
ohci: Update usb command timeouts to use usb_xfer_time()
vgabios: Fix broken build resulting from e5749978.
boot: Change ":rom%d" boot order rom instance to ":rom%x"
Minor - remove stray tab from src/fw/smm.c.
build: Update kconfig to version in Linux 3.16.
usb: Fix usb_xfer_time() to work when called in 16bit mode.
xhci: Call usb_desc2pipe() on xhci_update_pipe().
xhci: Remove 16bit code wrappers.
xhci: Use high memory instead of low memory for internal storage.
xhci: Move root hub and setup code to top of file.
xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
usb-hub: Enable power to all ports prior to calling usb_enumerate().
xhci: Change xhci_hub_detect() to use connect status instead of link state.
uhci: Repeatedly poll for device detect for 100ms.
ohci: Repeatedly poll for device detect for 100ms.
ehci: Stall uhci/ohci init only until default port routing is done.
usb: Perform device detect polling on all usb controllers.
ehci: Fix bug in hub port assignment
Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
pmm: Fix entry point to support non-zero %ss
Move stack hop code below call32/call16 code in stacks.c
Add need_hop_back() call that determines if stack_hop_back is needed
Update invoke_mouse_handler() to use need_hop_back()
Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
Track when entering via call32() and use the same mode for stack_hop_back()
Simplify farcall16 code
Update reset() to use call16_back()
build: Support declaring 32bit C functions that must reside in the f-segment
Move call16() functions from romlayout.S to inline assembler in stacks.c
Break up call32() into call32() and call32_sloppy()
Fully restore 16bit state during call16_sloppy()
Implement call32 mechanism using SMIs.
Move a20 code from system.c and ps2port.h to x86.h
Backup and restore a20 on call32_sloppy()
usb: Rename ?hci_control() to ?hci_send_control()
usb: Rename usb_getFrameExp() to usb_get_period()
usb: Rename findEndPointDesc() to usb_find_desc()
usb: Rename send_default_control() to usb_send_default_control()
usb: Rename free_pipe() to usb_free_pipe()
usb: Clarify usb freelist manipulations
xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
usb: Use usb_realloc_pipe for pipe alloc, update, and free.
Use 32bit memcpy in int1587 when applicable
Don't clobber %ax on ENTRY_INTO32 macro
Create assembler macros for saving and restoring 'struct bregs'
Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
Remove unused macro ENTRY_ST
vgabios: Don't declare custom internal BDA storage in std/bda.h
vgabios: Cache a pointer to the current mode struct in the BDA
vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
vgabios: Rename vbe_flags to flags
vgabios: Set cursor shape fixes
vgabios: Refactor get/set_cursor_shape() code
vgabios: Only init BDA device details in init_bios_area()
vgabios: Only set the dcc_index=8 if stdvga ports are available
vgabios: Move standard table definitions to std/vga.h
vgabios: Fill in available legacy modes in video_func_static at runtime
vgabios: Add support for reading framebuffer in "direct" mode
Fix PNP regression introduced in 99cb8f3e due to missed conversion
Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
vgabios: Support emulating text mode attributes while in graphics mode
vgabios: Add software cursor capability
Use an aligned stack offset when entering on the extra stack
Minor - comment updates in romlayout.S
Fix build issue on gcc34
pciinit: Fix build warning in mch_pci_slot_get_irq()
floppy: Make sure to yield() during floppy PIO
Minor - be consistent in placement of .code16/32 in romlayout.S
Use macros for .code16/32 mode switches in inline asm in stacks.c
Eliminate FUNCFSEG - only force portions of inline asm to f-segment
usb: Update USB hub code to support super speed hubs
Simplify README files - point to online documentation instead
sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
Add wiki documentation to repository
docs: Don't point to repo README files
docs: Add info on MODE16/MODESEGMENT compile time flags
docs: Add page describing SeaBIOS final object linking
scsi: Move cdb_* functions above scsi_* functions
scsi: Move process_scsi_op() to hw/blockcmd.c and rename
cdrom: call scsi_process_op() instead of cdb_read()
scsi: Don't export cdb_* functions
cdrom: Break up very large read requests into smaller requests
block: Check for read/write requests over 64K
usb: Add support for OHCI bulk transfers
readserial: Enhance pipe support
docs: Add documentation on using readserial.py script
uhci: Enable "depth" tree traversal for bulk transfers
uhci: Increase bulk transfer STACKTDS to 16
vgabios: Support emulated text in gfx_read_char()
ehci: No need to support td array wrapping
ehci: Simplify fillTDbuffer() and rename
ehci: Merge ehci_send_control with ehci_send_bulk
ohci: Merge ohci_send_control with ohci_send_bulk
uhci: Merge uhci_send_control with uhci_send_bulk
xhci: Merge xhci_send_control with xhci_send_bulk
usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
xhci: Move xhci_xfer_x() functions together
xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
usb: Control transfers always have an 8 byte command size
usb: Minor - properly free memory on get_device_config() error path
checkstack: Handle callw instruction
docs: Document why v1.6.3 release came after v0.6.2
docs: Update release history with dates of stable releases
docs: There is only one VAR16 flag now
docs: Note v1.8.0 release
Marcel Apfelbaum (1):
hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
Markus Armbruster (1):
boot: Fix boot order for SCSI target, lun > 9
Paolo Bonzini (5):
piix: add and use dev-piix.h
smm: complete SMM setup
smm: unify SMM handlers
vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
vgabios: implement read char in graphics mode
zhanghailiang (1):
acpi: use specified macro instead of magic-number
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-10 20:29 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-10 20:29 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Bandan Das, Andrey Korolyov, kraxel, kvm, qemu-devel
* Paolo Bonzini (pbonzini@redhat.com) wrote:
>
>
> On 10/03/2015 19:21, Bandan Das wrote:
> > Paolo Bonzini <pbonzini@redhat.com> writes:
> >
> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> >>> I'm seeing something similar; it's very intermittent and generally
> >>> happening right at boot of the guest; I'm running this on qemu
> >>> head+my postcopy world (but it's happening right at boot before postcopy
> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> >>> but hey maybe I'm seeing a different bug.
> >
> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> > From a cursory look and my limited understanding, it seems his failure is #GP
> > when executing video bios.
> >
> >> Same here on 3.16 + Xeon E5 v3 kernel.
> >
> > I will try to reproduce this on a v2.
>
> I see several failures, usually mine have suberror 1. With a 32-VCPU
> guest I can reproduce it roughly half of the time.
>
> Paolo
while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
(and leave about 2mins of runs before declaring good)
bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
bad: 21f5826a04d38e19488f917e1eef22751490c769
good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
good: c3edd62851098e6417786193ed9e9341781fcf57
good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
good: 73104fd399c6778112f64fe0d439319f24508d9a
good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
good: 4478aa768ccefcc5b234c23d035435fd71b932f6
good: 2.2.0
[root@virtlab413 qemu-world3]# git bisect bad
21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
commit 21f5826a04d38e19488f917e1eef22751490c769
Author: Gerd Hoffmann <kraxel@redhat.com>
Date: Thu Feb 19 09:33:03 2015 +0100
seabios: update to 1.8.0 release
'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
David Woodhouse (4):
Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
build: use -m16 where available instead of asm(".code16gcc")
romlayout: Use .code16 not .code16gcc
vgabios: Use .code16 not .code16gcc
Gerd Hoffmann (2):
add scripts/tarball.sh
build: set LC_ALL=C
Hannes Reinecke (1):
megasas: read addional PCI I/O bar
Ian Campbell (1):
romlayout: Use "rep ; nop" not "rep nop".
Kevin O'Connor (139):
vgabios: Return from handle_1011() if handler found.
edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
edd: Use sectors==-1 to detect removable media.
edd: Separate out ATA and virtio specific parts of fill_edd().
cdemu: store internal cdemu fields in standard "el-torito" spec format.
Move cdemu call interface and disk_ret helper code to disk.c.
smm: Replace SMI assembler code with C code.
smm: Use a C struct to define the layout of the SMM area.
smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
Don't enable thread preemption during S3 resume vga option rom execution.
Remove old Bochs bios fixed address string at 0xfff00.
Move most of the VAR16FIXED() defs to misc.c.
build: Avoid absolute paths during "whole-program" compiling.
Make sure handle_smi() and handle_smp() are compiled out if not enabled.
Remove the TODO file.
Abstract reset call (and possible 16bit mode switch) into reset() function.
build: Remove unused function getSectionsStart() from layoutrom.py.
build: Extract section visiting logic in layoutrom.py.
build: Refactor layoutrom.py gc() function.
build: Use customized entry point for each type of build.
build: Refactor findInit() function.
build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
build: Keep segmented sections separate until final link step.
build: Use fileid instead of category to write sections in layoutrom.py.
build: Only export needed fields in LayoutInfo in layoutrom.py.
build: Get fixed address variables from 32bit compile pass (not 16bit)
build: Minor - fix comments referring to old tools/ directory.
xhci: Update the times for usb command timeouts.
ehci: Update usb command timeouts to use usb_xfer_time()
uhci: Update usb command timeouts to use usb_xfer_time()
ohci: Update usb command timeouts to use usb_xfer_time()
vgabios: Fix broken build resulting from e5749978.
boot: Change ":rom%d" boot order rom instance to ":rom%x"
Minor - remove stray tab from src/fw/smm.c.
build: Update kconfig to version in Linux 3.16.
usb: Fix usb_xfer_time() to work when called in 16bit mode.
xhci: Call usb_desc2pipe() on xhci_update_pipe().
xhci: Remove 16bit code wrappers.
xhci: Use high memory instead of low memory for internal storage.
xhci: Move root hub and setup code to top of file.
xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
usb-hub: Enable power to all ports prior to calling usb_enumerate().
xhci: Change xhci_hub_detect() to use connect status instead of link state.
uhci: Repeatedly poll for device detect for 100ms.
ohci: Repeatedly poll for device detect for 100ms.
ehci: Stall uhci/ohci init only until default port routing is done.
usb: Perform device detect polling on all usb controllers.
ehci: Fix bug in hub port assignment
Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
pmm: Fix entry point to support non-zero %ss
Move stack hop code below call32/call16 code in stacks.c
Add need_hop_back() call that determines if stack_hop_back is needed
Update invoke_mouse_handler() to use need_hop_back()
Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
Track when entering via call32() and use the same mode for stack_hop_back()
Simplify farcall16 code
Update reset() to use call16_back()
build: Support declaring 32bit C functions that must reside in the f-segment
Move call16() functions from romlayout.S to inline assembler in stacks.c
Break up call32() into call32() and call32_sloppy()
Fully restore 16bit state during call16_sloppy()
Implement call32 mechanism using SMIs.
Move a20 code from system.c and ps2port.h to x86.h
Backup and restore a20 on call32_sloppy()
usb: Rename ?hci_control() to ?hci_send_control()
usb: Rename usb_getFrameExp() to usb_get_period()
usb: Rename findEndPointDesc() to usb_find_desc()
usb: Rename send_default_control() to usb_send_default_control()
usb: Rename free_pipe() to usb_free_pipe()
usb: Clarify usb freelist manipulations
xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
usb: Use usb_realloc_pipe for pipe alloc, update, and free.
Use 32bit memcpy in int1587 when applicable
Don't clobber %ax on ENTRY_INTO32 macro
Create assembler macros for saving and restoring 'struct bregs'
Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
Remove unused macro ENTRY_ST
vgabios: Don't declare custom internal BDA storage in std/bda.h
vgabios: Cache a pointer to the current mode struct in the BDA
vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
vgabios: Rename vbe_flags to flags
vgabios: Set cursor shape fixes
vgabios: Refactor get/set_cursor_shape() code
vgabios: Only init BDA device details in init_bios_area()
vgabios: Only set the dcc_index=8 if stdvga ports are available
vgabios: Move standard table definitions to std/vga.h
vgabios: Fill in available legacy modes in video_func_static at runtime
vgabios: Add support for reading framebuffer in "direct" mode
Fix PNP regression introduced in 99cb8f3e due to missed conversion
Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
vgabios: Support emulating text mode attributes while in graphics mode
vgabios: Add software cursor capability
Use an aligned stack offset when entering on the extra stack
Minor - comment updates in romlayout.S
Fix build issue on gcc34
pciinit: Fix build warning in mch_pci_slot_get_irq()
floppy: Make sure to yield() during floppy PIO
Minor - be consistent in placement of .code16/32 in romlayout.S
Use macros for .code16/32 mode switches in inline asm in stacks.c
Eliminate FUNCFSEG - only force portions of inline asm to f-segment
usb: Update USB hub code to support super speed hubs
Simplify README files - point to online documentation instead
sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
Add wiki documentation to repository
docs: Don't point to repo README files
docs: Add info on MODE16/MODESEGMENT compile time flags
docs: Add page describing SeaBIOS final object linking
scsi: Move cdb_* functions above scsi_* functions
scsi: Move process_scsi_op() to hw/blockcmd.c and rename
cdrom: call scsi_process_op() instead of cdb_read()
scsi: Don't export cdb_* functions
cdrom: Break up very large read requests into smaller requests
block: Check for read/write requests over 64K
usb: Add support for OHCI bulk transfers
readserial: Enhance pipe support
docs: Add documentation on using readserial.py script
uhci: Enable "depth" tree traversal for bulk transfers
uhci: Increase bulk transfer STACKTDS to 16
vgabios: Support emulated text in gfx_read_char()
ehci: No need to support td array wrapping
ehci: Simplify fillTDbuffer() and rename
ehci: Merge ehci_send_control with ehci_send_bulk
ohci: Merge ohci_send_control with ohci_send_bulk
uhci: Merge uhci_send_control with uhci_send_bulk
xhci: Merge xhci_send_control with xhci_send_bulk
usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
xhci: Move xhci_xfer_x() functions together
xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
usb: Control transfers always have an 8 byte command size
usb: Minor - properly free memory on get_device_config() error path
checkstack: Handle callw instruction
docs: Document why v1.6.3 release came after v0.6.2
docs: Update release history with dates of stable releases
docs: There is only one VAR16 flag now
docs: Note v1.8.0 release
Marcel Apfelbaum (1):
hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
Markus Armbruster (1):
boot: Fix boot order for SCSI target, lun > 9
Paolo Bonzini (5):
piix: add and use dev-piix.h
smm: complete SMM setup
smm: unify SMM handlers
vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
vgabios: implement read char in graphics mode
zhanghailiang (1):
acpi: use specified macro instead of magic-number
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-10 20:29 ` Dr. David Alan Gilbert
@ 2015-03-11 2:38 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 2:38 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>
>>
>> On 10/03/2015 19:21, Bandan Das wrote:
>> > Paolo Bonzini <pbonzini@redhat.com> writes:
>> >
>> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>> >>> I'm seeing something similar; it's very intermittent and generally
>> >>> happening right at boot of the guest; I'm running this on qemu
>> >>> head+my postcopy world (but it's happening right at boot before postcopy
>> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> >>> but hey maybe I'm seeing a different bug.
>> >
>> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
>> > From a cursory look and my limited understanding, it seems his failure is #GP
>> > when executing video bios.
>> >
>> >> Same here on 3.16 + Xeon E5 v3 kernel.
>> >
>> > I will try to reproduce this on a v2.
>>
>> I see several failures, usually mine have suberror 1. With a 32-VCPU
>> guest I can reproduce it roughly half of the time.
>>
>> Paolo
>
> while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> (and leave about 2mins of runs before declaring good)
>
> bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
> bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
> bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
> bad: 21f5826a04d38e19488f917e1eef22751490c769
> good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
> good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
> good: c3edd62851098e6417786193ed9e9341781fcf57
> good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
> good: 73104fd399c6778112f64fe0d439319f24508d9a
> good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
> good: 4478aa768ccefcc5b234c23d035435fd71b932f6
> good: 2.2.0
>
> [root@virtlab413 qemu-world3]# git bisect bad
> 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
I can reproduce this on E5-2620 v2 with David's "while true" test.
(The emulation failure I mean, not the suberror 2 that Andrey is seeing)
The commit that seems to have introduced this is -
commit 0673b7870063a3affbad9046fb6d385a4e734c19
Author: Kevin O'Connor <kevin@koconnor.net>
Date: Sat May 24 10:49:50 2014 -0400
smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
Change the multi-processor init code to trampoline into 32bit mode on
each of the additional processors. Implement an atomic lock so that
each processor performs its initialization serially.
I am not sure what in that change could cause this though..
Also, in my testing, "unrestricted_guest=0" avoids the failure.
> commit 21f5826a04d38e19488f917e1eef22751490c769
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date: Thu Feb 19 09:33:03 2015 +0100
>
> seabios: update to 1.8.0 release
>
> 'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
>
> David Woodhouse (4):
> Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
> build: use -m16 where available instead of asm(".code16gcc")
> romlayout: Use .code16 not .code16gcc
> vgabios: Use .code16 not .code16gcc
>
> Gerd Hoffmann (2):
> add scripts/tarball.sh
> build: set LC_ALL=C
>
> Hannes Reinecke (1):
> megasas: read addional PCI I/O bar
>
> Ian Campbell (1):
> romlayout: Use "rep ; nop" not "rep nop".
>
> Kevin O'Connor (139):
> vgabios: Return from handle_1011() if handler found.
> edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
> edd: Use sectors==-1 to detect removable media.
> edd: Separate out ATA and virtio specific parts of fill_edd().
> cdemu: store internal cdemu fields in standard "el-torito" spec format.
> Move cdemu call interface and disk_ret helper code to disk.c.
> smm: Replace SMI assembler code with C code.
> smm: Use a C struct to define the layout of the SMM area.
> smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> Don't enable thread preemption during S3 resume vga option rom execution.
> Remove old Bochs bios fixed address string at 0xfff00.
> Move most of the VAR16FIXED() defs to misc.c.
> build: Avoid absolute paths during "whole-program" compiling.
> Make sure handle_smi() and handle_smp() are compiled out if not enabled.
> Remove the TODO file.
> Abstract reset call (and possible 16bit mode switch) into reset() function.
> build: Remove unused function getSectionsStart() from layoutrom.py.
> build: Extract section visiting logic in layoutrom.py.
> build: Refactor layoutrom.py gc() function.
> build: Use customized entry point for each type of build.
> build: Refactor findInit() function.
> build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
> build: Keep segmented sections separate until final link step.
> build: Use fileid instead of category to write sections in layoutrom.py.
> build: Only export needed fields in LayoutInfo in layoutrom.py.
> build: Get fixed address variables from 32bit compile pass (not 16bit)
> build: Minor - fix comments referring to old tools/ directory.
> xhci: Update the times for usb command timeouts.
> ehci: Update usb command timeouts to use usb_xfer_time()
> uhci: Update usb command timeouts to use usb_xfer_time()
> ohci: Update usb command timeouts to use usb_xfer_time()
> vgabios: Fix broken build resulting from e5749978.
> boot: Change ":rom%d" boot order rom instance to ":rom%x"
> Minor - remove stray tab from src/fw/smm.c.
> build: Update kconfig to version in Linux 3.16.
> usb: Fix usb_xfer_time() to work when called in 16bit mode.
> xhci: Call usb_desc2pipe() on xhci_update_pipe().
> xhci: Remove 16bit code wrappers.
> xhci: Use high memory instead of low memory for internal storage.
> xhci: Move root hub and setup code to top of file.
> xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
> ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
> usb-hub: Enable power to all ports prior to calling usb_enumerate().
> xhci: Change xhci_hub_detect() to use connect status instead of link state.
> uhci: Repeatedly poll for device detect for 100ms.
> ohci: Repeatedly poll for device detect for 100ms.
> ehci: Stall uhci/ohci init only until default port routing is done.
> usb: Perform device detect polling on all usb controllers.
> ehci: Fix bug in hub port assignment
> Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
> pmm: Fix entry point to support non-zero %ss
> Move stack hop code below call32/call16 code in stacks.c
> Add need_hop_back() call that determines if stack_hop_back is needed
> Update invoke_mouse_handler() to use need_hop_back()
> Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
> Track when entering via call32() and use the same mode for stack_hop_back()
> Simplify farcall16 code
> Update reset() to use call16_back()
> build: Support declaring 32bit C functions that must reside in the f-segment
> Move call16() functions from romlayout.S to inline assembler in stacks.c
> Break up call32() into call32() and call32_sloppy()
> Fully restore 16bit state during call16_sloppy()
> Implement call32 mechanism using SMIs.
> Move a20 code from system.c and ps2port.h to x86.h
> Backup and restore a20 on call32_sloppy()
> usb: Rename ?hci_control() to ?hci_send_control()
> usb: Rename usb_getFrameExp() to usb_get_period()
> usb: Rename findEndPointDesc() to usb_find_desc()
> usb: Rename send_default_control() to usb_send_default_control()
> usb: Rename free_pipe() to usb_free_pipe()
> usb: Clarify usb freelist manipulations
> xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
> uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
> ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
> ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
> usb: Use usb_realloc_pipe for pipe alloc, update, and free.
> Use 32bit memcpy in int1587 when applicable
> Don't clobber %ax on ENTRY_INTO32 macro
> Create assembler macros for saving and restoring 'struct bregs'
> Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
> Remove unused macro ENTRY_ST
> vgabios: Don't declare custom internal BDA storage in std/bda.h
> vgabios: Cache a pointer to the current mode struct in the BDA
> vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
> vgabios: Rename vbe_flags to flags
> vgabios: Set cursor shape fixes
> vgabios: Refactor get/set_cursor_shape() code
> vgabios: Only init BDA device details in init_bios_area()
> vgabios: Only set the dcc_index=8 if stdvga ports are available
> vgabios: Move standard table definitions to std/vga.h
> vgabios: Fill in available legacy modes in video_func_static at runtime
> vgabios: Add support for reading framebuffer in "direct" mode
> Fix PNP regression introduced in 99cb8f3e due to missed conversion
> Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
> vgabios: Support emulating text mode attributes while in graphics mode
> vgabios: Add software cursor capability
> Use an aligned stack offset when entering on the extra stack
> Minor - comment updates in romlayout.S
> Fix build issue on gcc34
> pciinit: Fix build warning in mch_pci_slot_get_irq()
> floppy: Make sure to yield() during floppy PIO
> Minor - be consistent in placement of .code16/32 in romlayout.S
> Use macros for .code16/32 mode switches in inline asm in stacks.c
> Eliminate FUNCFSEG - only force portions of inline asm to f-segment
> usb: Update USB hub code to support super speed hubs
> Simplify README files - point to online documentation instead
> sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
> Add wiki documentation to repository
> docs: Don't point to repo README files
> docs: Add info on MODE16/MODESEGMENT compile time flags
> docs: Add page describing SeaBIOS final object linking
> scsi: Move cdb_* functions above scsi_* functions
> scsi: Move process_scsi_op() to hw/blockcmd.c and rename
> cdrom: call scsi_process_op() instead of cdb_read()
> scsi: Don't export cdb_* functions
> cdrom: Break up very large read requests into smaller requests
> block: Check for read/write requests over 64K
> usb: Add support for OHCI bulk transfers
> readserial: Enhance pipe support
> docs: Add documentation on using readserial.py script
> uhci: Enable "depth" tree traversal for bulk transfers
> uhci: Increase bulk transfer STACKTDS to 16
> vgabios: Support emulated text in gfx_read_char()
> ehci: No need to support td array wrapping
> ehci: Simplify fillTDbuffer() and rename
> ehci: Merge ehci_send_control with ehci_send_bulk
> ohci: Merge ohci_send_control with ohci_send_bulk
> uhci: Merge uhci_send_control with uhci_send_bulk
> xhci: Merge xhci_send_control with xhci_send_bulk
> usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
> xhci: Move xhci_xfer_x() functions together
> xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
> usb: Control transfers always have an 8 byte command size
> usb: Minor - properly free memory on get_device_config() error path
> checkstack: Handle callw instruction
> docs: Document why v1.6.3 release came after v0.6.2
> docs: Update release history with dates of stable releases
> docs: There is only one VAR16 flag now
> docs: Note v1.8.0 release
>
> Marcel Apfelbaum (1):
> hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
>
> Markus Armbruster (1):
> boot: Fix boot order for SCSI target, lun > 9
>
> Paolo Bonzini (5):
> piix: add and use dev-piix.h
> smm: complete SMM setup
> smm: unify SMM handlers
> vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
> vgabios: implement read char in graphics mode
>
> zhanghailiang (1):
> acpi: use specified macro instead of magic-number
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>
>
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 2:38 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 2:38 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Paolo Bonzini, Andrey Korolyov, kraxel, kvm, qemu-devel
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>
>>
>> On 10/03/2015 19:21, Bandan Das wrote:
>> > Paolo Bonzini <pbonzini@redhat.com> writes:
>> >
>> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>> >>> I'm seeing something similar; it's very intermittent and generally
>> >>> happening right at boot of the guest; I'm running this on qemu
>> >>> head+my postcopy world (but it's happening right at boot before postcopy
>> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> >>> but hey maybe I'm seeing a different bug.
>> >
>> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
>> > From a cursory look and my limited understanding, it seems his failure is #GP
>> > when executing video bios.
>> >
>> >> Same here on 3.16 + Xeon E5 v3 kernel.
>> >
>> > I will try to reproduce this on a v2.
>>
>> I see several failures, usually mine have suberror 1. With a 32-VCPU
>> guest I can reproduce it roughly half of the time.
>>
>> Paolo
>
> while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> (and leave about 2mins of runs before declaring good)
>
> bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
> bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
> bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
> bad: 21f5826a04d38e19488f917e1eef22751490c769
> good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
> good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
> good: c3edd62851098e6417786193ed9e9341781fcf57
> good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
> good: 73104fd399c6778112f64fe0d439319f24508d9a
> good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
> good: 4478aa768ccefcc5b234c23d035435fd71b932f6
> good: 2.2.0
>
> [root@virtlab413 qemu-world3]# git bisect bad
> 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
I can reproduce this on E5-2620 v2 with David's "while true" test.
(The emulation failure I mean, not the suberror 2 that Andrey is seeing)
The commit that seems to have introduced this is -
commit 0673b7870063a3affbad9046fb6d385a4e734c19
Author: Kevin O'Connor <kevin@koconnor.net>
Date: Sat May 24 10:49:50 2014 -0400
smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
Change the multi-processor init code to trampoline into 32bit mode on
each of the additional processors. Implement an atomic lock so that
each processor performs its initialization serially.
I am not sure what in that change could cause this though..
Also, in my testing, "unrestricted_guest=0" avoids the failure.
> commit 21f5826a04d38e19488f917e1eef22751490c769
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date: Thu Feb 19 09:33:03 2015 +0100
>
> seabios: update to 1.8.0 release
>
> 'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
>
> David Woodhouse (4):
> Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
> build: use -m16 where available instead of asm(".code16gcc")
> romlayout: Use .code16 not .code16gcc
> vgabios: Use .code16 not .code16gcc
>
> Gerd Hoffmann (2):
> add scripts/tarball.sh
> build: set LC_ALL=C
>
> Hannes Reinecke (1):
> megasas: read addional PCI I/O bar
>
> Ian Campbell (1):
> romlayout: Use "rep ; nop" not "rep nop".
>
> Kevin O'Connor (139):
> vgabios: Return from handle_1011() if handler found.
> edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
> edd: Use sectors==-1 to detect removable media.
> edd: Separate out ATA and virtio specific parts of fill_edd().
> cdemu: store internal cdemu fields in standard "el-torito" spec format.
> Move cdemu call interface and disk_ret helper code to disk.c.
> smm: Replace SMI assembler code with C code.
> smm: Use a C struct to define the layout of the SMM area.
> smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> Don't enable thread preemption during S3 resume vga option rom execution.
> Remove old Bochs bios fixed address string at 0xfff00.
> Move most of the VAR16FIXED() defs to misc.c.
> build: Avoid absolute paths during "whole-program" compiling.
> Make sure handle_smi() and handle_smp() are compiled out if not enabled.
> Remove the TODO file.
> Abstract reset call (and possible 16bit mode switch) into reset() function.
> build: Remove unused function getSectionsStart() from layoutrom.py.
> build: Extract section visiting logic in layoutrom.py.
> build: Refactor layoutrom.py gc() function.
> build: Use customized entry point for each type of build.
> build: Refactor findInit() function.
> build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
> build: Keep segmented sections separate until final link step.
> build: Use fileid instead of category to write sections in layoutrom.py.
> build: Only export needed fields in LayoutInfo in layoutrom.py.
> build: Get fixed address variables from 32bit compile pass (not 16bit)
> build: Minor - fix comments referring to old tools/ directory.
> xhci: Update the times for usb command timeouts.
> ehci: Update usb command timeouts to use usb_xfer_time()
> uhci: Update usb command timeouts to use usb_xfer_time()
> ohci: Update usb command timeouts to use usb_xfer_time()
> vgabios: Fix broken build resulting from e5749978.
> boot: Change ":rom%d" boot order rom instance to ":rom%x"
> Minor - remove stray tab from src/fw/smm.c.
> build: Update kconfig to version in Linux 3.16.
> usb: Fix usb_xfer_time() to work when called in 16bit mode.
> xhci: Call usb_desc2pipe() on xhci_update_pipe().
> xhci: Remove 16bit code wrappers.
> xhci: Use high memory instead of low memory for internal storage.
> xhci: Move root hub and setup code to top of file.
> xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
> ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
> usb-hub: Enable power to all ports prior to calling usb_enumerate().
> xhci: Change xhci_hub_detect() to use connect status instead of link state.
> uhci: Repeatedly poll for device detect for 100ms.
> ohci: Repeatedly poll for device detect for 100ms.
> ehci: Stall uhci/ohci init only until default port routing is done.
> usb: Perform device detect polling on all usb controllers.
> ehci: Fix bug in hub port assignment
> Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
> pmm: Fix entry point to support non-zero %ss
> Move stack hop code below call32/call16 code in stacks.c
> Add need_hop_back() call that determines if stack_hop_back is needed
> Update invoke_mouse_handler() to use need_hop_back()
> Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
> Track when entering via call32() and use the same mode for stack_hop_back()
> Simplify farcall16 code
> Update reset() to use call16_back()
> build: Support declaring 32bit C functions that must reside in the f-segment
> Move call16() functions from romlayout.S to inline assembler in stacks.c
> Break up call32() into call32() and call32_sloppy()
> Fully restore 16bit state during call16_sloppy()
> Implement call32 mechanism using SMIs.
> Move a20 code from system.c and ps2port.h to x86.h
> Backup and restore a20 on call32_sloppy()
> usb: Rename ?hci_control() to ?hci_send_control()
> usb: Rename usb_getFrameExp() to usb_get_period()
> usb: Rename findEndPointDesc() to usb_find_desc()
> usb: Rename send_default_control() to usb_send_default_control()
> usb: Rename free_pipe() to usb_free_pipe()
> usb: Clarify usb freelist manipulations
> xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
> uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
> ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
> ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
> usb: Use usb_realloc_pipe for pipe alloc, update, and free.
> Use 32bit memcpy in int1587 when applicable
> Don't clobber %ax on ENTRY_INTO32 macro
> Create assembler macros for saving and restoring 'struct bregs'
> Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
> Remove unused macro ENTRY_ST
> vgabios: Don't declare custom internal BDA storage in std/bda.h
> vgabios: Cache a pointer to the current mode struct in the BDA
> vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
> vgabios: Rename vbe_flags to flags
> vgabios: Set cursor shape fixes
> vgabios: Refactor get/set_cursor_shape() code
> vgabios: Only init BDA device details in init_bios_area()
> vgabios: Only set the dcc_index=8 if stdvga ports are available
> vgabios: Move standard table definitions to std/vga.h
> vgabios: Fill in available legacy modes in video_func_static at runtime
> vgabios: Add support for reading framebuffer in "direct" mode
> Fix PNP regression introduced in 99cb8f3e due to missed conversion
> Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
> vgabios: Support emulating text mode attributes while in graphics mode
> vgabios: Add software cursor capability
> Use an aligned stack offset when entering on the extra stack
> Minor - comment updates in romlayout.S
> Fix build issue on gcc34
> pciinit: Fix build warning in mch_pci_slot_get_irq()
> floppy: Make sure to yield() during floppy PIO
> Minor - be consistent in placement of .code16/32 in romlayout.S
> Use macros for .code16/32 mode switches in inline asm in stacks.c
> Eliminate FUNCFSEG - only force portions of inline asm to f-segment
> usb: Update USB hub code to support super speed hubs
> Simplify README files - point to online documentation instead
> sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
> Add wiki documentation to repository
> docs: Don't point to repo README files
> docs: Add info on MODE16/MODESEGMENT compile time flags
> docs: Add page describing SeaBIOS final object linking
> scsi: Move cdb_* functions above scsi_* functions
> scsi: Move process_scsi_op() to hw/blockcmd.c and rename
> cdrom: call scsi_process_op() instead of cdb_read()
> scsi: Don't export cdb_* functions
> cdrom: Break up very large read requests into smaller requests
> block: Check for read/write requests over 64K
> usb: Add support for OHCI bulk transfers
> readserial: Enhance pipe support
> docs: Add documentation on using readserial.py script
> uhci: Enable "depth" tree traversal for bulk transfers
> uhci: Increase bulk transfer STACKTDS to 16
> vgabios: Support emulated text in gfx_read_char()
> ehci: No need to support td array wrapping
> ehci: Simplify fillTDbuffer() and rename
> ehci: Merge ehci_send_control with ehci_send_bulk
> ohci: Merge ohci_send_control with ohci_send_bulk
> uhci: Merge uhci_send_control with uhci_send_bulk
> xhci: Merge xhci_send_control with xhci_send_bulk
> usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
> xhci: Move xhci_xfer_x() functions together
> xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
> usb: Control transfers always have an 8 byte command size
> usb: Minor - properly free memory on get_device_config() error path
> checkstack: Handle callw instruction
> docs: Document why v1.6.3 release came after v0.6.2
> docs: Update release history with dates of stable releases
> docs: There is only one VAR16 flag now
> docs: Note v1.8.0 release
>
> Marcel Apfelbaum (1):
> hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
>
> Markus Armbruster (1):
> boot: Fix boot order for SCSI target, lun > 9
>
> Paolo Bonzini (5):
> piix: add and use dev-piix.h
> smm: complete SMM setup
> smm: unify SMM handlers
> vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
> vgabios: implement read char in graphics mode
>
> zhanghailiang (1):
> acpi: use specified macro instead of magic-number
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>
>
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 2:38 ` Bandan Das
@ 2015-03-11 13:45 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 13:45 UTC (permalink / raw)
To: Bandan Das; +Cc: Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
* Bandan Das (bsd@redhat.com) wrote:
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
>
> > * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >>
> >>
> >> On 10/03/2015 19:21, Bandan Das wrote:
> >> > Paolo Bonzini <pbonzini@redhat.com> writes:
> >> >
> >> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> >> >>> I'm seeing something similar; it's very intermittent and generally
> >> >>> happening right at boot of the guest; I'm running this on qemu
> >> >>> head+my postcopy world (but it's happening right at boot before postcopy
> >> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> >> >>> but hey maybe I'm seeing a different bug.
> >> >
> >> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> >> > From a cursory look and my limited understanding, it seems his failure is #GP
> >> > when executing video bios.
> >> >
> >> >> Same here on 3.16 + Xeon E5 v3 kernel.
> >> >
> >> > I will try to reproduce this on a v2.
> >>
> >> I see several failures, usually mine have suberror 1. With a 32-VCPU
> >> guest I can reproduce it roughly half of the time.
> >>
> >> Paolo
> >
> > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> >
> > (and leave about 2mins of runs before declaring good)
> >
> > bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
> > bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
> > bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
> > bad: 21f5826a04d38e19488f917e1eef22751490c769
> > good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
> > good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
> > good: c3edd62851098e6417786193ed9e9341781fcf57
> > good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
> > good: 73104fd399c6778112f64fe0d439319f24508d9a
> > good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
> > good: 4478aa768ccefcc5b234c23d035435fd71b932f6
> > good: 2.2.0
> >
> > [root@virtlab413 qemu-world3]# git bisect bad
> > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
>
> I can reproduce this on E5-2620 v2 with David's "while true" test.
> (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> The commit that seems to have introduced this is -
>
> commit 0673b7870063a3affbad9046fb6d385a4e734c19
> Author: Kevin O'Connor <kevin@koconnor.net>
> Date: Sat May 24 10:49:50 2014 -0400
>
> smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
>
> Change the multi-processor init code to trampoline into 32bit mode on
> each of the additional processors. Implement an atomic lock so that
> each processor performs its initialization serially.
>
> I am not sure what in that change could cause this though..
> Also, in my testing, "unrestricted_guest=0" avoids the failure.
Turning on debug logging
( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2112
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
Found 1 cpu(s) max supported 128 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
WARNING - Unable to allocate resource at copy_mptable:62!
Copying SMBIOS entry point from 0x00006db0 to 0x000f6580
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
is where it dies for me; actually with debug console on it's
pretty rare for it to boot. Is that:
'WARNING - Unable to allocate resource at copy_mptable:62!' the
canary ?
Dave
> > commit 21f5826a04d38e19488f917e1eef22751490c769
> > Author: Gerd Hoffmann <kraxel@redhat.com>
> > Date: Thu Feb 19 09:33:03 2015 +0100
> >
> > seabios: update to 1.8.0 release
> >
> > 'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
> >
> > David Woodhouse (4):
> > Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
> > build: use -m16 where available instead of asm(".code16gcc")
> > romlayout: Use .code16 not .code16gcc
> > vgabios: Use .code16 not .code16gcc
> >
> > Gerd Hoffmann (2):
> > add scripts/tarball.sh
> > build: set LC_ALL=C
> >
> > Hannes Reinecke (1):
> > megasas: read addional PCI I/O bar
> >
> > Ian Campbell (1):
> > romlayout: Use "rep ; nop" not "rep nop".
> >
> > Kevin O'Connor (139):
> > vgabios: Return from handle_1011() if handler found.
> > edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
> > edd: Use sectors==-1 to detect removable media.
> > edd: Separate out ATA and virtio specific parts of fill_edd().
> > cdemu: store internal cdemu fields in standard "el-torito" spec format.
> > Move cdemu call interface and disk_ret helper code to disk.c.
> > smm: Replace SMI assembler code with C code.
> > smm: Use a C struct to define the layout of the SMM area.
> > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > Don't enable thread preemption during S3 resume vga option rom execution.
> > Remove old Bochs bios fixed address string at 0xfff00.
> > Move most of the VAR16FIXED() defs to misc.c.
> > build: Avoid absolute paths during "whole-program" compiling.
> > Make sure handle_smi() and handle_smp() are compiled out if not enabled.
> > Remove the TODO file.
> > Abstract reset call (and possible 16bit mode switch) into reset() function.
> > build: Remove unused function getSectionsStart() from layoutrom.py.
> > build: Extract section visiting logic in layoutrom.py.
> > build: Refactor layoutrom.py gc() function.
> > build: Use customized entry point for each type of build.
> > build: Refactor findInit() function.
> > build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
> > build: Keep segmented sections separate until final link step.
> > build: Use fileid instead of category to write sections in layoutrom.py.
> > build: Only export needed fields in LayoutInfo in layoutrom.py.
> > build: Get fixed address variables from 32bit compile pass (not 16bit)
> > build: Minor - fix comments referring to old tools/ directory.
> > xhci: Update the times for usb command timeouts.
> > ehci: Update usb command timeouts to use usb_xfer_time()
> > uhci: Update usb command timeouts to use usb_xfer_time()
> > ohci: Update usb command timeouts to use usb_xfer_time()
> > vgabios: Fix broken build resulting from e5749978.
> > boot: Change ":rom%d" boot order rom instance to ":rom%x"
> > Minor - remove stray tab from src/fw/smm.c.
> > build: Update kconfig to version in Linux 3.16.
> > usb: Fix usb_xfer_time() to work when called in 16bit mode.
> > xhci: Call usb_desc2pipe() on xhci_update_pipe().
> > xhci: Remove 16bit code wrappers.
> > xhci: Use high memory instead of low memory for internal storage.
> > xhci: Move root hub and setup code to top of file.
> > xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
> > ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
> > usb-hub: Enable power to all ports prior to calling usb_enumerate().
> > xhci: Change xhci_hub_detect() to use connect status instead of link state.
> > uhci: Repeatedly poll for device detect for 100ms.
> > ohci: Repeatedly poll for device detect for 100ms.
> > ehci: Stall uhci/ohci init only until default port routing is done.
> > usb: Perform device detect polling on all usb controllers.
> > ehci: Fix bug in hub port assignment
> > Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
> > pmm: Fix entry point to support non-zero %ss
> > Move stack hop code below call32/call16 code in stacks.c
> > Add need_hop_back() call that determines if stack_hop_back is needed
> > Update invoke_mouse_handler() to use need_hop_back()
> > Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
> > Track when entering via call32() and use the same mode for stack_hop_back()
> > Simplify farcall16 code
> > Update reset() to use call16_back()
> > build: Support declaring 32bit C functions that must reside in the f-segment
> > Move call16() functions from romlayout.S to inline assembler in stacks.c
> > Break up call32() into call32() and call32_sloppy()
> > Fully restore 16bit state during call16_sloppy()
> > Implement call32 mechanism using SMIs.
> > Move a20 code from system.c and ps2port.h to x86.h
> > Backup and restore a20 on call32_sloppy()
> > usb: Rename ?hci_control() to ?hci_send_control()
> > usb: Rename usb_getFrameExp() to usb_get_period()
> > usb: Rename findEndPointDesc() to usb_find_desc()
> > usb: Rename send_default_control() to usb_send_default_control()
> > usb: Rename free_pipe() to usb_free_pipe()
> > usb: Clarify usb freelist manipulations
> > xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
> > uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
> > ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
> > ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
> > usb: Use usb_realloc_pipe for pipe alloc, update, and free.
> > Use 32bit memcpy in int1587 when applicable
> > Don't clobber %ax on ENTRY_INTO32 macro
> > Create assembler macros for saving and restoring 'struct bregs'
> > Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
> > Remove unused macro ENTRY_ST
> > vgabios: Don't declare custom internal BDA storage in std/bda.h
> > vgabios: Cache a pointer to the current mode struct in the BDA
> > vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
> > vgabios: Rename vbe_flags to flags
> > vgabios: Set cursor shape fixes
> > vgabios: Refactor get/set_cursor_shape() code
> > vgabios: Only init BDA device details in init_bios_area()
> > vgabios: Only set the dcc_index=8 if stdvga ports are available
> > vgabios: Move standard table definitions to std/vga.h
> > vgabios: Fill in available legacy modes in video_func_static at runtime
> > vgabios: Add support for reading framebuffer in "direct" mode
> > Fix PNP regression introduced in 99cb8f3e due to missed conversion
> > Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
> > vgabios: Support emulating text mode attributes while in graphics mode
> > vgabios: Add software cursor capability
> > Use an aligned stack offset when entering on the extra stack
> > Minor - comment updates in romlayout.S
> > Fix build issue on gcc34
> > pciinit: Fix build warning in mch_pci_slot_get_irq()
> > floppy: Make sure to yield() during floppy PIO
> > Minor - be consistent in placement of .code16/32 in romlayout.S
> > Use macros for .code16/32 mode switches in inline asm in stacks.c
> > Eliminate FUNCFSEG - only force portions of inline asm to f-segment
> > usb: Update USB hub code to support super speed hubs
> > Simplify README files - point to online documentation instead
> > sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
> > Add wiki documentation to repository
> > docs: Don't point to repo README files
> > docs: Add info on MODE16/MODESEGMENT compile time flags
> > docs: Add page describing SeaBIOS final object linking
> > scsi: Move cdb_* functions above scsi_* functions
> > scsi: Move process_scsi_op() to hw/blockcmd.c and rename
> > cdrom: call scsi_process_op() instead of cdb_read()
> > scsi: Don't export cdb_* functions
> > cdrom: Break up very large read requests into smaller requests
> > block: Check for read/write requests over 64K
> > usb: Add support for OHCI bulk transfers
> > readserial: Enhance pipe support
> > docs: Add documentation on using readserial.py script
> > uhci: Enable "depth" tree traversal for bulk transfers
> > uhci: Increase bulk transfer STACKTDS to 16
> > vgabios: Support emulated text in gfx_read_char()
> > ehci: No need to support td array wrapping
> > ehci: Simplify fillTDbuffer() and rename
> > ehci: Merge ehci_send_control with ehci_send_bulk
> > ohci: Merge ohci_send_control with ohci_send_bulk
> > uhci: Merge uhci_send_control with uhci_send_bulk
> > xhci: Merge xhci_send_control with xhci_send_bulk
> > usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
> > xhci: Move xhci_xfer_x() functions together
> > xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
> > usb: Control transfers always have an 8 byte command size
> > usb: Minor - properly free memory on get_device_config() error path
> > checkstack: Handle callw instruction
> > docs: Document why v1.6.3 release came after v0.6.2
> > docs: Update release history with dates of stable releases
> > docs: There is only one VAR16 flag now
> > docs: Note v1.8.0 release
> >
> > Marcel Apfelbaum (1):
> > hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
> >
> > Markus Armbruster (1):
> > boot: Fix boot order for SCSI target, lun > 9
> >
> > Paolo Bonzini (5):
> > piix: add and use dev-piix.h
> > smm: complete SMM setup
> > smm: unify SMM handlers
> > vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
> > vgabios: implement read char in graphics mode
> >
> > zhanghailiang (1):
> > acpi: use specified macro instead of magic-number
> >
> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> >
> >
> >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 13:45 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 13:45 UTC (permalink / raw)
To: Bandan Das; +Cc: Paolo Bonzini, Andrey Korolyov, kraxel, kvm, qemu-devel
* Bandan Das (bsd@redhat.com) wrote:
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
>
> > * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >>
> >>
> >> On 10/03/2015 19:21, Bandan Das wrote:
> >> > Paolo Bonzini <pbonzini@redhat.com> writes:
> >> >
> >> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
> >> >>> I'm seeing something similar; it's very intermittent and generally
> >> >>> happening right at boot of the guest; I'm running this on qemu
> >> >>> head+my postcopy world (but it's happening right at boot before postcopy
> >> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
> >> >>> but hey maybe I'm seeing a different bug.
> >> >
> >> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
> >> > From a cursory look and my limited understanding, it seems his failure is #GP
> >> > when executing video bios.
> >> >
> >> >> Same here on 3.16 + Xeon E5 v3 kernel.
> >> >
> >> > I will try to reproduce this on a v2.
> >>
> >> I see several failures, usually mine have suberror 1. With a 32-VCPU
> >> guest I can reproduce it roughly half of the time.
> >>
> >> Paolo
> >
> > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> >
> > (and leave about 2mins of runs before declaring good)
> >
> > bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
> > bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
> > bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
> > bad: 21f5826a04d38e19488f917e1eef22751490c769
> > good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
> > good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
> > good: c3edd62851098e6417786193ed9e9341781fcf57
> > good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
> > good: 73104fd399c6778112f64fe0d439319f24508d9a
> > good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
> > good: 4478aa768ccefcc5b234c23d035435fd71b932f6
> > good: 2.2.0
> >
> > [root@virtlab413 qemu-world3]# git bisect bad
> > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
>
> I can reproduce this on E5-2620 v2 with David's "while true" test.
> (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> The commit that seems to have introduced this is -
>
> commit 0673b7870063a3affbad9046fb6d385a4e734c19
> Author: Kevin O'Connor <kevin@koconnor.net>
> Date: Sat May 24 10:49:50 2014 -0400
>
> smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
>
> Change the multi-processor init code to trampoline into 32bit mode on
> each of the additional processors. Implement an atomic lock so that
> each processor performs its initialization serially.
>
> I am not sure what in that change could cause this though..
> Also, in my testing, "unrestricted_guest=0" avoids the failure.
Turning on debug logging
( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2112
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
Found 1 cpu(s) max supported 128 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
WARNING - Unable to allocate resource at copy_mptable:62!
Copying SMBIOS entry point from 0x00006db0 to 0x000f6580
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
is where it dies for me; actually with debug console on it's
pretty rare for it to boot. Is that:
'WARNING - Unable to allocate resource at copy_mptable:62!' the
canary ?
Dave
> > commit 21f5826a04d38e19488f917e1eef22751490c769
> > Author: Gerd Hoffmann <kraxel@redhat.com>
> > Date: Thu Feb 19 09:33:03 2015 +0100
> >
> > seabios: update to 1.8.0 release
> >
> > 'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
> >
> > David Woodhouse (4):
> > Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
> > build: use -m16 where available instead of asm(".code16gcc")
> > romlayout: Use .code16 not .code16gcc
> > vgabios: Use .code16 not .code16gcc
> >
> > Gerd Hoffmann (2):
> > add scripts/tarball.sh
> > build: set LC_ALL=C
> >
> > Hannes Reinecke (1):
> > megasas: read addional PCI I/O bar
> >
> > Ian Campbell (1):
> > romlayout: Use "rep ; nop" not "rep nop".
> >
> > Kevin O'Connor (139):
> > vgabios: Return from handle_1011() if handler found.
> > edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
> > edd: Use sectors==-1 to detect removable media.
> > edd: Separate out ATA and virtio specific parts of fill_edd().
> > cdemu: store internal cdemu fields in standard "el-torito" spec format.
> > Move cdemu call interface and disk_ret helper code to disk.c.
> > smm: Replace SMI assembler code with C code.
> > smm: Use a C struct to define the layout of the SMM area.
> > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > Don't enable thread preemption during S3 resume vga option rom execution.
> > Remove old Bochs bios fixed address string at 0xfff00.
> > Move most of the VAR16FIXED() defs to misc.c.
> > build: Avoid absolute paths during "whole-program" compiling.
> > Make sure handle_smi() and handle_smp() are compiled out if not enabled.
> > Remove the TODO file.
> > Abstract reset call (and possible 16bit mode switch) into reset() function.
> > build: Remove unused function getSectionsStart() from layoutrom.py.
> > build: Extract section visiting logic in layoutrom.py.
> > build: Refactor layoutrom.py gc() function.
> > build: Use customized entry point for each type of build.
> > build: Refactor findInit() function.
> > build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
> > build: Keep segmented sections separate until final link step.
> > build: Use fileid instead of category to write sections in layoutrom.py.
> > build: Only export needed fields in LayoutInfo in layoutrom.py.
> > build: Get fixed address variables from 32bit compile pass (not 16bit)
> > build: Minor - fix comments referring to old tools/ directory.
> > xhci: Update the times for usb command timeouts.
> > ehci: Update usb command timeouts to use usb_xfer_time()
> > uhci: Update usb command timeouts to use usb_xfer_time()
> > ohci: Update usb command timeouts to use usb_xfer_time()
> > vgabios: Fix broken build resulting from e5749978.
> > boot: Change ":rom%d" boot order rom instance to ":rom%x"
> > Minor - remove stray tab from src/fw/smm.c.
> > build: Update kconfig to version in Linux 3.16.
> > usb: Fix usb_xfer_time() to work when called in 16bit mode.
> > xhci: Call usb_desc2pipe() on xhci_update_pipe().
> > xhci: Remove 16bit code wrappers.
> > xhci: Use high memory instead of low memory for internal storage.
> > xhci: Move root hub and setup code to top of file.
> > xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
> > ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
> > usb-hub: Enable power to all ports prior to calling usb_enumerate().
> > xhci: Change xhci_hub_detect() to use connect status instead of link state.
> > uhci: Repeatedly poll for device detect for 100ms.
> > ohci: Repeatedly poll for device detect for 100ms.
> > ehci: Stall uhci/ohci init only until default port routing is done.
> > usb: Perform device detect polling on all usb controllers.
> > ehci: Fix bug in hub port assignment
> > Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
> > pmm: Fix entry point to support non-zero %ss
> > Move stack hop code below call32/call16 code in stacks.c
> > Add need_hop_back() call that determines if stack_hop_back is needed
> > Update invoke_mouse_handler() to use need_hop_back()
> > Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
> > Track when entering via call32() and use the same mode for stack_hop_back()
> > Simplify farcall16 code
> > Update reset() to use call16_back()
> > build: Support declaring 32bit C functions that must reside in the f-segment
> > Move call16() functions from romlayout.S to inline assembler in stacks.c
> > Break up call32() into call32() and call32_sloppy()
> > Fully restore 16bit state during call16_sloppy()
> > Implement call32 mechanism using SMIs.
> > Move a20 code from system.c and ps2port.h to x86.h
> > Backup and restore a20 on call32_sloppy()
> > usb: Rename ?hci_control() to ?hci_send_control()
> > usb: Rename usb_getFrameExp() to usb_get_period()
> > usb: Rename findEndPointDesc() to usb_find_desc()
> > usb: Rename send_default_control() to usb_send_default_control()
> > usb: Rename free_pipe() to usb_free_pipe()
> > usb: Clarify usb freelist manipulations
> > xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
> > uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
> > ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
> > ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
> > usb: Use usb_realloc_pipe for pipe alloc, update, and free.
> > Use 32bit memcpy in int1587 when applicable
> > Don't clobber %ax on ENTRY_INTO32 macro
> > Create assembler macros for saving and restoring 'struct bregs'
> > Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
> > Remove unused macro ENTRY_ST
> > vgabios: Don't declare custom internal BDA storage in std/bda.h
> > vgabios: Cache a pointer to the current mode struct in the BDA
> > vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
> > vgabios: Rename vbe_flags to flags
> > vgabios: Set cursor shape fixes
> > vgabios: Refactor get/set_cursor_shape() code
> > vgabios: Only init BDA device details in init_bios_area()
> > vgabios: Only set the dcc_index=8 if stdvga ports are available
> > vgabios: Move standard table definitions to std/vga.h
> > vgabios: Fill in available legacy modes in video_func_static at runtime
> > vgabios: Add support for reading framebuffer in "direct" mode
> > Fix PNP regression introduced in 99cb8f3e due to missed conversion
> > Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
> > vgabios: Support emulating text mode attributes while in graphics mode
> > vgabios: Add software cursor capability
> > Use an aligned stack offset when entering on the extra stack
> > Minor - comment updates in romlayout.S
> > Fix build issue on gcc34
> > pciinit: Fix build warning in mch_pci_slot_get_irq()
> > floppy: Make sure to yield() during floppy PIO
> > Minor - be consistent in placement of .code16/32 in romlayout.S
> > Use macros for .code16/32 mode switches in inline asm in stacks.c
> > Eliminate FUNCFSEG - only force portions of inline asm to f-segment
> > usb: Update USB hub code to support super speed hubs
> > Simplify README files - point to online documentation instead
> > sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
> > Add wiki documentation to repository
> > docs: Don't point to repo README files
> > docs: Add info on MODE16/MODESEGMENT compile time flags
> > docs: Add page describing SeaBIOS final object linking
> > scsi: Move cdb_* functions above scsi_* functions
> > scsi: Move process_scsi_op() to hw/blockcmd.c and rename
> > cdrom: call scsi_process_op() instead of cdb_read()
> > scsi: Don't export cdb_* functions
> > cdrom: Break up very large read requests into smaller requests
> > block: Check for read/write requests over 64K
> > usb: Add support for OHCI bulk transfers
> > readserial: Enhance pipe support
> > docs: Add documentation on using readserial.py script
> > uhci: Enable "depth" tree traversal for bulk transfers
> > uhci: Increase bulk transfer STACKTDS to 16
> > vgabios: Support emulated text in gfx_read_char()
> > ehci: No need to support td array wrapping
> > ehci: Simplify fillTDbuffer() and rename
> > ehci: Merge ehci_send_control with ehci_send_bulk
> > ohci: Merge ohci_send_control with ohci_send_bulk
> > uhci: Merge uhci_send_control with uhci_send_bulk
> > xhci: Merge xhci_send_control with xhci_send_bulk
> > usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
> > xhci: Move xhci_xfer_x() functions together
> > xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
> > usb: Control transfers always have an 8 byte command size
> > usb: Minor - properly free memory on get_device_config() error path
> > checkstack: Handle callw instruction
> > docs: Document why v1.6.3 release came after v0.6.2
> > docs: Update release history with dates of stable releases
> > docs: There is only one VAR16 flag now
> > docs: Note v1.8.0 release
> >
> > Marcel Apfelbaum (1):
> > hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
> >
> > Markus Armbruster (1):
> > boot: Fix boot order for SCSI target, lun > 9
> >
> > Paolo Bonzini (5):
> > piix: add and use dev-piix.h
> > smm: complete SMM setup
> > smm: unify SMM handlers
> > vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
> > vgabios: implement read char in graphics mode
> >
> > zhanghailiang (1):
> > acpi: use specified macro instead of magic-number
> >
> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> >
> >
> >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 13:45 ` Dr. David Alan Gilbert
@ 2015-03-11 15:42 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 15:42 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> * Bandan Das (bsd@redhat.com) wrote:
> > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> > >
[...]
> > > [root@virtlab413 qemu-world3]# git bisect bad
> > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> >
> > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > The commit that seems to have introduced this is -
> >
> > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > Author: Kevin O'Connor <kevin@koconnor.net>
> > Date: Sat May 24 10:49:50 2014 -0400
> >
> > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
[...]
> Turning on debug logging
> ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 1 cpu(s) max supported 128 cpu(s)
Something is very odd here. When I run the above command (on an older
AMD machine) I get:
Found 128 cpu(s) max supported 128 cpu(s)
That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
That is, during smp init, SeaBIOS expects QEMU to tell it how many
cpus are active, and SeaBIOS waits until that many CPUs check in from
its SIPI request before proceeding.
I wonder if QEMU reported only 1 active cpu via that cmos register,
but more were actually active. If that was the case, it could
certainly explain the failure - as multiple cpus could be running
without the sipi trapoline in place.
What does the log look like on a non-failure case?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 15:42 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 15:42 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> * Bandan Das (bsd@redhat.com) wrote:
> > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> > >
[...]
> > > [root@virtlab413 qemu-world3]# git bisect bad
> > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> >
> > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > The commit that seems to have introduced this is -
> >
> > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > Author: Kevin O'Connor <kevin@koconnor.net>
> > Date: Sat May 24 10:49:50 2014 -0400
> >
> > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
[...]
> Turning on debug logging
> ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 1 cpu(s) max supported 128 cpu(s)
Something is very odd here. When I run the above command (on an older
AMD machine) I get:
Found 128 cpu(s) max supported 128 cpu(s)
That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
That is, during smp init, SeaBIOS expects QEMU to tell it how many
cpus are active, and SeaBIOS waits until that many CPUs check in from
its SIPI request before proceeding.
I wonder if QEMU reported only 1 active cpu via that cmos register,
but more were actually active. If that was the case, it could
certainly explain the failure - as multiple cpus could be running
without the sipi trapoline in place.
What does the log look like on a non-failure case?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 15:42 ` Kevin O'Connor
@ 2015-03-11 15:53 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 15:53 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > * Bandan Das (bsd@redhat.com) wrote:
> > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> > > >
> [...]
> > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > >
> > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > The commit that seems to have introduced this is -
> > >
> > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > Date: Sat May 24 10:49:50 2014 -0400
> > >
> > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> [...]
> > Turning on debug logging
> > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 1 cpu(s) max supported 128 cpu(s)
>
> Something is very odd here. When I run the above command (on an older
> AMD machine) I get:
>
> Found 128 cpu(s) max supported 128 cpu(s)
>
> That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> That is, during smp init, SeaBIOS expects QEMU to tell it how many
> cpus are active, and SeaBIOS waits until that many CPUs check in from
> its SIPI request before proceeding.
>
> I wonder if QEMU reported only 1 active cpu via that cmos register,
> but more were actually active. If that was the case, it could
> certainly explain the failure - as multiple cpus could be running
> without the sipi trapoline in place.
>
> What does the log look like on a non-failure case?
I had to drop down from 128 to get a working run with debug; here
are two runs with -smp 20 the first one worked, the second one
failed.
Dave
=========== Working ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2113
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
handle_smp: apic_id=12
handle_smp: apic_id=8
handle_smp: apic_id=14
handle_smp: apic_id=2
handle_smp: apic_id=13
handle_smp: apic_id=18
handle_smp: apic_id=1
handle_smp: apic_id=7
handle_smp: apic_id=3
handle_smp: apic_id=4
handle_smp: apic_id=6
handle_smp: apic_id=11
handle_smp: apic_id=10
handle_smp: apic_id=15
handle_smp: apic_id=9
handle_smp: apic_id=16
handle_smp: apic_id=17
handle_smp: apic_id=19
handle_smp: apic_id=5
Found 20 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
Copying MPTABLE from 0x00006db0/3ffa5c60 to 0x000f6340
Copying SMBIOS entry point from 0x00006db0 to 0x000f6320
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5e20-f6240
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from Floppy...
floppy error: 40 00 00 00 00 01 02
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from DVD/CD...
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Boot failed: Could not read from CDROM (code 0003)
enter handle_18:
NULL
Booting from ROM...
Booting from ca80:0361
=========== Broken ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2114
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
Found 1 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
Copying MPTABLE from 0x00006db0/3ffa5c60 to 0x000f6340
Copying SMBIOS entry point from 0x00006db0 to 0x000f6320
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 15:53 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 15:53 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > * Bandan Das (bsd@redhat.com) wrote:
> > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
> > > >
> [...]
> > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > >
> > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > The commit that seems to have introduced this is -
> > >
> > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > Date: Sat May 24 10:49:50 2014 -0400
> > >
> > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> [...]
> > Turning on debug logging
> > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 1 cpu(s) max supported 128 cpu(s)
>
> Something is very odd here. When I run the above command (on an older
> AMD machine) I get:
>
> Found 128 cpu(s) max supported 128 cpu(s)
>
> That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> That is, during smp init, SeaBIOS expects QEMU to tell it how many
> cpus are active, and SeaBIOS waits until that many CPUs check in from
> its SIPI request before proceeding.
>
> I wonder if QEMU reported only 1 active cpu via that cmos register,
> but more were actually active. If that was the case, it could
> certainly explain the failure - as multiple cpus could be running
> without the sipi trapoline in place.
>
> What does the log look like on a non-failure case?
I had to drop down from 128 to get a working run with debug; here
are two runs with -smp 20 the first one worked, the second one
failed.
Dave
=========== Working ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2113
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
handle_smp: apic_id=12
handle_smp: apic_id=8
handle_smp: apic_id=14
handle_smp: apic_id=2
handle_smp: apic_id=13
handle_smp: apic_id=18
handle_smp: apic_id=1
handle_smp: apic_id=7
handle_smp: apic_id=3
handle_smp: apic_id=4
handle_smp: apic_id=6
handle_smp: apic_id=11
handle_smp: apic_id=10
handle_smp: apic_id=15
handle_smp: apic_id=9
handle_smp: apic_id=16
handle_smp: apic_id=17
handle_smp: apic_id=19
handle_smp: apic_id=5
Found 20 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
Copying MPTABLE from 0x00006db0/3ffa5c60 to 0x000f6340
Copying SMBIOS entry point from 0x00006db0 to 0x000f6320
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5e20-f6240
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from Floppy...
floppy error: 40 00 00 00 00 01 02
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from DVD/CD...
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Boot failed: Could not read from CDROM (code 0003)
enter handle_18:
NULL
Booting from ROM...
Booting from ca80:0361
=========== Broken ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000dea20 to 0x3ffaed30 (size 70160)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2114
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
Found 1 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc98 to 0x000f65a0
Copying MPTABLE from 0x00006db0/3ffa5c60 to 0x000f6340
Copying SMBIOS entry point from 0x00006db0 to 0x000f6320
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006970 bp=00000000 sp=00006d0a cs=f000 ip=d239 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f4
pmm call arg1=0
VGA stack allocated at ef1b0
Running option rom at c980:0003
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-11 15:53 ` Dr. David Alan Gilbert
@ 2015-03-11 16:37 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 16:37 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 03:53:07PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > > * Bandan Das (bsd@redhat.com) wrote:
> > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
That is a truly impressive command line, BTW.
> > > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > > >
> > > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > > The commit that seems to have introduced this is -
> > > >
> > > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > > Date: Sat May 24 10:49:50 2014 -0400
> > > >
> > > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > [...]
> > > Turning on debug logging
> > > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> > >
> > > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> > [...]
> > > Found 1 cpu(s) max supported 128 cpu(s)
> >
> > Something is very odd here. When I run the above command (on an older
> > AMD machine) I get:
> >
> > Found 128 cpu(s) max supported 128 cpu(s)
> >
> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > its SIPI request before proceeding.
> >
> > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > but more were actually active. If that was the case, it could
> > certainly explain the failure - as multiple cpus could be running
> > without the sipi trapoline in place.
> >
> > What does the log look like on a non-failure case?
>
> I had to drop down from 128 to get a working run with debug; here
> are two runs with -smp 20 the first one worked, the second one
> failed.
[...]
> =========== Working ===========
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 20 cpu(s) max supported 20 cpu(s)
[...]
> =========== Broken ===========
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 1 cpu(s) max supported 20 cpu(s)
So, I couldn't get this to fail on my older AMD machine at all with
the default SeaBIOS code. But, when I change the code with the patch
below, it failed right away.
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2b8
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=000fd2c1 EFL=00000007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008300 DPL=0 TSS16-busy
GDT= 000f6a50 00000037
IDT= 000f6a8e 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 ba b8 d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb 3f 00 <72> f3 8b 25 00 ff fb 3f e8 d2 65 ff ff c7 05 04 ff fb 3f 00 00 00 00 f4 eb fd fa fc 66 b8
And the failed debug output looks like:
SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
[...]
cmos_smp_count0=20
[...]
cmos_smp_count=1
cmos_smp_count2=1/20
Found 1 cpu(s) max supported 20 cpu(s)
I'm going to check the assembly for a compiler error, but is it
possible QEMU is returning incorrect data in cmos index 0x5f?
David, any chance you can recompile seabios and double check your
output?
-Kevin
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -128,6 +128,7 @@ smp_setup(void)
// Wait for other CPUs to process the SIPI.
u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
@@ -140,6 +141,8 @@ smp_setup(void)
: "+m" (SMPLock), "+m" (SMPStack)
: : "cc", "memory");
yield();
+ dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
+ , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
// Restore memory.
*(u64*)BUILD_AP_BOOT_ADDR = old;
diff --git a/src/post.c b/src/post.c
index 9ea5620..dc11c72 100644
--- a/src/post.c
+++ b/src/post.c
@@ -170,6 +170,7 @@ platform_hardware_setup(void)
clock_setup();
// Platform specific setup
+ dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
qemu_platform_setup();
coreboot_platform_setup();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 16:37 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 16:37 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 03:53:07PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > > * Bandan Das (bsd@redhat.com) wrote:
> > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
That is a truly impressive command line, BTW.
> > > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > > >
> > > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > > The commit that seems to have introduced this is -
> > > >
> > > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > > Date: Sat May 24 10:49:50 2014 -0400
> > > >
> > > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > [...]
> > > Turning on debug logging
> > > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> > >
> > > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> > [...]
> > > Found 1 cpu(s) max supported 128 cpu(s)
> >
> > Something is very odd here. When I run the above command (on an older
> > AMD machine) I get:
> >
> > Found 128 cpu(s) max supported 128 cpu(s)
> >
> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > its SIPI request before proceeding.
> >
> > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > but more were actually active. If that was the case, it could
> > certainly explain the failure - as multiple cpus could be running
> > without the sipi trapoline in place.
> >
> > What does the log look like on a non-failure case?
>
> I had to drop down from 128 to get a working run with debug; here
> are two runs with -smp 20 the first one worked, the second one
> failed.
[...]
> =========== Working ===========
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 20 cpu(s) max supported 20 cpu(s)
[...]
> =========== Broken ===========
>
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
[...]
> Found 1 cpu(s) max supported 20 cpu(s)
So, I couldn't get this to fail on my older AMD machine at all with
the default SeaBIOS code. But, when I change the code with the patch
below, it failed right away.
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2b8
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=000fd2c1 EFL=00000007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008300 DPL=0 TSS16-busy
GDT= 000f6a50 00000037
IDT= 000f6a8e 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 ba b8 d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb 3f 00 <72> f3 8b 25 00 ff fb 3f e8 d2 65 ff ff c7 05 04 ff fb 3f 00 00 00 00 f4 eb fd fa fc 66 b8
And the failed debug output looks like:
SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
[...]
cmos_smp_count0=20
[...]
cmos_smp_count=1
cmos_smp_count2=1/20
Found 1 cpu(s) max supported 20 cpu(s)
I'm going to check the assembly for a compiler error, but is it
possible QEMU is returning incorrect data in cmos index 0x5f?
David, any chance you can recompile seabios and double check your
output?
-Kevin
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -128,6 +128,7 @@ smp_setup(void)
// Wait for other CPUs to process the SIPI.
u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
@@ -140,6 +141,8 @@ smp_setup(void)
: "+m" (SMPLock), "+m" (SMPStack)
: : "cc", "memory");
yield();
+ dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
+ , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
// Restore memory.
*(u64*)BUILD_AP_BOOT_ADDR = old;
diff --git a/src/post.c b/src/post.c
index 9ea5620..dc11c72 100644
--- a/src/post.c
+++ b/src/post.c
@@ -170,6 +170,7 @@ platform_hardware_setup(void)
clock_setup();
// Platform specific setup
+ dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
qemu_platform_setup();
coreboot_platform_setup();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 16:37 ` [Qemu-devel] " Kevin O'Connor
@ 2015-03-11 16:52 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 16:52 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 03:53:07PM +0000, Dr. David Alan Gilbert wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > > > * Bandan Das (bsd@redhat.com) wrote:
> > > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> That is a truly impressive command line, BTW.
Thanks :-)
> > > > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > > > >
> > > > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > > > The commit that seems to have introduced this is -
> > > > >
> > > > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > > > Date: Sat May 24 10:49:50 2014 -0400
> > > > >
> > > > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > > [...]
> > > > Turning on debug logging
> > > > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> > > >
> > > > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> > > [...]
> > > > Found 1 cpu(s) max supported 128 cpu(s)
> > >
> > > Something is very odd here. When I run the above command (on an older
> > > AMD machine) I get:
> > >
> > > Found 128 cpu(s) max supported 128 cpu(s)
> > >
> > > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > > its SIPI request before proceeding.
> > >
> > > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > > but more were actually active. If that was the case, it could
> > > certainly explain the failure - as multiple cpus could be running
> > > without the sipi trapoline in place.
> > >
> > > What does the log look like on a non-failure case?
> >
> > I had to drop down from 128 to get a working run with debug; here
> > are two runs with -smp 20 the first one worked, the second one
> > failed.
> [...]
> > =========== Working ===========
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 20 cpu(s) max supported 20 cpu(s)
> [...]
> > =========== Broken ===========
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 1 cpu(s) max supported 20 cpu(s)
>
> So, I couldn't get this to fail on my older AMD machine at all with
> the default SeaBIOS code. But, when I change the code with the patch
> below, it failed right away.
>
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2b8
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=000fd2c1 EFL=00000007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008300 DPL=0 TSS16-busy
> GDT= 000f6a50 00000037
> IDT= 000f6a8e 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 ba b8 d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb 3f 00 <72> f3 8b 25 00 ff fb 3f e8 d2 65 ff ff c7 05 04 ff fb 3f 00 00 00 00 f4 eb fd fa fc 66 b8
>
> And the failed debug output looks like:
>
> SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> [...]
> cmos_smp_count0=20
> [...]
> cmos_smp_count=1
> cmos_smp_count2=1/20
> Found 1 cpu(s) max supported 20 cpu(s)
>
> I'm going to check the assembly for a compiler error, but is it
> possible QEMU is returning incorrect data in cmos index 0x5f?
>
> David, any chance you can recompile seabios and double check your
> output?
Done;
=========== Working ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000df4e0 to 0x3ffaf570 (size 68048)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2116
cmos_smp_count0=20
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
cmos_smp_count=20
handle_smp: apic_id=8
handle_smp: apic_id=4
handle_smp: apic_id=2
handle_smp: apic_id=15
handle_smp: apic_id=16
handle_smp: apic_id=10
handle_smp: apic_id=3
handle_smp: apic_id=12
handle_smp: apic_id=19
handle_smp: apic_id=14
handle_smp: apic_id=11
handle_smp: apic_id=7
handle_smp: apic_id=17
handle_smp: apic_id=6
handle_smp: apic_id=18
handle_smp: apic_id=13
handle_smp: apic_id=9
handle_smp: apic_id=5
handle_smp: apic_id=1
cmos_smp_count2=20/20
Found 20 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc9c to 0x000f6720
Copying MPTABLE from 0x00006de8/3ffa64a0 to 0x000f64c0
Copying SMBIOS entry point from 0x00006de8 to 0x000f64a0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006af0 bp=00000000 sp=00006d66 cs=f000 ip=d244 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f6
pmm call arg1=0
VGA stack allocated at ef430
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5fa0-f63c0
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from Floppy...
floppy error: 40 00 00 00 00 01 02
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from DVD/CD...
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Boot failed: Could not read from CDROM (code 0003)
enter handle_18:
NULL
Booting from ROM...
Booting from ca80:0361
=========== Broken ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000df4e0 to 0x3ffaf570 (size 68048)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2115
cmos_smp_count0=20
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
cmos_smp_count=1
cmos_smp_count2=1/20
Found 1 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc9c to 0x000f6720
Copying MPTABLE from 0x00006de8/3ffa64a0 to 0x000f64c0
Copying SMBIOS entry point from 0x00006de8 to 0x000f64a0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006af0 bp=00000000 sp=00006d66 cs=f000 ip=d244 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f6
pmm call arg1=0
VGA stack allocated at ef430
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5fa0-f63c0
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
> -Kevin
>
>
> --- a/src/fw/smp.c
> +++ b/src/fw/smp.c
> @@ -128,6 +128,7 @@ smp_setup(void)
>
> // Wait for other CPUs to process the SIPI.
> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
> + dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
> while (cmos_smp_count != CountCPUs)
> asm volatile(
> // Release lock and allow other processors to use the stack.
> @@ -140,6 +141,8 @@ smp_setup(void)
> : "+m" (SMPLock), "+m" (SMPStack)
> : : "cc", "memory");
> yield();
> + dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
> + , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
>
> // Restore memory.
> *(u64*)BUILD_AP_BOOT_ADDR = old;
> diff --git a/src/post.c b/src/post.c
> index 9ea5620..dc11c72 100644
> --- a/src/post.c
> +++ b/src/post.c
> @@ -170,6 +170,7 @@ platform_hardware_setup(void)
> clock_setup();
>
> // Platform specific setup
> + dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
> qemu_platform_setup();
> coreboot_platform_setup();
> }
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 16:52 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 16:52 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 03:53:07PM +0000, Dr. David Alan Gilbert wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > On Wed, Mar 11, 2015 at 01:45:57PM +0000, Dr. David Alan Gilbert wrote:
> > > > * Bandan Das (bsd@redhat.com) wrote:
> > > > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> > > > > > while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> That is a truly impressive command line, BTW.
Thanks :-)
> > > > > > [root@virtlab413 qemu-world3]# git bisect bad
> > > > > > 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit
> > > > >
> > > > > I can reproduce this on E5-2620 v2 with David's "while true" test.
> > > > > (The emulation failure I mean, not the suberror 2 that Andrey is seeing)
> > > > > The commit that seems to have introduced this is -
> > > > >
> > > > > commit 0673b7870063a3affbad9046fb6d385a4e734c19
> > > > > Author: Kevin O'Connor <kevin@koconnor.net>
> > > > > Date: Sat May 24 10:49:50 2014 -0400
> > > > >
> > > > > smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
> > > [...]
> > > > Turning on debug logging
> > > > ( -chardev file,id=log,path=/tmp/debugcon.$$ -device isa-debugcon,chardev=log,iobase=0x402 )
> > > >
> > > > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> > > [...]
> > > > Found 1 cpu(s) max supported 128 cpu(s)
> > >
> > > Something is very odd here. When I run the above command (on an older
> > > AMD machine) I get:
> > >
> > > Found 128 cpu(s) max supported 128 cpu(s)
> > >
> > > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > > its SIPI request before proceeding.
> > >
> > > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > > but more were actually active. If that was the case, it could
> > > certainly explain the failure - as multiple cpus could be running
> > > without the sipi trapoline in place.
> > >
> > > What does the log look like on a non-failure case?
> >
> > I had to drop down from 128 to get a working run with debug; here
> > are two runs with -smp 20 the first one worked, the second one
> > failed.
> [...]
> > =========== Working ===========
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 20 cpu(s) max supported 20 cpu(s)
> [...]
> > =========== Broken ===========
> >
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-20150219_092859-nilsson.home.kraxel.org)
> [...]
> > Found 1 cpu(s) max supported 20 cpu(s)
>
> So, I couldn't get this to fail on my older AMD machine at all with
> the default SeaBIOS code. But, when I change the code with the patch
> below, it failed right away.
>
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=000fd2b8
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=000fd2c1 EFL=00000007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008300 DPL=0 TSS16-busy
> GDT= 000f6a50 00000037
> IDT= 000f6a8e 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 ba b8 d2 0f 00 e9 a2 fe f3 90 f0 0f ba 2d 04 ff fb 3f 00 <72> f3 8b 25 00 ff fb 3f e8 d2 65 ff ff c7 05 04 ff fb 3f 00 00 00 00 f4 eb fd fa fc 66 b8
>
> And the failed debug output looks like:
>
> SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> [...]
> cmos_smp_count0=20
> [...]
> cmos_smp_count=1
> cmos_smp_count2=1/20
> Found 1 cpu(s) max supported 20 cpu(s)
>
> I'm going to check the assembly for a compiler error, but is it
> possible QEMU is returning incorrect data in cmos index 0x5f?
>
> David, any chance you can recompile seabios and double check your
> output?
Done;
=========== Working ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000df4e0 to 0x3ffaf570 (size 68048)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2116
cmos_smp_count0=20
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
cmos_smp_count=20
handle_smp: apic_id=8
handle_smp: apic_id=4
handle_smp: apic_id=2
handle_smp: apic_id=15
handle_smp: apic_id=16
handle_smp: apic_id=10
handle_smp: apic_id=3
handle_smp: apic_id=12
handle_smp: apic_id=19
handle_smp: apic_id=14
handle_smp: apic_id=11
handle_smp: apic_id=7
handle_smp: apic_id=17
handle_smp: apic_id=6
handle_smp: apic_id=18
handle_smp: apic_id=13
handle_smp: apic_id=9
handle_smp: apic_id=5
handle_smp: apic_id=1
cmos_smp_count2=20/20
Found 20 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc9c to 0x000f6720
Copying MPTABLE from 0x00006de8/3ffa64a0 to 0x000f64c0
Copying SMBIOS entry point from 0x00006de8 to 0x000f64a0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006af0 bp=00000000 sp=00006d66 cs=f000 ip=d244 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f6
pmm call arg1=0
VGA stack allocated at ef430
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5fa0-f63c0
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from Floppy...
floppy error: 40 00 00 00 00 01 02
Boot failed: could not read the boot disk
enter handle_18:
NULL
Booting from DVD/CD...
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Boot failed: Could not read from CDROM (code 0003)
enter handle_18:
NULL
Booting from ROM...
Booting from ca80:0361
=========== Broken ===========
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
No Xen hypervisor found.
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000df4e0 to 0x3ffaf570 (size 68048)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
CPU Mhz=2115
cmos_smp_count0=20
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c04f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:03.0 bar 1, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c040, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 6, addr feb80000, size 00040000 [mem]
PCI: map device bdf=00:03.0 bar 0, addr febc0000, size 00020000 [mem]
PCI: map device bdf=00:02.0 bar 6, addr febe0000, size 00010000 [mem]
PCI: map device bdf=00:02.0 bar 2, addr febf0000, size 00001000 [mem]
PCI: map device bdf=00:02.0 bar 0, addr fd000000, size 01000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:02.0 id=1234:1111
PCI: init bdf=00:03.0 id=8086:100e
PCI: Using 00:02.0 for primary VGA
cmos_smp_count=1
cmos_smp_count2=1/20
Found 1 cpu(s) max supported 20 cpu(s)
Copying PIR from 0x3ffbfc9c to 0x000f6720
Copying MPTABLE from 0x00006de8/3ffa64a0 to 0x000f64c0
Copying SMBIOS entry point from 0x00006de8 to 0x000f64a0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.8.0-0-g4c59f5d-20150219_092912-nilsson.home.kraxel.org)
enter vga_post:
a=00000010 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=00006af0 bp=00000000 sp=00006d66 cs=f000 ip=d244 f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=fd000000, size 16 MB
Attempting to allocate VGA stack via pmm call to f000:d2f6
pmm call arg1=0
VGA stack allocated at ef430
Running option rom at c980:0003
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
All threads complete.
Found 1 lpt ports
Found 1 serial ports
Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at ca80:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
Running option rom at cb80:0003
Space available for UMB: ce000-ee800, f5fa0-f63c0
Returned 126976 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffdf000 = 1 RAM
4: 000000003ffdf000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
> -Kevin
>
>
> --- a/src/fw/smp.c
> +++ b/src/fw/smp.c
> @@ -128,6 +128,7 @@ smp_setup(void)
>
> // Wait for other CPUs to process the SIPI.
> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
> + dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
> while (cmos_smp_count != CountCPUs)
> asm volatile(
> // Release lock and allow other processors to use the stack.
> @@ -140,6 +141,8 @@ smp_setup(void)
> : "+m" (SMPLock), "+m" (SMPStack)
> : : "cc", "memory");
> yield();
> + dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
> + , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
>
> // Restore memory.
> *(u64*)BUILD_AP_BOOT_ADDR = old;
> diff --git a/src/post.c b/src/post.c
> index 9ea5620..dc11c72 100644
> --- a/src/post.c
> +++ b/src/post.c
> @@ -170,6 +170,7 @@ platform_hardware_setup(void)
> clock_setup();
>
> // Platform specific setup
> + dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
> qemu_platform_setup();
> coreboot_platform_setup();
> }
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 15:42 ` Kevin O'Connor
@ 2015-03-11 17:09 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 17:09 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Dr. David Alan Gilbert, Paolo Bonzini, kraxel, Andrey Korolyov,
qemu-devel, kvm
"Kevin O'Connor" <kevin@koconnor.net> writes:
...
>
> Something is very odd here. When I run the above command (on an older
> AMD machine) I get:
>
> Found 128 cpu(s) max supported 128 cpu(s)
>
> That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> That is, during smp init, SeaBIOS expects QEMU to tell it how many
> cpus are active, and SeaBIOS waits until that many CPUs check in from
> its SIPI request before proceeding.
>
> I wonder if QEMU reported only 1 active cpu via that cmos register,
> but more were actually active. If that was the case, it could
I was daring enough to try this and I don't see the crash :)
diff --git a/src/fw/smp.c b/src/fw/smp.c
index a466ea6..a346d46 100644
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
void VISIBLE32FLAT
handle_smp(void)
{
+ dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
if (!CONFIG_QEMU)
return;
@@ -128,6 +129,8 @@ smp_setup(void)
// Wait for other CPUs to process the SIPI.
u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ while (cmos_smp_count == 1)
+ cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
So, the while loop results in a race somehow ?
Bandan
> certainly explain the failure - as multiple cpus could be running
> without the sipi trapoline in place.
>
> What does the log look like on a non-failure case?
>
> -Kevin
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 17:09 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 17:09 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert, kraxel,
Paolo Bonzini
"Kevin O'Connor" <kevin@koconnor.net> writes:
...
>
> Something is very odd here. When I run the above command (on an older
> AMD machine) I get:
>
> Found 128 cpu(s) max supported 128 cpu(s)
>
> That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> That is, during smp init, SeaBIOS expects QEMU to tell it how many
> cpus are active, and SeaBIOS waits until that many CPUs check in from
> its SIPI request before proceeding.
>
> I wonder if QEMU reported only 1 active cpu via that cmos register,
> but more were actually active. If that was the case, it could
I was daring enough to try this and I don't see the crash :)
diff --git a/src/fw/smp.c b/src/fw/smp.c
index a466ea6..a346d46 100644
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
void VISIBLE32FLAT
handle_smp(void)
{
+ dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
if (!CONFIG_QEMU)
return;
@@ -128,6 +129,8 @@ smp_setup(void)
// Wait for other CPUs to process the SIPI.
u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ while (cmos_smp_count == 1)
+ cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
So, the while loop results in a race somehow ?
Bandan
> certainly explain the failure - as multiple cpus could be running
> without the sipi trapoline in place.
>
> What does the log look like on a non-failure case?
>
> -Kevin
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:09 ` Bandan Das
@ 2015-03-11 17:32 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 17:32 UTC (permalink / raw)
To: Bandan Das
Cc: Dr. David Alan Gilbert, Paolo Bonzini, kraxel, Andrey Korolyov,
qemu-devel, kvm
On Wed, Mar 11, 2015 at 01:09:42PM -0400, Bandan Das wrote:
> "Kevin O'Connor" <kevin@koconnor.net> writes:
> ...
> >
> > Something is very odd here. When I run the above command (on an older
> > AMD machine) I get:
> >
> > Found 128 cpu(s) max supported 128 cpu(s)
> >
> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > its SIPI request before proceeding.
> >
> > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > but more were actually active. If that was the case, it could
>
> I was daring enough to try this and I don't see the crash :)
>
> diff --git a/src/fw/smp.c b/src/fw/smp.c
> index a466ea6..a346d46 100644
> --- a/src/fw/smp.c
> +++ b/src/fw/smp.c
> @@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
> void VISIBLE32FLAT
> handle_smp(void)
> {
> + dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
> if (!CONFIG_QEMU)
> return;
>
> @@ -128,6 +129,8 @@ smp_setup(void)
>
> // Wait for other CPUs to process the SIPI.
> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
> + while (cmos_smp_count == 1)
> + cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
That would loop forever if you had "-smp 1".
> while (cmos_smp_count != CountCPUs)
> asm volatile(
> // Release lock and allow other processors to use the stack.
>
> So, the while loop results in a race somehow ?
No, the problem is that loop doesn't run at all, and as a result the
other cpus end up running random code. SeaBIOS sets up an entry
vector for multiple cpus, wakes up the cpus, then tears down the entry
vector. If it tears down the entry vector before all the cpus have
run through it, then the other cpus can end up running random code.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 17:32 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 17:32 UTC (permalink / raw)
To: Bandan Das
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert, kraxel,
Paolo Bonzini
On Wed, Mar 11, 2015 at 01:09:42PM -0400, Bandan Das wrote:
> "Kevin O'Connor" <kevin@koconnor.net> writes:
> ...
> >
> > Something is very odd here. When I run the above command (on an older
> > AMD machine) I get:
> >
> > Found 128 cpu(s) max supported 128 cpu(s)
> >
> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
> > cpus are active, and SeaBIOS waits until that many CPUs check in from
> > its SIPI request before proceeding.
> >
> > I wonder if QEMU reported only 1 active cpu via that cmos register,
> > but more were actually active. If that was the case, it could
>
> I was daring enough to try this and I don't see the crash :)
>
> diff --git a/src/fw/smp.c b/src/fw/smp.c
> index a466ea6..a346d46 100644
> --- a/src/fw/smp.c
> +++ b/src/fw/smp.c
> @@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
> void VISIBLE32FLAT
> handle_smp(void)
> {
> + dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
> if (!CONFIG_QEMU)
> return;
>
> @@ -128,6 +129,8 @@ smp_setup(void)
>
> // Wait for other CPUs to process the SIPI.
> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
> + while (cmos_smp_count == 1)
> + cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
That would loop forever if you had "-smp 1".
> while (cmos_smp_count != CountCPUs)
> asm volatile(
> // Release lock and allow other processors to use the stack.
>
> So, the while loop results in a race somehow ?
No, the problem is that loop doesn't run at all, and as a result the
other cpus end up running random code. SeaBIOS sets up an entry
vector for multiple cpus, wakes up the cpus, then tears down the entry
vector. If it tears down the entry vector before all the cpus have
run through it, then the other cpus can end up running random code.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 16:52 ` Dr. David Alan Gilbert
@ 2015-03-11 17:37 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 17:37 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > So, I couldn't get this to fail on my older AMD machine at all with
> > the default SeaBIOS code. But, when I change the code with the patch
> > below, it failed right away.
[...]
> > And the failed debug output looks like:
> >
> > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > [...]
> > cmos_smp_count0=20
> > [...]
> > cmos_smp_count=1
> > cmos_smp_count2=1/20
> > Found 1 cpu(s) max supported 20 cpu(s)
> >
> > I'm going to check the assembly for a compiler error, but is it
> > possible QEMU is returning incorrect data in cmos index 0x5f?
I checked the SeaBIOS assembler and it looks sane. So, I think the
question is, why is QEMU sometimes returning a 0 instead of 127 from
cmos 0x5f.
> > David, any chance you can recompile seabios and double check your
> > output?
>
> Done;
>
> =========== Working ===========
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
[...]
> cmos_smp_count0=20
> cmos_smp_count=20
> cmos_smp_count2=20/20
> Found 20 cpu(s) max supported 20 cpu(s)
[...]
> =========== Broken ===========
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
[...]
> cmos_smp_count0=20
> cmos_smp_count=1
> cmos_smp_count2=1/20
> Found 1 cpu(s) max supported 20 cpu(s)
That's the same pattern I see.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 17:37 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 17:37 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > So, I couldn't get this to fail on my older AMD machine at all with
> > the default SeaBIOS code. But, when I change the code with the patch
> > below, it failed right away.
[...]
> > And the failed debug output looks like:
> >
> > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > [...]
> > cmos_smp_count0=20
> > [...]
> > cmos_smp_count=1
> > cmos_smp_count2=1/20
> > Found 1 cpu(s) max supported 20 cpu(s)
> >
> > I'm going to check the assembly for a compiler error, but is it
> > possible QEMU is returning incorrect data in cmos index 0x5f?
I checked the SeaBIOS assembler and it looks sane. So, I think the
question is, why is QEMU sometimes returning a 0 instead of 127 from
cmos 0x5f.
> > David, any chance you can recompile seabios and double check your
> > output?
>
> Done;
>
> =========== Working ===========
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
[...]
> cmos_smp_count0=20
> cmos_smp_count=20
> cmos_smp_count2=20/20
> Found 20 cpu(s) max supported 20 cpu(s)
[...]
> =========== Broken ===========
> SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
[...]
> cmos_smp_count0=20
> cmos_smp_count=1
> cmos_smp_count2=1/20
> Found 1 cpu(s) max supported 20 cpu(s)
That's the same pattern I see.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:37 ` Kevin O'Connor
@ 2015-03-11 17:41 ` Paolo Bonzini
-1 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-11 17:41 UTC (permalink / raw)
To: Kevin O'Connor, Dr. David Alan Gilbert
Cc: Bandan Das, kraxel, Andrey Korolyov, qemu-devel, kvm
On 11/03/2015 18:37, Kevin O'Connor wrote:
> > I'm going to check the assembly for a compiler error, but is it
> > possible QEMU is returning incorrect data in cmos index 0x5f?
>
> I checked the SeaBIOS assembler and it looks sane. So, I think the
> question is, why is QEMU sometimes returning a 0 instead of 127 from
> cmos 0x5f.
Dave, can you get a KVM trace (trace-cmd record -e kvm/*
qemu-system-x86_64 ...) and send it to me privately?
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 17:41 ` Paolo Bonzini
0 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-03-11 17:41 UTC (permalink / raw)
To: Kevin O'Connor, Dr. David Alan Gilbert
Cc: Bandan Das, Andrey Korolyov, kraxel, kvm, qemu-devel
On 11/03/2015 18:37, Kevin O'Connor wrote:
> > I'm going to check the assembly for a compiler error, but is it
> > possible QEMU is returning incorrect data in cmos index 0x5f?
>
> I checked the SeaBIOS assembler and it looks sane. So, I think the
> question is, why is QEMU sometimes returning a 0 instead of 127 from
> cmos 0x5f.
Dave, can you get a KVM trace (trace-cmd record -e kvm/*
qemu-system-x86_64 ...) and send it to me privately?
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:37 ` Kevin O'Connor
@ 2015-03-11 17:59 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 17:59 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > So, I couldn't get this to fail on my older AMD machine at all with
> > > the default SeaBIOS code. But, when I change the code with the patch
> > > below, it failed right away.
> [...]
> > > And the failed debug output looks like:
> > >
> > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > > [...]
> > > cmos_smp_count0=20
> > > [...]
> > > cmos_smp_count=1
> > > cmos_smp_count2=1/20
> > > Found 1 cpu(s) max supported 20 cpu(s)
> > >
> > > I'm going to check the assembly for a compiler error, but is it
> > > possible QEMU is returning incorrect data in cmos index 0x5f?
>
> I checked the SeaBIOS assembler and it looks sane. So, I think the
> question is, why is QEMU sometimes returning a 0 instead of 127 from
> cmos 0x5f.
My reading of the logs I've just created is that qemu doesn't think
it's ever being asked to read 5f in the failed case:
good:
pc_cmos_init 5f setting smp_cpus=20
cmos: read index=0x0f val=0x00
cmos: read index=0x34 val=0x00
cmos: read index=0x35 val=0x3f
cmos: read index=0x38 val=0x30
cmos: read index=0x3d val=0x12
cmos: read index=0x38 val=0x30
cmos: read index=0x0b val=0x02
cmos: read index=0x0d val=0x80
cmos: read index=0x5f val=0x13 Yeh!
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
bad:
pc_cmos_init 5f setting smp_cpus=20
cmos: read index=0x0f val=0x00
cmos: read index=0x34 val=0x00
cmos: read index=0x35 val=0x3f
cmos: read index=0x38 val=0x30
cmos: read index=0x3d val=0x12
cmos: read index=0x38 val=0x30
cmos: read index=0x0b val=0x02
cmos: read index=0x0d val=0x80 Oh!
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
Dave
>
> > > David, any chance you can recompile seabios and double check your
> > > output?
> >
> > Done;
> >
> > =========== Working ===========
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
> [...]
> > cmos_smp_count0=20
> > cmos_smp_count=20
> > cmos_smp_count2=20/20
> > Found 20 cpu(s) max supported 20 cpu(s)
> [...]
> > =========== Broken ===========
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
> [...]
> > cmos_smp_count0=20
> > cmos_smp_count=1
> > cmos_smp_count2=1/20
> > Found 1 cpu(s) max supported 20 cpu(s)
>
> That's the same pattern I see.
>
> -Kevin
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 17:59 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 17:59 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > So, I couldn't get this to fail on my older AMD machine at all with
> > > the default SeaBIOS code. But, when I change the code with the patch
> > > below, it failed right away.
> [...]
> > > And the failed debug output looks like:
> > >
> > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > > [...]
> > > cmos_smp_count0=20
> > > [...]
> > > cmos_smp_count=1
> > > cmos_smp_count2=1/20
> > > Found 1 cpu(s) max supported 20 cpu(s)
> > >
> > > I'm going to check the assembly for a compiler error, but is it
> > > possible QEMU is returning incorrect data in cmos index 0x5f?
>
> I checked the SeaBIOS assembler and it looks sane. So, I think the
> question is, why is QEMU sometimes returning a 0 instead of 127 from
> cmos 0x5f.
My reading of the logs I've just created is that qemu doesn't think
it's ever being asked to read 5f in the failed case:
good:
pc_cmos_init 5f setting smp_cpus=20
cmos: read index=0x0f val=0x00
cmos: read index=0x34 val=0x00
cmos: read index=0x35 val=0x3f
cmos: read index=0x38 val=0x30
cmos: read index=0x3d val=0x12
cmos: read index=0x38 val=0x30
cmos: read index=0x0b val=0x02
cmos: read index=0x0d val=0x80
cmos: read index=0x5f val=0x13 Yeh!
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
bad:
pc_cmos_init 5f setting smp_cpus=20
cmos: read index=0x0f val=0x00
cmos: read index=0x34 val=0x00
cmos: read index=0x35 val=0x3f
cmos: read index=0x38 val=0x30
cmos: read index=0x3d val=0x12
cmos: read index=0x38 val=0x30
cmos: read index=0x0b val=0x02
cmos: read index=0x0d val=0x80 Oh!
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
cmos: read index=0x0f val=0x00
Dave
>
> > > David, any chance you can recompile seabios and double check your
> > > output?
> >
> > Done;
> >
> > =========== Working ===========
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
> [...]
> > cmos_smp_count0=20
> > cmos_smp_count=20
> > cmos_smp_count2=20/20
> > Found 20 cpu(s) max supported 20 cpu(s)
> [...]
> > =========== Broken ===========
> > SeaBIOS (version rel-1.8.0-0-g4c59f5d-dirty-20150311_164408-dgilbert-t530)
> [...]
> > cmos_smp_count0=20
> > cmos_smp_count=1
> > cmos_smp_count2=1/20
> > Found 1 cpu(s) max supported 20 cpu(s)
>
> That's the same pattern I see.
>
> -Kevin
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:32 ` Kevin O'Connor
@ 2015-03-11 18:01 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 18:01 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Dr. David Alan Gilbert, Paolo Bonzini, kraxel, Andrey Korolyov,
qemu-devel, kvm
"Kevin O'Connor" <kevin@koconnor.net> writes:
> On Wed, Mar 11, 2015 at 01:09:42PM -0400, Bandan Das wrote:
>> "Kevin O'Connor" <kevin@koconnor.net> writes:
>> ...
>> >
>> > Something is very odd here. When I run the above command (on an older
>> > AMD machine) I get:
>> >
>> > Found 128 cpu(s) max supported 128 cpu(s)
>> >
>> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
>> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
>> > cpus are active, and SeaBIOS waits until that many CPUs check in from
>> > its SIPI request before proceeding.
>> >
>> > I wonder if QEMU reported only 1 active cpu via that cmos register,
>> > but more were actually active. If that was the case, it could
>>
>> I was daring enough to try this and I don't see the crash :)
>>
>> diff --git a/src/fw/smp.c b/src/fw/smp.c
>> index a466ea6..a346d46 100644
>> --- a/src/fw/smp.c
>> +++ b/src/fw/smp.c
>> @@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
>> void VISIBLE32FLAT
>> handle_smp(void)
>> {
>> + dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
>> if (!CONFIG_QEMU)
>> return;
>>
>> @@ -128,6 +129,8 @@ smp_setup(void)
>>
>> // Wait for other CPUs to process the SIPI.
>> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
>> + while (cmos_smp_count == 1)
>> + cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
>
> That would loop forever if you had "-smp 1".
Sorry, I should have been more clear. What I meant is if I run
with "-smp 128" (from Dave's original reproducer), sticking this
while loop avoids the crash. So, the rtc_read eventually returns the
right number (127), as the above while loop keeps polling.
Bandan
>> while (cmos_smp_count != CountCPUs)
>> asm volatile(
>> // Release lock and allow other processors to use the stack.
>>
>> So, the while loop results in a race somehow ?
>
> No, the problem is that loop doesn't run at all, and as a result the
> other cpus end up running random code. SeaBIOS sets up an entry
> vector for multiple cpus, wakes up the cpus, then tears down the entry
> vector. If it tears down the entry vector before all the cpus have
> run through it, then the other cpus can end up running random code.
>
> -Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 18:01 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 18:01 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert, kraxel,
Paolo Bonzini
"Kevin O'Connor" <kevin@koconnor.net> writes:
> On Wed, Mar 11, 2015 at 01:09:42PM -0400, Bandan Das wrote:
>> "Kevin O'Connor" <kevin@koconnor.net> writes:
>> ...
>> >
>> > Something is very odd here. When I run the above command (on an older
>> > AMD machine) I get:
>> >
>> > Found 128 cpu(s) max supported 128 cpu(s)
>> >
>> > That first value (1 vs 128) comes from QEMU (via cmos index 0x5f).
>> > That is, during smp init, SeaBIOS expects QEMU to tell it how many
>> > cpus are active, and SeaBIOS waits until that many CPUs check in from
>> > its SIPI request before proceeding.
>> >
>> > I wonder if QEMU reported only 1 active cpu via that cmos register,
>> > but more were actually active. If that was the case, it could
>>
>> I was daring enough to try this and I don't see the crash :)
>>
>> diff --git a/src/fw/smp.c b/src/fw/smp.c
>> index a466ea6..a346d46 100644
>> --- a/src/fw/smp.c
>> +++ b/src/fw/smp.c
>> @@ -49,6 +49,7 @@ int apic_id_is_present(u8 apic_id)
>> void VISIBLE32FLAT
>> handle_smp(void)
>> {
>> + dprintf(DEBUG_HDL_smp, "Calling handle_smp\n");
>> if (!CONFIG_QEMU)
>> return;
>>
>> @@ -128,6 +129,8 @@ smp_setup(void)
>>
>> // Wait for other CPUs to process the SIPI.
>> u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
>> + while (cmos_smp_count == 1)
>> + cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
>
> That would loop forever if you had "-smp 1".
Sorry, I should have been more clear. What I meant is if I run
with "-smp 128" (from Dave's original reproducer), sticking this
while loop avoids the crash. So, the rtc_read eventually returns the
right number (127), as the above while loop keeps polling.
Bandan
>> while (cmos_smp_count != CountCPUs)
>> asm volatile(
>> // Release lock and allow other processors to use the stack.
>>
>> So, the while loop results in a race somehow ?
>
> No, the problem is that loop doesn't run at all, and as a result the
> other cpus end up running random code. SeaBIOS sets up an entry
> vector for multiple cpus, wakes up the cpus, then tears down the entry
> vector. If it tears down the entry vector before all the cpus have
> run through it, then the other cpus can end up running random code.
>
> -Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:59 ` Dr. David Alan Gilbert
@ 2015-03-11 18:24 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 18:24 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Andrey Korolyov, kvm, qemu-devel, kraxel,
Paolo Bonzini
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
>> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> > > So, I couldn't get this to fail on my older AMD machine at all with
>> > > the default SeaBIOS code. But, when I change the code with the patch
>> > > below, it failed right away.
>> [...]
>> > > And the failed debug output looks like:
>> > >
>> > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
>> > > [...]
>> > > cmos_smp_count0=20
>> > > [...]
>> > > cmos_smp_count=1
>> > > cmos_smp_count2=1/20
>> > > Found 1 cpu(s) max supported 20 cpu(s)
>> > >
>> > > I'm going to check the assembly for a compiler error, but is it
>> > > possible QEMU is returning incorrect data in cmos index 0x5f?
>>
>> I checked the SeaBIOS assembler and it looks sane. So, I think the
>> question is, why is QEMU sometimes returning a 0 instead of 127 from
>> cmos 0x5f.
>
> My reading of the logs I've just created is that qemu doesn't think
> it's ever being asked to read 5f in the failed case:
>
> good:
>
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80
> cmos: read index=0x5f val=0x13 Yeh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
>
> bad:
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80 Oh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
rtc_read in hw/rt.c is:
u8
rtc_read(u8 index)
{
index |= NMI_DISABLE_BIT;
outb(index, PORT_CMOS_INDEX);
return inb(PORT_CMOS_DATA);
}
Is it possible that the read would happen before the index has been committed ?
> Dave
>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 18:24 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 18:24 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Kevin O'Connor, kraxel,
Paolo Bonzini
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
>> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> > > So, I couldn't get this to fail on my older AMD machine at all with
>> > > the default SeaBIOS code. But, when I change the code with the patch
>> > > below, it failed right away.
>> [...]
>> > > And the failed debug output looks like:
>> > >
>> > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
>> > > [...]
>> > > cmos_smp_count0=20
>> > > [...]
>> > > cmos_smp_count=1
>> > > cmos_smp_count2=1/20
>> > > Found 1 cpu(s) max supported 20 cpu(s)
>> > >
>> > > I'm going to check the assembly for a compiler error, but is it
>> > > possible QEMU is returning incorrect data in cmos index 0x5f?
>>
>> I checked the SeaBIOS assembler and it looks sane. So, I think the
>> question is, why is QEMU sometimes returning a 0 instead of 127 from
>> cmos 0x5f.
>
> My reading of the logs I've just created is that qemu doesn't think
> it's ever being asked to read 5f in the failed case:
>
> good:
>
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80
> cmos: read index=0x5f val=0x13 Yeh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
>
> bad:
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80 Oh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
rtc_read in hw/rt.c is:
u8
rtc_read(u8 index)
{
index |= NMI_DISABLE_BIT;
outb(index, PORT_CMOS_INDEX);
return inb(PORT_CMOS_DATA);
}
Is it possible that the read would happen before the index has been committed ?
> Dave
>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 17:59 ` Dr. David Alan Gilbert
@ 2015-03-11 18:40 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 18:40 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
On Wed, Mar 11, 2015 at 05:59:04PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> > > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > > So, I couldn't get this to fail on my older AMD machine at all with
> > > > the default SeaBIOS code. But, when I change the code with the patch
> > > > below, it failed right away.
> > [...]
> > > > And the failed debug output looks like:
> > > >
> > > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > > > [...]
> > > > cmos_smp_count0=20
> > > > [...]
> > > > cmos_smp_count=1
> > > > cmos_smp_count2=1/20
> > > > Found 1 cpu(s) max supported 20 cpu(s)
> > > >
> > > > I'm going to check the assembly for a compiler error, but is it
> > > > possible QEMU is returning incorrect data in cmos index 0x5f?
> >
> > I checked the SeaBIOS assembler and it looks sane. So, I think the
> > question is, why is QEMU sometimes returning a 0 instead of 127 from
> > cmos 0x5f.
>
> My reading of the logs I've just created is that qemu doesn't think
> it's ever being asked to read 5f in the failed case:
>
> good:
>
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80
> cmos: read index=0x5f val=0x13 Yeh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
>
> bad:
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80 Oh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
For what it's worth, I can't seem to trigger the problem if I move the
cmos read above the SIPI/LAPIC code (see patch below).
I used this command line:
while true; do (sleep 5; echo -e '\001cq\n')| ../qemu/qemu-git/x86_64-softmmu/qemu-system-x86_64 -chardev file,path=foo.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga -L test 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
This is on an "AMD Phenom(tm) II X6 1090T Processor" machine.
-Kevin
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -107,6 +107,8 @@ smp_setup(void)
| (((u32)entry_smp - BUILD_BIOS_ADDR) << 8));
*(u64*)BUILD_AP_BOOT_ADDR = new;
+ u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+
// enable local APIC
u32 val = readl(APIC_SVR);
writel(APIC_SVR, val | APIC_ENABLED);
@@ -127,7 +129,7 @@ smp_setup(void)
writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector);
// Wait for other CPUs to process the SIPI.
- u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
@@ -140,6 +142,8 @@ smp_setup(void)
: "+m" (SMPLock), "+m" (SMPStack)
: : "cc", "memory");
yield();
+ dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
+ , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
// Restore memory.
*(u64*)BUILD_AP_BOOT_ADDR = old;
diff --git a/src/post.c b/src/post.c
index 9ea5620..dc11c72 100644
--- a/src/post.c
+++ b/src/post.c
@@ -170,6 +170,7 @@ platform_hardware_setup(void)
clock_setup();
// Platform specific setup
+ dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
qemu_platform_setup();
coreboot_platform_setup();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 18:40 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 18:40 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 05:59:04PM +0000, Dr. David Alan Gilbert wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
> > On Wed, Mar 11, 2015 at 04:52:03PM +0000, Dr. David Alan Gilbert wrote:
> > > * Kevin O'Connor (kevin@koconnor.net) wrote:
> > > > So, I couldn't get this to fail on my older AMD machine at all with
> > > > the default SeaBIOS code. But, when I change the code with the patch
> > > > below, it failed right away.
> > [...]
> > > > And the failed debug output looks like:
> > > >
> > > > SeaBIOS (version rel-1.8.0-7-gd23eba6-dirty-20150311_121819-morn.localdomain)
> > > > [...]
> > > > cmos_smp_count0=20
> > > > [...]
> > > > cmos_smp_count=1
> > > > cmos_smp_count2=1/20
> > > > Found 1 cpu(s) max supported 20 cpu(s)
> > > >
> > > > I'm going to check the assembly for a compiler error, but is it
> > > > possible QEMU is returning incorrect data in cmos index 0x5f?
> >
> > I checked the SeaBIOS assembler and it looks sane. So, I think the
> > question is, why is QEMU sometimes returning a 0 instead of 127 from
> > cmos 0x5f.
>
> My reading of the logs I've just created is that qemu doesn't think
> it's ever being asked to read 5f in the failed case:
>
> good:
>
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80
> cmos: read index=0x5f val=0x13 Yeh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
>
> bad:
> pc_cmos_init 5f setting smp_cpus=20
> cmos: read index=0x0f val=0x00
> cmos: read index=0x34 val=0x00
> cmos: read index=0x35 val=0x3f
> cmos: read index=0x38 val=0x30
> cmos: read index=0x3d val=0x12
> cmos: read index=0x38 val=0x30
> cmos: read index=0x0b val=0x02
> cmos: read index=0x0d val=0x80 Oh!
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
> cmos: read index=0x0f val=0x00
For what it's worth, I can't seem to trigger the problem if I move the
cmos read above the SIPI/LAPIC code (see patch below).
I used this command line:
while true; do (sleep 5; echo -e '\001cq\n')| ../qemu/qemu-git/x86_64-softmmu/qemu-system-x86_64 -chardev file,path=foo.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga -L test 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
This is on an "AMD Phenom(tm) II X6 1090T Processor" machine.
-Kevin
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -107,6 +107,8 @@ smp_setup(void)
| (((u32)entry_smp - BUILD_BIOS_ADDR) << 8));
*(u64*)BUILD_AP_BOOT_ADDR = new;
+ u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+
// enable local APIC
u32 val = readl(APIC_SVR);
writel(APIC_SVR, val | APIC_ENABLED);
@@ -127,7 +129,7 @@ smp_setup(void)
writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector);
// Wait for other CPUs to process the SIPI.
- u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+ dprintf(1, "cmos_smp_count=%d\n", cmos_smp_count);
while (cmos_smp_count != CountCPUs)
asm volatile(
// Release lock and allow other processors to use the stack.
@@ -140,6 +142,8 @@ smp_setup(void)
: "+m" (SMPLock), "+m" (SMPStack)
: : "cc", "memory");
yield();
+ dprintf(1, "cmos_smp_count2=%d/%d\n", cmos_smp_count
+ , rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
// Restore memory.
*(u64*)BUILD_AP_BOOT_ADDR = old;
diff --git a/src/post.c b/src/post.c
index 9ea5620..dc11c72 100644
--- a/src/post.c
+++ b/src/post.c
@@ -170,6 +170,7 @@ platform_hardware_setup(void)
clock_setup();
// Platform specific setup
+ dprintf(1, "cmos_smp_count0=%d\n", rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
qemu_platform_setup();
coreboot_platform_setup();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 18:40 ` Kevin O'Connor
@ 2015-03-11 18:45 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 18:45 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> For what it's worth, I can't seem to trigger the problem if I move the
> cmos read above the SIPI/LAPIC code (see patch below).
Ugh!
That's a seabios bug. Main processor modifies the rtc index
(rtc_read()) while APs try to clear the NMI bit by modifying the rtc
index (romlayout.S:transition32).
I'll put together a fix.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 18:45 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 18:45 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> For what it's worth, I can't seem to trigger the problem if I move the
> cmos read above the SIPI/LAPIC code (see patch below).
Ugh!
That's a seabios bug. Main processor modifies the rtc index
(rtc_read()) while APs try to clear the NMI bit by modifying the rtc
index (romlayout.S:transition32).
I'll put together a fix.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 18:45 ` Kevin O'Connor
@ 2015-03-11 19:19 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 19:19 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> > For what it's worth, I can't seem to trigger the problem if I move the
> > cmos read above the SIPI/LAPIC code (see patch below).
>
> Ugh!
>
> That's a seabios bug. Main processor modifies the rtc index
> (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> index (romlayout.S:transition32).
>
> I'll put together a fix.
The seabios patch below resolves the issue for me.
-Kevin
--- a/src/romlayout.S
+++ b/src/romlayout.S
@@ -22,7 +22,8 @@
// %edx = return location (in 32bit mode)
// Clobbers: ecx, flags, segment registers, cr0, idt/gdt
DECLFUNC transition32
-transition32_for_smi:
+transition32_nmi_off:
+ // transition32 when NMI and A20 are already initialized
movl %eax, %ecx
jmp 1f
transition32:
@@ -205,7 +206,7 @@ __farcall16:
entry_smi:
// Transition to 32bit mode.
movl $1f + BUILD_BIOS_ADDR, %edx
- jmp transition32_for_smi
+ jmp transition32_nmi_off
.code32
1: movl $BUILD_SMM_ADDR + 0x8000, %esp
calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
@@ -216,8 +217,10 @@ entry_smi:
DECLFUNC entry_smp
entry_smp:
// Transition to 32bit mode.
+ cli
+ cld
movl $2f + BUILD_BIOS_ADDR, %edx
- jmp transition32
+ jmp transition32_nmi_off
.code32
// Acquire lock and take ownership of shared stack
1: rep ; nop
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 19:19 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-11 19:19 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> > For what it's worth, I can't seem to trigger the problem if I move the
> > cmos read above the SIPI/LAPIC code (see patch below).
>
> Ugh!
>
> That's a seabios bug. Main processor modifies the rtc index
> (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> index (romlayout.S:transition32).
>
> I'll put together a fix.
The seabios patch below resolves the issue for me.
-Kevin
--- a/src/romlayout.S
+++ b/src/romlayout.S
@@ -22,7 +22,8 @@
// %edx = return location (in 32bit mode)
// Clobbers: ecx, flags, segment registers, cr0, idt/gdt
DECLFUNC transition32
-transition32_for_smi:
+transition32_nmi_off:
+ // transition32 when NMI and A20 are already initialized
movl %eax, %ecx
jmp 1f
transition32:
@@ -205,7 +206,7 @@ __farcall16:
entry_smi:
// Transition to 32bit mode.
movl $1f + BUILD_BIOS_ADDR, %edx
- jmp transition32_for_smi
+ jmp transition32_nmi_off
.code32
1: movl $BUILD_SMM_ADDR + 0x8000, %esp
calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
@@ -216,8 +217,10 @@ entry_smi:
DECLFUNC entry_smp
entry_smp:
// Transition to 32bit mode.
+ cli
+ cld
movl $2f + BUILD_BIOS_ADDR, %edx
- jmp transition32
+ jmp transition32_nmi_off
.code32
// Acquire lock and take ownership of shared stack
1: rep ; nop
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 19:19 ` Kevin O'Connor
@ 2015-03-11 19:33 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 19:33 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Bandan Das, Paolo Bonzini, kraxel, Andrey Korolyov, qemu-devel, kvm
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> > > For what it's worth, I can't seem to trigger the problem if I move the
> > > cmos read above the SIPI/LAPIC code (see patch below).
> >
> > Ugh!
> >
> > That's a seabios bug. Main processor modifies the rtc index
> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> > index (romlayout.S:transition32).
> >
> > I'll put together a fix.
>
> The seabios patch below resolves the issue for me.
Thanks! Looks good here.
Andrey, Paolo, Bandan: Does it fix it for you as well?
Dave
> -Kevin
>
>
> --- a/src/romlayout.S
> +++ b/src/romlayout.S
> @@ -22,7 +22,8 @@
> // %edx = return location (in 32bit mode)
> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> DECLFUNC transition32
> -transition32_for_smi:
> +transition32_nmi_off:
> + // transition32 when NMI and A20 are already initialized
> movl %eax, %ecx
> jmp 1f
> transition32:
> @@ -205,7 +206,7 @@ __farcall16:
> entry_smi:
> // Transition to 32bit mode.
> movl $1f + BUILD_BIOS_ADDR, %edx
> - jmp transition32_for_smi
> + jmp transition32_nmi_off
> .code32
> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> @@ -216,8 +217,10 @@ entry_smi:
> DECLFUNC entry_smp
> entry_smp:
> // Transition to 32bit mode.
> + cli
> + cld
> movl $2f + BUILD_BIOS_ADDR, %edx
> - jmp transition32
> + jmp transition32_nmi_off
> .code32
> // Acquire lock and take ownership of shared stack
> 1: rep ; nop
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 19:33 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 19:33 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Bandan Das, kraxel, Paolo Bonzini
* Kevin O'Connor (kevin@koconnor.net) wrote:
> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> > > For what it's worth, I can't seem to trigger the problem if I move the
> > > cmos read above the SIPI/LAPIC code (see patch below).
> >
> > Ugh!
> >
> > That's a seabios bug. Main processor modifies the rtc index
> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> > index (romlayout.S:transition32).
> >
> > I'll put together a fix.
>
> The seabios patch below resolves the issue for me.
Thanks! Looks good here.
Andrey, Paolo, Bandan: Does it fix it for you as well?
Dave
> -Kevin
>
>
> --- a/src/romlayout.S
> +++ b/src/romlayout.S
> @@ -22,7 +22,8 @@
> // %edx = return location (in 32bit mode)
> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> DECLFUNC transition32
> -transition32_for_smi:
> +transition32_nmi_off:
> + // transition32 when NMI and A20 are already initialized
> movl %eax, %ecx
> jmp 1f
> transition32:
> @@ -205,7 +206,7 @@ __farcall16:
> entry_smi:
> // Transition to 32bit mode.
> movl $1f + BUILD_BIOS_ADDR, %edx
> - jmp transition32_for_smi
> + jmp transition32_nmi_off
> .code32
> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> @@ -216,8 +217,10 @@ entry_smi:
> DECLFUNC entry_smp
> entry_smp:
> // Transition to 32bit mode.
> + cli
> + cld
> movl $2f + BUILD_BIOS_ADDR, %edx
> - jmp transition32
> + jmp transition32_nmi_off
> .code32
> // Acquire lock and take ownership of shared stack
> 1: rep ; nop
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 19:33 ` Dr. David Alan Gilbert
@ 2015-03-11 19:47 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 19:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Paolo Bonzini, kraxel, Andrey Korolyov,
qemu-devel, kvm
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> > > For what it's worth, I can't seem to trigger the problem if I move the
>> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >
>> > Ugh!
>> >
>> > That's a seabios bug. Main processor modifies the rtc index
>> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> > index (romlayout.S:transition32).
>> >
>> > I'll put together a fix.
>>
>> The seabios patch below resolves the issue for me.
>
> Thanks! Looks good here.
>
> Andrey, Paolo, Bandan: Does it fix it for you as well?
Works for me too, thanks Kevin!
Bandan
> Dave
>
>> -Kevin
>>
>>
>> --- a/src/romlayout.S
>> +++ b/src/romlayout.S
>> @@ -22,7 +22,8 @@
>> // %edx = return location (in 32bit mode)
>> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> DECLFUNC transition32
>> -transition32_for_smi:
>> +transition32_nmi_off:
>> + // transition32 when NMI and A20 are already initialized
>> movl %eax, %ecx
>> jmp 1f
>> transition32:
>> @@ -205,7 +206,7 @@ __farcall16:
>> entry_smi:
>> // Transition to 32bit mode.
>> movl $1f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32_for_smi
>> + jmp transition32_nmi_off
>> .code32
>> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> @@ -216,8 +217,10 @@ entry_smi:
>> DECLFUNC entry_smp
>> entry_smp:
>> // Transition to 32bit mode.
>> + cli
>> + cld
>> movl $2f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32
>> + jmp transition32_nmi_off
>> .code32
>> // Acquire lock and take ownership of shared stack
>> 1: rep ; nop
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 19:47 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-11 19:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Andrey Korolyov, kvm, qemu-devel, Kevin O'Connor, kraxel,
Paolo Bonzini
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> > > For what it's worth, I can't seem to trigger the problem if I move the
>> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >
>> > Ugh!
>> >
>> > That's a seabios bug. Main processor modifies the rtc index
>> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> > index (romlayout.S:transition32).
>> >
>> > I'll put together a fix.
>>
>> The seabios patch below resolves the issue for me.
>
> Thanks! Looks good here.
>
> Andrey, Paolo, Bandan: Does it fix it for you as well?
Works for me too, thanks Kevin!
Bandan
> Dave
>
>> -Kevin
>>
>>
>> --- a/src/romlayout.S
>> +++ b/src/romlayout.S
>> @@ -22,7 +22,8 @@
>> // %edx = return location (in 32bit mode)
>> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> DECLFUNC transition32
>> -transition32_for_smi:
>> +transition32_nmi_off:
>> + // transition32 when NMI and A20 are already initialized
>> movl %eax, %ecx
>> jmp 1f
>> transition32:
>> @@ -205,7 +206,7 @@ __farcall16:
>> entry_smi:
>> // Transition to 32bit mode.
>> movl $1f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32_for_smi
>> + jmp transition32_nmi_off
>> .code32
>> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> @@ -216,8 +217,10 @@ entry_smi:
>> DECLFUNC entry_smp
>> entry_smp:
>> // Transition to 32bit mode.
>> + cli
>> + cld
>> movl $2f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32
>> + jmp transition32_nmi_off
>> .code32
>> // Acquire lock and take ownership of shared stack
>> 1: rep ; nop
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 19:33 ` Dr. David Alan Gilbert
@ 2015-03-11 19:47 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-11 19:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> > > For what it's worth, I can't seem to trigger the problem if I move the
>> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >
>> > Ugh!
>> >
>> > That's a seabios bug. Main processor modifies the rtc index
>> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> > index (romlayout.S:transition32).
>> >
>> > I'll put together a fix.
>>
>> The seabios patch below resolves the issue for me.
>
> Thanks! Looks good here.
>
> Andrey, Paolo, Bandan: Does it fix it for you as well?
>
Thanks Kevin, Dave,
I`m afraid that I`m hitting something different not only because
different suberror code but also because of mine version of seabios -
I am using 1.7.5 and corresponding code in the proposed patch looks
different - there is no smp-related code patch is about of. Those
mentioned devices went to production successfully and I`m afraid I
cannot afford playing on them anymore, even if I re-trigger the issue
with patched 1.8.1-rc, there is no way to switch to a different kernel
and retest due to specific conditions of this production suite. I`ve
ordered a pair of new shoes^W 2620v2-s which should arrive to me next
Monday, so I`ll be able to test a) against 1.8.0-release, b) against
patched bios code, c) reproduce initial error on master/3.19 (may be
I`ll take them before weekend by going into this computer shop in
person). Until then, I have a very deep feeling that mine issue is not
there :) Also I became very curious on how a lack of IDT feature may
completely eliminate the issue appearance for me, the only possible
explanation is a clock-related race which is kinda stupid suggestion
and unlikely to exist in nature.
Thanks again for everyone for throughout testing and ideas!
>
>> -Kevin
>>
>>
>> --- a/src/romlayout.S
>> +++ b/src/romlayout.S
>> @@ -22,7 +22,8 @@
>> // %edx = return location (in 32bit mode)
>> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> DECLFUNC transition32
>> -transition32_for_smi:
>> +transition32_nmi_off:
>> + // transition32 when NMI and A20 are already initialized
>> movl %eax, %ecx
>> jmp 1f
>> transition32:
>> @@ -205,7 +206,7 @@ __farcall16:
>> entry_smi:
>> // Transition to 32bit mode.
>> movl $1f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32_for_smi
>> + jmp transition32_nmi_off
>> .code32
>> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> @@ -216,8 +217,10 @@ entry_smi:
>> DECLFUNC entry_smp
>> entry_smp:
>> // Transition to 32bit mode.
>> + cli
>> + cld
>> movl $2f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32
>> + jmp transition32_nmi_off
>> .code32
>> // Acquire lock and take ownership of shared stack
>> 1: rep ; nop
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 19:47 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-11 19:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Kevin O'Connor (kevin@koconnor.net) wrote:
>> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> > > For what it's worth, I can't seem to trigger the problem if I move the
>> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >
>> > Ugh!
>> >
>> > That's a seabios bug. Main processor modifies the rtc index
>> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> > index (romlayout.S:transition32).
>> >
>> > I'll put together a fix.
>>
>> The seabios patch below resolves the issue for me.
>
> Thanks! Looks good here.
>
> Andrey, Paolo, Bandan: Does it fix it for you as well?
>
Thanks Kevin, Dave,
I`m afraid that I`m hitting something different not only because
different suberror code but also because of mine version of seabios -
I am using 1.7.5 and corresponding code in the proposed patch looks
different - there is no smp-related code patch is about of. Those
mentioned devices went to production successfully and I`m afraid I
cannot afford playing on them anymore, even if I re-trigger the issue
with patched 1.8.1-rc, there is no way to switch to a different kernel
and retest due to specific conditions of this production suite. I`ve
ordered a pair of new shoes^W 2620v2-s which should arrive to me next
Monday, so I`ll be able to test a) against 1.8.0-release, b) against
patched bios code, c) reproduce initial error on master/3.19 (may be
I`ll take them before weekend by going into this computer shop in
person). Until then, I have a very deep feeling that mine issue is not
there :) Also I became very curious on how a lack of IDT feature may
completely eliminate the issue appearance for me, the only possible
explanation is a clock-related race which is kinda stupid suggestion
and unlikely to exist in nature.
Thanks again for everyone for throughout testing and ideas!
>
>> -Kevin
>>
>>
>> --- a/src/romlayout.S
>> +++ b/src/romlayout.S
>> @@ -22,7 +22,8 @@
>> // %edx = return location (in 32bit mode)
>> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> DECLFUNC transition32
>> -transition32_for_smi:
>> +transition32_nmi_off:
>> + // transition32 when NMI and A20 are already initialized
>> movl %eax, %ecx
>> jmp 1f
>> transition32:
>> @@ -205,7 +206,7 @@ __farcall16:
>> entry_smi:
>> // Transition to 32bit mode.
>> movl $1f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32_for_smi
>> + jmp transition32_nmi_off
>> .code32
>> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> @@ -216,8 +217,10 @@ entry_smi:
>> DECLFUNC entry_smp
>> entry_smp:
>> // Transition to 32bit mode.
>> + cli
>> + cld
>> movl $2f + BUILD_BIOS_ADDR, %edx
>> - jmp transition32
>> + jmp transition32_nmi_off
>> .code32
>> // Acquire lock and take ownership of shared stack
>> 1: rep ; nop
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 19:47 ` Andrey Korolyov
@ 2015-03-11 19:59 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 19:59 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> >> > > For what it's worth, I can't seem to trigger the problem if I move the
> >> > > cmos read above the SIPI/LAPIC code (see patch below).
> >> >
> >> > Ugh!
> >> >
> >> > That's a seabios bug. Main processor modifies the rtc index
> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> >> > index (romlayout.S:transition32).
> >> >
> >> > I'll put together a fix.
> >>
> >> The seabios patch below resolves the issue for me.
> >
> > Thanks! Looks good here.
> >
> > Andrey, Paolo, Bandan: Does it fix it for you as well?
> >
>
> Thanks Kevin, Dave,
>
> I`m afraid that I`m hitting something different not only because
> different suberror code but also because of mine version of seabios -
> I am using 1.7.5 and corresponding code in the proposed patch looks
> different - there is no smp-related code patch is about of. Those
> mentioned devices went to production successfully and I`m afraid I
> cannot afford playing on them anymore, even if I re-trigger the issue
> with patched 1.8.1-rc, there is no way to switch to a different kernel
> and retest due to specific conditions of this production suite. I`ve
> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
was pretty simple. If you can suggest any flags I should add etc to the
test I'd be happy to give it a go.
Dave
> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
> patched bios code, c) reproduce initial error on master/3.19 (may be
> I`ll take them before weekend by going into this computer shop in
> person). Until then, I have a very deep feeling that mine issue is not
> there :) Also I became very curious on how a lack of IDT feature may
> completely eliminate the issue appearance for me, the only possible
> explanation is a clock-related race which is kinda stupid suggestion
> and unlikely to exist in nature.
>
> Thanks again for everyone for throughout testing and ideas!
>
> >
> >> -Kevin
> >>
> >>
> >> --- a/src/romlayout.S
> >> +++ b/src/romlayout.S
> >> @@ -22,7 +22,8 @@
> >> // %edx = return location (in 32bit mode)
> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> >> DECLFUNC transition32
> >> -transition32_for_smi:
> >> +transition32_nmi_off:
> >> + // transition32 when NMI and A20 are already initialized
> >> movl %eax, %ecx
> >> jmp 1f
> >> transition32:
> >> @@ -205,7 +206,7 @@ __farcall16:
> >> entry_smi:
> >> // Transition to 32bit mode.
> >> movl $1f + BUILD_BIOS_ADDR, %edx
> >> - jmp transition32_for_smi
> >> + jmp transition32_nmi_off
> >> .code32
> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> >> @@ -216,8 +217,10 @@ entry_smi:
> >> DECLFUNC entry_smp
> >> entry_smp:
> >> // Transition to 32bit mode.
> >> + cli
> >> + cld
> >> movl $2f + BUILD_BIOS_ADDR, %edx
> >> - jmp transition32
> >> + jmp transition32_nmi_off
> >> .code32
> >> // Acquire lock and take ownership of shared stack
> >> 1: rep ; nop
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 19:59 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-11 19:59 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> >> > > For what it's worth, I can't seem to trigger the problem if I move the
> >> > > cmos read above the SIPI/LAPIC code (see patch below).
> >> >
> >> > Ugh!
> >> >
> >> > That's a seabios bug. Main processor modifies the rtc index
> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> >> > index (romlayout.S:transition32).
> >> >
> >> > I'll put together a fix.
> >>
> >> The seabios patch below resolves the issue for me.
> >
> > Thanks! Looks good here.
> >
> > Andrey, Paolo, Bandan: Does it fix it for you as well?
> >
>
> Thanks Kevin, Dave,
>
> I`m afraid that I`m hitting something different not only because
> different suberror code but also because of mine version of seabios -
> I am using 1.7.5 and corresponding code in the proposed patch looks
> different - there is no smp-related code patch is about of. Those
> mentioned devices went to production successfully and I`m afraid I
> cannot afford playing on them anymore, even if I re-trigger the issue
> with patched 1.8.1-rc, there is no way to switch to a different kernel
> and retest due to specific conditions of this production suite. I`ve
> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
was pretty simple. If you can suggest any flags I should add etc to the
test I'd be happy to give it a go.
Dave
> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
> patched bios code, c) reproduce initial error on master/3.19 (may be
> I`ll take them before weekend by going into this computer shop in
> person). Until then, I have a very deep feeling that mine issue is not
> there :) Also I became very curious on how a lack of IDT feature may
> completely eliminate the issue appearance for me, the only possible
> explanation is a clock-related race which is kinda stupid suggestion
> and unlikely to exist in nature.
>
> Thanks again for everyone for throughout testing and ideas!
>
> >
> >> -Kevin
> >>
> >>
> >> --- a/src/romlayout.S
> >> +++ b/src/romlayout.S
> >> @@ -22,7 +22,8 @@
> >> // %edx = return location (in 32bit mode)
> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> >> DECLFUNC transition32
> >> -transition32_for_smi:
> >> +transition32_nmi_off:
> >> + // transition32 when NMI and A20 are already initialized
> >> movl %eax, %ecx
> >> jmp 1f
> >> transition32:
> >> @@ -205,7 +206,7 @@ __farcall16:
> >> entry_smi:
> >> // Transition to 32bit mode.
> >> movl $1f + BUILD_BIOS_ADDR, %edx
> >> - jmp transition32_for_smi
> >> + jmp transition32_nmi_off
> >> .code32
> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> >> @@ -216,8 +217,10 @@ entry_smi:
> >> DECLFUNC entry_smp
> >> entry_smp:
> >> // Transition to 32bit mode.
> >> + cli
> >> + cld
> >> movl $2f + BUILD_BIOS_ADDR, %edx
> >> - jmp transition32
> >> + jmp transition32_nmi_off
> >> .code32
> >> // Acquire lock and take ownership of shared stack
> >> 1: rep ; nop
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 19:59 ` Dr. David Alan Gilbert
@ 2015-03-11 20:09 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-11 20:09 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> >> > > For what it's worth, I can't seem to trigger the problem if I move the
>> >> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >> >
>> >> > Ugh!
>> >> >
>> >> > That's a seabios bug. Main processor modifies the rtc index
>> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> >> > index (romlayout.S:transition32).
>> >> >
>> >> > I'll put together a fix.
>> >>
>> >> The seabios patch below resolves the issue for me.
>> >
>> > Thanks! Looks good here.
>> >
>> > Andrey, Paolo, Bandan: Does it fix it for you as well?
>> >
>>
>> Thanks Kevin, Dave,
>>
>> I`m afraid that I`m hitting something different not only because
>> different suberror code but also because of mine version of seabios -
>> I am using 1.7.5 and corresponding code in the proposed patch looks
>> different - there is no smp-related code patch is about of. Those
>> mentioned devices went to production successfully and I`m afraid I
>> cannot afford playing on them anymore, even if I re-trigger the issue
>> with patched 1.8.1-rc, there is no way to switch to a different kernel
>> and retest due to specific conditions of this production suite. I`ve
>> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
>
> Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
> was pretty simple. If you can suggest any flags I should add etc to the
> test I'd be happy to give it a go.
>
> Dave
Here is mine launch string:
qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
-realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
-device sga -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
-device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
512,slots=31,maxmem=16384M -object
memory-backend-ram,id=mem0,size=512M -device
pc-dimm,id=dimm0,node=0,memdev=mem0
I omitted disk backend in this example, but there is a chance that my
problem is not reproducible without some calls made explicitly by a
bootloader (not sure what to say for mid-runtime failures).
>
>> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
>> patched bios code, c) reproduce initial error on master/3.19 (may be
>> I`ll take them before weekend by going into this computer shop in
>> person). Until then, I have a very deep feeling that mine issue is not
>> there :) Also I became very curious on how a lack of IDT feature may
>> completely eliminate the issue appearance for me, the only possible
>> explanation is a clock-related race which is kinda stupid suggestion
>> and unlikely to exist in nature.
>>
>> Thanks again for everyone for throughout testing and ideas!
>>
>> >
>> >> -Kevin
>> >>
>> >>
>> >> --- a/src/romlayout.S
>> >> +++ b/src/romlayout.S
>> >> @@ -22,7 +22,8 @@
>> >> // %edx = return location (in 32bit mode)
>> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> >> DECLFUNC transition32
>> >> -transition32_for_smi:
>> >> +transition32_nmi_off:
>> >> + // transition32 when NMI and A20 are already initialized
>> >> movl %eax, %ecx
>> >> jmp 1f
>> >> transition32:
>> >> @@ -205,7 +206,7 @@ __farcall16:
>> >> entry_smi:
>> >> // Transition to 32bit mode.
>> >> movl $1f + BUILD_BIOS_ADDR, %edx
>> >> - jmp transition32_for_smi
>> >> + jmp transition32_nmi_off
>> >> .code32
>> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> >> @@ -216,8 +217,10 @@ entry_smi:
>> >> DECLFUNC entry_smp
>> >> entry_smp:
>> >> // Transition to 32bit mode.
>> >> + cli
>> >> + cld
>> >> movl $2f + BUILD_BIOS_ADDR, %edx
>> >> - jmp transition32
>> >> + jmp transition32_nmi_off
>> >> .code32
>> >> // Acquire lock and take ownership of shared stack
>> >> 1: rep ; nop
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-11 20:09 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-11 20:09 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> >> > > For what it's worth, I can't seem to trigger the problem if I move the
>> >> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >> >
>> >> > Ugh!
>> >> >
>> >> > That's a seabios bug. Main processor modifies the rtc index
>> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> >> > index (romlayout.S:transition32).
>> >> >
>> >> > I'll put together a fix.
>> >>
>> >> The seabios patch below resolves the issue for me.
>> >
>> > Thanks! Looks good here.
>> >
>> > Andrey, Paolo, Bandan: Does it fix it for you as well?
>> >
>>
>> Thanks Kevin, Dave,
>>
>> I`m afraid that I`m hitting something different not only because
>> different suberror code but also because of mine version of seabios -
>> I am using 1.7.5 and corresponding code in the proposed patch looks
>> different - there is no smp-related code patch is about of. Those
>> mentioned devices went to production successfully and I`m afraid I
>> cannot afford playing on them anymore, even if I re-trigger the issue
>> with patched 1.8.1-rc, there is no way to switch to a different kernel
>> and retest due to specific conditions of this production suite. I`ve
>> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
>
> Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
> was pretty simple. If you can suggest any flags I should add etc to the
> test I'd be happy to give it a go.
>
> Dave
Here is mine launch string:
qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
-realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
-device sga -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
-device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
512,slots=31,maxmem=16384M -object
memory-backend-ram,id=mem0,size=512M -device
pc-dimm,id=dimm0,node=0,memdev=mem0
I omitted disk backend in this example, but there is a chance that my
problem is not reproducible without some calls made explicitly by a
bootloader (not sure what to say for mid-runtime failures).
>
>> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
>> patched bios code, c) reproduce initial error on master/3.19 (may be
>> I`ll take them before weekend by going into this computer shop in
>> person). Until then, I have a very deep feeling that mine issue is not
>> there :) Also I became very curious on how a lack of IDT feature may
>> completely eliminate the issue appearance for me, the only possible
>> explanation is a clock-related race which is kinda stupid suggestion
>> and unlikely to exist in nature.
>>
>> Thanks again for everyone for throughout testing and ideas!
>>
>> >
>> >> -Kevin
>> >>
>> >>
>> >> --- a/src/romlayout.S
>> >> +++ b/src/romlayout.S
>> >> @@ -22,7 +22,8 @@
>> >> // %edx = return location (in 32bit mode)
>> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> >> DECLFUNC transition32
>> >> -transition32_for_smi:
>> >> +transition32_nmi_off:
>> >> + // transition32 when NMI and A20 are already initialized
>> >> movl %eax, %ecx
>> >> jmp 1f
>> >> transition32:
>> >> @@ -205,7 +206,7 @@ __farcall16:
>> >> entry_smi:
>> >> // Transition to 32bit mode.
>> >> movl $1f + BUILD_BIOS_ADDR, %edx
>> >> - jmp transition32_for_smi
>> >> + jmp transition32_nmi_off
>> >> .code32
>> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> >> @@ -216,8 +217,10 @@ entry_smi:
>> >> DECLFUNC entry_smp
>> >> entry_smp:
>> >> // Transition to 32bit mode.
>> >> + cli
>> >> + cld
>> >> movl $2f + BUILD_BIOS_ADDR, %edx
>> >> - jmp transition32
>> >> + jmp transition32_nmi_off
>> >> .code32
>> >> // Acquire lock and take ownership of shared stack
>> >> 1: rep ; nop
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-11 20:09 ` Andrey Korolyov
@ 2015-03-12 9:59 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-12 9:59 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Andrey Korolyov (andrey@xdel.ru) wrote:
> >> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
> >> <dgilbert@redhat.com> wrote:
> >> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> >> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> >> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> >> >> > > For what it's worth, I can't seem to trigger the problem if I move the
> >> >> > > cmos read above the SIPI/LAPIC code (see patch below).
> >> >> >
> >> >> > Ugh!
> >> >> >
> >> >> > That's a seabios bug. Main processor modifies the rtc index
> >> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> >> >> > index (romlayout.S:transition32).
> >> >> >
> >> >> > I'll put together a fix.
> >> >>
> >> >> The seabios patch below resolves the issue for me.
> >> >
> >> > Thanks! Looks good here.
> >> >
> >> > Andrey, Paolo, Bandan: Does it fix it for you as well?
> >> >
> >>
> >> Thanks Kevin, Dave,
> >>
> >> I`m afraid that I`m hitting something different not only because
> >> different suberror code but also because of mine version of seabios -
> >> I am using 1.7.5 and corresponding code in the proposed patch looks
> >> different - there is no smp-related code patch is about of. Those
> >> mentioned devices went to production successfully and I`m afraid I
> >> cannot afford playing on them anymore, even if I re-trigger the issue
> >> with patched 1.8.1-rc, there is no way to switch to a different kernel
> >> and retest due to specific conditions of this production suite. I`ve
> >> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
> >
> > Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
> > was pretty simple. If you can suggest any flags I should add etc to the
> > test I'd be happy to give it a go.
> >
> > Dave
>
> Here is mine launch string:
>
> qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
> -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
> node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
> -device sga -rtc base=utc,driftfix=slew -global
> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
> -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
> 512,slots=31,maxmem=16384M -object
> memory-backend-ram,id=mem0,size=512M -device
> pc-dimm,id=dimm0,node=0,memdev=mem0
>
> I omitted disk backend in this example, but there is a chance that my
> problem is not reproducible without some calls made explicitly by a
> bootloader (not sure what to say for mid-runtime failures).
It seems to survive OK:
while true; do (sleep 1; echo -e '\001cc\n'; sleep 5; echo -e 'q\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -enable-kvm -name vmtest -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512 -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -device sga -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m 512,slots=31,maxmem=16384M -object memory-backend-ram,id=mem0,size=512M -device pc-dimm,id=dimm0,node=0,memdev=mem0 ~/pi.vfd 2>&1 | tee /tmp/qemu.op; grep "internal
error" /tmp/qemu.op -q && break; done
Dave
>
> >
> >> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
> >> patched bios code, c) reproduce initial error on master/3.19 (may be
> >> I`ll take them before weekend by going into this computer shop in
> >> person). Until then, I have a very deep feeling that mine issue is not
> >> there :) Also I became very curious on how a lack of IDT feature may
> >> completely eliminate the issue appearance for me, the only possible
> >> explanation is a clock-related race which is kinda stupid suggestion
> >> and unlikely to exist in nature.
> >>
> >> Thanks again for everyone for throughout testing and ideas!
> >>
> >> >
> >> >> -Kevin
> >> >>
> >> >>
> >> >> --- a/src/romlayout.S
> >> >> +++ b/src/romlayout.S
> >> >> @@ -22,7 +22,8 @@
> >> >> // %edx = return location (in 32bit mode)
> >> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> >> >> DECLFUNC transition32
> >> >> -transition32_for_smi:
> >> >> +transition32_nmi_off:
> >> >> + // transition32 when NMI and A20 are already initialized
> >> >> movl %eax, %ecx
> >> >> jmp 1f
> >> >> transition32:
> >> >> @@ -205,7 +206,7 @@ __farcall16:
> >> >> entry_smi:
> >> >> // Transition to 32bit mode.
> >> >> movl $1f + BUILD_BIOS_ADDR, %edx
> >> >> - jmp transition32_for_smi
> >> >> + jmp transition32_nmi_off
> >> >> .code32
> >> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> >> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> >> >> @@ -216,8 +217,10 @@ entry_smi:
> >> >> DECLFUNC entry_smp
> >> >> entry_smp:
> >> >> // Transition to 32bit mode.
> >> >> + cli
> >> >> + cld
> >> >> movl $2f + BUILD_BIOS_ADDR, %edx
> >> >> - jmp transition32
> >> >> + jmp transition32_nmi_off
> >> >> .code32
> >> >> // Acquire lock and take ownership of shared stack
> >> >> 1: rep ; nop
> >> > --
> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-12 9:59 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-12 9:59 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
* Andrey Korolyov (andrey@xdel.ru) wrote:
> On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Andrey Korolyov (andrey@xdel.ru) wrote:
> >> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
> >> <dgilbert@redhat.com> wrote:
> >> > * Kevin O'Connor (kevin@koconnor.net) wrote:
> >> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
> >> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
> >> >> > > For what it's worth, I can't seem to trigger the problem if I move the
> >> >> > > cmos read above the SIPI/LAPIC code (see patch below).
> >> >> >
> >> >> > Ugh!
> >> >> >
> >> >> > That's a seabios bug. Main processor modifies the rtc index
> >> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
> >> >> > index (romlayout.S:transition32).
> >> >> >
> >> >> > I'll put together a fix.
> >> >>
> >> >> The seabios patch below resolves the issue for me.
> >> >
> >> > Thanks! Looks good here.
> >> >
> >> > Andrey, Paolo, Bandan: Does it fix it for you as well?
> >> >
> >>
> >> Thanks Kevin, Dave,
> >>
> >> I`m afraid that I`m hitting something different not only because
> >> different suberror code but also because of mine version of seabios -
> >> I am using 1.7.5 and corresponding code in the proposed patch looks
> >> different - there is no smp-related code patch is about of. Those
> >> mentioned devices went to production successfully and I`m afraid I
> >> cannot afford playing on them anymore, even if I re-trigger the issue
> >> with patched 1.8.1-rc, there is no way to switch to a different kernel
> >> and retest due to specific conditions of this production suite. I`ve
> >> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
> >
> > Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
> > was pretty simple. If you can suggest any flags I should add etc to the
> > test I'd be happy to give it a go.
> >
> > Dave
>
> Here is mine launch string:
>
> qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
> -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
> node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
> -device sga -rtc base=utc,driftfix=slew -global
> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
> -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
> 512,slots=31,maxmem=16384M -object
> memory-backend-ram,id=mem0,size=512M -device
> pc-dimm,id=dimm0,node=0,memdev=mem0
>
> I omitted disk backend in this example, but there is a chance that my
> problem is not reproducible without some calls made explicitly by a
> bootloader (not sure what to say for mid-runtime failures).
It seems to survive OK:
while true; do (sleep 1; echo -e '\001cc\n'; sleep 5; echo -e 'q\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -enable-kvm -name vmtest -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512 -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -device sga -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m 512,slots=31,maxmem=16384M -object memory-backend-ram,id=mem0,size=512M -device pc-dimm,id=dimm0,node=0,memdev=mem0 ~/pi.vfd 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
Dave
>
> >
> >> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
> >> patched bios code, c) reproduce initial error on master/3.19 (may be
> >> I`ll take them before weekend by going into this computer shop in
> >> person). Until then, I have a very deep feeling that mine issue is not
> >> there :) Also I became very curious on how a lack of IDT feature may
> >> completely eliminate the issue appearance for me, the only possible
> >> explanation is a clock-related race which is kinda stupid suggestion
> >> and unlikely to exist in nature.
> >>
> >> Thanks again for everyone for throughout testing and ideas!
> >>
> >> >
> >> >> -Kevin
> >> >>
> >> >>
> >> >> --- a/src/romlayout.S
> >> >> +++ b/src/romlayout.S
> >> >> @@ -22,7 +22,8 @@
> >> >> // %edx = return location (in 32bit mode)
> >> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
> >> >> DECLFUNC transition32
> >> >> -transition32_for_smi:
> >> >> +transition32_nmi_off:
> >> >> + // transition32 when NMI and A20 are already initialized
> >> >> movl %eax, %ecx
> >> >> jmp 1f
> >> >> transition32:
> >> >> @@ -205,7 +206,7 @@ __farcall16:
> >> >> entry_smi:
> >> >> // Transition to 32bit mode.
> >> >> movl $1f + BUILD_BIOS_ADDR, %edx
> >> >> - jmp transition32_for_smi
> >> >> + jmp transition32_nmi_off
> >> >> .code32
> >> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
> >> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
> >> >> @@ -216,8 +217,10 @@ entry_smi:
> >> >> DECLFUNC entry_smp
> >> >> entry_smp:
> >> >> // Transition to 32bit mode.
> >> >> + cli
> >> >> + cld
> >> >> movl $2f + BUILD_BIOS_ADDR, %edx
> >> >> - jmp transition32
> >> >> + jmp transition32_nmi_off
> >> >> .code32
> >> >> // Acquire lock and take ownership of shared stack
> >> >> 1: rep ; nop
> >> > --
> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-12 9:59 ` Dr. David Alan Gilbert
@ 2015-03-12 10:47 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-12 10:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 12, 2015 at 12:59 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Andrey Korolyov (andrey@xdel.ru) wrote:
>> >> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
>> >> <dgilbert@redhat.com> wrote:
>> >> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> >> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> >> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> >> >> > > For what it's worth, I can't seem to trigger the problem if I move the
>> >> >> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >> >> >
>> >> >> > Ugh!
>> >> >> >
>> >> >> > That's a seabios bug. Main processor modifies the rtc index
>> >> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> >> >> > index (romlayout.S:transition32).
>> >> >> >
>> >> >> > I'll put together a fix.
>> >> >>
>> >> >> The seabios patch below resolves the issue for me.
>> >> >
>> >> > Thanks! Looks good here.
>> >> >
>> >> > Andrey, Paolo, Bandan: Does it fix it for you as well?
>> >> >
>> >>
>> >> Thanks Kevin, Dave,
>> >>
>> >> I`m afraid that I`m hitting something different not only because
>> >> different suberror code but also because of mine version of seabios -
>> >> I am using 1.7.5 and corresponding code in the proposed patch looks
>> >> different - there is no smp-related code patch is about of. Those
>> >> mentioned devices went to production successfully and I`m afraid I
>> >> cannot afford playing on them anymore, even if I re-trigger the issue
>> >> with patched 1.8.1-rc, there is no way to switch to a different kernel
>> >> and retest due to specific conditions of this production suite. I`ve
>> >> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
>> >
>> > Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
>> > was pretty simple. If you can suggest any flags I should add etc to the
>> > test I'd be happy to give it a go.
>> >
>> > Dave
>>
>> Here is mine launch string:
>>
>> qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
>> -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
>> node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
>> -device sga -rtc base=utc,driftfix=slew -global
>> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
>> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
>> -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
>> 512,slots=31,maxmem=16384M -object
>> memory-backend-ram,id=mem0,size=512M -device
>> pc-dimm,id=dimm0,node=0,memdev=mem0
>>
>> I omitted disk backend in this example, but there is a chance that my
>> problem is not reproducible without some calls made explicitly by a
>> bootloader (not sure what to say for mid-runtime failures).
>
> It seems to survive OK:
Thanks David, I`ll go through test sequence and report. Unfortunately
my orchestration does not have even a hundred millisecond precision
for libvirt events, so I can`t tell if the immediate start-up failures
happened before bootloader execution or during it, all I have for
those is a less-than-two-second interval between actual pass of a
launch command and paused state event. QEMU logging also does not give
me timestamps for an emulation errors even with appropriate timestamp
arg.
>
> while true; do (sleep 1; echo -e '\001cc\n'; sleep 5; echo -e 'q\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -enable-kvm -name vmtest -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512 -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -device sga -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m 512,slots=31,maxmem=16384M -object memory-backend-ram,id=mem0,size=512M -device pc-dimm,id=dimm0,node=0,memdev=mem0 ~/pi.vfd 2>&1 | tee /tmp/qemu.op; grep "interna
l error" /tmp/qemu.op -q && break; done
>
> Dave
>
>>
>> >
>> >> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
>> >> patched bios code, c) reproduce initial error on master/3.19 (may be
>> >> I`ll take them before weekend by going into this computer shop in
>> >> person). Until then, I have a very deep feeling that mine issue is not
>> >> there :) Also I became very curious on how a lack of IDT feature may
>> >> completely eliminate the issue appearance for me, the only possible
>> >> explanation is a clock-related race which is kinda stupid suggestion
>> >> and unlikely to exist in nature.
>> >>
>> >> Thanks again for everyone for throughout testing and ideas!
>> >>
>> >> >
>> >> >> -Kevin
>> >> >>
>> >> >>
>> >> >> --- a/src/romlayout.S
>> >> >> +++ b/src/romlayout.S
>> >> >> @@ -22,7 +22,8 @@
>> >> >> // %edx = return location (in 32bit mode)
>> >> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> >> >> DECLFUNC transition32
>> >> >> -transition32_for_smi:
>> >> >> +transition32_nmi_off:
>> >> >> + // transition32 when NMI and A20 are already initialized
>> >> >> movl %eax, %ecx
>> >> >> jmp 1f
>> >> >> transition32:
>> >> >> @@ -205,7 +206,7 @@ __farcall16:
>> >> >> entry_smi:
>> >> >> // Transition to 32bit mode.
>> >> >> movl $1f + BUILD_BIOS_ADDR, %edx
>> >> >> - jmp transition32_for_smi
>> >> >> + jmp transition32_nmi_off
>> >> >> .code32
>> >> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> >> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> >> >> @@ -216,8 +217,10 @@ entry_smi:
>> >> >> DECLFUNC entry_smp
>> >> >> entry_smp:
>> >> >> // Transition to 32bit mode.
>> >> >> + cli
>> >> >> + cld
>> >> >> movl $2f + BUILD_BIOS_ADDR, %edx
>> >> >> - jmp transition32
>> >> >> + jmp transition32_nmi_off
>> >> >> .code32
>> >> >> // Acquire lock and take ownership of shared stack
>> >> >> 1: rep ; nop
>> >> > --
>> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-12 10:47 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-12 10:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 12, 2015 at 12:59 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Andrey Korolyov (andrey@xdel.ru) wrote:
>> On Wed, Mar 11, 2015 at 10:59 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Andrey Korolyov (andrey@xdel.ru) wrote:
>> >> On Wed, Mar 11, 2015 at 10:33 PM, Dr. David Alan Gilbert
>> >> <dgilbert@redhat.com> wrote:
>> >> > * Kevin O'Connor (kevin@koconnor.net) wrote:
>> >> >> On Wed, Mar 11, 2015 at 02:45:31PM -0400, Kevin O'Connor wrote:
>> >> >> > On Wed, Mar 11, 2015 at 02:40:39PM -0400, Kevin O'Connor wrote:
>> >> >> > > For what it's worth, I can't seem to trigger the problem if I move the
>> >> >> > > cmos read above the SIPI/LAPIC code (see patch below).
>> >> >> >
>> >> >> > Ugh!
>> >> >> >
>> >> >> > That's a seabios bug. Main processor modifies the rtc index
>> >> >> > (rtc_read()) while APs try to clear the NMI bit by modifying the rtc
>> >> >> > index (romlayout.S:transition32).
>> >> >> >
>> >> >> > I'll put together a fix.
>> >> >>
>> >> >> The seabios patch below resolves the issue for me.
>> >> >
>> >> > Thanks! Looks good here.
>> >> >
>> >> > Andrey, Paolo, Bandan: Does it fix it for you as well?
>> >> >
>> >>
>> >> Thanks Kevin, Dave,
>> >>
>> >> I`m afraid that I`m hitting something different not only because
>> >> different suberror code but also because of mine version of seabios -
>> >> I am using 1.7.5 and corresponding code in the proposed patch looks
>> >> different - there is no smp-related code patch is about of. Those
>> >> mentioned devices went to production successfully and I`m afraid I
>> >> cannot afford playing on them anymore, even if I re-trigger the issue
>> >> with patched 1.8.1-rc, there is no way to switch to a different kernel
>> >> and retest due to specific conditions of this production suite. I`ve
>> >> ordered a pair of new shoes^W 2620v2-s which should arrive to me next
>> >
>> > Well I was testing on a pair of 'E5-2620 v2'; but as you saw my test case
>> > was pretty simple. If you can suggest any flags I should add etc to the
>> > test I'd be happy to give it a go.
>> >
>> > Dave
>>
>> Here is mine launch string:
>>
>> qemu-system-x86_64 -enable-kvm -name vmtest -S -machine
>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
>> -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa
>> node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -nodefaults
>> -device sga -rtc base=utc,driftfix=slew -global
>> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global
>> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
>> -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m
>> 512,slots=31,maxmem=16384M -object
>> memory-backend-ram,id=mem0,size=512M -device
>> pc-dimm,id=dimm0,node=0,memdev=mem0
>>
>> I omitted disk backend in this example, but there is a chance that my
>> problem is not reproducible without some calls made explicitly by a
>> bootloader (not sure what to say for mid-runtime failures).
>
> It seems to survive OK:
Thanks David, I`ll go through test sequence and report. Unfortunately
my orchestration does not have even a hundred millisecond precision
for libvirt events, so I can`t tell if the immediate start-up failures
happened before bootloader execution or during it, all I have for
those is a less-than-two-second interval between actual pass of a
launch command and paused state event. QEMU logging also does not give
me timestamps for an emulation errors even with appropriate timestamp
arg.
>
> while true; do (sleep 1; echo -e '\001cc\n'; sleep 5; echo -e 'q\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -enable-kvm -name vmtest -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512 -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0-11,mem=512 -nographic -no-user-config -device sga -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -m 512,slots=31,maxmem=16384M -object memory-backend-ram,id=mem0,size=512M -device pc-dimm,id=dimm0,node=0,memdev=mem0 ~/pi.vfd 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> Dave
>
>>
>> >
>> >> Monday, so I`ll be able to test a) against 1.8.0-release, b) against
>> >> patched bios code, c) reproduce initial error on master/3.19 (may be
>> >> I`ll take them before weekend by going into this computer shop in
>> >> person). Until then, I have a very deep feeling that mine issue is not
>> >> there :) Also I became very curious on how a lack of IDT feature may
>> >> completely eliminate the issue appearance for me, the only possible
>> >> explanation is a clock-related race which is kinda stupid suggestion
>> >> and unlikely to exist in nature.
>> >>
>> >> Thanks again for everyone for throughout testing and ideas!
>> >>
>> >> >
>> >> >> -Kevin
>> >> >>
>> >> >>
>> >> >> --- a/src/romlayout.S
>> >> >> +++ b/src/romlayout.S
>> >> >> @@ -22,7 +22,8 @@
>> >> >> // %edx = return location (in 32bit mode)
>> >> >> // Clobbers: ecx, flags, segment registers, cr0, idt/gdt
>> >> >> DECLFUNC transition32
>> >> >> -transition32_for_smi:
>> >> >> +transition32_nmi_off:
>> >> >> + // transition32 when NMI and A20 are already initialized
>> >> >> movl %eax, %ecx
>> >> >> jmp 1f
>> >> >> transition32:
>> >> >> @@ -205,7 +206,7 @@ __farcall16:
>> >> >> entry_smi:
>> >> >> // Transition to 32bit mode.
>> >> >> movl $1f + BUILD_BIOS_ADDR, %edx
>> >> >> - jmp transition32_for_smi
>> >> >> + jmp transition32_nmi_off
>> >> >> .code32
>> >> >> 1: movl $BUILD_SMM_ADDR + 0x8000, %esp
>> >> >> calll _cfunc32flat_handle_smi - BUILD_BIOS_ADDR
>> >> >> @@ -216,8 +217,10 @@ entry_smi:
>> >> >> DECLFUNC entry_smp
>> >> >> entry_smp:
>> >> >> // Transition to 32bit mode.
>> >> >> + cli
>> >> >> + cld
>> >> >> movl $2f + BUILD_BIOS_ADDR, %edx
>> >> >> - jmp transition32
>> >> >> + jmp transition32_nmi_off
>> >> >> .code32
>> >> >> // Acquire lock and take ownership of shared stack
>> >> >> 1: rep ; nop
>> >> > --
>> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-12 10:47 ` Andrey Korolyov
@ 2015-03-16 19:17 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-16 19:17 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
it appearance is very rare (compared to the number of actual launches)
and most probably bounded to the physical characteristics of my
production nodes. As soon as I reach any reproducible path for a
regular workstation environment, I`ll let everyone know. Also I am
starting to think that issue can belong to the particular motherboard
firmware revision, despite fact that the CPU microcode is the same
everywhere.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-16 19:17 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-16 19:17 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
it appearance is very rare (compared to the number of actual launches)
and most probably bounded to the physical characteristics of my
production nodes. As soon as I reach any reproducible path for a
regular workstation environment, I`ll let everyone know. Also I am
starting to think that issue can belong to the particular motherboard
firmware revision, despite fact that the CPU microcode is the same
everywhere.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-16 19:17 ` Andrey Korolyov
@ 2015-03-16 19:26 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-16 19:26 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
* Andrey Korolyov (andrey@xdel.ru) wrote:
> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> it appearance is very rare (compared to the number of actual launches)
> and most probably bounded to the physical characteristics of my
> production nodes. As soon as I reach any reproducible path for a
> regular workstation environment, I`ll let everyone know. Also I am
> starting to think that issue can belong to the particular motherboard
> firmware revision, despite fact that the CPU microcode is the same
> everywhere.
OK - so you're still seeing it with the new ROM that went in today?
( remotes/kraxel/tags/pull-seabios-1.8.1-20150316-1 ) and it doesn't
trigger with my one line script?
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-16 19:26 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 157+ messages in thread
From: Dr. David Alan Gilbert @ 2015-03-16 19:26 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
* Andrey Korolyov (andrey@xdel.ru) wrote:
> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> it appearance is very rare (compared to the number of actual launches)
> and most probably bounded to the physical characteristics of my
> production nodes. As soon as I reach any reproducible path for a
> regular workstation environment, I`ll let everyone know. Also I am
> starting to think that issue can belong to the particular motherboard
> firmware revision, despite fact that the CPU microcode is the same
> everywhere.
OK - so you're still seeing it with the new ROM that went in today?
( remotes/kraxel/tags/pull-seabios-1.8.1-20150316-1 ) and it doesn't
trigger with my one line script?
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-16 19:17 ` Andrey Korolyov
@ 2015-03-25 20:43 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 20:43 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> it appearance is very rare (compared to the number of actual launches)
> and most probably bounded to the physical characteristics of my
> production nodes. As soon as I reach any reproducible path for a
> regular workstation environment, I`ll let everyone know. Also I am
> starting to think that issue can belong to the particular motherboard
> firmware revision, despite fact that the CPU microcode is the same
> everywhere.
Hello everyone, I`ve managed to reproduce this issue
*deterministically* with latest seabios with smp fix and 3.18.3. The
error occuring just *once* per vm until hypervisor reboots, at least
in my setup, this is definitely crazy...
- launch two VMs (Centos 7 in my case),
- wait a little while they are booting,
- attach serial console (I am using virsh list for this exact purpose),
- issue acpi reboot or reset, does not matter,
- VM always hangs at boot, most times with sgabios initialization
string printed out [1], but sometimes it hangs a bit later [2],
- no matter how many times I try to relaunch the QEMU afterwards, the
issue does not appear on VM which experienced problem once;
- trace and sample args can be seen in [3] and [4] respectively.
1)
Google, Inc.
Serial Graphics Adapter 06/11/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
(pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
Term: 211x62
4 0
2)
Google, Inc.
Serial Graphics Adapter 06/11/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
(pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
Term: 211x62
4 0
[...empty screen...]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
3)
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6cb0 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
ba 2d d4 fe fb 3f
4)
/usr/bin/qemu-system-x86_64 -name centos71 -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
/usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
12,sockets=1,cores=12,threads=12 -uuid
3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
-nodefaults -device sga -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
-msg timestamp=on
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 20:43 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 20:43 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> it appearance is very rare (compared to the number of actual launches)
> and most probably bounded to the physical characteristics of my
> production nodes. As soon as I reach any reproducible path for a
> regular workstation environment, I`ll let everyone know. Also I am
> starting to think that issue can belong to the particular motherboard
> firmware revision, despite fact that the CPU microcode is the same
> everywhere.
Hello everyone, I`ve managed to reproduce this issue
*deterministically* with latest seabios with smp fix and 3.18.3. The
error occuring just *once* per vm until hypervisor reboots, at least
in my setup, this is definitely crazy...
- launch two VMs (Centos 7 in my case),
- wait a little while they are booting,
- attach serial console (I am using virsh list for this exact purpose),
- issue acpi reboot or reset, does not matter,
- VM always hangs at boot, most times with sgabios initialization
string printed out [1], but sometimes it hangs a bit later [2],
- no matter how many times I try to relaunch the QEMU afterwards, the
issue does not appear on VM which experienced problem once;
- trace and sample args can be seen in [3] and [4] respectively.
1)
Google, Inc.
Serial Graphics Adapter 06/11/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
(pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
Term: 211x62
4 0
2)
Google, Inc.
Serial Graphics Adapter 06/11/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
(pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
Term: 211x62
4 0
[...empty screen...]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
3)
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6cb0 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
ba 2d d4 fe fb 3f
4)
/usr/bin/qemu-system-x86_64 -name centos71 -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
/usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
12,sockets=1,cores=12,threads=12 -uuid
3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
-nodefaults -device sga -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
-msg timestamp=on
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-25 20:43 ` [Qemu-devel] " Andrey Korolyov
@ 2015-03-25 20:46 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 20:46 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
> - attach serial console (I am using virsh list for this exact purpose),
virsh console of course, sorry
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 20:46 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 20:46 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: kvm, qemu-devel, Bandan Das, Kevin O'Connor, Gerd Hoffmann,
Paolo Bonzini
> - attach serial console (I am using virsh list for this exact purpose),
virsh console of course, sorry
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 20:43 ` [Qemu-devel] " Andrey Korolyov
@ 2015-03-25 20:54 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-25 20:54 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Wed, Mar 25, 2015 at 11:43:31PM +0300, Andrey Korolyov wrote:
> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> > it appearance is very rare (compared to the number of actual launches)
> > and most probably bounded to the physical characteristics of my
> > production nodes. As soon as I reach any reproducible path for a
> > regular workstation environment, I`ll let everyone know. Also I am
> > starting to think that issue can belong to the particular motherboard
> > firmware revision, despite fact that the CPU microcode is the same
> > everywhere.
>
>
> Hello everyone, I`ve managed to reproduce this issue
> *deterministically* with latest seabios with smp fix and 3.18.3. The
> error occuring just *once* per vm until hypervisor reboots, at least
> in my setup, this is definitely crazy...
>
> - launch two VMs (Centos 7 in my case),
> - wait a little while they are booting,
> - attach serial console (I am using virsh list for this exact purpose),
> - issue acpi reboot or reset, does not matter,
> - VM always hangs at boot, most times with sgabios initialization
> string printed out [1], but sometimes it hangs a bit later [2],
> - no matter how many times I try to relaunch the QEMU afterwards, the
> issue does not appear on VM which experienced problem once;
> - trace and sample args can be seen in [3] and [4] respectively.
Can you add something like:
-chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
to the qemu command line and forward the resulting log from both a
succesful boot and a failed one?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 20:54 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-25 20:54 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Gerd Hoffmann, Paolo Bonzini
On Wed, Mar 25, 2015 at 11:43:31PM +0300, Andrey Korolyov wrote:
> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
> > it appearance is very rare (compared to the number of actual launches)
> > and most probably bounded to the physical characteristics of my
> > production nodes. As soon as I reach any reproducible path for a
> > regular workstation environment, I`ll let everyone know. Also I am
> > starting to think that issue can belong to the particular motherboard
> > firmware revision, despite fact that the CPU microcode is the same
> > everywhere.
>
>
> Hello everyone, I`ve managed to reproduce this issue
> *deterministically* with latest seabios with smp fix and 3.18.3. The
> error occuring just *once* per vm until hypervisor reboots, at least
> in my setup, this is definitely crazy...
>
> - launch two VMs (Centos 7 in my case),
> - wait a little while they are booting,
> - attach serial console (I am using virsh list for this exact purpose),
> - issue acpi reboot or reset, does not matter,
> - VM always hangs at boot, most times with sgabios initialization
> string printed out [1], but sometimes it hangs a bit later [2],
> - no matter how many times I try to relaunch the QEMU afterwards, the
> issue does not appear on VM which experienced problem once;
> - trace and sample args can be seen in [3] and [4] respectively.
Can you add something like:
-chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
to the qemu command line and forward the resulting log from both a
succesful boot and a failed one?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 20:54 ` Kevin O'Connor
@ 2015-03-25 22:31 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 22:31 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]
On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Wed, Mar 25, 2015 at 11:43:31PM +0300, Andrey Korolyov wrote:
>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>> > it appearance is very rare (compared to the number of actual launches)
>> > and most probably bounded to the physical characteristics of my
>> > production nodes. As soon as I reach any reproducible path for a
>> > regular workstation environment, I`ll let everyone know. Also I am
>> > starting to think that issue can belong to the particular motherboard
>> > firmware revision, despite fact that the CPU microcode is the same
>> > everywhere.
>>
>>
>> Hello everyone, I`ve managed to reproduce this issue
>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>> error occuring just *once* per vm until hypervisor reboots, at least
>> in my setup, this is definitely crazy...
>>
>> - launch two VMs (Centos 7 in my case),
>> - wait a little while they are booting,
>> - attach serial console (I am using virsh list for this exact purpose),
>> - issue acpi reboot or reset, does not matter,
>> - VM always hangs at boot, most times with sgabios initialization
>> string printed out [1], but sometimes it hangs a bit later [2],
>> - no matter how many times I try to relaunch the QEMU afterwards, the
>> issue does not appear on VM which experienced problem once;
>> - trace and sample args can be seen in [3] and [4] respectively.
>
> Can you add something like:
>
> -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
>
> to the qemu command line and forward the resulting log from both a
> succesful boot and a failed one?
>
> -Kevin
Of course, logs are attached.
[-- Attachment #2: reboot.failed --]
[-- Type: application/octet-stream, Size: 6053 bytes --]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2103
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=4
handle_smp: apic_id=11
handle_smp: apic_id=3
handle_smp: apic_id=6
handle_smp: apic_id=2
handle_smp: apic_id=8
handle_smp: apic_id=5
handle_smp: apic_id=9
handle_smp: apic_id=1
handle_smp: apic_id=7
handle_smp: apic_id=10
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2102
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=2
handle_smp: apic_id=5
handle_smp: apic_id=1
handle_smp: apic_id=4
handle_smp: apic_id=9
handle_smp: apic_id=8
handle_smp: apic_id=11
handle_smp: apic_id=6
handle_smp: apic_id=7
handle_smp: apic_id=10
handle_smp: apic_id=3
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
[-- Attachment #3: reboot.succeeded --]
[-- Type: application/octet-stream, Size: 6126 bytes --]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2102
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=4
handle_smp: apic_id=2
handle_smp: apic_id=1
handle_smp: apic_id=9
handle_smp: apic_id=6
handle_smp: apic_id=11
handle_smp: apic_id=7
handle_smp: apic_id=5
handle_smp: apic_id=3
handle_smp: apic_id=8
handle_smp: apic_id=10
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2105
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=9
handle_smp: apic_id=7
handle_smp: apic_id=1
handle_smp: apic_id=6
handle_smp: apic_id=5
handle_smp: apic_id=10
handle_smp: apic_id=4
handle_smp: apic_id=8
handle_smp: apic_id=11
handle_smp: apic_id=2
handle_smp: apic_id=3
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 22:31 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 22:31 UTC (permalink / raw)
To: Kevin O'Connor
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Gerd Hoffmann, Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]
On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Wed, Mar 25, 2015 at 11:43:31PM +0300, Andrey Korolyov wrote:
>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>> > it appearance is very rare (compared to the number of actual launches)
>> > and most probably bounded to the physical characteristics of my
>> > production nodes. As soon as I reach any reproducible path for a
>> > regular workstation environment, I`ll let everyone know. Also I am
>> > starting to think that issue can belong to the particular motherboard
>> > firmware revision, despite fact that the CPU microcode is the same
>> > everywhere.
>>
>>
>> Hello everyone, I`ve managed to reproduce this issue
>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>> error occuring just *once* per vm until hypervisor reboots, at least
>> in my setup, this is definitely crazy...
>>
>> - launch two VMs (Centos 7 in my case),
>> - wait a little while they are booting,
>> - attach serial console (I am using virsh list for this exact purpose),
>> - issue acpi reboot or reset, does not matter,
>> - VM always hangs at boot, most times with sgabios initialization
>> string printed out [1], but sometimes it hangs a bit later [2],
>> - no matter how many times I try to relaunch the QEMU afterwards, the
>> issue does not appear on VM which experienced problem once;
>> - trace and sample args can be seen in [3] and [4] respectively.
>
> Can you add something like:
>
> -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
>
> to the qemu command line and forward the resulting log from both a
> succesful boot and a failed one?
>
> -Kevin
Of course, logs are attached.
[-- Attachment #2: reboot.failed --]
[-- Type: application/octet-stream, Size: 6053 bytes --]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2103
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=4
handle_smp: apic_id=11
handle_smp: apic_id=3
handle_smp: apic_id=6
handle_smp: apic_id=2
handle_smp: apic_id=8
handle_smp: apic_id=5
handle_smp: apic_id=9
handle_smp: apic_id=1
handle_smp: apic_id=7
handle_smp: apic_id=10
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2102
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=2
handle_smp: apic_id=5
handle_smp: apic_id=1
handle_smp: apic_id=4
handle_smp: apic_id=9
handle_smp: apic_id=8
handle_smp: apic_id=11
handle_smp: apic_id=6
handle_smp: apic_id=7
handle_smp: apic_id=10
handle_smp: apic_id=3
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
[-- Attachment #3: reboot.succeeded --]
[-- Type: application/octet-stream, Size: 6126 bytes --]
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2102
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=4
handle_smp: apic_id=2
handle_smp: apic_id=1
handle_smp: apic_id=9
handle_smp: apic_id=6
handle_smp: apic_id=11
handle_smp: apic_id=7
handle_smp: apic_id=5
handle_smp: apic_id=3
handle_smp: apic_id=8
handle_smp: apic_id=10
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e11b0 to 0x3ffafd10 (size 66096)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2105
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=9
handle_smp: apic_id=7
handle_smp: apic_id=1
handle_smp: apic_id=6
handle_smp: apic_id=5
handle_smp: apic_id=10
handle_smp: apic_id=4
handle_smp: apic_id=8
handle_smp: apic_id=11
handle_smp: apic_id=2
handle_smp: apic_id=3
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1360
Copying MPTABLE from 0x00006d49/3ffa6c20 to 0x000f1270
Copying SMBIOS entry point from 0x00006d59 to 0x000f10d0
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150325_230423-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1050: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1050
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 22:31 ` Andrey Korolyov
@ 2015-03-25 23:02 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-25 23:02 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 01:31:11AM +0300, Andrey Korolyov wrote:
> On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> >
> > Can you add something like:
> >
> > -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
> >
> > to the qemu command line and forward the resulting log from both a
> > succesful boot and a failed one?
> >
> > -Kevin
>
> Of course, logs are attached.
Thanks. From a diff of the two logs:
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
-enter handle_19:
- NULL
-Booting from Hard Disk...
-Booting from 0000:7c00
So, it got most of the way through the reboot - there's only a few
function calls between the e820 map being dumped and the handle_19
call. The fault also seems to show it stopped in the BIOS in 16bit
mode:
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
Can you add the patch below, force the fault, and forward the log.
Also, if you recreate the failure can you take the EIP from the fault
(eg, d331) and search for the corresponding function in the output of:
objdump -m i386 -M i8086 -M suffix -ldr out/rom16.o | less
(That is, search for "d331:".) If that's too much of a pain, just
send me a direct email with the seabios out/rom16.o file and the new
EIP of the fault. (I need the out/rom16.o that was used to build the
version of SeaBIOS that faulted.)
-Kevin
diff --git a/src/post.c b/src/post.c
index 9ea5620..bbd19c0 100644
--- a/src/post.c
+++ b/src/post.c
@@ -185,21 +185,24 @@ prepareboot(void)
pmm_prepboot();
malloc_prepboot();
memmap_prepboot();
+ dprintf(1, "a\n");
HaveRunPost = 2;
// Setup bios checksum.
BiosChecksum -= checksum((u8*)BUILD_BIOS_ADDR, BUILD_BIOS_SIZE);
+ dprintf(1, "b\n");
}
// Begin the boot process by invoking an int0x19 in 16bit mode.
void VISIBLE32FLAT
startBoot(void)
{
+ dprintf(1, "e\n");
// Clear low-memory allocations (required by PMM spec).
memset((void*)BUILD_STACK_ADDR, 0, BUILD_EBDA_MINIMUM - BUILD_STACK_ADDR);
- dprintf(3, "Jump to int19\n");
+ dprintf(1, "Jump to int19 (vector=%x)\n", GET_IVT(0x19).segoff);
struct bregs br;
memset(&br, 0, sizeof(br));
br.flags = F_IF;
@@ -239,9 +242,11 @@ maininit(void)
// Prepare for boot.
prepareboot();
+ dprintf(1, "c\n");
// Write protect bios memory.
make_bios_readonly();
+ dprintf(1, "d\n");
// Invoke int 19 to start boot process.
startBoot();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 23:02 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-25 23:02 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 01:31:11AM +0300, Andrey Korolyov wrote:
> On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> >
> > Can you add something like:
> >
> > -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
> >
> > to the qemu command line and forward the resulting log from both a
> > succesful boot and a failed one?
> >
> > -Kevin
>
> Of course, logs are attached.
Thanks. From a diff of the two logs:
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
-enter handle_19:
- NULL
-Booting from Hard Disk...
-Booting from 0000:7c00
So, it got most of the way through the reboot - there's only a few
function calls between the e820 map being dumped and the handle_19
call. The fault also seems to show it stopped in the BIOS in 16bit
mode:
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
Can you add the patch below, force the fault, and forward the log.
Also, if you recreate the failure can you take the EIP from the fault
(eg, d331) and search for the corresponding function in the output of:
objdump -m i386 -M i8086 -M suffix -ldr out/rom16.o | less
(That is, search for "d331:".) If that's too much of a pain, just
send me a direct email with the seabios out/rom16.o file and the new
EIP of the fault. (I need the out/rom16.o that was used to build the
version of SeaBIOS that faulted.)
-Kevin
diff --git a/src/post.c b/src/post.c
index 9ea5620..bbd19c0 100644
--- a/src/post.c
+++ b/src/post.c
@@ -185,21 +185,24 @@ prepareboot(void)
pmm_prepboot();
malloc_prepboot();
memmap_prepboot();
+ dprintf(1, "a\n");
HaveRunPost = 2;
// Setup bios checksum.
BiosChecksum -= checksum((u8*)BUILD_BIOS_ADDR, BUILD_BIOS_SIZE);
+ dprintf(1, "b\n");
}
// Begin the boot process by invoking an int0x19 in 16bit mode.
void VISIBLE32FLAT
startBoot(void)
{
+ dprintf(1, "e\n");
// Clear low-memory allocations (required by PMM spec).
memset((void*)BUILD_STACK_ADDR, 0, BUILD_EBDA_MINIMUM - BUILD_STACK_ADDR);
- dprintf(3, "Jump to int19\n");
+ dprintf(1, "Jump to int19 (vector=%x)\n", GET_IVT(0x19).segoff);
struct bregs br;
memset(&br, 0, sizeof(br));
br.flags = F_IF;
@@ -239,9 +242,11 @@ maininit(void)
// Prepare for boot.
prepareboot();
+ dprintf(1, "c\n");
// Write protect bios memory.
make_bios_readonly();
+ dprintf(1, "d\n");
// Invoke int 19 to start boot process.
startBoot();
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 23:02 ` Kevin O'Connor
@ 2015-03-25 23:35 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 23:35 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
[-- Attachment #1: Type: text/plain, Size: 3324 bytes --]
On Thu, Mar 26, 2015 at 2:02 AM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 01:31:11AM +0300, Andrey Korolyov wrote:
>> On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> >
>> > Can you add something like:
>> >
>> > -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
>> >
>> > to the qemu command line and forward the resulting log from both a
>> > succesful boot and a failed one?
>> >
>> > -Kevin
>>
>> Of course, logs are attached.
>
> Thanks. From a diff of the two logs:
>
> 4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
> 5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
> 6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
> -enter handle_19:
> - NULL
> -Booting from Hard Disk...
> -Booting from 0000:7c00
>
> So, it got most of the way through the reboot - there's only a few
> function calls between the e820 map being dumped and the handle_19
> call. The fault also seems to show it stopped in the BIOS in 16bit
> mode:
>
>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>
> Can you add the patch below, force the fault, and forward the log.
>
> Also, if you recreate the failure can you take the EIP from the fault
> (eg, d331) and search for the corresponding function in the output of:
> objdump -m i386 -M i8086 -M suffix -ldr out/rom16.o | less
> (That is, search for "d331:".) If that's too much of a pain, just
> send me a direct email with the seabios out/rom16.o file and the new
> EIP of the fault. (I need the out/rom16.o that was used to build the
> version of SeaBIOS that faulted.)
>
> -Kevin
>
>
> diff --git a/src/post.c b/src/post.c
> index 9ea5620..bbd19c0 100644
> --- a/src/post.c
> +++ b/src/post.c
> @@ -185,21 +185,24 @@ prepareboot(void)
> pmm_prepboot();
> malloc_prepboot();
> memmap_prepboot();
> + dprintf(1, "a\n");
>
> HaveRunPost = 2;
>
> // Setup bios checksum.
> BiosChecksum -= checksum((u8*)BUILD_BIOS_ADDR, BUILD_BIOS_SIZE);
> + dprintf(1, "b\n");
> }
>
> // Begin the boot process by invoking an int0x19 in 16bit mode.
> void VISIBLE32FLAT
> startBoot(void)
> {
> + dprintf(1, "e\n");
> // Clear low-memory allocations (required by PMM spec).
> memset((void*)BUILD_STACK_ADDR, 0, BUILD_EBDA_MINIMUM - BUILD_STACK_ADDR);
>
> - dprintf(3, "Jump to int19\n");
> + dprintf(1, "Jump to int19 (vector=%x)\n", GET_IVT(0x19).segoff);
> struct bregs br;
> memset(&br, 0, sizeof(br));
> br.flags = F_IF;
> @@ -239,9 +242,11 @@ maininit(void)
> // Prepare for boot.
> prepareboot();
>
> + dprintf(1, "c\n");
> // Write protect bios memory.
> make_bios_readonly();
>
> + dprintf(1, "d\n");
> // Invoke int 19 to start boot process.
> startBoot();
> }
Thanks, strangely the reboot is always failing now and always reaching
seabios greeting. May be prints straightened up a race (e.g. it is not
int19 problem really).
object file part:
0000d331 <irq_trampoline_0x19>:
irq_trampoline_0x19():
/root/seabios-1.8.1/src/romlayout.S:195
d331: cd 19 int $0x19
d333: cb lretw
[-- Attachment #2: reboot.failed --]
[-- Type: application/octet-stream, Size: 6137 bytes --]
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e1130 to 0x3ffafce0 (size 66144)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2103
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=11
handle_smp: apic_id=3
handle_smp: apic_id=8
handle_smp: apic_id=4
handle_smp: apic_id=9
handle_smp: apic_id=1
handle_smp: apic_id=2
handle_smp: apic_id=7
handle_smp: apic_id=6
handle_smp: apic_id=10
handle_smp: apic_id=5
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1310
Copying MPTABLE from 0x00006d49/3ffa6bf0 to 0x000f1220
Copying SMBIOS entry point from 0x00006d59 to 0x000f1080
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1000: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1000
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
a
b
c
d
e
Jump to int19 (vector=f000e6f2)
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e1130 to 0x3ffafce0 (size 66144)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2108
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=1
handle_smp: apic_id=2
handle_smp: apic_id=3
handle_smp: apic_id=4
handle_smp: apic_id=5
handle_smp: apic_id=6
handle_smp: apic_id=7
handle_smp: apic_id=8
handle_smp: apic_id=9
handle_smp: apic_id=10
handle_smp: apic_id=11
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1310
Copying MPTABLE from 0x00006d49/3ffa6bf0 to 0x000f1220
Copying SMBIOS entry point from 0x00006d59 to 0x000f1080
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1000: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1000
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
a
b
c
d
e
Jump to int19 (vector=f000e6f2)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-25 23:35 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-25 23:35 UTC (permalink / raw)
To: Kevin O'Connor
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Gerd Hoffmann, Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 3324 bytes --]
On Thu, Mar 26, 2015 at 2:02 AM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 01:31:11AM +0300, Andrey Korolyov wrote:
>> On Wed, Mar 25, 2015 at 11:54 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> >
>> > Can you add something like:
>> >
>> > -chardev file,path=seabioslog.`date +%s`,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios
>> >
>> > to the qemu command line and forward the resulting log from both a
>> > succesful boot and a failed one?
>> >
>> > -Kevin
>>
>> Of course, logs are attached.
>
> Thanks. From a diff of the two logs:
>
> 4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
> 5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
> 6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
> -enter handle_19:
> - NULL
> -Booting from Hard Disk...
> -Booting from 0000:7c00
>
> So, it got most of the way through the reboot - there's only a few
> function calls between the e820 map being dumped and the handle_19
> call. The fault also seems to show it stopped in the BIOS in 16bit
> mode:
>
>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>
> Can you add the patch below, force the fault, and forward the log.
>
> Also, if you recreate the failure can you take the EIP from the fault
> (eg, d331) and search for the corresponding function in the output of:
> objdump -m i386 -M i8086 -M suffix -ldr out/rom16.o | less
> (That is, search for "d331:".) If that's too much of a pain, just
> send me a direct email with the seabios out/rom16.o file and the new
> EIP of the fault. (I need the out/rom16.o that was used to build the
> version of SeaBIOS that faulted.)
>
> -Kevin
>
>
> diff --git a/src/post.c b/src/post.c
> index 9ea5620..bbd19c0 100644
> --- a/src/post.c
> +++ b/src/post.c
> @@ -185,21 +185,24 @@ prepareboot(void)
> pmm_prepboot();
> malloc_prepboot();
> memmap_prepboot();
> + dprintf(1, "a\n");
>
> HaveRunPost = 2;
>
> // Setup bios checksum.
> BiosChecksum -= checksum((u8*)BUILD_BIOS_ADDR, BUILD_BIOS_SIZE);
> + dprintf(1, "b\n");
> }
>
> // Begin the boot process by invoking an int0x19 in 16bit mode.
> void VISIBLE32FLAT
> startBoot(void)
> {
> + dprintf(1, "e\n");
> // Clear low-memory allocations (required by PMM spec).
> memset((void*)BUILD_STACK_ADDR, 0, BUILD_EBDA_MINIMUM - BUILD_STACK_ADDR);
>
> - dprintf(3, "Jump to int19\n");
> + dprintf(1, "Jump to int19 (vector=%x)\n", GET_IVT(0x19).segoff);
> struct bregs br;
> memset(&br, 0, sizeof(br));
> br.flags = F_IF;
> @@ -239,9 +242,11 @@ maininit(void)
> // Prepare for boot.
> prepareboot();
>
> + dprintf(1, "c\n");
> // Write protect bios memory.
> make_bios_readonly();
>
> + dprintf(1, "d\n");
> // Invoke int 19 to start boot process.
> startBoot();
> }
Thanks, strangely the reboot is always failing now and always reaching
seabios greeting. May be prints straightened up a race (e.g. it is not
int19 problem really).
object file part:
0000d331 <irq_trampoline_0x19>:
irq_trampoline_0x19():
/root/seabios-1.8.1/src/romlayout.S:195
d331: cd 19 int $0x19
d333: cb lretw
[-- Attachment #2: reboot.failed --]
[-- Type: application/octet-stream, Size: 6137 bytes --]
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e1130 to 0x3ffafce0 (size 66144)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2103
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=11
handle_smp: apic_id=3
handle_smp: apic_id=8
handle_smp: apic_id=4
handle_smp: apic_id=9
handle_smp: apic_id=1
handle_smp: apic_id=2
handle_smp: apic_id=7
handle_smp: apic_id=6
handle_smp: apic_id=10
handle_smp: apic_id=5
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1310
Copying MPTABLE from 0x00006d49/3ffa6bf0 to 0x000f1220
Copying SMBIOS entry point from 0x00006d59 to 0x000f1080
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1000: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1000
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
a
b
c
d
e
Jump to int19 (vector=f000e6f2)
enter handle_19:
NULL
Booting from Hard Disk...
Booting from 0000:7c00
In resume (status=0)
In 32bit resume
Attempting a hard reboot
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Running on QEMU (i440fx)
Running on KVM
RamSize: 0x40000000 [cmos]
Relocating init from 0x000e1130 to 0x3ffafce0 (size 66144)
Found QEMU fw_cfg
RamBlock: addr 0x0000000000000000 len 0x0000000040000000 [e820]
Moving pm_base to 0x600
boot order:
1: /pci@i0cf8/scsi@5/disk@0,0
2: HALT
CPU Mhz=2108
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c06f
PCI: 32: 0000000080000000 - 00000000fec00000
PCI: map device bdf=00:05.0 bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:04.0 bar 0, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1 bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:03.0 bar 0, addr febf8000, size 00004000 [mem]
PCI: map device bdf=00:04.0 bar 1, addr febfc000, size 00001000 [mem]
PCI: map device bdf=00:05.0 bar 1, addr febfd000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0x608
PCI: init bdf=00:03.0 id=1033:0194
PCI: init bdf=00:04.0 id=1af4:1003
PCI: init bdf=00:05.0 id=1af4:1001
PCI: No VGA devices found
handle_smp: apic_id=1
handle_smp: apic_id=2
handle_smp: apic_id=3
handle_smp: apic_id=4
handle_smp: apic_id=5
handle_smp: apic_id=6
handle_smp: apic_id=7
handle_smp: apic_id=8
handle_smp: apic_id=9
handle_smp: apic_id=10
handle_smp: apic_id=11
Found 12 cpu(s) max supported 12 cpu(s)
Copying PIR from 0x3ffbfd08 to 0x000f1310
Copying MPTABLE from 0x00006d49/3ffa6bf0 to 0x000f1220
Copying SMBIOS entry point from 0x00006d59 to 0x000f1080
Scan for VGA option rom
Running option rom at c000:0003
Turning on vga text mode console
SeaBIOS (version 1.8.1-20150326_021758-testnode)
Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
All threads complete.
Found 0 lpt ports
Found 1 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:5
Searching bootorder for: /pci@i0cf8/*@5
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin
Searching bootorder for: HALT
drive 0x000f1000: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=16777216
Running option rom at c100:0003
Space available for UMB: c3800-ed800, f0000-f1000
Returned 131072 bytes of ZoneHigh
e820 map has 7 items:
0: 0000000000000000 - 000000000009f800 = 1 RAM
1: 000000000009f800 - 00000000000a0000 = 2 RESERVED
2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
3: 0000000000100000 - 000000003ffe0000 = 1 RAM
4: 000000003ffe0000 - 0000000040000000 = 2 RESERVED
5: 00000000feffc000 - 00000000ff000000 = 2 RESERVED
6: 00000000fffc0000 - 0000000100000000 = 2 RESERVED
a
b
c
d
e
Jump to int19 (vector=f000e6f2)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 23:35 ` Andrey Korolyov
@ 2015-03-26 0:05 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 0:05 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> Thanks, strangely the reboot is always failing now and always reaching
> seabios greeting. May be prints straightened up a race (e.g. it is not
> int19 problem really).
>
> object file part:
>
> 0000d331 <irq_trampoline_0x19>:
> irq_trampoline_0x19():
> /root/seabios-1.8.1/src/romlayout.S:195
> d331: cd 19 int $0x19
> d333: cb lretw
[...]
> Jump to int19 (vector=f000e6f2)
Thanks. So, it dies on the "int $0x19" instruction itself. The
vector looks correct and I don't see anything in the cpu register
state that looks wrong. Maybe one of the kvm developers will have an
idea what could cause a fault there.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 0:05 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 0:05 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> Thanks, strangely the reboot is always failing now and always reaching
> seabios greeting. May be prints straightened up a race (e.g. it is not
> int19 problem really).
>
> object file part:
>
> 0000d331 <irq_trampoline_0x19>:
> irq_trampoline_0x19():
> /root/seabios-1.8.1/src/romlayout.S:195
> d331: cd 19 int $0x19
> d333: cb lretw
[...]
> Jump to int19 (vector=f000e6f2)
Thanks. So, it dies on the "int $0x19" instruction itself. The
vector looks correct and I don't see anything in the cpu register
state that looks wrong. Maybe one of the kvm developers will have an
idea what could cause a fault there.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-25 20:43 ` [Qemu-devel] " Andrey Korolyov
@ 2015-03-26 2:47 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-26 2:47 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Dr. David Alan Gilbert, Kevin O'Connor, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
Hi Andrey,
Andrey Korolyov <andrey@xdel.ru> writes:
> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>> it appearance is very rare (compared to the number of actual launches)
>> and most probably bounded to the physical characteristics of my
>> production nodes. As soon as I reach any reproducible path for a
>> regular workstation environment, I`ll let everyone know. Also I am
>> starting to think that issue can belong to the particular motherboard
>> firmware revision, despite fact that the CPU microcode is the same
>> everywhere.
I will take the risk and say this - "could it be a processor bug ?" :)
>
> Hello everyone, I`ve managed to reproduce this issue
> *deterministically* with latest seabios with smp fix and 3.18.3. The
> error occuring just *once* per vm until hypervisor reboots, at least
> in my setup, this is definitely crazy...
>
> - launch two VMs (Centos 7 in my case),
> - wait a little while they are booting,
> - attach serial console (I am using virsh list for this exact purpose),
> - issue acpi reboot or reset, does not matter,
> - VM always hangs at boot, most times with sgabios initialization
> string printed out [1], but sometimes it hangs a bit later [2],
> - no matter how many times I try to relaunch the QEMU afterwards, the
> issue does not appear on VM which experienced problem once;
> - trace and sample args can be seen in [3] and [4] respectively.
My system is a Dell R720 dual socket which has 2620v2s. I tried your
setup but couldn't reproduce (my qemu cmdline isn't exactly the same
as yours), although, if you could simplify your command line a bit,
I can try again.
Bandan
> 1)
> Google, Inc.
> Serial Graphics Adapter 06/11/14
> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
> Term: 211x62
> 4 0
>
> 2)
> Google, Inc.
> Serial Graphics Adapter 06/11/14
> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
> Term: 211x62
> 4 0
> [...empty screen...]
> SeaBIOS (version 1.8.1-20150325_230423-testnode)
> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>
>
> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>
> 3)
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6cb0 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
> ba 2d d4 fe fb 3f
>
> 4)
> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
> 12,sockets=1,cores=12,threads=12 -uuid
> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
> -nodefaults -device sga -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
> PIIX4_PM.disable_s4=1 -boot strict=on -device
> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
> -msg timestamp=on
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 2:47 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-26 2:47 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
Hi Andrey,
Andrey Korolyov <andrey@xdel.ru> writes:
> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>> it appearance is very rare (compared to the number of actual launches)
>> and most probably bounded to the physical characteristics of my
>> production nodes. As soon as I reach any reproducible path for a
>> regular workstation environment, I`ll let everyone know. Also I am
>> starting to think that issue can belong to the particular motherboard
>> firmware revision, despite fact that the CPU microcode is the same
>> everywhere.
I will take the risk and say this - "could it be a processor bug ?" :)
>
> Hello everyone, I`ve managed to reproduce this issue
> *deterministically* with latest seabios with smp fix and 3.18.3. The
> error occuring just *once* per vm until hypervisor reboots, at least
> in my setup, this is definitely crazy...
>
> - launch two VMs (Centos 7 in my case),
> - wait a little while they are booting,
> - attach serial console (I am using virsh list for this exact purpose),
> - issue acpi reboot or reset, does not matter,
> - VM always hangs at boot, most times with sgabios initialization
> string printed out [1], but sometimes it hangs a bit later [2],
> - no matter how many times I try to relaunch the QEMU afterwards, the
> issue does not appear on VM which experienced problem once;
> - trace and sample args can be seen in [3] and [4] respectively.
My system is a Dell R720 dual socket which has 2620v2s. I tried your
setup but couldn't reproduce (my qemu cmdline isn't exactly the same
as yours), although, if you could simplify your command line a bit,
I can try again.
Bandan
> 1)
> Google, Inc.
> Serial Graphics Adapter 06/11/14
> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
> Term: 211x62
> 4 0
>
> 2)
> Google, Inc.
> Serial Graphics Adapter 06/11/14
> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
> Term: 211x62
> 4 0
> [...empty screen...]
> SeaBIOS (version 1.8.1-20150325_230423-testnode)
> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>
>
> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>
> 3)
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6cb0 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
> ba 2d d4 fe fb 3f
>
> 4)
> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
> 12,sockets=1,cores=12,threads=12 -uuid
> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
> -nodefaults -device sga -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
> PIIX4_PM.disable_s4=1 -boot strict=on -device
> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
> -msg timestamp=on
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 2:47 ` Bandan Das
@ 2015-03-26 9:18 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 9:18 UTC (permalink / raw)
To: Bandan Das
Cc: Dr. David Alan Gilbert, Kevin O'Connor, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
[-- Attachment #1: Type: text/plain, Size: 5196 bytes --]
On Thu, Mar 26, 2015 at 5:47 AM, Bandan Das <bsd@redhat.com> wrote:
> Hi Andrey,
>
> Andrey Korolyov <andrey@xdel.ru> writes:
>
>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>>> it appearance is very rare (compared to the number of actual launches)
>>> and most probably bounded to the physical characteristics of my
>>> production nodes. As soon as I reach any reproducible path for a
>>> regular workstation environment, I`ll let everyone know. Also I am
>>> starting to think that issue can belong to the particular motherboard
>>> firmware revision, despite fact that the CPU microcode is the same
>>> everywhere.
>
> I will take the risk and say this - "could it be a processor bug ?" :)
>
>>
>> Hello everyone, I`ve managed to reproduce this issue
>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>> error occuring just *once* per vm until hypervisor reboots, at least
>> in my setup, this is definitely crazy...
>>
>> - launch two VMs (Centos 7 in my case),
>> - wait a little while they are booting,
>> - attach serial console (I am using virsh list for this exact purpose),
>> - issue acpi reboot or reset, does not matter,
>> - VM always hangs at boot, most times with sgabios initialization
>> string printed out [1], but sometimes it hangs a bit later [2],
>> - no matter how many times I try to relaunch the QEMU afterwards, the
>> issue does not appear on VM which experienced problem once;
>> - trace and sample args can be seen in [3] and [4] respectively.
>
> My system is a Dell R720 dual socket which has 2620v2s. I tried your
> setup but couldn't reproduce (my qemu cmdline isn't exactly the same
> as yours), although, if you could simplify your command line a bit,
> I can try again.
>
> Bandan
>
>> 1)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>>
>> 2)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>> [...empty screen...]
>> SeaBIOS (version 1.8.1-20150325_230423-testnode)
>> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>>
>>
>> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>>
>> 3)
>>
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000ef
>> extra data[1]: 80000b0d
>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT= 000f6cb0 00000037
>> IDT= 00000000 000003ff
>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
>> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
>> ba 2d d4 fe fb 3f
>>
>> 4)
>> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
>> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
>> 12,sockets=1,cores=12,threads=12 -uuid
>> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
>> -nodefaults -device sga -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
>> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -chardev
>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
>> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
>> -msg timestamp=on
Hehe, 2.2 works just perfectly but 2.1 isn`t. I`ll bisect the issue in
a next couple of days and post the right commit (but as can remember
none of commits b/w 2.1 and 2.2 can fix simular issue by a purpose).
I`ve attached a reference xml to simplify playing with libvirt if
anyone willing to do so.
[-- Attachment #2: centos71.xml --]
[-- Type: text/xml, Size: 3392 bytes --]
<domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>centos71</name>
<uuid>3c78721f-7317-4f85-bcbe-f5ad46d293a1</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static' cpuset='0-11'>12</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
<loader>/usr/share/seabios/bios.bin</loader>
<boot dev='hd'/>
<bios useserial='yes'/>
</os>
<features>
<acpi/>
<apic eoi='on'/>
<pae/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>SandyBridge</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='12' threads='12'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' io='native'/>
<auth username='qemukvm'>
<secret type='ceph' uuid='ec833836-71ae-f516-0666-7d2f6e123097'/>
</auth>
<source protocol='rbd' name='dev-rack2/centos7-1.raw'>
<host name='10.6.0.1' port='6789'/>
<host name='10.6.0.3' port='6789'/>
<host name='10.6.0.4' port='6789'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<iotune>
<read_bytes_sec>30000000</read_bytes_sec>
<write_bytes_sec>15000000</write_bytes_sec>
<read_iops_sec>100</read_iops_sec>
<write_iops_sec>50</write_iops_sec>
</iotune>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<controller type='usb' index='0' model='nec-xhci'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/centos71.sock'/>
<target type='virtio' name='org.qemu.guest_agent.1'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<memballoon model='none'>
<alias name='balloon0'/>
</memballoon>
</devices>
<seclabel type='none'/>
<qemu:commandline>
<qemu:arg value='-chardev'/>
<qemu:arg value='file,path=/tmp/seabioslog-vm1.log,id=seabios'/>
<qemu:arg value='-device'/>
<qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
</qemu:commandline>
</domain>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 9:18 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 9:18 UTC (permalink / raw)
To: Bandan Das
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 5196 bytes --]
On Thu, Mar 26, 2015 at 5:47 AM, Bandan Das <bsd@redhat.com> wrote:
> Hi Andrey,
>
> Andrey Korolyov <andrey@xdel.ru> writes:
>
>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>>> it appearance is very rare (compared to the number of actual launches)
>>> and most probably bounded to the physical characteristics of my
>>> production nodes. As soon as I reach any reproducible path for a
>>> regular workstation environment, I`ll let everyone know. Also I am
>>> starting to think that issue can belong to the particular motherboard
>>> firmware revision, despite fact that the CPU microcode is the same
>>> everywhere.
>
> I will take the risk and say this - "could it be a processor bug ?" :)
>
>>
>> Hello everyone, I`ve managed to reproduce this issue
>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>> error occuring just *once* per vm until hypervisor reboots, at least
>> in my setup, this is definitely crazy...
>>
>> - launch two VMs (Centos 7 in my case),
>> - wait a little while they are booting,
>> - attach serial console (I am using virsh list for this exact purpose),
>> - issue acpi reboot or reset, does not matter,
>> - VM always hangs at boot, most times with sgabios initialization
>> string printed out [1], but sometimes it hangs a bit later [2],
>> - no matter how many times I try to relaunch the QEMU afterwards, the
>> issue does not appear on VM which experienced problem once;
>> - trace and sample args can be seen in [3] and [4] respectively.
>
> My system is a Dell R720 dual socket which has 2620v2s. I tried your
> setup but couldn't reproduce (my qemu cmdline isn't exactly the same
> as yours), although, if you could simplify your command line a bit,
> I can try again.
>
> Bandan
>
>> 1)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>>
>> 2)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>> [...empty screen...]
>> SeaBIOS (version 1.8.1-20150325_230423-testnode)
>> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>>
>>
>> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>>
>> 3)
>>
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000ef
>> extra data[1]: 80000b0d
>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT= 000f6cb0 00000037
>> IDT= 00000000 000003ff
>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
>> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
>> ba 2d d4 fe fb 3f
>>
>> 4)
>> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
>> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
>> 12,sockets=1,cores=12,threads=12 -uuid
>> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
>> -nodefaults -device sga -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
>> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -chardev
>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
>> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
>> -msg timestamp=on
Hehe, 2.2 works just perfectly but 2.1 isn`t. I`ll bisect the issue in
a next couple of days and post the right commit (but as can remember
none of commits b/w 2.1 and 2.2 can fix simular issue by a purpose).
I`ve attached a reference xml to simplify playing with libvirt if
anyone willing to do so.
[-- Attachment #2: centos71.xml --]
[-- Type: text/xml, Size: 3392 bytes --]
<domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>centos71</name>
<uuid>3c78721f-7317-4f85-bcbe-f5ad46d293a1</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static' cpuset='0-11'>12</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
<loader>/usr/share/seabios/bios.bin</loader>
<boot dev='hd'/>
<bios useserial='yes'/>
</os>
<features>
<acpi/>
<apic eoi='on'/>
<pae/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>SandyBridge</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='12' threads='12'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback' io='native'/>
<auth username='qemukvm'>
<secret type='ceph' uuid='ec833836-71ae-f516-0666-7d2f6e123097'/>
</auth>
<source protocol='rbd' name='dev-rack2/centos7-1.raw'>
<host name='10.6.0.1' port='6789'/>
<host name='10.6.0.3' port='6789'/>
<host name='10.6.0.4' port='6789'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<iotune>
<read_bytes_sec>30000000</read_bytes_sec>
<write_bytes_sec>15000000</write_bytes_sec>
<read_iops_sec>100</read_iops_sec>
<write_iops_sec>50</write_iops_sec>
</iotune>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<controller type='usb' index='0' model='nec-xhci'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/centos71.sock'/>
<target type='virtio' name='org.qemu.guest_agent.1'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<memballoon model='none'>
<alias name='balloon0'/>
</memballoon>
</devices>
<seclabel type='none'/>
<qemu:commandline>
<qemu:arg value='-chardev'/>
<qemu:arg value='file,path=/tmp/seabioslog-vm1.log,id=seabios'/>
<qemu:arg value='-device'/>
<qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
</qemu:commandline>
</domain>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 9:18 ` Andrey Korolyov
@ 2015-03-26 15:05 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 15:05 UTC (permalink / raw)
To: Bandan Das
Cc: Dr. David Alan Gilbert, Kevin O'Connor, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
On Thu, Mar 26, 2015 at 12:18 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Thu, Mar 26, 2015 at 5:47 AM, Bandan Das <bsd@redhat.com> wrote:
>> Hi Andrey,
>>
>> Andrey Korolyov <andrey@xdel.ru> writes:
>>
>>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>>>> it appearance is very rare (compared to the number of actual launches)
>>>> and most probably bounded to the physical characteristics of my
>>>> production nodes. As soon as I reach any reproducible path for a
>>>> regular workstation environment, I`ll let everyone know. Also I am
>>>> starting to think that issue can belong to the particular motherboard
>>>> firmware revision, despite fact that the CPU microcode is the same
>>>> everywhere.
>>
>> I will take the risk and say this - "could it be a processor bug ?" :)
>>
>>>
>>> Hello everyone, I`ve managed to reproduce this issue
>>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>>> error occuring just *once* per vm until hypervisor reboots, at least
>>> in my setup, this is definitely crazy...
>>>
>>> - launch two VMs (Centos 7 in my case),
>>> - wait a little while they are booting,
>>> - attach serial console (I am using virsh list for this exact purpose),
>>> - issue acpi reboot or reset, does not matter,
>>> - VM always hangs at boot, most times with sgabios initialization
>>> string printed out [1], but sometimes it hangs a bit later [2],
>>> - no matter how many times I try to relaunch the QEMU afterwards, the
>>> issue does not appear on VM which experienced problem once;
>>> - trace and sample args can be seen in [3] and [4] respectively.
>>
>> My system is a Dell R720 dual socket which has 2620v2s. I tried your
>> setup but couldn't reproduce (my qemu cmdline isn't exactly the same
>> as yours), although, if you could simplify your command line a bit,
>> I can try again.
>>
>> Bandan
>>
>>> 1)
>>> Google, Inc.
>>> Serial Graphics Adapter 06/11/14
>>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>>> Term: 211x62
>>> 4 0
>>>
>>> 2)
>>> Google, Inc.
>>> Serial Graphics Adapter 06/11/14
>>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>>> Term: 211x62
>>> 4 0
>>> [...empty screen...]
>>> SeaBIOS (version 1.8.1-20150325_230423-testnode)
>>> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>>>
>>>
>>> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>>>
>>> 3)
>>>
>>> KVM internal error. Suberror: 2
>>> extra data[0]: 800000ef
>>> extra data[1]: 80000b0d
>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
>>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>> ES =0000 00000000 0000ffff 00009300
>>> CS =f000 000f0000 0000ffff 00009b00
>>> SS =0000 00000000 0000ffff 00009300
>>> DS =0000 00000000 0000ffff 00009300
>>> FS =0000 00000000 0000ffff 00009300
>>> GS =0000 00000000 0000ffff 00009300
>>> LDT=0000 00000000 0000ffff 00008200
>>> TR =0000 00000000 0000ffff 00008b00
>>> GDT= 000f6cb0 00000037
>>> IDT= 00000000 000003ff
>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>> DR3=0000000000000000
>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>> EFER=0000000000000000
>>> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
>>> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
>>> ba 2d d4 fe fb 3f
>>>
>>> 4)
>>> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
>>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
>>> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
>>> 12,sockets=1,cores=12,threads=12 -uuid
>>> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
>>> -nodefaults -device sga -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>>> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>>> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
>>> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>> -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -chardev
>>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
>>> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
>>> -msg timestamp=on
>
> Hehe, 2.2 works just perfectly but 2.1 isn`t. I`ll bisect the issue in
> a next couple of days and post the right commit (but as can remember
> none of commits b/w 2.1 and 2.2 can fix simular issue by a purpose).
> I`ve attached a reference xml to simplify playing with libvirt if
> anyone willing to do so.
Sorry, 2.2 hangs as well but more rarely. Looks like it is important
to conduct the test sequence on a freshly booted host, as issue tends
to not reappear during the hypervisor boot cycle. Please let me know
if host kernel config is needed, for example if nobody will be able to
reproduce this.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 15:05 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 15:05 UTC (permalink / raw)
To: Bandan Das
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 12:18 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Thu, Mar 26, 2015 at 5:47 AM, Bandan Das <bsd@redhat.com> wrote:
>> Hi Andrey,
>>
>> Andrey Korolyov <andrey@xdel.ru> writes:
>>
>>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>>>> it appearance is very rare (compared to the number of actual launches)
>>>> and most probably bounded to the physical characteristics of my
>>>> production nodes. As soon as I reach any reproducible path for a
>>>> regular workstation environment, I`ll let everyone know. Also I am
>>>> starting to think that issue can belong to the particular motherboard
>>>> firmware revision, despite fact that the CPU microcode is the same
>>>> everywhere.
>>
>> I will take the risk and say this - "could it be a processor bug ?" :)
>>
>>>
>>> Hello everyone, I`ve managed to reproduce this issue
>>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>>> error occuring just *once* per vm until hypervisor reboots, at least
>>> in my setup, this is definitely crazy...
>>>
>>> - launch two VMs (Centos 7 in my case),
>>> - wait a little while they are booting,
>>> - attach serial console (I am using virsh list for this exact purpose),
>>> - issue acpi reboot or reset, does not matter,
>>> - VM always hangs at boot, most times with sgabios initialization
>>> string printed out [1], but sometimes it hangs a bit later [2],
>>> - no matter how many times I try to relaunch the QEMU afterwards, the
>>> issue does not appear on VM which experienced problem once;
>>> - trace and sample args can be seen in [3] and [4] respectively.
>>
>> My system is a Dell R720 dual socket which has 2620v2s. I tried your
>> setup but couldn't reproduce (my qemu cmdline isn't exactly the same
>> as yours), although, if you could simplify your command line a bit,
>> I can try again.
>>
>> Bandan
>>
>>> 1)
>>> Google, Inc.
>>> Serial Graphics Adapter 06/11/14
>>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>>> Term: 211x62
>>> 4 0
>>>
>>> 2)
>>> Google, Inc.
>>> Serial Graphics Adapter 06/11/14
>>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>>> Term: 211x62
>>> 4 0
>>> [...empty screen...]
>>> SeaBIOS (version 1.8.1-20150325_230423-testnode)
>>> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>>>
>>>
>>> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>>>
>>> 3)
>>>
>>> KVM internal error. Suberror: 2
>>> extra data[0]: 800000ef
>>> extra data[1]: 80000b0d
>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
>>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>> ES =0000 00000000 0000ffff 00009300
>>> CS =f000 000f0000 0000ffff 00009b00
>>> SS =0000 00000000 0000ffff 00009300
>>> DS =0000 00000000 0000ffff 00009300
>>> FS =0000 00000000 0000ffff 00009300
>>> GS =0000 00000000 0000ffff 00009300
>>> LDT=0000 00000000 0000ffff 00008200
>>> TR =0000 00000000 0000ffff 00008b00
>>> GDT= 000f6cb0 00000037
>>> IDT= 00000000 000003ff
>>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>> DR3=0000000000000000
>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>> EFER=0000000000000000
>>> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
>>> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
>>> ba 2d d4 fe fb 3f
>>>
>>> 4)
>>> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
>>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
>>> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
>>> 12,sockets=1,cores=12,threads=12 -uuid
>>> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
>>> -nodefaults -device sga -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>>> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>>> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
>>> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>> -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -chardev
>>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
>>> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
>>> -msg timestamp=on
>
> Hehe, 2.2 works just perfectly but 2.1 isn`t. I`ll bisect the issue in
> a next couple of days and post the right commit (but as can remember
> none of commits b/w 2.1 and 2.2 can fix simular issue by a purpose).
> I`ve attached a reference xml to simplify playing with libvirt if
> anyone willing to do so.
Sorry, 2.2 hangs as well but more rarely. Looks like it is important
to conduct the test sequence on a freshly booted host, as issue tends
to not reappear during the hypervisor boot cycle. Please let me know
if host kernel config is needed, for example if nobody will be able to
reproduce this.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 0:05 ` Kevin O'Connor
@ 2015-03-26 15:58 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 15:58 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-25 20:05-0400, Kevin O'Connor:
> On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> > Thanks, strangely the reboot is always failing now and always reaching
> > seabios greeting. May be prints straightened up a race (e.g. it is not
> > int19 problem really).
> >
> > object file part:
> >
> > 0000d331 <irq_trampoline_0x19>:
> > irq_trampoline_0x19():
> > /root/seabios-1.8.1/src/romlayout.S:195
> > d331: cd 19 int $0x19
> > d333: cb lretw
>
> [...]
> > Jump to int19 (vector=f000e6f2)
>
> Thanks. So, it dies on the "int $0x19" instruction itself. The
> vector looks correct and I don't see anything in the cpu register
> state that looks wrong. Maybe one of the kvm developers will have an
> idea what could cause a fault there.
The place agrees with the "<cd> 19 cb" part of KVM error output.
Suberror 2 means that we were interrupted while delivering a vector,
here it is disected: (delivering 'vect_info')
vect_info (extra data[0]: 800000ef)
- vector 0xef
- INTR_TYPE_EXT_INTR (0x000)
- no error code (0x000)
- valid (0x80000000)
intr_info (extra data[1]: 80000b0d)
- #GP (0x0d)
- INTR_TYPE_HARD_EXCEPTION (0x300)
- error code on stack (0x800) [Hunk at the bottom exposes it.]
- valid (0x80000000)
Notice the 0xef. My best hypothesis so far is that we fail at resetting
devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
(The bug happens at the first place that enables interrupts.)
This should be handled without an internal error, though.
(It's possible we are hitting two bugs here ...)
Can you provide KVM event trace as well? (`trace-cmd record -e 'kvm*'`)
Thanks.
---
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 50c675b46901..541a29476a56 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5088,9 +5088,10 @@ static int handle_exception(struct kvm_vcpu *vcpu)
!(is_page_fault(intr_info) && !(error_code & PFERR_RSVD_MASK))) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_SIMUL_EX;
- vcpu->run->internal.ndata = 2;
+ vcpu->run->internal.ndata = 3;
vcpu->run->internal.data[0] = vect_info;
vcpu->run->internal.data[1] = intr_info;
+ vcpu->run->internal.data[2] = error_code;
return 0;
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 15:58 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 15:58 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert,
Bandan Das, Gerd Hoffmann, Paolo Bonzini
2015-03-25 20:05-0400, Kevin O'Connor:
> On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> > Thanks, strangely the reboot is always failing now and always reaching
> > seabios greeting. May be prints straightened up a race (e.g. it is not
> > int19 problem really).
> >
> > object file part:
> >
> > 0000d331 <irq_trampoline_0x19>:
> > irq_trampoline_0x19():
> > /root/seabios-1.8.1/src/romlayout.S:195
> > d331: cd 19 int $0x19
> > d333: cb lretw
>
> [...]
> > Jump to int19 (vector=f000e6f2)
>
> Thanks. So, it dies on the "int $0x19" instruction itself. The
> vector looks correct and I don't see anything in the cpu register
> state that looks wrong. Maybe one of the kvm developers will have an
> idea what could cause a fault there.
The place agrees with the "<cd> 19 cb" part of KVM error output.
Suberror 2 means that we were interrupted while delivering a vector,
here it is disected: (delivering 'vect_info')
vect_info (extra data[0]: 800000ef)
- vector 0xef
- INTR_TYPE_EXT_INTR (0x000)
- no error code (0x000)
- valid (0x80000000)
intr_info (extra data[1]: 80000b0d)
- #GP (0x0d)
- INTR_TYPE_HARD_EXCEPTION (0x300)
- error code on stack (0x800) [Hunk at the bottom exposes it.]
- valid (0x80000000)
Notice the 0xef. My best hypothesis so far is that we fail at resetting
devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
(The bug happens at the first place that enables interrupts.)
This should be handled without an internal error, though.
(It's possible we are hitting two bugs here ...)
Can you provide KVM event trace as well? (`trace-cmd record -e 'kvm*'`)
Thanks.
---
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 50c675b46901..541a29476a56 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5088,9 +5088,10 @@ static int handle_exception(struct kvm_vcpu *vcpu)
!(is_page_fault(intr_info) && !(error_code & PFERR_RSVD_MASK))) {
vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_SIMUL_EX;
- vcpu->run->internal.ndata = 2;
+ vcpu->run->internal.ndata = 3;
vcpu->run->internal.data[0] = vect_info;
vcpu->run->internal.data[1] = intr_info;
+ vcpu->run->internal.data[2] = error_code;
return 0;
}
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-26 15:58 ` Radim Krčmář
@ 2015-03-26 16:36 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 16:36 UTC (permalink / raw)
To: Radim Krčmář
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert,
Bandan Das, Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
> 2015-03-25 20:05-0400, Kevin O'Connor:
> > On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> > > Thanks, strangely the reboot is always failing now and always reaching
> > > seabios greeting. May be prints straightened up a race (e.g. it is not
> > > int19 problem really).
> > >
> > > object file part:
> > >
> > > 0000d331 <irq_trampoline_0x19>:
> > > irq_trampoline_0x19():
> > > /root/seabios-1.8.1/src/romlayout.S:195
> > > d331: cd 19 int $0x19
> > > d333: cb lretw
> >
> > [...]
> > > Jump to int19 (vector=f000e6f2)
> >
> > Thanks. So, it dies on the "int $0x19" instruction itself. The
> > vector looks correct and I don't see anything in the cpu register
> > state that looks wrong. Maybe one of the kvm developers will have an
> > idea what could cause a fault there.
>
> The place agrees with the "<cd> 19 cb" part of KVM error output.
> Suberror 2 means that we were interrupted while delivering a vector,
> here it is disected: (delivering 'vect_info')
>
> vect_info (extra data[0]: 800000ef)
> - vector 0xef
> - INTR_TYPE_EXT_INTR (0x000)
> - no error code (0x000)
> - valid (0x80000000)
>
> intr_info (extra data[1]: 80000b0d)
> - #GP (0x0d)
> - INTR_TYPE_HARD_EXCEPTION (0x300)
> - error code on stack (0x800) [Hunk at the bottom exposes it.]
> - valid (0x80000000)
Thanks for the background info.
> Notice the 0xef. My best hypothesis so far is that we fail at resetting
> devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
> (The bug happens at the first place that enables interrupts.)
FYI, the "int $0x19" isn't the first place SeaBIOS will enable
interrupts. Each screen print (every character in the seabios banner
and uuid string) will call the vga bios (int $0x10) with irqs enabled
(see output.c:screenc).
Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
simple "iretw".
Things that are unusual about the "int $0x19" call:
- it is likely the first place that the cpu is transitioned into
16bit real mode as opposed to "big real" mode. (That is, the
first place interrupts are enabled with the segment limits set to
0xffff.)
- it's right after the fw/shadow.c:make_bios_readonly() call, which
attempts to configures the memory at 0xf0000-0x100000 as
read-only. That code also issues a wbinvd() call.
I'm not sure if the crash always happens at the "int $0x19" location
though. Andrey, does the crash always happen with "EIP=d331" and/or
with "Code=... <cd> 19"?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 16:36 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 16:36 UTC (permalink / raw)
To: Radim Krčmář
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert,
Bandan Das, Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
> 2015-03-25 20:05-0400, Kevin O'Connor:
> > On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
> > > Thanks, strangely the reboot is always failing now and always reaching
> > > seabios greeting. May be prints straightened up a race (e.g. it is not
> > > int19 problem really).
> > >
> > > object file part:
> > >
> > > 0000d331 <irq_trampoline_0x19>:
> > > irq_trampoline_0x19():
> > > /root/seabios-1.8.1/src/romlayout.S:195
> > > d331: cd 19 int $0x19
> > > d333: cb lretw
> >
> > [...]
> > > Jump to int19 (vector=f000e6f2)
> >
> > Thanks. So, it dies on the "int $0x19" instruction itself. The
> > vector looks correct and I don't see anything in the cpu register
> > state that looks wrong. Maybe one of the kvm developers will have an
> > idea what could cause a fault there.
>
> The place agrees with the "<cd> 19 cb" part of KVM error output.
> Suberror 2 means that we were interrupted while delivering a vector,
> here it is disected: (delivering 'vect_info')
>
> vect_info (extra data[0]: 800000ef)
> - vector 0xef
> - INTR_TYPE_EXT_INTR (0x000)
> - no error code (0x000)
> - valid (0x80000000)
>
> intr_info (extra data[1]: 80000b0d)
> - #GP (0x0d)
> - INTR_TYPE_HARD_EXCEPTION (0x300)
> - error code on stack (0x800) [Hunk at the bottom exposes it.]
> - valid (0x80000000)
Thanks for the background info.
> Notice the 0xef. My best hypothesis so far is that we fail at resetting
> devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
> (The bug happens at the first place that enables interrupts.)
FYI, the "int $0x19" isn't the first place SeaBIOS will enable
interrupts. Each screen print (every character in the seabios banner
and uuid string) will call the vga bios (int $0x10) with irqs enabled
(see output.c:screenc).
Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
simple "iretw".
Things that are unusual about the "int $0x19" call:
- it is likely the first place that the cpu is transitioned into
16bit real mode as opposed to "big real" mode. (That is, the
first place interrupts are enabled with the segment limits set to
0xffff.)
- it's right after the fw/shadow.c:make_bios_readonly() call, which
attempts to configures the memory at 0xf0000-0x100000 as
read-only. That code also issues a wbinvd() call.
I'm not sure if the crash always happens at the "int $0x19" location
though. Andrey, does the crash always happen with "EIP=d331" and/or
with "Code=... <cd> 19"?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 16:36 ` [Qemu-devel] " Kevin O'Connor
@ 2015-03-26 16:48 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 16:48 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Radim Krčmář,
Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
>> 2015-03-25 20:05-0400, Kevin O'Connor:
>> > On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
>> > > Thanks, strangely the reboot is always failing now and always reaching
>> > > seabios greeting. May be prints straightened up a race (e.g. it is not
>> > > int19 problem really).
>> > >
>> > > object file part:
>> > >
>> > > 0000d331 <irq_trampoline_0x19>:
>> > > irq_trampoline_0x19():
>> > > /root/seabios-1.8.1/src/romlayout.S:195
>> > > d331: cd 19 int $0x19
>> > > d333: cb lretw
>> >
>> > [...]
>> > > Jump to int19 (vector=f000e6f2)
>> >
>> > Thanks. So, it dies on the "int $0x19" instruction itself. The
>> > vector looks correct and I don't see anything in the cpu register
>> > state that looks wrong. Maybe one of the kvm developers will have an
>> > idea what could cause a fault there.
>>
>> The place agrees with the "<cd> 19 cb" part of KVM error output.
>> Suberror 2 means that we were interrupted while delivering a vector,
>> here it is disected: (delivering 'vect_info')
>>
>> vect_info (extra data[0]: 800000ef)
>> - vector 0xef
>> - INTR_TYPE_EXT_INTR (0x000)
>> - no error code (0x000)
>> - valid (0x80000000)
>>
>> intr_info (extra data[1]: 80000b0d)
>> - #GP (0x0d)
>> - INTR_TYPE_HARD_EXCEPTION (0x300)
>> - error code on stack (0x800) [Hunk at the bottom exposes it.]
>> - valid (0x80000000)
>
> Thanks for the background info.
>
>> Notice the 0xef. My best hypothesis so far is that we fail at resetting
>> devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
>> (The bug happens at the first place that enables interrupts.)
>
> FYI, the "int $0x19" isn't the first place SeaBIOS will enable
> interrupts. Each screen print (every character in the seabios banner
> and uuid string) will call the vga bios (int $0x10) with irqs enabled
> (see output.c:screenc).
>
> Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
> simple "iretw".
>
> Things that are unusual about the "int $0x19" call:
> - it is likely the first place that the cpu is transitioned into
> 16bit real mode as opposed to "big real" mode. (That is, the
> first place interrupts are enabled with the segment limits set to
> 0xffff.)
> - it's right after the fw/shadow.c:make_bios_readonly() call, which
> attempts to configures the memory at 0xf0000-0x100000 as
> read-only. That code also issues a wbinvd() call.
>
> I'm not sure if the crash always happens at the "int $0x19" location
> though. Andrey, does the crash always happen with "EIP=d331" and/or
> with "Code=... <cd> 19"?
>
> -Kevin
There are also rare occurences for d3f9 (in the middle of ep) and d334
ep (less than one tenth of events for both). I`ll post a sample event
capture with and without Radim`s proposed patch maybe today or
tomorrow.
/root/seabios-1.8.1/src/romlayout.S:289
d3eb: 66 50 pushl %eax
d3ed: 66 51 pushl %ecx
d3ef: 66 52 pushl %edx
d3f1: 66 53 pushl %ebx
d3f3: 66 55 pushl %ebp
d3f5: 66 56 pushl %esi
d3f7: 66 57 pushl %edi
d3f9: 06 pushw %es
d3fa: 1e pushw %ds
0000d334 <irq_trampoline_0x1c>:
irq_trampoline_0x1c():
/root/seabios-1.8.1/src/romlayout.S:196
d334: cd 1c int $0x1c
d336: cb lretw
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 16:48 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 16:48 UTC (permalink / raw)
To: Kevin O'Connor
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
>> 2015-03-25 20:05-0400, Kevin O'Connor:
>> > On Thu, Mar 26, 2015 at 02:35:58AM +0300, Andrey Korolyov wrote:
>> > > Thanks, strangely the reboot is always failing now and always reaching
>> > > seabios greeting. May be prints straightened up a race (e.g. it is not
>> > > int19 problem really).
>> > >
>> > > object file part:
>> > >
>> > > 0000d331 <irq_trampoline_0x19>:
>> > > irq_trampoline_0x19():
>> > > /root/seabios-1.8.1/src/romlayout.S:195
>> > > d331: cd 19 int $0x19
>> > > d333: cb lretw
>> >
>> > [...]
>> > > Jump to int19 (vector=f000e6f2)
>> >
>> > Thanks. So, it dies on the "int $0x19" instruction itself. The
>> > vector looks correct and I don't see anything in the cpu register
>> > state that looks wrong. Maybe one of the kvm developers will have an
>> > idea what could cause a fault there.
>>
>> The place agrees with the "<cd> 19 cb" part of KVM error output.
>> Suberror 2 means that we were interrupted while delivering a vector,
>> here it is disected: (delivering 'vect_info')
>>
>> vect_info (extra data[0]: 800000ef)
>> - vector 0xef
>> - INTR_TYPE_EXT_INTR (0x000)
>> - no error code (0x000)
>> - valid (0x80000000)
>>
>> intr_info (extra data[1]: 80000b0d)
>> - #GP (0x0d)
>> - INTR_TYPE_HARD_EXCEPTION (0x300)
>> - error code on stack (0x800) [Hunk at the bottom exposes it.]
>> - valid (0x80000000)
>
> Thanks for the background info.
>
>> Notice the 0xef. My best hypothesis so far is that we fail at resetting
>> devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
>> (The bug happens at the first place that enables interrupts.)
>
> FYI, the "int $0x19" isn't the first place SeaBIOS will enable
> interrupts. Each screen print (every character in the seabios banner
> and uuid string) will call the vga bios (int $0x10) with irqs enabled
> (see output.c:screenc).
>
> Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
> simple "iretw".
>
> Things that are unusual about the "int $0x19" call:
> - it is likely the first place that the cpu is transitioned into
> 16bit real mode as opposed to "big real" mode. (That is, the
> first place interrupts are enabled with the segment limits set to
> 0xffff.)
> - it's right after the fw/shadow.c:make_bios_readonly() call, which
> attempts to configures the memory at 0xf0000-0x100000 as
> read-only. That code also issues a wbinvd() call.
>
> I'm not sure if the crash always happens at the "int $0x19" location
> though. Andrey, does the crash always happen with "EIP=d331" and/or
> with "Code=... <cd> 19"?
>
> -Kevin
There are also rare occurences for d3f9 (in the middle of ep) and d334
ep (less than one tenth of events for both). I`ll post a sample event
capture with and without Radim`s proposed patch maybe today or
tomorrow.
/root/seabios-1.8.1/src/romlayout.S:289
d3eb: 66 50 pushl %eax
d3ed: 66 51 pushl %ecx
d3ef: 66 52 pushl %edx
d3f1: 66 53 pushl %ebx
d3f3: 66 55 pushl %ebp
d3f5: 66 56 pushl %esi
d3f7: 66 57 pushl %edi
d3f9: 06 pushw %es
d3fa: 1e pushw %ds
0000d334 <irq_trampoline_0x1c>:
irq_trampoline_0x1c():
/root/seabios-1.8.1/src/romlayout.S:196
d334: cd 1c int $0x1c
d336: cb lretw
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 16:48 ` Andrey Korolyov
@ 2015-03-26 17:06 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 17:06 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Radim Krčmář,
Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> > I'm not sure if the crash always happens at the "int $0x19" location
> > though. Andrey, does the crash always happen with "EIP=d331" and/or
> > with "Code=... <cd> 19"?
>
> There are also rare occurences for d3f9 (in the middle of ep) and d334
> ep (less than one tenth of events for both). I`ll post a sample event
> capture with and without Radim`s proposed patch maybe today or
> tomorrow.
>
> /root/seabios-1.8.1/src/romlayout.S:289
> d3eb: 66 50 pushl %eax
> d3ed: 66 51 pushl %ecx
> d3ef: 66 52 pushl %edx
> d3f1: 66 53 pushl %ebx
> d3f3: 66 55 pushl %ebp
> d3f5: 66 56 pushl %esi
> d3f7: 66 57 pushl %edi
> d3f9: 06 pushw %es
> d3fa: 1e pushw %ds
>
> 0000d334 <irq_trampoline_0x1c>:
> irq_trampoline_0x1c():
> /root/seabios-1.8.1/src/romlayout.S:196
> d334: cd 1c int $0x1c
> d336: cb lretw
Thanks. The d334 looks very similar to the d331 report (code=<cd>
1c). That path could happen during post (big real mode) or
immiediately after post (real mode).
The d3f9 report does not look like the others - interrupts are
disabled there. If you still have the error logs, can you post the
full kvm crash report for d3f9?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:06 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 17:06 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> > I'm not sure if the crash always happens at the "int $0x19" location
> > though. Andrey, does the crash always happen with "EIP=d331" and/or
> > with "Code=... <cd> 19"?
>
> There are also rare occurences for d3f9 (in the middle of ep) and d334
> ep (less than one tenth of events for both). I`ll post a sample event
> capture with and without Radim`s proposed patch maybe today or
> tomorrow.
>
> /root/seabios-1.8.1/src/romlayout.S:289
> d3eb: 66 50 pushl %eax
> d3ed: 66 51 pushl %ecx
> d3ef: 66 52 pushl %edx
> d3f1: 66 53 pushl %ebx
> d3f3: 66 55 pushl %ebp
> d3f5: 66 56 pushl %esi
> d3f7: 66 57 pushl %edi
> d3f9: 06 pushw %es
> d3fa: 1e pushw %ds
>
> 0000d334 <irq_trampoline_0x1c>:
> irq_trampoline_0x1c():
> /root/seabios-1.8.1/src/romlayout.S:196
> d334: cd 1c int $0x1c
> d336: cb lretw
Thanks. The d334 looks very similar to the d331 report (code=<cd>
1c). That path could happen during post (big real mode) or
immiediately after post (real mode).
The d3f9 report does not look like the others - interrupts are
disabled there. If you still have the error logs, can you post the
full kvm crash report for d3f9?
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 17:06 ` Kevin O'Connor
@ 2015-03-26 17:08 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 17:08 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Radim Krčmář,
Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
>> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> > I'm not sure if the crash always happens at the "int $0x19" location
>> > though. Andrey, does the crash always happen with "EIP=d331" and/or
>> > with "Code=... <cd> 19"?
>>
>> There are also rare occurences for d3f9 (in the middle of ep) and d334
>> ep (less than one tenth of events for both). I`ll post a sample event
>> capture with and without Radim`s proposed patch maybe today or
>> tomorrow.
>>
>> /root/seabios-1.8.1/src/romlayout.S:289
>> d3eb: 66 50 pushl %eax
>> d3ed: 66 51 pushl %ecx
>> d3ef: 66 52 pushl %edx
>> d3f1: 66 53 pushl %ebx
>> d3f3: 66 55 pushl %ebp
>> d3f5: 66 56 pushl %esi
>> d3f7: 66 57 pushl %edi
>> d3f9: 06 pushw %es
>> d3fa: 1e pushw %ds
>>
>> 0000d334 <irq_trampoline_0x1c>:
>> irq_trampoline_0x1c():
>> /root/seabios-1.8.1/src/romlayout.S:196
>> d334: cd 1c int $0x1c
>> d336: cb lretw
>
> Thanks. The d334 looks very similar to the d331 report (code=<cd>
> 1c). That path could happen during post (big real mode) or
> immiediately after post (real mode).
>
> The d3f9 report does not look like the others - interrupts are
> disabled there. If you still have the error logs, can you post the
> full kvm crash report for d3f9?
>
Here you go:
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6e98 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
b8 00 e0 00 00 8e
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:08 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 17:08 UTC (permalink / raw)
To: Kevin O'Connor
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
>> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> > I'm not sure if the crash always happens at the "int $0x19" location
>> > though. Andrey, does the crash always happen with "EIP=d331" and/or
>> > with "Code=... <cd> 19"?
>>
>> There are also rare occurences for d3f9 (in the middle of ep) and d334
>> ep (less than one tenth of events for both). I`ll post a sample event
>> capture with and without Radim`s proposed patch maybe today or
>> tomorrow.
>>
>> /root/seabios-1.8.1/src/romlayout.S:289
>> d3eb: 66 50 pushl %eax
>> d3ed: 66 51 pushl %ecx
>> d3ef: 66 52 pushl %edx
>> d3f1: 66 53 pushl %ebx
>> d3f3: 66 55 pushl %ebp
>> d3f5: 66 56 pushl %esi
>> d3f7: 66 57 pushl %edi
>> d3f9: 06 pushw %es
>> d3fa: 1e pushw %ds
>>
>> 0000d334 <irq_trampoline_0x1c>:
>> irq_trampoline_0x1c():
>> /root/seabios-1.8.1/src/romlayout.S:196
>> d334: cd 1c int $0x1c
>> d336: cb lretw
>
> Thanks. The d334 looks very similar to the d331 report (code=<cd>
> 1c). That path could happen during post (big real mode) or
> immiediately after post (real mode).
>
> The d3f9 report does not look like the others - interrupts are
> disabled there. If you still have the error logs, can you post the
> full kvm crash report for d3f9?
>
Here you go:
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
EAX=00000003 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006cd4
EIP=0000d3f9 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6e98 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
b8 00 e0 00 00 8e
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 17:08 ` Andrey Korolyov
@ 2015-03-26 17:18 ` Kevin O'Connor
-1 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 17:18 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Radim Krčmář,
Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 08:08:52PM +0300, Andrey Korolyov wrote:
> On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> > On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
> >> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> >> > I'm not sure if the crash always happens at the "int $0x19" location
> >> > though. Andrey, does the crash always happen with "EIP=d331" and/or
> >> > with "Code=... <cd> 19"?
> >>
> >> There are also rare occurences for d3f9 (in the middle of ep) and d334
> >> ep (less than one tenth of events for both). I`ll post a sample event
> >> capture with and without Radim`s proposed patch maybe today or
> >> tomorrow.
> >>
> >> /root/seabios-1.8.1/src/romlayout.S:289
> >> d3eb: 66 50 pushl %eax
> >> d3ed: 66 51 pushl %ecx
> >> d3ef: 66 52 pushl %edx
> >> d3f1: 66 53 pushl %ebx
> >> d3f3: 66 55 pushl %ebp
> >> d3f5: 66 56 pushl %esi
> >> d3f7: 66 57 pushl %edi
> >> d3f9: 06 pushw %es
> >> d3fa: 1e pushw %ds
> >>
> >> 0000d334 <irq_trampoline_0x1c>:
> >> irq_trampoline_0x1c():
> >> /root/seabios-1.8.1/src/romlayout.S:196
> >> d334: cd 1c int $0x1c
> >> d336: cb lretw
> >
> > Thanks. The d334 looks very similar to the d331 report (code=<cd>
> > 1c). That path could happen during post (big real mode) or
> > immiediately after post (real mode).
> >
> > The d3f9 report does not look like the others - interrupts are
> > disabled there. If you still have the error logs, can you post the
> > full kvm crash report for d3f9?
> >
>
> Here you go:
Thanks. While we're at, can you verify if all your reports are
showing the cpu in "real mode". That is, do they all have "0000ffff"
in the third column of the segment registers - as in:
> ES =0000 00000000 0000ffff 00009300
[...]
> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> b8 00 e0 00 00 8e
KVM reports the code as "int $0x10" here. Was it possible this report
was from a different build of seabios (that had a different code
layout)?
Interestingly, this "int $0x10" is also in real-mode and not "big real
mode", so I think it would have occurred after post completed.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:18 ` Kevin O'Connor
0 siblings, 0 replies; 157+ messages in thread
From: Kevin O'Connor @ 2015-03-26 17:18 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 26, 2015 at 08:08:52PM +0300, Andrey Korolyov wrote:
> On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> > On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
> >> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> >> > I'm not sure if the crash always happens at the "int $0x19" location
> >> > though. Andrey, does the crash always happen with "EIP=d331" and/or
> >> > with "Code=... <cd> 19"?
> >>
> >> There are also rare occurences for d3f9 (in the middle of ep) and d334
> >> ep (less than one tenth of events for both). I`ll post a sample event
> >> capture with and without Radim`s proposed patch maybe today or
> >> tomorrow.
> >>
> >> /root/seabios-1.8.1/src/romlayout.S:289
> >> d3eb: 66 50 pushl %eax
> >> d3ed: 66 51 pushl %ecx
> >> d3ef: 66 52 pushl %edx
> >> d3f1: 66 53 pushl %ebx
> >> d3f3: 66 55 pushl %ebp
> >> d3f5: 66 56 pushl %esi
> >> d3f7: 66 57 pushl %edi
> >> d3f9: 06 pushw %es
> >> d3fa: 1e pushw %ds
> >>
> >> 0000d334 <irq_trampoline_0x1c>:
> >> irq_trampoline_0x1c():
> >> /root/seabios-1.8.1/src/romlayout.S:196
> >> d334: cd 1c int $0x1c
> >> d336: cb lretw
> >
> > Thanks. The d334 looks very similar to the d331 report (code=<cd>
> > 1c). That path could happen during post (big real mode) or
> > immiediately after post (real mode).
> >
> > The d3f9 report does not look like the others - interrupts are
> > disabled there. If you still have the error logs, can you post the
> > full kvm crash report for d3f9?
> >
>
> Here you go:
Thanks. While we're at, can you verify if all your reports are
showing the cpu in "real mode". That is, do they all have "0000ffff"
in the third column of the segment registers - as in:
> ES =0000 00000000 0000ffff 00009300
[...]
> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
> b8 00 e0 00 00 8e
KVM reports the code as "int $0x10" here. Was it possible this report
was from a different build of seabios (that had a different code
layout)?
Interestingly, this "int $0x10" is also in real-mode and not "big real
mode", so I think it would have occurred after post completed.
-Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 17:18 ` Kevin O'Connor
@ 2015-03-26 17:33 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 17:33 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Radim Krčmář,
Dr. David Alan Gilbert, Bandan Das, Paolo Bonzini, Gerd Hoffmann,
qemu-devel, kvm
On Thu, Mar 26, 2015 at 8:18 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 08:08:52PM +0300, Andrey Korolyov wrote:
>> On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> > On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
>> >> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> >> > I'm not sure if the crash always happens at the "int $0x19" location
>> >> > though. Andrey, does the crash always happen with "EIP=d331" and/or
>> >> > with "Code=... <cd> 19"?
>> >>
>> >> There are also rare occurences for d3f9 (in the middle of ep) and d334
>> >> ep (less than one tenth of events for both). I`ll post a sample event
>> >> capture with and without Radim`s proposed patch maybe today or
>> >> tomorrow.
>> >>
>> >> /root/seabios-1.8.1/src/romlayout.S:289
>> >> d3eb: 66 50 pushl %eax
>> >> d3ed: 66 51 pushl %ecx
>> >> d3ef: 66 52 pushl %edx
>> >> d3f1: 66 53 pushl %ebx
>> >> d3f3: 66 55 pushl %ebp
>> >> d3f5: 66 56 pushl %esi
>> >> d3f7: 66 57 pushl %edi
>> >> d3f9: 06 pushw %es
>> >> d3fa: 1e pushw %ds
>> >>
>> >> 0000d334 <irq_trampoline_0x1c>:
>> >> irq_trampoline_0x1c():
>> >> /root/seabios-1.8.1/src/romlayout.S:196
>> >> d334: cd 1c int $0x1c
>> >> d336: cb lretw
>> >
>> > Thanks. The d334 looks very similar to the d331 report (code=<cd>
>> > 1c). That path could happen during post (big real mode) or
>> > immiediately after post (real mode).
>> >
>> > The d3f9 report does not look like the others - interrupts are
>> > disabled there. If you still have the error logs, can you post the
>> > full kvm crash report for d3f9?
>> >
>>
>> Here you go:
>
> Thanks. While we're at, can you verify if all your reports are
> showing the cpu in "real mode". That is, do they all have "0000ffff"
> in the third column of the segment registers - as in:
>
>> ES =0000 00000000 0000ffff 00009300
>
That`s positive.
> [...]
>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> b8 00 e0 00 00 8e
>
> KVM reports the code as "int $0x10" here. Was it possible this report
> was from a different build of seabios (that had a different code
> layout)?
>
Yep, sorry, I`ve mixed in logs just from before transition out of 1.7.5.
> Interestingly, this "int $0x10" is also in real-mode and not "big real
> mode", so I think it would have occurred after post completed.
>
> -Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:33 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 17:33 UTC (permalink / raw)
To: Kevin O'Connor
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das, Gerd Hoffmann,
Paolo Bonzini
On Thu, Mar 26, 2015 at 8:18 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
> On Thu, Mar 26, 2015 at 08:08:52PM +0300, Andrey Korolyov wrote:
>> On Thu, Mar 26, 2015 at 8:06 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> > On Thu, Mar 26, 2015 at 07:48:09PM +0300, Andrey Korolyov wrote:
>> >> On Thu, Mar 26, 2015 at 7:36 PM, Kevin O'Connor <kevin@koconnor.net> wrote:
>> >> > I'm not sure if the crash always happens at the "int $0x19" location
>> >> > though. Andrey, does the crash always happen with "EIP=d331" and/or
>> >> > with "Code=... <cd> 19"?
>> >>
>> >> There are also rare occurences for d3f9 (in the middle of ep) and d334
>> >> ep (less than one tenth of events for both). I`ll post a sample event
>> >> capture with and without Radim`s proposed patch maybe today or
>> >> tomorrow.
>> >>
>> >> /root/seabios-1.8.1/src/romlayout.S:289
>> >> d3eb: 66 50 pushl %eax
>> >> d3ed: 66 51 pushl %ecx
>> >> d3ef: 66 52 pushl %edx
>> >> d3f1: 66 53 pushl %ebx
>> >> d3f3: 66 55 pushl %ebp
>> >> d3f5: 66 56 pushl %esi
>> >> d3f7: 66 57 pushl %edi
>> >> d3f9: 06 pushw %es
>> >> d3fa: 1e pushw %ds
>> >>
>> >> 0000d334 <irq_trampoline_0x1c>:
>> >> irq_trampoline_0x1c():
>> >> /root/seabios-1.8.1/src/romlayout.S:196
>> >> d334: cd 1c int $0x1c
>> >> d336: cb lretw
>> >
>> > Thanks. The d334 looks very similar to the d331 report (code=<cd>
>> > 1c). That path could happen during post (big real mode) or
>> > immiediately after post (real mode).
>> >
>> > The d3f9 report does not look like the others - interrupts are
>> > disabled there. If you still have the error logs, can you post the
>> > full kvm crash report for d3f9?
>> >
>>
>> Here you go:
>
> Thanks. While we're at, can you verify if all your reports are
> showing the cpu in "real mode". That is, do they all have "0000ffff"
> in the third column of the segment registers - as in:
>
>> ES =0000 00000000 0000ffff 00009300
>
That`s positive.
> [...]
>> Code=48 18 67 8c 00 8c d1 8e d9 66 5a 66 58 66 5d 66 c3 cd 02 cb <cd>
>> 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb cd 19 cb cd 1c cb fa fc 66
>> b8 00 e0 00 00 8e
>
> KVM reports the code as "int $0x10" here. Was it possible this report
> was from a different build of seabios (that had a different code
> layout)?
>
Yep, sorry, I`ve mixed in logs just from before transition out of 1.7.5.
> Interestingly, this "int $0x10" is also in real-mode and not "big real
> mode", so I think it would have occurred after post completed.
>
> -Kevin
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 16:36 ` [Qemu-devel] " Kevin O'Connor
@ 2015-03-26 17:34 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:34 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-26 12:36-0400, Kevin O'Connor:
> On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
> > Notice the 0xef. My best hypothesis so far is that we fail at resetting
> > devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
> > (The bug happens at the first place that enables interrupts.)
>
> FYI, the "int $0x19" isn't the first place SeaBIOS will enable
> interrupts. Each screen print (every character in the seabios banner
> and uuid string) will call the vga bios (int $0x10) with irqs enabled
> (see output.c:screenc).
Most useful, thank you.
So interrupt can't be "forgotten" there on reboot ... it's possible that
a pending timer injects it later.
(I'd like to grasp the reason behind 0xef first.)
> Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
> simple "iretw".
The #GP error code could help a bit here.
> Things that are unusual about the "int $0x19" call:
> - it is likely the first place that the cpu is transitioned into
> 16bit real mode as opposed to "big real" mode. (That is, the
> first place interrupts are enabled with the segment limits set to
> 0xffff.)
> - it's right after the fw/shadow.c:make_bios_readonly() call, which
> attempts to configures the memory at 0xf0000-0x100000 as
> read-only. That code also issues a wbinvd() call.
(I'll wait for the trace before doing more wild guesses ...)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:34 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:34 UTC (permalink / raw)
To: Kevin O'Connor
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert,
Bandan Das, Gerd Hoffmann, Paolo Bonzini
2015-03-26 12:36-0400, Kevin O'Connor:
> On Thu, Mar 26, 2015 at 04:58:07PM +0100, Radim Krčmář wrote:
> > Notice the 0xef. My best hypothesis so far is that we fail at resetting
> > devices, and 0xef is LOCAL_TIMER_VECTOR from Linux before we rebooted.
> > (The bug happens at the first place that enables interrupts.)
>
> FYI, the "int $0x19" isn't the first place SeaBIOS will enable
> interrupts. Each screen print (every character in the seabios banner
> and uuid string) will call the vga bios (int $0x10) with irqs enabled
> (see output.c:screenc).
Most useful, thank you.
So interrupt can't be "forgotten" there on reboot ... it's possible that
a pending timer injects it later.
(I'd like to grasp the reason behind 0xef first.)
> Also, SeaBIOS loads a default vector (f000:ff53) at 0xef which does a
> simple "iretw".
The #GP error code could help a bit here.
> Things that are unusual about the "int $0x19" call:
> - it is likely the first place that the cpu is transitioned into
> 16bit real mode as opposed to "big real" mode. (That is, the
> first place interrupts are enabled with the segment limits set to
> 0xffff.)
> - it's right after the fw/shadow.c:make_bios_readonly() call, which
> attempts to configures the memory at 0xf0000-0x100000 as
> read-only. That code also issues a wbinvd() call.
(I'll wait for the trace before doing more wild guesses ...)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 16:48 ` Andrey Korolyov
@ 2015-03-26 17:35 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:35 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-26 19:48+0300, Andrey Korolyov:
> I`ll post a sample event
> capture with and without Radim`s proposed patch maybe today or
> tomorrow.
Thanks.
The patch doesn't change runtime behavior, it just adds another data
field to the error report, so there is no need to test both cases.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:35 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:35 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-26 19:48+0300, Andrey Korolyov:
> I`ll post a sample event
> capture with and without Radim`s proposed patch maybe today or
> tomorrow.
Thanks.
The patch doesn't change runtime behavior, it just adds another data
field to the error report, so there is no need to test both cases.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 17:08 ` Andrey Korolyov
@ 2015-03-26 17:40 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:40 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-26 20:08+0300, Andrey Korolyov:
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
Btw. does this part ever change?
I see that first report had:
KVM internal error. Suberror: 2
extra data[0]: 800000d1
extra data[1]: 80000b0d
Was that a Windows guest by any chance?
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 17:40 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 17:40 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-26 20:08+0300, Andrey Korolyov:
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
Btw. does this part ever change?
I see that first report had:
KVM internal error. Suberror: 2
extra data[0]: 800000d1
extra data[1]: 80000b0d
Was that a Windows guest by any chance?
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 17:40 ` Radim Krčmář
@ 2015-03-26 18:24 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 18:24 UTC (permalink / raw)
To: Radim Krčmář
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-26 20:08+0300, Andrey Korolyov:
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000ef
>> extra data[1]: 80000b0d
>
> Btw. does this part ever change?
>
> I see that first report had:
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000d1
> extra data[1]: 80000b0d
>
> Was that a Windows guest by any chance?
Yes, exactly, different extra data output was from a Windows VMs.
Thanks for clarifying things for your patch, I hadn`t looked at the
vmx code yet and thought that it changing things.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 18:24 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-26 18:24 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-26 20:08+0300, Andrey Korolyov:
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000ef
>> extra data[1]: 80000b0d
>
> Btw. does this part ever change?
>
> I see that first report had:
>
> KVM internal error. Suberror: 2
> extra data[0]: 800000d1
> extra data[1]: 80000b0d
>
> Was that a Windows guest by any chance?
Yes, exactly, different extra data output was from a Windows VMs.
Thanks for clarifying things for your patch, I hadn`t looked at the
vmx code yet and thought that it changing things.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 18:24 ` Andrey Korolyov
@ 2015-03-26 20:40 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 20:40 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-26 21:24+0300, Andrey Korolyov:
> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> > 2015-03-26 20:08+0300, Andrey Korolyov:
> >> KVM internal error. Suberror: 2
> >> extra data[0]: 800000ef
> >> extra data[1]: 80000b0d
> >
> > Btw. does this part ever change?
> >
> > I see that first report had:
> >
> > KVM internal error. Suberror: 2
> > extra data[0]: 800000d1
> > extra data[1]: 80000b0d
> >
> > Was that a Windows guest by any chance?
>
> Yes, exactly, different extra data output was from a Windows VMs.
Windows uses vector 0xd1 for timer interrupts.
I second Bandan -- checking that it reproduces on other machine would be
great for sanity :) (Although a bug in our APICv is far more likely.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 20:40 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-26 20:40 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-26 21:24+0300, Andrey Korolyov:
> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> > 2015-03-26 20:08+0300, Andrey Korolyov:
> >> KVM internal error. Suberror: 2
> >> extra data[0]: 800000ef
> >> extra data[1]: 80000b0d
> >
> > Btw. does this part ever change?
> >
> > I see that first report had:
> >
> > KVM internal error. Suberror: 2
> > extra data[0]: 800000d1
> > extra data[1]: 80000b0d
> >
> > Was that a Windows guest by any chance?
>
> Yes, exactly, different extra data output was from a Windows VMs.
Windows uses vector 0xd1 for timer interrupts.
I second Bandan -- checking that it reproduces on other machine would be
great for sanity :) (Although a bug in our APICv is far more likely.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 20:40 ` Radim Krčmář
@ 2015-03-26 21:03 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-26 21:03 UTC (permalink / raw)
To: Radim Krčmář
Cc: Andrey Korolyov, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
Radim Krčmář <rkrcmar@redhat.com> writes:
> 2015-03-26 21:24+0300, Andrey Korolyov:
>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>> >> KVM internal error. Suberror: 2
>> >> extra data[0]: 800000ef
>> >> extra data[1]: 80000b0d
>> >
>> > Btw. does this part ever change?
>> >
>> > I see that first report had:
>> >
>> > KVM internal error. Suberror: 2
>> > extra data[0]: 800000d1
>> > extra data[1]: 80000b0d
>> >
>> > Was that a Windows guest by any chance?
>>
>> Yes, exactly, different extra data output was from a Windows VMs.
>
> Windows uses vector 0xd1 for timer interrupts.
> I second Bandan -- checking that it reproduces on other machine would be
> great for sanity :) (Although a bug in our APICv is far more likely.)
If it's APICv related, a run without apicv enabled could give more hints.
Your "devices not getting reset" hypothesis makes the most sense to me,
maybe the timer vector in the error message is just one part of
the whole story. Another misbehaving interrupt from the dark comes in at the
same time and leads to a double fault.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-26 21:03 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-26 21:03 UTC (permalink / raw)
To: Radim Krčmář
Cc: Andrey Korolyov, kvm, qemu-devel, Dr. David Alan Gilbert,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
Radim Krčmář <rkrcmar@redhat.com> writes:
> 2015-03-26 21:24+0300, Andrey Korolyov:
>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>> >> KVM internal error. Suberror: 2
>> >> extra data[0]: 800000ef
>> >> extra data[1]: 80000b0d
>> >
>> > Btw. does this part ever change?
>> >
>> > I see that first report had:
>> >
>> > KVM internal error. Suberror: 2
>> > extra data[0]: 800000d1
>> > extra data[1]: 80000b0d
>> >
>> > Was that a Windows guest by any chance?
>>
>> Yes, exactly, different extra data output was from a Windows VMs.
>
> Windows uses vector 0xd1 for timer interrupts.
> I second Bandan -- checking that it reproduces on other machine would be
> great for sanity :) (Although a bug in our APICv is far more likely.)
If it's APICv related, a run without apicv enabled could give more hints.
Your "devices not getting reset" hypothesis makes the most sense to me,
maybe the timer vector in the error message is just one part of
the whole story. Another misbehaving interrupt from the dark comes in at the
same time and leads to a double fault.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 21:03 ` Bandan Das
@ 2015-03-27 10:16 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-27 10:16 UTC (permalink / raw)
To: Bandan Das
Cc: Radim Krčmář,
Kevin O'Connor, Dr. David Alan Gilbert, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> Radim Krčmář <rkrcmar@redhat.com> writes:
>
>> 2015-03-26 21:24+0300, Andrey Korolyov:
>>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>>> >> KVM internal error. Suberror: 2
>>> >> extra data[0]: 800000ef
>>> >> extra data[1]: 80000b0d
>>> >
>>> > Btw. does this part ever change?
>>> >
>>> > I see that first report had:
>>> >
>>> > KVM internal error. Suberror: 2
>>> > extra data[0]: 800000d1
>>> > extra data[1]: 80000b0d
>>> >
>>> > Was that a Windows guest by any chance?
>>>
>>> Yes, exactly, different extra data output was from a Windows VMs.
>>
>> Windows uses vector 0xd1 for timer interrupts.
>
>> I second Bandan -- checking that it reproduces on other machine would be
>> great for sanity :) (Although a bug in our APICv is far more likely.)
>
> If it's APICv related, a run without apicv enabled could give more hints.
>
> Your "devices not getting reset" hypothesis makes the most sense to me,
> maybe the timer vector in the error message is just one part of
> the whole story. Another misbehaving interrupt from the dark comes in at the
> same time and leads to a double fault.
Default trace (APICv enabled, first reboot introduced the issue):
http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
Trace without APICv (three reboots, just to make sure to hit the
problematic condition of supposed DF, as it still have not one hundred
percent reproducibility):
http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
It would be great of course to reproduce this somewhere else,
otherwise all this thread may end in fixing a bug which exists only at
my particular platform. Right now I have no hardware except a lot of
well-known (in terms of existing issues) Supermicro boards of one
model.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-27 10:16 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-27 10:16 UTC (permalink / raw)
To: Bandan Das
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> Radim Krčmář <rkrcmar@redhat.com> writes:
>
>> 2015-03-26 21:24+0300, Andrey Korolyov:
>>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>>> >> KVM internal error. Suberror: 2
>>> >> extra data[0]: 800000ef
>>> >> extra data[1]: 80000b0d
>>> >
>>> > Btw. does this part ever change?
>>> >
>>> > I see that first report had:
>>> >
>>> > KVM internal error. Suberror: 2
>>> > extra data[0]: 800000d1
>>> > extra data[1]: 80000b0d
>>> >
>>> > Was that a Windows guest by any chance?
>>>
>>> Yes, exactly, different extra data output was from a Windows VMs.
>>
>> Windows uses vector 0xd1 for timer interrupts.
>
>> I second Bandan -- checking that it reproduces on other machine would be
>> great for sanity :) (Although a bug in our APICv is far more likely.)
>
> If it's APICv related, a run without apicv enabled could give more hints.
>
> Your "devices not getting reset" hypothesis makes the most sense to me,
> maybe the timer vector in the error message is just one part of
> the whole story. Another misbehaving interrupt from the dark comes in at the
> same time and leads to a double fault.
Default trace (APICv enabled, first reboot introduced the issue):
http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
Trace without APICv (three reboots, just to make sure to hit the
problematic condition of supposed DF, as it still have not one hundred
percent reproducibility):
http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
It would be great of course to reproduce this somewhere else,
otherwise all this thread may end in fixing a bug which exists only at
my particular platform. Right now I have no hardware except a lot of
well-known (in terms of existing issues) Supermicro boards of one
model.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-26 20:40 ` Radim Krčmář
@ 2015-03-27 11:54 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-27 11:54 UTC (permalink / raw)
To: Radim Krčmář
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Thu, Mar 26, 2015 at 11:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-26 21:24+0300, Andrey Korolyov:
>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>> >> KVM internal error. Suberror: 2
>> >> extra data[0]: 800000ef
>> >> extra data[1]: 80000b0d
>> >
>> > Btw. does this part ever change?
>> >
>> > I see that first report had:
>> >
>> > KVM internal error. Suberror: 2
>> > extra data[0]: 800000d1
>> > extra data[1]: 80000b0d
>> >
>> > Was that a Windows guest by any chance?
>>
>> Yes, exactly, different extra data output was from a Windows VMs.
>
> Windows uses vector 0xd1 for timer interrupts.
>
> I second Bandan -- checking that it reproduces on other machine would be
> great for sanity :) (Although a bug in our APICv is far more likely.)
Trace with new bits:
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
extra data[2]: 77b
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d24
EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6cb0 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
ba 2d d4 fe fb 3f
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-27 11:54 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-27 11:54 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Thu, Mar 26, 2015 at 11:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-26 21:24+0300, Andrey Korolyov:
>> On Thu, Mar 26, 2015 at 8:40 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-26 20:08+0300, Andrey Korolyov:
>> >> KVM internal error. Suberror: 2
>> >> extra data[0]: 800000ef
>> >> extra data[1]: 80000b0d
>> >
>> > Btw. does this part ever change?
>> >
>> > I see that first report had:
>> >
>> > KVM internal error. Suberror: 2
>> > extra data[0]: 800000d1
>> > extra data[1]: 80000b0d
>> >
>> > Was that a Windows guest by any chance?
>>
>> Yes, exactly, different extra data output was from a Windows VMs.
>
> Windows uses vector 0xd1 for timer interrupts.
>
> I second Bandan -- checking that it reproduces on other machine would be
> great for sanity :) (Although a bug in our APICv is far more likely.)
Trace with new bits:
KVM internal error. Suberror: 2
extra data[0]: 800000ef
extra data[1]: 80000b0d
extra data[2]: 77b
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d24
EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 000f6cb0 00000037
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
ba 2d d4 fe fb 3f
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-27 10:16 ` Andrey Korolyov
@ 2015-03-30 18:56 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-30 18:56 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-27 13:16+0300, Andrey Korolyov:
> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> > Radim Krčmář <rkrcmar@redhat.com> writes:
> >> I second Bandan -- checking that it reproduces on other machine would be
> >> great for sanity :) (Although a bug in our APICv is far more likely.)
> >
> > If it's APICv related, a run without apicv enabled could give more hints.
> >
> > Your "devices not getting reset" hypothesis makes the most sense to me,
> > maybe the timer vector in the error message is just one part of
> > the whole story. Another misbehaving interrupt from the dark comes in at the
> > same time and leads to a double fault.
>
> Default trace (APICv enabled, first reboot introduced the issue):
> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
The relevant part is here,
prefixed with "qemu-system-x86-4180 [002] 697.111550:"
kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
kvm_cr: cr_write 0 = 0x10
kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
kvm_inj_virq: irq 8
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
kvm_page_fault: address ffea5 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
kvm_page_fault: address fe990 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
> Trace without APICv (three reboots, just to make sure to hit the
> problematic condition of supposed DF, as it still have not one hundred
> percent reproducibility):
> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
The trace here contains a well matching excerpt, just instead of the
EXCEPTION_NMI, it does
169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
169.905102: kvm_page_fault: address feffd066 error_code 181
and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
of TSS. (I guess it is pre-fetch for following IO instruction.)
Nothing strikes me when looking at it, but some APICv boots don't fail,
so it would be interesting to compare them ... hosts's 0xf6 interrupt
(IRQ_WORK_VECTOR) is a possible source of races. (We could look more
closely. It is fired too often for my liking as well.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-30 18:56 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-30 18:56 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-27 13:16+0300, Andrey Korolyov:
> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> > Radim Krčmář <rkrcmar@redhat.com> writes:
> >> I second Bandan -- checking that it reproduces on other machine would be
> >> great for sanity :) (Although a bug in our APICv is far more likely.)
> >
> > If it's APICv related, a run without apicv enabled could give more hints.
> >
> > Your "devices not getting reset" hypothesis makes the most sense to me,
> > maybe the timer vector in the error message is just one part of
> > the whole story. Another misbehaving interrupt from the dark comes in at the
> > same time and leads to a double fault.
>
> Default trace (APICv enabled, first reboot introduced the issue):
> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
The relevant part is here,
prefixed with "qemu-system-x86-4180 [002] 697.111550:"
kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
kvm_cr: cr_write 0 = 0x10
kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
kvm_inj_virq: irq 8
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
kvm_page_fault: address ffea5 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
kvm_page_fault: address fe990 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
kvm_entry: vcpu 0
kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
> Trace without APICv (three reboots, just to make sure to hit the
> problematic condition of supposed DF, as it still have not one hundred
> percent reproducibility):
> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
The trace here contains a well matching excerpt, just instead of the
EXCEPTION_NMI, it does
169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
169.905102: kvm_page_fault: address feffd066 error_code 181
and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
of TSS. (I guess it is pre-fetch for following IO instruction.)
Nothing strikes me when looking at it, but some APICv boots don't fail,
so it would be interesting to compare them ... hosts's 0xf6 interrupt
(IRQ_WORK_VECTOR) is a possible source of races. (We could look more
closely. It is fired too often for my liking as well.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-27 11:54 ` Andrey Korolyov
@ 2015-03-30 19:28 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-30 19:28 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Kevin O'Connor, Dr. David Alan Gilbert, Bandan Das,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-27 14:54+0300, Andrey Korolyov:
> Trace with new bits:
Thanks.
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
> extra data[2]: 77b
The #GP code looks formatted as documented under INT in SDM,
(vector << 3) | 2 | ext
where 'ext' stands for 'external' (as opposed to software).
0x77b == (0xef << 3) | 2 | 1
It was 0xef and wasn't triggered by an INT instruction.
The weird part is that it looks like a protected mode error, but CR0
says we are in real mode.
(If CPU interpreted the vector in protected mode, then it would violate
the IDT limit and throw a #GP ...
It's too late for coffee today, so I'll try to lure some ideas later.)
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d24
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6cb0 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
> ba 2d d4 fe fb 3f
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-30 19:28 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-30 19:28 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-27 14:54+0300, Andrey Korolyov:
> Trace with new bits:
Thanks.
> KVM internal error. Suberror: 2
> extra data[0]: 800000ef
> extra data[1]: 80000b0d
> extra data[2]: 77b
The #GP code looks formatted as documented under INT in SDM,
(vector << 3) | 2 | ext
where 'ext' stands for 'external' (as opposed to software).
0x77b == (0xef << 3) | 2 | 1
It was 0xef and wasn't triggered by an INT instruction.
The weird part is that it looks like a protected mode error, but CR0
says we are in real mode.
(If CPU interpreted the vector in protected mode, then it would violate
the IDT limit and throw a #GP ...
It's too late for coffee today, so I'll try to lure some ideas later.)
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d24
> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 000f6cb0 00000037
> IDT= 00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
> ba 2d d4 fe fb 3f
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-30 18:56 ` [Qemu-devel] " Radim Krčmář
@ 2015-03-30 19:32 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-30 19:32 UTC (permalink / raw)
To: Radim Krčmář
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-27 13:16+0300, Andrey Korolyov:
>> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
>> > Radim Krčmář <rkrcmar@redhat.com> writes:
>> >> I second Bandan -- checking that it reproduces on other machine would be
>> >> great for sanity :) (Although a bug in our APICv is far more likely.)
>> >
>> > If it's APICv related, a run without apicv enabled could give more hints.
>> >
>> > Your "devices not getting reset" hypothesis makes the most sense to me,
>> > maybe the timer vector in the error message is just one part of
>> > the whole story. Another misbehaving interrupt from the dark comes in at the
>> > same time and leads to a double fault.
>>
>> Default trace (APICv enabled, first reboot introduced the issue):
>> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
>
> The relevant part is here,
> prefixed with "qemu-system-x86-4180 [002] 697.111550:"
>
> kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> kvm_cr: cr_write 0 = 0x10
> kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
> kvm_inj_virq: irq 8
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
> kvm_page_fault: address ffea5 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
> kvm_page_fault: address fe990 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
> kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>
>> Trace without APICv (three reboots, just to make sure to hit the
>> problematic condition of supposed DF, as it still have not one hundred
>> percent reproducibility):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
>
> The trace here contains a well matching excerpt, just instead of the
> EXCEPTION_NMI, it does
>
> 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
> 169.905102: kvm_page_fault: address feffd066 error_code 181
>
> and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
> of TSS. (I guess it is pre-fetch for following IO instruction.)
>
> Nothing strikes me when looking at it, but some APICv boots don't fail,
> so it would be interesting to compare them ... hosts's 0xf6 interrupt
> (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
> closely. It is fired too often for my liking as well.)
Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
(missed right button in mailer previously)
The related bits looks the same as with enable_apicv=0 for me.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-30 19:32 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-30 19:32 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-27 13:16+0300, Andrey Korolyov:
>> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
>> > Radim Krčmář <rkrcmar@redhat.com> writes:
>> >> I second Bandan -- checking that it reproduces on other machine would be
>> >> great for sanity :) (Although a bug in our APICv is far more likely.)
>> >
>> > If it's APICv related, a run without apicv enabled could give more hints.
>> >
>> > Your "devices not getting reset" hypothesis makes the most sense to me,
>> > maybe the timer vector in the error message is just one part of
>> > the whole story. Another misbehaving interrupt from the dark comes in at the
>> > same time and leads to a double fault.
>>
>> Default trace (APICv enabled, first reboot introduced the issue):
>> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
>
> The relevant part is here,
> prefixed with "qemu-system-x86-4180 [002] 697.111550:"
>
> kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> kvm_cr: cr_write 0 = 0x10
> kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
> kvm_inj_virq: irq 8
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
> kvm_page_fault: address ffea5 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
> kvm_page_fault: address fe990 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
> kvm_entry: vcpu 0
> kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
> kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>
>> Trace without APICv (three reboots, just to make sure to hit the
>> problematic condition of supposed DF, as it still have not one hundred
>> percent reproducibility):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
>
> The trace here contains a well matching excerpt, just instead of the
> EXCEPTION_NMI, it does
>
> 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
> 169.905102: kvm_page_fault: address feffd066 error_code 181
>
> and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
> of TSS. (I guess it is pre-fetch for following IO instruction.)
>
> Nothing strikes me when looking at it, but some APICv boots don't fail,
> so it would be interesting to compare them ... hosts's 0xf6 interrupt
> (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
> closely. It is fired too often for my liking as well.)
Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
(missed right button in mailer previously)
The related bits looks the same as with enable_apicv=0 for me.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-30 19:32 ` Andrey Korolyov
@ 2015-03-31 13:45 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-31 13:45 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-30 22:32+0300, Andrey Korolyov:
> On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> > 2015-03-27 13:16+0300, Andrey Korolyov:
> >> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> >> > Radim Krčmář <rkrcmar@redhat.com> writes:
> >> >> I second Bandan -- checking that it reproduces on other machine would be
> >> >> great for sanity :) (Although a bug in our APICv is far more likely.)
> >> >
> >> > If it's APICv related, a run without apicv enabled could give more hints.
> >> >
> >> > Your "devices not getting reset" hypothesis makes the most sense to me,
> >> > maybe the timer vector in the error message is just one part of
> >> > the whole story. Another misbehaving interrupt from the dark comes in at the
> >> > same time and leads to a double fault.
> >>
> >> Default trace (APICv enabled, first reboot introduced the issue):
> >> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
> >
> > The relevant part is here,
> > prefixed with "qemu-system-x86-4180 [002] 697.111550:"
> >
> > kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> > kvm_cr: cr_write 0 = 0x10
> > kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> > kvm_entry: vcpu 0
> > kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> > kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> > kvm_emulate_insn: f0000:d280: 31 c0
> > kvm_emulate_insn: f0000:d282: 8e e0
> > kvm_emulate_insn: f0000:d284: 8e e8
> > kvm_emulate_insn: f0000:d286: 8e c0
> > kvm_emulate_insn: f0000:d288: 8e d8
> > kvm_emulate_insn: f0000:d28a: 8e d0
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> > kvm_page_fault: address f8dd0 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> > kvm_page_fault: address f76d6 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
> > kvm_inj_virq: irq 8
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
> > kvm_page_fault: address ffea5 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
> > kvm_page_fault: address fe990 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
> > kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
> >
> >> Trace without APICv (three reboots, just to make sure to hit the
> >> problematic condition of supposed DF, as it still have not one hundred
> >> percent reproducibility):
> >> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
> >
> > The trace here contains a well matching excerpt, just instead of the
> > EXCEPTION_NMI, it does
> >
> > 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
> > 169.905102: kvm_page_fault: address feffd066 error_code 181
> >
> > and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
> > of TSS. (I guess it is pre-fetch for following IO instruction.)
> >
> > Nothing strikes me when looking at it, but some APICv boots don't fail,
> > so it would be interesting to compare them ... hosts's 0xf6 interrupt
> > (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
> > closely. It is fired too often for my liking as well.)
>
> Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
>
> The related bits looks the same as with enable_apicv=0 for me.
Yeah,
qemu-system-x86-4201 [007] 159.297337:
kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
kvm_cr: cr_write 0 = 0x10
kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xd331 info 181 0
kvm_page_fault: address feffd066 error_code 181
kvm_inj_virq: irq 25
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xe6f2 info 184 0
kvm_page_fault: address fe6f2 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason IO_INSTRUCTION rip 0xd202 info 700040 0
kvm_pio: pio_write at 0x70 size 1 count 1 val 0x8f
kvm_userspace_exit: reason KVM_EXIT_IO (2)
I noticed three differences from the failing version:
1) Page fault 0xfeffd066 @ 0xd331, KVM also injects vector 0x19 there
2) No PENDING_INTERRUPT irq 8 (not sure why it happened on APICv)
3) No EXTERNAL_INTERRUPT vector 0xf6 (lucky race)
Chasing the culprit this way could take a long time, so a new tracepoint
that shows if 0xef is set on entry would let us guess the bug faster ...
Please provide a failing trace with the following patch:
---
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 7c7bc8bef21f..8cf85dcaa1ee 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -742,6 +742,29 @@ TRACE_EVENT(kvm_emulate_insn,
#define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
#define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
+TRACE_EVENT(kvm_0xef,
+ TP_PROTO(bool irr, bool isr, u32 vmcs),
+ TP_ARGS(irr, isr, vmcs),
+
+ TP_STRUCT__entry(
+ __field(bool, irr )
+ __field(bool, isr )
+ __field(u32, vmcs)
+ ),
+
+ TP_fast_assign(
+ __entry->irr = irr;
+ __entry->isr = isr;
+ __entry->vmcs = vmcs;
+ ),
+
+ TP_printk("irr %s, isr %s, vmcs 0x%x",
+ __entry->irr ? "set " : "clear",
+ __entry->isr ? "set " : "clear",
+ __entry->vmcs
+ )
+ );
+
TRACE_EVENT(
vcpu_match_mmio,
TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index eee63dc33d89..254cca12c2ec 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -8129,6 +8129,13 @@ static void atomic_switch_perf_msrs(struct vcpu_vmx *vmx)
msrs[i].host);
}
+#define VEC_POS(v) ((v) & (32 - 1))
+#define REG_POS(v) (((v) >> 5) << 4)
+static inline int apic_test_vector(int vec, void *bitmap)
+{
+ return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
+}
+
static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -8143,6 +8150,10 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
if (vmx->emulation_required)
return;
+ trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
+ apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
+ vmcs_read32(VM_ENTRY_INTR_INFO_FIELD));
+
if (vmx->ple_window_dirty) {
vmx->ple_window_dirty = false;
vmcs_write32(PLE_WINDOW, vmx->ple_window);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32bf19ef3115..a45fa01bd354 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 13:45 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-31 13:45 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-30 22:32+0300, Andrey Korolyov:
> On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> > 2015-03-27 13:16+0300, Andrey Korolyov:
> >> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
> >> > Radim Krčmář <rkrcmar@redhat.com> writes:
> >> >> I second Bandan -- checking that it reproduces on other machine would be
> >> >> great for sanity :) (Although a bug in our APICv is far more likely.)
> >> >
> >> > If it's APICv related, a run without apicv enabled could give more hints.
> >> >
> >> > Your "devices not getting reset" hypothesis makes the most sense to me,
> >> > maybe the timer vector in the error message is just one part of
> >> > the whole story. Another misbehaving interrupt from the dark comes in at the
> >> > same time and leads to a double fault.
> >>
> >> Default trace (APICv enabled, first reboot introduced the issue):
> >> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
> >
> > The relevant part is here,
> > prefixed with "qemu-system-x86-4180 [002] 697.111550:"
> >
> > kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> > kvm_cr: cr_write 0 = 0x10
> > kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> > kvm_entry: vcpu 0
> > kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> > kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> > kvm_emulate_insn: f0000:d280: 31 c0
> > kvm_emulate_insn: f0000:d282: 8e e0
> > kvm_emulate_insn: f0000:d284: 8e e8
> > kvm_emulate_insn: f0000:d286: 8e c0
> > kvm_emulate_insn: f0000:d288: 8e d8
> > kvm_emulate_insn: f0000:d28a: 8e d0
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> > kvm_page_fault: address f8dd0 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> > kvm_page_fault: address f76d6 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
> > kvm_inj_virq: irq 8
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
> > kvm_page_fault: address ffea5 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
> > kvm_page_fault: address fe990 error_code 184
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
> > kvm_entry: vcpu 0
> > kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
> > kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
> >
> >> Trace without APICv (three reboots, just to make sure to hit the
> >> problematic condition of supposed DF, as it still have not one hundred
> >> percent reproducibility):
> >> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
> >
> > The trace here contains a well matching excerpt, just instead of the
> > EXCEPTION_NMI, it does
> >
> > 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
> > 169.905102: kvm_page_fault: address feffd066 error_code 181
> >
> > and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
> > of TSS. (I guess it is pre-fetch for following IO instruction.)
> >
> > Nothing strikes me when looking at it, but some APICv boots don't fail,
> > so it would be interesting to compare them ... hosts's 0xf6 interrupt
> > (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
> > closely. It is fired too often for my liking as well.)
>
> Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
>
> The related bits looks the same as with enable_apicv=0 for me.
Yeah,
qemu-system-x86-4201 [007] 159.297337:
kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
kvm_cr: cr_write 0 = 0x10
kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xd331 info 181 0
kvm_page_fault: address feffd066 error_code 181
kvm_inj_virq: irq 25
kvm_entry: vcpu 0
kvm_exit: reason EPT_VIOLATION rip 0xe6f2 info 184 0
kvm_page_fault: address fe6f2 error_code 184
kvm_entry: vcpu 0
kvm_exit: reason IO_INSTRUCTION rip 0xd202 info 700040 0
kvm_pio: pio_write at 0x70 size 1 count 1 val 0x8f
kvm_userspace_exit: reason KVM_EXIT_IO (2)
I noticed three differences from the failing version:
1) Page fault 0xfeffd066 @ 0xd331, KVM also injects vector 0x19 there
2) No PENDING_INTERRUPT irq 8 (not sure why it happened on APICv)
3) No EXTERNAL_INTERRUPT vector 0xf6 (lucky race)
Chasing the culprit this way could take a long time, so a new tracepoint
that shows if 0xef is set on entry would let us guess the bug faster ...
Please provide a failing trace with the following patch:
---
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 7c7bc8bef21f..8cf85dcaa1ee 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -742,6 +742,29 @@ TRACE_EVENT(kvm_emulate_insn,
#define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
#define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
+TRACE_EVENT(kvm_0xef,
+ TP_PROTO(bool irr, bool isr, u32 vmcs),
+ TP_ARGS(irr, isr, vmcs),
+
+ TP_STRUCT__entry(
+ __field(bool, irr )
+ __field(bool, isr )
+ __field(u32, vmcs)
+ ),
+
+ TP_fast_assign(
+ __entry->irr = irr;
+ __entry->isr = isr;
+ __entry->vmcs = vmcs;
+ ),
+
+ TP_printk("irr %s, isr %s, vmcs 0x%x",
+ __entry->irr ? "set " : "clear",
+ __entry->isr ? "set " : "clear",
+ __entry->vmcs
+ )
+ );
+
TRACE_EVENT(
vcpu_match_mmio,
TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index eee63dc33d89..254cca12c2ec 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -8129,6 +8129,13 @@ static void atomic_switch_perf_msrs(struct vcpu_vmx *vmx)
msrs[i].host);
}
+#define VEC_POS(v) ((v) & (32 - 1))
+#define REG_POS(v) (((v) >> 5) << 4)
+static inline int apic_test_vector(int vec, void *bitmap)
+{
+ return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
+}
+
static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -8143,6 +8150,10 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
if (vmx->emulation_required)
return;
+ trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
+ apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
+ vmcs_read32(VM_ENTRY_INTR_INFO_FIELD));
+
if (vmx->ple_window_dirty) {
vmx->ple_window_dirty = false;
vmcs_write32(PLE_WINDOW, vmx->ple_window);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32bf19ef3115..a45fa01bd354 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-31 13:45 ` [Qemu-devel] " Radim Krčmář
@ 2015-03-31 14:56 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 14:56 UTC (permalink / raw)
To: Radim Krčmář
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Tue, Mar 31, 2015 at 4:45 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-30 22:32+0300, Andrey Korolyov:
>> On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-27 13:16+0300, Andrey Korolyov:
>> >> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
>> >> > Radim Krčmář <rkrcmar@redhat.com> writes:
>> >> >> I second Bandan -- checking that it reproduces on other machine would be
>> >> >> great for sanity :) (Although a bug in our APICv is far more likely.)
>> >> >
>> >> > If it's APICv related, a run without apicv enabled could give more hints.
>> >> >
>> >> > Your "devices not getting reset" hypothesis makes the most sense to me,
>> >> > maybe the timer vector in the error message is just one part of
>> >> > the whole story. Another misbehaving interrupt from the dark comes in at the
>> >> > same time and leads to a double fault.
>> >>
>> >> Default trace (APICv enabled, first reboot introduced the issue):
>> >> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
>> >
>> > The relevant part is here,
>> > prefixed with "qemu-system-x86-4180 [002] 697.111550:"
>> >
>> > kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
>> > kvm_cr: cr_write 0 = 0x10
>> > kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
>> > kvm_entry: vcpu 0
>> > kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
>> > kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
>> > kvm_emulate_insn: f0000:d280: 31 c0
>> > kvm_emulate_insn: f0000:d282: 8e e0
>> > kvm_emulate_insn: f0000:d284: 8e e8
>> > kvm_emulate_insn: f0000:d286: 8e c0
>> > kvm_emulate_insn: f0000:d288: 8e d8
>> > kvm_emulate_insn: f0000:d28a: 8e d0
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
>> > kvm_page_fault: address f8dd0 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
>> > kvm_page_fault: address f76d6 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
>> > kvm_inj_virq: irq 8
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
>> > kvm_page_fault: address ffea5 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
>> > kvm_page_fault: address fe990 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
>> > kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>> >
>> >> Trace without APICv (three reboots, just to make sure to hit the
>> >> problematic condition of supposed DF, as it still have not one hundred
>> >> percent reproducibility):
>> >> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
>> >
>> > The trace here contains a well matching excerpt, just instead of the
>> > EXCEPTION_NMI, it does
>> >
>> > 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
>> > 169.905102: kvm_page_fault: address feffd066 error_code 181
>> >
>> > and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
>> > of TSS. (I guess it is pre-fetch for following IO instruction.)
>> >
>> > Nothing strikes me when looking at it, but some APICv boots don't fail,
>> > so it would be interesting to compare them ... hosts's 0xf6 interrupt
>> > (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
>> > closely. It is fired too often for my liking as well.)
>>
>> Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
>>
>> The related bits looks the same as with enable_apicv=0 for me.
>
> Yeah,
>
> qemu-system-x86-4201 [007] 159.297337:
> kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> kvm_cr: cr_write 0 = 0x10
> kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xd331 info 181 0
> kvm_page_fault: address feffd066 error_code 181
> kvm_inj_virq: irq 25
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xe6f2 info 184 0
> kvm_page_fault: address fe6f2 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0xd202 info 700040 0
> kvm_pio: pio_write at 0x70 size 1 count 1 val 0x8f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
>
> I noticed three differences from the failing version:
> 1) Page fault 0xfeffd066 @ 0xd331, KVM also injects vector 0x19 there
> 2) No PENDING_INTERRUPT irq 8 (not sure why it happened on APICv)
> 3) No EXTERNAL_INTERRUPT vector 0xf6 (lucky race)
>
> Chasing the culprit this way could take a long time, so a new tracepoint
> that shows if 0xef is set on entry would let us guess the bug faster ...
>
> Please provide a failing trace with the following patch:
>
> ---
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index 7c7bc8bef21f..8cf85dcaa1ee 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -742,6 +742,29 @@ TRACE_EVENT(kvm_emulate_insn,
> #define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
> #define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
>
> +TRACE_EVENT(kvm_0xef,
> + TP_PROTO(bool irr, bool isr, u32 vmcs),
> + TP_ARGS(irr, isr, vmcs),
> +
> + TP_STRUCT__entry(
> + __field(bool, irr )
> + __field(bool, isr )
> + __field(u32, vmcs)
> + ),
> +
> + TP_fast_assign(
> + __entry->irr = irr;
> + __entry->isr = isr;
> + __entry->vmcs = vmcs;
> + ),
> +
> + TP_printk("irr %s, isr %s, vmcs 0x%x",
> + __entry->irr ? "set " : "clear",
> + __entry->isr ? "set " : "clear",
> + __entry->vmcs
> + )
> + );
> +
> TRACE_EVENT(
> vcpu_match_mmio,
> TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index eee63dc33d89..254cca12c2ec 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -8129,6 +8129,13 @@ static void atomic_switch_perf_msrs(struct vcpu_vmx *vmx)
> msrs[i].host);
> }
>
> +#define VEC_POS(v) ((v) & (32 - 1))
> +#define REG_POS(v) (((v) >> 5) << 4)
> +static inline int apic_test_vector(int vec, void *bitmap)
> +{
> + return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
> +}
> +
> static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> {
> struct vcpu_vmx *vmx = to_vmx(vcpu);
> @@ -8143,6 +8150,10 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> if (vmx->emulation_required)
> return;
>
> + trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
> + apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
> + vmcs_read32(VM_ENTRY_INTR_INFO_FIELD));
> +
> if (vmx->ple_window_dirty) {
> vmx->ple_window_dirty = false;
> vmcs_write32(PLE_WINDOW, vmx->ple_window);
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 32bf19ef3115..a45fa01bd354 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
Thanks, please see below:
http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 14:56 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 14:56 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Tue, Mar 31, 2015 at 4:45 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-30 22:32+0300, Andrey Korolyov:
>> On Mon, Mar 30, 2015 at 9:56 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
>> > 2015-03-27 13:16+0300, Andrey Korolyov:
>> >> On Fri, Mar 27, 2015 at 12:03 AM, Bandan Das <bsd@redhat.com> wrote:
>> >> > Radim Krčmář <rkrcmar@redhat.com> writes:
>> >> >> I second Bandan -- checking that it reproduces on other machine would be
>> >> >> great for sanity :) (Although a bug in our APICv is far more likely.)
>> >> >
>> >> > If it's APICv related, a run without apicv enabled could give more hints.
>> >> >
>> >> > Your "devices not getting reset" hypothesis makes the most sense to me,
>> >> > maybe the timer vector in the error message is just one part of
>> >> > the whole story. Another misbehaving interrupt from the dark comes in at the
>> >> > same time and leads to a double fault.
>> >>
>> >> Default trace (APICv enabled, first reboot introduced the issue):
>> >> http://xdel.ru/downloads/kvm-e5v2-issue/hanged-reboot-apic-on.dat.gz
>> >
>> > The relevant part is here,
>> > prefixed with "qemu-system-x86-4180 [002] 697.111550:"
>> >
>> > kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
>> > kvm_cr: cr_write 0 = 0x10
>> > kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
>> > kvm_entry: vcpu 0
>> > kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
>> > kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
>> > kvm_emulate_insn: f0000:d280: 31 c0
>> > kvm_emulate_insn: f0000:d282: 8e e0
>> > kvm_emulate_insn: f0000:d284: 8e e8
>> > kvm_emulate_insn: f0000:d286: 8e c0
>> > kvm_emulate_insn: f0000:d288: 8e d8
>> > kvm_emulate_insn: f0000:d28a: 8e d0
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xd28f info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
>> > kvm_page_fault: address f8dd0 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8dd0 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
>> > kvm_page_fault: address f76d6 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0x76d6 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason PENDING_INTERRUPT rip 0xd331 info 0 0
>> > kvm_inj_virq: irq 8
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0xfea5 info 184 0
>> > kvm_page_fault: address ffea5 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xfea5 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EPT_VIOLATION rip 0xe990 info 184 0
>> > kvm_page_fault: address fe990 error_code 184
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXTERNAL_INTERRUPT rip 0xe990 info 0 800000f6
>> > kvm_entry: vcpu 0
>> > kvm_exit: reason EXCEPTION_NMI rip 0xd334 info 0 80000b0d
>> > kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>> >
>> >> Trace without APICv (three reboots, just to make sure to hit the
>> >> problematic condition of supposed DF, as it still have not one hundred
>> >> percent reproducibility):
>> >> http://xdel.ru/downloads/kvm-e5v2-issue/apic-off.dat.gz
>> >
>> > The trace here contains a well matching excerpt, just instead of the
>> > EXCEPTION_NMI, it does
>> >
>> > 169.905098: kvm_exit: reason EPT_VIOLATION rip 0xd334 info 181 0
>> > 169.905102: kvm_page_fault: address feffd066 error_code 181
>> >
>> > and works. Page fault says we tried to read 0xfeffd066 -- probably IOPB
>> > of TSS. (I guess it is pre-fetch for following IO instruction.)
>> >
>> > Nothing strikes me when looking at it, but some APICv boots don't fail,
>> > so it would be interesting to compare them ... hosts's 0xf6 interrupt
>> > (IRQ_WORK_VECTOR) is a possible source of races. (We could look more
>> > closely. It is fired too often for my liking as well.)
>>
>> Thanks Radim, http://xdel.ru/downloads/kvm-e5v2-issue/no-fail-with-apicv.dat.gz
>>
>> The related bits looks the same as with enable_apicv=0 for me.
>
> Yeah,
>
> qemu-system-x86-4201 [007] 159.297337:
> kvm_exit: reason CR_ACCESS rip 0xd272 info 0 0
> kvm_cr: cr_write 0 = 0x10
> kvm_mmu_get_page: existing sp gfn 0 0/4 q0 direct --- !pge !nxe root 0 sync
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xd331 info 181 0
> kvm_page_fault: address feffd066 error_code 181
> kvm_inj_virq: irq 25
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0xe6f2 info 184 0
> kvm_page_fault: address fe6f2 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0xd202 info 700040 0
> kvm_pio: pio_write at 0x70 size 1 count 1 val 0x8f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
>
> I noticed three differences from the failing version:
> 1) Page fault 0xfeffd066 @ 0xd331, KVM also injects vector 0x19 there
> 2) No PENDING_INTERRUPT irq 8 (not sure why it happened on APICv)
> 3) No EXTERNAL_INTERRUPT vector 0xf6 (lucky race)
>
> Chasing the culprit this way could take a long time, so a new tracepoint
> that shows if 0xef is set on entry would let us guess the bug faster ...
>
> Please provide a failing trace with the following patch:
>
> ---
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index 7c7bc8bef21f..8cf85dcaa1ee 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -742,6 +742,29 @@ TRACE_EVENT(kvm_emulate_insn,
> #define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
> #define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
>
> +TRACE_EVENT(kvm_0xef,
> + TP_PROTO(bool irr, bool isr, u32 vmcs),
> + TP_ARGS(irr, isr, vmcs),
> +
> + TP_STRUCT__entry(
> + __field(bool, irr )
> + __field(bool, isr )
> + __field(u32, vmcs)
> + ),
> +
> + TP_fast_assign(
> + __entry->irr = irr;
> + __entry->isr = isr;
> + __entry->vmcs = vmcs;
> + ),
> +
> + TP_printk("irr %s, isr %s, vmcs 0x%x",
> + __entry->irr ? "set " : "clear",
> + __entry->isr ? "set " : "clear",
> + __entry->vmcs
> + )
> + );
> +
> TRACE_EVENT(
> vcpu_match_mmio,
> TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index eee63dc33d89..254cca12c2ec 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -8129,6 +8129,13 @@ static void atomic_switch_perf_msrs(struct vcpu_vmx *vmx)
> msrs[i].host);
> }
>
> +#define VEC_POS(v) ((v) & (32 - 1))
> +#define REG_POS(v) (((v) >> 5) << 4)
> +static inline int apic_test_vector(int vec, void *bitmap)
> +{
> + return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
> +}
> +
> static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> {
> struct vcpu_vmx *vmx = to_vmx(vcpu);
> @@ -8143,6 +8150,10 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> if (vmx->emulation_required)
> return;
>
> + trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
> + apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
> + vmcs_read32(VM_ENTRY_INTR_INFO_FIELD));
> +
> if (vmx->ple_window_dirty) {
> vmx->ple_window_dirty = false;
> vmcs_write32(PLE_WINDOW, vmx->ple_window);
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 32bf19ef3115..a45fa01bd354 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
Thanks, please see below:
http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-31 14:56 ` Andrey Korolyov
@ 2015-03-31 16:45 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-31 16:45 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-31 17:56+0300, Andrey Korolyov:
> > Chasing the culprit this way could take a long time, so a new tracepoint
> > that shows if 0xef is set on entry would let us guess the bug faster ...
> >
> > Please provide a failing trace with the following patch:
>
> Thanks, please see below:
>
> http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
qemu-system-x86-4022 [006] 255.915978:
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EXCEPTION_NMI rip 0xd331 info 0 80000b0d
kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
Ok, nothing obvious here either ... I've desperately added all
information I know about. Please run it again, thanks.
(The patch has to be applied instead of the previous one.)
---
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 7c7bc8bef21f..f986636ad9d0 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -742,6 +742,41 @@ TRACE_EVENT(kvm_emulate_insn,
#define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
#define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
+TRACE_EVENT(kvm_0xef,
+ TP_PROTO(bool irr, bool isr, u32 info, bool on, bool pir, u16 status),
+ TP_ARGS(irr, isr, info, on, pir, status),
+
+ TP_STRUCT__entry(
+ __field(bool, irr )
+ __field(bool, isr )
+ __field(u32, info)
+ __field(bool, on )
+ __field(bool, pir )
+ __field(u8, rvi )
+ __field(u8, svi )
+ ),
+
+ TP_fast_assign(
+ __entry->irr = irr;
+ __entry->isr = isr;
+ __entry->info = info;
+ __entry->on = on;
+ __entry->pir = pir;
+ __entry->rvi = status & 0xff;
+ __entry->svi = status >> 8;
+ ),
+
+ TP_printk("irr %s, isr %s, info 0x%x, on %s, pir %s, rvi 0x%x, svi 0x%x",
+ __entry->irr ? "set " : "clear",
+ __entry->isr ? "set " : "clear",
+ __entry->info,
+ __entry->on ? "set " : "clear",
+ __entry->pir ? "set " : "clear",
+ __entry->rvi,
+ __entry->svi
+ )
+ );
+
TRACE_EVENT(
vcpu_match_mmio,
TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index eee63dc33d89..b461edc93d53 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5047,6 +5047,25 @@ static int handle_machine_check(struct kvm_vcpu *vcpu)
return 1;
}
+#define VEC_POS(v) ((v) & (32 - 1))
+#define REG_POS(v) (((v) >> 5) << 4)
+static inline int apic_test_vector(int vec, void *bitmap)
+{
+ return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
+}
+
+static inline void random_trace(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+ trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
+ apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
+ vmcs_read32(VM_ENTRY_INTR_INFO_FIELD),
+ test_bit(POSTED_INTR_ON, (unsigned long *)&vmx->pi_desc.control),
+ test_bit(0xef, (unsigned long *)vmx->pi_desc.pir),
+ vmcs_read16(GUEST_INTR_STATUS));
+}
+
static int handle_exception(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -5077,6 +5096,8 @@ static int handle_exception(struct kvm_vcpu *vcpu)
return 1;
}
+ random_trace(vcpu);
+
error_code = 0;
if (intr_info & INTR_INFO_DELIVER_CODE_MASK)
error_code = vmcs_read32(VM_EXIT_INTR_ERROR_CODE);
@@ -8143,6 +8164,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
if (vmx->emulation_required)
return;
+ random_trace(vcpu);
+
if (vmx->ple_window_dirty) {
vmx->ple_window_dirty = false;
vmcs_write32(PLE_WINDOW, vmx->ple_window);
@@ -8312,6 +8335,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
vmx->exit_reason = vmcs_read32(VM_EXIT_REASON);
trace_kvm_exit(vmx->exit_reason, vcpu, KVM_ISA_VMX);
+ random_trace(vcpu);
+
/*
* the KVM_REQ_EVENT optimization bit is only on for one entry, and if
* we did not inject a still-pending event to L1 now because of
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32bf19ef3115..a45fa01bd354 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 16:45 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-03-31 16:45 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-31 17:56+0300, Andrey Korolyov:
> > Chasing the culprit this way could take a long time, so a new tracepoint
> > that shows if 0xef is set on entry would let us guess the bug faster ...
> >
> > Please provide a failing trace with the following patch:
>
> Thanks, please see below:
>
> http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
qemu-system-x86-4022 [006] 255.915978:
kvm_entry: vcpu 0
kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
kvm_emulate_insn: f0000:d280: 31 c0
kvm_emulate_insn: f0000:d282: 8e e0
kvm_emulate_insn: f0000:d284: 8e e8
kvm_emulate_insn: f0000:d286: 8e c0
kvm_emulate_insn: f0000:d288: 8e d8
kvm_emulate_insn: f0000:d28a: 8e d0
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
kvm_page_fault: address f8dd0 error_code 184
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
kvm_page_fault: address f76d6 error_code 184
kvm_entry: vcpu 0
kvm_0xef: irr clear, isr clear, vmcs 0x0
kvm_exit: reason EXCEPTION_NMI rip 0xd331 info 0 80000b0d
kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
Ok, nothing obvious here either ... I've desperately added all
information I know about. Please run it again, thanks.
(The patch has to be applied instead of the previous one.)
---
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 7c7bc8bef21f..f986636ad9d0 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -742,6 +742,41 @@ TRACE_EVENT(kvm_emulate_insn,
#define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
#define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
+TRACE_EVENT(kvm_0xef,
+ TP_PROTO(bool irr, bool isr, u32 info, bool on, bool pir, u16 status),
+ TP_ARGS(irr, isr, info, on, pir, status),
+
+ TP_STRUCT__entry(
+ __field(bool, irr )
+ __field(bool, isr )
+ __field(u32, info)
+ __field(bool, on )
+ __field(bool, pir )
+ __field(u8, rvi )
+ __field(u8, svi )
+ ),
+
+ TP_fast_assign(
+ __entry->irr = irr;
+ __entry->isr = isr;
+ __entry->info = info;
+ __entry->on = on;
+ __entry->pir = pir;
+ __entry->rvi = status & 0xff;
+ __entry->svi = status >> 8;
+ ),
+
+ TP_printk("irr %s, isr %s, info 0x%x, on %s, pir %s, rvi 0x%x, svi 0x%x",
+ __entry->irr ? "set " : "clear",
+ __entry->isr ? "set " : "clear",
+ __entry->info,
+ __entry->on ? "set " : "clear",
+ __entry->pir ? "set " : "clear",
+ __entry->rvi,
+ __entry->svi
+ )
+ );
+
TRACE_EVENT(
vcpu_match_mmio,
TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index eee63dc33d89..b461edc93d53 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5047,6 +5047,25 @@ static int handle_machine_check(struct kvm_vcpu *vcpu)
return 1;
}
+#define VEC_POS(v) ((v) & (32 - 1))
+#define REG_POS(v) (((v) >> 5) << 4)
+static inline int apic_test_vector(int vec, void *bitmap)
+{
+ return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
+}
+
+static inline void random_trace(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+ trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
+ apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
+ vmcs_read32(VM_ENTRY_INTR_INFO_FIELD),
+ test_bit(POSTED_INTR_ON, (unsigned long *)&vmx->pi_desc.control),
+ test_bit(0xef, (unsigned long *)vmx->pi_desc.pir),
+ vmcs_read16(GUEST_INTR_STATUS));
+}
+
static int handle_exception(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -5077,6 +5096,8 @@ static int handle_exception(struct kvm_vcpu *vcpu)
return 1;
}
+ random_trace(vcpu);
+
error_code = 0;
if (intr_info & INTR_INFO_DELIVER_CODE_MASK)
error_code = vmcs_read32(VM_EXIT_INTR_ERROR_CODE);
@@ -8143,6 +8164,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
if (vmx->emulation_required)
return;
+ random_trace(vcpu);
+
if (vmx->ple_window_dirty) {
vmx->ple_window_dirty = false;
vmcs_write32(PLE_WINDOW, vmx->ple_window);
@@ -8312,6 +8335,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
vmx->exit_reason = vmcs_read32(VM_EXIT_REASON);
trace_kvm_exit(vmx->exit_reason, vcpu, KVM_ISA_VMX);
+ random_trace(vcpu);
+
/*
* the KVM_REQ_EVENT optimization bit is only on for one entry, and if
* we did not inject a still-pending event to L1 now because of
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32bf19ef3115..a45fa01bd354 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
^ permalink raw reply related [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-31 16:45 ` [Qemu-devel] " Radim Krčmář
@ 2015-03-31 17:40 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 17:40 UTC (permalink / raw)
To: Radim Krčmář
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Tue, Mar 31, 2015 at 7:45 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-31 17:56+0300, Andrey Korolyov:
>> > Chasing the culprit this way could take a long time, so a new tracepoint
>> > that shows if 0xef is set on entry would let us guess the bug faster ...
>> >
>> > Please provide a failing trace with the following patch:
>>
>> Thanks, please see below:
>>
>> http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
>
> qemu-system-x86-4022 [006] 255.915978:
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EXCEPTION_NMI rip 0xd331 info 0 80000b0d
> kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>
> Ok, nothing obvious here either ... I've desperately added all
> information I know about. Please run it again, thanks.
>
> (The patch has to be applied instead of the previous one.)
> ---
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index 7c7bc8bef21f..f986636ad9d0 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -742,6 +742,41 @@ TRACE_EVENT(kvm_emulate_insn,
> #define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
> #define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
>
> +TRACE_EVENT(kvm_0xef,
> + TP_PROTO(bool irr, bool isr, u32 info, bool on, bool pir, u16 status),
> + TP_ARGS(irr, isr, info, on, pir, status),
> +
> + TP_STRUCT__entry(
> + __field(bool, irr )
> + __field(bool, isr )
> + __field(u32, info)
> + __field(bool, on )
> + __field(bool, pir )
> + __field(u8, rvi )
> + __field(u8, svi )
> + ),
> +
> + TP_fast_assign(
> + __entry->irr = irr;
> + __entry->isr = isr;
> + __entry->info = info;
> + __entry->on = on;
> + __entry->pir = pir;
> + __entry->rvi = status & 0xff;
> + __entry->svi = status >> 8;
> + ),
> +
> + TP_printk("irr %s, isr %s, info 0x%x, on %s, pir %s, rvi 0x%x, svi 0x%x",
> + __entry->irr ? "set " : "clear",
> + __entry->isr ? "set " : "clear",
> + __entry->info,
> + __entry->on ? "set " : "clear",
> + __entry->pir ? "set " : "clear",
> + __entry->rvi,
> + __entry->svi
> + )
> + );
> +
> TRACE_EVENT(
> vcpu_match_mmio,
> TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index eee63dc33d89..b461edc93d53 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5047,6 +5047,25 @@ static int handle_machine_check(struct kvm_vcpu *vcpu)
> return 1;
> }
>
> +#define VEC_POS(v) ((v) & (32 - 1))
> +#define REG_POS(v) (((v) >> 5) << 4)
> +static inline int apic_test_vector(int vec, void *bitmap)
> +{
> + return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
> +}
> +
> +static inline void random_trace(struct kvm_vcpu *vcpu)
> +{
> + struct vcpu_vmx *vmx = to_vmx(vcpu);
> +
> + trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
> + apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
> + vmcs_read32(VM_ENTRY_INTR_INFO_FIELD),
> + test_bit(POSTED_INTR_ON, (unsigned long *)&vmx->pi_desc.control),
> + test_bit(0xef, (unsigned long *)vmx->pi_desc.pir),
> + vmcs_read16(GUEST_INTR_STATUS));
> +}
> +
> static int handle_exception(struct kvm_vcpu *vcpu)
> {
> struct vcpu_vmx *vmx = to_vmx(vcpu);
> @@ -5077,6 +5096,8 @@ static int handle_exception(struct kvm_vcpu *vcpu)
> return 1;
> }
>
> + random_trace(vcpu);
> +
> error_code = 0;
> if (intr_info & INTR_INFO_DELIVER_CODE_MASK)
> error_code = vmcs_read32(VM_EXIT_INTR_ERROR_CODE);
> @@ -8143,6 +8164,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> if (vmx->emulation_required)
> return;
>
> + random_trace(vcpu);
> +
> if (vmx->ple_window_dirty) {
> vmx->ple_window_dirty = false;
> vmcs_write32(PLE_WINDOW, vmx->ple_window);
> @@ -8312,6 +8335,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> vmx->exit_reason = vmcs_read32(VM_EXIT_REASON);
> trace_kvm_exit(vmx->exit_reason, vcpu, KVM_ISA_VMX);
>
> + random_trace(vcpu);
> +
> /*
> * the KVM_REQ_EVENT optimization bit is only on for one entry, and if
> * we did not inject a still-pending event to L1 now because of
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 32bf19ef3115..a45fa01bd354 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
Something a bit more interesting, but the mess is happening just
*after* NMI firing.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 17:40 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 17:40 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Tue, Mar 31, 2015 at 7:45 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-31 17:56+0300, Andrey Korolyov:
>> > Chasing the culprit this way could take a long time, so a new tracepoint
>> > that shows if 0xef is set on entry would let us guess the bug faster ...
>> >
>> > Please provide a failing trace with the following patch:
>>
>> Thanks, please see below:
>>
>> http://xdel.ru/downloads/kvm-e5v2-issue/new-tracepoint-fail-with-apicv.dat.gz
>
> qemu-system-x86-4022 [006] 255.915978:
> kvm_entry: vcpu 0
> kvm_emulate_insn: f0000:d275: ea 7a d2 00 f0
> kvm_emulate_insn: f0000:d27a: 2e 0f 01 1e f0 6c
> kvm_emulate_insn: f0000:d280: 31 c0
> kvm_emulate_insn: f0000:d282: 8e e0
> kvm_emulate_insn: f0000:d284: 8e e8
> kvm_emulate_insn: f0000:d286: 8e c0
> kvm_emulate_insn: f0000:d288: 8e d8
> kvm_emulate_insn: f0000:d28a: 8e d0
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EPT_VIOLATION rip 0x8dd0 info 184 0
> kvm_page_fault: address f8dd0 error_code 184
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EPT_VIOLATION rip 0x76d6 info 184 0
> kvm_page_fault: address f76d6 error_code 184
> kvm_entry: vcpu 0
> kvm_0xef: irr clear, isr clear, vmcs 0x0
> kvm_exit: reason EXCEPTION_NMI rip 0xd331 info 0 80000b0d
> kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17)
>
> Ok, nothing obvious here either ... I've desperately added all
> information I know about. Please run it again, thanks.
>
> (The patch has to be applied instead of the previous one.)
> ---
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index 7c7bc8bef21f..f986636ad9d0 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -742,6 +742,41 @@ TRACE_EVENT(kvm_emulate_insn,
> #define trace_kvm_emulate_insn_start(vcpu) trace_kvm_emulate_insn(vcpu, 0)
> #define trace_kvm_emulate_insn_failed(vcpu) trace_kvm_emulate_insn(vcpu, 1)
>
> +TRACE_EVENT(kvm_0xef,
> + TP_PROTO(bool irr, bool isr, u32 info, bool on, bool pir, u16 status),
> + TP_ARGS(irr, isr, info, on, pir, status),
> +
> + TP_STRUCT__entry(
> + __field(bool, irr )
> + __field(bool, isr )
> + __field(u32, info)
> + __field(bool, on )
> + __field(bool, pir )
> + __field(u8, rvi )
> + __field(u8, svi )
> + ),
> +
> + TP_fast_assign(
> + __entry->irr = irr;
> + __entry->isr = isr;
> + __entry->info = info;
> + __entry->on = on;
> + __entry->pir = pir;
> + __entry->rvi = status & 0xff;
> + __entry->svi = status >> 8;
> + ),
> +
> + TP_printk("irr %s, isr %s, info 0x%x, on %s, pir %s, rvi 0x%x, svi 0x%x",
> + __entry->irr ? "set " : "clear",
> + __entry->isr ? "set " : "clear",
> + __entry->info,
> + __entry->on ? "set " : "clear",
> + __entry->pir ? "set " : "clear",
> + __entry->rvi,
> + __entry->svi
> + )
> + );
> +
> TRACE_EVENT(
> vcpu_match_mmio,
> TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match),
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index eee63dc33d89..b461edc93d53 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5047,6 +5047,25 @@ static int handle_machine_check(struct kvm_vcpu *vcpu)
> return 1;
> }
>
> +#define VEC_POS(v) ((v) & (32 - 1))
> +#define REG_POS(v) (((v) >> 5) << 4)
> +static inline int apic_test_vector(int vec, void *bitmap)
> +{
> + return test_bit(VEC_POS(vec), (bitmap) + REG_POS(vec));
> +}
> +
> +static inline void random_trace(struct kvm_vcpu *vcpu)
> +{
> + struct vcpu_vmx *vmx = to_vmx(vcpu);
> +
> + trace_kvm_0xef(apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_IRR),
> + apic_test_vector(0xef, vcpu->arch.apic->regs + APIC_ISR),
> + vmcs_read32(VM_ENTRY_INTR_INFO_FIELD),
> + test_bit(POSTED_INTR_ON, (unsigned long *)&vmx->pi_desc.control),
> + test_bit(0xef, (unsigned long *)vmx->pi_desc.pir),
> + vmcs_read16(GUEST_INTR_STATUS));
> +}
> +
> static int handle_exception(struct kvm_vcpu *vcpu)
> {
> struct vcpu_vmx *vmx = to_vmx(vcpu);
> @@ -5077,6 +5096,8 @@ static int handle_exception(struct kvm_vcpu *vcpu)
> return 1;
> }
>
> + random_trace(vcpu);
> +
> error_code = 0;
> if (intr_info & INTR_INFO_DELIVER_CODE_MASK)
> error_code = vmcs_read32(VM_EXIT_INTR_ERROR_CODE);
> @@ -8143,6 +8164,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> if (vmx->emulation_required)
> return;
>
> + random_trace(vcpu);
> +
> if (vmx->ple_window_dirty) {
> vmx->ple_window_dirty = false;
> vmcs_write32(PLE_WINDOW, vmx->ple_window);
> @@ -8312,6 +8335,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
> vmx->exit_reason = vmcs_read32(VM_EXIT_REASON);
> trace_kvm_exit(vmx->exit_reason, vcpu, KVM_ISA_VMX);
>
> + random_trace(vcpu);
> +
> /*
> * the KVM_REQ_EVENT optimization bit is only on for one entry, and if
> * we did not inject a still-pending event to L1 now because of
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 32bf19ef3115..a45fa01bd354 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -7881,3 +7881,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_write_tsc_offset);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ple_window);
> EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pml_full);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_0xef);
http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
Something a bit more interesting, but the mess is happening just
*after* NMI firing.
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-31 17:40 ` Andrey Korolyov
@ 2015-03-31 18:01 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-31 18:01 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Radim Krčmář,
Kevin O'Connor, Dr. David Alan Gilbert, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
Andrey Korolyov <andrey@xdel.ru> writes:
...
> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>
> Something a bit more interesting, but the mess is happening just
> *after* NMI firing.
What happens if NMI is turned off on the host ?
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 18:01 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-31 18:01 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
Andrey Korolyov <andrey@xdel.ru> writes:
...
> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>
> Something a bit more interesting, but the mess is happening just
> *after* NMI firing.
What happens if NMI is turned off on the host ?
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-03-31 18:01 ` Bandan Das
@ 2015-03-31 18:04 ` Bandan Das
-1 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-31 18:04 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
Bandan Das <bsd@redhat.com> writes:
> Andrey Korolyov <andrey@xdel.ru> writes:
> ...
>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>
>> Something a bit more interesting, but the mess is happening just
>> *after* NMI firing.
>
> What happens if NMI is turned off on the host ?
Sorry, I meant the watchdog..
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 18:04 ` Bandan Das
0 siblings, 0 replies; 157+ messages in thread
From: Bandan Das @ 2015-03-31 18:04 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
Bandan Das <bsd@redhat.com> writes:
> Andrey Korolyov <andrey@xdel.ru> writes:
> ...
>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>
>> Something a bit more interesting, but the mess is happening just
>> *after* NMI firing.
>
> What happens if NMI is turned off on the host ?
Sorry, I meant the watchdog..
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-31 18:04 ` [Qemu-devel] " Bandan Das
@ 2015-03-31 18:23 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 18:23 UTC (permalink / raw)
To: Bandan Das
Cc: Radim Krčmář,
Kevin O'Connor, Dr. David Alan Gilbert, Paolo Bonzini,
Gerd Hoffmann, qemu-devel, kvm
On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
> Bandan Das <bsd@redhat.com> writes:
>
>> Andrey Korolyov <andrey@xdel.ru> writes:
>> ...
>>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>>
>>> Something a bit more interesting, but the mess is happening just
>>> *after* NMI firing.
>>
>> What happens if NMI is turned off on the host ?
>
> Sorry, I meant the watchdog..
Thanks, everything goes well (as it probably should go there):
http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-03-31 18:23 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-03-31 18:23 UTC (permalink / raw)
To: Bandan Das
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Kevin O'Connor,
Gerd Hoffmann, Paolo Bonzini
On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
> Bandan Das <bsd@redhat.com> writes:
>
>> Andrey Korolyov <andrey@xdel.ru> writes:
>> ...
>>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>>
>>> Something a bit more interesting, but the mess is happening just
>>> *after* NMI firing.
>>
>> What happens if NMI is turned off on the host ?
>
> Sorry, I meant the watchdog..
Thanks, everything goes well (as it probably should go there):
http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-03-31 18:23 ` Andrey Korolyov
@ 2015-04-01 11:49 ` Radim Krčmář
-1 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-04-01 11:49 UTC (permalink / raw)
To: Andrey Korolyov
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
2015-03-31 21:23+0300, Andrey Korolyov:
> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
> > Bandan Das <bsd@redhat.com> writes:
> >> Andrey Korolyov <andrey@xdel.ru> writes:
> >> ...
> >>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
> >>>
> >>> Something a bit more interesting, but the mess is happening just
> >>> *after* NMI firing.
> >>
> >> What happens if NMI is turned off on the host ?
> >
> > Sorry, I meant the watchdog..
>
> Thanks, everything goes well (as it probably should go there):
> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
Nice revelation!
KVM doesn't expect host's NMIs to look like this so it doesn't pass them
to the host. What was the watchdog that casually sent NMIs?
(It worked after "nmi_watchdog=0" on the host?)
(Guest's NMI should have a different result as well. NMI_EXCEPTION is
an expected exit reason for guest's hard exceptions, they are then
differentiated by intr_info and nothing hinted that this was a NMI.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 11:49 ` Radim Krčmář
0 siblings, 0 replies; 157+ messages in thread
From: Radim Krčmář @ 2015-04-01 11:49 UTC (permalink / raw)
To: Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
2015-03-31 21:23+0300, Andrey Korolyov:
> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
> > Bandan Das <bsd@redhat.com> writes:
> >> Andrey Korolyov <andrey@xdel.ru> writes:
> >> ...
> >>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
> >>>
> >>> Something a bit more interesting, but the mess is happening just
> >>> *after* NMI firing.
> >>
> >> What happens if NMI is turned off on the host ?
> >
> > Sorry, I meant the watchdog..
>
> Thanks, everything goes well (as it probably should go there):
> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
Nice revelation!
KVM doesn't expect host's NMIs to look like this so it doesn't pass them
to the host. What was the watchdog that casually sent NMIs?
(It worked after "nmi_watchdog=0" on the host?)
(Guest's NMI should have a different result as well. NMI_EXCEPTION is
an expected exit reason for guest's hard exceptions, they are then
differentiated by intr_info and nothing hinted that this was a NMI.)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 11:49 ` Radim Krčmář
@ 2015-04-01 12:05 ` Paolo Bonzini
-1 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-04-01 12:05 UTC (permalink / raw)
To: Radim Krčmář, Andrey Korolyov
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Gerd Hoffmann, qemu-devel, kvm
On 01/04/2015 13:49, Radim Krčmář wrote:
> 2015-03-31 21:23+0300, Andrey Korolyov:
>> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
>>> Bandan Das <bsd@redhat.com> writes:
>>>> Andrey Korolyov <andrey@xdel.ru> writes:
>>>> ...
>>>>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>>>>
>>>>> Something a bit more interesting, but the mess is happening just
>>>>> *after* NMI firing.
>>>>
>>>> What happens if NMI is turned off on the host ?
>>>
>>> Sorry, I meant the watchdog..
>>
>> Thanks, everything goes well (as it probably should go there):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
>
> Nice revelation!
Yes, pretty random but good to know. Can you try again with the
nmi/nmi_handler tracepoint also?
Paolo
> KVM doesn't expect host's NMIs to look like this so it doesn't pass them
> to the host. What was the watchdog that casually sent NMIs?
> (It worked after "nmi_watchdog=0" on the host?)
>
> (Guest's NMI should have a different result as well. NMI_EXCEPTION is
> an expected exit reason for guest's hard exceptions, they are then
> differentiated by intr_info and nothing hinted that this was a NMI.)
>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 12:05 ` Paolo Bonzini
0 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-04-01 12:05 UTC (permalink / raw)
To: Radim Krčmář, Andrey Korolyov
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
On 01/04/2015 13:49, Radim Krčmář wrote:
> 2015-03-31 21:23+0300, Andrey Korolyov:
>> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
>>> Bandan Das <bsd@redhat.com> writes:
>>>> Andrey Korolyov <andrey@xdel.ru> writes:
>>>> ...
>>>>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>>>>>
>>>>> Something a bit more interesting, but the mess is happening just
>>>>> *after* NMI firing.
>>>>
>>>> What happens if NMI is turned off on the host ?
>>>
>>> Sorry, I meant the watchdog..
>>
>> Thanks, everything goes well (as it probably should go there):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
>
> Nice revelation!
Yes, pretty random but good to know. Can you try again with the
nmi/nmi_handler tracepoint also?
Paolo
> KVM doesn't expect host's NMIs to look like this so it doesn't pass them
> to the host. What was the watchdog that casually sent NMIs?
> (It worked after "nmi_watchdog=0" on the host?)
>
> (Guest's NMI should have a different result as well. NMI_EXCEPTION is
> an expected exit reason for guest's hard exceptions, they are then
> differentiated by intr_info and nothing hinted that this was a NMI.)
>
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 11:49 ` Radim Krčmář
@ 2015-04-01 12:26 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 12:26 UTC (permalink / raw)
To: Radim Krčmář
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Paolo Bonzini, Gerd Hoffmann, qemu-devel, kvm
On Wed, Apr 1, 2015 at 2:49 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-31 21:23+0300, Andrey Korolyov:
>> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
>> > Bandan Das <bsd@redhat.com> writes:
>> >> Andrey Korolyov <andrey@xdel.ru> writes:
>> >> ...
>> >>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>> >>>
>> >>> Something a bit more interesting, but the mess is happening just
>> >>> *after* NMI firing.
>> >>
>> >> What happens if NMI is turned off on the host ?
>> >
>> > Sorry, I meant the watchdog..
>>
>> Thanks, everything goes well (as it probably should go there):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
>
> Nice revelation!
>
> KVM doesn't expect host's NMIs to look like this so it doesn't pass them
> to the host. What was the watchdog that casually sent NMIs?
> (It worked after "nmi_watchdog=0" on the host?)
>
> (Guest's NMI should have a different result as well. NMI_EXCEPTION is
> an expected exit reason for guest's hard exceptions, they are then
> differentiated by intr_info and nothing hinted that this was a NMI.)
Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
would look different and they had no reasons to be fired at this stage
inside guest. I`d suspect a hypervisor hardware misbehavior there but
have a very little idea on how APICv behavior (which is completely
microcode-dependent and CPU-dependent but decoupled from peripheral
hardware) may vary at this point, I am using 1.20140913.1 ucode
version from debian if this can matter. Will send trace suggested by
Paolo in a next couple of hours. Also it would be awesome to ask
hardware folks from Intel who can prove or disprove my abovementioned
statement (as I was unable to catch the problem on 2603v2 so far, this
hypothesis has some chance to be real).
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 12:26 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 12:26 UTC (permalink / raw)
To: Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann, Paolo Bonzini
On Wed, Apr 1, 2015 at 2:49 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2015-03-31 21:23+0300, Andrey Korolyov:
>> On Tue, Mar 31, 2015 at 9:04 PM, Bandan Das <bsd@redhat.com> wrote:
>> > Bandan Das <bsd@redhat.com> writes:
>> >> Andrey Korolyov <andrey@xdel.ru> writes:
>> >> ...
>> >>> http://xdel.ru/downloads/kvm-e5v2-issue/another-tracepoint-fail-with-apicv.dat.gz
>> >>>
>> >>> Something a bit more interesting, but the mess is happening just
>> >>> *after* NMI firing.
>> >>
>> >> What happens if NMI is turned off on the host ?
>> >
>> > Sorry, I meant the watchdog..
>>
>> Thanks, everything goes well (as it probably should go there):
>> http://xdel.ru/downloads/kvm-e5v2-issue/apicv-enabled-nmi-disabled.dat.gz
>
> Nice revelation!
>
> KVM doesn't expect host's NMIs to look like this so it doesn't pass them
> to the host. What was the watchdog that casually sent NMIs?
> (It worked after "nmi_watchdog=0" on the host?)
>
> (Guest's NMI should have a different result as well. NMI_EXCEPTION is
> an expected exit reason for guest's hard exceptions, they are then
> differentiated by intr_info and nothing hinted that this was a NMI.)
Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
would look different and they had no reasons to be fired at this stage
inside guest. I`d suspect a hypervisor hardware misbehavior there but
have a very little idea on how APICv behavior (which is completely
microcode-dependent and CPU-dependent but decoupled from peripheral
hardware) may vary at this point, I am using 1.20140913.1 ucode
version from debian if this can matter. Will send trace suggested by
Paolo in a next couple of hours. Also it would be awesome to ask
hardware folks from Intel who can prove or disprove my abovementioned
statement (as I was unable to catch the problem on 2603v2 so far, this
hypothesis has some chance to be real).
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 12:26 ` Andrey Korolyov
@ 2015-04-01 13:19 ` Paolo Bonzini
-1 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-04-01 13:19 UTC (permalink / raw)
To: Andrey Korolyov, Radim Krčmář
Cc: Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Gerd Hoffmann, qemu-devel, kvm
On 01/04/2015 14:26, Andrey Korolyov wrote:
> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
> would look different and they had no reasons to be fired at this stage
> inside guest. I`d suspect a hypervisor hardware misbehavior there but
> have a very little idea on how APICv behavior (which is completely
> microcode-dependent and CPU-dependent but decoupled from peripheral
> hardware) may vary at this point, I am using 1.20140913.1 ucode
> version from debian if this can matter. Will send trace suggested by
> Paolo in a next couple of hours. Also it would be awesome to ask
> hardware folks from Intel who can prove or disprove my abovementioned
> statement (as I was unable to catch the problem on 2603v2 so far, this
> hypothesis has some chance to be real).
Yes, the interaction with the NMI watchdog is unexpected and makes a
processor erratum somewhat more likely.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 13:19 ` Paolo Bonzini
0 siblings, 0 replies; 157+ messages in thread
From: Paolo Bonzini @ 2015-04-01 13:19 UTC (permalink / raw)
To: Andrey Korolyov, Radim Krčmář
Cc: kvm, qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
On 01/04/2015 14:26, Andrey Korolyov wrote:
> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
> would look different and they had no reasons to be fired at this stage
> inside guest. I`d suspect a hypervisor hardware misbehavior there but
> have a very little idea on how APICv behavior (which is completely
> microcode-dependent and CPU-dependent but decoupled from peripheral
> hardware) may vary at this point, I am using 1.20140913.1 ucode
> version from debian if this can matter. Will send trace suggested by
> Paolo in a next couple of hours. Also it would be awesome to ask
> hardware folks from Intel who can prove or disprove my abovementioned
> statement (as I was unable to catch the problem on 2603v2 so far, this
> hypothesis has some chance to be real).
Yes, the interaction with the NMI watchdog is unexpected and makes a
processor erratum somewhat more likely.
Paolo
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 13:19 ` Paolo Bonzini
@ 2015-04-01 15:37 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 15:37 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Radim Krčmář,
Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Gerd Hoffmann, qemu-devel, kvm
On Wed, Apr 1, 2015 at 4:19 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 01/04/2015 14:26, Andrey Korolyov wrote:
>> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
>> would look different and they had no reasons to be fired at this stage
>> inside guest. I`d suspect a hypervisor hardware misbehavior there but
>> have a very little idea on how APICv behavior (which is completely
>> microcode-dependent and CPU-dependent but decoupled from peripheral
>> hardware) may vary at this point, I am using 1.20140913.1 ucode
>> version from debian if this can matter. Will send trace suggested by
>> Paolo in a next couple of hours. Also it would be awesome to ask
>> hardware folks from Intel who can prove or disprove my abovementioned
>> statement (as I was unable to catch the problem on 2603v2 so far, this
>> hypothesis has some chance to be real).
>
> Yes, the interaction with the NMI watchdog is unexpected and makes a
> processor erratum somewhat more likely.
>
> Paolo
http://xdel.ru/downloads/kvm-e5v2-issue/trace-nmi-apicv-fail-at-reboot.dat.gz
err, no NMI entries nearby failure event, though capture should be correct:
/sys/kernel/debug/tracing/events/kvm*/filter
/sys/kernel/debug/tracing/events/*/kvm*/filter
/sys/kernel/debug/tracing/events/nmi*/filter
/sys/kernel/debug/tracing/events/*/nmi*/filter
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 15:37 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 15:37 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
On Wed, Apr 1, 2015 at 4:19 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 01/04/2015 14:26, Andrey Korolyov wrote:
>> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
>> would look different and they had no reasons to be fired at this stage
>> inside guest. I`d suspect a hypervisor hardware misbehavior there but
>> have a very little idea on how APICv behavior (which is completely
>> microcode-dependent and CPU-dependent but decoupled from peripheral
>> hardware) may vary at this point, I am using 1.20140913.1 ucode
>> version from debian if this can matter. Will send trace suggested by
>> Paolo in a next couple of hours. Also it would be awesome to ask
>> hardware folks from Intel who can prove or disprove my abovementioned
>> statement (as I was unable to catch the problem on 2603v2 so far, this
>> hypothesis has some chance to be real).
>
> Yes, the interaction with the NMI watchdog is unexpected and makes a
> processor erratum somewhat more likely.
>
> Paolo
http://xdel.ru/downloads/kvm-e5v2-issue/trace-nmi-apicv-fail-at-reboot.dat.gz
err, no NMI entries nearby failure event, though capture should be correct:
/sys/kernel/debug/tracing/events/kvm*/filter
/sys/kernel/debug/tracing/events/*/kvm*/filter
/sys/kernel/debug/tracing/events/nmi*/filter
/sys/kernel/debug/tracing/events/*/nmi*/filter
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 15:37 ` Andrey Korolyov
@ 2015-04-01 16:29 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 16:29 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Radim Krčmář,
Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Gerd Hoffmann, qemu-devel, kvm
On Wed, Apr 1, 2015 at 6:37 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Wed, Apr 1, 2015 at 4:19 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 01/04/2015 14:26, Andrey Korolyov wrote:
>>> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
>>> would look different and they had no reasons to be fired at this stage
>>> inside guest. I`d suspect a hypervisor hardware misbehavior there but
>>> have a very little idea on how APICv behavior (which is completely
>>> microcode-dependent and CPU-dependent but decoupled from peripheral
>>> hardware) may vary at this point, I am using 1.20140913.1 ucode
>>> version from debian if this can matter. Will send trace suggested by
>>> Paolo in a next couple of hours. Also it would be awesome to ask
>>> hardware folks from Intel who can prove or disprove my abovementioned
>>> statement (as I was unable to catch the problem on 2603v2 so far, this
>>> hypothesis has some chance to be real).
>>
>> Yes, the interaction with the NMI watchdog is unexpected and makes a
>> processor erratum somewhat more likely.
>>
>> Paolo
>
>
> http://xdel.ru/downloads/kvm-e5v2-issue/trace-nmi-apicv-fail-at-reboot.dat.gz
>
> err, no NMI entries nearby failure event, though capture should be correct:
> /sys/kernel/debug/tracing/events/kvm*/filter
> /sys/kernel/debug/tracing/events/*/kvm*/filter
> /sys/kernel/debug/tracing/events/nmi*/filter
> /sys/kernel/debug/tracing/events/*/nmi*/filter
Moved 2603v2s back and issue is still here. I used wrong pattern for
the issue on a previous series of tests on those CPUs in the middle of
month, continuously respawning VMs when the real issue is hiding in
*first* reboot events starting from the hypervisor reboot (or module
load). So either it should be reproducible anywhere or this is not a
hardware issue (or it is related to the mainboard instead of CPU
itself :) ).
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 16:29 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 16:29 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
On Wed, Apr 1, 2015 at 6:37 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Wed, Apr 1, 2015 at 4:19 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 01/04/2015 14:26, Andrey Korolyov wrote:
>>> Yes, I disabled host watchdog during runtime. Indeed guest-induced NMI
>>> would look different and they had no reasons to be fired at this stage
>>> inside guest. I`d suspect a hypervisor hardware misbehavior there but
>>> have a very little idea on how APICv behavior (which is completely
>>> microcode-dependent and CPU-dependent but decoupled from peripheral
>>> hardware) may vary at this point, I am using 1.20140913.1 ucode
>>> version from debian if this can matter. Will send trace suggested by
>>> Paolo in a next couple of hours. Also it would be awesome to ask
>>> hardware folks from Intel who can prove or disprove my abovementioned
>>> statement (as I was unable to catch the problem on 2603v2 so far, this
>>> hypothesis has some chance to be real).
>>
>> Yes, the interaction with the NMI watchdog is unexpected and makes a
>> processor erratum somewhat more likely.
>>
>> Paolo
>
>
> http://xdel.ru/downloads/kvm-e5v2-issue/trace-nmi-apicv-fail-at-reboot.dat.gz
>
> err, no NMI entries nearby failure event, though capture should be correct:
> /sys/kernel/debug/tracing/events/kvm*/filter
> /sys/kernel/debug/tracing/events/*/kvm*/filter
> /sys/kernel/debug/tracing/events/nmi*/filter
> /sys/kernel/debug/tracing/events/*/nmi*/filter
Moved 2603v2s back and issue is still here. I used wrong pattern for
the issue on a previous series of tests on those CPUs in the middle of
month, continuously respawning VMs when the real issue is hiding in
*first* reboot events starting from the hypervisor reboot (or module
load). So either it should be reproducible anywhere or this is not a
hardware issue (or it is related to the mainboard instead of CPU
itself :) ).
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
2015-04-01 16:29 ` Andrey Korolyov
@ 2015-04-01 22:58 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 22:58 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Radim Krčmář,
Bandan Das, Kevin O'Connor, Dr. David Alan Gilbert,
Gerd Hoffmann, qemu-devel, kvm
*putting my tinfoil hat on*
After thinking a little bit more, the observable behavior is a quite
good match for a bios-level hypervisor (hardware trojan in a modern
terminology), as it likely is sensitive to timing[1], does not appear
more than once per VM during boot cycle and seemingly does not regard
a fact if kvm-intel was reloaded once or twice (or more) and not
reproducible outside of domain of a single board model. If nobody has
a better suggestions to try on, I`ll do a couple of steps in a next
days:
- extract and compare bios to the vendor`s image with SPI programmer,
- extract and compare BMC image with public version (should be easy as well),
- try to analyze switch timings by writing sample code for a bare
hardware (there can be a hint that the L2 Linux guest can expose
larger execution time difference with L1 on host with top-level
hypervisor than on supposedly 'non-infected' one),
- try to analyze binary BIOS code itself, though it can be VERY
problematic, I am even not talking for same possibility for BMC.
Sorry for posting such a naive and stupid stuff in the public ml, but
I am really out of clues of what`s happening there and why it is not
reproducible anywhere else.
1. https://xakep.ru/2011/12/26/58104/ (russian text, but can be read
through g-translate without lack of details)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-01 22:58 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-01 22:58 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
*putting my tinfoil hat on*
After thinking a little bit more, the observable behavior is a quite
good match for a bios-level hypervisor (hardware trojan in a modern
terminology), as it likely is sensitive to timing[1], does not appear
more than once per VM during boot cycle and seemingly does not regard
a fact if kvm-intel was reloaded once or twice (or more) and not
reproducible outside of domain of a single board model. If nobody has
a better suggestions to try on, I`ll do a couple of steps in a next
days:
- extract and compare bios to the vendor`s image with SPI programmer,
- extract and compare BMC image with public version (should be easy as well),
- try to analyze switch timings by writing sample code for a bare
hardware (there can be a hint that the L2 Linux guest can expose
larger execution time difference with L1 on host with top-level
hypervisor than on supposedly 'non-infected' one),
- try to analyze binary BIOS code itself, though it can be VERY
problematic, I am even not talking for same possibility for BMC.
Sorry for posting such a naive and stupid stuff in the public ml, but
I am really out of clues of what`s happening there and why it is not
reproducible anywhere else.
1. https://xakep.ru/2011/12/26/58104/ (russian text, but can be read
through g-translate without lack of details)
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: E5-2620v2 - emulation stop error
2015-04-01 22:58 ` Andrey Korolyov
@ 2015-04-05 14:12 ` Andrey Korolyov
-1 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-05 14:12 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
A small update:
the behavior is caused by setting unrestricted_guest feature to N, I
had this feature disabled everywhere from approx. three years ago when
its enablement was one of suspects of the host crashes with
contemporary then KVM module. Also nVMX is likely to not work at all
and produce the same traces as in https://lkml.org/lkml/2014/7/17/12
without unrestricted_guest=1. I think this fact actually explaining
all real mode weirdness we`ve seen before and this should be probably
ended either by putting appropriate bits in a README or module
information or making strict dependency between
apicv/unrestricted_guest+nested/unrestricted_guest or fixing the issue
at its root if this is possible or appropriate solution. Thanks
everyone for keeping up with ideas through this thread!
^ permalink raw reply [flat|nested] 157+ messages in thread
* Re: [Qemu-devel] E5-2620v2 - emulation stop error
@ 2015-04-05 14:12 ` Andrey Korolyov
0 siblings, 0 replies; 157+ messages in thread
From: Andrey Korolyov @ 2015-04-05 14:12 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, Radim Krčmář,
qemu-devel, Dr. David Alan Gilbert, Bandan Das,
Kevin O'Connor, Gerd Hoffmann
A small update:
the behavior is caused by setting unrestricted_guest feature to N, I
had this feature disabled everywhere from approx. three years ago when
its enablement was one of suspects of the host crashes with
contemporary then KVM module. Also nVMX is likely to not work at all
and produce the same traces as in https://lkml.org/lkml/2014/7/17/12
without unrestricted_guest=1. I think this fact actually explaining
all real mode weirdness we`ve seen before and this should be probably
ended either by putting appropriate bits in a README or module
information or making strict dependency between
apicv/unrestricted_guest+nested/unrestricted_guest or fixing the issue
at its root if this is possible or appropriate solution. Thanks
everyone for keeping up with ideas through this thread!
^ permalink raw reply [flat|nested] 157+ messages in thread
end of thread, other threads:[~2015-04-05 14:12 UTC | newest]
Thread overview: 157+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-03-05 22:14 E5-2620v2 - emulation stop error Andrey Korolyov
2015-03-05 22:14 ` [Qemu-devel] " Andrey Korolyov
2015-03-05 23:44 ` Andrey Korolyov
2015-03-05 23:44 ` [Qemu-devel] " Andrey Korolyov
2015-03-06 16:57 ` Bandan Das
2015-03-06 16:57 ` Bandan Das
2015-03-07 0:00 ` Andrey Korolyov
2015-03-10 14:24 ` Andrey Korolyov
2015-03-10 16:57 ` Dr. David Alan Gilbert
2015-03-10 18:08 ` Andrey Korolyov
2015-03-10 18:16 ` Dr. David Alan Gilbert
2015-03-10 18:21 ` Andrey Korolyov
2015-03-10 19:30 ` Paolo Bonzini
2015-03-10 18:10 ` Paolo Bonzini
2015-03-10 18:21 ` Bandan Das
2015-03-10 18:21 ` Bandan Das
2015-03-10 19:25 ` Paolo Bonzini
2015-03-10 19:25 ` Paolo Bonzini
2015-03-10 19:37 ` Dr. David Alan Gilbert
2015-03-10 20:29 ` Dr. David Alan Gilbert
2015-03-10 20:29 ` Dr. David Alan Gilbert
2015-03-11 2:38 ` Bandan Das
2015-03-11 2:38 ` Bandan Das
2015-03-11 13:45 ` Dr. David Alan Gilbert
2015-03-11 13:45 ` Dr. David Alan Gilbert
2015-03-11 15:42 ` Kevin O'Connor
2015-03-11 15:42 ` Kevin O'Connor
2015-03-11 15:53 ` Dr. David Alan Gilbert
2015-03-11 15:53 ` Dr. David Alan Gilbert
2015-03-11 16:37 ` Kevin O'Connor
2015-03-11 16:37 ` [Qemu-devel] " Kevin O'Connor
2015-03-11 16:52 ` Dr. David Alan Gilbert
2015-03-11 16:52 ` Dr. David Alan Gilbert
2015-03-11 17:37 ` Kevin O'Connor
2015-03-11 17:37 ` Kevin O'Connor
2015-03-11 17:41 ` Paolo Bonzini
2015-03-11 17:41 ` Paolo Bonzini
2015-03-11 17:59 ` Dr. David Alan Gilbert
2015-03-11 17:59 ` Dr. David Alan Gilbert
2015-03-11 18:24 ` Bandan Das
2015-03-11 18:24 ` Bandan Das
2015-03-11 18:40 ` Kevin O'Connor
2015-03-11 18:40 ` Kevin O'Connor
2015-03-11 18:45 ` Kevin O'Connor
2015-03-11 18:45 ` Kevin O'Connor
2015-03-11 19:19 ` Kevin O'Connor
2015-03-11 19:19 ` Kevin O'Connor
2015-03-11 19:33 ` Dr. David Alan Gilbert
2015-03-11 19:33 ` Dr. David Alan Gilbert
2015-03-11 19:47 ` Bandan Das
2015-03-11 19:47 ` Bandan Das
2015-03-11 19:47 ` Andrey Korolyov
2015-03-11 19:47 ` Andrey Korolyov
2015-03-11 19:59 ` Dr. David Alan Gilbert
2015-03-11 19:59 ` Dr. David Alan Gilbert
2015-03-11 20:09 ` Andrey Korolyov
2015-03-11 20:09 ` Andrey Korolyov
2015-03-12 9:59 ` Dr. David Alan Gilbert
2015-03-12 9:59 ` Dr. David Alan Gilbert
2015-03-12 10:47 ` Andrey Korolyov
2015-03-12 10:47 ` Andrey Korolyov
2015-03-16 19:17 ` Andrey Korolyov
2015-03-16 19:17 ` Andrey Korolyov
2015-03-16 19:26 ` Dr. David Alan Gilbert
2015-03-16 19:26 ` Dr. David Alan Gilbert
2015-03-25 20:43 ` Andrey Korolyov
2015-03-25 20:43 ` [Qemu-devel] " Andrey Korolyov
2015-03-25 20:46 ` Andrey Korolyov
2015-03-25 20:46 ` [Qemu-devel] " Andrey Korolyov
2015-03-25 20:54 ` Kevin O'Connor
2015-03-25 20:54 ` Kevin O'Connor
2015-03-25 22:31 ` Andrey Korolyov
2015-03-25 22:31 ` Andrey Korolyov
2015-03-25 23:02 ` Kevin O'Connor
2015-03-25 23:02 ` Kevin O'Connor
2015-03-25 23:35 ` Andrey Korolyov
2015-03-25 23:35 ` Andrey Korolyov
2015-03-26 0:05 ` Kevin O'Connor
2015-03-26 0:05 ` Kevin O'Connor
2015-03-26 15:58 ` Radim Krčmář
2015-03-26 15:58 ` Radim Krčmář
2015-03-26 16:36 ` Kevin O'Connor
2015-03-26 16:36 ` [Qemu-devel] " Kevin O'Connor
2015-03-26 16:48 ` Andrey Korolyov
2015-03-26 16:48 ` Andrey Korolyov
2015-03-26 17:06 ` Kevin O'Connor
2015-03-26 17:06 ` Kevin O'Connor
2015-03-26 17:08 ` Andrey Korolyov
2015-03-26 17:08 ` Andrey Korolyov
2015-03-26 17:18 ` Kevin O'Connor
2015-03-26 17:18 ` Kevin O'Connor
2015-03-26 17:33 ` Andrey Korolyov
2015-03-26 17:33 ` Andrey Korolyov
2015-03-26 17:40 ` Radim Krčmář
2015-03-26 17:40 ` Radim Krčmář
2015-03-26 18:24 ` Andrey Korolyov
2015-03-26 18:24 ` Andrey Korolyov
2015-03-26 20:40 ` Radim Krčmář
2015-03-26 20:40 ` Radim Krčmář
2015-03-26 21:03 ` Bandan Das
2015-03-26 21:03 ` Bandan Das
2015-03-27 10:16 ` Andrey Korolyov
2015-03-27 10:16 ` Andrey Korolyov
2015-03-30 18:56 ` Radim Krčmář
2015-03-30 18:56 ` [Qemu-devel] " Radim Krčmář
2015-03-30 19:32 ` Andrey Korolyov
2015-03-30 19:32 ` Andrey Korolyov
2015-03-31 13:45 ` Radim Krčmář
2015-03-31 13:45 ` [Qemu-devel] " Radim Krčmář
2015-03-31 14:56 ` Andrey Korolyov
2015-03-31 14:56 ` Andrey Korolyov
2015-03-31 16:45 ` Radim Krčmář
2015-03-31 16:45 ` [Qemu-devel] " Radim Krčmář
2015-03-31 17:40 ` Andrey Korolyov
2015-03-31 17:40 ` Andrey Korolyov
2015-03-31 18:01 ` Bandan Das
2015-03-31 18:01 ` Bandan Das
2015-03-31 18:04 ` Bandan Das
2015-03-31 18:04 ` [Qemu-devel] " Bandan Das
2015-03-31 18:23 ` Andrey Korolyov
2015-03-31 18:23 ` Andrey Korolyov
2015-04-01 11:49 ` Radim Krčmář
2015-04-01 11:49 ` Radim Krčmář
2015-04-01 12:05 ` Paolo Bonzini
2015-04-01 12:05 ` Paolo Bonzini
2015-04-01 12:26 ` Andrey Korolyov
2015-04-01 12:26 ` Andrey Korolyov
2015-04-01 13:19 ` Paolo Bonzini
2015-04-01 13:19 ` Paolo Bonzini
2015-04-01 15:37 ` Andrey Korolyov
2015-04-01 15:37 ` Andrey Korolyov
2015-04-01 16:29 ` Andrey Korolyov
2015-04-01 16:29 ` Andrey Korolyov
2015-04-01 22:58 ` Andrey Korolyov
2015-04-01 22:58 ` Andrey Korolyov
2015-04-05 14:12 ` Andrey Korolyov
2015-04-05 14:12 ` [Qemu-devel] " Andrey Korolyov
2015-03-27 11:54 ` Andrey Korolyov
2015-03-27 11:54 ` Andrey Korolyov
2015-03-30 19:28 ` Radim Krčmář
2015-03-30 19:28 ` Radim Krčmář
2015-03-26 17:35 ` Radim Krčmář
2015-03-26 17:35 ` Radim Krčmář
2015-03-26 17:34 ` Radim Krčmář
2015-03-26 17:34 ` Radim Krčmář
2015-03-26 2:47 ` Bandan Das
2015-03-26 2:47 ` Bandan Das
2015-03-26 9:18 ` Andrey Korolyov
2015-03-26 9:18 ` Andrey Korolyov
2015-03-26 15:05 ` Andrey Korolyov
2015-03-26 15:05 ` Andrey Korolyov
2015-03-11 17:09 ` Bandan Das
2015-03-11 17:09 ` Bandan Das
2015-03-11 17:32 ` Kevin O'Connor
2015-03-11 17:32 ` Kevin O'Connor
2015-03-11 18:01 ` Bandan Das
2015-03-11 18:01 ` Bandan Das
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.