From: Andrew Cooper <andrew.cooper3@citrix.com>
To: "Thimo E." <abc@digithi.de>
Cc: Keir Fraser <keir@xen.org>, Jan Beulich <JBeulich@suse.com>,
Eddie Dong <eddie.dong@intel.com>,
Xen-develList <xen-devel@lists.xen.org>,
Jun Nakajima <jun.nakajima@intel.com>,
Xiantao Zhang <xiantao.zhang@intel.com>
Subject: Re: cpuidle and un-eoid interrupts at the local apic
Date: Fri, 9 Aug 2013 22:44:40 +0100 [thread overview]
Message-ID: <520562C8.8080703@citrix.com> (raw)
In-Reply-To: <520561E1.8020809@citrix.com>
[-- Attachment #1.1: Type: text/plain, Size: 13052 bytes --]
On 09/08/13 22:40, Andrew Cooper wrote:
> On 09/08/13 22:27, Thimo E. wrote:
>> Next crash occured, debugging output included.
>>
>> One Remark: Over the last days (besides many linux PV guests) 1
>> Windows Guest (with PV drivers) was running, today I've started
>> another Windows guest and during 3 hours two crashed occured,
>> coincidence ?
>>
>> Best regards
>> Thimo
>
> So according to my debugging, we really have just pushed the same irq
> which we have subsequently seen again unexpectedly.
>
> This bug has only ever been seen on Haswell hardware, and appears
> linked to running HVM guests.
>
> So either there is an erroneous ACK the LAPIC which is clearing the
> ISR before the PEOI stack is expecting (which I
"can't"
Apologies for the confusion.
~Andrew
> obviously see, looking at the code), or something more funky is going
> on with the hardware.
>
> CC'ing in the Intel maintainers: Do you have any ideas? Could this
> be related to APICv?
>
> ~Andrew
>
>>
>> (XEN) **Pending EOI error
>> (XEN) irq 29, vector 0x24
>> (XEN) s[0] irq 29, vec 0x24, ready 0, ISR 00000001, TMR 00000000,
>> IRR 00000000
>> (XEN) All LAPIC state:
>> (XEN) [vector] ISR TMR IRR
>> (XEN) [1f:00] 00000000 00000000 00000000
>> (XEN) [3f:20] 00000010 76efa12e 00000000
>> (XEN) [5f:40] 00000000 e6f0f2fc 00000000
>> (XEN) [7f:60] 00000000 32d096ca 00000000
>> (XEN) [9f:80] 00000000 78fcf87a 00000000
>> (XEN) [bf:a0] 00000000 f9b9fe4e 00000000
>> (XEN) [df:c0] 00000000 ffdfe7ab 00000000
>> (XEN) [ff:e0] 00000000 00000000 00000000
>> (XEN) Peoi stack trace records:
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Guest interrupt information:
>> (XEN) IRQ: 0 affinity:1 vec:f0 type=IO-APIC-edge
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 1 affinity:1 vec:38 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 1(----),
>> (XEN) IRQ: 2 affinity:f vec:00 type=XT-PIC
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 3 affinity:1 vec:40 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 4 affinity:1 vec:48 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 5 affinity:1 vec:50 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 5(----),
>> (XEN) IRQ: 6 affinity:1 vec:58 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 7 affinity:1 vec:60 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 8 affinity:1 vec:68 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 8(----),
>> (XEN) IRQ: 9 affinity:1 vec:70 type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 9(----),
>> (XEN) IRQ: 10 affinity:1 vec:78 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 11 affinity:1 vec:88 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 12 affinity:1 vec:90 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 13 affinity:1 vec:98 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 14 affinity:1 vec:a0 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 15 affinity:1 vec:a8 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 16 affinity:1 vec:db type=IO-APIC-level
>> status=00000010 in-flight=0 domain-list=0: 16(----),
>> (XEN) IRQ: 18 affinity:1 vec:2c type=IO-APIC-level
>> status=00000010 in-flight=0 domain-list=0: 18(----),
>> (XEN) IRQ: 19 affinity:1 vec:51 type=IO-APIC-level
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 20 affinity:1 vec:29 type=IO-APIC-level
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 22 affinity:1 vec:bb type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 22(----),
>> (XEN) IRQ: 23 affinity:8 vec:c2 type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 23(----),
>> (XEN) IRQ: 24 affinity:1 vec:28 type=DMA_MSI
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 25 affinity:1 vec:30 type=DMA_MSI
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 26 affinity:f vec:c0 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 27 affinity:f vec:c8 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 28 affinity:f vec:d0 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 29 affinity:2 vec:24 type=PCI-MSI
>> status=00000010 in-flight=0 domain-list=0:276(----),
>> (XEN) IRQ: 30 affinity:4 vec:93 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:275(----),
>> (XEN) IRQ: 31 affinity:2 vec:4a type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:274(----),
>> (XEN) IRQ: 32 affinity:2 vec:73 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:273(----),
>> (XEN) IRQ: 33 affinity:1 vec:49 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:272(----),
>> (XEN) IRQ: 34 affinity:8 vec:5f type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:271(----),
>> (XEN) IO-APIC interrupt information:
>> (XEN) IRQ 0 Vec240:
>> (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 1 Vec 56:
>> (XEN) Apic 0x00, Pin 1: vec=38 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 3 Vec 64:
>> (XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 4 Vec 72:
>> (XEN) Apic 0x00, Pin 4: vec=48 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 5 Vec 80:
>> (XEN) Apic 0x00, Pin 5: vec=50 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 6 Vec 88:
>> (XEN) Apic 0x00, Pin 6: vec=58 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 7 Vec 96:
>> (XEN) Apic 0x00, Pin 7: vec=60 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 8 Vec104:
>> (XEN) Apic 0x00, Pin 8: vec=68 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 9 Vec112:
>> (XEN) Apic 0x00, Pin 9: vec=70 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 10 Vec120:
>> (XEN) Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 11 Vec136:
>> (XEN) Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 12 Vec144:
>> (XEN) Apic 0x00, Pin 12: vec=90 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 13 Vec152:
>> (XEN) Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 14 Vec160:
>> (XEN) Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 15 Vec168:
>> (XEN) Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 16 Vec219:
>> (XEN) Apic 0x00, Pin 16: vec=db delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 18 Vec 44:
>> (XEN) Apic 0x00, Pin 18: vec=2c delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 19 Vec 81:
>> (XEN) Apic 0x00, Pin 19: vec=51 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN) IRQ 20 Vec 41:
>> (XEN) Apic 0x00, Pin 20: vec=29 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN) IRQ 22 Vec187:
>> (XEN) Apic 0x00, Pin 22: vec=bb delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 23 Vec194:
>> (XEN) Apic 0x00, Pin 23: vec=c2 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) number of MP IRQ sources: 15.
>> (XEN) number of IO-APIC #2 registers: 24.
>> (XEN) testing the IO APIC.......................
>> (XEN) IO APIC #2......
>> (XEN) .... register #00: 02000000
>> (XEN) ....... : physical APIC id: 02
>> (XEN) ....... : Delivery Type: 0
>> (XEN) ....... : LTS : 0
>> (XEN) .... register #01: 00170020
>> (XEN) ....... : max redirection entries: 0017
>> (XEN) ....... : PRQ implemented: 0
>> (XEN) ....... : IO APIC version: 0020
>> (XEN) .... IRQ redirection table:
>> (XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
>> (XEN) 00 000 00 1 0 0 0 0 0 0 00
>> (XEN) 01 000 00 0 0 0 0 0 1 1 38
>> (XEN) 02 000 00 0 0 0 0 0 1 1 F0
>> (XEN) 03 000 00 0 0 0 0 0 1 1 40
>> (XEN) 04 000 00 0 0 0 0 0 1 1 48
>> (XEN) 05 000 00 0 0 0 0 0 1 1 50
>> (XEN) 06 000 00 0 0 0 0 0 1 1 58
>> (XEN) 07 000 00 0 0 0 0 0 1 1 60
>> (XEN) 08 000 00 0 0 0 0 0 1 1 68
>> (XEN) 09 000 00 0 1 0 0 0 1 1 70
>> (XEN) 0a 000 00 0 0 0 0 0 1 1 78
>> (XEN) 0b 000 00 0 0 0 0 0 1 1 88
>> (XEN) 0c 000 00 0 0 0 0 0 1 1 90
>> (XEN) 0d 000 00 0 0 0 0 0 1 1 98
>> (XEN) 0e 000 00 0 0 0 0 0 1 1 A0
>> (XEN) 0f 000 00 0 0 0 0 0 1 1 A8
>> (XEN) 10 000 00 0 1 0 1 0 1 1 DB
>> (XEN) 11 000 00 1 0 0 0 0 0 0 00
>> (XEN) 12 000 00 0 1 0 1 0 1 1 2C
>> (XEN) 13 000 00 1 1 0 1 0 1 1 51
>> (XEN) 14 000 00 1 1 0 1 0 1 1 29
>> (XEN) 15 07A 0A 1 0 0 0 0 0 2 B4
>> (XEN) 16 000 00 0 1 0 1 0 1 1 BB
>> (XEN) 17 000 00 0 1 0 1 0 1 1 C2
>> (XEN) Using vector-based indexing
>> (XEN) IRQ to pin mappings:
>> (XEN) IRQ240 -> 0:2
>> (XEN) IRQ56 -> 0:1
>> (XEN) IRQ64 -> 0:3
>> (XEN) IRQ72 -> 0:4
>> (XEN) IRQ80 -> 0:5
>> (XEN) IRQ88 -> 0:6
>> (XEN) IRQ96 -> 0:7
>> (XEN) IRQ104 -> 0:8
>> (XEN) IRQ112 -> 0:9
>> (XEN) IRQ120 -> 0:10
>> (XEN) IRQ136 -> 0:11
>> (XEN) IRQ144 -> 0:12
>> (XEN) IRQ152 -> 0:13
>> (XEN) IRQ160 -> 0:14
>> (XEN) IRQ168 -> 0:15
>> (XEN) IRQ219 -> 0:16
>> (XEN) IRQ44 -> 0:18
>> (XEN) IRQ81 -> 0:19
>> (XEN) IRQ41 -> 0:20
>> (XEN) IRQ187 -> 0:22
>> (XEN) IRQ194 -> 0:23
>> (XEN) .................................... done.
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 1:
>> (XEN) CA-107844****************************************
>> (XEN)
>> (XEN) Reboot in five seconds...
>> (XEN) Executing crash image
>>
>>
>> Am 05.08.2013 16:51, schrieb Andrew Cooper:
>>> All of these crashes are coming out of mwait_idle, so the cpu in
>>> question has literally just been in an lower power state.
>>>
>>> I am wondering whether there is some caching issue where an update to
>>> the Pending EOI stack pointer got "lost", but this seems like a little
>>> too specific to be reasonably explained as a caching issue.
>>>
>>> A new debugging patch is on its way (Sorry - it has been a very busy few
>>> days)
>>>
>>> ~Andrew
>>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
[-- Attachment #1.2: Type: text/html, Size: 22925 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-08-09 21:44 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-31 20:32 cpuidle and un-eoid interrupts at the local apic Andrew Cooper
2013-06-03 14:30 ` Jan Beulich
2013-07-31 8:30 ` Thimo E.
2013-07-31 9:47 ` Andrew Cooper
2013-08-02 22:50 ` Thimo E.
2013-08-02 23:32 ` Andrew Cooper
2013-08-05 12:45 ` Jan Beulich
2013-08-05 14:51 ` Andrew Cooper
2013-08-09 21:27 ` Thimo E.
2013-08-09 21:40 ` Andrew Cooper
2013-08-09 21:44 ` Andrew Cooper [this message]
2013-08-11 17:46 ` Thimo E.
2013-08-12 6:02 ` Zhang, Yang Z
2013-08-12 8:49 ` Zhang, Yang Z
2013-08-12 8:57 ` Jan Beulich
2013-08-12 11:52 ` Thimo E
2013-08-12 12:04 ` Andrew Cooper
2013-08-19 15:14 ` Thimo E.
2013-08-20 5:43 ` Thimo Eichstädt
2013-08-20 8:40 ` Jan Beulich
2013-08-20 8:50 ` Zhang, Yang Z
2013-08-23 7:22 ` Thimo Eichstädt
2013-08-23 7:30 ` Zhang, Yang Z
2013-08-27 1:03 ` Zhang, Yang Z
2013-09-04 18:32 ` Thimo E.
2013-09-04 18:55 ` Andrew Cooper
2013-09-04 19:56 ` Thimo E.
2013-09-04 20:54 ` Andrew Cooper
2013-09-05 1:45 ` Zhang, Yang Z
2013-09-05 7:20 ` Thimo E.
2013-09-05 1:15 ` Zhang, Yang Z
2013-09-17 2:09 ` Zhang, Yang Z
2013-09-17 7:39 ` Thimo E.
2013-09-17 7:43 ` Zhang, Yang Z
2013-09-17 21:04 ` Thimo E.
2013-09-18 1:18 ` Zhang, Xiantao
2013-09-18 17:24 ` Thimo E.
2013-09-18 12:06 ` Andrew Cooper
2013-08-12 13:54 ` Thimo E
2013-08-12 14:06 ` Andrew Cooper
2013-08-13 1:43 ` Zhang, Yang Z
2013-08-13 6:39 ` Thimo E.
2013-08-13 11:39 ` Wu, Feng
2013-08-13 12:46 ` Andrew Cooper
2013-08-12 9:10 ` Andrew Cooper
2013-08-12 5:50 ` Zhang, Yang Z
2013-08-12 8:20 ` Jan Beulich
2013-08-12 9:28 ` Andrew Cooper
2013-08-12 10:05 ` Jan Beulich
2013-08-12 10:27 ` Andrew Cooper
2013-08-14 2:53 ` Zhang, Yang Z
2013-08-14 7:51 ` Thimo E.
2013-08-14 9:52 ` Andrew Cooper
2013-09-07 13:27 ` Thimo E.
2013-09-07 17:02 ` Andrew Cooper
2013-09-07 23:37 ` Thimo E.
2013-09-08 9:53 ` Andrew Cooper
2013-09-08 10:24 ` Thimo E.
2013-09-09 13:16 ` Andrew Cooper
2013-09-09 14:48 ` Thimo Eichstädt
2013-09-09 15:12 ` Andrew Cooper
2013-09-09 7:59 ` Jan Beulich
2013-09-09 12:53 ` Andrew Cooper
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=520562C8.8080703@citrix.com \
--to=andrew.cooper3@citrix.com \
--cc=JBeulich@suse.com \
--cc=abc@digithi.de \
--cc=eddie.dong@intel.com \
--cc=jun.nakajima@intel.com \
--cc=keir@xen.org \
--cc=xen-devel@lists.xen.org \
--cc=xiantao.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.