From: Thimo E <abc@digithi.de>
To: "Zhang, Yang Z" <yang.z.zhang@intel.com>
Cc: Keir Fraser <keir@xen.org>, Jan Beulich <JBeulich@suse.com>,
Andrew Cooper <andrew.cooper3@citrix.com>,
"Dong, Eddie" <eddie.dong@intel.com>,
Xen-develList <xen-devel@lists.xen.org>,
"Nakajima, Jun" <jun.nakajima@intel.com>,
"Zhang, Xiantao" <xiantao.zhang@intel.com>
Subject: Re: cpuidle and un-eoid interrupts at the local apic
Date: Mon, 12 Aug 2013 15:54:59 +0200 [thread overview]
Message-ID: <5208E933.1020609@digithi.de> (raw)
In-Reply-To: <A9667DDFB95DB7438FA9D7D576C3D87E0A8E11A4@SHSMSX104.ccr.corp.intel.com>
[-- Attachment #1.1: Type: text/plain, Size: 1869 bytes --]
Hello Yang,
and attached the next crash dump which occured today, only some minutes
after I've created the logfiles I've sent in the mail just before.
Perhaps together with the logfiles of the former mail it gives you a
better understand of what is going on.
I've disabled Interrupt remapping now.
> 4.....
> can you add some debug message in the guest EOI code path(like
_irq_guest_eoi())) to track the EOI?
@Andrew: Is it possible for you to integrate the requested changes from
Yang into your Xen debugging version ?
Best regards
Thimo
Am 12.08.2013 10:49, schrieb Zhang, Yang Z:
>
> Hi Thimo,
>
> From your previous experience and log, it shows:
>
> 1.The interrupt that triggers the issue is a MSI.
>
> 2.MSI are treated as edge-triggered interrupts nomally, except when
> there is no way to mask the device. In this case, your previous log
> indicates the device is unmaskable(What special device are you
> using?Modern PCI devcie should be maskable).
>
> 3.The IRQ 29 is belong to dom0, it seems it is not a HVM related issue.
>
> 4.The status of IRQ 29 is 10 which means the guest already issues the
> EOI because the bit IRQ_GUEST_EOI_PENDING is cleared, so there should
> be no pending EOI in the EOI stack. If possible, can you add some
> debug message in the guest EOI code path(like _irq_guest_eoi())) to
> track the EOI?
>
> 5.Both of the log show when the issue occured, most of the other
> interrupts which owned by dom0 were in IRQ_MOVE_PENDING status. Is it
> a coincidence? Or it happened only on the special condition like heavy
> of IRQ migration?Perhaps you can disable irq balance in dom0 and pin
> the IRQ manually.
>
|6.I guess the interrupt remapping is enabled in your machine. Can you
try to disable IR to see whether it still reproduceable?
>
> Also, please provide the whole Xen log.
>
> Best regards,
>
> Yang
>
[-- Attachment #1.2: Type: text/html, Size: 11804 bytes --]
[-- Attachment #2: crash20130812.txt --]
[-- Type: text/plain, Size: 11060 bytes --]
(XEN) **Pending EOI error^M
(XEN) irq 29, vector 0x26^M
(XEN) s[0] irq 29, vec 0x26, ready 0, ISR 00000001, TMR 00000000, IRR 00000000^M
(XEN) All LAPIC state:^M
(XEN) [vector] ISR TMR IRR^M
(XEN) [1f:00] 00000000 00000000 00000000^M
(XEN) [3f:20] 00000040 74fe50aa 00000000^M
(XEN) [5f:40] 00000000 ecbaeed2 00000000^M
(XEN) [7f:60] 00000000 be38f2d6 00000000^M
(XEN) [9f:80] 00000000 f0bc768a 00000000^M
(XEN) [bf:a0] 00000000 fadff8dc 00000000^M
(XEN) [df:c0] 00000000 df20fe40 00000000^M
(XEN) [ff:e0] 00000000 00000000 00000000^M
(XEN) Peoi stack trace records:^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 29, vec 0x26}^M
(XEN) Marked {sp 0, irq 29, vec 0x26} ready^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 29, vec 0x26}^M
(XEN) Marked {sp 0, irq 29, vec 0x26} ready^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 29, vec 0x26}^M
(XEN) Marked {sp 0, irq 29, vec 0x26} ready^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 29, vec 0x26}^M
(XEN) Marked {sp 0, irq 29, vec 0x26} ready^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 29, vec 0x26}^M
(XEN) Marked {sp 0, irq 29, vec 0x26} ready^M
(XEN) Pushed {sp 0, irq 29, vec 0x26}^M
(XEN) Poped entry {sp 1, irq 30, vec 0x45}^M
(XEN) Marked {sp 0, irq 30, vec 0x4d} ready^M
(XEN) Pushed {sp 0, irq 30, vec 0x45}^M
(XEN) Poped entry {sp 1, irq 30, vec 0x45}^M
(XEN) Marked {sp 0, irq 30, vec 0x45} ready^M
(XEN) Pushed {sp 0, irq 30, vec 0x45}^M
(XEN) Poped entry {sp 1, irq 30, vec 0x45}^M
(XEN) Marked {sp 0, irq 30, vec 0x45} ready^M
(XEN) Pushed {sp 0, irq 30, vec 0x45}^M
(XEN) Poped entry {sp 1, irq 30, vec 0x45}^M
(XEN) Marked {sp 0, irq 30, vec 0x45} ready^M
(XEN) Pushed {sp 0, irq 30, vec 0x45}^M
(XEN) Poped entry {sp 1, irq 30, vec 0x45}^M
(XEN) Marked {sp 0, irq 30, vec 0x45} ready^M
(XEN) Pushed {sp 0, irq 30, vec 0x45}^M
(XEN) Poped entry {sp 1, irq 30, vec 0xd3}^M
(XEN) Guest interrupt information:^M
(XEN) IRQ: 0 affinity:1 vec:f0 type=IO-APIC-edge status=00000000 mapped, unbound^M
(XEN) IRQ: 1 affinity:1 vec:38 type=IO-APIC-edge status=00000054 in-flight=0 domain-list=0: 1(----),^M
(XEN) IRQ: 2 affinity:f vec:00 type=XT-PIC status=00000000 mapped, unbound^M
(XEN) IRQ: 3 affinity:1 vec:40 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 4 affinity:1 vec:48 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 5 affinity:1 vec:50 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 5(----),^M
(XEN) IRQ: 6 affinity:1 vec:58 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 7 affinity:1 vec:60 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 8 affinity:1 vec:68 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 8(----),^M
(XEN) IRQ: 9 affinity:1 vec:70 type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 9(----),^M
(XEN) IRQ: 10 affinity:1 vec:78 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 11 affinity:1 vec:88 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 12 affinity:1 vec:90 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 13 affinity:1 vec:98 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 14 affinity:1 vec:a0 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 15 affinity:1 vec:a8 type=IO-APIC-edge status=00000002 mapped, unbound^M
(XEN) IRQ: 16 affinity:8 vec:dd type=IO-APIC-level status=00000010 in-flight=1 domain-list=0: 16(PS-M),^M
(XEN) IRQ: 18 affinity:8 vec:2e type=IO-APIC-level status=00000010 in-flight=0 domain-list=0: 18(-S--),^M
(XEN) IRQ: 19 affinity:1 vec:39 type=IO-APIC-level status=00000002 mapped, unbound^M
(XEN) IRQ: 20 affinity:1 vec:29 type=IO-APIC-level status=00000002 mapped, unbound^M
(XEN) IRQ: 22 affinity:8 vec:d5 type=IO-APIC-level status=00000010 in-flight=0 domain-list=0: 22(-S--),^M
(XEN) IRQ: 23 affinity:2 vec:9b type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 23(-S--),^M
(XEN) IRQ: 24 affinity:1 vec:28 type=DMA_MSI status=00000000 mapped, unbound^M
(XEN) IRQ: 25 affinity:1 vec:30 type=DMA_MSI status=00000000 mapped, unbound^M
(XEN) IRQ: 26 affinity:f vec:c0 type=PCI-MSI status=00000002 mapped, unbound^M
(XEN) IRQ: 27 affinity:f vec:c8 type=PCI-MSI status=00000002 mapped, unbound^M
(XEN) IRQ: 28 affinity:f vec:d0 type=PCI-MSI status=00000002 mapped, unbound^M
(XEN) IRQ: 29 affinity:1 vec:26 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:276(-S--),^M
(XEN) IRQ: 30 affinity:4 vec:cd type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(-S--),^M
(XEN) IRQ: 31 affinity:1 vec:24 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:274(-S--),^M
(XEN) IRQ: 32 affinity:2 vec:2c type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(-S--),^M
(XEN) IRQ: 33 affinity:8 vec:7b type=PCI-MSI status=00000050 in-flight=0 domain-list=0:272(-S--),^M
(XEN) IRQ: 34 affinity:1 vec:59 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:271(-S--),^M
(XEN) IO-APIC interrupt information:^M
(XEN) IRQ 0 Vec240:^M
(XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 1 Vec 56:^M
(XEN) Apic 0x00, Pin 1: vec=38 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 3 Vec 64:^M
(XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 4 Vec 72:^M
(XEN) Apic 0x00, Pin 4: vec=48 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 5 Vec 80:^M
(XEN) Apic 0x00, Pin 5: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 6 Vec 88:^M
(XEN) Apic 0x00, Pin 6: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 7 Vec 96:^M
(XEN) Apic 0x00, Pin 7: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 8 Vec104:^M
(XEN) Apic 0x00, Pin 8: vec=68 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 9 Vec112:^M
(XEN) Apic 0x00, Pin 9: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:0^M
(XEN) IRQ 10 Vec120:^M
(XEN) Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 11 Vec136:^M
(XEN) Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 12 Vec144:^M
(XEN) Apic 0x00, Pin 12: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 13 Vec152:^M
(XEN) Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 14 Vec160:^M
(XEN) Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 15 Vec168:^M
(XEN) Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0^M
(XEN) IRQ 16 Vec221:^M
(XEN) Apic 0x00, Pin 16: vec=dd delivery=LoPri dest=L status=1 polarity=1 irr=1 trig=L mask=0 dest_id:0^M
(XEN) IRQ 18 Vec 46:^M
(XEN) Apic 0x00, Pin 18: vec=2e delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0^M
(XEN) IRQ 19 Vec 57:^M
(XEN) Apic 0x00, Pin 19: vec=39 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0^M
(XEN) IRQ 20 Vec 41:^M
(XEN) Apic 0x00, Pin 20: vec=29 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0^M
(XEN) IRQ 22 Vec213:^M
(XEN) Apic 0x00, Pin 22: vec=d5 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0^M
(XEN) IRQ 23 Vec155:^M
(XEN) Apic 0x00, Pin 23: vec=9b delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0^M
(XEN) number of MP IRQ sources: 15.^M
(XEN) number of IO-APIC #2 registers: 24.^M
(XEN) testing the IO APIC.......................^M
(XEN) IO APIC #2......^M
(XEN) .... register #00: 02000000^M
(XEN) ....... : physical APIC id: 02^M
(XEN) ....... : Delivery Type: 0^M
(XEN) ....... : LTS : 0^M
(XEN) .... register #01: 00170020^M
(XEN) ....... : max redirection entries: 0017^M
(XEN) ....... : PRQ implemented: 0^M
(XEN) ....... : IO APIC version: 0020^M
(XEN) .... IRQ redirection table:^M
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: ^M
(XEN) 00 000 00 1 0 0 0 0 0 0 00^M
(XEN) 01 000 00 0 0 0 0 0 1 1 38^M
(XEN) 02 000 00 0 0 0 0 0 1 1 F0^M
(XEN) 03 000 00 0 0 0 0 0 1 1 40^M
(XEN) 04 000 00 0 0 0 0 0 1 1 48^M
(XEN) 05 000 00 0 0 0 0 0 1 1 50^M
(XEN) 06 000 00 0 0 0 0 0 1 1 58^M
(XEN) 07 000 00 0 0 0 0 0 1 1 60^M
(XEN) 08 000 00 0 0 0 0 0 1 1 68^M
(XEN) 09 000 00 0 1 0 0 0 1 1 70^M
(XEN) 0a 000 00 0 0 0 0 0 1 1 78^M
(XEN) 0b 000 00 0 0 0 0 0 1 1 88^M
(XEN) 0c 000 00 0 0 0 0 0 1 1 90^M
(XEN) 0d 000 00 0 0 0 0 0 1 1 98^M
(XEN) 0e 000 00 0 0 0 0 0 1 1 A0^M
(XEN) 0f 000 00 0 0 0 0 0 1 1 A8^M
(XEN) 10 000 00 0 1 1 1 1 1 1 DD^M
(XEN) 11 000 00 1 0 0 0 0 0 0 00^M
(XEN) 12 000 00 0 1 0 1 0 1 1 2E^M
(XEN) 13 000 00 1 1 0 1 0 1 1 39^M
(XEN) 14 000 00 1 1 0 1 0 1 1 29^M
(XEN) 15 07A 0A 1 0 0 0 0 0 2 B4^M
(XEN) 16 000 00 0 1 0 1 0 1 1 D5^M
(XEN) 17 000 00 0 1 0 1 0 1 1 9B^M
(XEN) Using vector-based indexing^M
(XEN) IRQ to pin mappings:^M
(XEN) IRQ240 -> 0:2^M
(XEN) IRQ56 -> 0:1^M
(XEN) IRQ64 -> 0:3^M
(XEN) IRQ72 -> 0:4^M
(XEN) IRQ80 -> 0:5^M
(XEN) IRQ88 -> 0:6^M
(XEN) IRQ96 -> 0:7^M
(XEN) IRQ104 -> 0:8^M
(XEN) IRQ112 -> 0:9^M
(XEN) IRQ120 -> 0:10^M
(XEN) IRQ136 -> 0:11^M
(XEN) IRQ144 -> 0:12^M
(XEN) IRQ152 -> 0:13^M
(XEN) IRQ160 -> 0:14^M
(XEN) IRQ168 -> 0:15^M
(XEN) IRQ221 -> 0:16^M
(XEN) IRQ46 -> 0:18^M
(XEN) IRQ57 -> 0:19^M
(XEN) IRQ41 -> 0:20^M
(XEN) IRQ213 -> 0:22^M
(XEN) IRQ155 -> 0:23^M
(XEN) .................................... done.^M
(XEN) ^M
(XEN) ****************************************^M
(XEN) Panic on CPU 0:^M
(XEN) CA-107844****************************************^M
(XEN) ^M
(XEN) Reboot in five seconds...^M
(XEN) Executing crash image^M
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-08-12 13:54 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-31 20:32 cpuidle and un-eoid interrupts at the local apic Andrew Cooper
2013-06-03 14:30 ` Jan Beulich
2013-07-31 8:30 ` Thimo E.
2013-07-31 9:47 ` Andrew Cooper
2013-08-02 22:50 ` Thimo E.
2013-08-02 23:32 ` Andrew Cooper
2013-08-05 12:45 ` Jan Beulich
2013-08-05 14:51 ` Andrew Cooper
2013-08-09 21:27 ` Thimo E.
2013-08-09 21:40 ` Andrew Cooper
2013-08-09 21:44 ` Andrew Cooper
2013-08-11 17:46 ` Thimo E.
2013-08-12 6:02 ` Zhang, Yang Z
2013-08-12 8:49 ` Zhang, Yang Z
2013-08-12 8:57 ` Jan Beulich
2013-08-12 11:52 ` Thimo E
2013-08-12 12:04 ` Andrew Cooper
2013-08-19 15:14 ` Thimo E.
2013-08-20 5:43 ` Thimo Eichstädt
2013-08-20 8:40 ` Jan Beulich
2013-08-20 8:50 ` Zhang, Yang Z
2013-08-23 7:22 ` Thimo Eichstädt
2013-08-23 7:30 ` Zhang, Yang Z
2013-08-27 1:03 ` Zhang, Yang Z
2013-09-04 18:32 ` Thimo E.
2013-09-04 18:55 ` Andrew Cooper
2013-09-04 19:56 ` Thimo E.
2013-09-04 20:54 ` Andrew Cooper
2013-09-05 1:45 ` Zhang, Yang Z
2013-09-05 7:20 ` Thimo E.
2013-09-05 1:15 ` Zhang, Yang Z
2013-09-17 2:09 ` Zhang, Yang Z
2013-09-17 7:39 ` Thimo E.
2013-09-17 7:43 ` Zhang, Yang Z
2013-09-17 21:04 ` Thimo E.
2013-09-18 1:18 ` Zhang, Xiantao
2013-09-18 17:24 ` Thimo E.
2013-09-18 12:06 ` Andrew Cooper
2013-08-12 13:54 ` Thimo E [this message]
2013-08-12 14:06 ` Andrew Cooper
2013-08-13 1:43 ` Zhang, Yang Z
2013-08-13 6:39 ` Thimo E.
2013-08-13 11:39 ` Wu, Feng
2013-08-13 12:46 ` Andrew Cooper
2013-08-12 9:10 ` Andrew Cooper
2013-08-12 5:50 ` Zhang, Yang Z
2013-08-12 8:20 ` Jan Beulich
2013-08-12 9:28 ` Andrew Cooper
2013-08-12 10:05 ` Jan Beulich
2013-08-12 10:27 ` Andrew Cooper
2013-08-14 2:53 ` Zhang, Yang Z
2013-08-14 7:51 ` Thimo E.
2013-08-14 9:52 ` Andrew Cooper
2013-09-07 13:27 ` Thimo E.
2013-09-07 17:02 ` Andrew Cooper
2013-09-07 23:37 ` Thimo E.
2013-09-08 9:53 ` Andrew Cooper
2013-09-08 10:24 ` Thimo E.
2013-09-09 13:16 ` Andrew Cooper
2013-09-09 14:48 ` Thimo Eichstädt
2013-09-09 15:12 ` Andrew Cooper
2013-09-09 7:59 ` Jan Beulich
2013-09-09 12:53 ` Andrew Cooper
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5208E933.1020609@digithi.de \
--to=abc@digithi.de \
--cc=JBeulich@suse.com \
--cc=andrew.cooper3@citrix.com \
--cc=eddie.dong@intel.com \
--cc=jun.nakajima@intel.com \
--cc=keir@xen.org \
--cc=xen-devel@lists.xen.org \
--cc=xiantao.zhang@intel.com \
--cc=yang.z.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.