* Interrupt injection with ISR set on Intel hardware
@ 2018-10-15 10:30 Roger Pau Monné
2018-10-15 12:06 ` Andrew Cooper
` (2 more replies)
0 siblings, 3 replies; 30+ messages in thread
From: Roger Pau Monné @ 2018-10-15 10:30 UTC (permalink / raw)
To: xen-devel; +Cc: Andrew Cooper, Kevin Tian, Wei Liu, Jun Nakajima, Jan Beulich
[-- Attachment #1: Type: text/plain, Size: 13813 bytes --]
Hello,
Wei recently discovered an issue when running a Linux PVH Dom0 on a
box with a Intel Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw
000906e9) CPU, we are not sure whether the issue is limited to a PVH
Dom0, or it just happens to be easier to trigger in this scenario.
The issue is caused by what seems to be an interrupt injection while
Xen is still servicing a previous interrupt (ie: the interrupt hasn't
been EOI'ed and ISR for the vector is set) with the same or lower
priority than the interrupt currently being serviced. This injection
always happen when returning from idle from a state ACPI_STATE_C3 or
lower.
Note that I haven't been able to reproduce this issue when using
mwait-idle=0 or max_cstate=2 on the Xen command line, but again
without knowing the underlying issue it's impossible to tell whether
it's relevant.
Andrew provided a debug patch which I've expanded to also log power
state transition, and is attached to this email.
Here is a trace of a crash, together with the debug info.
(XEN) *** Pending EOI error ***
(XEN) cpu #1, irq 30, vector 0x21, sp 1
(XEN) Peoi stack: sp 1
(XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
(XEN) Peoi stack trace records:
(XEN) [22619] POP {sp 1, irq 30, vec 0x21}
(XEN) [22620] POWER TYPE 4
(XEN) [22621] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22622] WAKE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22623] ACK_PRE PPR 0x000000f0
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) [22624] ACK_POST PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22625] POWER TYPE 5
(XEN) [22626] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22627] WAKE PPR 0x00000010
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22628] PUSH {sp 0, irq 30, vec 0x21}
(XEN) [22629] POWER TYPE 5
(XEN) [22630] IDLE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22631] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22632] POWER TYPE 5
(XEN) [22633] IDLE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22634] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000004
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22635] ACK_PRE PPR 0x000000f0
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000004
(XEN) [22636] ACK_POST PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22637] READY {sp 1, irq 30, vec 0x21}
(XEN) [22638] ACK_PRE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22639] ACK_POST PPR 0x00000010
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [22640] POP {sp 1, irq 30, vec 0x21}
(XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
(XEN) [22642] POWER TYPE 4
(XEN) [22643] IDLE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22644] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22645] POWER TYPE 3
(XEN) [22646] IDLE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22647] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22648] POWER TYPE 3
(XEN) [22649] IDLE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [22650] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) All LAPIC state:
(XEN) [vector] ISR TMR IRR
(XEN) [1f:00] 00000000 00000000 00000000
(XEN) [3f:20] 00000002 00000000 00000000
(XEN) [5f:40] 00000000 00000000 00000000
(XEN) [7f:60] 00000000 00000000 00000000
(XEN) [9f:80] 00000000 00000000 00000000
(XEN) [bf:a0] 00000000 00000000 00000000
(XEN) [df:c0] 00000000 00000000 00000000
(XEN) [ff:e0] 00000000 00000000 04000000
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
(XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 1
(XEN) RIP: e008:[<ffff82d08028737d>] do_IRQ+0x8df/0xacb
(XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor
(XEN) rax: ffff83086c67202c rbx: 0000000000000180 rcx: 0000000000000000
(XEN) rdx: ffff83086c68ffff rsi: 000000000000000a rdi: ffff83086c601e24
(XEN) rbp: ffff83086c68fd98 rsp: ffff83086c68fd38 r8: ffff83086c690000
(XEN) r9: 0000000000000030 r10: 0000000004000000 r11: 0000000000000007
(XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
(XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4: 00000000003526e0
(XEN) cr3: 0000000855ba7000 cr2: 0000556bfa53c040
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen code around <ffff82d08028737d> (do_IRQ+0x8df/0xacb):
(XEN) 8d 7e 24 e8 51 66 fb ff <0f> 0b 0f 0b 0f 0b 0f 0b b8 00 00 00 00 eb 4e 83
(XEN) Xen stack trace from rsp=ffff83086c68fd38:
(XEN) ffff82d000000000 ffff83086c601e24 0000000000000000 ffff83086c6724e0
(XEN) ffff82d08037b841 ffff82d08037b835 ffff82d08037b841 0000000000000000
(XEN) 0000000000000000 0000000000000000 ffff83086c68ffff 0000000000000000
(XEN) 00007cf793970237 ffff82d08037b8aa 00000003040712e5 0000000000000008
(XEN) ffff83086c671448 ffff83086c671390 ffff83086c68fec0 00000003040b3015
(XEN) ffff83086c672d08 ffff83086c6724e0 ffff83086c672d28 0000000000000180
(XEN) ffff83086c67202c 0000000000000000 ffff83086c68ffff 0000000000002ccf
(XEN) ffff83086c6713c0 0000002100000000 ffff82d0802e2403 000000000000e008
(XEN) 0000000000000202 ffff83086c68fe50 0000000000000000 ffff830088dd4000
(XEN) 00000020ffffffff 0000000000000000 ffff83086c68fee8 ffff82d08059bd00
(XEN) 0000000000000000 0000000000000000 000002d90000017f ffff82d0805a3c80
(XEN) 0000000000000001 ffff82d08059bd00 0000000000000001 0000000000000001
(XEN) ffff830856085000 ffff83086c68fef0 ffff82d08027755d ffff83086c6a5000
(XEN) ffff830088dd4000 ffff830088bfa000 ffff83086c6a5000 ffff83086c68fdb8
(XEN) 0000000000000000 0000000000000000 ffff880269a3bd00 ffff880269a3bd00
(XEN) 0000000000000005 0000000000000005 0000000000000000 0000000000000120
(XEN) 0000000000000000 000000002059d803 ffffffff816fe980 ffff88027335a7c0
(XEN) ffffffff82049af8 ffff88027335a7c0 00000000dade4600 0000beef0000beef
(XEN) ffffffff816fec52 000000bf0000beef 0000000000000246 ffffc90000d13e98
(XEN) 000000000000beef ffff83086c68beef 000000000000beef 000000000000beef
(XEN) Xen call trace:
(XEN) [<ffff82d08028737d>] do_IRQ+0x8df/0xacb
(XEN) [<ffff82d08037b8aa>] common_interrupt+0x10a/0x120
(XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
(XEN) [<ffff82d08027755d>] domain.c#idle_loop+0xb3/0xb5
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
Finally I'm also proving the surrounding context of the instructions
pointers in the trace above:
(XEN) [<ffff82d08028737d>] do_IRQ+0x8df/0xacb
xen/arch/x86/irq.c:1340:
1325 if ( action->ack_type == ACKTYPE_EOI )
1326 {
1327 sp = pending_eoi_sp(peoi);
1328 if ( !((sp == 0) || (peoi[sp-1].vector < vector)) )
1329 {
1330 printk("*** Pending EOI error ***\n");
1331 printk(" cpu #%u, irq %d, vector 0x%x, sp %d\n",
1332 smp_processor_id(), irq, vector, sp);
1333
1334 dump_peoi_stack(sp);
1335 dump_peoi_records();
1336 dump_lapic();
1337
1338 spin_unlock(&desc->lock);
1339
->1340 assert_failed("(sp == 0) || (peoi[sp-1].vector < vector)");
1341 }
1342
1343 ASSERT(sp < (NR_DYNAMIC_VECTORS-1));
1344 peoi[sp].irq = irq;
1345 peoi[sp].vector = vector;
1346 peoi[sp].ready = 0;
1347 pending_eoi_sp(peoi) = sp+1;
1348 cpumask_set_cpu(smp_processor_id(), action->cpu_eoi_map);
(XEN) [<ffff82d08037b8aa>] common_interrupt+0x10a/0x120
xen/arch/x86/x86_64/entry.S:58
47 /* Inject exception if pending. */
48 lea VCPU_trap_bounce(%rbx), %rdx
49 testb $TBF_EXCEPTION, TRAPBOUNCE_flags(%rdx)
50 jnz .Lprocess_trapbounce
51
52 cmpb $0, VCPU_mce_pending(%rbx)
53 jne process_mce
54 .Ltest_guest_nmi:
55 cmpb $0, VCPU_nmi_pending(%rbx)
56 jne process_nmi
57 test_guest_events:
-> 58 movq VCPU_vcpu_info(%rbx), %rax
59 movzwl VCPUINFO_upcall_pending(%rax), %eax
60 decl %eax
61 cmpl $0xfe, %eax
62 ja restore_all_guest
63 /*process_guest_events:*/
64 sti
65 leaq VCPU_trap_bounce(%rbx), %rdx
66 movq VCPU_event_addr(%rbx), %rax
67 movq %rax, TRAPBOUNCE_eip(%rdx)
68 movb $TBF_INTERRUPT, TRAPBOUNCE_flags(%rdx)
69 call create_bounce_frame
70 jmp test_all_events
(XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
xen/arch/x86/cpu/mwait-idle.c:802
788 if (cpu_is_haltable(cpu))
789 mwait_idle_with_hints(eax, MWAIT_ECX_INTERRUPT_BREAK);
790
791 after = cpuidle_get_tick();
792
793 cstate_restore_tsc();
794 trace_exit_reason(irq_traced);
795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
796 irq_traced[0], irq_traced[1], irq_traced[2], irq_traced[3]);
797
798 /* Now back in C0. */
799 update_idle_stats(power, cx, before, after);
800 local_irq_enable();
801
-> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
803 lapic_timer_on();
804
805 sched_tick_resume();
806 cpufreq_dbs_timer_resume();
(XEN) [<ffff82d08027755d>] domain.c#idle_loop+0xb3/0xb5
xen/arch/x86/domain.c:144
129 for ( ; ; )
130 {
131 if ( cpu_is_offline(cpu) )
132 play_dead();
133
134 /* Are we here for running vcpu context tasklets, or for idling? */
135 if ( unlikely(tasklet_work_to_do(cpu)) )
136 do_tasklet();
137 /*
138 * Test softirqs twice --- first to see if should even try scrubbing
139 * and then, after it is done, whether softirqs became pending
140 * while we were scrubbing.
141 */
142 else if ( !softirq_pending(cpu) && !scrub_free_pages() &&
143 !softirq_pending(cpu) )
-> 144 pm_idle();
145 do_softirq();
146 /*
147 * We MUST be last (or before pm_idle). Otherwise after we get the
148 * softirq we would execute pm_idle (and sleep) and not patch.
149 */
150 check_for_livepatch_work();
151 }
[-- Attachment #2: 0001-PEOI-debug.patch --]
[-- Type: text/plain, Size: 11230 bytes --]
>From 65f5703b9a7634bea440ebfea908120aaa6b35de Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Mon, 15 Oct 2018 12:26:42 +0200
Subject: [PATCH] PEOI debug
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
[power state tracing]
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
xen/arch/x86/acpi/cpu_idle.c | 4 +
xen/arch/x86/cpu/mwait-idle.c | 2 +
xen/arch/x86/irq.c | 182 +++++++++++++++++++++++++++++++++-
xen/include/asm-x86/apic.h | 14 +++
xen/include/asm-x86/irq.h | 44 ++++++++
5 files changed, 244 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/acpi/cpu_idle.c b/xen/arch/x86/acpi/cpu_idle.c
index 14b02789c5..08e3ffd996 100644
--- a/xen/arch/x86/acpi/cpu_idle.c
+++ b/xen/arch/x86/acpi/cpu_idle.c
@@ -420,6 +420,8 @@ void mwait_idle_with_hints(unsigned int eax, unsigned int ecx)
{
struct cpu_info *info = get_cpu_info();
+ peoi_debug_apic(PEOI_IDLE);
+
cpumask_set_cpu(cpu, &cpuidle_mwait_flags);
spec_ctrl_enter_idle(info);
@@ -427,6 +429,8 @@ void mwait_idle_with_hints(unsigned int eax, unsigned int ecx)
spec_ctrl_exit_idle(info);
cpumask_clear_cpu(cpu, &cpuidle_mwait_flags);
+
+ peoi_debug_apic(PEOI_WAKE);
}
if ( expires <= NOW() && expires > 0 )
diff --git a/xen/arch/x86/cpu/mwait-idle.c b/xen/arch/x86/cpu/mwait-idle.c
index 77fc3ddacc..8ccbdc97ae 100644
--- a/xen/arch/x86/cpu/mwait-idle.c
+++ b/xen/arch/x86/cpu/mwait-idle.c
@@ -781,6 +781,8 @@ static void mwait_idle(void)
before = cpuidle_get_tick();
TRACE_4D(TRC_PM_IDLE_ENTRY, cx->type, before, exp, pred);
+ peoi_debug_power(PEOI_POWER, cx->type);
+
update_last_cx_stat(power, cx, before);
if (cpu_is_haltable(cpu))
diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
index 35e7de594f..5f12f2497c 100644
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -1148,6 +1148,161 @@ static void irq_guest_eoi_timer_fn(void *data)
spin_unlock_irqrestore(&desc->lock, flags);
}
+DEFINE_PER_CPU(struct peoi_dbg_record, peoi_dbg[NR_PEOI_RECORDS]);
+DEFINE_PER_CPU(unsigned int, peoi_dbg_idx);
+
+void peoi_debug_stack(enum peoi_dbg_type action, unsigned int sp,
+ unsigned int irq, unsigned int vector)
+{
+ unsigned int *idx = &this_cpu(peoi_dbg_idx);
+ struct peoi_dbg_record *rec =
+ &this_cpu(peoi_dbg)[*idx & (NR_PEOI_RECORDS - 1)];
+ struct peoi_dbg_stack *stack = &rec->stack;
+
+ rec->seq = *idx;
+ rec->action = action;
+
+ stack->sp = sp;
+ stack->irq = irq;
+ stack->vector = vector;
+
+ (*idx)++;
+}
+
+void peoi_debug_apic(enum peoi_dbg_type action)
+{
+ unsigned int i, *idx = &this_cpu(peoi_dbg_idx);
+ struct peoi_dbg_record *rec =
+ &this_cpu(peoi_dbg)[*idx & (NR_PEOI_RECORDS - 1)];
+ struct peoi_dbg_apic *apic = &rec->apic;
+ uint32_t *irr = _p(apic->irr);
+ uint32_t *isr = _p(apic->isr);
+
+ rec->seq = *idx;
+ rec->action = action;
+
+ for ( i = 0; i < APIC_ISR_NR; i++ )
+ {
+ irr[i] = apic_read(APIC_IRR + i * 0x10);
+ isr[i] = apic_read(APIC_ISR + i * 0x10);
+ }
+
+ apic->ppr = apic_read(APIC_PROCPRI);
+
+ (*idx)++;
+}
+
+void peoi_debug_power(enum peoi_dbg_type action, unsigned int state)
+{
+ unsigned int *idx = &this_cpu(peoi_dbg_idx);
+ struct peoi_dbg_record *rec =
+ &this_cpu(peoi_dbg)[*idx & (NR_PEOI_RECORDS - 1)];
+ struct peoi_dbg_power *power = &rec->power;
+
+ rec->seq = *idx;
+ rec->action = action;
+
+ power->state = state;
+
+ (*idx)++;
+}
+
+static void dump_peoi_record(const struct peoi_dbg_record *r)
+{
+ const struct peoi_dbg_stack *s = &r->stack;
+ const struct peoi_dbg_apic *a = &r->apic;
+ const struct peoi_dbg_power *p = &r->power;
+
+ switch ( r->action )
+ {
+ case PEOI_PUSH:
+ printk(" [%5u] PUSH {sp %2d, irq %3d, vec 0x%02x}\n",
+ r->seq, s->sp, s->irq, s->vector);
+ break;
+
+ case PEOI_SETREADY:
+ printk(" [%5u] READY {sp %2d, irq %3d, vec 0x%02x}\n",
+ r->seq, s->sp, s->irq, s->vector);
+ break;
+
+ case PEOI_FLUSH:
+ printk(" [%5u] FLUSH %d -> 0\n", r->seq, s->sp);
+ break;
+
+ case PEOI_POP:
+ printk(" [%5u] POP {sp %2d, irq %3d, vec 0x%02x}\n",
+ r->seq, s->sp, s->irq, s->vector);
+ break;
+
+ case PEOI_IDLE:
+ printk(" [%5u] IDLE PPR 0x%08x\n", r->seq, a->ppr);
+
+ dump_apic_bitmaps:
+ printk(" IRR %*phN\n"
+ " ISR %*phN\n",
+ (int)sizeof(a->irr), a->irr, (int)sizeof(a->isr), a->isr);
+ break;
+
+ case PEOI_WAKE:
+ printk(" [%5u] WAKE PPR 0x%08x\n", r->seq, a->ppr);
+ goto dump_apic_bitmaps;
+
+ case PEOI_ACK_PRE:
+ printk(" [%5u] ACK_PRE PPR 0x%08x\n", r->seq, a->ppr);
+ goto dump_apic_bitmaps;
+
+ case PEOI_ACK_POST:
+ printk(" [%5u] ACK_POST PPR 0x%08x\n", r->seq, a->ppr);
+ goto dump_apic_bitmaps;
+
+ case PEOI_POWER:
+ printk(" [%5u] POWER TYPE %u\n", r->seq, p->state);
+ break;
+
+ default:
+ printk(" [%5u] ??? %d\n", r->seq, r->action);
+ break;
+ }
+}
+
+static void dump_peoi_records(void)
+{
+ unsigned int i, idx = this_cpu(peoi_dbg_idx);
+ struct peoi_dbg_record *rec = this_cpu(peoi_dbg);
+
+ printk("Peoi stack trace records:\n");
+ for ( i = 0; i < NR_PEOI_RECORDS; ++i )
+ dump_peoi_record(&rec[(idx + i) & (NR_PEOI_RECORDS - 1)]);
+}
+
+static void dump_lapic(void)
+{
+ unsigned int i;
+
+ printk("All LAPIC state:\n");
+ printk(" [vector] %8s %8s %8s\n", "ISR", "TMR", "IRR");
+ for ( i = 0; i < APIC_ISR_NR; ++i )
+ printk(" [%02x:%02x] %08"PRIx32" %08"PRIx32" %08"PRIx32"\n",
+ (i * 32) + 31, i * 32,
+ apic_read(APIC_ISR + i * 0x10),
+ apic_read(APIC_TMR + i * 0x10),
+ apic_read(APIC_IRR + i * 0x10));
+}
+
+static void dump_peoi_stack(int sp)
+{
+ struct pending_eoi *peoi = this_cpu(pending_eoi);
+ int i;
+
+ printk("Peoi stack: sp %d\n", sp);
+ for ( i = sp - 1; i >= 0; --i )
+ printk(" [%2d] irq %3d, vec 0x%02x, ready %u, ISR %u, TMR %u, IRR %u\n",
+ i, peoi[i].irq, peoi[i].vector, peoi[i].ready,
+ apic_isr_read(peoi[i].vector),
+ apic_tmr_read(peoi[i].vector),
+ apic_irr_read(peoi[i].vector));
+}
+
static void __do_IRQ_guest(int irq)
{
struct irq_desc *desc = irq_to_desc(irq);
@@ -1170,13 +1325,29 @@ static void __do_IRQ_guest(int irq)
if ( action->ack_type == ACKTYPE_EOI )
{
sp = pending_eoi_sp(peoi);
- ASSERT((sp == 0) || (peoi[sp-1].vector < vector));
+ if ( !((sp == 0) || (peoi[sp-1].vector < vector)) )
+ {
+ printk("*** Pending EOI error ***\n");
+ printk(" cpu #%u, irq %d, vector 0x%x, sp %d\n",
+ smp_processor_id(), irq, vector, sp);
+
+ dump_peoi_stack(sp);
+ dump_peoi_records();
+ dump_lapic();
+
+ spin_unlock(&desc->lock);
+
+ assert_failed("(sp == 0) || (peoi[sp-1].vector < vector)");
+ }
+
ASSERT(sp < (NR_DYNAMIC_VECTORS-1));
peoi[sp].irq = irq;
peoi[sp].vector = vector;
peoi[sp].ready = 0;
pending_eoi_sp(peoi) = sp+1;
cpumask_set_cpu(smp_processor_id(), action->cpu_eoi_map);
+
+ peoi_debug_stack(PEOI_PUSH, sp, irq, peoi[sp].vector);
}
for ( i = 0; i < action->nr_guests; i++ )
@@ -1383,6 +1554,8 @@ static void flush_ready_eoi(void)
if ( desc->handler->end )
desc->handler->end(desc, peoi[sp].vector);
spin_unlock(&desc->lock);
+
+ peoi_debug_stack(PEOI_POP, sp + 1, irq, peoi[sp].vector);
}
pending_eoi_sp(peoi) = sp+1;
@@ -1409,6 +1582,8 @@ static void __set_eoi_ready(struct irq_desc *desc)
} while ( peoi[--sp].irq != irq );
ASSERT(!peoi[sp].ready);
peoi[sp].ready = 1;
+
+ peoi_debug_stack(PEOI_SETREADY, sp + 1, irq, desc->arch.vector);
}
/* Mark specified IRQ as ready-for-EOI (if it really is) and attempt to EOI. */
@@ -2276,7 +2451,7 @@ void free_domain_pirqs(struct domain *d)
pcidevs_unlock();
}
-static void dump_irqs(unsigned char key)
+void dump_irqs(unsigned char key)
{
int i, irq, pirq;
struct irq_desc *desc;
@@ -2448,6 +2623,9 @@ void fixup_eoi(void)
/* Flush the interrupt EOI stack. */
peoi = this_cpu(pending_eoi);
+
+ peoi_debug_stack(PEOI_FLUSH, pending_eoi_sp(peoi), -1, -1);
+
for ( sp = 0; sp < pending_eoi_sp(peoi); sp++ )
peoi[sp].ready = 1;
flush_ready_eoi();
diff --git a/xen/include/asm-x86/apic.h b/xen/include/asm-x86/apic.h
index 9d7ec93042..77a60f8ac1 100644
--- a/xen/include/asm-x86/apic.h
+++ b/xen/include/asm-x86/apic.h
@@ -149,6 +149,16 @@ static __inline bool_t apic_isr_read(u8 vector)
(vector & 0x1f)) & 1;
}
+static inline bool apic_tmr_read(u8 vector)
+{
+ return apic_read(APIC_TMR + ((vector & ~0x1f) >> 1)) >> (vector & 0x1f);
+}
+
+static inline bool apic_irr_read(u8 vector)
+{
+ return apic_read(APIC_IRR + ((vector & ~0x1f) >> 1)) >> (vector & 0x1f);
+}
+
static __inline u32 get_apic_id(void) /* Get the physical APIC id */
{
u32 id = apic_read(APIC_ID);
@@ -161,8 +171,12 @@ int get_physical_broadcast(void);
static inline void ack_APIC_irq(void)
{
+ peoi_debug_apic(PEOI_ACK_PRE);
+
/* Docs say use 0 for future compatibility */
apic_write(APIC_EOI, 0);
+
+ peoi_debug_apic(PEOI_ACK_POST);
}
extern int get_maxlvt(void);
diff --git a/xen/include/asm-x86/irq.h b/xen/include/asm-x86/irq.h
index 4b39997f09..938e5ef6b5 100644
--- a/xen/include/asm-x86/irq.h
+++ b/xen/include/asm-x86/irq.h
@@ -27,6 +27,50 @@ typedef struct {
DECLARE_BITMAP(_bits,NR_VECTORS);
} vmask_t;
+struct peoi_dbg_record
+{
+ unsigned int seq;
+
+ enum peoi_dbg_type {
+ PEOI_PUSH,
+ PEOI_SETREADY,
+ PEOI_FLUSH,
+ PEOI_POP,
+
+ PEOI_IDLE,
+ PEOI_WAKE,
+ PEOI_ACK_PRE,
+ PEOI_ACK_POST,
+
+ PEOI_POWER,
+ } action;
+
+ union {
+ struct peoi_dbg_stack {
+ unsigned int sp, irq, vector;
+ } stack;
+
+ struct peoi_dbg_apic {
+ DECLARE_BITMAP(irr, NR_VECTORS);
+ DECLARE_BITMAP(isr, NR_VECTORS);
+ unsigned int ppr;
+ } apic;
+
+ struct peoi_dbg_power {
+ unsigned int state;
+ } power;
+ };
+};
+
+#define NR_PEOI_RECORDS 32
+DECLARE_PER_CPU(struct peoi_dbg_record, peoi_dbg[NR_PEOI_RECORDS]);
+DECLARE_PER_CPU(unsigned int, peoi_dbg_idx);
+
+void peoi_debug_stack(enum peoi_dbg_type action, unsigned int sp,
+ unsigned int irq, unsigned int vector);
+void peoi_debug_apic(enum peoi_dbg_type action);
+void peoi_debug_power(enum peoi_dbg_type action, unsigned int state);
+
struct irq_desc;
struct arch_irq_desc {
--
2.19.1
[-- Attachment #3: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-15 10:30 Interrupt injection with ISR set on Intel hardware Roger Pau Monné
@ 2018-10-15 12:06 ` Andrew Cooper
2018-10-22 7:33 ` Chao Gao
2018-10-25 12:51 ` Jan Beulich
2018-10-29 16:33 ` Jan Beulich
2018-12-12 10:36 ` Tian, Kevin
2 siblings, 2 replies; 30+ messages in thread
From: Andrew Cooper @ 2018-10-15 12:06 UTC (permalink / raw)
To: xen-devel, Jun Nakajima, Kevin Tian
Cc: Wei Liu, Jan Beulich, Roger Pau Monné
On 15/10/18 11:30, Roger Pau Monné wrote:
> Hello,
>
> Wei recently discovered an issue when running a Linux PVH Dom0 on a
> box with a Intel Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw
> 000906e9) CPU, we are not sure whether the issue is limited to a PVH
> Dom0, or it just happens to be easier to trigger in this scenario.
This issue has been seen very occasionally for years. My debugging
patch dates back to 2013, and it has been observed on Haswell systems as
well. There have also been a handful of reports on xen-devel over the
years.
Wei is the first person to get a reliable enough repro to debug. It is
not exclusive to PVH Dom0, but that appears to be the easiest way to
tickle the problem.
> The issue is caused by what seems to be an interrupt injection while
> Xen is still servicing a previous interrupt (ie: the interrupt hasn't
> been EOI'ed and ISR for the vector is set) with the same or lower
> priority than the interrupt currently being serviced. This injection
> always happen when returning from idle from a state ACPI_STATE_C3 or
> lower.
As a bit of background, for some guest irqs, we need to inject the
interrupt into the guest and wait for an explicit ack.
If the irq source doesn't have a mask bit which Xen can use, the only
option we have is to avoid repeated interruption is to leave the irq in
service at the LAPIC. The purpose of the Pending EOI stack is to manage
these as acks arrive back from guest context.
For reasons which aren't clear, guest-bound MSI vectors which don't have
a mask bit also use this PEOI stack mechanism. I think this is probably
a Xen bug, but it also relevant to the issue.
In Wei's case, the interrupt in question is an MSI non-maskable
interrupt from the USB controller.
> Note that I haven't been able to reproduce this issue when using
> mwait-idle=0 or max_cstate=2 on the Xen command line, but again
> without knowing the underlying issue it's impossible to tell whether
> it's relevant.
>
> Andrew provided a debug patch which I've expanded to also log power
> state transition, and is attached to this email.
>
> Here is a trace of a crash, together with the debug info.
>
> (XEN) *** Pending EOI error ***
> (XEN) cpu #1, irq 30, vector 0x21, sp 1
> (XEN) Peoi stack: sp 1
> (XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
> (XEN) Peoi stack trace records:
> (XEN) [22619] POP {sp 1, irq 30, vec 0x21}
> (XEN) [22620] POWER TYPE 4
> (XEN) [22621] IDLE PPR 0x00000010
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22622] WAKE PPR 0x00000010
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22623] ACK_PRE PPR 0x000000f0
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
> (XEN) [22624] ACK_POST PPR 0x00000010
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22625] POWER TYPE 5
> (XEN) [22626] IDLE PPR 0x00000010
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22627] WAKE PPR 0x00000010
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22628] PUSH {sp 0, irq 30, vec 0x21}
> (XEN) [22629] POWER TYPE 5
> (XEN) [22630] IDLE PPR 0x00000020
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22631] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22632] POWER TYPE 5
> (XEN) [22633] IDLE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22634] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000004
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22635] ACK_PRE PPR 0x000000f0
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000004
> (XEN) [22636] ACK_POST PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22637] READY {sp 1, irq 30, vec 0x21}
> (XEN) [22638] ACK_PRE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22639] ACK_POST PPR 0x00000010
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) [22640] POP {sp 1, irq 30, vec 0x21}
> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
> (XEN) [22642] POWER TYPE 4
> (XEN) [22643] IDLE PPR 0x00000020
> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22644] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22645] POWER TYPE 3
> (XEN) [22646] IDLE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22647] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22648] POWER TYPE 3
> (XEN) [22649] IDLE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) [22650] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
What has happened here is that, despite vector 0x21 being in service
(starting at the PUSH), we see it injected a second time. The ASSERT()
fires because we find this vector still on the pending EOI stack.
After that, we go idle a few times, but never haven't yet acked the
vector (i.e. whatever we're waiting for the guest to acknowledge hasn't
happened yet, and Xen has nothing else to do on this CPU).
From the debugging, we see that PPR/IRR/ISR appear to retain their state
across the mwait, and there is nothing in the manual which I can see
discussing the interaction of LAPIC state and C states.
However, from the behaviour seen here, we occasionally get woken from
mwait by an interrupt which already pending. I can only conclude that
there is some issue with priority calculations for edge triggered
interrupts when idle, which allows another one to slip in. The fact
that we can't reproduce this at all with max_cstate=2 is quite telling.
Any thoughts?
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-15 12:06 ` Andrew Cooper
@ 2018-10-22 7:33 ` Chao Gao
2018-10-22 7:57 ` Andrew Cooper
2018-10-29 11:22 ` Roger Pau Monné
2018-10-25 12:51 ` Jan Beulich
1 sibling, 2 replies; 30+ messages in thread
From: Chao Gao @ 2018-10-22 7:33 UTC (permalink / raw)
To: Andrew Cooper
Cc: Kevin Tian, Wei Liu, Jun Nakajima, Jan Beulich, xen-devel,
Roger Pau Monné
On Mon, Oct 15, 2018 at 01:06:12PM +0100, Andrew Cooper wrote:
>On 15/10/18 11:30, Roger Pau Monné wrote:
>> Hello,
>>
>> Wei recently discovered an issue when running a Linux PVH Dom0 on a
>> box with a Intel Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw
>> 000906e9) CPU, we are not sure whether the issue is limited to a PVH
>> Dom0, or it just happens to be easier to trigger in this scenario.
>
>This issue has been seen very occasionally for years. My debugging
>patch dates back to 2013, and it has been observed on Haswell systems as
>well. There have also been a handful of reports on xen-devel over the
>years.
>
>Wei is the first person to get a reliable enough repro to debug. It is
>not exclusive to PVH Dom0, but that appears to be the easiest way to
>tickle the problem.
>
>> The issue is caused by what seems to be an interrupt injection while
>> Xen is still servicing a previous interrupt (ie: the interrupt hasn't
>> been EOI'ed and ISR for the vector is set) with the same or lower
>> priority than the interrupt currently being serviced. This injection
>> always happen when returning from idle from a state ACPI_STATE_C3 or
>> lower.
>
>As a bit of background, for some guest irqs, we need to inject the
>interrupt into the guest and wait for an explicit ack.
>
>If the irq source doesn't have a mask bit which Xen can use, the only
>option we have is to avoid repeated interruption is to leave the irq in
>service at the LAPIC. The purpose of the Pending EOI stack is to manage
>these as acks arrive back from guest context.
>
>For reasons which aren't clear, guest-bound MSI vectors which don't have
>a mask bit also use this PEOI stack mechanism. I think this is probably
>a Xen bug, but it also relevant to the issue.
>
>In Wei's case, the interrupt in question is an MSI non-maskable
>interrupt from the USB controller.
>
>> Note that I haven't been able to reproduce this issue when using
>> mwait-idle=0 or max_cstate=2 on the Xen command line, but again
>> without knowing the underlying issue it's impossible to tell whether
>> it's relevant.
>>
>> Andrew provided a debug patch which I've expanded to also log power
>> state transition, and is attached to this email.
>>
>> Here is a trace of a crash, together with the debug info.
>>
>> (XEN) *** Pending EOI error ***
>> (XEN) cpu #1, irq 30, vector 0x21, sp 1
>> (XEN) Peoi stack: sp 1
>> (XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
>> (XEN) Peoi stack trace records:
>> (XEN) [22619] POP {sp 1, irq 30, vec 0x21}
>> (XEN) [22620] POWER TYPE 4
>> (XEN) [22621] IDLE PPR 0x00000010
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22622] WAKE PPR 0x00000010
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22623] ACK_PRE PPR 0x000000f0
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
>> (XEN) [22624] ACK_POST PPR 0x00000010
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22625] POWER TYPE 5
>> (XEN) [22626] IDLE PPR 0x00000010
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22627] WAKE PPR 0x00000010
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22628] PUSH {sp 0, irq 30, vec 0x21}
>> (XEN) [22629] POWER TYPE 5
>> (XEN) [22630] IDLE PPR 0x00000020
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22631] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22632] POWER TYPE 5
>> (XEN) [22633] IDLE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22634] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000004
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22635] ACK_PRE PPR 0x000000f0
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000004
>> (XEN) [22636] ACK_POST PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22637] READY {sp 1, irq 30, vec 0x21}
>> (XEN) [22638] ACK_PRE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22639] ACK_POST PPR 0x00000010
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) [22640] POP {sp 1, irq 30, vec 0x21}
>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>> (XEN) [22642] POWER TYPE 4
>> (XEN) [22643] IDLE PPR 0x00000020
>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22644] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22645] POWER TYPE 3
>> (XEN) [22646] IDLE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22647] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22648] POWER TYPE 3
>> (XEN) [22649] IDLE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) [22650] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>
>What has happened here is that, despite vector 0x21 being in service
>(starting at the PUSH), we see it injected a second time. The ASSERT()
>fires because we find this vector still on the pending EOI stack.
>
>After that, we go idle a few times, but never haven't yet acked the
>vector (i.e. whatever we're waiting for the guest to acknowledge hasn't
>happened yet, and Xen has nothing else to do on this CPU).
>
>From the debugging, we see that PPR/IRR/ISR appear to retain their state
>across the mwait, and there is nothing in the manual which I can see
>discussing the interaction of LAPIC state and C states.
>
>However, from the behaviour seen here, we occasionally get woken from
>mwait by an interrupt which already pending. I can only conclude that
>there is some issue with priority calculations for edge triggered
>interrupts when idle, which allows another one to slip in. The fact
Hi, Roger, Andrew and Wei,
Jan's patch
(https://lists.xen.org/archives/html/xen-devel/2018-10/msg01031.html)
fixs an issue in handling SVI. Currently, when dealing with EOI from guest, the
SVI was cleared. But the correct way is clearing the corresponding bit in VISR
and then setting SVI to the highest index of bit set in VISR (please refer to
SDM 29.1.4). If SVI is set to a value lower than the vector of the highest
priority interrupt that is in service, the PPR virtualization (29.1.3) might
set the VPPR to a lower value on VMEntry too. Thus an interrupt with same or
lower priority, which should be blocked by VPPR, slips in.
Could you apply Jan's patch and try to reproduce it again?
Thanks
Chao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-22 7:33 ` Chao Gao
@ 2018-10-22 7:57 ` Andrew Cooper
2018-10-29 11:22 ` Roger Pau Monné
1 sibling, 0 replies; 30+ messages in thread
From: Andrew Cooper @ 2018-10-22 7:57 UTC (permalink / raw)
To: Chao Gao
Cc: Kevin Tian, Wei Liu, Jun Nakajima, Jan Beulich, xen-devel,
Roger Pau Monné
On 22/10/2018 08:33, Chao Gao wrote:
> On Mon, Oct 15, 2018 at 01:06:12PM +0100, Andrew Cooper wrote:
>> On 15/10/18 11:30, Roger Pau Monné wrote:
>>> Hello,
>>>
>>> Wei recently discovered an issue when running a Linux PVH Dom0 on a
>>> box with a Intel Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw
>>> 000906e9) CPU, we are not sure whether the issue is limited to a PVH
>>> Dom0, or it just happens to be easier to trigger in this scenario.
>> This issue has been seen very occasionally for years. My debugging
>> patch dates back to 2013, and it has been observed on Haswell systems as
>> well. There have also been a handful of reports on xen-devel over the
>> years.
>>
>> Wei is the first person to get a reliable enough repro to debug. It is
>> not exclusive to PVH Dom0, but that appears to be the easiest way to
>> tickle the problem.
>>
>>> The issue is caused by what seems to be an interrupt injection while
>>> Xen is still servicing a previous interrupt (ie: the interrupt hasn't
>>> been EOI'ed and ISR for the vector is set) with the same or lower
>>> priority than the interrupt currently being serviced. This injection
>>> always happen when returning from idle from a state ACPI_STATE_C3 or
>>> lower.
>> As a bit of background, for some guest irqs, we need to inject the
>> interrupt into the guest and wait for an explicit ack.
>>
>> If the irq source doesn't have a mask bit which Xen can use, the only
>> option we have is to avoid repeated interruption is to leave the irq in
>> service at the LAPIC. The purpose of the Pending EOI stack is to manage
>> these as acks arrive back from guest context.
>>
>> For reasons which aren't clear, guest-bound MSI vectors which don't have
>> a mask bit also use this PEOI stack mechanism. I think this is probably
>> a Xen bug, but it also relevant to the issue.
>>
>> In Wei's case, the interrupt in question is an MSI non-maskable
>> interrupt from the USB controller.
>>
>>> Note that I haven't been able to reproduce this issue when using
>>> mwait-idle=0 or max_cstate=2 on the Xen command line, but again
>>> without knowing the underlying issue it's impossible to tell whether
>>> it's relevant.
>>>
>>> Andrew provided a debug patch which I've expanded to also log power
>>> state transition, and is attached to this email.
>>>
>>> Here is a trace of a crash, together with the debug info.
>>>
>>> (XEN) *** Pending EOI error ***
>>> (XEN) cpu #1, irq 30, vector 0x21, sp 1
>>> (XEN) Peoi stack: sp 1
>>> (XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
>>> (XEN) Peoi stack trace records:
>>> (XEN) [22619] POP {sp 1, irq 30, vec 0x21}
>>> (XEN) [22620] POWER TYPE 4
>>> (XEN) [22621] IDLE PPR 0x00000010
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22622] WAKE PPR 0x00000010
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22623] ACK_PRE PPR 0x000000f0
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
>>> (XEN) [22624] ACK_POST PPR 0x00000010
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22625] POWER TYPE 5
>>> (XEN) [22626] IDLE PPR 0x00000010
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22627] WAKE PPR 0x00000010
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22628] PUSH {sp 0, irq 30, vec 0x21}
>>> (XEN) [22629] POWER TYPE 5
>>> (XEN) [22630] IDLE PPR 0x00000020
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22631] WAKE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22632] POWER TYPE 5
>>> (XEN) [22633] IDLE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22634] WAKE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000004
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22635] ACK_PRE PPR 0x000000f0
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000004
>>> (XEN) [22636] ACK_POST PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22637] READY {sp 1, irq 30, vec 0x21}
>>> (XEN) [22638] ACK_PRE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22639] ACK_POST PPR 0x00000010
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) [22640] POP {sp 1, irq 30, vec 0x21}
>>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>>> (XEN) [22642] POWER TYPE 4
>>> (XEN) [22643] IDLE PPR 0x00000020
>>> (XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22644] WAKE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22645] POWER TYPE 3
>>> (XEN) [22646] IDLE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22647] WAKE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22648] POWER TYPE 3
>>> (XEN) [22649] IDLE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) [22650] WAKE PPR 0x00000020
>>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>> What has happened here is that, despite vector 0x21 being in service
>> (starting at the PUSH), we see it injected a second time. The ASSERT()
>> fires because we find this vector still on the pending EOI stack.
>>
>> After that, we go idle a few times, but never haven't yet acked the
>> vector (i.e. whatever we're waiting for the guest to acknowledge hasn't
>> happened yet, and Xen has nothing else to do on this CPU).
>>
> >From the debugging, we see that PPR/IRR/ISR appear to retain their state
>> across the mwait, and there is nothing in the manual which I can see
>> discussing the interaction of LAPIC state and C states.
>>
>> However, from the behaviour seen here, we occasionally get woken from
>> mwait by an interrupt which already pending. I can only conclude that
>> there is some issue with priority calculations for edge triggered
>> interrupts when idle, which allows another one to slip in. The fact
> Hi, Roger, Andrew and Wei,
>
> Jan's patch
> (https://lists.xen.org/archives/html/xen-devel/2018-10/msg01031.html)
> fixs an issue in handling SVI. Currently, when dealing with EOI from guest, the
> SVI was cleared. But the correct way is clearing the corresponding bit in VISR
> and then setting SVI to the highest index of bit set in VISR (please refer to
> SDM 29.1.4). If SVI is set to a value lower than the vector of the highest
> priority interrupt that is in service, the PPR virtualization (29.1.3) might
> set the VPPR to a lower value on VMEntry too. Thus an interrupt with same or
> lower priority, which should be blocked by VPPR, slips in.
>
> Could you apply Jan's patch and try to reproduce it again?
Hello,
I'm aware of Jan's patch, but pertains to Xen's emulation of the virtual
Local APIC for a guest.
This bug is with the real hardware APIC, as it pertains waking from
MWAIT. At the point that things go wrong, there is no VT-x involved at all.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-15 12:06 ` Andrew Cooper
2018-10-22 7:33 ` Chao Gao
@ 2018-10-25 12:51 ` Jan Beulich
2018-10-25 13:02 ` Andrew Cooper
1 sibling, 1 reply; 30+ messages in thread
From: Jan Beulich @ 2018-10-25 12:51 UTC (permalink / raw)
To: Andrew Cooper, Jun Nakajima, Kevin Tian
Cc: xen-devel, Wei Liu, Roger Pau Monne
>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
> From the debugging, we see that PPR/IRR/ISR appear to retain their state
> across the mwait, and there is nothing in the manual which I can see
> discussing the interaction of LAPIC state and C states.
Is it perhaps a bad idea to go idle with an un-acked interrupt?
Quite possibly that's not something an ordinary OS would do.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-25 12:51 ` Jan Beulich
@ 2018-10-25 13:02 ` Andrew Cooper
2018-10-25 13:57 ` Jan Beulich
0 siblings, 1 reply; 30+ messages in thread
From: Andrew Cooper @ 2018-10-25 13:02 UTC (permalink / raw)
To: Jan Beulich, Jun Nakajima, Kevin Tian; +Cc: xen-devel, Wei Liu, Roger Pau Monne
On 25/10/18 13:51, Jan Beulich wrote:
>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
>> From the debugging, we see that PPR/IRR/ISR appear to retain their state
>> across the mwait, and there is nothing in the manual which I can see
>> discussing the interaction of LAPIC state and C states.
> Is it perhaps a bad idea to go idle with an un-acked interrupt?
Most likely.
Then again, going idle with an un-acked line interrupt does appear to
work. It is only un-acked edge interrupts which appear to hit this issue.
Still - I'd prefer some guidance from the hardware folk as to what can
realistically be expected here.
> Quite possibly that's not something an ordinary OS would do.
This is definitely not something an ordinary OS would do during normal
operation. Xen suffers more than most other hypervisors because our
device drivers are in dom0.
That said, can't we just mask the line at the PIC/IO-APIC and forgo the
PEOI stack entirely? The more I think about how our interrupt handling
works, the more I think vastly over complicated.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-25 13:02 ` Andrew Cooper
@ 2018-10-25 13:57 ` Jan Beulich
2018-10-30 6:59 ` Tian, Kevin
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE2BAB0@SHSMSX101.ccr.corp.intel.com>
0 siblings, 2 replies; 30+ messages in thread
From: Jan Beulich @ 2018-10-25 13:57 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Kevin Tian, Wei Liu, Jun Nakajima, Roger Pau Monne
>>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
> On 25/10/18 13:51, Jan Beulich wrote:
>>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
>>> From the debugging, we see that PPR/IRR/ISR appear to retain their state
>>> across the mwait, and there is nothing in the manual which I can see
>>> discussing the interaction of LAPIC state and C states.
>> Is it perhaps a bad idea to go idle with an un-acked interrupt?
>
> Most likely.
>
> Then again, going idle with an un-acked line interrupt does appear to
> work. It is only un-acked edge interrupts which appear to hit this issue.
Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
ack mode, which should not be used on recent hardware because of
directed EOI presumably being available everywhere) where the ack
gets deferred until the .end hook (i.e. after the handler was run).
IOW AFAICT line interrupts would never be pending when we go idle.
> Still - I'd prefer some guidance from the hardware folk as to what can
> realistically be expected here.
Fully agree.
>> Quite possibly that's not something an ordinary OS would do.
>
> This is definitely not something an ordinary OS would do during normal
> operation. Xen suffers more than most other hypervisors because our
> device drivers are in dom0.
>
> That said, can't we just mask the line at the PIC/IO-APIC and forgo the
> PEOI stack entirely? The more I think about how our interrupt handling
> works, the more I think vastly over complicated.
MSIs obviously can't be masked at the PIC/IO-APIC; we do
mask (level) IO-APIC IRQs already (when directed EOI is not in
use). As to avoiding the PEOI stack - perhaps with directed EOI
we could, as the strict ordering of EOIs then is not a requirement.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-22 7:33 ` Chao Gao
2018-10-22 7:57 ` Andrew Cooper
@ 2018-10-29 11:22 ` Roger Pau Monné
1 sibling, 0 replies; 30+ messages in thread
From: Roger Pau Monné @ 2018-10-29 11:22 UTC (permalink / raw)
To: Chao Gao
Cc: Kevin Tian, Wei Liu, Jun Nakajima, Andrew Cooper, Jan Beulich, xen-devel
On Mon, Oct 22, 2018 at 03:33:07PM +0800, Chao Gao wrote:
> Hi, Roger, Andrew and Wei,
>
> Jan's patch
> (https://lists.xen.org/archives/html/xen-devel/2018-10/msg01031.html)
> fixs an issue in handling SVI. Currently, when dealing with EOI from guest, the
> SVI was cleared. But the correct way is clearing the corresponding bit in VISR
> and then setting SVI to the highest index of bit set in VISR (please refer to
> SDM 29.1.4). If SVI is set to a value lower than the vector of the highest
> priority interrupt that is in service, the PPR virtualization (29.1.3) might
> set the VPPR to a lower value on VMEntry too. Thus an interrupt with same or
> lower priority, which should be blocked by VPPR, slips in.
AFAICT the patch is for the emulated lapic, and the issue we are
seeing here is with the hardware lapic.
> Could you apply Jan's patch and try to reproduce it again?
I've applied Jan's patch and I'm still able to reproduce the issue.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-15 10:30 Interrupt injection with ISR set on Intel hardware Roger Pau Monné
2018-10-15 12:06 ` Andrew Cooper
@ 2018-10-29 16:33 ` Jan Beulich
2018-10-29 16:44 ` Andrew Cooper
2018-10-29 16:55 ` Roger Pau Monné
2018-12-12 10:36 ` Tian, Kevin
2 siblings, 2 replies; 30+ messages in thread
From: Jan Beulich @ 2018-10-29 16:33 UTC (permalink / raw)
To: Roger Pau Monne
Cc: Andrew Cooper, Kevin Tian, Wei Liu, Jun Nakajima, xen-devel
>>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
This is the last push or pop.
> (XEN) [22650] WAKE PPR 0x00000020
> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
For one I'm having trouble understanding why IRR here is different
from ...
> (XEN) All LAPIC state:
> (XEN) [vector] ISR TMR IRR
> (XEN) [1f:00] 00000000 00000000 00000000
> (XEN) [3f:20] 00000002 00000000 00000000
> (XEN) [5f:40] 00000000 00000000 00000000
> (XEN) [7f:60] 00000000 00000000 00000000
> (XEN) [9f:80] 00000000 00000000 00000000
> (XEN) [bf:a0] 00000000 00000000 00000000
> (XEN) [df:c0] 00000000 00000000 00000000
> (XEN) [ff:e0] 00000000 00000000 04000000
... IRR here.
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> (XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
> (XEN) CPU: 1
> (XEN) RIP: e008:[<ffff82d08028737d>] do_IRQ+0x8df/0xacb
> (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor
> (XEN) rax: ffff83086c67202c rbx: 0000000000000180 rcx: 0000000000000000
> (XEN) rdx: ffff83086c68ffff rsi: 000000000000000a rdi: ffff83086c601e24
> (XEN) rbp: ffff83086c68fd98 rsp: ffff83086c68fd38 r8: ffff83086c690000
> (XEN) r9: 0000000000000030 r10: 0000000004000000 r11: 0000000000000007
> (XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
> (XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4: 00000000003526e0
And then I'm having trouble guessing which register holds
"vector" here: r9 is the only one where I could sort of guess
it might be a vector, but then the assertion would not have
triggered. There's in particular no register with the low byte
being 0x21, nor is there any with it being 0xfa (to match the
bit that became set in IRR).
Could you please check or provide the disassembly?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-29 16:33 ` Jan Beulich
@ 2018-10-29 16:44 ` Andrew Cooper
2018-10-29 16:58 ` Jan Beulich
2018-10-29 16:55 ` Roger Pau Monné
1 sibling, 1 reply; 30+ messages in thread
From: Andrew Cooper @ 2018-10-29 16:44 UTC (permalink / raw)
To: Jan Beulich, Roger Pau Monne; +Cc: xen-devel, Kevin Tian, Wei Liu, Jun Nakajima
On 29/10/18 16:33, Jan Beulich wrote:
>>>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
> This is the last push or pop.
>
>> (XEN) [22650] WAKE PPR 0x00000020
>> (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
> For one I'm having trouble understanding why IRR here is different
> from ...
>
>> (XEN) All LAPIC state:
>> (XEN) [vector] ISR TMR IRR
>> (XEN) [1f:00] 00000000 00000000 00000000
>> (XEN) [3f:20] 00000002 00000000 00000000
>> (XEN) [5f:40] 00000000 00000000 00000000
>> (XEN) [7f:60] 00000000 00000000 00000000
>> (XEN) [9f:80] 00000000 00000000 00000000
>> (XEN) [bf:a0] 00000000 00000000 00000000
>> (XEN) [df:c0] 00000000 00000000 00000000
>> (XEN) [ff:e0] 00000000 00000000 04000000
> ... IRR here.
You shouldn't expect them to be the same.
The WAKE line is sampled before we enable interrupts, and the "All LAPIC
state" is after we enable interrupts and (erroneously) accept vector
0x21 a second time.
In the meantime, a TLB flush has become pending, but interrupts are
currently disabled so it has yet to be accepted. Remember that bits
accumulate in IRR entirely asynchronously.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-29 16:33 ` Jan Beulich
2018-10-29 16:44 ` Andrew Cooper
@ 2018-10-29 16:55 ` Roger Pau Monné
1 sibling, 0 replies; 30+ messages in thread
From: Roger Pau Monné @ 2018-10-29 16:55 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, Kevin Tian, Wei Liu, Jun Nakajima, xen-devel
On Mon, Oct 29, 2018 at 10:33:11AM -0600, Jan Beulich wrote:
> >>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
> > (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>
> This is the last push or pop.
>
> > (XEN) [22650] WAKE PPR 0x00000020
> > (XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
> > (XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
>
> For one I'm having trouble understanding why IRR here is different
> from ...
>
> > (XEN) All LAPIC state:
> > (XEN) [vector] ISR TMR IRR
> > (XEN) [1f:00] 00000000 00000000 00000000
> > (XEN) [3f:20] 00000002 00000000 00000000
> > (XEN) [5f:40] 00000000 00000000 00000000
> > (XEN) [7f:60] 00000000 00000000 00000000
> > (XEN) [9f:80] 00000000 00000000 00000000
> > (XEN) [bf:a0] 00000000 00000000 00000000
> > (XEN) [df:c0] 00000000 00000000 00000000
> > (XEN) [ff:e0] 00000000 00000000 04000000
>
> ... IRR here.
I expect (and this is all hypothesis ATM) that the IRR bit is clear
because the interrupt has been injected, so IRR has been cleared and
ISR has been set (except that ISR was already set).
> > (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> > (XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
> > (XEN) CPU: 1
> > (XEN) RIP: e008:[<ffff82d08028737d>] do_IRQ+0x8df/0xacb
> > (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor
> > (XEN) rax: ffff83086c67202c rbx: 0000000000000180 rcx: 0000000000000000
> > (XEN) rdx: ffff83086c68ffff rsi: 000000000000000a rdi: ffff83086c601e24
> > (XEN) rbp: ffff83086c68fd98 rsp: ffff83086c68fd38 r8: ffff83086c690000
> > (XEN) r9: 0000000000000030 r10: 0000000004000000 r11: 0000000000000007
> > (XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
> > (XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4: 00000000003526e0
>
> And then I'm having trouble guessing which register holds
> "vector" here: r9 is the only one where I could sort of guess
> it might be a vector, but then the assertion would not have
> triggered. There's in particular no register with the low byte
> being 0x21, nor is there any with it being 0xfa (to match the
> bit that became set in IRR).
>
> Could you please check or provide the disassembly?
The code chunk that you mention is the following (keep in mind this
has the debug patch attached):
1325 if ( action->ack_type == ACKTYPE_EOI )
1326 {
1327 sp = pending_eoi_sp(peoi);
1328 if ( !((sp == 0) || (peoi[sp-1].vector < vector)) )
1329 {
1330 printk("*** Pending EOI error ***\n");
1331 printk(" cpu #%u, irq %d, vector 0x%x, sp %d\n",
1332 smp_processor_id(), irq, vector, sp);
1333
1334 dump_peoi_stack(sp);
1335 dump_peoi_records();
1336 dump_lapic();
1337
1338 spin_unlock(&desc->lock);
1339
1340 assert_failed("(sp == 0) || (peoi[sp-1].vector < vector)");
So by the time the code reaches assert_failed the vector value has
long left the registers.
If you want a trace without the debug patch, here it is:
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1173
(XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 7
(XEN) RIP: e008:[<ffff82d08028737c>] do_IRQ+0x496/0x680
(XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor
(XEN) rax: ffff83085609e4c0 rbx: ffff83086c601e00 rcx: 0000000000000001
(XEN) rdx: 0000000000000021 rsi: 0000000000000021 rdi: 0000000000000001
(XEN) rbp: ffff830856097d98 rsp: ffff830856097d38 r8: 0000000000000021
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: ffff83086c6f1f10 r13: 0000000000000021 r14: ffff83086c601e00
(XEN) r15: 000000000000001e cr0: 0000000080050033 cr4: 00000000003526e0
(XEN) cr3: 0000000855ba7000 cr2: 000055c6e4a1c0f0
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
In this case there are several registers that contain 0x21.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-29 16:44 ` Andrew Cooper
@ 2018-10-29 16:58 ` Jan Beulich
2018-10-29 17:06 ` Andrew Cooper
0 siblings, 1 reply; 30+ messages in thread
From: Jan Beulich @ 2018-10-29 16:58 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Kevin Tian, Wei Liu, Jun Nakajima, Roger Pau Monne
>>> On 29.10.18 at 17:44, <andrew.cooper3@citrix.com> wrote:
> On 29/10/18 16:33, Jan Beulich wrote:
>>>>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
>>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>> This is the last push or pop.
>>
>>> (XEN) [22650] WAKE PPR 0x00000020
>>> (XEN) IRR
> 0000000002000000000000000000000000000000000000000000000000000000
>>> (XEN) ISR
> 0000000002000000000000000000000000000000000000000000000000000000
>> For one I'm having trouble understanding why IRR here is different
>> from ...
>>
>>> (XEN) All LAPIC state:
>>> (XEN) [vector] ISR TMR IRR
>>> (XEN) [1f:00] 00000000 00000000 00000000
>>> (XEN) [3f:20] 00000002 00000000 00000000
>>> (XEN) [5f:40] 00000000 00000000 00000000
>>> (XEN) [7f:60] 00000000 00000000 00000000
>>> (XEN) [9f:80] 00000000 00000000 00000000
>>> (XEN) [bf:a0] 00000000 00000000 00000000
>>> (XEN) [df:c0] 00000000 00000000 00000000
>>> (XEN) [ff:e0] 00000000 00000000 04000000
>> ... IRR here.
>
> You shouldn't expect them to be the same.
>
> The WAKE line is sampled before we enable interrupts, and the "All LAPIC
> state" is after we enable interrupts and (erroneously) accept vector
> 0x21 a second time.
Oh, right - I had overlooked that the debugging patch was actually
attached to Roger's mail.
> In the meantime, a TLB flush has become pending, but interrupts are
> currently disabled so it has yet to be accepted. Remember that bits
> accumulate in IRR entirely asynchronously.
Well, bits newly set are of course to be expected at any time. My
issue was just with bit 0x21 having got cleared. (FTR I think it's an
LAPIC timer interrupt which has become pending, not a TLB flush
IPI, but I don't think the difference matters here at all.)
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-29 16:58 ` Jan Beulich
@ 2018-10-29 17:06 ` Andrew Cooper
2018-10-30 7:32 ` Jan Beulich
0 siblings, 1 reply; 30+ messages in thread
From: Andrew Cooper @ 2018-10-29 17:06 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel, Kevin Tian, Wei Liu, Jun Nakajima, Roger Pau Monne
On 29/10/18 16:58, Jan Beulich wrote:
>>>> On 29.10.18 at 17:44, <andrew.cooper3@citrix.com> wrote:
>> On 29/10/18 16:33, Jan Beulich wrote:
>>>>>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
>>>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>>> This is the last push or pop.
>>>
>>>> (XEN) [22650] WAKE PPR 0x00000020
>>>> (XEN) IRR
>> 0000000002000000000000000000000000000000000000000000000000000000
>>>> (XEN) ISR
>> 0000000002000000000000000000000000000000000000000000000000000000
>>> For one I'm having trouble understanding why IRR here is different
>>> from ...
>>>
>>>> (XEN) All LAPIC state:
>>>> (XEN) [vector] ISR TMR IRR
>>>> (XEN) [1f:00] 00000000 00000000 00000000
>>>> (XEN) [3f:20] 00000002 00000000 00000000
>>>> (XEN) [5f:40] 00000000 00000000 00000000
>>>> (XEN) [7f:60] 00000000 00000000 00000000
>>>> (XEN) [9f:80] 00000000 00000000 00000000
>>>> (XEN) [bf:a0] 00000000 00000000 00000000
>>>> (XEN) [df:c0] 00000000 00000000 00000000
>>>> (XEN) [ff:e0] 00000000 00000000 04000000
>>> ... IRR here.
>> You shouldn't expect them to be the same.
>>
>> The WAKE line is sampled before we enable interrupts, and the "All LAPIC
>> state" is after we enable interrupts and (erroneously) accept vector
>> 0x21 a second time.
> Oh, right - I had overlooked that the debugging patch was actually
> attached to Roger's mail.
>
>> In the meantime, a TLB flush has become pending, but interrupts are
>> currently disabled so it has yet to be accepted. Remember that bits
>> accumulate in IRR entirely asynchronously.
> Well, bits newly set are of course to be expected at any time. My
> issue was just with bit 0x21 having got cleared.
What is unexpected about that?
Accepting a vector clears it out of IRR and sets it in ISR. A second
interrupt can then be queued in IRR while the first is being serviced.
> (FTR I think it's an
> LAPIC timer interrupt which has become pending, not a TLB flush
> IPI, but I don't think the difference matters here at all.)
Hmm possibly. I thought I traced it once before, but the vector layout
here is slightly dynamic based on boot conditions. I agree that the
difference is immaterial.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-25 13:57 ` Jan Beulich
@ 2018-10-30 6:59 ` Tian, Kevin
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE2BAB0@SHSMSX101.ccr.corp.intel.com>
1 sibling, 0 replies; 30+ messages in thread
From: Tian, Kevin @ 2018-10-30 6:59 UTC (permalink / raw)
To: Jan Beulich, Andrew Cooper
Cc: xen-devel, Wei Liu, Nakajima, Jun, Roger Pau Monne
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Thursday, October 25, 2018 9:58 PM
>
> >>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
> > On 25/10/18 13:51, Jan Beulich wrote:
> >>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
> >>> From the debugging, we see that PPR/IRR/ISR appear to retain their
> state
> >>> across the mwait, and there is nothing in the manual which I can see
> >>> discussing the interaction of LAPIC state and C states.
> >> Is it perhaps a bad idea to go idle with an un-acked interrupt?
> >
> > Most likely.
> >
> > Then again, going idle with an un-acked line interrupt does appear to
> > work. It is only un-acked edge interrupts which appear to hit this issue.
>
> Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
> ack mode, which should not be used on recent hardware because of
> directed EOI presumably being available everywhere) where the ack
> gets deferred until the .end hook (i.e. after the handler was run).
> IOW AFAICT line interrupts would never be pending when we go idle.
>
> > Still - I'd prefer some guidance from the hardware folk as to what can
> > realistically be expected here.
>
> Fully agree.
Just sent a mail internally to get clarification.
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-29 17:06 ` Andrew Cooper
@ 2018-10-30 7:32 ` Jan Beulich
0 siblings, 0 replies; 30+ messages in thread
From: Jan Beulich @ 2018-10-30 7:32 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Kevin Tian, Wei Liu, Jun Nakajima, Roger Pau Monne
>>> On 29.10.18 at 18:06, <andrew.cooper3@citrix.com> wrote:
> On 29/10/18 16:58, Jan Beulich wrote:
>>>>> On 29.10.18 at 17:44, <andrew.cooper3@citrix.com> wrote:
>>> On 29/10/18 16:33, Jan Beulich wrote:
>>>>>>> On 15.10.18 at 12:30, <roger.pau@citrix.com> wrote:
>>>>> (XEN) [22641] PUSH {sp 0, irq 30, vec 0x21}
>>>> This is the last push or pop.
>>>>
>>>>> (XEN) [22650] WAKE PPR 0x00000020
>>>>> (XEN) IRR
>>> 0000000002000000000000000000000000000000000000000000000000000000
>>>>> (XEN) ISR
>>> 0000000002000000000000000000000000000000000000000000000000000000
>>>> For one I'm having trouble understanding why IRR here is different
>>>> from ...
>>>>
>>>>> (XEN) All LAPIC state:
>>>>> (XEN) [vector] ISR TMR IRR
>>>>> (XEN) [1f:00] 00000000 00000000 00000000
>>>>> (XEN) [3f:20] 00000002 00000000 00000000
>>>>> (XEN) [5f:40] 00000000 00000000 00000000
>>>>> (XEN) [7f:60] 00000000 00000000 00000000
>>>>> (XEN) [9f:80] 00000000 00000000 00000000
>>>>> (XEN) [bf:a0] 00000000 00000000 00000000
>>>>> (XEN) [df:c0] 00000000 00000000 00000000
>>>>> (XEN) [ff:e0] 00000000 00000000 04000000
>>>> ... IRR here.
>>> You shouldn't expect them to be the same.
>>>
>>> The WAKE line is sampled before we enable interrupts, and the "All LAPIC
>>> state" is after we enable interrupts and (erroneously) accept vector
>>> 0x21 a second time.
>> Oh, right - I had overlooked that the debugging patch was actually
>> attached to Roger's mail.
>>
>>> In the meantime, a TLB flush has become pending, but interrupts are
>>> currently disabled so it has yet to be accepted. Remember that bits
>>> accumulate in IRR entirely asynchronously.
>> Well, bits newly set are of course to be expected at any time. My
>> issue was just with bit 0x21 having got cleared.
>
> What is unexpected about that?
>
> Accepting a vector clears it out of IRR and sets it in ISR. A second
> interrupt can then be queued in IRR while the first is being serviced.
"Unexpected" was meant in the sense of how the sequence of
logged lines looked, without having looked at how the debugging
patch actually produces them.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE2BAB0@SHSMSX101.ccr.corp.intel.com>
@ 2018-11-01 0:40 ` Tian, Kevin
2018-11-01 9:18 ` Andrew Cooper
0 siblings, 1 reply; 30+ messages in thread
From: Tian, Kevin @ 2018-11-01 0:40 UTC (permalink / raw)
To: 'Jan Beulich', Andrew Cooper
Cc: xen-devel, Wei Liu, Nakajima, Jun, Roger Pau Monne
> From: Tian, Kevin
> Sent: Tuesday, October 30, 2018 3:00 PM
>
> > From: Jan Beulich [mailto:JBeulich@suse.com]
> > Sent: Thursday, October 25, 2018 9:58 PM
> >
> > >>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
> > > On 25/10/18 13:51, Jan Beulich wrote:
> > >>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
> > >>> From the debugging, we see that PPR/IRR/ISR appear to retain their
> > state
> > >>> across the mwait, and there is nothing in the manual which I can see
> > >>> discussing the interaction of LAPIC state and C states.
> > >> Is it perhaps a bad idea to go idle with an un-acked interrupt?
> > >
> > > Most likely.
> > >
> > > Then again, going idle with an un-acked line interrupt does appear to
> > > work. It is only un-acked edge interrupts which appear to hit this issue.
> >
> > Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
> > ack mode, which should not be used on recent hardware because of
> > directed EOI presumably being available everywhere) where the ack
> > gets deferred until the .end hook (i.e. after the handler was run).
> > IOW AFAICT line interrupts would never be pending when we go idle.
> >
> > > Still - I'd prefer some guidance from the hardware folk as to what can
> > > realistically be expected here.
> >
> > Fully agree.
>
> Just sent a mail internally to get clarification.
>
One question.
in the first mail, Roger mentioned:
--
The issue is caused by what seems to be an interrupt injection while
Xen is still servicing a previous interrupt (ie: the interrupt hasn't
been EOI'ed and ISR for the vector is set) with **the same or lower
priority** than the interrupt currently being serviced.
--
from the debug log, it's actually the exact same vector (0x21) as
what is being in service in peoi stack.
Do you actually see the scenario "with the same or lower priority"?
If yes, can you post the debug log too?
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-11-01 0:40 ` Tian, Kevin
@ 2018-11-01 9:18 ` Andrew Cooper
2018-11-28 9:19 ` Roger Pau Monné
0 siblings, 1 reply; 30+ messages in thread
From: Andrew Cooper @ 2018-11-01 9:18 UTC (permalink / raw)
To: Tian, Kevin, 'Jan Beulich'
Cc: xen-devel, Wei Liu, Nakajima, Jun, Roger Pau Monne
On 01/11/2018 00:40, Tian, Kevin wrote:
>> From: Tian, Kevin
>> Sent: Tuesday, October 30, 2018 3:00 PM
>>
>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>> Sent: Thursday, October 25, 2018 9:58 PM
>>>
>>>>>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
>>>> On 25/10/18 13:51, Jan Beulich wrote:
>>>>>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
>>>>>> From the debugging, we see that PPR/IRR/ISR appear to retain their
>>> state
>>>>>> across the mwait, and there is nothing in the manual which I can see
>>>>>> discussing the interaction of LAPIC state and C states.
>>>>> Is it perhaps a bad idea to go idle with an un-acked interrupt?
>>>> Most likely.
>>>>
>>>> Then again, going idle with an un-acked line interrupt does appear to
>>>> work. It is only un-acked edge interrupts which appear to hit this issue.
>>> Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
>>> ack mode, which should not be used on recent hardware because of
>>> directed EOI presumably being available everywhere) where the ack
>>> gets deferred until the .end hook (i.e. after the handler was run).
>>> IOW AFAICT line interrupts would never be pending when we go idle.
>>>
>>>> Still - I'd prefer some guidance from the hardware folk as to what can
>>>> realistically be expected here.
>>> Fully agree.
>> Just sent a mail internally to get clarification.
>>
> One question.
>
> in the first mail, Roger mentioned:
> --
> The issue is caused by what seems to be an interrupt injection while
> Xen is still servicing a previous interrupt (ie: the interrupt hasn't
> been EOI'ed and ISR for the vector is set) with **the same or lower
> priority** than the interrupt currently being serviced.
> --
>
> from the debug log, it's actually the exact same vector (0x21) as
> what is being in service in peoi stack.
Yes - the problem is a repeat delivery of an interrupt which Xen thinks
it is already in the middle of processing.
>
> Do you actually see the scenario "with the same or lower priority"?
> If yes, can you post the debug log too?
I'm afraid that I don't understand the question. A repeat delivery of
vector 0x21 is the same priority.
I haven't seen an example of a lower priority interrupt being accepted,
but that might just be down to the repro scenario. Unfortunately, XTF
isn't usable on native hardware yet so I can't experiment cleanly in
this area.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-11-01 9:18 ` Andrew Cooper
@ 2018-11-28 9:19 ` Roger Pau Monné
2018-12-02 8:52 ` Tian, Kevin
0 siblings, 1 reply; 30+ messages in thread
From: Roger Pau Monné @ 2018-11-28 9:19 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Tian, Kevin, Wei Liu, Nakajima, Jun, 'Jan Beulich'
On Thu, Nov 01, 2018 at 09:18:14AM +0000, Andrew Cooper wrote:
> On 01/11/2018 00:40, Tian, Kevin wrote:
> >> From: Tian, Kevin
> >> Sent: Tuesday, October 30, 2018 3:00 PM
> >>
> >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>> Sent: Thursday, October 25, 2018 9:58 PM
> >>>
> >>>>>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
> >>>> On 25/10/18 13:51, Jan Beulich wrote:
> >>>>>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
> >>>>>> From the debugging, we see that PPR/IRR/ISR appear to retain their
> >>> state
> >>>>>> across the mwait, and there is nothing in the manual which I can see
> >>>>>> discussing the interaction of LAPIC state and C states.
> >>>>> Is it perhaps a bad idea to go idle with an un-acked interrupt?
> >>>> Most likely.
> >>>>
> >>>> Then again, going idle with an un-acked line interrupt does appear to
> >>>> work. It is only un-acked edge interrupts which appear to hit this issue.
> >>> Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
> >>> ack mode, which should not be used on recent hardware because of
> >>> directed EOI presumably being available everywhere) where the ack
> >>> gets deferred until the .end hook (i.e. after the handler was run).
> >>> IOW AFAICT line interrupts would never be pending when we go idle.
> >>>
> >>>> Still - I'd prefer some guidance from the hardware folk as to what can
> >>>> realistically be expected here.
> >>> Fully agree.
> >> Just sent a mail internally to get clarification.
> >>
> > One question.
> >
> > in the first mail, Roger mentioned:
> > --
> > The issue is caused by what seems to be an interrupt injection while
> > Xen is still servicing a previous interrupt (ie: the interrupt hasn't
> > been EOI'ed and ISR for the vector is set) with **the same or lower
> > priority** than the interrupt currently being serviced.
> > --
> >
> > from the debug log, it's actually the exact same vector (0x21) as
> > what is being in service in peoi stack.
>
> Yes - the problem is a repeat delivery of an interrupt which Xen thinks
> it is already in the middle of processing.
>
> >
> > Do you actually see the scenario "with the same or lower priority"?
> > If yes, can you post the debug log too?
>
> I'm afraid that I don't understand the question. A repeat delivery of
> vector 0x21 is the same priority.
>
> I haven't seen an example of a lower priority interrupt being accepted,
> but that might just be down to the repro scenario. Unfortunately, XTF
> isn't usable on native hardware yet so I can't experiment cleanly in
> this area.
Hello,
Is there any news on this?
I would like to have a fix before the 4.12 release if possible.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-11-28 9:19 ` Roger Pau Monné
@ 2018-12-02 8:52 ` Tian, Kevin
0 siblings, 0 replies; 30+ messages in thread
From: Tian, Kevin @ 2018-12-02 8:52 UTC (permalink / raw)
To: Roger Pau Monné, Andrew Cooper
Cc: xen-devel, Lai, Paul C, Wei Liu, Nakajima, Jun, 'Jan Beulich'
> From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> Sent: Wednesday, November 28, 2018 5:20 PM
>
> On Thu, Nov 01, 2018 at 09:18:14AM +0000, Andrew Cooper wrote:
> > On 01/11/2018 00:40, Tian, Kevin wrote:
> > >> From: Tian, Kevin
> > >> Sent: Tuesday, October 30, 2018 3:00 PM
> > >>
> > >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> > >>> Sent: Thursday, October 25, 2018 9:58 PM
> > >>>
> > >>>>>> On 25.10.18 at 15:02, <andrew.cooper3@citrix.com> wrote:
> > >>>> On 25/10/18 13:51, Jan Beulich wrote:
> > >>>>>>>> On 15.10.18 at 14:06, <andrew.cooper3@citrix.com> wrote:
> > >>>>>> From the debugging, we see that PPR/IRR/ISR appear to retain
> their
> > >>> state
> > >>>>>> across the mwait, and there is nothing in the manual which I can
> see
> > >>>>>> discussing the interaction of LAPIC state and C states.
> > >>>>> Is it perhaps a bad idea to go idle with an un-acked interrupt?
> > >>>> Most likely.
> > >>>>
> > >>>> Then again, going idle with an un-acked line interrupt does appear
> to
> > >>>> work. It is only un-acked edge interrupts which appear to hit this
> issue.
> > >>> Well, non-maskable MSI are the only ones (outside of "new" IO-APIC
> > >>> ack mode, which should not be used on recent hardware because of
> > >>> directed EOI presumably being available everywhere) where the ack
> > >>> gets deferred until the .end hook (i.e. after the handler was run).
> > >>> IOW AFAICT line interrupts would never be pending when we go idle.
> > >>>
> > >>>> Still - I'd prefer some guidance from the hardware folk as to what
> can
> > >>>> realistically be expected here.
> > >>> Fully agree.
> > >> Just sent a mail internally to get clarification.
> > >>
> > > One question.
> > >
> > > in the first mail, Roger mentioned:
> > > --
> > > The issue is caused by what seems to be an interrupt injection while
> > > Xen is still servicing a previous interrupt (ie: the interrupt hasn't
> > > been EOI'ed and ISR for the vector is set) with **the same or lower
> > > priority** than the interrupt currently being serviced.
> > > --
> > >
> > > from the debug log, it's actually the exact same vector (0x21) as
> > > what is being in service in peoi stack.
> >
> > Yes - the problem is a repeat delivery of an interrupt which Xen thinks
> > it is already in the middle of processing.
> >
> > >
> > > Do you actually see the scenario "with the same or lower priority"?
> > > If yes, can you post the debug log too?
> >
> > I'm afraid that I don't understand the question. A repeat delivery of
> > vector 0x21 is the same priority.
> >
> > I haven't seen an example of a lower priority interrupt being accepted,
> > but that might just be down to the repro scenario. Unfortunately, XTF
> > isn't usable on native hardware yet so I can't experiment cleanly in
> > this area.
>
> Hello,
>
> Is there any news on this?
>
> I would like to have a fix before the 4.12 release if possible.
>
sorry still waiting for a formal clarification from internal team...
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-10-15 10:30 Interrupt injection with ISR set on Intel hardware Roger Pau Monné
2018-10-15 12:06 ` Andrew Cooper
2018-10-29 16:33 ` Jan Beulich
@ 2018-12-12 10:36 ` Tian, Kevin
2018-12-12 11:24 ` Roger Pau Monné
2 siblings, 1 reply; 30+ messages in thread
From: Tian, Kevin @ 2018-12-12 10:36 UTC (permalink / raw)
To: Roger Pau Monné, xen-devel
Cc: Andrew Cooper, Wei Liu, Nakajima, Jun, Jan Beulich
> From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> Sent: Monday, October 15, 2018 6:30 PM
> (XEN) [22642] POWER TYPE 4
> (XEN) [22643] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [22644] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
is it a fixed pattern after how many rounds of Cstate enter/exit with
pending IRR(0x21) then you see assertion happened (in this example
it happens at 3rd time)?
> (XEN) [22645] POWER TYPE 3
> (XEN) [22646] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [22647] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [22648] POWER TYPE 3
> (XEN) [22649] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [22650] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) All LAPIC state:
> (XEN) [vector] ISR TMR IRR
> (XEN) [1f:00] 00000000 00000000 00000000
> (XEN) [3f:20] 00000002 00000000 00000000
> (XEN) [5f:40] 00000000 00000000 00000000
> (XEN) [7f:60] 00000000 00000000 00000000
> (XEN) [9f:80] 00000000 00000000 00000000
> (XEN) [bf:a0] 00000000 00000000 00000000
> (XEN) [df:c0] 00000000 00000000 00000000
> (XEN) [ff:e0] 00000000 00000000 04000000
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> (XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
> (XEN) CPU: 1
> (XEN) RIP: e008:[<ffff82d08028737d>] do_IRQ+0x8df/0xacb
> (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor
> (XEN) rax: ffff83086c67202c rbx: 0000000000000180 rcx:
> 0000000000000000
> (XEN) rdx: ffff83086c68ffff rsi: 000000000000000a rdi: ffff83086c601e24
> (XEN) rbp: ffff83086c68fd98 rsp: ffff83086c68fd38 r8: ffff83086c690000
> (XEN) r9: 0000000000000030 r10: 0000000004000000 r11:
> 0000000000000007
> (XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
> (XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4:
> 00000000003526e0
> (XEN) cr3: 0000000855ba7000 cr2: 0000556bfa53c040
> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss:
> 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> (XEN) Xen code around <ffff82d08028737d> (do_IRQ+0x8df/0xacb):
> (XEN) 8d 7e 24 e8 51 66 fb ff <0f> 0b 0f 0b 0f 0b 0f 0b b8 00 00 00 00 eb 4e
> 83
> (XEN) Xen stack trace from rsp=ffff83086c68fd38:
> (XEN) ffff82d000000000 ffff83086c601e24 0000000000000000
> ffff83086c6724e0
> (XEN) ffff82d08037b841 ffff82d08037b835 ffff82d08037b841
> 0000000000000000
> (XEN) 0000000000000000 0000000000000000 ffff83086c68ffff
> 0000000000000000
> (XEN) 00007cf793970237 ffff82d08037b8aa 00000003040712e5
> 0000000000000008
> (XEN) ffff83086c671448 ffff83086c671390 ffff83086c68fec0
> 00000003040b3015
> (XEN) ffff83086c672d08 ffff83086c6724e0 ffff83086c672d28
> 0000000000000180
> (XEN) ffff83086c67202c 0000000000000000 ffff83086c68ffff
> 0000000000002ccf
> (XEN) ffff83086c6713c0 0000002100000000 ffff82d0802e2403
> 000000000000e008
> (XEN) 0000000000000202 ffff83086c68fe50 0000000000000000
> ffff830088dd4000
> (XEN) 00000020ffffffff 0000000000000000 ffff83086c68fee8
> ffff82d08059bd00
> (XEN) 0000000000000000 0000000000000000 000002d90000017f
> ffff82d0805a3c80
> (XEN) 0000000000000001 ffff82d08059bd00 0000000000000001
> 0000000000000001
> (XEN) ffff830856085000 ffff83086c68fef0 ffff82d08027755d
> ffff83086c6a5000
> (XEN) ffff830088dd4000 ffff830088bfa000 ffff83086c6a5000
> ffff83086c68fdb8
> (XEN) 0000000000000000 0000000000000000 ffff880269a3bd00
> ffff880269a3bd00
> (XEN) 0000000000000005 0000000000000005 0000000000000000
> 0000000000000120
> (XEN) 0000000000000000 000000002059d803 ffffffff816fe980
> ffff88027335a7c0
> (XEN) ffffffff82049af8 ffff88027335a7c0 00000000dade4600
> 0000beef0000beef
> (XEN) ffffffff816fec52 000000bf0000beef 0000000000000246
> ffffc90000d13e98
> (XEN) 000000000000beef ffff83086c68beef 000000000000beef
> 000000000000beef
> (XEN) Xen call trace:
> (XEN) [<ffff82d08028737d>] do_IRQ+0x8df/0xacb
> (XEN) [<ffff82d08037b8aa>] common_interrupt+0x10a/0x120
> (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> (XEN) [<ffff82d08027755d>] domain.c#idle_loop+0xb3/0xb5
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> Finally I'm also proving the surrounding context of the instructions
> pointers in the trace above:
>
> (XEN) [<ffff82d08028737d>] do_IRQ+0x8df/0xacb
> xen/arch/x86/irq.c:1340:
>
> 1325 if ( action->ack_type == ACKTYPE_EOI )
> 1326 {
> 1327 sp = pending_eoi_sp(peoi);
> 1328 if ( !((sp == 0) || (peoi[sp-1].vector < vector)) )
> 1329 {
> 1330 printk("*** Pending EOI error ***\n");
> 1331 printk(" cpu #%u, irq %d, vector 0x%x, sp %d\n",
> 1332 smp_processor_id(), irq, vector, sp);
> 1333
> 1334 dump_peoi_stack(sp);
> 1335 dump_peoi_records();
> 1336 dump_lapic();
> 1337
> 1338 spin_unlock(&desc->lock);
> 1339
> ->1340 assert_failed("(sp == 0) || (peoi[sp-1].vector < vector)");
> 1341 }
> 1342
> 1343 ASSERT(sp < (NR_DYNAMIC_VECTORS-1));
> 1344 peoi[sp].irq = irq;
> 1345 peoi[sp].vector = vector;
> 1346 peoi[sp].ready = 0;
> 1347 pending_eoi_sp(peoi) = sp+1;
> 1348 cpumask_set_cpu(smp_processor_id(), action->cpu_eoi_map);
>
> (XEN) [<ffff82d08037b8aa>] common_interrupt+0x10a/0x120
> xen/arch/x86/x86_64/entry.S:58
>
> 47 /* Inject exception if pending. */
> 48 lea VCPU_trap_bounce(%rbx), %rdx
> 49 testb $TBF_EXCEPTION, TRAPBOUNCE_flags(%rdx)
> 50 jnz .Lprocess_trapbounce
> 51
> 52 cmpb $0, VCPU_mce_pending(%rbx)
> 53 jne process_mce
> 54 .Ltest_guest_nmi:
> 55 cmpb $0, VCPU_nmi_pending(%rbx)
> 56 jne process_nmi
> 57 test_guest_events:
> -> 58 movq VCPU_vcpu_info(%rbx), %rax
> 59 movzwl VCPUINFO_upcall_pending(%rax), %eax
> 60 decl %eax
> 61 cmpl $0xfe, %eax
> 62 ja restore_all_guest
> 63 /*process_guest_events:*/
> 64 sti
> 65 leaq VCPU_trap_bounce(%rbx), %rdx
> 66 movq VCPU_event_addr(%rbx), %rax
> 67 movq %rax, TRAPBOUNCE_eip(%rdx)
> 68 movb $TBF_INTERRUPT, TRAPBOUNCE_flags(%rdx)
> 69 call create_bounce_frame
> 70 jmp test_all_events
>
> (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> xen/arch/x86/cpu/mwait-idle.c:802
>
> 788 if (cpu_is_haltable(cpu))
> 789 mwait_idle_with_hints(eax,
> MWAIT_ECX_INTERRUPT_BREAK);
> 790
> 791 after = cpuidle_get_tick();
> 792
> 793 cstate_restore_tsc();
> 794 trace_exit_reason(irq_traced);
> 795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
> 796 irq_traced[0], irq_traced[1], irq_traced[2],
> irq_traced[3]);
> 797
> 798 /* Now back in C0. */
> 799 update_idle_stats(power, cx, before, after);
> 800 local_irq_enable();
> 801
> -> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
> 803 lapic_timer_on();
> 804
> 805 sched_tick_resume();
> 806 cpufreq_dbs_timer_resume();
>
> (XEN) [<ffff82d08027755d>] domain.c#idle_loop+0xb3/0xb5
> xen/arch/x86/domain.c:144
>
> 129 for ( ; ; )
> 130 {
> 131 if ( cpu_is_offline(cpu) )
> 132 play_dead();
> 133
> 134 /* Are we here for running vcpu context tasklets, or for idling?
> */
> 135 if ( unlikely(tasklet_work_to_do(cpu)) )
> 136 do_tasklet();
> 137 /*
> 138 * Test softirqs twice --- first to see if should even try scrubbing
> 139 * and then, after it is done, whether softirqs became pending
> 140 * while we were scrubbing.
> 141 */
> 142 else if ( !softirq_pending(cpu) && !scrub_free_pages() &&
> 143 !softirq_pending(cpu) )
> -> 144 pm_idle();
> 145 do_softirq();
> 146 /*
> 147 * We MUST be last (or before pm_idle). Otherwise after we get
> the
> 148 * softirq we would execute pm_idle (and sleep) and not patch.
> 149 */
> 150 check_for_livepatch_work();
> 151 }
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-12 10:36 ` Tian, Kevin
@ 2018-12-12 11:24 ` Roger Pau Monné
2018-12-12 11:48 ` Tian, Kevin
0 siblings, 1 reply; 30+ messages in thread
From: Roger Pau Monné @ 2018-12-12 11:24 UTC (permalink / raw)
To: Tian, Kevin; +Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > Sent: Monday, October 15, 2018 6:30 PM
> > (XEN) [22642] POWER TYPE 4
> > (XEN) [22643] IDLE PPR 0x00000020
> > (XEN) IRR
> > 00000000000000000000000000000000000000000000000000000000000000
> > 00
> > (XEN) ISR
> > 00000000020000000000000000000000000000000000000000000000000000
> > 00
> > (XEN) [22644] WAKE PPR 0x00000020
> > (XEN) IRR
> > 00000000020000000000000000000000000000000000000000000000000000
> > 00
> > (XEN) ISR
> > 00000000020000000000000000000000000000000000000000000000000000
> > 00
>
> looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
Yes, that's correct. Having a pending IRR and going idle doesn't
always trigger the spurious interrupt re-injection.
> is it a fixed pattern after how many rounds of Cstate enter/exit with
> pending IRR(0x21) then you see assertion happened (in this example
> it happens at 3rd time)?
It's not a fixed pattern, here's another trace with IRR(0x21) being
pending just once during the Cstate transitions:
(XEN) *** Pending EOI error ***
(XEN) cpu #1, irq 30, vector 0x21, sp 1
(XEN) Peoi stack: sp 1
(XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
(XEN) Peoi stack trace records:
(XEN) [ 7886] ACK_POST PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7887] POWER TYPE 5
(XEN) [ 7888] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7889] WAKE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7890] ACK_PRE PPR 0x000000f0
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) [ 7891] ACK_POST PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7892] POWER TYPE 5
(XEN) [ 7893] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7894] WAKE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7895] ACK_PRE PPR 0x000000f0
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) [ 7896] ACK_POST PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7897] POWER TYPE 5
(XEN) [ 7898] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7899] WAKE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7900] ACK_PRE PPR 0x000000f0
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000004
(XEN) [ 7901] ACK_POST PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7902] POWER TYPE 5
(XEN) [ 7903] IDLE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7904] WAKE PPR 0x00000010
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) [ 7905] PUSH {sp 0, irq 30, vec 0x21}
(XEN) [ 7906] POWER TYPE 3
(XEN) [ 7907] IDLE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7908] WAKE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7909] POWER TYPE 3
(XEN) [ 7910] IDLE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7911] WAKE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7912] POWER TYPE 3
(XEN) [ 7913] IDLE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7914] WAKE PPR 0x00000020
(XEN) IRR 0000000000000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7915] POWER TYPE 3
(XEN) [ 7916] IDLE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) [ 7917] WAKE PPR 0x00000020
(XEN) IRR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) ISR 0000000002000000000000000000000000000000000000000000000000000000
(XEN) All LAPIC state:
(XEN) [vector] ISR TMR IRR
(XEN) [1f:00] 00000000 00000000 00000000
(XEN) [3f:20] 00000002 00000000 00000000
(XEN) [5f:40] 00000000 00000000 00000000
(XEN) [7f:60] 00000000 00000000 00000000
(XEN) [9f:80] 00000000 00000000 00000000
(XEN) [bf:a0] 00000000 00000000 00000000
(XEN) [df:c0] 00000000 00000000 00000000
(XEN) [ff:e0] 00000000 00000000 04000000
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
(XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 1
(XEN) RIP: e008:[<ffff82d080288e1d>] do_IRQ+0x91c/0xb08
(XEN) RFLAGS: 0000000000010012 CONTEXT: hypervisor
(XEN) rax: ffff83086c67002c rbx: 0000000000000180 rcx: 0000000000000000
(XEN) rdx: ffff83086c687fff rsi: 000000000000000a rdi: ffff83086c601e24
(XEN) rbp: ffff83086c687d98 rsp: ffff83086c687d38 r8: ffff83086c690000
(XEN) r9: 0000000000000030 r10: 0000000004000000 r11: 0000000000000007
(XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
(XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4: 00000000003526e0
(XEN) cr3: 0000000855b9f000 cr2: 00007f86c9ffd708
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen code around <ffff82d080288e1d> (do_IRQ+0x91c/0xb08):
(XEN) 8d 7e 24 e8 6b 4c fb ff <0f> 0b 0f 0b 0f 0b 0f 0b b8 00 00 00 00 eb 4e 83
(XEN) Xen stack trace from rsp=ffff83086c687d38:
(XEN) ffff82d000000000 ffff83086c601e24 0000000000000000 ffff83086c6704e0
(XEN) ffff82d080384841 ffff82d080384835 ffff82d080384841 0000000000000000
(XEN) 0000000000000000 0000000000000000 ffff83086c687fff 0000000000000000
(XEN) 00007cf793978237 ffff82d0803848aa 00000002c9f5819b 0000000000000008
(XEN) ffff83086c6738d8 ffff83086c673820 ffff83086c687ec0 00000002c9f631f5
(XEN) ffff83086c6708f8 ffff83086c6704e0 ffff83086c670918 0000000000000180
(XEN) ffff83086c67002c 0000000000000000 ffff83086c687fff 0000000000000ffb
(XEN) ffff83086c673850 0000002100000000 ffff82d0802e8ba9 000000000000e008
(XEN) 0000000000000202 ffff83086c687e50 0000000000000000 ffff83086c68f000
(XEN) 00000020ffffffff 0000000000000000 ffff83086c687ee8 ffff82d0805a3900
(XEN) 0000000000000000 0000000000000000 000003530000012a ffff82d0805ab880
(XEN) 0000000000000001 ffff82d0805a3900 0000000000000001 0000000000000001
(XEN) ffff83085607d000 ffff83086c687ef0 ffff82d080278a68 ffff83086c6a5000
(XEN) ffff83086c68f000 ffff830851580000 ffff83086c6a5000 ffff83086c687da8
(XEN) 0000000000000000 0000000000000000 ffff880269a39e80 ffff880269a39e80
(XEN) 0000000000000004 0000000000000004 0000000000000001 00000000000000f3
(XEN) 0000000000000000 000000000b2777bc ffffffff816fe980 ffff88027331a7c0
(XEN) ffffffff82049af8 ffff88027331a7c0 00000000cb5efd00 0000beef0000beef
(XEN) ffffffff816fec52 000000bf0000beef 0000000000000246 ffffc90000d0be98
(XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
(XEN) Xen call trace:
(XEN) [<ffff82d080288e1d>] do_IRQ+0x91c/0xb08
(XEN) [<ffff82d0803848aa>] common_interrupt+0x10a/0x120
(XEN) [<ffff82d0802e8ba9>] mwait-idle.c#mwait_idle+0x2a5/0x381
(XEN) [<ffff82d080278a68>] domain.c#idle_loop+0xb3/0xb5
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-12 11:24 ` Roger Pau Monné
@ 2018-12-12 11:48 ` Tian, Kevin
2018-12-12 12:17 ` Roger Pau Monné
0 siblings, 1 reply; 30+ messages in thread
From: Tian, Kevin @ 2018-12-12 11:48 UTC (permalink / raw)
To: Roger Pau Monné
Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
> From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> Sent: Wednesday, December 12, 2018 7:25 PM
>
> On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > Sent: Monday, October 15, 2018 6:30 PM
> > > (XEN) [22642] POWER TYPE 4
> > > (XEN) [22643] IDLE PPR 0x00000020
> > > (XEN) IRR
> > >
> 00000000000000000000000000000000000000000000000000000000000000
> > > 00
> > > (XEN) ISR
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > 00
> > > (XEN) [22644] WAKE PPR 0x00000020
> > > (XEN) IRR
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > 00
> > > (XEN) ISR
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > 00
> >
> > looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
>
> Yes, that's correct. Having a pending IRR and going idle doesn't
> always trigger the spurious interrupt re-injection.
>
> > is it a fixed pattern after how many rounds of Cstate enter/exit with
> > pending IRR(0x21) then you see assertion happened (in this example
> > it happens at 3rd time)?
>
> It's not a fixed pattern, here's another trace with IRR(0x21) being
> pending just once during the Cstate transitions:
did you observe a case where such asset may occur when IRR(0x21)
is cleared but ISR (0x21) is set?
I want to understand whether pending ISR alone triggers such
problem or another pending IRR together is necessary...
>
> (XEN) *** Pending EOI error ***
> (XEN) cpu #1, irq 30, vector 0x21, sp 1
> (XEN) Peoi stack: sp 1
> (XEN) [ 0] irq 30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
> (XEN) Peoi stack trace records:
> (XEN) [ 7886] ACK_POST PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7887] POWER TYPE 5
> (XEN) [ 7888] IDLE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7889] WAKE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7890] ACK_PRE PPR 0x000000f0
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) [ 7891] ACK_POST PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7892] POWER TYPE 5
> (XEN) [ 7893] IDLE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7894] WAKE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7895] ACK_PRE PPR 0x000000f0
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) [ 7896] ACK_POST PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7897] POWER TYPE 5
> (XEN) [ 7898] IDLE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7899] WAKE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7900] ACK_PRE PPR 0x000000f0
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 04
> (XEN) [ 7901] ACK_POST PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7902] POWER TYPE 5
> (XEN) [ 7903] IDLE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7904] WAKE PPR 0x00000010
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7905] PUSH {sp 0, irq 30, vec 0x21}
> (XEN) [ 7906] POWER TYPE 3
> (XEN) [ 7907] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7908] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7909] POWER TYPE 3
> (XEN) [ 7910] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7911] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7912] POWER TYPE 3
> (XEN) [ 7913] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7914] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000000000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7915] POWER TYPE 3
> (XEN) [ 7916] IDLE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) [ 7917] WAKE PPR 0x00000020
> (XEN) IRR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) ISR
> 00000000020000000000000000000000000000000000000000000000000000
> 00
> (XEN) All LAPIC state:
> (XEN) [vector] ISR TMR IRR
> (XEN) [1f:00] 00000000 00000000 00000000
> (XEN) [3f:20] 00000002 00000000 00000000
> (XEN) [5f:40] 00000000 00000000 00000000
> (XEN) [7f:60] 00000000 00000000 00000000
> (XEN) [9f:80] 00000000 00000000 00000000
> (XEN) [bf:a0] 00000000 00000000 00000000
> (XEN) [df:c0] 00000000 00000000 00000000
> (XEN) [ff:e0] 00000000 00000000 04000000
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> (XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
> (XEN) CPU: 1
> (XEN) RIP: e008:[<ffff82d080288e1d>] do_IRQ+0x91c/0xb08
> (XEN) RFLAGS: 0000000000010012 CONTEXT: hypervisor
> (XEN) rax: ffff83086c67002c rbx: 0000000000000180 rcx:
> 0000000000000000
> (XEN) rdx: ffff83086c687fff rsi: 000000000000000a rdi: ffff83086c601e24
> (XEN) rbp: ffff83086c687d98 rsp: ffff83086c687d38 r8: ffff83086c690000
> (XEN) r9: 0000000000000030 r10: 0000000004000000 r11:
> 0000000000000007
> (XEN) r12: 000000000000011f r13: 00000000ffffffff r14: ffff83086c601e00
> (XEN) r15: ffff82cfffffb100 cr0: 0000000080050033 cr4:
> 00000000003526e0
> (XEN) cr3: 0000000855b9f000 cr2: 00007f86c9ffd708
> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss:
> 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> (XEN) Xen code around <ffff82d080288e1d> (do_IRQ+0x91c/0xb08):
> (XEN) 8d 7e 24 e8 6b 4c fb ff <0f> 0b 0f 0b 0f 0b 0f 0b b8 00 00 00 00 eb 4e
> 83
> (XEN) Xen stack trace from rsp=ffff83086c687d38:
> (XEN) ffff82d000000000 ffff83086c601e24 0000000000000000
> ffff83086c6704e0
> (XEN) ffff82d080384841 ffff82d080384835 ffff82d080384841
> 0000000000000000
> (XEN) 0000000000000000 0000000000000000 ffff83086c687fff
> 0000000000000000
> (XEN) 00007cf793978237 ffff82d0803848aa 00000002c9f5819b
> 0000000000000008
> (XEN) ffff83086c6738d8 ffff83086c673820 ffff83086c687ec0
> 00000002c9f631f5
> (XEN) ffff83086c6708f8 ffff83086c6704e0 ffff83086c670918
> 0000000000000180
> (XEN) ffff83086c67002c 0000000000000000 ffff83086c687fff
> 0000000000000ffb
> (XEN) ffff83086c673850 0000002100000000 ffff82d0802e8ba9
> 000000000000e008
> (XEN) 0000000000000202 ffff83086c687e50 0000000000000000
> ffff83086c68f000
> (XEN) 00000020ffffffff 0000000000000000 ffff83086c687ee8
> ffff82d0805a3900
> (XEN) 0000000000000000 0000000000000000 000003530000012a
> ffff82d0805ab880
> (XEN) 0000000000000001 ffff82d0805a3900 0000000000000001
> 0000000000000001
> (XEN) ffff83085607d000 ffff83086c687ef0 ffff82d080278a68
> ffff83086c6a5000
> (XEN) ffff83086c68f000 ffff830851580000 ffff83086c6a5000
> ffff83086c687da8
> (XEN) 0000000000000000 0000000000000000 ffff880269a39e80
> ffff880269a39e80
> (XEN) 0000000000000004 0000000000000004 0000000000000001
> 00000000000000f3
> (XEN) 0000000000000000 000000000b2777bc ffffffff816fe980
> ffff88027331a7c0
> (XEN) ffffffff82049af8 ffff88027331a7c0 00000000cb5efd00
> 0000beef0000beef
> (XEN) ffffffff816fec52 000000bf0000beef 0000000000000246
> ffffc90000d0be98
> (XEN) 000000000000beef 000000000000beef 000000000000beef
> 000000000000beef
> (XEN) Xen call trace:
> (XEN) [<ffff82d080288e1d>] do_IRQ+0x91c/0xb08
> (XEN) [<ffff82d0803848aa>] common_interrupt+0x10a/0x120
> (XEN) [<ffff82d0802e8ba9>] mwait-idle.c#mwait_idle+0x2a5/0x381
> (XEN) [<ffff82d080278a68>] domain.c#idle_loop+0xb3/0xb5
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1340
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-12 11:48 ` Tian, Kevin
@ 2018-12-12 12:17 ` Roger Pau Monné
2018-12-13 1:28 ` Tian, Kevin
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE9E951@SHSMSX101.ccr.corp.intel.com>
0 siblings, 2 replies; 30+ messages in thread
From: Roger Pau Monné @ 2018-12-12 12:17 UTC (permalink / raw)
To: Tian, Kevin; +Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
On Wed, Dec 12, 2018 at 11:48:52AM +0000, Tian, Kevin wrote:
> > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > Sent: Wednesday, December 12, 2018 7:25 PM
> >
> > On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > Sent: Monday, October 15, 2018 6:30 PM
> > > > (XEN) [22642] POWER TYPE 4
> > > > (XEN) [22643] IDLE PPR 0x00000020
> > > > (XEN) IRR
> > > >
> > 00000000000000000000000000000000000000000000000000000000000000
> > > > 00
> > > > (XEN) ISR
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > 00
> > > > (XEN) [22644] WAKE PPR 0x00000020
> > > > (XEN) IRR
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > 00
> > > > (XEN) ISR
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > 00
> > >
> > > looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
> >
> > Yes, that's correct. Having a pending IRR and going idle doesn't
> > always trigger the spurious interrupt re-injection.
> >
> > > is it a fixed pattern after how many rounds of Cstate enter/exit with
> > > pending IRR(0x21) then you see assertion happened (in this example
> > > it happens at 3rd time)?
> >
> > It's not a fixed pattern, here's another trace with IRR(0x21) being
> > pending just once during the Cstate transitions:
>
> did you observe a case where such asset may occur when IRR(0x21)
> is cleared but ISR (0x21) is set?
No, I've always seen both ISR and IRR set when the interrupt injection
happens. This of course doesn't mean it's not possible, but I have not
seen any trace with ISR(0x21) set and IRR(0x21) clear.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-12 12:17 ` Roger Pau Monné
@ 2018-12-13 1:28 ` Tian, Kevin
2018-12-13 8:36 ` Jan Beulich
2018-12-13 8:52 ` Roger Pau Monné
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE9E951@SHSMSX101.ccr.corp.intel.com>
1 sibling, 2 replies; 30+ messages in thread
From: Tian, Kevin @ 2018-12-13 1:28 UTC (permalink / raw)
To: Roger Pau Monné
Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
> From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> Sent: Wednesday, December 12, 2018 8:18 PM
>
> On Wed, Dec 12, 2018 at 11:48:52AM +0000, Tian, Kevin wrote:
> > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > Sent: Wednesday, December 12, 2018 7:25 PM
> > >
> > > On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > > Sent: Monday, October 15, 2018 6:30 PM
> > > > > (XEN) [22642] POWER TYPE 4
> > > > > (XEN) [22643] IDLE PPR 0x00000020
> > > > > (XEN) IRR
> > > > >
> > >
> 00000000000000000000000000000000000000000000000000000000000000
> > > > > 00
> > > > > (XEN) ISR
> > > > >
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > 00
> > > > > (XEN) [22644] WAKE PPR 0x00000020
> > > > > (XEN) IRR
> > > > >
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > 00
> > > > > (XEN) ISR
> > > > >
> > >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > 00
> > > >
> > > > looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
> > >
> > > Yes, that's correct. Having a pending IRR and going idle doesn't
> > > always trigger the spurious interrupt re-injection.
> > >
> > > > is it a fixed pattern after how many rounds of Cstate enter/exit with
> > > > pending IRR(0x21) then you see assertion happened (in this example
> > > > it happens at 3rd time)?
> > >
> > > It's not a fixed pattern, here's another trace with IRR(0x21) being
> > > pending just once during the Cstate transitions:
> >
> > did you observe a case where such asset may occur when IRR(0x21)
> > is cleared but ISR (0x21) is set?
>
> No, I've always seen both ISR and IRR set when the interrupt injection
> happens. This of course doesn't mean it's not possible, but I have not
> seen any trace with ISR(0x21) set and IRR(0x21) clear.
>
sorry but let me double confirm. You always see ISR[21]/IRR[21] being
set "before and after entering C3" to hit the problem, right? When
interrupt injection happens later, ISR[21] is set but IRR[21] is cleared (as
expected for normal interrupt delivery process).
btw I checked your original mail:
(XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
xen/arch/x86/cpu/mwait-idle.c:802
788 if (cpu_is_haltable(cpu))
789 mwait_idle_with_hints(eax, MWAIT_ECX_INTERRUPT_BREAK);
790
791 after = cpuidle_get_tick();
792
793 cstate_restore_tsc();
794 trace_exit_reason(irq_traced);
795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
796 irq_traced[0], irq_traced[1], irq_traced[2], irq_traced[3]);
797
798 /* Now back in C0. */
799 update_idle_stats(power, cx, before, after);
800 local_irq_enable();
801
-> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
803 lapic_timer_on();
804
805 sched_tick_resume();
806 cpufreq_dbs_timer_resume();
Looks above code is different from staging:
acpi_processor_idle:
acpi_idle_do_entry:
acpi_processor_ffh_cstate_enter:
mwait_idle_with_hints
there is no mwait_idle alone. and even with compiler optimization I didn't
find code sequence like above...
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE9E951@SHSMSX101.ccr.corp.intel.com>
@ 2018-12-13 2:44 ` Tian, Kevin
2018-12-13 8:39 ` Roger Pau Monné
0 siblings, 1 reply; 30+ messages in thread
From: Tian, Kevin @ 2018-12-13 2:44 UTC (permalink / raw)
To: 'Roger Pau Monné'
Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
btw can you also capture ISR/IRR/PPR right before local_irq_enable()?
though I didn't see a reason why code in-between may impact those
bits, it doesn't hurt to capture the context right before interrupt is
raised. :-)
> -----Original Message-----
> From: Tian, Kevin
> Sent: Thursday, December 13, 2018 9:28 AM
> To: Roger Pau Monné <roger.pau@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Wei Liu <wei.liu2@citrix.com>; Jan
> Beulich <JBeulich@suse.com>; Andrew Cooper
> <andrew.cooper3@citrix.com>; Nakajima, Jun <jun.nakajima@intel.com>
> Subject: RE: Interrupt injection with ISR set on Intel hardware
>
> > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > Sent: Wednesday, December 12, 2018 8:18 PM
> >
> > On Wed, Dec 12, 2018 at 11:48:52AM +0000, Tian, Kevin wrote:
> > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > Sent: Wednesday, December 12, 2018 7:25 PM
> > > >
> > > > On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > > > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > > > Sent: Monday, October 15, 2018 6:30 PM
> > > > > > (XEN) [22642] POWER TYPE 4
> > > > > > (XEN) [22643] IDLE PPR 0x00000020
> > > > > > (XEN) IRR
> > > > > >
> > > >
> >
> 00000000000000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) ISR
> > > > > >
> > > >
> >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) [22644] WAKE PPR 0x00000020
> > > > > > (XEN) IRR
> > > > > >
> > > >
> >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) ISR
> > > > > >
> > > >
> >
> 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > >
> > > > > looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
> > > >
> > > > Yes, that's correct. Having a pending IRR and going idle doesn't
> > > > always trigger the spurious interrupt re-injection.
> > > >
> > > > > is it a fixed pattern after how many rounds of Cstate enter/exit with
> > > > > pending IRR(0x21) then you see assertion happened (in this example
> > > > > it happens at 3rd time)?
> > > >
> > > > It's not a fixed pattern, here's another trace with IRR(0x21) being
> > > > pending just once during the Cstate transitions:
> > >
> > > did you observe a case where such asset may occur when IRR(0x21)
> > > is cleared but ISR (0x21) is set?
> >
> > No, I've always seen both ISR and IRR set when the interrupt injection
> > happens. This of course doesn't mean it's not possible, but I have not
> > seen any trace with ISR(0x21) set and IRR(0x21) clear.
> >
>
> sorry but let me double confirm. You always see ISR[21]/IRR[21] being
> set "before and after entering C3" to hit the problem, right? When
> interrupt injection happens later, ISR[21] is set but IRR[21] is cleared (as
> expected for normal interrupt delivery process).
>
> btw I checked your original mail:
>
> (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> xen/arch/x86/cpu/mwait-idle.c:802
>
> 788 if (cpu_is_haltable(cpu))
> 789 mwait_idle_with_hints(eax,
> MWAIT_ECX_INTERRUPT_BREAK);
> 790
> 791 after = cpuidle_get_tick();
> 792
> 793 cstate_restore_tsc();
> 794 trace_exit_reason(irq_traced);
> 795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
> 796 irq_traced[0], irq_traced[1], irq_traced[2],
> irq_traced[3]);
> 797
> 798 /* Now back in C0. */
> 799 update_idle_stats(power, cx, before, after);
> 800 local_irq_enable();
> 801
> -> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
> 803 lapic_timer_on();
> 804
> 805 sched_tick_resume();
> 806 cpufreq_dbs_timer_resume();
>
> Looks above code is different from staging:
>
> acpi_processor_idle:
> acpi_idle_do_entry:
> acpi_processor_ffh_cstate_enter:
> mwait_idle_with_hints
>
> there is no mwait_idle alone. and even with compiler optimization I didn't
> find code sequence like above...
>
> Thanks
> Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-13 1:28 ` Tian, Kevin
@ 2018-12-13 8:36 ` Jan Beulich
2018-12-13 9:03 ` Tian, Kevin
2018-12-13 8:52 ` Roger Pau Monné
1 sibling, 1 reply; 30+ messages in thread
From: Jan Beulich @ 2018-12-13 8:36 UTC (permalink / raw)
To: Kevin Tian
Cc: Andrew Cooper, Wei Liu, xen-devel, Jun Nakajima, Roger Pau Monne
>>> On 13.12.18 at 02:28, <kevin.tian@intel.com> wrote:
> btw I checked your original mail:
>
> (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> xen/arch/x86/cpu/mwait-idle.c:802
>
> 788 if (cpu_is_haltable(cpu))
> 789 mwait_idle_with_hints(eax, MWAIT_ECX_INTERRUPT_BREAK);
> 790
> 791 after = cpuidle_get_tick();
> 792
> 793 cstate_restore_tsc();
> 794 trace_exit_reason(irq_traced);
> 795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
> 796 irq_traced[0], irq_traced[1], irq_traced[2], irq_traced[3]);
> 797
> 798 /* Now back in C0. */
> 799 update_idle_stats(power, cx, before, after);
> 800 local_irq_enable();
> 801
> -> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
> 803 lapic_timer_on();
> 804
> 805 sched_tick_resume();
> 806 cpufreq_dbs_timer_resume();
>
> Looks above code is different from staging:
>
> acpi_processor_idle:
> acpi_idle_do_entry:
> acpi_processor_ffh_cstate_enter:
> mwait_idle_with_hints
>
> there is no mwait_idle alone. and even with compiler optimization I didn't
> find code sequence like above...
You're looking at two entirely different code paths, only one of
which can be in use in any particular case: Either the idle
entering routine used is acpi_processor_idle(), or (when the
processor is supported by that driver code) it is mwait_idle().
See mwait_idle_init() for when the latter gets used; the former
may get installed at the point the Dom0 kernel reports ACPI
C-state data (and only when mwait_idle() is not in use).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-13 2:44 ` Tian, Kevin
@ 2018-12-13 8:39 ` Roger Pau Monné
2018-12-13 9:04 ` Tian, Kevin
0 siblings, 1 reply; 30+ messages in thread
From: Roger Pau Monné @ 2018-12-13 8:39 UTC (permalink / raw)
To: Tian, Kevin; +Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
On Thu, Dec 13, 2018 at 02:44:00AM +0000, Tian, Kevin wrote:
> btw can you also capture ISR/IRR/PPR right before local_irq_enable()?
> though I didn't see a reason why code in-between may impact those
> bits, it doesn't hurt to capture the context right before interrupt is
> raised. :-)
I've done that and the result is the same as the ones that are
currently printed on the trace, there's no change to the registers
between the point where they are printed and the call to
local_irq_enable.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-13 1:28 ` Tian, Kevin
2018-12-13 8:36 ` Jan Beulich
@ 2018-12-13 8:52 ` Roger Pau Monné
1 sibling, 0 replies; 30+ messages in thread
From: Roger Pau Monné @ 2018-12-13 8:52 UTC (permalink / raw)
To: Tian, Kevin; +Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
On Thu, Dec 13, 2018 at 01:28:23AM +0000, Tian, Kevin wrote:
> > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > Sent: Wednesday, December 12, 2018 8:18 PM
> >
> > On Wed, Dec 12, 2018 at 11:48:52AM +0000, Tian, Kevin wrote:
> > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > Sent: Wednesday, December 12, 2018 7:25 PM
> > > >
> > > > On Wed, Dec 12, 2018 at 10:36:44AM +0000, Tian, Kevin wrote:
> > > > > > From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> > > > > > Sent: Monday, October 15, 2018 6:30 PM
> > > > > > (XEN) [22642] POWER TYPE 4
> > > > > > (XEN) [22643] IDLE PPR 0x00000020
> > > > > > (XEN) IRR
> > > > > >
> > > >
> > 00000000000000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) ISR
> > > > > >
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) [22644] WAKE PPR 0x00000020
> > > > > > (XEN) IRR
> > > > > >
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > > > (XEN) ISR
> > > > > >
> > > >
> > 00000000020000000000000000000000000000000000000000000000000000
> > > > > > 00
> > > > >
> > > > > looks pending IRR (0x21) doesn't always trigger a spurious interrupt?
> > > >
> > > > Yes, that's correct. Having a pending IRR and going idle doesn't
> > > > always trigger the spurious interrupt re-injection.
> > > >
> > > > > is it a fixed pattern after how many rounds of Cstate enter/exit with
> > > > > pending IRR(0x21) then you see assertion happened (in this example
> > > > > it happens at 3rd time)?
> > > >
> > > > It's not a fixed pattern, here's another trace with IRR(0x21) being
> > > > pending just once during the Cstate transitions:
> > >
> > > did you observe a case where such asset may occur when IRR(0x21)
> > > is cleared but ISR (0x21) is set?
> >
> > No, I've always seen both ISR and IRR set when the interrupt injection
> > happens. This of course doesn't mean it's not possible, but I have not
> > seen any trace with ISR(0x21) set and IRR(0x21) clear.
> >
>
> sorry but let me double confirm. You always see ISR[21]/IRR[21] being
> set "before and after entering C3" to hit the problem, right?
Yes, that's correct.
> When
> interrupt injection happens later, ISR[21] is set but IRR[21] is cleared (as
> expected for normal interrupt delivery process).
>
> btw I checked your original mail:
>
> (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> xen/arch/x86/cpu/mwait-idle.c:802
>
> 788 if (cpu_is_haltable(cpu))
> 789 mwait_idle_with_hints(eax, MWAIT_ECX_INTERRUPT_BREAK);
> 790
> 791 after = cpuidle_get_tick();
> 792
> 793 cstate_restore_tsc();
> 794 trace_exit_reason(irq_traced);
> 795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
> 796 irq_traced[0], irq_traced[1], irq_traced[2], irq_traced[3]);
> 797
> 798 /* Now back in C0. */
> 799 update_idle_stats(power, cx, before, after);
> 800 local_irq_enable();
> 801
> -> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
> 803 lapic_timer_on();
> 804
> 805 sched_tick_resume();
> 806 cpufreq_dbs_timer_resume();
>
> Looks above code is different from staging:
The code matches staging at the point where I posted the original bug
report, 2 months ago.
> acpi_processor_idle:
> acpi_idle_do_entry:
> acpi_processor_ffh_cstate_enter:
> mwait_idle_with_hints
There's another caller of mwait_idle_with_hints, the mwait_idle
function. This gets setup by mwait_idle_init.
> there is no mwait_idle alone.
There's a mwait_idle in current staging code, check:
xen/arch/x86/cpu/mwait-idle.c:718
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/cpu/mwait-idle.c;h=f89c52f2565e8deb7d4aec309124fe6fbd57b27d;hb=refs/heads/staging#l718
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-13 8:36 ` Jan Beulich
@ 2018-12-13 9:03 ` Tian, Kevin
0 siblings, 0 replies; 30+ messages in thread
From: Tian, Kevin @ 2018-12-13 9:03 UTC (permalink / raw)
To: Jan Beulich
Cc: Andrew Cooper, Wei Liu, xen-devel, Nakajima, Jun, Roger Pau Monne
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Thursday, December 13, 2018 4:37 PM
>
> >>> On 13.12.18 at 02:28, <kevin.tian@intel.com> wrote:
> > btw I checked your original mail:
> >
> > (XEN) [<ffff82d0802e2403>] mwait-idle.c#mwait_idle+0x2a5/0x381
> > xen/arch/x86/cpu/mwait-idle.c:802
> >
> > 788 if (cpu_is_haltable(cpu))
> > 789 mwait_idle_with_hints(eax,
> MWAIT_ECX_INTERRUPT_BREAK);
> > 790
> > 791 after = cpuidle_get_tick();
> > 792
> > 793 cstate_restore_tsc();
> > 794 trace_exit_reason(irq_traced);
> > 795 TRACE_6D(TRC_PM_IDLE_EXIT, cx->type, after,
> > 796 irq_traced[0], irq_traced[1], irq_traced[2],
> irq_traced[3]);
> > 797
> > 798 /* Now back in C0. */
> > 799 update_idle_stats(power, cx, before, after);
> > 800 local_irq_enable();
> > 801
> > -> 802 if (!(lapic_timer_reliable_states & (1 << cstate)))
> > 803 lapic_timer_on();
> > 804
> > 805 sched_tick_resume();
> > 806 cpufreq_dbs_timer_resume();
> >
> > Looks above code is different from staging:
> >
> > acpi_processor_idle:
> > acpi_idle_do_entry:
> > acpi_processor_ffh_cstate_enter:
> > mwait_idle_with_hints
> >
> > there is no mwait_idle alone. and even with compiler optimization I didn't
> > find code sequence like above...
>
> You're looking at two entirely different code paths, only one of
> which can be in use in any particular case: Either the idle
> entering routine used is acpi_processor_idle(), or (when the
> processor is supported by that driver code) it is mwait_idle().
> See mwait_idle_init() for when the latter gets used; the former
> may get installed at the point the Dom0 kernel reports ACPI
> C-state data (and only when mwait_idle() is not in use).
>
yes, I missed the other path.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: Interrupt injection with ISR set on Intel hardware
2018-12-13 8:39 ` Roger Pau Monné
@ 2018-12-13 9:04 ` Tian, Kevin
0 siblings, 0 replies; 30+ messages in thread
From: Tian, Kevin @ 2018-12-13 9:04 UTC (permalink / raw)
To: Roger Pau Monné
Cc: xen-devel, Wei Liu, Nakajima, Jun, Jan Beulich, Andrew Cooper
> From: Roger Pau Monné [mailto:roger.pau@citrix.com]
> Sent: Thursday, December 13, 2018 4:40 PM
>
> On Thu, Dec 13, 2018 at 02:44:00AM +0000, Tian, Kevin wrote:
> > btw can you also capture ISR/IRR/PPR right before local_irq_enable()?
> > though I didn't see a reason why code in-between may impact those
> > bits, it doesn't hurt to capture the context right before interrupt is
> > raised. :-)
>
> I've done that and the result is the same as the ones that are
> currently printed on the trace, there's no change to the registers
> between the point where they are printed and the call to
> local_irq_enable.
>
Thanks for confirmation.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2018-12-13 9:04 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-15 10:30 Interrupt injection with ISR set on Intel hardware Roger Pau Monné
2018-10-15 12:06 ` Andrew Cooper
2018-10-22 7:33 ` Chao Gao
2018-10-22 7:57 ` Andrew Cooper
2018-10-29 11:22 ` Roger Pau Monné
2018-10-25 12:51 ` Jan Beulich
2018-10-25 13:02 ` Andrew Cooper
2018-10-25 13:57 ` Jan Beulich
2018-10-30 6:59 ` Tian, Kevin
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE2BAB0@SHSMSX101.ccr.corp.intel.com>
2018-11-01 0:40 ` Tian, Kevin
2018-11-01 9:18 ` Andrew Cooper
2018-11-28 9:19 ` Roger Pau Monné
2018-12-02 8:52 ` Tian, Kevin
2018-10-29 16:33 ` Jan Beulich
2018-10-29 16:44 ` Andrew Cooper
2018-10-29 16:58 ` Jan Beulich
2018-10-29 17:06 ` Andrew Cooper
2018-10-30 7:32 ` Jan Beulich
2018-10-29 16:55 ` Roger Pau Monné
2018-12-12 10:36 ` Tian, Kevin
2018-12-12 11:24 ` Roger Pau Monné
2018-12-12 11:48 ` Tian, Kevin
2018-12-12 12:17 ` Roger Pau Monné
2018-12-13 1:28 ` Tian, Kevin
2018-12-13 8:36 ` Jan Beulich
2018-12-13 9:03 ` Tian, Kevin
2018-12-13 8:52 ` Roger Pau Monné
[not found] ` <AADFC41AFE54684AB9EE6CBC0274A5D19BE9E951@SHSMSX101.ccr.corp.intel.com>
2018-12-13 2:44 ` Tian, Kevin
2018-12-13 8:39 ` Roger Pau Monné
2018-12-13 9:04 ` Tian, Kevin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.