From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: Jan Beulich <jbeulich@suse.com>, <xen-devel@lists.xenproject.org>
Subject: Re: NetBSD dom0 PVH: hardware interrupts stalls
Date: Sat, 28 Nov 2020 15:53:11 +0100 [thread overview]
Message-ID: <20201128145311.3gmzq5lnkz6ajdtr@Air-de-Roger> (raw)
In-Reply-To: <20201127214420.GA637@antioche.eu.org>
[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]
On Fri, Nov 27, 2020 at 10:44:20PM +0100, Manuel Bouyer wrote:
> On Fri, Nov 27, 2020 at 09:22:11PM +0100, Roger Pau Monné wrote:
> > >
> > > But I can confirm that now, entering ^A^A^A gets interrupt going in again
> >
> > I think there are some weird things with dpci interrupts that I'm
> > trying to understand. I have a patch now that will panic when the
> > buffer is full, so we will hopefully be able to see the whole trace of
> > events. There will be no need for you to press the 'T' key now, the
> > system will panic when the buffer is full.
> >
> > Note this patch also removes the deassert done in pt_irq_time_out.
>
> thanks
> the trace is at
> http://www-soc.lip6.fr/~bouyer/xen-log13.txt
Thanks! I think I've found the issue and I'm attaching a possible fix
(fix.patch) to this email. In any case I've also attached a further
debug patch, in case the fix turns out to be wrong. Please test the
fix first, as the debug patch will end up triggering a panic when the
buffer is full.
Roger.
[-- Attachment #2: fix.patch --]
[-- Type: text/plain, Size: 1389 bytes --]
From 232112a292c3b82b3063ea6c7aab56afc8e03f67 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Sat, 28 Nov 2020 15:06:26 +0100
Subject: [PATCH] x86/vioapic: fix usage of index in place of GSI in
vioapic_write_redirent
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The usage of idx instead of the GSI in vioapic_write_redirent when
accessing gsi_assert_count can cause a PVH dom0 with multiple
vIO-APICs to lose interrupts in case a pin of a IO-APIC different than
the first one is unmasked with pending interrupts.
Switch to use gsi instead to fix the issue.
Fixes: 9f44b08f7d0e4 ('x86/vioapic: introduce support for multiple vIO APICS')
Signed-off-by: Roger Pau Monné <roge.rpau@citrix.com>
---
xen/arch/x86/hvm/vioapic.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
index 67d4a6237f..e64abee7a9 100644
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -260,7 +260,7 @@ static void vioapic_write_redirent(
pent->fields.remote_irr = 0;
else if ( !ent.fields.mask &&
!ent.fields.remote_irr &&
- hvm_irq->gsi_assert_count[idx] )
+ hvm_irq->gsi_assert_count[gsi] )
{
pent->fields.remote_irr = 1;
vioapic_deliver(vioapic, idx);
--
2.29.2
[-- Attachment #3: debug.patch --]
[-- Type: text/plain, Size: 7609 bytes --]
diff --git a/xen/arch/x86/hvm/irq.c b/xen/arch/x86/hvm/irq.c
index 38ac5fb6c7..9db3dcc957 100644
--- a/xen/arch/x86/hvm/irq.c
+++ b/xen/arch/x86/hvm/irq.c
@@ -187,6 +187,10 @@ void hvm_gsi_assert(struct domain *d, unsigned int gsi)
* to know if the GSI is pending or not.
*/
spin_lock(&d->arch.hvm.irq_lock);
+ if ( gsi == TRACK_IRQ )
+ debugtrace_printk("hvm_gsi_assert irq %u trig %u assert count %u\n",
+ gsi, trig, hvm_irq->gsi_assert_count[gsi]);
+
if ( trig == VIOAPIC_EDGE_TRIG || !hvm_irq->gsi_assert_count[gsi] )
{
if ( trig == VIOAPIC_LEVEL_TRIG )
diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
index e64abee7a9..df82147f9b 100644
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -257,7 +257,11 @@ static void vioapic_write_redirent(
vlapic_adjust_i8259_target(d);
}
else if ( ent.fields.trig_mode == VIOAPIC_EDGE_TRIG )
+ {
+ if ( gsi == TRACK_IRQ )
+ debugtrace_printk("vIO-APIC set edge trigger irq %u\n", gsi);
pent->fields.remote_irr = 0;
+ }
else if ( !ent.fields.mask &&
!ent.fields.remote_irr &&
hvm_irq->gsi_assert_count[gsi] )
@@ -278,6 +282,12 @@ static void vioapic_write_redirent(
*/
int ret = vioapic_hwdom_map_gsi(gsi, ent.fields.trig_mode,
ent.fields.polarity);
+
+ if ( gsi == TRACK_IRQ )
+ debugtrace_printk("vIO-APIC UNMASK irq %u irr %u mask %u assert count %u\n",
+ gsi, pent->fields.remote_irr, pent->fields.mask,
+ hvm_irq->gsi_assert_count[gsi]);
+
if ( ret )
{
gprintk(XENLOG_ERR,
@@ -285,6 +295,9 @@ static void vioapic_write_redirent(
unmasked = 0;
}
}
+ else if ( is_hardware_domain(d) && gsi == TRACK_IRQ )
+ debugtrace_printk("vIO-APIC MASK irq %u\n", gsi);
+
if ( gsi == 0 || unmasked )
pt_may_unmask_irq(d, NULL);
@@ -405,6 +418,10 @@ static void vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int pin)
ASSERT(spin_is_locked(&d->arch.hvm.irq_lock));
+ if ( irq == TRACK_IRQ )
+ debugtrace_printk("vIO-APIC deliver irq %u vector %u\n",
+ irq, vector);
+
HVM_DBG_LOG(DBG_LEVEL_IOAPIC,
"dest=%x dest_mode=%x delivery_mode=%x "
"vector=%x trig_mode=%x",
diff --git a/xen/arch/x86/io_apic.c b/xen/arch/x86/io_apic.c
index 49bd778484..db7167eb4b 100644
--- a/xen/arch/x86/io_apic.c
+++ b/xen/arch/x86/io_apic.c
@@ -1641,6 +1641,9 @@ static void mask_and_ack_level_ioapic_irq(struct irq_desc *desc)
unsigned long v;
int i;
+ if ( desc->irq == TRACK_IRQ )
+ debugtrace_printk("ACK irq %u\n", desc->irq);
+
irq_complete_move(desc);
if ( !directed_eoi_enabled )
@@ -1688,6 +1691,9 @@ static void mask_and_ack_level_ioapic_irq(struct irq_desc *desc)
static void end_level_ioapic_irq_old(struct irq_desc *desc, u8 vector)
{
+ if ( desc->irq == TRACK_IRQ )
+ debugtrace_printk("END irq %u\n", desc->irq);
+
if ( directed_eoi_enabled )
{
if ( !(desc->status & (IRQ_DISABLED|IRQ_MOVE_PENDING)) )
diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
index 8d1f9a9fc6..ec52e44cb7 100644
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -1109,6 +1109,10 @@ static void irq_guest_eoi_timer_fn(void *data)
unsigned int i, irq = desc - irq_desc;
irq_guest_action_t *action;
+ if ( desc->irq == TRACK_IRQ )
+ debugtrace_printk("irq_guest_eoi_timer_fn irq %u status %x\n",
+ desc->irq, desc->status);
+
spin_lock_irq(&desc->lock);
if ( !(desc->status & IRQ_GUEST) )
@@ -1118,6 +1122,10 @@ static void irq_guest_eoi_timer_fn(void *data)
ASSERT(action->ack_type != ACKTYPE_NONE);
+ if ( desc->irq == TRACK_IRQ )
+ debugtrace_printk("ack_type %u in_flight %u\n",
+ action->ack_type, action->in_flight);
+
/*
* Is no IRQ in flight at all, or another instance of this timer already
* running? Skip everything to avoid forcing an EOI early.
@@ -1837,6 +1845,12 @@ static void do_IRQ_guest(struct irq_desc *desc, unsigned int vector)
unsigned int i;
struct pending_eoi *peoi = this_cpu(pending_eoi);
+ if ( desc->irq == TRACK_IRQ )
+ debugtrace_printk("do_IRQ_guest irq %u nr_guests %u ack_type %u in_flight %u\n",
+ desc->irq, action->nr_guests, action->ack_type,
+ action->in_flight);
+
+
if ( unlikely(!action->nr_guests) )
{
/* An interrupt may slip through while freeing an ACKTYPE_EOI irq. */
diff --git a/xen/common/debugtrace.c b/xen/common/debugtrace.c
index f3794b9453..b22c09297d 100644
--- a/xen/common/debugtrace.c
+++ b/xen/common/debugtrace.c
@@ -130,14 +130,14 @@ static void debugtrace_toggle(void)
void debugtrace_dump(void)
{
- unsigned long flags;
+ //unsigned long flags;
watchdog_disable();
- spin_lock_irqsave(&debugtrace_lock, flags);
+ //spin_lock_irqsave(&debugtrace_lock, flags);
debugtrace_dump_worker();
- spin_unlock_irqrestore(&debugtrace_lock, flags);
+ //spin_unlock_irqrestore(&debugtrace_lock, flags);
watchdog_enable();
}
@@ -152,7 +152,10 @@ static void debugtrace_add_to_buf(char *buf)
{
data->buf[data->prd++] = *p;
if ( data->prd == debugtrace_bytes )
+ {
+ panic("END of buffer\n");
data->prd = 0;
+ }
}
}
diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c
index 6b1305a3e5..e0949b7057 100644
--- a/xen/drivers/passthrough/io.c
+++ b/xen/drivers/passthrough/io.c
@@ -174,7 +174,10 @@ static void pt_irq_time_out(void *data)
* In the identity mapped case the EOI can also be done now, this way
* the iteration over the list of domain pirqs is avoided.
*/
- hvm_gsi_deassert(irq_map->dom, dpci_pirq(irq_map)->pirq);
+ if ( dpci_pirq(irq_map)->pirq == TRACK_IRQ )
+ debugtrace_printk("pt_irq_time_out irq %u\n",
+ dpci_pirq(irq_map)->pirq);
+ //hvm_gsi_deassert(irq_map->dom, dpci_pirq(irq_map)->pirq);
irq_map->flags |= HVM_IRQ_DPCI_EOI_LATCH;
pt_irq_guest_eoi(irq_map->dom, irq_map, NULL);
spin_unlock(&irq_map->dom->event_lock);
@@ -828,6 +831,9 @@ int hvm_do_IRQ_dpci(struct domain *d, struct pirq *pirq)
!pirq_dpci || !(pirq_dpci->flags & HVM_IRQ_DPCI_MAPPED) )
return 0;
+ if ( pirq->pirq == TRACK_IRQ )
+ debugtrace_printk("hvm_do_IRQ_dpci irq %u\n", pirq->pirq);
+
pirq_dpci->masked = 1;
raise_softirq_for(pirq_dpci);
return 1;
@@ -1010,6 +1016,10 @@ void hvm_dpci_eoi(struct domain *d, unsigned int guest_gsi,
if ( !is_iommu_enabled(d) )
return;
+ if ( guest_gsi == TRACK_IRQ )
+ debugtrace_printk("hvm_dpci_eoi irq %u irr %u\n", guest_gsi,
+ ent->fields.remote_irr);
+
if ( is_hardware_domain(d) )
{
spin_lock(&d->event_lock);
diff --git a/xen/include/xen/irq.h b/xen/include/xen/irq.h
index 43d567fe44..91579c33b9 100644
--- a/xen/include/xen/irq.h
+++ b/xen/include/xen/irq.h
@@ -174,4 +174,6 @@ unsigned int arch_hwdom_irqs(domid_t);
void arch_evtchn_bind_pirq(struct domain *, int pirq);
#endif
+#define TRACK_IRQ 17
+
#endif /* __XEN_IRQ_H__ */
--
2.29.2
next prev parent reply other threads:[~2020-11-28 14:53 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-17 15:09 NetBSD dom0 PVH: hardware interrupts stalls Manuel Bouyer
2020-11-17 15:58 ` Roger Pau Monné
2020-11-17 16:40 ` Manuel Bouyer
2020-11-18 8:57 ` Roger Pau Monné
2020-11-18 9:24 ` Manuel Bouyer
2020-11-18 10:00 ` Roger Pau Monné
2020-11-18 12:14 ` Manuel Bouyer
2020-11-18 14:39 ` Roger Pau Monné
2020-11-18 14:59 ` Jan Beulich
2020-11-19 14:19 ` Roger Pau Monné
2020-11-19 15:57 ` Manuel Bouyer
2020-11-19 16:57 ` Manuel Bouyer
2020-11-19 17:57 ` Manuel Bouyer
2020-11-20 8:09 ` Jan Beulich
2020-11-20 8:28 ` Roger Pau Monné
2020-11-20 8:52 ` Manuel Bouyer
2020-11-20 8:59 ` Jan Beulich
2020-11-20 9:27 ` Manuel Bouyer
2020-11-20 10:00 ` Jan Beulich
2020-11-20 10:38 ` Manuel Bouyer
2020-11-23 9:57 ` Roger Pau Monné
2020-11-23 11:32 ` Manuel Bouyer
2020-11-23 12:51 ` Roger Pau Monné
2020-11-23 14:31 ` Manuel Bouyer
2020-11-23 17:06 ` Roger Pau Monné
2020-11-23 17:39 ` Manuel Bouyer
2020-11-24 10:05 ` Jan Beulich
2020-11-24 12:21 ` Roger Pau Monné
2020-11-24 13:59 ` Manuel Bouyer
2020-11-24 14:09 ` Jan Beulich
2020-11-24 14:27 ` Manuel Bouyer
2020-11-24 14:33 ` Jan Beulich
2020-11-24 14:36 ` Jan Beulich
2020-11-24 14:52 ` Jan Beulich
2020-11-24 15:00 ` Roger Pau Monné
2020-11-24 15:08 ` Manuel Bouyer
2020-11-24 15:49 ` Roger Pau Monné
2020-11-24 16:09 ` Manuel Bouyer
2020-11-26 13:34 ` Roger Pau Monné
2020-11-26 14:16 ` Manuel Bouyer
2020-11-26 14:26 ` Roger Pau Monné
2020-11-26 15:09 ` Roger Pau Monné
2020-11-26 17:20 ` Manuel Bouyer
2020-11-27 10:59 ` Roger Pau Monné
2020-11-27 11:18 ` Jan Beulich
2020-11-27 11:19 ` Manuel Bouyer
2020-11-27 11:21 ` Jan Beulich
2020-11-27 13:10 ` Manuel Bouyer
2020-11-27 13:14 ` Jan Beulich
2020-11-27 13:18 ` Manuel Bouyer
2020-11-27 11:29 ` Jan Beulich
2020-11-27 13:13 ` Manuel Bouyer
2020-11-27 13:18 ` Jan Beulich
2020-11-27 13:31 ` Manuel Bouyer
2020-11-27 13:40 ` Jan Beulich
2020-11-27 13:49 ` Jürgen Groß
2020-11-27 13:59 ` Manuel Bouyer
2020-11-27 20:22 ` Roger Pau Monné
2020-11-27 21:44 ` Manuel Bouyer
2020-11-28 14:53 ` Roger Pau Monné [this message]
2020-11-28 17:14 ` Manuel Bouyer
2020-11-29 9:23 ` Manuel Bouyer
2020-11-30 10:00 ` Jan Beulich
2020-11-30 10:28 ` Manuel Bouyer
2020-11-30 11:35 ` Manuel Bouyer
2020-11-30 11:44 ` Jan Beulich
2020-11-30 11:50 ` Manuel Bouyer
2020-11-30 12:09 ` Jan Beulich
2020-11-24 14:42 ` Jan Beulich
2020-11-24 14:59 ` Roger Pau Monné
2020-11-24 15:18 ` Manuel Bouyer
2020-11-24 15:23 ` Jürgen Groß
2020-11-20 8:54 ` Jan Beulich
2020-11-20 9:13 ` Manuel Bouyer
2020-11-23 9:49 ` Roger Pau Monné
2020-11-18 15:03 ` Manuel Bouyer
2020-11-18 9:16 ` Jan Beulich
2020-11-18 9:28 ` Manuel Bouyer
2020-11-18 9:43 ` Jan Beulich
2020-11-18 10:14 ` Manuel Bouyer
2020-11-18 11:17 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201128145311.3gmzq5lnkz6ajdtr@Air-de-Roger \
--to=roger.pau@citrix.com \
--cc=bouyer@antioche.eu.org \
--cc=jbeulich@suse.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).