From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Jan Beulich <jbeulich@suse.com>, Manuel Bouyer <bouyer@antioche.eu.org>
Cc: <xen-devel@lists.xenproject.org>
Subject: Re: NetBSD dom0 PVH: hardware interrupts stalls
Date: Thu, 19 Nov 2020 15:19:15 +0100 [thread overview]
Message-ID: <20201119141915.igyb7djkw47rf2dt@Air-de-Roger> (raw)
In-Reply-To: <bb2b6182-f3a6-61e5-ee70-90a65ae56435@suse.com>
[-- Attachment #1: Type: text/plain, Size: 5267 bytes --]
On Wed, Nov 18, 2020 at 03:59:44PM +0100, Jan Beulich wrote:
> On 18.11.2020 15:39, Roger Pau Monné wrote:
> > On Wed, Nov 18, 2020 at 01:14:03PM +0100, Manuel Bouyer wrote:
> >> I did some more instrumentation from the NetBSD kernel, including dumping
> >> the iopic2 pin2 register.
> >>
> >> At the time of the command timeout, the register value is 0x0000a067,
> >> which, if I understant it properly, menas that there's no interrupt
> >> pending (bit IOAPIC_REDLO_RIRR, 0x00004000, is not set).
> >> From the NetBSD ddb, I can dump this register multiple times, waiting
> >> several seconds, etc .., it doens't change).
> >> Now if I call ioapic_dump_raw() from the debugger, which triggers some
> >> XEN printf:
> >> db{0}> call ioapic_dump_raw^M
> >> Register dump of ioapic0^M
> >> [ 203.5489060] 00 08000000 00170011 08000000(XEN) vioapic.c:124:d0v0 apic_mem_re
> >> adl:undefined ioregsel 3
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 4
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 5
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 6
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 7
> >> 00000000^M
> >> [ 203.5489060] 08(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 8
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 9
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel a
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel b
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel c
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel d
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel e
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel f
> >> 00000000^M
> >> [ 203.5489060] 10 00010000 00000000 00010000 00000000 00010000 00000000 00010000 00000000^M
> >> [...]
> >> [ 203.5489060] Register dump of ioapic2^M
> >> [ 203.5489060] 00 0a000000 00070011 0a000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 3
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 4
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 5
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 6
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 7
> >> 00000000^M
> >> [ 203.5489060] 08(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 8
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 9
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel a
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel b
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel c
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel d
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel e
> >> 00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel f
> >> 00000000^M
> >> [ 203.5489060] 10 00010000 00000000 00010000 00000000 0000e067 00000000 00010000 00000000^M
> >>
> >> then the register switches to 0000e067, with the IOAPIC_REDLO_RIRR bit set.
> >> From here, if I continue from ddb, the dom0 boots.
> >>
> >> I can get the same effect by just doing ^A^A^A so my guess is that it's
> >> not accessing the iopic's register which changes the IOAPIC_REDLO_RIRR bit,
> >> but the XEN printf. Also, from NetBSD, using a dump fuinction which
> >> doesn't access undefined registers - and so doesn't trigger XEN printfs -
> >> doens't change the IOAPIC_REDLO_RIRR bit either.
> >
> > I'm thinking about further ways to debug this. I see that all active
> > IO-APIC pins are routed to vCPU0, but does it make a difference if you
> > boot with dom0_max_vcpus=1 on the Xen command line? (thus limiting
> > NertBSD dom0 to a single CPU)
>
> I too have been pondering possible approaches. One thing I thought might
> help is accompany all places setting remote_irr (and calling
> vioapic_deliver()) with a conditional log message, turning on the
> condition immediately before the first "undefined ioregsel" gets logged.
> (And turn it off again once the last RTE was read in sequence, just to
> avoid spamming the console.) From Manuel's description above, there has
> to be something that sets the bit and causes the delivery _without_ any
> active action by the guest (i.e. neither EOI nor RTE write) and
> _without_ any new instance of the IRQ appearing. I have some vague hope
> that knowing how we end up making the system make progress again may
> also help understand how it got stuck.
I've got two different debug patches for you to try. I'm attaching both
to this email but I think we should start with Jan's suggestion
(conditional_print.patch). That patch will only print extra messages
between the ioregsel 3 ... ioregsel f existing debug messages, you
will have to trigger this from NetBSD by using ioapic_dump_raw AFAICT.
The other patch (verbose_intr.patch) adds some messages related to
interrupts, but I'm afraid it's likely to interfere with the issue we
are trying to debug, since it will add a lot of extra printk's (and
likely flood your console).
Thanks, Roger.
[-- Attachment #2: conditional_print.patch --]
[-- Type: text/plain, Size: 2002 bytes --]
From 9f1cf4f5f4f143be2d9e87d1b2f4c4cf4c286b69 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Thu, 19 Nov 2020 14:11:43 +0100
Subject: [PATCH]
---
xen/arch/x86/hvm/vioapic.c | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
index 67d4a6237f..9c0423b26e 100644
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -43,7 +43,13 @@
/* HACK: Route IRQ0 only to VCPU0 to prevent time jumps. */
#define IRQ0_SPECIAL_ROUTING 1
-static void vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int irq);
+static void _vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int irq);
+
+static bool __read_mostly print;
+#define vioapic_deliver(vioapic, irq) ({\
+ if ( print && irq == 34 ) \
+ printk("%s:%d:%s: vioapic_deliver\n", __FILE__, __LINE__, __func__); \
+ _vioapic_deliver(vioapic, irq); })
static struct hvm_vioapic *addr_vioapic(const struct domain *d,
unsigned long addr)
@@ -119,6 +125,16 @@ static uint32_t vioapic_read_indirect(const struct hvm_vioapic *vioapic)
if ( redir_index >= vioapic->nr_pins )
{
+ switch ( vioapic->ioregsel )
+ {
+ case 3:
+ print = true;
+ break;
+
+ case 0xf:
+ print = false;
+ break;
+ }
gdprintk(XENLOG_WARNING, "apic_mem_readl:undefined ioregsel %x\n",
vioapic->ioregsel);
break;
@@ -391,7 +407,7 @@ static void ioapic_inj_irq(
vlapic_set_irq(target, vector, trig_mode);
}
-static void vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int pin)
+static void _vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int pin)
{
uint16_t dest = vioapic->redirtbl[pin].fields.dest_id;
uint8_t dest_mode = vioapic->redirtbl[pin].fields.dest_mode;
--
2.29.2
[-- Attachment #3: verbose_intr.patch --]
[-- Type: text/plain, Size: 2950 bytes --]
From 381eebe51657a3b3101dc80880fa3350dcfb1e23 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Thu, 19 Nov 2020 10:45:00 +0100
Subject: [PATCH]
---
xen/arch/x86/hvm/vioapic.c | 9 +++++++++
xen/arch/x86/irq.c | 3 +++
xen/drivers/passthrough/io.c | 3 +++
xen/include/asm-x86/irq.h | 2 ++
4 files changed, 17 insertions(+)
diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
index 67d4a6237f..1702434f0d 100644
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -235,6 +235,10 @@ static void vioapic_write_redirent(
pent = &vioapic->redirtbl[idx];
ent = *pent;
+ if ( gsi == TRACK_IRQ )
+ printk("vioapic write %s word %d val %#x current %#lx\n",
+ top_word ? "top" : "low", val, ent.bits);
+
if ( top_word )
{
/* Contains only the dest_id. */
@@ -410,6 +414,9 @@ static void vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int pin)
"vector=%x trig_mode=%x",
dest, dest_mode, delivery_mode, vector, trig_mode);
+ if ( irq == TRACK_IRQ )
+ printk("vioapic inject vector %#x\n", vector);
+
switch ( delivery_mode )
{
case dest_LowestPrio:
@@ -517,6 +524,8 @@ void vioapic_update_EOI(struct domain *d, u8 vector)
continue;
ent->fields.remote_irr = 0;
+ if ( vioapic->base_gsi + pin == TRACK_IRQ )
+ printk("vioapic EOI\n");
if ( is_iommu_enabled(d) )
{
diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
index 8d1f9a9fc6..34f50a24ea 100644
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -1837,6 +1837,9 @@ static void do_IRQ_guest(struct irq_desc *desc, unsigned int vector)
unsigned int i;
struct pending_eoi *peoi = this_cpu(pending_eoi);
+ if ( desc->irq == TRACK_IRQ )
+ printk("do_IRQ_guest\n");
+
if ( unlikely(!action->nr_guests) )
{
/* An interrupt may slip through while freeing an ACKTYPE_EOI irq. */
diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c
index 6b1305a3e5..ddbda10593 100644
--- a/xen/drivers/passthrough/io.c
+++ b/xen/drivers/passthrough/io.c
@@ -997,6 +997,9 @@ static void hvm_gsi_eoi(struct domain *d, unsigned int gsi,
if ( !pirq_dpci(pirq) )
return;
+ if ( gsi == TRACK_IRQ )
+ printk("dpci EOI\n");
+
hvm_gsi_deassert(d, gsi);
hvm_pirq_eoi(pirq, ent);
}
diff --git a/xen/include/asm-x86/irq.h b/xen/include/asm-x86/irq.h
index 7c825e9d9c..bb450e5e59 100644
--- a/xen/include/asm-x86/irq.h
+++ b/xen/include/asm-x86/irq.h
@@ -218,4 +218,6 @@ int allocate_and_map_gsi_pirq(struct domain *d, int index, int *pirq_p);
int allocate_and_map_msi_pirq(struct domain *d, int index, int *pirq_p,
int type, struct msi_info *msi);
+#define TRACK_IRQ 34
+
#endif /* _ASM_HW_IRQ_H */
--
2.29.2
next prev parent reply other threads:[~2020-11-19 14:20 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-17 15:09 NetBSD dom0 PVH: hardware interrupts stalls Manuel Bouyer
2020-11-17 15:58 ` Roger Pau Monné
2020-11-17 16:40 ` Manuel Bouyer
2020-11-18 8:57 ` Roger Pau Monné
2020-11-18 9:24 ` Manuel Bouyer
2020-11-18 10:00 ` Roger Pau Monné
2020-11-18 12:14 ` Manuel Bouyer
2020-11-18 14:39 ` Roger Pau Monné
2020-11-18 14:59 ` Jan Beulich
2020-11-19 14:19 ` Roger Pau Monné [this message]
2020-11-19 15:57 ` Manuel Bouyer
2020-11-19 16:57 ` Manuel Bouyer
2020-11-19 17:57 ` Manuel Bouyer
2020-11-20 8:09 ` Jan Beulich
2020-11-20 8:28 ` Roger Pau Monné
2020-11-20 8:52 ` Manuel Bouyer
2020-11-20 8:59 ` Jan Beulich
2020-11-20 9:27 ` Manuel Bouyer
2020-11-20 10:00 ` Jan Beulich
2020-11-20 10:38 ` Manuel Bouyer
2020-11-23 9:57 ` Roger Pau Monné
2020-11-23 11:32 ` Manuel Bouyer
2020-11-23 12:51 ` Roger Pau Monné
2020-11-23 14:31 ` Manuel Bouyer
2020-11-23 17:06 ` Roger Pau Monné
2020-11-23 17:39 ` Manuel Bouyer
2020-11-24 10:05 ` Jan Beulich
2020-11-24 12:21 ` Roger Pau Monné
2020-11-24 13:59 ` Manuel Bouyer
2020-11-24 14:09 ` Jan Beulich
2020-11-24 14:27 ` Manuel Bouyer
2020-11-24 14:33 ` Jan Beulich
2020-11-24 14:36 ` Jan Beulich
2020-11-24 14:52 ` Jan Beulich
2020-11-24 15:00 ` Roger Pau Monné
2020-11-24 15:08 ` Manuel Bouyer
2020-11-24 15:49 ` Roger Pau Monné
2020-11-24 16:09 ` Manuel Bouyer
2020-11-26 13:34 ` Roger Pau Monné
2020-11-26 14:16 ` Manuel Bouyer
2020-11-26 14:26 ` Roger Pau Monné
2020-11-26 15:09 ` Roger Pau Monné
2020-11-26 17:20 ` Manuel Bouyer
2020-11-27 10:59 ` Roger Pau Monné
2020-11-27 11:18 ` Jan Beulich
2020-11-27 11:19 ` Manuel Bouyer
2020-11-27 11:21 ` Jan Beulich
2020-11-27 13:10 ` Manuel Bouyer
2020-11-27 13:14 ` Jan Beulich
2020-11-27 13:18 ` Manuel Bouyer
2020-11-27 11:29 ` Jan Beulich
2020-11-27 13:13 ` Manuel Bouyer
2020-11-27 13:18 ` Jan Beulich
2020-11-27 13:31 ` Manuel Bouyer
2020-11-27 13:40 ` Jan Beulich
2020-11-27 13:49 ` Jürgen Groß
2020-11-27 13:59 ` Manuel Bouyer
2020-11-27 20:22 ` Roger Pau Monné
2020-11-27 21:44 ` Manuel Bouyer
2020-11-28 14:53 ` Roger Pau Monné
2020-11-28 17:14 ` Manuel Bouyer
2020-11-29 9:23 ` Manuel Bouyer
2020-11-30 10:00 ` Jan Beulich
2020-11-30 10:28 ` Manuel Bouyer
2020-11-30 11:35 ` Manuel Bouyer
2020-11-30 11:44 ` Jan Beulich
2020-11-30 11:50 ` Manuel Bouyer
2020-11-30 12:09 ` Jan Beulich
2020-11-24 14:42 ` Jan Beulich
2020-11-24 14:59 ` Roger Pau Monné
2020-11-24 15:18 ` Manuel Bouyer
2020-11-24 15:23 ` Jürgen Groß
2020-11-20 8:54 ` Jan Beulich
2020-11-20 9:13 ` Manuel Bouyer
2020-11-23 9:49 ` Roger Pau Monné
2020-11-18 15:03 ` Manuel Bouyer
2020-11-18 9:16 ` Jan Beulich
2020-11-18 9:28 ` Manuel Bouyer
2020-11-18 9:43 ` Jan Beulich
2020-11-18 10:14 ` Manuel Bouyer
2020-11-18 11:17 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201119141915.igyb7djkw47rf2dt@Air-de-Roger \
--to=roger.pau@citrix.com \
--cc=bouyer@antioche.eu.org \
--cc=jbeulich@suse.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).