From: Manuel Bouyer <bouyer@antioche.eu.org>
To: "Roger Pau Monné" <roger.pau@citrix.com>
Cc: xen-devel@lists.xenproject.org
Subject: Re: NetBSD dom0 PVH: hardware interrupts stalls
Date: Wed, 18 Nov 2020 13:14:03 +0100 [thread overview]
Message-ID: <20201118121403.GC3126@antioche.eu.org> (raw)
In-Reply-To: <20201118100025.ic7r3kfsbdnr6muz@Air-de-Roger>
On Wed, Nov 18, 2020 at 11:00:25AM +0100, Roger Pau Monné wrote:
> On Wed, Nov 18, 2020 at 10:24:25AM +0100, Manuel Bouyer wrote:
> > On Wed, Nov 18, 2020 at 09:57:38AM +0100, Roger Pau Monné wrote:
> > > On Tue, Nov 17, 2020 at 05:40:33PM +0100, Manuel Bouyer wrote:
> > > > On Tue, Nov 17, 2020 at 04:58:07PM +0100, Roger Pau Monné wrote:
> > > > > [...]
> > > > >
> > > > > I have attached a patch below that will dump the vIO-APIC info as part
> > > > > of the 'i' debug key output, can you paste the whole output of the 'i'
> > > > > debug key when the system stalls?
> > > >
> > > > see attached file. Note that the kernel did unstall while 'i' output was
> > > > being printed, so it is mixed with some NetBSD kernel output.
> > > > The idt entry of the 'ioapic2 pin2' interrupt is 103 on CPU 0.
> > > >
> > > > I also put the whole sequence at
> > > > http://www-soc.lip6.fr/~bouyer/xen-log3.txt
> > >
> > > On one of the instances the pin shows up as masked, but I'm not sure
> > > if that's relevant since later it shows up as unmasked. Might just be
> > > part of how NetBSD handles such interrupts.
> >
> > Yes, NetBSD can mask an interrupt source if the interrupts needs to be delayed.
> > It will be unmasked once the interrupt has been handled.
>
> Yes, I think that's roughly the same model that FreeBSD uses for
> level IO-APIC interrupts: mask it until the handlers have been run.
>
> > Would it be possible that Xen misses an unmask write, or fails to
> > call the vector if the interrupt is again pending at the time of the
> > unmask ?
>
> Well, it should work properly, but we cannot discard anything.
I did some more instrumentation from the NetBSD kernel, including dumping
the iopic2 pin2 register.
At the time of the command timeout, the register value is 0x0000a067,
which, if I understant it properly, menas that there's no interrupt
pending (bit IOAPIC_REDLO_RIRR, 0x00004000, is not set).
From the NetBSD ddb, I can dump this register multiple times, waiting
several seconds, etc .., it doens't change).
Now if I call ioapic_dump_raw() from the debugger, which triggers some
XEN printf:
db{0}> call ioapic_dump_raw^M
Register dump of ioapic0^M
[ 203.5489060] 00 08000000 00170011 08000000(XEN) vioapic.c:124:d0v0 apic_mem_re
adl:undefined ioregsel 3
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 4
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 5
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 6
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 7
00000000^M
[ 203.5489060] 08(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 8
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 9
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel a
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel b
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel c
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel d
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel e
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel f
00000000^M
[ 203.5489060] 10 00010000 00000000 00010000 00000000 00010000 00000000 00010000 00000000^M
[...]
[ 203.5489060] Register dump of ioapic2^M
[ 203.5489060] 00 0a000000 00070011 0a000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 3
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 4
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 5
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 6
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 7
00000000^M
[ 203.5489060] 08(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 8
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel 9
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel a
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel b
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel c
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel d
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel e
00000000(XEN) vioapic.c:124:d0v0 apic_mem_readl:undefined ioregsel f
00000000^M
[ 203.5489060] 10 00010000 00000000 00010000 00000000 0000e067 00000000 00010000 00000000^M
then the register switches to 0000e067, with the IOAPIC_REDLO_RIRR bit set.
From here, if I continue from ddb, the dom0 boots.
I can get the same effect by just doing ^A^A^A so my guess is that it's
not accessing the iopic's register which changes the IOAPIC_REDLO_RIRR bit,
but the XEN printf. Also, from NetBSD, using a dump fuinction which
doesn't access undefined registers - and so doesn't trigger XEN printfs -
doens't change the IOAPIC_REDLO_RIRR bit either.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
next prev parent reply other threads:[~2020-11-18 12:14 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-17 15:09 NetBSD dom0 PVH: hardware interrupts stalls Manuel Bouyer
2020-11-17 15:58 ` Roger Pau Monné
2020-11-17 16:40 ` Manuel Bouyer
2020-11-18 8:57 ` Roger Pau Monné
2020-11-18 9:24 ` Manuel Bouyer
2020-11-18 10:00 ` Roger Pau Monné
2020-11-18 12:14 ` Manuel Bouyer [this message]
2020-11-18 14:39 ` Roger Pau Monné
2020-11-18 14:59 ` Jan Beulich
2020-11-19 14:19 ` Roger Pau Monné
2020-11-19 15:57 ` Manuel Bouyer
2020-11-19 16:57 ` Manuel Bouyer
2020-11-19 17:57 ` Manuel Bouyer
2020-11-20 8:09 ` Jan Beulich
2020-11-20 8:28 ` Roger Pau Monné
2020-11-20 8:52 ` Manuel Bouyer
2020-11-20 8:59 ` Jan Beulich
2020-11-20 9:27 ` Manuel Bouyer
2020-11-20 10:00 ` Jan Beulich
2020-11-20 10:38 ` Manuel Bouyer
2020-11-23 9:57 ` Roger Pau Monné
2020-11-23 11:32 ` Manuel Bouyer
2020-11-23 12:51 ` Roger Pau Monné
2020-11-23 14:31 ` Manuel Bouyer
2020-11-23 17:06 ` Roger Pau Monné
2020-11-23 17:39 ` Manuel Bouyer
2020-11-24 10:05 ` Jan Beulich
2020-11-24 12:21 ` Roger Pau Monné
2020-11-24 13:59 ` Manuel Bouyer
2020-11-24 14:09 ` Jan Beulich
2020-11-24 14:27 ` Manuel Bouyer
2020-11-24 14:33 ` Jan Beulich
2020-11-24 14:36 ` Jan Beulich
2020-11-24 14:52 ` Jan Beulich
2020-11-24 15:00 ` Roger Pau Monné
2020-11-24 15:08 ` Manuel Bouyer
2020-11-24 15:49 ` Roger Pau Monné
2020-11-24 16:09 ` Manuel Bouyer
2020-11-26 13:34 ` Roger Pau Monné
2020-11-26 14:16 ` Manuel Bouyer
2020-11-26 14:26 ` Roger Pau Monné
2020-11-26 15:09 ` Roger Pau Monné
2020-11-26 17:20 ` Manuel Bouyer
2020-11-27 10:59 ` Roger Pau Monné
2020-11-27 11:18 ` Jan Beulich
2020-11-27 11:19 ` Manuel Bouyer
2020-11-27 11:21 ` Jan Beulich
2020-11-27 13:10 ` Manuel Bouyer
2020-11-27 13:14 ` Jan Beulich
2020-11-27 13:18 ` Manuel Bouyer
2020-11-27 11:29 ` Jan Beulich
2020-11-27 13:13 ` Manuel Bouyer
2020-11-27 13:18 ` Jan Beulich
2020-11-27 13:31 ` Manuel Bouyer
2020-11-27 13:40 ` Jan Beulich
2020-11-27 13:49 ` Jürgen Groß
2020-11-27 13:59 ` Manuel Bouyer
2020-11-27 20:22 ` Roger Pau Monné
2020-11-27 21:44 ` Manuel Bouyer
2020-11-28 14:53 ` Roger Pau Monné
2020-11-28 17:14 ` Manuel Bouyer
2020-11-29 9:23 ` Manuel Bouyer
2020-11-30 10:00 ` Jan Beulich
2020-11-30 10:28 ` Manuel Bouyer
2020-11-30 11:35 ` Manuel Bouyer
2020-11-30 11:44 ` Jan Beulich
2020-11-30 11:50 ` Manuel Bouyer
2020-11-30 12:09 ` Jan Beulich
2020-11-24 14:42 ` Jan Beulich
2020-11-24 14:59 ` Roger Pau Monné
2020-11-24 15:18 ` Manuel Bouyer
2020-11-24 15:23 ` Jürgen Groß
2020-11-20 8:54 ` Jan Beulich
2020-11-20 9:13 ` Manuel Bouyer
2020-11-23 9:49 ` Roger Pau Monné
2020-11-18 15:03 ` Manuel Bouyer
2020-11-18 9:16 ` Jan Beulich
2020-11-18 9:28 ` Manuel Bouyer
2020-11-18 9:43 ` Jan Beulich
2020-11-18 10:14 ` Manuel Bouyer
2020-11-18 11:17 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201118121403.GC3126@antioche.eu.org \
--to=bouyer@antioche.eu.org \
--cc=roger.pau@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).