From: Chris J Arges <chris.j.arges@canonical.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Rafael David Tinoco <inaddy@ubuntu.com>,
Ingo Molnar <mingo@kernel.org>, Peter Anvin <hpa@zytor.com>,
Jiang Liu <jiang.liu@linux.intel.com>,
Peter Zijlstra <peterz@infradead.org>,
LKML <linux-kernel@vger.kernel.org>, Jens Axboe <axboe@kernel.dk>,
Frederic Weisbecker <fweisbec@gmail.com>,
Gema Gomez <gema.gomez-solano@canonical.com>,
the arch/x86 maintainers <x86@kernel.org>
Subject: Re: smp_call_function_single lockups
Date: Wed, 1 Apr 2015 09:32:36 -0500 [thread overview]
Message-ID: <20150401143236.GB12730@canonical.com> (raw)
In-Reply-To: <CA+55aFyJXDdLAG8=N-ewiTRDrAs_457gL-XQGa6Q9o2poxpNGQ@mail.gmail.com>
On Tue, Mar 31, 2015 at 04:07:32PM -0700, Linus Torvalds wrote:
> On Tue, Mar 31, 2015 at 3:23 PM, Chris J Arges
> <chris.j.arges@canonical.com> wrote:
> >
> > I had a few runs with your patch plus modifications, and got the following
> > results (modified patch inlined below):
>
> Ok, thanks.
>
> > [ 14.423916] ack_APIC_irq: vector = d1, irq = ffffffff
> > [ 176.060005] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [qemu-system-x86:1630]
> >
> > [ 17.995298] ack_APIC_irq: vector = d1, irq = ffffffff
> > [ 182.993828] ack_APIC_irq: vector = e1, irq = ffffffff
> > [ 202.919691] ack_APIC_irq: vector = 22, irq = ffffffff
> > [ 484.132006] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [qemu-system-x86:1586]
> >
> > [ 15.592032] ack_APIC_irq: vector = d1, irq = ffffffff
> > [ 304.993490] ack_APIC_irq: vector = e1, irq = ffffffff
> > [ 315.174755] ack_APIC_irq: vector = 22, irq = ffffffff
> > [ 360.108007] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ksmd:26]
> .. snip snip ..
>
> So yeah, that's VECTOR_UNDEFINED, and while it could happen as part of
> irq setup, I'm not seeing that being something that your load should
> trigger.
>
> It could also obviously just be the vector being somehow corrupted,
> either due to crazy hardware or software.
>
> But quite frankly, the most likely reason is that whole irq vector movement.
>
> Especially since it sounds from your other email that when you apply
> Ingo's patches, the ack_APIC_irq warnings go away. Is that correct? Or
> did you just grep for "move" in the messages?
>
> If you do get both movement messages (from Info's patch) _and_ the
> ack_APIC_irq warnings (from mine), it would be interesting to see if
> the vectors line up somehow..
>
> Linus
>
Linus,
I included the full patch in reply to Ingo's email, and when running with that
I no longer get the ack_APIC_irq WARNs.
My next homework assignments are:
- Testing with irqbalance disabled
- Testing w/ the appropriate dump_stack() in Ingo's patch
- L0 testing
Thanks,
--chris
next prev parent reply other threads:[~2015-04-01 14:33 UTC|newest]
Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-11 13:19 smp_call_function_single lockups Rafael David Tinoco
2015-02-11 18:18 ` Linus Torvalds
2015-02-11 19:59 ` Linus Torvalds
2015-02-11 20:42 ` Linus Torvalds
2015-02-12 16:38 ` Rafael David Tinoco
2015-02-18 22:25 ` Peter Zijlstra
2015-02-19 15:42 ` Rafael David Tinoco
2015-02-19 16:14 ` Linus Torvalds
2015-02-23 14:01 ` Rafael David Tinoco
2015-02-23 19:32 ` Linus Torvalds
2015-02-23 20:50 ` Peter Zijlstra
2015-02-23 21:02 ` Rafael David Tinoco
2015-02-19 16:16 ` Peter Zijlstra
2015-02-19 16:26 ` Linus Torvalds
2015-02-19 16:32 ` Rafael David Tinoco
2015-02-19 16:59 ` Linus Torvalds
2015-02-19 17:30 ` Rafael David Tinoco
2015-02-19 17:39 ` Linus Torvalds
2015-02-19 20:29 ` Linus Torvalds
2015-02-19 21:59 ` Linus Torvalds
2015-02-19 22:45 ` Linus Torvalds
2015-03-31 3:15 ` Chris J Arges
2015-03-31 4:28 ` Linus Torvalds
2015-03-31 10:56 ` [debug PATCHes] " Ingo Molnar
2015-03-31 22:38 ` Chris J Arges
2015-04-01 12:39 ` Ingo Molnar
2015-04-01 14:10 ` Chris J Arges
2015-04-01 14:55 ` Ingo Molnar
2015-03-31 4:46 ` Linus Torvalds
2015-03-31 15:08 ` Linus Torvalds
2015-03-31 22:23 ` Chris J Arges
2015-03-31 23:07 ` Linus Torvalds
2015-04-01 14:32 ` Chris J Arges [this message]
2015-04-01 15:36 ` Linus Torvalds
2015-04-02 9:55 ` Ingo Molnar
2015-04-02 17:35 ` Linus Torvalds
2015-04-01 12:43 ` Ingo Molnar
2015-04-01 16:10 ` Chris J Arges
2015-04-01 16:14 ` Linus Torvalds
2015-04-01 21:59 ` Chris J Arges
2015-04-02 17:31 ` Linus Torvalds
2015-04-02 18:26 ` Ingo Molnar
2015-04-02 18:51 ` Chris J Arges
2015-04-02 19:07 ` Ingo Molnar
2015-04-02 20:57 ` Linus Torvalds
2015-04-02 21:13 ` Chris J Arges
2015-04-03 5:43 ` [PATCH] smp/call: Detect stuck CSD locks Ingo Molnar
2015-04-03 5:47 ` Ingo Molnar
2015-04-06 16:58 ` Chris J Arges
2015-04-06 17:32 ` Linus Torvalds
2015-04-07 9:21 ` Ingo Molnar
2015-04-07 20:59 ` Chris J Arges
2015-04-07 21:15 ` Linus Torvalds
2015-04-08 6:47 ` Ingo Molnar
2015-04-13 3:56 ` Chris J Arges
2015-04-13 6:14 ` Ingo Molnar
2015-04-15 19:54 ` Chris J Arges
2015-04-16 11:04 ` Ingo Molnar
2015-04-16 15:58 ` Chris J Arges
2015-04-16 16:31 ` Ingo Molnar
2015-04-29 21:08 ` Chris J Arges
2015-05-11 14:00 ` Ingo Molnar
2015-05-20 18:19 ` Chris J Arges
2015-04-03 5:45 ` smp_call_function_single lockups Ingo Molnar
2015-04-06 17:23 ` Chris J Arges
2015-02-20 9:30 ` Ingo Molnar
2015-02-20 16:49 ` Linus Torvalds
2015-02-20 19:41 ` Ingo Molnar
2015-02-20 20:03 ` Linus Torvalds
2015-02-20 20:11 ` Ingo Molnar
2015-03-20 10:15 ` Peter Zijlstra
2015-03-20 16:26 ` Linus Torvalds
2015-03-20 17:14 ` Mike Galbraith
2015-04-01 14:22 ` Frederic Weisbecker
2015-04-18 10:13 ` [tip:locking/urgent] smp: Fix smp_call_function_single_async() locking tip-bot for Linus Torvalds
2015-02-22 8:59 smp_call_function_single lockups Daniel J Blueman
2015-02-22 10:37 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150401143236.GB12730@canonical.com \
--to=chris.j.arges@canonical.com \
--cc=axboe@kernel.dk \
--cc=fweisbec@gmail.com \
--cc=gema.gomez-solano@canonical.com \
--cc=hpa@zytor.com \
--cc=inaddy@ubuntu.com \
--cc=jiang.liu@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).