From: Peter Zijlstra <peterz@infradead.org>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Qian Cai <cai@lca.pw>, "Paul E. McKenney" <paulmck@kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Michael Ellerman <mpe@ellerman.id.au>,
linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
Borislav Petkov <bp@alien8.de>
Subject: Re: Endless soft-lockups for compiling workload since next-20200519
Date: Thu, 21 May 2020 11:39:38 +0200 [thread overview]
Message-ID: <20200521093938.GG325280@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <20200521004035.GA15455@lenoir>
On Thu, May 21, 2020 at 02:40:36AM +0200, Frederic Weisbecker wrote:
> On Wed, May 20, 2020 at 02:50:56PM +0200, Peter Zijlstra wrote:
> > On Tue, May 19, 2020 at 11:58:17PM -0400, Qian Cai wrote:
> > > Just a head up. Repeatedly compiling kernels for a while would trigger
> > > endless soft-lockups since next-20200519 on both x86_64 and powerpc.
> > > .config are in,
> >
> > Could be 90b5363acd47 ("sched: Clean up scheduler_ipi()"), although I've
> > not seen anything like that myself. Let me go have a look.
> >
> >
> > In as far as the logs are readable (they're a wrapped mess, please don't
> > do that!), they contain very little useful, as is typical with IPIs :/
> >
> > > [ 1167.993773][ C1] WARNING: CPU: 1 PID: 0 at kernel/smp.c:127
> > > flush_smp_call_function_queue+0x1fa/0x2e0
>
> So I've tried to think of a race that could produce that and here is
> the only thing I could come up with. It's a bit complicated unfortunately:
This:
> smp_call_function_single_async() { smp_call_function_single_async() {
> // verified csd->flags != CSD_LOCK // verified csd->flags != CSD_LOCK
> csd->flags = CSD_LOCK csd->flags = CSD_LOCK
concurrent smp_call_function_single_async() using the same csd is what
I'm looking at as well. Now in the ILB case there is an easy cure:
(because there is only a single ilb target)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 01f94cf52783..b6d8a7b991f0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10033,7 +10033,7 @@ static void kick_ilb(unsigned int flags)
* is idle. And the softirq performing nohz idle load balance
* will be run before returning from the IPI.
*/
- smp_call_function_single_async(ilb_cpu, &cpu_rq(ilb_cpu)->nohz_csd);
+ smp_call_function_single_async(ilb_cpu, &this_rq()->nohz_csd);
}
/*
Qian, can you give that a spin?
But I'm still not convinced of your scenario:
> kick_ilb() {
> atomic_fetch_or(...., nohz_flags(0))
> atomic_fetch_or(...., nohz_flags(0)) #VMENTER
> smp_call_function_single_async() { smp_call_function_single_async() {
> // verified csd->flags != CSD_LOCK // verified csd->flags != CSD_LOCK
> csd->flags = CSD_LOCK csd->flags = CSD_LOCK
Note that we check the return value of atomic_fetch_or() and bail if
someone else set a flag in KICK_MASK before us.
Aah, I suppose you're saying this can happen when:
!(flags & NOHZ_KICK_MASK)
? That's not supposed to happen though.
Anyway, let me go stare at the remove wake-up case, because i'm afraid
that might have the same problem too...
next prev parent reply other threads:[~2020-05-21 9:46 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-20 3:58 Endless soft-lockups for compiling workload since next-20200519 Qian Cai
2020-05-20 12:50 ` Peter Zijlstra
2020-05-20 14:06 ` Qian Cai
2020-05-21 0:40 ` Frederic Weisbecker
2020-05-21 9:39 ` Peter Zijlstra [this message]
2020-05-21 10:49 ` Peter Zijlstra
2020-05-21 11:00 ` Peter Zijlstra
2020-05-21 12:41 ` Frederic Weisbecker
2020-05-25 13:21 ` Peter Zijlstra
2020-05-25 14:05 ` Frederic Weisbecker
2020-05-25 14:38 ` Peter Zijlstra
2020-05-25 14:17 ` Peter Zijlstra
2020-05-22 2:00 ` Qian Cai
2020-05-21 10:10 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200521093938.GG325280@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=bp@alien8.de \
--cc=cai@lca.pw \
--cc=frederic@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mpe@ellerman.id.au \
--cc=paulmck@kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).