From: Mark Rutland <mark.rutland@arm.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
jeremy.linton@arm.com
Subject: Re: Perf hotplug lockup in v4.9-rc8
Date: Wed, 7 Dec 2016 19:56:44 +0000 [thread overview]
Message-ID: <20161207195643.GA9027@leverpostej> (raw)
In-Reply-To: <20161207183455.GQ3124@twins.programming.kicks-ass.net>
On Wed, Dec 07, 2016 at 07:34:55PM +0100, Peter Zijlstra wrote:
> On Wed, Dec 07, 2016 at 05:53:47PM +0000, Mark Rutland wrote:
> > On Wed, Dec 07, 2016 at 01:52:17PM +0000, Mark Rutland wrote:
> > > Hi all
> > >
> > > Jeremy noticed a kernel lockup on arm64 when the perf tool was used in
> > > parallel with hotplug, which I've reproduced on arm64 and x86(-64) with
> > > v4.9-rc8. In both cases I'm using defconfig; I've tried enabling lockdep
> > > but it was silent for arm64 and x86.
> >
> > It looks like we're trying to install a task-bound event into a context
> > where task_cpu(ctx->task) is dead, and thus the cpu_function_call() in
> > perf_install_in_context() fails. We retry repeatedly.
> >
> > On !PREEMPT (as with x86 defconfig), we manage to prevent the hotplug
> > machinery from making progress, and this turns into a livelock.
> >
> > On PREEMPT (as with arm64 defconfig), I'm somewhat lost.
>
> So the problem is that even with PREEMPT we can hit a blocked task
> that has a 'dead' cpu.
>
> We'll spin until either the task wakes up or the CPU does, either can
> take a very long time.
>
> How exactly your test-case triggers this, all it executes is 'true' and
> that really shouldn't block much, is a mystery still.
The perf tool forks a helper process, which blocks on a pipe, and once
signalled, execs the target (i.e. true). The main perf process opens
(enable-on-exec) events on that, then writes to the pipe to wake up the
helper.
... so now I see why that makes us see a dead task_cpu(); thanks for the
explanation above!
[...]
> @@ -2352,6 +2357,28 @@ perf_install_in_context(struct perf_event_context *ctx,
> return;
> }
> raw_spin_unlock_irq(&ctx->lock);
> +
> + raw_spin_lock_irq(&task->pi_lock);
> + if (!(task->state == TASK_RUNNING || task->state == TASK_WAKING)) {
For a moment I thought there was a remaining race here with the lazy
ctx-switch if the new task was RUNNING on an online CPU, but I guess
we'll retry the cpu_function_call() in that case.
I'll attack this tomorrow when I can think again...
Thanks,
Mark.
next prev parent reply other threads:[~2016-12-07 19:57 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-12-07 13:53 Perf hotplug lockup in v4.9-rc8 Mark Rutland
2016-12-07 14:30 ` Mark Rutland
2016-12-07 16:39 ` Mark Rutland
2016-12-07 17:53 ` Mark Rutland
2016-12-07 18:34 ` Peter Zijlstra
2016-12-07 19:56 ` Mark Rutland [this message]
2016-12-09 13:59 ` Peter Zijlstra
2016-12-12 11:46 ` Will Deacon
2016-12-12 12:42 ` Peter Zijlstra
2016-12-22 8:45 ` Peter Zijlstra
2016-12-22 14:00 ` Peter Zijlstra
2016-12-22 16:33 ` Paul E. McKenney
2017-01-11 14:59 ` Mark Rutland
2017-01-11 16:03 ` Peter Zijlstra
2017-01-11 16:26 ` Mark Rutland
2017-01-11 19:51 ` Peter Zijlstra
2017-01-14 12:28 ` [tip:perf/urgent] perf/core: Fix sys_perf_event_open() vs. hotplug tip-bot for Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161207195643.GA9027@leverpostej \
--to=mark.rutland@arm.com \
--cc=acme@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=jeremy.linton@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.