From: Don Zickus <dzickus@redhat.com>
To: Guenter Roeck <linux@roeck-us.net>
Cc: Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
linux-kernel@vger.kernel.org, Ingo Molnar <mingo@kernel.org>
Subject: Re: Crashes in perf_event_ctx_lock_nested
Date: Tue, 31 Oct 2017 14:50:59 -0400 [thread overview]
Message-ID: <20171031185059.2yl4qrxvrqqzra3d@redhat.com> (raw)
In-Reply-To: <20171031171622.GA28688@roeck-us.net>
On Tue, Oct 31, 2017 at 10:16:22AM -0700, Guenter Roeck wrote:
> On Tue, Oct 31, 2017 at 02:48:50PM +0100, Peter Zijlstra wrote:
> > On Mon, Oct 30, 2017 at 03:45:12PM -0700, Guenter Roeck wrote:
> > > I added some logging and a long msleep() in hardlockup_detector_perf_cleanup().
> > > Here is the result:
> > >
> > > [ 0.274361] NMI watchdog: ############ hardlockup_detector_perf_init
> > > [ 0.274915] NMI watchdog: ############ hardlockup_detector_event_create(0)
> > > [ 0.277049] NMI watchdog: ############ hardlockup_detector_perf_cleanup
> > > [ 0.277593] NMI watchdog: ############ hardlockup_detector_perf_enable(0)
> > > [ 0.278027] NMI watchdog: ############ hardlockup_detector_event_create(0)
> > > [ 1.312044] NMI watchdog: ############ hardlockup_detector_perf_cleanup done
> > > [ 1.385122] NMI watchdog: ############ hardlockup_detector_perf_enable(1)
> > > [ 1.386028] NMI watchdog: ############ hardlockup_detector_event_create(1)
> > > [ 1.466102] NMI watchdog: ############ hardlockup_detector_perf_enable(2)
> > > [ 1.475536] NMI watchdog: ############ hardlockup_detector_event_create(2)
> > > [ 1.535099] NMI watchdog: ############ hardlockup_detector_perf_enable(3)
> > > [ 1.535101] NMI watchdog: ############ hardlockup_detector_event_create(3)
> >
> > > [ 7.222816] NMI watchdog: ############ hardlockup_detector_perf_disable(0)
> > > [ 7.230567] NMI watchdog: ############ hardlockup_detector_perf_disable(1)
> > > [ 7.243138] NMI watchdog: ############ hardlockup_detector_perf_disable(2)
> > > [ 7.250966] NMI watchdog: ############ hardlockup_detector_perf_disable(3)
> > > [ 7.258826] NMI watchdog: ############ hardlockup_detector_perf_enable(1)
> > > [ 7.258827] NMI watchdog: ############ hardlockup_detector_perf_cleanup
> > > [ 7.258831] NMI watchdog: ############ hardlockup_detector_perf_enable(2)
> > > [ 7.258833] NMI watchdog: ############ hardlockup_detector_perf_enable(0)
> > > [ 7.258834] NMI watchdog: ############ hardlockup_detector_event_create(2)
> > > [ 7.258835] NMI watchdog: ############ hardlockup_detector_event_create(0)
> > > [ 7.260169] NMI watchdog: ############ hardlockup_detector_perf_enable(3)
> > > [ 7.260170] NMI watchdog: ############ hardlockup_detector_event_create(3)
> > > [ 7.494251] NMI watchdog: ############ hardlockup_detector_event_create(1)
> > > [ 8.287135] NMI watchdog: ############ hardlockup_detector_perf_cleanup done
> > >
> > > Looks like there are a number of problems: hardlockup_detector_event_create()
> > > creates the event data structure even if it is already created,
> >
> > Right, that does look dodgy. And on its own should be fairly straight
> > forward to cure. But I'd like to understand the rest of it first.
> >
> > > and hardlockup_detector_perf_cleanup() runs unprotected and in
> > > parallel to the enable/create functions.
> >
> > Well, looking at the code, cpu_maps_update_begin() aka.
> > cpu_add_remove_lock is serializing cpu_up() and cpu_down() and _should_
> > thereby also serialize cleanup vs the smp_hotplug_thread operations.
> >
> > Your trace does indeed indicate this is not the case, but I cannot, from
> > the code, see how this could happen.
> >
> > Could you use trace_printk() instead and boot with
> > "trace_options=stacktrace" ?
> >
> Attached. Let me know if you need more information. Note this is with
> msleep(1000) in the cleanup function to avoid the crash.
>
> > > ALso, the following message is seen twice.
> > >
> > > [ 0.278758] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
> > > [ 7.258838] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
> > >
> > > I don't offer a proposed patch since I have no idea how to best solve the
> > > problem.
> > >
> > > Also, is the repeated enable/disable/cleanup as part of the normal boot
> > > really necessary ?
> >
> > That's weird, I don't see that on my machines. We very much only bring
> > up the CPUs _once_. Also note they're 7s apart. Did you do something
> > funny like resume-from-disk or so?
>
> No, just whatever Chrome OS does when it starts the kernel. The hardware
> used in this test is a Google Pixelbook, though we have also seen the problem
> with other Chromebooks.
Is Chrome OS, changing the default timeout from 10s to something else?
That would explain it as a script is executed late in the boot cycle and
explain the quick restart.
Cheers,
Don
next prev parent reply other threads:[~2017-10-31 18:51 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-30 22:45 Crashes in perf_event_ctx_lock_nested Guenter Roeck
2017-10-31 13:48 ` Peter Zijlstra
2017-10-31 17:16 ` Guenter Roeck
2017-10-31 18:50 ` Don Zickus [this message]
2017-10-31 20:12 ` Guenter Roeck
2017-10-31 20:23 ` Don Zickus
2017-10-31 21:32 ` Thomas Gleixner
2017-10-31 22:11 ` Guenter Roeck
2017-11-01 18:11 ` Don Zickus
2017-11-01 18:34 ` Guenter Roeck
2017-11-01 19:46 ` [tip:core/urgent] watchdog/hardlockup/perf: Use atomics to track in-use cpu counter tip-bot for Don Zickus
2017-11-01 20:28 ` tip-bot for Don Zickus
2017-11-01 18:22 ` Crashes in perf_event_ctx_lock_nested Thomas Gleixner
2017-11-01 8:14 ` Peter Zijlstra
2017-11-01 8:26 ` Thomas Gleixner
2017-11-01 19:46 ` [tip:core/urgent] watchdog/harclockup/perf: Revert a33d44843d45 ("watchdog/hardlockup/perf: Simplify deferred event destroy") tip-bot for Thomas Gleixner
2017-11-01 20:32 ` Guenter Roeck
2017-11-01 20:52 ` Thomas Gleixner
2017-11-01 20:27 ` tip-bot for Thomas Gleixner
2017-10-31 18:48 ` Crashes in perf_event_ctx_lock_nested Don Zickus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171031185059.2yl4qrxvrqqzra3d@redhat.com \
--to=dzickus@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@roeck-us.net \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).