Re: Yet more softlockups.

[Dave Jones <davej@redhat.com>]

On Fri, Jul 12, 2013 at 08:55:31AM -0700, Dave Hansen wrote:

 > I mean that somebody turned 'active_events' on without actually wanting
 > perf to be on.  I'd be curious how it got set to something nonzero.
 > Could you stick a WARN_ONCE() or printk_ratelimit() on the three sites
 > that modify it?
 
I'll add to my list of things to get to, but it probably won't be until post-weekend.

 > > Here's a fun trick:
 > > 
 > > trinity -c perf_event_open -C4 -q -l off
 > > 
 > > Within about a minute, that brings any of my boxes to its knees.
 > > The softlockup detector starts going nuts, and then the box wedges solid.
 > 
 > *But* dropping the perf sample rate has been really effective at keeping
 > me from hitting it, and I've had to use _lots_ of CPUs (60-160) doing
 > those NMIs at once to trigger the lockups.
 > 
 > Being able to trigger it with so few CPUs is interesting though.  I'll
 > try on my hardware.

I hacked trinity to pause for 24s after each call, and changed the kernel
to taint on lockup (so that trinity would immediatly stop).
My hope was to find the combination of perf_event_open calls that triggered this.
After 12 hours, it's still ticking along. It's now done about a thousand
more calls than is usually necessary to hit the bug. Which makes me wonder
if this is timing related somehow.

Perhaps also worth noting that the majority of calls trinity makes will -EINVAL

	Dave