Re: [PATCH v3 2/2] Make hard lockup detection use timestamps

[ZAK Magnus <zakmagnus@google.com>]

No news?

I've been testing and looking into issues and I realized dump_stack()
calls touch_nmi_watchdog(). That wrecks what the patch is trying to do
so I'm changing it to save the trace and print it later after the
stall has completed. This would also resolve some other things you
were saying weren't so good. Hopefully the logic is similar enough
that some things you may have learned still apply.

On Mon, Jul 25, 2011 at 5:44 AM, Don Zickus <dzickus@redhat.com> wrote:
> On Fri, Jul 22, 2011 at 03:34:37PM -0700, ZAK Magnus wrote:
>> On Fri, Jul 22, 2011 at 12:53 PM, Don Zickus <dzickus@redhat.com> wrote:
>>
>> > So I played with the hardlockup case and I kinda like the timestamp thing.
>> > It seems to give useful data.  In fact I feel like I can shrink the
>> > hardlockup window, run some tests and see where the latencies are in a
>> > system.  The patch itself I think is ok, I'll review on Monday or Tuesday
>> > when I get some more free time.
>> >
>> > However, I ran the softlockup case and the output was a mess.  I think
>> > rcu_sched stalls were being detected and as a result it was NMI dumping
>> > stack traces for all cpus.  I can't tell if it was your patch or some
>> > uncovered bug.
>> >
>> > I'll dig into on Monday.  Not sure if you were able to see that.
>> >
>> > Thanks,
>> > Don
>> >
>> I'm not sure what you mean. One problem could be the wording I used.
>> For the soft stalls I just called it LOCKUP, mostly to be very showy
>> in order to cover that case where it's unclear what exactly is
>> happening. This doesn't do much to distinguish soft and hard lockups,
>> and I see LOCKUP otherwise seems to refer to hard lockup, so maybe
>> that's misleading.
>
> It had nothing to do with the wording.  It was spewing a ton of stack
> traces.  Most of them related to rcu_sched stalls which requested stack
> traces for each cpu (and the machine I as on had 16 cpus) repeatedly.
>
> So from a user perspective, I just saw a flood of stack traces scroll
> across the screen forever for a minute.  It was impossible to determine
> what was going on without reviewing the logs once everything calmed down.
> That is never a good thing.  It probably has nothing to do with your
> patch, but it is something that should be looked at.
>
> I'll try and poke today or tomorrow.
>
> Cheers,
> Don
>