RE: [PATCH V2] kernel/watchdog: fix spurious hard lockups

["Liang, Kan" <kan.liang@intel.com>]

> On Mon, Jun 26, 2017 at 04:19:27PM -0400, Don Zickus wrote:
> > On Fri, Jun 23, 2017 at 11:50:25PM +0200, Thomas Gleixner wrote:
> > > On Fri, 23 Jun 2017, Don Zickus wrote:
> > > > Hmm, all this work for a temp fix.  Kan, how much longer until the
> > > > real fix of having perf count the right cycles?
> > >
> > > Quite a while. The approach is wilfully breaking the user space ABI,
> > > which is not going to happen.
> > >
> > > And there is a simpler solution as well, as I said here:
> > >
> > >
> > > http://lkml.kernel.org/r/alpine.DEB.2.20.1706221730520.1885@nanos
> >
> > Hi Thomas,
> >
> > So, you are saying instead of slowing down the perf counter, speed up
> > the hrtimer to sample more frequently like so:
> >
> > diff --git a/kernel/watchdog.c b/kernel/watchdog.c index
> > 03e0b69..8ff49de 100644
> > --- a/kernel/watchdog.c
> > +++ b/kernel/watchdog.c
> > @@ -160,7 +160,7 @@ static void set_sample_period(void)
> >  	 * and hard thresholds) to increment before the
> >  	 * hardlockup detector generates a warning
> >  	 */
> > -	sample_period = get_softlockup_thresh() * ((u64)NSEC_PER_SEC / 5);
> > +	sample_period = get_softlockup_thresh() * ((u64)NSEC_PER_SEC /
> 10);
> >  }
> 
> Hi Kan,
> 
> Will the above patch work for you?

Hi Don & Thomas,

Sorry for the late response. We just finished the tests for all proposed patches.

There are three proposed patches so far.
Patch 1: The patch as above which speed up the hrtimer.
Patch 2: Thomas's first proposal.
https://patchwork.kernel.org/patch/9803033/
https://patchwork.kernel.org/patch/9805903/
Patch 3: my original proposal which increase the NMI watchdog timeout by 3X
https://patchwork.kernel.org/patch/9802053/

According to our test, only patch 3 works well.
The other two patches will hang the system eventually.
For patch 1, the system hang after running our test case for ~1 hour.
For patch 2, the system hang in running the overnight test.
There is no error message shown when the system hang. So I don't know the
root cause yet.

BTW: We set 1 to watchdog_thresh when we did the test.
It's believed that can speed up the failure.

Thanks,
Kan