Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Ross Green <rgkernel@gmail.com>
Cc: "John Stultz" <john.stultz@linaro.org>,
	"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>,
	"Thomas Gleixner" <tglx@linutronix.de>,
	"Peter Zijlstra" <peterz@infradead.org>,
	lkml <linux-kernel@vger.kernel.org>,
	"Ingo Molnar" <mingo@kernel.org>,
	"Lai Jiangshan" <jiangshanlai@gmail.com>,
	dipankar@in.ibm.com, "Andrew Morton" <akpm@linux-foundation.org>,
	"Josh Triplett" <josh@joshtriplett.org>,
	rostedt <rostedt@goodmis.org>,
	"David Howells" <dhowells@redhat.com>,
	"Eric Dumazet" <edumazet@google.com>,
	"Darren Hart" <dvhart@linux.intel.com>,
	"Frédéric Weisbecker" <fweisbec@gmail.com>,
	"Oleg Nesterov" <oleg@redhat.com>,
	"pranith kumar" <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Date: Fri, 19 Feb 2016 22:32:48 -0800	[thread overview]
Message-ID: <20160220063248.GE3522@linux.vnet.ibm.com> (raw)
In-Reply-To: <CANfgCY3JGavgR_Mim8i0g38ikOwRAwZKRqin6u9EX4Q-71oBYw@mail.gmail.com>

On Sat, Feb 20, 2016 at 03:34:30PM +1100, Ross Green wrote:
> On Sat, Feb 20, 2016 at 4:33 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Thu, Feb 18, 2016 at 08:13:18PM -0800, John Stultz wrote:
> >> On Thu, Feb 18, 2016 at 7:56 PM, Ross Green <rgkernel@gmail.com> wrote:
> >> > Well a bonus extra!
> >> > Kept everything running and there was another stall.
> >> > So i have included the demsg output for perusal.
> >> >
> >> > Just to clear things up there is no hotplug involved in this system.
> >> > It is a standard Pandaboard ES Ti4460 two processor system.
> >> > I use this for testing as a generic armv7 processor, plus can keep it
> >> > just running along for testing for a long time. the system has a total
> >> > of 23-25 process running on average. Mainly standard daemons. There is
> >> > certainly no heavy processing going on. I run a series of benchmarks
> >> > that are cpu intensive for the first 20 miinutes after boot and then
> >> > just leave it idle away. checking every so often to see how it has
> >> > gone.
> >> > As mentioned I have observed these stalls going back to 3.17 kernel.
> >> > It will often take up to a week to record such a stall. I will
> >> > typically test every new release kernel, so the -rc? series will get
> >> > around a weeks testing.
> >>
> >> Sorry. Kind of hopping in a bit late here. Is this always happening
> >> with just the pandaboard? Or are you seeing this on different
> >> machines?
> >>
> >> Have you tried enabling CONFIG_DEBUG_TIMEKEEPING just in case
> >> something is going awry there?
> >
> > Excellent point -- timekeeping issues have caused this sort of issue
> > in the past.
> >
> > Ross, on your next test, could you please enable CONFIG_DEBUG_TIMEKEEPING
> > as John suggests?
> >
> >                                                         Thanx, Paul
> >
> As John has suggested have already enabled CONFIG_DEBUG_TIMEKEEPING.
> 
> So far just on 1 day running.
> 
> Sigh...!! Nothing to report as yet, only one day on the clock.
> Its like watching grass grow!

I hear you!  Though I was thinking in terms of watching paint dry...

							Thanx, Paul