Re: Crashes with 874bbfe600a6 in 3.18.25

From: Ben Hutchings <ben@decadent.org.uk>
To: Tejun Heo <tj@kernel.org>, Sasha Levin <sasha.levin@oracle.com>
Cc: Jan Kara <jack@suse.cz>, Shaohua Li <shli@fb.com>,
	LKML <linux-kernel@vger.kernel.org>,
	stable@vger.kernel.org, Daniel Bilik <daniel.bilik@neosystem.cz>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: Crashes with 874bbfe600a6 in 3.18.25
Date: Sat, 23 Jan 2016 02:20:23 +0000	[thread overview]
Message-ID: <1453515623.3734.156.camel@decadent.org.uk> (raw)
In-Reply-To: <20160122160903.GH32380@htj.duckdns.org>

[-- Attachment #1: Type: text/plain, Size: 2196 bytes --]

On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> (cc'ing Thomas)
> 
> On Thu, Jan 21, 2016 at 08:10:20PM -0500, Sasha Levin wrote:
> > On 01/21/2016 04:52 AM, Jan Kara wrote:
> > > On Wed 20-01-16 13:39:01, Shaohua Li wrote:
> > > > On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
> > > > > Hello,
> > > > > 
> > > > > a friend of mine started seeing crashes with 3.18.25 kernel - once
> > > > > appropriate load is put on the machine it crashes within minutes. He
> > > > > tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
> > > > > Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
> > > > > sure delayed work run in local cpu" makes the kernel stable again. I'm
> > > > > attaching screenshot of the crash - sadly the initial part is missing but
> > > > > it seems that we crashed when processing timers on otherwise idle CPU. This
> > > > > is a production machine so experimentation is not easy but if we really
> > > > > need more information it may be possible to reproduce the issue again and
> > > > > gather it.
> > > > > 
> > > > > Anyone has idea what is going on? I was looking into the code for a while
> > > > > but so far I have no good explanation.  It would be good to understand the
> > > > > cause instead of just blindly reverting the commit from stable tree...
> > > > 
> > > > Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?
> > > 
> > > That doesn't seem to be included in 3.18-stable although it was CCed to stable.
> > > Sasha?
> > 
> > Looks like it requires more than trivial backport (I think). Tejun?
> 
> The timer migration has changed quite a bit.  Given that we've never
> seen vmstat work crashing in 3.18 era, I wonder whether the right
> thing to do here is reverting 874bbfe600a6 from 3.18 stable?

It's not just 3.18 that has this; 874bbfe600a6 was backported to all
stable branches from 3.10 onward.  Only the 4.2-ckt branch has
22b886dd10180939.

Ben.

-- 
Ben Hutchings
Life is what happens to you while you're busy making other plans.
                                                               - John Lennon

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 811 bytes --]