Re: timer code oops when calling mod_delayed_work

From: Jeff Layton <jlayton@poochiereds.net>
To: Chris Worley <chris.worley@primarydata.com>
Cc: Tejun Heo <tj@kernel.org>,
	linux-kernel@vger.kernel.org, bfields@fieldses.org,
	Michael Skralivetsky <michael.skralivetsky@primarydata.com>,
	Trond Myklebust <trond.myklebust@primarydata.com>,
	Lai Jiangshan <laijs@cn.fujitsu.com>
Subject: Re: timer code oops when calling mod_delayed_work
Date: Mon, 2 Nov 2015 14:56:33 -0500	[thread overview]
Message-ID: <20151102145633.5329f3da@tlielax.poochiereds.net> (raw)
In-Reply-To: <CABz_YQ7fKYYL7jbp+XfFLHG61ZhxPUA8cAq92Vn2vkKzEtAR3Q@mail.gmail.com>

On Mon, 2 Nov 2015 09:48:41 -1000
Chris Worley <chris.worley@primarydata.com> wrote:

> On Sat, Oct 31, 2015 at 11:54 AM, Jeff Layton <jlayton@poochiereds.net> wrote:
> > On Sat, 31 Oct 2015 17:31:07 -0400
> > Tejun Heo <tj@kernel.org> wrote:
> ...
> >>
> >> > I have asked Chris and Michael to see if they can bisect it down, but
> >> > it may be a bit before they can get that done. Any insight you might
> >> > have in the meantime would helpful.
> >>
> >> Yeah, I'd love to find out how reproducible the issue is.  If the
> >> problem is rarely reproducible, it might make sense to try
> >> instrumentation before trying bisection as it *could* be a latent bug
> >> which has been there all along and bisecting to the commit introducing
> >> the code wouldn't help us too much.
> >>
> >
> > It seems fairly reproducible, at least on v4.3-rc7 kernels:
> >
> > This came about when I asked them to perf test some nfsd patches that I
> > have queued up. I patched a Fedora 4.3-rc7 kernel and wanted to see
> > what the perf delta was (with NFSv3, fwiw):
> >
> >     Patched kernels here: http://koji.fedoraproject.org/koji/taskinfo?taskID=11598089
> >
> >     Unpatched kernels here: http://koji.fedoraproject.org/koji/buildinfo?buildID=694377
> >
> > Michael was using the SPEC SFS VDI workload to test, and was able to
> > get the same panic on both kernels. So it does seem to be reproducible.
> > It might even be possible to tune the VM to make the shrinker fire more
> > often, which may help tickle this more.
> >
> > In any case, I've asked them to try something v4.2-ish and see if it's
> > reproducible there, and then try v4.1 if it is. I figure anything
> > earlier is probably not worth testing if it still fails on v4.1. If it
> > turns out not to be reproducible on those earlier kernels then we can
> > bisect from there to track it down.
> 
> The trick seems to be the NFS thread count: I initially though this
> was SFS/VDI specific, but when I ratcheted up the thread count to what
> Michael was using, 256 threads oopses on fio (throughput) benchmarks
> too.
> 
> In bisecting kernels, it appeared between 4.2.3-200 and 4.2.5-200 (all
> the 4.2.4 kernels were bad).
> 
> Jeff has a lead on this...
> 
> Chris

Thanks Chris. This commit went in between those kernels:

commit a7c571f2e3ff9243ba41c242486f53dbca37d864
Author: Shaohua Li <shli@fb.com>
Date:   Wed Sep 30 09:05:30 2015 -0700

    workqueue: make sure delayed work run in local cpu

    commit 874bbfe600a660cba9c776b3957b1ce393151b76 upstream.

...and I suspect that it's the culprit. That commit causes this code to
always use add_timer_on, which seems to have different semantics from
add_timer.

I'm going to build a v4.2.5 kernel with that patch reverted to confirm
it, but it seems likely...

-- 
Jeff Layton <jlayton@poochiereds.net>