From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753341AbbKCBdq (ORCPT ); Mon, 2 Nov 2015 20:33:46 -0500 Received: from mail-qg0-f53.google.com ([209.85.192.53]:34486 "EHLO mail-qg0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751313AbbKCBdp (ORCPT ); Mon, 2 Nov 2015 20:33:45 -0500 Date: Mon, 2 Nov 2015 20:33:39 -0500 From: Jeff Layton To: Chris Worley Cc: Tejun Heo , linux-kernel@vger.kernel.org, bfields@fieldses.org, Michael Skralivetsky , Trond Myklebust , Shaohua Li Subject: Re: timer code oops when calling mod_delayed_work Message-ID: <20151102203339.7ed8f2bb@synchrony.poochiereds.net> In-Reply-To: <20151102145633.5329f3da@tlielax.poochiereds.net> References: <20151029103113.2f893924@tlielax.poochiereds.net> <20151029135836.02ad9000@synchrony.poochiereds.net> <20151031020012.GH3582@mtj.duckdns.org> <20151031073400.2cf05d77@tlielax.poochiereds.net> <20151031213107.GA23841@mtj.duckdns.org> <20151031175404.3c57a17a@tlielax.poochiereds.net> <20151102145633.5329f3da@tlielax.poochiereds.net> X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.28; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2 Nov 2015 14:56:33 -0500 Jeff Layton wrote: > On Mon, 2 Nov 2015 09:48:41 -1000 > Chris Worley wrote: > > > On Sat, Oct 31, 2015 at 11:54 AM, Jeff Layton wrote: > > > On Sat, 31 Oct 2015 17:31:07 -0400 > > > Tejun Heo wrote: > > ... > > >> > > >> > I have asked Chris and Michael to see if they can bisect it down, but > > >> > it may be a bit before they can get that done. Any insight you might > > >> > have in the meantime would helpful. > > >> > > >> Yeah, I'd love to find out how reproducible the issue is. If the > > >> problem is rarely reproducible, it might make sense to try > > >> instrumentation before trying bisection as it *could* be a latent bug > > >> which has been there all along and bisecting to the commit introducing > > >> the code wouldn't help us too much. > > >> > > > > > > It seems fairly reproducible, at least on v4.3-rc7 kernels: > > > > > > This came about when I asked them to perf test some nfsd patches that I > > > have queued up. I patched a Fedora 4.3-rc7 kernel and wanted to see > > > what the perf delta was (with NFSv3, fwiw): > > > > > > Patched kernels here: http://koji.fedoraproject.org/koji/taskinfo?taskID=11598089 > > > > > > Unpatched kernels here: http://koji.fedoraproject.org/koji/buildinfo?buildID=694377 > > > > > > Michael was using the SPEC SFS VDI workload to test, and was able to > > > get the same panic on both kernels. So it does seem to be reproducible. > > > It might even be possible to tune the VM to make the shrinker fire more > > > often, which may help tickle this more. > > > > > > In any case, I've asked them to try something v4.2-ish and see if it's > > > reproducible there, and then try v4.1 if it is. I figure anything > > > earlier is probably not worth testing if it still fails on v4.1. If it > > > turns out not to be reproducible on those earlier kernels then we can > > > bisect from there to track it down. > > > > The trick seems to be the NFS thread count: I initially though this > > was SFS/VDI specific, but when I ratcheted up the thread count to what > > Michael was using, 256 threads oopses on fio (throughput) benchmarks > > too. > > > > In bisecting kernels, it appeared between 4.2.3-200 and 4.2.5-200 (all > > the 4.2.4 kernels were bad). > > > > Jeff has a lead on this... > > > > Chris > > Thanks Chris. This commit went in between those kernels: > > commit a7c571f2e3ff9243ba41c242486f53dbca37d864 > Author: Shaohua Li > Date: Wed Sep 30 09:05:30 2015 -0700 > > workqueue: make sure delayed work run in local cpu > > commit 874bbfe600a660cba9c776b3957b1ce393151b76 upstream. > > > ...and I suspect that it's the culprit. That commit causes this code to > always use add_timer_on, which seems to have different semantics from > add_timer. > > I'm going to build a v4.2.5 kernel with that patch reverted to confirm > it, but it seems likely... > (dropping Lai's address, and adding Shaohua Li's) Ok, I built a kernel with that patch reverted and that seems to fix the problem. Looking at the patch, I guess the main difference is that we're no longer using add_timer for unbound workqueue tasks. That said, we should have possession of the PENDING bit before calling add_timer_on. So...I'm a little stumped. Any thoughts as to where the race could be? -- Jeff Layton