From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751940AbbJaVyL (ORCPT ); Sat, 31 Oct 2015 17:54:11 -0400 Received: from mail-qg0-f54.google.com ([209.85.192.54]:36111 "EHLO mail-qg0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751845AbbJaVyH (ORCPT ); Sat, 31 Oct 2015 17:54:07 -0400 Date: Sat, 31 Oct 2015 17:54:04 -0400 From: Jeff Layton To: Tejun Heo Cc: linux-kernel@vger.kernel.org, bfields@fieldses.org, Michael Skralivetsky , Chris Worley , Trond Myklebust , Lai Jiangshan Subject: Re: timer code oops when calling mod_delayed_work Message-ID: <20151031175404.3c57a17a@tlielax.poochiereds.net> In-Reply-To: <20151031213107.GA23841@mtj.duckdns.org> References: <20151029103113.2f893924@tlielax.poochiereds.net> <20151029135836.02ad9000@synchrony.poochiereds.net> <20151031020012.GH3582@mtj.duckdns.org> <20151031073400.2cf05d77@tlielax.poochiereds.net> <20151031213107.GA23841@mtj.duckdns.org> X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.28; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, 31 Oct 2015 17:31:07 -0400 Tejun Heo wrote: > Hello, Jeff. > > On Sat, Oct 31, 2015 at 07:34:00AM -0400, Jeff Layton wrote: > > > Heh, this one is tricky. Yeah, try_to_grab_pending() missing PENDING > > > would explain the failure but I can't see how it'd leak at the moment. > > > > Thanks Tejun. Yeah, I realized that after sending the response above. > > > > If you successfully delete the timer the timer then the PENDING bit > > should already be set. Might be worth throwing in something like this, > > just before the return 1: > > > > WARN_ON(!test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) > > > > ...but I doubt it would fire. I think it's likely that the bug is > > elsewhere. > > I doubt that'd trigger either but it'd still be a good idea to throw > that in just in case. > > > The other thing is that we've had this code in place for a couple of > > years now, and this is the first time I've seen an oops like this. I > > suspect that this may be a recent regression, but I don't know that for > > sure. > > I can't think of any recent changes which could affect this. This > part of code has remained unchanged for years at least from the > workqueue side. > > > I have asked Chris and Michael to see if they can bisect it down, but > > it may be a bit before they can get that done. Any insight you might > > have in the meantime would helpful. > > Yeah, I'd love to find out how reproducible the issue is. If the > problem is rarely reproducible, it might make sense to try > instrumentation before trying bisection as it *could* be a latent bug > which has been there all along and bisecting to the commit introducing > the code wouldn't help us too much. > It seems fairly reproducible, at least on v4.3-rc7 kernels: This came about when I asked them to perf test some nfsd patches that I have queued up. I patched a Fedora 4.3-rc7 kernel and wanted to see what the perf delta was (with NFSv3, fwiw): Patched kernels here: http://koji.fedoraproject.org/koji/taskinfo?taskID=11598089 Unpatched kernels here: http://koji.fedoraproject.org/koji/buildinfo?buildID=694377 Michael was using the SPEC SFS VDI workload to test, and was able to get the same panic on both kernels. So it does seem to be reproducible. It might even be possible to tune the VM to make the shrinker fire more often, which may help tickle this more. In any case, I've asked them to try something v4.2-ish and see if it's reproducible there, and then try v4.1 if it is. I figure anything earlier is probably not worth testing if it still fails on v4.1. If it turns out not to be reproducible on those earlier kernels then we can bisect from there to track it down. Thanks again! -- Jeff Layton