From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754300AbdLHQgk (ORCPT ); Fri, 8 Dec 2017 11:36:40 -0500 Received: from mail-wm0-f53.google.com ([74.125.82.53]:33777 "EHLO mail-wm0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753445AbdLHQge (ORCPT ); Fri, 8 Dec 2017 11:36:34 -0500 X-Google-Smtp-Source: AGs4zMbZyIWkr/q/ULYZJvBljYS7DOWauvX+DgG1WdLyawKXtW6BetcGMrUg61gxeQ+pJuGD/dYtbg== Date: Fri, 8 Dec 2017 17:36:28 +0100 From: Daniel Vetter To: Peter Zijlstra Cc: LKML , DRI Development , Intel Graphics Development , Tvrtko Ursulin , Marta Lofstedt , Byungchul Park , Ingo Molnar , Tejun Heo , Kees Cook , Thomas Gleixner , Shaohua Li , Andrew Morton , Jens Axboe , Greg Kroah-Hartman , Jonathan Corbet , Oleg Nesterov , Daniel Vetter Subject: Re: [Intel-gfx] [PATCH] kthread: finer-grained lockdep/cross-release completion Message-ID: <20171208163628.juefyg3x32umj7um@phenom.ffwll.local> Mail-Followup-To: Peter Zijlstra , LKML , DRI Development , Intel Graphics Development , Tvrtko Ursulin , Marta Lofstedt , Byungchul Park , Ingo Molnar , Tejun Heo , Kees Cook , Thomas Gleixner , Shaohua Li , Andrew Morton , Jens Axboe , Greg Kroah-Hartman , Jonathan Corbet , Oleg Nesterov , Daniel Vetter References: <20171207100849.407-1-daniel.vetter@ffwll.ch> <20171207122255.zi5ishny24k66ik7@hirez.programming.kicks-ass.net> <20171207145828.z52kjbrrlcl75m7m@phenom.ffwll.local> <20171207195709.cxcil57boc6czdot@hirez.programming.kicks-ass.net> <20171207205657.wkt6xj3e2opegqeq@phenom.ffwll.local> <20171208101416.hu4uwan3jgcbb6dw@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171208101416.hu4uwan3jgcbb6dw@hirez.programming.kicks-ass.net> X-Operating-System: Linux phenom 4.13.0-1-amd64 User-Agent: NeoMutt/20170609 (1.8.3) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Dec 08, 2017 at 11:14:16AM +0100, Peter Zijlstra wrote: > On Thu, Dec 07, 2017 at 09:56:57PM +0100, Daniel Vetter wrote: > > On Thu, Dec 07, 2017 at 08:57:09PM +0100, Peter Zijlstra wrote: > > > > Is what it says I suppose. Now I don't know enough about that i915 code > > > to say if that breadcrumbs_signal thread can ever trigger a fault or > > > not. I got properly lost in that dma_fence callback maze. > > > > > > You're saying not? > > > > Our own kthread, no. At least a tons of run on our CI with the kthread > > patch applied shut up lockdep splats for good. And since we have all the > > i915 kthreads still with the same lockdep_map even with the patch applied, > > since they are all created in the same function, I think that's pretty > > solid evidence. > > > > [There's also really no reasonable reason for it to fault, but I trust > > automated tools more to check this stuff than my own brain. The test suite > > we're running is fairly nasty and does all kinds of corner case > > thrashing. Note that the dma_fence callbacks can be provideded by any > > other driver (think multi-gpu desktops and stuff), but the contract is > > that they must be able to handle hardirq context. Faulting's definitely > > not on the table.] > > OK, good. Aside: Could/should we take some fake lockdep locks around these callbacks, since not all drivers call them from a hardirq context? Just to validate that everyone follows the contract. I need to ponder proper lockdep annotations for dma_fence anyway, since they're just completions which also have some support for direct hw->hw signalling. > > The problem lockdep seems to complain about is that some random other > > kthread could fault, end up in the i915 fault handler, and get stuck until > > i915_reset_device is done doing what it needs to do. But as long as that > > kthread is in turn not providing a service that i915_reset_device needs, I > > don't see how that can deadlock. And if we have that case (there was > > definitely plenty of that stuff that cross-release uncovered in our code, > > we had to shuffle a bunch of allocations and things out from under > > dev->struct_mutex), then there should be another lock or completion that > > closes the loop again. > > Indeed so. > > > > (also, that comment near need_resched() doesn't make sense to me) > > > > I assume you mean the one in intel_breadcrumbs_signaler(). The hw design > > is somewhat screwed up and depends upon ridiculously low interrupt > > servicing time. We get there by essentially implementing something like > > netdev polled mode, from irq context. Like net polling if the scheduler > > gets pissed at us we stop and dump it all into a kthread. From a latency > > and ops/sec pov a gpu is pretty close to networking sometimes. > > > > [Note: I just have a rough idea what the code is supposed to do, I didn't > > write/review/debug that one.] > > The thing is though; that calling schedule() from an RT thread doesn't > help anything if it goes running instantly again. > > And looking more; that uses the waitqueue code 'creatively' it doesn't > actually have a condition to wait on, so wtf is it doing with a > waitqueue? Yeah that looks fishy. I discussed it with Chris, and the waitqueue stuff is indeed broken. Chris has a patch to address that. The main wakeup logic of the thread is done through wake_up_process(breadcrumb->signaller) directly, so no ordering issue wrt adding to the waitqueue. And then the proper waker/wakee pattern holds: - set tast state - check condition - schedule() Link to Chris' patch: https://www.spinics.net/lists/intel-gfx/msg149891.html Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch