From: Daniel Vetter <daniel@ffwll.ch>
To: Peter Zijlstra <peterz@infradead.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
DRI Development <dri-devel@lists.freedesktop.org>,
Intel Graphics Development <intel-gfx@lists.freedesktop.org>,
Tvrtko Ursulin <tvrtko.ursulin@intel.com>,
Marta Lofstedt <marta.lofstedt@intel.com>,
Byungchul Park <byungchul.park@lge.com>,
Ingo Molnar <mingo@kernel.org>, Tejun Heo <tj@kernel.org>,
Kees Cook <keescook@chromium.org>,
Thomas Gleixner <tglx@linutronix.de>, Shaohua Li <shli@fb.com>,
Andrew Morton <akpm@linux-foundation.org>,
Jens Axboe <axboe@kernel.dk>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Jonathan Corbet <corbet@lwn.net>, Oleg Nesterov <oleg@redhat.com>,
Daniel Vetter <daniel.vetter@intel.com>
Subject: Re: [Intel-gfx] [PATCH] kthread: finer-grained lockdep/cross-release completion
Date: Fri, 8 Dec 2017 17:36:28 +0100 [thread overview]
Message-ID: <20171208163628.juefyg3x32umj7um@phenom.ffwll.local> (raw)
In-Reply-To: <20171208101416.hu4uwan3jgcbb6dw@hirez.programming.kicks-ass.net>
On Fri, Dec 08, 2017 at 11:14:16AM +0100, Peter Zijlstra wrote:
> On Thu, Dec 07, 2017 at 09:56:57PM +0100, Daniel Vetter wrote:
> > On Thu, Dec 07, 2017 at 08:57:09PM +0100, Peter Zijlstra wrote:
>
> > > Is what it says I suppose. Now I don't know enough about that i915 code
> > > to say if that breadcrumbs_signal thread can ever trigger a fault or
> > > not. I got properly lost in that dma_fence callback maze.
> > >
> > > You're saying not?
> >
> > Our own kthread, no. At least a tons of run on our CI with the kthread
> > patch applied shut up lockdep splats for good. And since we have all the
> > i915 kthreads still with the same lockdep_map even with the patch applied,
> > since they are all created in the same function, I think that's pretty
> > solid evidence.
> >
> > [There's also really no reasonable reason for it to fault, but I trust
> > automated tools more to check this stuff than my own brain. The test suite
> > we're running is fairly nasty and does all kinds of corner case
> > thrashing. Note that the dma_fence callbacks can be provideded by any
> > other driver (think multi-gpu desktops and stuff), but the contract is
> > that they must be able to handle hardirq context. Faulting's definitely
> > not on the table.]
>
> OK, good.
Aside: Could/should we take some fake lockdep locks around these
callbacks, since not all drivers call them from a hardirq context? Just to
validate that everyone follows the contract.
I need to ponder proper lockdep annotations for dma_fence anyway, since
they're just completions which also have some support for direct hw->hw
signalling.
> > The problem lockdep seems to complain about is that some random other
> > kthread could fault, end up in the i915 fault handler, and get stuck until
> > i915_reset_device is done doing what it needs to do. But as long as that
> > kthread is in turn not providing a service that i915_reset_device needs, I
> > don't see how that can deadlock. And if we have that case (there was
> > definitely plenty of that stuff that cross-release uncovered in our code,
> > we had to shuffle a bunch of allocations and things out from under
> > dev->struct_mutex), then there should be another lock or completion that
> > closes the loop again.
>
> Indeed so.
>
> > > (also, that comment near need_resched() doesn't make sense to me)
> >
> > I assume you mean the one in intel_breadcrumbs_signaler(). The hw design
> > is somewhat screwed up and depends upon ridiculously low interrupt
> > servicing time. We get there by essentially implementing something like
> > netdev polled mode, from irq context. Like net polling if the scheduler
> > gets pissed at us we stop and dump it all into a kthread. From a latency
> > and ops/sec pov a gpu is pretty close to networking sometimes.
> >
> > [Note: I just have a rough idea what the code is supposed to do, I didn't
> > write/review/debug that one.]
>
> The thing is though; that calling schedule() from an RT thread doesn't
> help anything if it goes running instantly again.
>
> And looking more; that uses the waitqueue code 'creatively' it doesn't
> actually have a condition to wait on, so wtf is it doing with a
> waitqueue?
Yeah that looks fishy. I discussed it with Chris, and the waitqueue stuff
is indeed broken. Chris has a patch to address that.
The main wakeup logic of the thread is done through
wake_up_process(breadcrumb->signaller) directly, so no ordering issue wrt
adding to the waitqueue. And then the proper waker/wakee pattern holds:
- set tast state
- check condition
- schedule()
Link to Chris' patch:
https://www.spinics.net/lists/intel-gfx/msg149891.html
Thanks, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
next prev parent reply other threads:[~2017-12-08 16:36 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-07 10:08 [PATCH] kthread: finer-grained lockdep/cross-release completion Daniel Vetter
2017-12-07 12:22 ` Peter Zijlstra
2017-12-07 14:58 ` Daniel Vetter
2017-12-07 19:57 ` Peter Zijlstra
2017-12-07 20:56 ` [Intel-gfx] " Daniel Vetter
2017-12-08 10:14 ` Peter Zijlstra
2017-12-08 16:36 ` Daniel Vetter [this message]
2017-12-08 18:40 ` Peter Zijlstra
2017-12-08 10:54 ` Peter Zijlstra
2017-12-11 9:19 ` Daniel Vetter
2017-12-18 7:11 ` Daniel Vetter
2017-12-18 17:42 ` Linus Torvalds
2017-12-19 9:59 ` Daniel Vetter
2017-12-20 1:14 ` Byungchul Park
2018-03-15 10:31 ` Daniel Vetter
2018-03-15 12:41 ` Peter Zijlstra
2018-03-15 23:26 ` Byungchul Park
2018-10-18 8:56 ` Daniel Vetter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171208163628.juefyg3x32umj7um@phenom.ffwll.local \
--to=daniel@ffwll.ch \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=byungchul.park@lge.com \
--cc=corbet@lwn.net \
--cc=daniel.vetter@intel.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=gregkh@linuxfoundation.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=marta.lofstedt@intel.com \
--cc=mingo@kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=shli@fb.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=tvrtko.ursulin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).