[PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Chris Wilson]

The signal threads may be running concurrently with the GPU reset. The
completion from the GPU run asynchronous with the reset and two threads
may see different snapshots of the state, and the signaler may mark a
request as complete as we try to reset it. We don't tolerate 2 different
views of the same state and complain if we try to mark a request as
failed if it is already complete. Disable the signal threads during
reset to prevent this conflict (even though the conflict implies that
the state we resetting to is invalid, we have already made our
decision!).

References: https://bugs.freedesktop.org/show_bug.cgi?id=99671
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c          | 16 +++++++++++++++-
 drivers/gpu/drm/i915/intel_breadcrumbs.c |  3 +++
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 48922ff454e6..95582295b219 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -35,6 +35,7 @@
 #include "intel_frontbuffer.h"
 #include "intel_mocs.h"
 #include <linux/dma-fence-array.h>
+#include <linux/kthread.h>
 #include <linux/reservation.h>
 #include <linux/shmem_fs.h>
 #include <linux/slab.h>
@@ -2643,6 +2644,17 @@ int i915_gem_reset_prepare(struct drm_i915_private *dev_priv)
 	for_each_engine(engine, dev_priv, id) {
 		struct drm_i915_gem_request *request;
 
+		/* Prevent the signaler thread from updating the request
+		 * state (by calling dma_fence_signal) as we are processing
+		 * the reset. The write from the GPU of the seqno is
+		 * asynchronous and the signaler thread may see a different
+		 * value to us and declare the request complete, even though
+		 * the reset routine have picked that request as the active
+		 * (incomplete) request. This conflict is not handled
+		 * gracefully!
+		 */
+		kthread_park(engine->breadcrumbs.signaler);
+
 		/* Prevent request submission to the hardware until we have
 		 * completed the reset in i915_gem_reset_finish(). If a request
 		 * is completed by one engine, it may then queue a request
@@ -2796,8 +2808,10 @@ void i915_gem_reset_finish(struct drm_i915_private *dev_priv)
 
 	lockdep_assert_held(&dev_priv->drm.struct_mutex);
 
-	for_each_engine(engine, dev_priv, id)
+	for_each_engine(engine, dev_priv, id) {
 		tasklet_enable(&engine->irq_tasklet);
+		kthread_unpark(engine->breadcrumbs.signaler);
+	}
 }
 
 static void nop_submit_request(struct drm_i915_gem_request *request)
diff --git a/drivers/gpu/drm/i915/intel_breadcrumbs.c b/drivers/gpu/drm/i915/intel_breadcrumbs.c
index 9fd002bcebb6..f5e05343110a 100644
--- a/drivers/gpu/drm/i915/intel_breadcrumbs.c
+++ b/drivers/gpu/drm/i915/intel_breadcrumbs.c
@@ -490,6 +490,9 @@ static int intel_breadcrumbs_signaler(void *arg)
 				break;
 
 			schedule();
+
+			if (kthread_should_park())
+				kthread_parkme();
 		}
 	} while (1);
 	__set_current_state(TASK_RUNNING);
-- 
2.11.0

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.BAT: success for drm/i915: Park the breadcrumbs signaler across a GPU reset

[Patchwork]

== Series Details ==

Series: drm/i915: Park the breadcrumbs signaler across a GPU reset
URL   : https://patchwork.freedesktop.org/series/19510/
State : success

== Summary ==

Series 19510v1 drm/i915: Park the breadcrumbs signaler across a GPU reset
https://patchwork.freedesktop.org/api/1.0/series/19510/revisions/1/mbox/

fi-bdw-5557u     total:252  pass:241  dwarn:0   dfail:0   fail:0   skip:11 
fi-bsw-n3050     total:252  pass:213  dwarn:0   dfail:0   fail:0   skip:39 
fi-bxt-j4205     total:252  pass:233  dwarn:0   dfail:0   fail:0   skip:19 
fi-bxt-t5700     total:83   pass:70   dwarn:0   dfail:0   fail:0   skip:12 
fi-byt-j1900     total:252  pass:225  dwarn:0   dfail:0   fail:0   skip:27 
fi-byt-n2820     total:252  pass:221  dwarn:0   dfail:0   fail:0   skip:31 
fi-hsw-4770      total:252  pass:236  dwarn:0   dfail:0   fail:0   skip:16 
fi-hsw-4770r     total:252  pass:236  dwarn:0   dfail:0   fail:0   skip:16 
fi-ilk-650       total:252  pass:202  dwarn:0   dfail:0   fail:0   skip:50 
fi-ivb-3520m     total:252  pass:234  dwarn:0   dfail:0   fail:0   skip:18 
fi-ivb-3770      total:252  pass:234  dwarn:0   dfail:0   fail:0   skip:18 
fi-kbl-7500u     total:252  pass:234  dwarn:0   dfail:0   fail:0   skip:18 
fi-skl-6260u     total:252  pass:242  dwarn:0   dfail:0   fail:0   skip:10 
fi-skl-6700hq    total:252  pass:235  dwarn:0   dfail:0   fail:0   skip:17 
fi-skl-6700k     total:252  pass:230  dwarn:4   dfail:0   fail:0   skip:18 
fi-skl-6770hq    total:252  pass:242  dwarn:0   dfail:0   fail:0   skip:10 
fi-snb-2520m     total:252  pass:224  dwarn:0   dfail:0   fail:0   skip:28 
fi-snb-2600      total:252  pass:223  dwarn:0   dfail:0   fail:0   skip:29 

4dbd7c0fbb78579ff491ef1184f78087055c5aa5 drm-tip: 2017y-02m-10d-21h-45m-14s UTC integration manifest
8e1a828 drm/i915: Park the breadcrumbs signaler across a GPU reset

== Logs ==

For more details see: https://intel-gfx-ci.01.org/CI/Patchwork_3778/
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Chris Wilson]

On Sun, Feb 12, 2017 at 12:44:49AM +0000, Chris Wilson wrote:
> The signal threads may be running concurrently with the GPU reset. The
> completion from the GPU run asynchronous with the reset and two threads
> may see different snapshots of the state, and the signaler may mark a
> request as complete as we try to reset it. We don't tolerate 2 different
> views of the same state and complain if we try to mark a request as
> failed if it is already complete. Disable the signal threads during
> reset to prevent this conflict (even though the conflict implies that
> the state we resetting to is invalid, we have already made our
> decision!).
> 
> References: https://bugs.freedesktop.org/show_bug.cgi?id=99671
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Mika Kuoppala <mika.kuoppala@intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Also
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=99733
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Mika Kuoppala]

Chris Wilson <chris@chris-wilson.co.uk> writes:

> The signal threads may be running concurrently with the GPU reset. The
> completion from the GPU run asynchronous with the reset and two threads
> may see different snapshots of the state, and the signaler may mark a
> request as complete as we try to reset it. We don't tolerate 2 different
> views of the same state and complain if we try to mark a request as
> failed if it is already complete. Disable the signal threads during
> reset to prevent this conflict (even though the conflict implies that
> the state we resetting to is invalid, we have already made our
> decision!).
>
> References: https://bugs.freedesktop.org/show_bug.cgi?id=99671
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Mika Kuoppala <mika.kuoppala@intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> ---
>  drivers/gpu/drm/i915/i915_gem.c          | 16 +++++++++++++++-
>  drivers/gpu/drm/i915/intel_breadcrumbs.c |  3 +++
>  2 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index 48922ff454e6..95582295b219 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -35,6 +35,7 @@
>  #include "intel_frontbuffer.h"
>  #include "intel_mocs.h"
>  #include <linux/dma-fence-array.h>
> +#include <linux/kthread.h>
>  #include <linux/reservation.h>
>  #include <linux/shmem_fs.h>
>  #include <linux/slab.h>
> @@ -2643,6 +2644,17 @@ int i915_gem_reset_prepare(struct drm_i915_private *dev_priv)
>  	for_each_engine(engine, dev_priv, id) {
>  		struct drm_i915_gem_request *request;
>  
> +		/* Prevent the signaler thread from updating the request
> +		 * state (by calling dma_fence_signal) as we are processing
> +		 * the reset. The write from the GPU of the seqno is
> +		 * asynchronous and the signaler thread may see a different
> +		 * value to us and declare the request complete, even though
> +		 * the reset routine have picked that request as the active
> +		 * (incomplete) request. This conflict is not handled
> +		 * gracefully!
> +		 */
> +		kthread_park(engine->breadcrumbs.signaler);
> +

I was wondering here if it would be best to disable the tasklet first.
But stopping the car before we stop the engine seems wise thing to do.

Another unrelated thing is that we might be better off just to
make find_active_request always seqno coherent and fork a
noncoherent version for error capture.

Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

>  		/* Prevent request submission to the hardware until we have
>  		 * completed the reset in i915_gem_reset_finish(). If a request
>  		 * is completed by one engine, it may then queue a request
> @@ -2796,8 +2808,10 @@ void i915_gem_reset_finish(struct drm_i915_private *dev_priv)
>  
>  	lockdep_assert_held(&dev_priv->drm.struct_mutex);
>  
> -	for_each_engine(engine, dev_priv, id)
> +	for_each_engine(engine, dev_priv, id) {
>  		tasklet_enable(&engine->irq_tasklet);
> +		kthread_unpark(engine->breadcrumbs.signaler);
> +	}
>  }
>  
>  static void nop_submit_request(struct drm_i915_gem_request *request)
> diff --git a/drivers/gpu/drm/i915/intel_breadcrumbs.c b/drivers/gpu/drm/i915/intel_breadcrumbs.c
> index 9fd002bcebb6..f5e05343110a 100644
> --- a/drivers/gpu/drm/i915/intel_breadcrumbs.c
> +++ b/drivers/gpu/drm/i915/intel_breadcrumbs.c
> @@ -490,6 +490,9 @@ static int intel_breadcrumbs_signaler(void *arg)
>  				break;
>  
>  			schedule();
> +
> +			if (kthread_should_park())
> +				kthread_parkme();
>  		}
>  	} while (1);
>  	__set_current_state(TASK_RUNNING);
> -- 
> 2.11.0
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Chris Wilson]

On Mon, Feb 13, 2017 at 11:56:46AM +0200, Mika Kuoppala wrote:
> Chris Wilson <chris@chris-wilson.co.uk> writes:
> 
> > The signal threads may be running concurrently with the GPU reset. The
> > completion from the GPU run asynchronous with the reset and two threads
> > may see different snapshots of the state, and the signaler may mark a
> > request as complete as we try to reset it. We don't tolerate 2 different
> > views of the same state and complain if we try to mark a request as
> > failed if it is already complete. Disable the signal threads during
> > reset to prevent this conflict (even though the conflict implies that
> > the state we resetting to is invalid, we have already made our
> > decision!).
> >
> > References: https://bugs.freedesktop.org/show_bug.cgi?id=99671
> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Mika Kuoppala <mika.kuoppala@intel.com>
> > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > ---
> >  drivers/gpu/drm/i915/i915_gem.c          | 16 +++++++++++++++-
> >  drivers/gpu/drm/i915/intel_breadcrumbs.c |  3 +++
> >  2 files changed, 18 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> > index 48922ff454e6..95582295b219 100644
> > --- a/drivers/gpu/drm/i915/i915_gem.c
> > +++ b/drivers/gpu/drm/i915/i915_gem.c
> > @@ -35,6 +35,7 @@
> >  #include "intel_frontbuffer.h"
> >  #include "intel_mocs.h"
> >  #include <linux/dma-fence-array.h>
> > +#include <linux/kthread.h>
> >  #include <linux/reservation.h>
> >  #include <linux/shmem_fs.h>
> >  #include <linux/slab.h>
> > @@ -2643,6 +2644,17 @@ int i915_gem_reset_prepare(struct drm_i915_private *dev_priv)
> >  	for_each_engine(engine, dev_priv, id) {
> >  		struct drm_i915_gem_request *request;
> >  
> > +		/* Prevent the signaler thread from updating the request
> > +		 * state (by calling dma_fence_signal) as we are processing
> > +		 * the reset. The write from the GPU of the seqno is
> > +		 * asynchronous and the signaler thread may see a different
> > +		 * value to us and declare the request complete, even though
> > +		 * the reset routine have picked that request as the active
> > +		 * (incomplete) request. This conflict is not handled
> > +		 * gracefully!
> > +		 */
> > +		kthread_park(engine->breadcrumbs.signaler);
> > +
> 
> I was wondering here if it would be best to disable the tasklet first.
> But stopping the car before we stop the engine seems wise thing to do.

Note that we would have to do two passes over the engines to stop the
signalers, and even then we still have other sources (other devices)
that can cause tasklets to be invoked. There is not a foolproof answer
here, they each serve a different purpose and can be disabled
orthogonally.
 
> Another unrelated thing is that we might be better off just to
> make find_active_request always seqno coherent and fork a
> noncoherent version for error capture.

It's slightly more than that. In the worst case irq_barrier is not
sufficient, and a second call to active_request may return a different
victim - which again raises some awkwards issues in how we assign blame.

At least we now do the irq_barrier hammer once at the start in reset_prepare,
so we should be better, but I'm wondering if we want to store the
request from prepare and then double check in the actual reset.

Definitely a follow on patch to test the waters.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Mika Kuoppala]

Chris Wilson <chris@chris-wilson.co.uk> writes:

> On Mon, Feb 13, 2017 at 11:56:46AM +0200, Mika Kuoppala wrote:
>> Chris Wilson <chris@chris-wilson.co.uk> writes:
>> 
>> > The signal threads may be running concurrently with the GPU reset. The
>> > completion from the GPU run asynchronous with the reset and two threads
>> > may see different snapshots of the state, and the signaler may mark a
>> > request as complete as we try to reset it. We don't tolerate 2 different
>> > views of the same state and complain if we try to mark a request as
>> > failed if it is already complete. Disable the signal threads during
>> > reset to prevent this conflict (even though the conflict implies that
>> > the state we resetting to is invalid, we have already made our
>> > decision!).
>> >
>> > References: https://bugs.freedesktop.org/show_bug.cgi?id=99671
>> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
>> > Cc: Mika Kuoppala <mika.kuoppala@intel.com>
>> > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>> > ---
>> >  drivers/gpu/drm/i915/i915_gem.c          | 16 +++++++++++++++-
>> >  drivers/gpu/drm/i915/intel_breadcrumbs.c |  3 +++
>> >  2 files changed, 18 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
>> > index 48922ff454e6..95582295b219 100644
>> > --- a/drivers/gpu/drm/i915/i915_gem.c
>> > +++ b/drivers/gpu/drm/i915/i915_gem.c
>> > @@ -35,6 +35,7 @@
>> >  #include "intel_frontbuffer.h"
>> >  #include "intel_mocs.h"
>> >  #include <linux/dma-fence-array.h>
>> > +#include <linux/kthread.h>
>> >  #include <linux/reservation.h>
>> >  #include <linux/shmem_fs.h>
>> >  #include <linux/slab.h>
>> > @@ -2643,6 +2644,17 @@ int i915_gem_reset_prepare(struct drm_i915_private *dev_priv)
>> >  	for_each_engine(engine, dev_priv, id) {
>> >  		struct drm_i915_gem_request *request;
>> >  
>> > +		/* Prevent the signaler thread from updating the request
>> > +		 * state (by calling dma_fence_signal) as we are processing
>> > +		 * the reset. The write from the GPU of the seqno is
>> > +		 * asynchronous and the signaler thread may see a different
>> > +		 * value to us and declare the request complete, even though
>> > +		 * the reset routine have picked that request as the active
>> > +		 * (incomplete) request. This conflict is not handled
>> > +		 * gracefully!
>> > +		 */
>> > +		kthread_park(engine->breadcrumbs.signaler);
>> > +
>> 
>> I was wondering here if it would be best to disable the tasklet first.
>> But stopping the car before we stop the engine seems wise thing to do.
>
> Note that we would have to do two passes over the engines to stop the
> signalers, and even then we still have other sources (other devices)
> that can cause tasklets to be invoked. There is not a foolproof answer
> here, they each serve a different purpose and can be disabled
> orthogonally.
>  
>> Another unrelated thing is that we might be better off just to
>> make find_active_request always seqno coherent and fork a
>> noncoherent version for error capture.
>
> It's slightly more than that. In the worst case irq_barrier is not
> sufficient, and a second call to active_request may return a different
> victim - which again raises some awkwards issues in how we assign blame.
>
> At least we now do the irq_barrier hammer once at the start in reset_prepare,
> so we should be better, but I'm wondering if we want to store the
> request from prepare and then double check in the actual reset.
>

#1 store seqno from hangcheck
#2 get mutex for reset
#3 barrier
#4 find_request (only once) 
#5 on prepare path, check the submachinery
against this req and if inconsistent, queue hangcheck
and return from prepare without resetting.

?
-Mika
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Park the breadcrumbs signaler across a GPU reset

[Chris Wilson]

On Mon, Feb 13, 2017 at 12:34:26PM +0200, Mika Kuoppala wrote:
> Chris Wilson <chris@chris-wilson.co.uk> writes:
> > At least we now do the irq_barrier hammer once at the start in reset_prepare,
> > so we should be better, but I'm wondering if we want to store the
> > request from prepare and then double check in the actual reset.
> >
> 
> #1 store seqno from hangcheck
> #2 get mutex for reset
> #3 barrier
> #4 find_request (only once) 
> #5 on prepare path, check the submachinery
> against this req and if inconsistent, queue hangcheck
> and return from prepare without resetting.
> 
> ?

Per-engine, or global voting.

The issue I just found with the find_request once plan is that it does
have to be after the reset / hw is truly idle. And then we should employ
a really big hammer on top. The problem is that for legacy ringbuffer
sumbission, we restart from the point of last retirement and so if our
retirement is inaccurate we may replay one too many requests. Hmm. we
may get corruption either way of course, so perhaps this is not as big
an issue as I thought.

Aborting the reset if we detect just before we do that the GPU recovered
is definitely an improvement over my plan.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx