[PATCH] drm/i915/oa: Disable OA on Haswell

[PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

We observe that the OA architecture is clobbering random memory. Disable
it until this can be resolved.

References: https://bugs.freedesktop.org/show_bug.cgi?id=106379
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: stable@vger.kernel.org
---
 drivers/gpu/drm/i915/i915_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_perf.c b/drivers/gpu/drm/i915/i915_perf.c
index 019bd2d073ad..20187f3bf350 100644
--- a/drivers/gpu/drm/i915/i915_perf.c
+++ b/drivers/gpu/drm/i915/i915_perf.c
@@ -3425,7 +3425,7 @@ static struct ctl_table dev_root[] = {
  */
 void i915_perf_init(struct drm_i915_private *dev_priv)
 {
-	if (IS_HASWELL(dev_priv)) {
+	if (IS_HASWELL(dev_priv) && 0) {
 		dev_priv->perf.oa.ops.is_valid_b_counter_reg =
 			gen7_is_valid_b_counter_addr;
 		dev_priv->perf.oa.ops.is_valid_mux_reg =
-- 
2.17.0

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

My understanding of the virtual memory addressing from the GPU is limited...
But how can the GPU poke at the kernel's allocated data?
I thought we mapped into the GPU's address space only what is allocated 
through gem.

-
Lionel

On 11/05/18 14:56, Chris Wilson wrote:
> We observe that the OA architecture is clobbering random memory. Disable
> it until this can be resolved.
>
> References: https://bugs.freedesktop.org/show_bug.cgi?id=106379
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
> Cc: Matthew Auld <matthew.auld@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Cc: Jani Nikula <jani.nikula@intel.com>
> Cc: stable@vger.kernel.org
> ---
>   drivers/gpu/drm/i915/i915_perf.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/i915/i915_perf.c b/drivers/gpu/drm/i915/i915_perf.c
> index 019bd2d073ad..20187f3bf350 100644
> --- a/drivers/gpu/drm/i915/i915_perf.c
> +++ b/drivers/gpu/drm/i915/i915_perf.c
> @@ -3425,7 +3425,7 @@ static struct ctl_table dev_root[] = {
>    */
>   void i915_perf_init(struct drm_i915_private *dev_priv)
>   {
> -	if (IS_HASWELL(dev_priv)) {
> +	if (IS_HASWELL(dev_priv) && 0) {
>   		dev_priv->perf.oa.ops.is_valid_b_counter_reg =
>   			gen7_is_valid_b_counter_addr;
>   		dev_priv->perf.oa.ops.is_valid_mux_reg =

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 15:14:13)
> My understanding of the virtual memory addressing from the GPU is limited...
> But how can the GPU poke at the kernel's allocated data?
> I thought we mapped into the GPU's address space only what is allocated 
> through gem.

Correct. The HW should only be accessing the pages through the GTT and
the GTT should only contain known pages (or a pointer to the scratch
page). There is maybe a hole where we are freeing the memory before
the HW has finished using it (still writing through stale TLB and
whatnot even though the system has reallocated the pages), but other
than that quite, quite scary. Hence this awooga.
-Chris

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

On 11/05/18 15:18, Chris Wilson wrote:
> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
>> My understanding of the virtual memory addressing from the GPU is limited...
>> But how can the GPU poke at the kernel's allocated data?
>> I thought we mapped into the GPU's address space only what is allocated
>> through gem.
> Correct. The HW should only be accessing the pages through the GTT and
> the GTT should only contain known pages (or a pointer to the scratch
> page). There is maybe a hole where we are freeing the memory before
> the HW has finished using it (still writing through stale TLB and
> whatnot even though the system has reallocated the pages), but other
> than that quite, quite scary. Hence this awooga.
> -Chris
>
Oh right...

So this patch is a backup if you previous one won't fix the issue we see 
on CI?

-
Lionel

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 15:28:24)
> On 11/05/18 15:18, Chris Wilson wrote:
> > Quoting Lionel Landwerlin (2018-05-11 15:14:13)
> >> My understanding of the virtual memory addressing from the GPU is limited...
> >> But how can the GPU poke at the kernel's allocated data?
> >> I thought we mapped into the GPU's address space only what is allocated
> >> through gem.
> > Correct. The HW should only be accessing the pages through the GTT and
> > the GTT should only contain known pages (or a pointer to the scratch
> > page). There is maybe a hole where we are freeing the memory before
> > the HW has finished using it (still writing through stale TLB and
> > whatnot even though the system has reallocated the pages), but other
> > than that quite, quite scary. Hence this awooga.
> >
> Oh right...
> 
> So this patch is a backup if you previous one won't fix the issue we see 
> on CI?

Yes. Try to fix, it we can't, we disable until we can. It may also be
purely coincidental that we've seen this bug a few times around the same
test only on this machine... ;)
-Chris

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

On 11/05/18 15:34, Chris Wilson wrote:
> Quoting Lionel Landwerlin (2018-05-11 15:28:24)
>> On 11/05/18 15:18, Chris Wilson wrote:
>>> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
>>>> My understanding of the virtual memory addressing from the GPU is limited...
>>>> But how can the GPU poke at the kernel's allocated data?
>>>> I thought we mapped into the GPU's address space only what is allocated
>>>> through gem.
>>> Correct. The HW should only be accessing the pages through the GTT and
>>> the GTT should only contain known pages (or a pointer to the scratch
>>> page). There is maybe a hole where we are freeing the memory before
>>> the HW has finished using it (still writing through stale TLB and
>>> whatnot even though the system has reallocated the pages), but other
>>> than that quite, quite scary. Hence this awooga.
>>>
>> Oh right...
>>
>> So this patch is a backup if you previous one won't fix the issue we see
>> on CI?
> Yes. Try to fix, it we can't, we disable until we can. It may also be
> purely coincidental that we've seen this bug a few times around the same
> test only on this machine... ;)
> -Chris
>
Trying on a laptop.

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

On 11/05/18 15:34, Chris Wilson wrote:
> Quoting Lionel Landwerlin (2018-05-11 15:28:24)
>> On 11/05/18 15:18, Chris Wilson wrote:
>>> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
>>>> My understanding of the virtual memory addressing from the GPU is limited...
>>>> But how can the GPU poke at the kernel's allocated data?
>>>> I thought we mapped into the GPU's address space only what is allocated
>>>> through gem.
>>> Correct. The HW should only be accessing the pages through the GTT and
>>> the GTT should only contain known pages (or a pointer to the scratch
>>> page). There is maybe a hole where we are freeing the memory before
>>> the HW has finished using it (still writing through stale TLB and
>>> whatnot even though the system has reallocated the pages), but other
>>> than that quite, quite scary. Hence this awooga.
>>>
>> Oh right...
>>
>> So this patch is a backup if you previous one won't fix the issue we see
>> on CI?
> Yes. Try to fix, it we can't, we disable until we can. It may also be
> purely coincidental that we've seen this bug a few times around the same
> test only on this machine... ;)
> -Chris
>
Trying on a laptop.

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.BAT: success for drm/i915/oa: Disable OA on Haswell

[Patchwork]

== Series Details ==

Series: drm/i915/oa: Disable OA on Haswell
URL   : https://patchwork.freedesktop.org/series/43056/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4168 -> Patchwork_8982 =

== Summary - WARNING ==

  Minor unknown changes coming with Patchwork_8982 need to be verified
  manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_8982, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/43056/revisions/1/mbox/

== Possible new issues ==

  Here are the unknown changes that may have been introduced in Patchwork_8982:

  === IGT changes ===

    ==== Warnings ====

    igt@gem_exec_gttfill@basic:
      fi-pnv-d510:        PASS -> SKIP

    


== Participating hosts (41 -> 36) ==

  Missing    (5): fi-byt-j1900 fi-byt-squawks fi-ctg-p8600 fi-ilk-m540 fi-skl-6700hq 


== Build changes ==

    * Linux: CI_DRM_4168 -> Patchwork_8982

  CI_DRM_4168: 8718b981fec6cff68f2a94ef0aaeaecc2cd6a659 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_4475: 35f08c12aa216d5b62a5b9984b575cee6905098f @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  Patchwork_8982: e327acc3264bcc2c7f88e812238d1d618308f488 @ git://anongit.freedesktop.org/gfx-ci/linux
  piglit_4475: 3ba0657bff4216d1ec7179935590261855f1651e @ git://anongit.freedesktop.org/piglit


== Linux commits ==

e327acc3264b drm/i915/oa: Disable OA on Haswell

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_8982/issues.html
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

On 11/05/18 15:18, Chris Wilson wrote:
> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
>> My understanding of the virtual memory addressing from the GPU is limited...
>> But how can the GPU poke at the kernel's allocated data?
>> I thought we mapped into the GPU's address space only what is allocated
>> through gem.
> Correct. The HW should only be accessing the pages through the GTT and
> the GTT should only contain known pages (or a pointer to the scratch
> page). There is maybe a hole where we are freeing the memory before
> the HW has finished using it (still writing through stale TLB and
> whatnot even though the system has reallocated the pages), but other
> than that quite, quite scary. Hence this awooga.
> -Chris
>

I managed to reproduce a kasan backtrace on the same test.
So it's not just the CI machine.

But I can't even startup a gdm on that machine with drm-tip. So maybe 
there is some much more broken...


i915/perf unpins the object correctly before freeing (at which point it 
could be reused).
Should we ensure i915_vma_destroy() i915/perf maybe?

It almost seems like this is an issue that could arise in other part of 
the driver too.

-
Lionel

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 16:43:02)
> On 11/05/18 15:18, Chris Wilson wrote:
> > Quoting Lionel Landwerlin (2018-05-11 15:14:13)
> >> My understanding of the virtual memory addressing from the GPU is limited...
> >> But how can the GPU poke at the kernel's allocated data?
> >> I thought we mapped into the GPU's address space only what is allocated
> >> through gem.
> > Correct. The HW should only be accessing the pages through the GTT and
> > the GTT should only contain known pages (or a pointer to the scratch
> > page). There is maybe a hole where we are freeing the memory before
> > the HW has finished using it (still writing through stale TLB and
> > whatnot even though the system has reallocated the pages), but other
> > than that quite, quite scary. Hence this awooga.
> > -Chris
> >
> 
> I managed to reproduce a kasan backtrace on the same test.
> So it's not just the CI machine.
> 
> But I can't even startup a gdm on that machine with drm-tip. So maybe 
> there is some much more broken...

Don't leave us in suspense...

> i915/perf unpins the object correctly before freeing (at which point it 
> could be reused).

Sure, but does perf know that the OA unit has stopped writing at that
point... That's not so clear (from my pov).

> Should we ensure i915_vma_destroy() i915/perf maybe?
> 
> It almost seems like this is an issue that could arise in other part of 
> the driver too.

The problem of the HW continuing to access the pages after unbinding is
inherent to the system (and what actually happens if we change PTE in
flight is usually undefined), hence the great care we go to track HW
activity and try not to release pages while it is still using them.
-Chris

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

On 11/05/18 16:51, Chris Wilson wrote:
> Quoting Lionel Landwerlin (2018-05-11 16:43:02)
>> On 11/05/18 15:18, Chris Wilson wrote:
>>> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
>>>> My understanding of the virtual memory addressing from the GPU is limited...
>>>> But how can the GPU poke at the kernel's allocated data?
>>>> I thought we mapped into the GPU's address space only what is allocated
>>>> through gem.
>>> Correct. The HW should only be accessing the pages through the GTT and
>>> the GTT should only contain known pages (or a pointer to the scratch
>>> page). There is maybe a hole where we are freeing the memory before
>>> the HW has finished using it (still writing through stale TLB and
>>> whatnot even though the system has reallocated the pages), but other
>>> than that quite, quite scary. Hence this awooga.
>>> -Chris
>>>
>> I managed to reproduce a kasan backtrace on the same test.
>> So it's not just the CI machine.
>>
>> But I can't even startup a gdm on that machine with drm-tip. So maybe
>> there is some much more broken...
> Don't leave us in suspense...

Your first patch (check that OA is actually disabled) seems to get rid 
of the issue on my machine.
Thanks a lot a for finding that!

Trying to find when HSW when wrong now. Same kernel works just fine on 
my SKL.

>
>> i915/perf unpins the object correctly before freeing (at which point it
>> could be reused).
> Sure, but does perf know that the OA unit has stopped writing at that
> point... That's not so clear (from my pov).

Clearly it wasn't :(

>
>> Should we ensure i915_vma_destroy() i915/perf maybe?
>>
>> It almost seems like this is an issue that could arise in other part of
>> the driver too.
> The problem of the HW continuing to access the pages after unbinding is
> inherent to the system (and what actually happens if we change PTE in
> flight is usually undefined), hence the great care we go to track HW
> activity and try not to release pages while it is still using them.
> -Chris
>

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 16:58:27)
> On 11/05/18 16:51, Chris Wilson wrote:
> > Quoting Lionel Landwerlin (2018-05-11 16:43:02)
> >> On 11/05/18 15:18, Chris Wilson wrote:
> >>> Quoting Lionel Landwerlin (2018-05-11 15:14:13)
> >>>> My understanding of the virtual memory addressing from the GPU is limited...
> >>>> But how can the GPU poke at the kernel's allocated data?
> >>>> I thought we mapped into the GPU's address space only what is allocated
> >>>> through gem.
> >>> Correct. The HW should only be accessing the pages through the GTT and
> >>> the GTT should only contain known pages (or a pointer to the scratch
> >>> page). There is maybe a hole where we are freeing the memory before
> >>> the HW has finished using it (still writing through stale TLB and
> >>> whatnot even though the system has reallocated the pages), but other
> >>> than that quite, quite scary. Hence this awooga.
> >>> -Chris
> >>>
> >> I managed to reproduce a kasan backtrace on the same test.
> >> So it's not just the CI machine.
> >>
> >> But I can't even startup a gdm on that machine with drm-tip. So maybe
> >> there is some much more broken...
> > Don't leave us in suspense...
> 
> Your first patch (check that OA is actually disabled) seems to get rid 
> of the issue on my machine.
> Thanks a lot a for finding that!

Care to add a t-b and we'll close the bug?
-Chris

✓ Fi.CI.IGT: success for drm/i915/oa: Disable OA on Haswell

[Patchwork]

== Series Details ==

Series: drm/i915/oa: Disable OA on Haswell
URL   : https://patchwork.freedesktop.org/series/43056/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4168_full -> Patchwork_8982_full =

== Summary - WARNING ==

  Minor unknown changes coming with Patchwork_8982_full need to be verified
  manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_8982_full, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/43056/revisions/1/mbox/

== Possible new issues ==

  Here are the unknown changes that may have been introduced in Patchwork_8982_full:

  === IGT changes ===

    ==== Warnings ====

    igt@gem_mocs_settings@mocs-rc6-bsd1:
      shard-kbl:          PASS -> SKIP

    igt@gem_mocs_settings@mocs-rc6-dirty-render:
      shard-kbl:          SKIP -> PASS

    igt@kms_pipe_crc_basic@read-crc-pipe-c-frame-sequence:
      shard-apl:          SKIP -> PASS +2

    igt@perf@blocking:
      shard-hsw:          PASS -> SKIP +25

    
== Known issues ==

  Here are the changes found in Patchwork_8982_full that come from known issues:

  === IGT changes ===

    ==== Issues hit ====

    igt@kms_flip@flip-vs-expired-vblank:
      shard-glk:          PASS -> FAIL (fdo#105707)

    igt@kms_setmode@basic:
      shard-kbl:          PASS -> FAIL (fdo#99912)

    
    ==== Possible fixes ====

    igt@gem_eio@in-flight-10ms:
      shard-glk:          FAIL (fdo#105957) -> PASS

    igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible:
      shard-kbl:          FAIL (fdo#100368) -> PASS

    igt@kms_flip@flip-vs-expired-vblank-interruptible:
      shard-glk:          FAIL (fdo#102887) -> PASS

    igt@kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-msflip-blt:
      shard-apl:          FAIL (fdo#104724, fdo#103167) -> PASS

    
  fdo#100368 https://bugs.freedesktop.org/show_bug.cgi?id=100368
  fdo#102887 https://bugs.freedesktop.org/show_bug.cgi?id=102887
  fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
  fdo#104724 https://bugs.freedesktop.org/show_bug.cgi?id=104724
  fdo#105707 https://bugs.freedesktop.org/show_bug.cgi?id=105707
  fdo#105957 https://bugs.freedesktop.org/show_bug.cgi?id=105957
  fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912


== Participating hosts (5 -> 5) ==

  No changes in participating hosts


== Build changes ==

    * Linux: CI_DRM_4168 -> Patchwork_8982

  CI_DRM_4168: 8718b981fec6cff68f2a94ef0aaeaecc2cd6a659 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_4475: 35f08c12aa216d5b62a5b9984b575cee6905098f @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  Patchwork_8982: e327acc3264bcc2c7f88e812238d1d618308f488 @ git://anongit.freedesktop.org/gfx-ci/linux
  piglit_4475: 3ba0657bff4216d1ec7179935590261855f1651e @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_8982/shards.html
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Lionel Landwerlin]

[-- Attachment #1.1: Type: text/plain, Size: 258 bytes --]

On 11/05/18 16:51, Chris Wilson wrote:
>> But I can't even startup a gdm on that machine with drm-tip. So maybe
>> there is some much more broken...
> Don't leave us in suspense...
>
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890614


Not our bug :)


[-- Attachment #1.2: Type: text/html, Size: 859 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 18:41:28)
> On 11/05/18 16:51, Chris Wilson wrote:
> 
>         But I can't even startup a gdm on that machine with drm-tip. So maybe
>         there is some much more broken...
> 
>     Don't leave us in suspense...
> 
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890614
> 
> 
> Not our bug :)

You would not believe how relieved I am that someone else has bugs in
their code.
-Chris

Re: [PATCH] drm/i915/oa: Disable OA on Haswell

[Chris Wilson]

Quoting Lionel Landwerlin (2018-05-11 16:43:02)
> On 11/05/18 15:18, Chris Wilson wrote:
> > Quoting Lionel Landwerlin (2018-05-11 15:14:13)
> >> My understanding of the virtual memory addressing from the GPU is limited...
> >> But how can the GPU poke at the kernel's allocated data?
> >> I thought we mapped into the GPU's address space only what is allocated
> >> through gem.
> > Correct. The HW should only be accessing the pages through the GTT and
> > the GTT should only contain known pages (or a pointer to the scratch
> > page). There is maybe a hole where we are freeing the memory before
> > the HW has finished using it (still writing through stale TLB and
> > whatnot even though the system has reallocated the pages), but other
> > than that quite, quite scary. Hence this awooga.
> >
> 
> I managed to reproduce a kasan backtrace on the same test.
> So it's not just the CI machine.

For the record, CI also seems much happier with the wait for OACONTROL
before unpinning, so no need to panic!
-Chris