Sandy Bridge GPU hang reproducer...

Sandy Bridge GPU hang reproducer...

[Daniel J Blueman]

Hi Chris et al,

On my Sandy Bridge GPU (8086:0126 rev 09) laptop, I often see hangs
that are correctly recovered and sometimes ones which aren't (causing
X lockup or kernel hard lock), hurting usability.

I'm able to reproduce GPU hangs often with the composite tests in
rendercheck (may need to restart a few times):

$ ./rendercheck -t composite,cacomposite
Begin composite mask test on a8
<command hang, maybe GPU hang too>

I'm on the internal LVDS panel (1600x900), and can produce this with
2.6.39-rc7 and Xorg 1.10.1 (Ubuntu 11.04 + gnome 3). Kernel logs are
captured booting with 'drm.debug=0x04' [1], and i915 error state is at
http://quora.org/2011/i915_error_state.txt.bz2 .

Any joy in reproducing it there?

Thanks,
  Daniel

--- [1]

[drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung
[drm] capturing error event; look for more information in
/debug/dri/0/i915_error_state
[drm:i915_wait_request] *ERROR* i915_wait_request returns -11
(awaiting 797473 at 797462, next 797474)
[drm:drm_crtc_helper_set_mode], [CRTC:3]
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:ironlake_disable_fbc], disabled FBC
[drm:sandybridge_update_wm], FIFO watermarks For pipe A - plane 7, cursor: 6
[drm:ironlake_check_srwm], watermark 1: display plane 13, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 2: display plane 17, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 3: display plane 81, fbc lines 3, cursor 6
[drm:intel_update_fbc],
[drm:ironlake_enable_fbc], enabled fbc on plane 0
[drm:intel_crtc_mode_set], using SSC reference clock of 120 MHz
[drm:intel_crtc_mode_set], Mode for pipe A:
[drm:drm_mode_debug_printmodeline], Modeline 8:"1600x900" 60 114400
1600 1664 1706 2062 900 903 906 924 0x48 0x9
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:intel_pipe_set_base_atomic], Writing base 005E6000 00000000 0 0 6656
[drm:intel_update_fbc],
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:ironlake_enable_fbc], enabled fbc on plane 0
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:sandybridge_update_wm], FIFO watermarks For pipe A - plane 7, cursor: 6
[drm:ironlake_check_srwm], watermark 1: display plane 13, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 2: display plane 17, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 3: display plane 81, fbc lines 3, cursor 6
[drm:drm_crtc_helper_set_mode], [ENCODER:6:LVDS-6] set [MODE:8:1600x900]
[drm:sandybridge_update_wm], FIFO watermarks For pipe A - plane 7, cursor: 6
[drm:ironlake_check_srwm], watermark 1: display plane 13, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 2: display plane 17, fbc lines 3, cursor 6
[drm:ironlake_check_srwm], watermark 3: display plane 81, fbc lines 3, cursor 6
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:gen6_fdi_link_train], FDI_RX_IIR 0x100
[drm:gen6_fdi_link_train], FDI train 1 done.
[drm:gen6_fdi_link_train], FDI_RX_IIR 0x600
[drm:gen6_fdi_link_train], FDI train 2 done.
[drm:gen6_fdi_link_train], FDI train done.
[drm:intel_update_fbc],
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:ironlake_enable_fbc], enabled fbc on plane 0
[drm:ironlake_crtc_dpms], crtc 1/1 dpms off
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_crtc_cursor_set],
[drm:intel_crtc_cursor_set], cursor off
[drm:intel_ironlake_crt_detect_hotplug], trigger hotplug detect cycle:
adpa=0xf40000
[drm:intel_ironlake_crt_detect_hotplug], ironlake hotplug
adpa=0xf40000, result 0
[drm:intel_crt_detect], CRT not detected via hotplug
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:drm_crtc_helper_set_config],
[drm:drm_crtc_helper_set_config], [CRTC:3] [FB:9] #connectors=1 (x y) (0 0)
[drm:drm_crtc_helper_set_config], [CONNECTOR:5:LVDS-1] to [CRTC:3]
[drm:intel_pipe_set_base_atomic], Writing base 00063000 00000000 0 0 6400
[drm:intel_update_fbc],
[drm:intel_update_fbc], framebuffer not tiled, disabling compression
[drm:intel_update_fbc], unsupported config, disabling FBC
[drm:ironlake_disable_fbc], disabled FBC
[drm:intel_wait_for_vblank], vblank wait timed out
[drm:drm_crtc_helper_set_config],
[drm:drm_crtc_helper_set_config], [CRTC:4] [NOFB]
[drm:drm_crtc_helper_set_config], [CONNECTOR:5:LVDS-1] to [CRTC:3]
[drm:drm_crtc_helper_set_config],
[drm:drm_crtc_helper_set_config], [CRTC:3] [FB:9] #connectors=1 (x y) (0 0)
[drm:drm_crtc_helper_set_config], [CONNECTOR:5:LVDS-1] to [CRTC:3]
[drm:drm_crtc_helper_set_config],
[drm:drm_crtc_helper_set_config], [CRTC:3] [FB:9] #connectors=1 (x y) (0 0)
[drm:drm_crtc_helper_set_config], [CONNECTOR:5:LVDS-1] to [CRTC:3]
[drm:intel_ironlake_crt_detect_hotplug], ironlake hotplug
adpa=0xf40000, result 0
[drm:intel_crt_detect], CRT not detected via hotplug
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_dp_aux_ch], dp_aux_ch timeout status 0x5143003e
[drm:ironlake_dp_detect], DPCD: 0000
[drm:intel_ironlake_crt_detect_hotplug], ironlake hotplug
adpa=0xf40000, result 0
[drm:intel_crt_detect], CRT not detected via hotplug
-- 
Daniel J Blueman

Re: Sandy Bridge GPU hang reproducer...

[Chris Wilson]

On Wed, 18 May 2011 12:38:44 +0100, Daniel J Blueman <daniel.blueman@gmail.com> wrote:
> Hi Chris et al,
> 
> On my Sandy Bridge GPU (8086:0126 rev 09) laptop, I often see hangs
> that are correctly recovered and sometimes ones which aren't (causing
> X lockup or kernel hard lock), hurting usability.
> 
> I'm able to reproduce GPU hangs often with the composite tests in
> rendercheck (may need to restart a few times):
> 
> $ ./rendercheck -t composite,cacomposite
> Begin composite mask test on a8
> <command hang, maybe GPU hang too>

Reproducing use rendercheck is unusual, as that is something that we do
run frequently.

Looking through the i915_error_state made me wince (lots of 1x1 copies
over the same pixel...) but it does remind me of

  https://bugzilla.kernel.org/show_bug.cgi?id=27892

for which the workaround is to flush the caches after every op (Option
"DebugFlushCaches" "True").
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: Sandy Bridge GPU hang reproducer...

[Daniel J Blueman]

On 18 May 2011 19:04, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Wed, 18 May 2011 12:38:44 +0100, Daniel J Blueman <daniel.blueman@gmail.com> wrote:
>> Hi Chris et al,
>>
>> On my Sandy Bridge GPU (8086:0126 rev 09) laptop, I often see hangs
>> that are correctly recovered and sometimes ones which aren't (causing
>> X lockup or kernel hard lock), hurting usability.
>>
>> I'm able to reproduce GPU hangs often with the composite tests in
>> rendercheck (may need to restart a few times):
>>
>> $ ./rendercheck -t composite,cacomposite
>> Begin composite mask test on a8
>> <command hang, maybe GPU hang too>
>
> Reproducing use rendercheck is unusual, as that is something that we do
> run frequently.
>
> Looking through the i915_error_state made me wince (lots of 1x1 copies
> over the same pixel...) but it does remind me of
>
>  https://bugzilla.kernel.org/show_bug.cgi?id=27892
>
> for which the workaround is to flush the caches after every op (Option
> "DebugFlushCaches" "True").

After considerable time with 'DebugFlushCaches' enabled, I still hit a
crash. With just 'DebugFlushBatches', it's solid so far (>20 sigma).
Sometimes, I still see the hangcheck timer get hit [1] but nothing
more.

This is quite a heavy-handed workaround and is instrumental to
stability, so it would be really good going to find a more specific
workaround that will be enabled by default.

Thanks,
  Daniel

--- [1]

[   43.015860] [drm:i915_hangcheck_ring_idle] *ERROR* Hangcheck timer
elapsed... blt ring idle [waiting on 8471, at 8471], missed IRQ?
[30413.917749] [drm:i915_hangcheck_ring_idle] *ERROR* Hangcheck timer
elapsed... blt ring idle [waiting on 1390703, at 1390703], missed IRQ?
[33033.624549] [drm:i915_hangcheck_ring_idle] *ERROR* Hangcheck timer
elapsed... blt ring idle [waiting on 4186964, at 4186964], missed IRQ?
-- 
Daniel J Blueman