| Bug ID | 102322 |
|---|---|
| Summary | System crashes after "[drm] IP block:gmc_v8_0 is hung!&q= uot; / [drm] IP block:sdma_v3_0 is hung! |
| Product | DRI |
| Version | DRI git |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | critical |
| Priority | medium |
| Component | DRM/AMDgpu |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | jb5sgc1n.nya@20mm.eu |
I consistently experience complete system crashes when browsin= g web pages using firefox for about 30 minutes, with the following dmesg output from the amdg= pu driver: [ 2330.720711] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeou= t, last signaled seq=3D40778, last emitted seq=3D40780 [ 2330.720768] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma1 timeou= t, last signaled seq=3D31305, last emitted seq=3D31306 [ 2330.720771] [drm] IP block:gmc_v8_0 is hung! [ 2330.720774] [drm] IP block:gmc_v8_0 is hung! [ 2330.720775] [drm] IP block:sdma_v3_0 is hung! [ 2330.720778] [drm] IP block:sdma_v3_0 is hung! (Above cited messages are the last to make it to a network-filesystem by running "dmesg -w" before the system stops to do anything.) I am running a kernel compiled from https://cgit.freedesktop.org/~agd5f/linux/log/?h=3Damd-staging-drm= -next as of "commit 94097b0f7f1bfa54b3b1f8b0d74bbd271a0564e4" (so the very la= test as of today). My GPU is an RX 460. Notice that this bug may be the same symptom as reported in https://bugs.freedesktop.org/show_bug.c= gi?id=3D98874 However, the system crashes for me occur usually while vertically scrolling through some (ordinary) web page.
Sadly, not only did this bug not attract any attention, it als= o still occurs, and seemingly even more frequent than before, on current bleeding-edge kern= els from amd-staging-drm-next, and also with the now current Firefox 57 and the= now current versions of Xorg, Mesa etc. from Arch Linux.
Just to mention this once again: These system crashes still oc= cur, and way too frequently to consider the amdgpu driver stable enough for professional use. Sample dmesg output from today: Feb 24 18:26:55 [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, last signaled seq=3D5430589, last emitted seq=3D5430591 Feb 24 18:26:55 [drm] IP block:gmc_v8_0 is hung! Feb 24 18:26:55 [drm] IP block:gfx_v8_0 is hung! Feb 24 18:27:02 [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeo= ut, last signaled seq=3D185928, last emitted seq=3D185930 Feb 24 18:27:02 [drm] IP block:gmc_v8_0 is hung! Feb 24 18:27:02 [drm] IP block:gfx_v8_0 is hung! Feb 24 18:27:05 [drm:amdgpu_dm_atomic_check [amdgpu]] *ERROR* [CRTC:43:crtc= -0] hw_done or flip_done timed out
Just for the record, others have reported similar symptoms - h= ere is a recent example: https://bugs.freedesktop.org/show_bug.= cgi?id=3D106666
I was asked in https://www.p= horonix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux= /1027705-amdgpu-on-linux-4-18-to-offer-greater-vega-power-savings-displaypo= rt-1-4-fixes?p=3D1027933#post1027933 to mention here that I have experienced this kind of bug only when using the "new" display code (amdgpu.dc=3D1). I cannot strictly rule out that it could also happen with dc=3D0, since I h= ave tried dc=3D0 only for short periods occasionally, but during those periods = I did not see this kind of crash.
Just for the record: To rule out my personally compiled kernel= s are somehow "more buggy than what others compile", I tried the current Arch-L= inux-supplied Linux 4.17.2-1-ARCH kernel. Survives about 5 minutes of Firefox-browsing between crashes with: Jun 20 00:01:11 ryzen kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ri= ng sdma0 timeout, last signaled seq=3D1895, last em> Jun 20 00:01:11 ryzen kernel: [drm] IP block:gmc_v8_0 is hung! (4.13.* did at least survive days.)
Verify you are using latest AMD firmware and up to date MESA/L= LVM Firmware here (amdgpu folder) - https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linu= x-firmware.git/ Andrey
(In reply to Andrey Grodzovsky from comment #6) > Verify you are using latest AMD firmware and up = to date MESA/LLVM Firmware: pacman -Q linux-firmware linux-firmware 20180606.d114732-1 ll /usr/lib/firmware/amdgpu/vega10_vce.bin -rw-r--r-- 1 root root 165344 Jun 7 08:01 /usr/lib/firmware/amdgpu/vega10_vce.bin MESA: pacman -Q mesa mesa 18.1.2-1 LLVM: pacman -Q llvm-libs llvm-libs 6.0.0-4 Is this new enough? BTW: In a forum somebody asked what the dmesg output on crash looked like i= f I enabled amdgpu.gpu_recovery=3D1 - the result is a few lines more of output,= but still a fatal system crash: Jun 26 00:50:09 ryzen kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ri= ng gfx timeout, last signaled seq=3D12277, last emitted seq=3D12279 Jun 26 00:50:09 ryzen kernel: [drm] IP block:gmc_v8_0 is hung! Jun 26 00:50:09 ryzen kernel: [drm] IP block:gfx_v8_0 is hung! Jun 26 00:50:09 ryzen kernel: amdgpu 0000:0a:00.0: GPU reset begin! Jun 26 00:50:15 ryzen kernel: [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:42:crtc-0] flip_done timed out Jun 26 00:50:15 ryzen kernel: [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:42:crtc-0] flip_done timed out Jun 26 00:50:25 ryzen kernel: [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:40:plane-4] flip_done timed out
(In reply to dwagner from comment #7) > (In reply to Andrey Grodzovsky from comment #6) > > Verify you are using latest AMD firmware and up to date MESA/LLVM >=20 > Firmware: >=20 > pacman -Q linux-firmware > linux-firmware 20180606.d114732-1 >=20 > ll /usr/lib/firmware/amdgpu/vega10_vce.bin > -rw-r--r-- 1 root root 165344 Jun 7 08:01 > /usr/lib/firmware/amdgpu/vega10_vce.bin >=20 >=20 > MESA: >=20 > pacman -Q mesa > mesa 18.1.2-1 >=20 >=20 > LLVM: > pacman -Q llvm-libs > llvm-libs 6.0.0-4 >=20 > Is this new enough? The kernel and MESA seems new enough, LLVM is 6 so maybe you should try 7. The firmware also looks pretty late but I still would advise to manually override all firmware files with files from here https://git.kernel.org/pub/scm/linux/kernel/git/fi= rmware/linux-firmware.git/tree/amdgpu Just backup your existing firmware/amdgpu folder for any case. >=20 >=20 > BTW: In a forum somebody asked what the dmesg output on crash looked l= ike if > I enabled amdgpu.gpu_recovery=3D1 - the result is a few lines more of = output, > but still a fatal system crash: >=20 > Jun 26 00:50:09 ryzen kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERRO= R* > ring gfx timeout, last signaled seq=3D12277, last emitted seq=3D12279 > Jun 26 00:50:09 ryzen kernel: [drm] IP block:gmc_v8_0 is hung! > Jun 26 00:50:09 ryzen kernel: [drm] IP block:gfx_v8_0 is hung! > Jun 26 00:50:09 ryzen kernel: amdgpu 0000:0a:00.0: GPU reset begin! > Jun 26 00:50:15 ryzen kernel: [drm:drm_atomic_helper_wait_for_flip_done > [drm_kms_helper]] *ERROR* [CRTC:42:crtc-0] flip_done timed out > Jun 26 00:50:15 ryzen kernel: [drm:drm_atomic_helper_wait_for_dependen= cies > [drm_kms_helper]] *ERROR* [CRTC:42:crtc-0] flip_done timed out > Jun 26 00:50:25 ryzen kernel: [drm:drm_atomic_helper_wait_for_dependen= cies > [drm_kms_helper]] *ERROR* [PLANE:40:plane-4] flip_done timed out It's a know issue, try the patch I attached to resolve the deadlock , but y= ou will probably experience other failures after that anyway.=20 Andrey
Created attachment 140345 = [details] [review] Deadlock fix
(In reply to Andrey Grodzovsky from comment #8) > The kernel and MESA seems new enough, LLVM is 6 = so maybe you should try 7. LLVM 7 has not been released, and replacing LLVM 6 with the current subvers= ion head of LLVM 7 means to basically recompile and reinstall half of the opera= ting system (starting at radeonsi, then Xorg, then its dependencies...) I'm fine with using experimental new kernels to find a more stable amdgpu driver - but if a kernel driver crashes just because some user-space application (X11) utilizes a wrong compiler version at run time, then some = part of the driver design is very wrong.=20 > The firmware also looks pretty late but I still = would advise to manually > override all firmware files with files from here > https://git.kernel.org/pub/scm/linux/kernel/git/firmware= /linux-firmware.git/ > tree/amdgpu I did a "diff -r" on the git files with the ones installed by Arc= h, they are all binary identical. > > Jun 26 00:50:25 ryzen kernel: [drm:drm_atom= ic_helper_wait_for_dependencies > > [drm_kms_helper]] *ERROR* [PLANE:40:plane-4] flip_done timed out >=20 > It's a know issue, try the patch I attached to resolve the deadlock , = but > you will probably experience other failures after that anyway. Ok, thanks for the patch, will try this next time I compile a new kernel.
(In reply to Andrey Grodzovsky from comment #8) > The kernel and MESA seems new enough, LLVM is 6 = so maybe you should try 7. LLVM 6 is fine.
(In reply to dwagner from comment #2) > Just to mention this once again: These system cr= ashes still occur, and way > too frequently to consider the amdgpu driver stable enough for profess= ional > use. Sample dmesg output from today: >=20 > Feb 24 18:26:55 [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx ti= meout, > last signaled seq=3D5430589, last emitted seq=3D5430591 > Feb 24 18:26:55 [drm] IP block:gmc_v8_0 is hung! > Feb 24 18:26:55 [drm] IP block:gfx_v8_0 is hung! > Feb 24 18:27:02 [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 > timeout, last signaled seq=3D185928, last emitted seq=3D185930 > Feb 24 18:27:02 [drm] IP block:gmc_v8_0 is hung! > Feb 24 18:27:02 [drm] IP block:gfx_v8_0 is hung! > Feb 24 18:27:05 [drm:amdgpu_dm_atomic_check [amdgpu]] *ERROR* > [CRTC:43:crtc-0] hw_done or flip_done timed out Can you load the kernel with grub command line amdgpu.vm_update_mode=3D3 to= force CPU VM update mode and see if this helps ?
(In reply to Andrey Grodzovsky from comment #12) > Can you load the kernel with grub command line a= mdgpu.vm_update_mode=3D3 to > force CPU VM update mode and see if this helps ? Sure. Too early yet to say "hurray", but at an uptime of one hour= , currently, 4.17.2 survived with amdgpu.vm_update_mode=3D3 already about 20 times longe= r than without that option before the first crash. One (probably just informal) message is emitted by the kernel: [ 19.319565] CPU update of VM recommended only for large BAR system Can you explain a little: What is a "large BAR system", and what = does the vm_update_mode=3D3 option actually cause? Should I expect any weird side ef= fects to look for? BTW: Not a result of that option, but of the kernel version, seems to be the fact that the shader clock keeps at a pretty high frequency all the time - = even without any 3d or compute load, just displaying a quiet 4k/60Hz desktop ima= ge: cat pp_dpm_sclk 0: 214Mhz=20 1: 481Mhz=20 2: 760Mhz=20 3: 1020Mhz=20 4: 1102Mhz=20 5: 1138Mhz=20 6: 1180Mhz * 7: 1220Mhz=20 Much lower shader clocks are used only if I lower the refresh rate of the screen. Is there a reason why the shader clocks should stay high even in the absence of 3d/compute load? (I would have better understood if the minimum memory clock was depending on the refresh rate, but memory clock stays as low as with the older kernels.)=
(In reply to dwagner from comment #13) >=20 > Much lower shader clocks are used only if I lower the refresh rate of = the > screen. Is there a reason why the shader clocks should stay high even = in the > absence of 3d/compute load? > Certain display requirements can cause the engine clock to be kept higher as well.
(In reply to dwagner from comment #13) > (In reply to Andrey Grodzovsky from comment #12) > > Can you load the kernel with grub command line amdgpu.vm_update_m= ode=3D3 to > > force CPU VM update mode and see if this helps ? >=20 > Sure. Too early yet to say "hurray", but at an uptime of one= hour, > currently, 4.17.2 survived with amdgpu.vm_update_mode=3D3 already abou= t 20 > times longer than without that option before the first crash. >=20 > One (probably just informal) message is emitted by the kernel: > [ 19.319565] CPU update of VM recommended only for large BAR system >=20 > Can you explain a little: What is a "large BAR system", and = what does the > vm_update_mode=3D3 option actually cause? Should I expect any weird si= de > effects to look for? I think it just means systems with large VRAM so it will require large BAR = for mapping. But I am not sure on that point. vm_update_mode=3D3 means GPUVM page tables update is done using CPU. By def= ault we do it using DMA engine on the ASIC. The log showed a hang in this engine= so I assumed there is something wrong with SDMA commands we submit. I assume more CPU utilization as a side effect and maybe slower rendering. >=20 >=20 > BTW: Not a result of that option, but of the kernel version, seems to = be the > fact that the shader clock keeps at a pretty high frequency all the ti= me - > even without any 3d or compute load, just displaying a quiet 4k/60Hz d= esktop > image: >=20 > cat pp_dpm_sclk > 0: 214Mhz=20 > 1: 481Mhz=20 > 2: 760Mhz=20 > 3: 1020Mhz=20 > 4: 1102Mhz=20 > 5: 1138Mhz=20 > 6: 1180Mhz * > 7: 1220Mhz=20 >=20 > Much lower shader clocks are used only if I lower the refresh rate of = the > screen. Is there a reason why the shader clocks should stay high even = in the > absence of 3d/compute load? >=20 > (I would have better understood if the minimum memory clock was depend= ing on > the refresh rate, but memory clock stays as low as with the older kern= els.)
(In reply to Andrey Grodzovsky from comment #15) > I think it just means systems with large VRAM so= it will require large BAR > for mapping. But I am not sure on that point. That's correct. the updates are done with the CPU rather than the GPU (SDM= A).=20 The default BAR size on most systems is usually 256MB for 32 bit compatibil= ity so the window for CPU access to vram (where the page tables live) is limite= d.
(In reply to Alex Deucher from comment #16) > (In reply to Andrey Grodzovsky from comment #15) > > I think it just means systems with large VRAM so it will require = large BAR > > for mapping. But I am not sure on that point. >=20 > That's correct. the updates are done with the CPU rather than the GPU > (SDMA). The default BAR size on most systems is usually 256MB for 32 = bit > compatibility so the window for CPU access to vram (where the page tab= les > live) is limited. Thanks Alex. dwagner, this is obviously just a work around and not a fix. It points to s= ome problem with SDMA packets, if you want to continue exploring we can try to = dump some fence traces and SDMA HW ring content to examine the latest packets be= fore the hang happened.
The good news: So far no crashes during normal uptime with amdgpu.vm_update_mode=3D3 The bad news: System crashes immediately upon S3 resume (with messages quite different from the ones I saw with earlier S3-resume crashes) - I filed bug report https://bugs.freedesktop.org/show_bug.= cgi?id=3D107065 on this. (In reply to Andrey Grodzovsky from comment #17) > dwagner, this is obviously just a work around an= d not a fix. It points to > some problem with SDMA packets, if you want to continue exploring we c= an try > to dump some fence traces and SDMA HW ring content to examine the late= st > packets before the hang happened. If you can include some debug output into "amd-staging-drm-next" = that helps finding the root cause, I might be able to provide some output - if the ker= nel survives long enough after the crash to write the system journal - this has= not always been the case.
Can you use addr2line or gdb with 'list' command to give the l= ine number matching (In reply to dwagner from comment #18) > The good news: So far no crashes during normal u= ptime with > amdgpu.vm_update_mode=3D3 >=20 > The bad news: System crashes immediately upon S3 resume (with messages= quite > different from the ones I saw with earlier S3-resume crashes) - I file= d bug > report https://bugs.freedesktop.org/show_bug.= cgi?id=3D107065 on this. >=20 > (In reply to Andrey Grodzovsky from comment #17) > > dwagner, this is obviously just a work around and not a fix. It p= oints to > > some problem with SDMA packets, if you want to continue exploring= we can try > > to dump some fence traces and SDMA HW ring content to examine the= latest > > packets before the hang happened. >=20 > If you can include some debug output into "amd-staging-drm-next&q= uot; that helps > finding the root cause, I might be able to provide some output - if the > kernel survives long enough after the crash to write the system journa= l - > this has not always been the case. No need to recompile, just need to see what is the content of SDMA ring buf= fer when the hang occurs. Clone and build our register analyzer from here - https://cgit.freedesktop.= org/amd/umr/ and once the hang happens just run=20 sudo umr -lb sudo umr -R gfx[.] sudo umr -R sdma0[.] sudo umr -R sdma1[.] I will probably need more info later but let's try this first.
(In reply to Andrey Grodzovsky from comment #19) > No need to recompile, just need to see what is t= he content of SDMA ring > buffer when the hang occurs. >=20 > Clone and build our register analyzer from here - > https://cgit.freedes= ktop.org/amd/umr/ and once the hang happens just run=20 >=20 > sudo umr -lb > sudo umr -R gfx[.] > sudo umr -R sdma0[.] > sudo umr -R sdma1[.] >=20 > I will probably need more info later but let's try this first. How can I run "umr" on a crashed system? I guess those register v= alues are retained over a press of the reset button / reboot?
(I meant to write "I guess those register values are NOT = retained over a reboot, right?")