| Bug ID | 111481 |
|---|---|
| Summary | AMD Navi GPU frequent freezes on both Manjaro/Ubuntu with ker= nel 5.3 and mesa 19.2 -git/llvm9 |
| Product | DRI |
| Version | unspecified |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | critical |
| Priority | not set |
| Component | DRM/AMDgpu |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | popovic.marko@protonmail.com |
I've tried my AMD Radeon RX 5700 XT on both ubuntu (llvm 9 / m=
esa 19.3 - Oibaf
PPA) and Manjaro (llvm 10 git / mesa-git).
On both I've been using Gnome shell and in both cases I had frequent lockups
and freezes. Once my GPU disconnected to Monitor and remained so until I
rebooted, other times desktop would just freeze and crash the whole system.
Software tried: LLVM 10 git / MESA 19.3 - git on Manjaro
LLVM 9 / MESA 19.3 git from Oibaf PPA
Kernels tried: Manjaro 5.3 RC4, Ubuntu 5.3 RC5 generic, Ubuntu drm-tip 5.3
daily
Error log:
avg 24 22:53:58 Marko-PC kernel: [drm:amdgpu_dm_commit_planes.constprop.0
[amdgpu]] ERROR Waiting for fences timed out or interrupted!
avg 24 22:53:58 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] ERROR r=
ing
gfx_0.0.0 timeout, signaled seq=3D94235, emitted seq=3D94237
avg 24 22:53:58 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] ERROR
Process information: process citra-qt pid 27356 thread citra-qt:cs0 pid 273=
66
Happened on all setups, bug was pretty much the same, lockups weren't extre=
mely
frequent but frequent enough that they were very noticable (5-6 freezes per=
day
on average)
Faulty hardware is probably out of options since I never had a hiccup or
anything even close to crash or freeze on my Windows desktop.
| What | Removed | Added |
|---|---|---|
| Priority | not set | highest |
Adding error log from Manjaro: avg 23 16:05:37 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=3D1742, emitted seq=3D1743 avg 23 16:05:37 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process gnome-shell pid 975 thread gnome-shell:cs0 pid= 988 avg 23 16:05:37 Marko-PC kernel: [drm] GPU recovery disabled. Pretty much same-type error happens in different situations and very often = at random while using the desktop. These 2 logs one is from launching an OpenGL from Citra emulator which is reproducable every time and the second one from Manjaro is while browsing the Gnome shell and it would crash without any cl= ear triggers.
| What | Removed | Added |
|---|---|---|
| Priority | highest | not set |
I confirm that I have this bug or a very similar one. It, for some reason, happens most when i'm using my IDE (Intellij based). It will append the most when I type code and the crash occur when the IDE is supposed to propose some code completion. I do have one to two crash a day. Video card is RX5700 CPU is Ryzen R7-2700X Software tested LLVM 9 git libdrm, mesa, ddx updated from GIT very frequently. Bug is there since I have the card, like 3 weeks ago.
I don't know if i'm encountering the same bug, but it is at le=
ast similar.
I don't get hard freezes/lockups, but i get a strange "stutterting&quo=
t;, as if the
whole OS halted for a few seconds, then continued for a few seconds...and t=
he
halted times grew while the "usable seconds" got shorter quickly =
to the point
of unusability...
It doesn't happen regularly (seems like anything between 30min and 120min) =
and
i haven't yet made out a direct cause, but in journalctl, it seems the same
messages appear every time when it begins:
kernel: amdgpu: [powerplay] Failed to send message 0xf, response 0xfffffffb,
param 0xfd6000
kernel: amdgpu: [powerplay] Failed to send message 0xf, response 0xfffffffb,
param 0xfd6000
kernel: amdgpu 0000:0f:00.0: [mmhub] VMC page fault (src_id:0 ring:169 vmi=
d:0
pasid:0)
kernel: amdgpu 0000:0f:00.0: at page 0x0000600000fd6000 from 18
kernel: amdgpu 0000:0f:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00041152
after that there are a lot of these:
kernel: amdgpu: [powerplay] Failed to send message 0x40, response 0xffffffc2
param 0x2
kernel: amdgpu: [powerplay] Failed to send message 0xe, response 0xffffffc2,
param 0x80
until shutdown/hardreset.
Maybe some observation that might help to narrow it down:
The first time it occured, i had to do a few reboots that showed this behav=
iour
right after startup until it finally worked again - for about 45min.
As it didn't work again after around 10 reboots, i tried uninstalling corec=
trl
(that i used to have a custom fan-curve) - and it finally booted normal aga=
in!
I then installed radeon-profile to have fan-controll (i don't want to have =
the
fans stand still on desktop, as the card gets over 80=C2=B0 C hot before th=
e fans
kick in...).
The issue still occurs with radeon-profile, but at least every reboot is
running fine...
Other thing i noticed is that after the first "freeze" with radeo=
n-profile
lm_sensors stopped reporting the fanspeed for the card, it always stays at
zero.
So maybe it is related to fan-control or the sysfs interface in general?
Forgot to mention: running Manjaro 5.3rc6.d0826.ga55aa89-1, me= sa-git 1:19.3.0_devel.114849.0142dcb990e-1 and llvm-libs-git 10.0.0_r325376.70e158e09e9-1 And if it matters: firmware from https://aur.archlinux.org/packages/linux-firmware-agd5f-radeon-navi= 10/ v2019.08.26.14.36-1
It probably really depend of what we do on our desktop. I just= remember now how I did stop using FileZilla since I got that GPU as it was crashing almost a= ll the time I was using it (Like I never not crashed while that thing was open= and running). Still use it for work but I keep it to minimum (open, upload, clo= se) instead of keeping it running.
| What | Removed | Added |
|---|---|---|
| CC | alexandr.kara@gmail.com |
Might be related to https://bugs.freedesktop.org/show_bug.= cgi?id=3D111269. I also get the "ring gfx_0.0.0 timeout" error (but not the "ring sd= ma0 timeout" error).=20 Using LLVM from git + Mesa 19.2.0-rc1 on Fedora 30 with kernel from Fedora = 31 (5.3.0-0.rc5.git0.1.fc31.x86_64). GPU AMD Radeon RX 5700 XT, CPU AMD Ryzen 7 1700, 32 GB RAM (EDD).
Created attachment 145225<= /a> [details] [review] Merge last adg5f code Ok, I did look at the recent kernel patch and commit and they seam to have fixed a couple bugs. I do not know it it include these but I did not crash = one time since I merged that into the kernel 5.3-rc6. (that code is staged for = 5.4 merge window). I did attach the patch so you can merge that if you wish to try. It add all= the latest bits for AMDGPU into 5.3-rc6, including Renoir support.
(In reply to Mathieu Belanger from comment #7) > Created atta= chment 145225 [details] [review] [re= view] > Merge last adg5f code >=20 > Ok, I did look at the recent kernel patch and commit and they seam to = have > fixed a couple bugs. I do not know it it include these but I did not c= rash > one time since I merged that into the kernel 5.3-rc6. (that code is st= aged > for 5.4 merge window). >=20 > I did attach the patch so you can merge that if you wish to try. It ad= d all > the latest bits for AMDGPU into 5.3-rc6, including Renoir support. How do I merge the patch myself? :) I'd like to try it
On my side i can report that the issue does not occur if i don= 't use a tool to modify the FANs - does anyone of you use something of the like or are this seperate issues?
(In reply to Matthias M=C3=BCller from comment #9) > On my side i can report that the issue does not = occur if i don't use a tool > to modify the FANs - does anyone of you use something of the like or a= re > this seperate issues? I don't use any tools, all is stock. (In reply to Mathieu Belanger from = comment #7) > Created atta= chment 145225 [details] [review] [re= view] > Merge last adg5f code >=20 > Ok, I did look at the recent kernel patch and commit and they seam to = have > fixed a couple bugs. I do not know it it include these but I did not c= rash > one time since I merged that into the kernel 5.3-rc6. (that code is st= aged > for 5.4 merge window). >=20 > I did attach the patch so you can merge that if you wish to try. It ad= d all > the latest bits for AMDGPU into 5.3-rc6, including Renoir support. After applying the patch, same type of error occurs, luckily very easy to reproduce with Citra emulator, apparently it does something that AMD's driv= er really doesn't like and makes chances higher for error to occur. Also when = CPU is under heavy I/O load error seems more likely to occur as well on my end. Last log after applying the latest patch from the merge posted in the attachment: sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out! sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=3D16312, emitted seq=3D16314 sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process citra-qt pid 2928 thread citra-qt:cs0 pid 2938 sep 01 02:29:10 Marko-PC kernel: [drm] GPU recovery disabled. If we could get any official AMD responses to at least make sure that we're= at least being listened to would be very nice.
Same bug is also reproducable when launching native version of= Rocket League. Here are the logs: sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:158 vmid:0 pasid:0, for process pid 0 thread pid 0) sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: in page starting at address 0x0000000000fff000 from client 27 sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00001B3C sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: MORE_FAULTS:= 0x0 sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: WALKER_ERROR: 0x6 sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0:=20=20=20=20=20=20=20= =20=20 PERMISSION_FAULTS: 0x3 sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: MAPPING_ERRO= R: 0x1 sep 01 12:20:56 Marko-PC kernel: amdgpu 0000:03:00.0: RW: 0x0 sep 01 12:21:12 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=3D7198, emitted seq=3D7200 sep 01 12:21:12 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process RocketLeague pid 3035 thread RocketLeag:cs0 pid 3042
I did not crash and have a > 24h uptime. I could not test Citra as I don't have a 3DS and the roms I found are encrypted.. I could not test on Rocket League as it would require me to spend for a gam= e I will not play. I will continue to test later today.
(In reply to Marko Popovic from comment #10) > (In reply to Matthias M=C3=BCller from comment #9) > > On my side i can report that the issue does not occur if i don't = use a tool > > to modify the FANs - does anyone of you use something of the like= or are > > this seperate issues? >=20 > I don't use any tools, all is stock. >=20 > (In reply to Mathieu Belanger from comment #7) > > Created attachment 145225= [details] [review] [review] [review] > > Merge last adg5f code > >=20 > > Ok, I did look at the recent kernel patch and commit and they sea= m to have > > fixed a couple bugs. I do not know it it include these but I did = not crash > > one time since I merged that into the kernel 5.3-rc6. (that code = is staged > > for 5.4 merge window). > >=20 > > I did attach the patch so you can merge that if you wish to try. = It add all > > the latest bits for AMDGPU into 5.3-rc6, including Renoir support. >=20 > After applying the patch, same type of error occurs, luckily very easy= to > reproduce with Citra emulator, apparently it does something that AMD's > driver really doesn't like and makes chances higher for error to occur= . Also > when CPU is under heavy I/O load error seems more likely to occur as w= ell on > my end. >=20 > Last log after applying the latest patch from the merge posted in the > attachment: > sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_dm_atomic_commit_tail [am= dgpu]] > *ERROR* Waiting for fences timed out! > sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *E= RROR* > ring gfx_0.0.0 timeout, signaled seq=3D16312, emitted seq=3D16314 > sep 01 02:29:10 Marko-PC kernel: [drm:amdgpu_job_timedout [amdgpu]] *E= RROR* > Process information: process citra-qt pid 2928 thread citra-qt:cs0 pid= 2938 > sep 01 02:29:10 Marko-PC kernel: [drm] GPU recovery disabled. >=20 > If we could get any official AMD responses to at least make sure that = we're > at least being listened to would be very nice. I was able to reproduce that Citra crash. Followed the instruction, it did crash instantly after choosing continue (o= r a fraction of a second after, the music lagged a lil and complete system crash (was able so sync/umount/reboot with the magics key)). Is your crash exactly at the same place? If so then it's very reproducible = and=20 it might be a good idea to run a opengl trace to see what commands was sent last to provoke the crash. I am not familiar with the Ubuntu stuff, is these got compiled on your syst= em? if no do you know the build date of your Mesa, libdrm and xf86-video-amdgpu (x11 ddx). Also can you tell what microcode files dates you do have? Libdrm : 07:49:10 PM 08/27/2019 Mesa : 05:37:07 PM 08/30/2019 Xorg amdgpu DDX : 07:55:17 PM 08/27/2019 The microcode files where not available on my distribution when I installed them. I did download/install them on August 6 but they where from July 15 i= sh I think, I remember that the latest microcode at that time where crashing wit= h a black screen on module load and that's why I did install an older version.<= /pre>
(In reply to Mathieu Belanger from comment #13)=20 > I was able to reproduce that Citra crash. > Followed the instruction, it did crash instantly after choosing contin= ue (or > a fraction of a second after, the music lagged a lil and complete syst= em > crash (was able so sync/umount/reboot with the magics key)). >=20 > Is your crash exactly at the same place? If so then it's very reproduc= ible > and it might be a good idea to run a opengl trace to see what command= s was > sent last to provoke the crash. >=20 > I am not familiar with the Ubuntu stuff, is these got compiled on your > system? if no do you know the build date of your Mesa, libdrm and > xf86-video-amdgpu (x11 ddx). >=20 > Also can you tell what microcode files dates you do have? >=20 > Libdrm : 07:49:10 PM 08/27/2019 > Mesa : 05:37:07 PM 08/30/2019 > Xorg amdgpu DDX : 07:55:17 PM 08/27/2019 >=20 > The microcode files where not available on my distribution when I inst= alled > them. I did download/install them on August 6 but they where from July= 15 > ish I think, I remember that the latest microcode at that time where > crashing with a black screen on module load and that's why I did insta= ll an > older version. Yes, always happens at the same place with Citra emulator, however what bot= hers me more about the bug is that sometimes it happens completely randomly on my system without any really obvious triggers while just browsing and using my desktop, so it's not Citra exclusive, but luckily I've found the Citra meth= od to provode the bug so we can do more detailed logging. Further observations: - Bug is the same-type as other crashes and is not Citra emulator exclusive, happens on Rocket League on launch as well and sometimes randomly while usi= ng the desktop - Same type of crash IS NOT reproducable on Windows on the same GPU - Same type of bug IS NOT reproducable on my IntelHD laptop with same versi= ons of MESA/LLVM which probably means either faulty AMD kernel driver or faulty Firmware binaries. My versions are: MESA: Mesa 19.3.0-devel (git-6775a52 2019-09-02 eoan-oibaf-ppa) Kernel: Ubuntu mainline 5.3 daily build (I ALSO tried amd-drm-next-5.4, same bug is reproducable) Firmware binaries: 2019-08-26 from /~agd5f/radeon_ucode/navi10
(In reply to Marko Popovic from comment #14) >=20 > Yes, always happens at the same place with Citra emulator Could you capture a trace of the problem (using Apitrace or Renderdoc)? This would be very helpful to fix it.
Created attachment=
145232 [details]
APITrace log from Citra crash
(In reply to Pierre-Eric Pelloux-Prayer from comment #15) > (In reply to Marko Popovic from comment #14) > >=20 > > Yes, always happens at the same place with Citra emulator >=20 > Could you capture a trace of the problem (using Apitrace or Renderdoc)? >=20 > This would be very helpful to fix it. I added reproduced Citra crash recorded by using command: apitrace trace ./citra-qt I hope this is correct, if you need anything else or done differently please just let me know!
Created att=
achment 145233 [details]
APITrace log from RocketLeague crash
I am adding Rocket League crash output from apitrace.
(In reply to Marko Popovic from comment #17) > (In reply to Pierre-Eric Pelloux-Prayer from comment #15) > > (In reply to Marko Popovic from comment #14) > > >=20 > > > Yes, always happens at the same place with Citra emulator > >=20 > > Could you capture a trace of the problem (using Apitrace or Rende= rdoc)? > >=20 > > This would be very helpful to fix it. >=20 > I added reproduced Citra crash recorded by using command: > apitrace trace ./citra-qt >=20 > I hope this is correct, if you need anything else or done differently = please > just let me know! Thanks for the trace! Replaying the trace a few times is enough to reliably to reproduce the hang. Using AMD_DEBUG=3Dnongg seems to prevent it so it could be a temporary work= around until a proper fix is found. Could you confirm this on your system? >=20 > I am adding Rocket League crash output from apitrace. This trace file is very small (only one frame) and doesn't hang here.