[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 3011 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

            Bug ID: 107432
           Summary: Periodic complete system lockup with Vega M and Kernel
                    4.18-rc6+
           Product: DRI
           Version: unspecified
          Hardware: x86-64 (AMD64)
                OS: Linux (All)
            Status: NEW
          Severity: normal
          Priority: medium
         Component: DRM/AMDgpu
          Assignee: dri-devel@lists.freedesktop.org
          Reporter: rstrube@gmail.com

Created attachment 140902
  --> https://bugs.freedesktop.org/attachment.cgi?id=140902&action=edit
System log leading up to hard crash

Description:

Periodically my system will begin to slow down dramatically (the mouse cursor
hitches as I try to move it) and I am unable to interact with anything on the
screen.  Eventually the mouse cursor disappears altogether.  Trying to switch
to a tty I do get prompted to login, but after entering my credentials nothing
happens.  It appears to be a hard lockup.  The only solution is to manually
power down my machine and reboot.

This probably happens one or two times a day, normally after starting a new
application.

Hardware:
Dell XPS 15 9575 2 in 1 (Kaby Lake G)

Versions:
Kernel 4.18-rc7
Mesa 18.1.5
Xorg 1.19.6
uCode for Vega M from Linux Firmware git (master) which includes the latest
18.20 uCode from AMD that was recently merged into Linux Firmware

I do have the two sinks available (one for the Intel iGPU and one for the AMD
Vega M), running:

xrandr --listproviders

Lists the following:

Providers: number : 2
Provider 0: id: 0x6f cap: 0x9, Source Output, Sink Offload crtcs: 3 outputs: 7
associated providers: 1 name:modesetting
Provider 1: id: 0x45 cap: 0x6, Sink Output, Source Offload crtcs: 6 outputs: 0
associated providers: 1 name:Unknown AMD Radeon GPU @ pci:0000:01:00.0

And running:

env DRI_PRIME=1 glxinfo | grep "OpenGL renderer"

Lists:

OpenGL renderer string: AMD VEGAM (DRM 3.26.0, 4.18.0-041800rc7-generic, LLVM
6.0.0)

So the Vega M is active and available in my system.

I noticed that this problem started happening after the release of kernel
4.18-rc6 and continues with 4.18-rc7. I've been using 4.18 since rc1 without
issue.  This entry in the changelog caught my eye:

Leo Liu (1):
      drm/amdgpu: Make sure IB tests flushed after IP resume

Not sure if this is at all related, but the reason I bring this up is because 
the errors I see in my logs everytime I encounter this problem are:

kernel: amdgpu 0000:01:00.0: GPU pci config reset
kernel: [drm:amdgpu_device_ip_suspend [amdgpu]] *ERROR* suspend of IP block
<uvd_v6_0> failed -12

Please note that so far I have only encountered this problem when launching
applications that use my Intel iGPU (i.e. I am not setting DRI_PRIME=1).

I've attached my entire log to provide more context.

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 4484 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 631 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

Robert Strube <rstrube@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #140902|0                           |1
        is obsolete|                            |

--- Comment #1 from Robert Strube <rstrube@gmail.com> ---
Created attachment 140903
  --> https://bugs.freedesktop.org/attachment.cgi?id=140903&action=edit
dmesg log leading up to system crash (more detailed)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2172 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 427 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

Michel Dänzer <michel@daenzer.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #140903|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1109 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 598 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #2 from Michel Dänzer <michel@daenzer.net> ---
Created attachment 140905
  --> https://bugs.freedesktop.org/attachment.cgi?id=140905&action=edit
Use kvmalloc in amdgpu_uvd_suspend

Does this patch help by any chance?

If not, can you bisect between 4.18-rc1 and -rc6? Note that from your
description, you'll need to test for at least one day before declaring a commit
good (if you hit a failure, you can immediately declare that commit bad).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1658 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 1867 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #3 from Robert Strube <rstrube@gmail.com> ---
(In reply to Michel Dänzer from comment #2)
> Created attachment 140905 [details] [review]
> Use kvmalloc in amdgpu_uvd_suspend
> 
> Does this patch help by any chance?
> 
> If not, can you bisect between 4.18-rc1 and -rc6? Note that from your
> description, you'll need to test for at least one day before declaring a
> commit good (if you hit a failure, you can immediately declare that commit
> bad).

Hi Michel,

Thank you for the patch.  I've rebuilt the kernel with the changes in your
patch and am currently going to test it out over the next several days.

I've noticed that the problem seems to occur when there is a large amount of
memory pressure (e.g. I'm running a VM where I've allocated lots of memory),
and almost always after I've just opened a new application windows.  Perhaps a
web browser, text editor, etc.

Today I had a scenario (running the vanilla 4.18-rc7) where I simply ran out of
memory *BUT* this occurred in the absence of opening up a new application
window, and the system was able to recover gracefully.

I do have 16GB of RAM in my system, but I can easily hit the limit by running a
VM and opening several applications.

Should I conduct tests with memory pressure applied to see if your patch
addresses the issue?  Are we trying to simulate the same scenario as before?

I'll report back my results.

P.S. I've attached another dmesg.log from the out of memory problems I ran into
today (again running on vanilla 4.18-rc7 and not using your patch) so you can
compare the two scenarios.  This scenario did not result in a complete system
lockup, so something different must have occurred.

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 3083 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 614 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #4 from Robert Strube <rstrube@gmail.com> ---
Created attachment 140932
  --> https://bugs.freedesktop.org/attachment.cgi?id=140932&action=edit
dmesg log leading up to out of memory scenario (no crash this time)

In this scenario the memory pressure the system was experiencing did not lead
to a system crash.  The main difference here was that I did not open a new
application, the applications I already had open simply exhausted the memory I
had available.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1652 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 427 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

Michel Dänzer <michel@daenzer.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #140932|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1109 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 1250 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #5 from Robert Strube <rstrube@gmail.com> ---
So I've been conducting lots of additional investigation with both the vanilla
kernel (4.18-rc7) and the kernel with your patch.

I took more time to try to recreate the scenarios that cause the crash
(monitoring system resources, etc.) and this is when I realized that my
swapfile was very small (only 2GB).

Short story - Upon further investigation I don't believe this is a bug with
DRM/amdgpu but rather the crash was caused because I simply ran out of memory
*and* swapspace combined.

I feel a little silly about this, I'm running Ubuntu 18.04 and I guess the
default swapfile size is 2GB.  I'm used to using swap partions which are the
same size as the system RAM, so I never considered that I could be running out
of *both*.

I think at this point it's safe to close the bug.  I'm going to increase my
swapfile size to 16GB and monitor my system more closely.  If I get the hard
system crash I'll first determine if I ran out of swap, and then if it appears
I had enough swap, I'll reopen this bug.

Thanks for your assistance!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2062 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 430 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

Robert Strube <rstrube@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |NOTABUG

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1273 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 504 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #6 from Michel Dänzer <michel@daenzer.net> ---
Well, https://bugs.freedesktop.org/attachment.cgi?id=140903 definitely shows an
amdgpu issue exacerbating the memory pressure situation — it tries to allocate
4M of physically contiguous memory. My patch fixes that. Can you confirm that
the patch at least doesn't cause any additional issues of its own?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1442 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 1029 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #7 from Robert Strube <rstrube@gmail.com> ---
(In reply to Michel Dänzer from comment #6)
> Well, https://bugs.freedesktop.org/attachment.cgi?id=140903 definitely shows
> an amdgpu issue exacerbating the memory pressure situation — it tries to
> allocate 4M of physically contiguous memory. My patch fixes that. Can you
> confirm that the patch at least doesn't cause any additional issues of its
> own?

Hey! Good point.

I ran the custom kernel for a couple days without issue.  Would you like me to
do some more testing? I went back to vanilla 4.18-rc7 - but I'd be happy to
make my daily driver 4.18-rc7 with the patch.

My understanding is that kvmalloc is a slightly safer way of allocating memory
as compared to kmalloc - in that it doesn't necessarily need the memory to be
contiguous.  The downside is that it's not quite a performant.  Is this
correct?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2048 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 679 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #8 from Robert Strube <rstrube@gmail.com> ---
(In reply to Michel Dänzer from comment #6)
> Well, https://bugs.freedesktop.org/attachment.cgi?id=140903 definitely shows
> an amdgpu issue exacerbating the memory pressure situation — it tries to
> allocate 4M of physically contiguous memory. My patch fixes that. Can you
> confirm that the patch at least doesn't cause any additional issues of its
> own?

I've now moved to 4.18-rc8.  Would you like me to apply your patch to this
release and report back?

Thanks!
Rob

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1698 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 264 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #9 from Michel Dänzer <michel@daenzer.net> ---
Please test https://patchwork.freedesktop.org/patch/242563/ instead.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1196 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 496 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #10 from Robert Strube <rstrube@gmail.com> ---
Hello Michel,

Apologies, I've been pretty busy with work the last month or so.  I'm now
available again to test out your patch (not sure if this has already made it's
way into mainline?).

I'm currently running 4.18.7.  Please let me know if I can start to help out on
this issue again.

Rob

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 226 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #11 from Michel Dänzer <michel@daenzer.net> ---
The patch landed in 4.19-rc1.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1097 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[Bug 107432] Periodic complete system lockup with Vega M and Kernel 4.18-rc6+

[bugzilla-daemon]

[-- Attachment #1.1: Type: text/plain, Size: 324 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=107432

--- Comment #12 from Robert Strube <rstrube@gmail.com> ---
Thanks Michel!

I'm currently running 4.19.  I'll put my system under memory pressure and see
if things are working OK.

Rob

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1193 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel