| Bug ID | 106225 |
|---|---|
| Summary | Kernel panic after modesetting (not on every boot) on ryzen 5= 2400g |
| Product | DRI |
| Version | unspecified |
| Hardware | x86-64 (AMD64) |
| OS | Linux (All) |
| Status | NEW |
| Severity | normal |
| Priority | medium |
| Component | DRM/AMDgpu |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | f.pinamartins@gmail.com |
Created attachment 139081 [details]
journaltcl log file
On a new Ryzen 5 2400G build, I am using linux-4.16.3 (Arch Linux).
On some boots (~1 out of every 3) I get a kernel panic after modesetting
occurs.
I have attached the relevant systemd logfile (for the entire boot process).
I am not sure what component is failing, but I suspect AMDGPU, since it hap=
pens
suspiciously after modesetting.
If it matters, the mainboard is a Gigabyte "AB350N-Gaminig Wifi",=
with the
latest available BIOS as of writing (BIOS F23d 04/17/2018).
Please advise if more information is required.
The journalctl log file shows oopses from the evdev driver, no= obvious amdgpu related issues. Can you attach a picture of the output from the actual kernel panic?
| What | Removed | Added |
|---|---|---|
| Attachment #139081 is obsolete | 1 |
Created attachment 139114 [details]
journaltcl log file
Apologies, as this was the wrong logfile (I was getting some journalctl
corruption).
But it seems you are correct and this is not a kernel panic, as I still can
reboot using the "Magic SysRq key" (which I usually can't during =
kernel
panics), albeit the "Caps Lock" key is unresponsive.
Attached is a log that should show the issue. Note the line:
abr 25 23:23:09 ZenBox systemd-udevd[297]: worker [308] failed while handli=
ng
'/devices/pci0000:00/0000:00:08.1/0000:09:00.0'
Where '/devices/pci0000:00/0000:00:08.1/0000:09:00.0' seems to be the "=
;graphics
card".
Symptoms are the same. During some boots, modesetting "seems" to =
occur, but I
get a black screen instead. System is unresponsive, including "Caps Lo=
ck"
lights not changing on key press. Magic SysRq key does seem to successfully
reboot the system.
How should I edit the title to correctly to reflect this?
| What | Removed | Added |
|---|---|---|
| CC | harry.wentland@amd.com |
There's a general protection fault within kmem_cache_alloc_tra= ce when dcn10_create_resource_pool calls kzalloc (which looks innocuous). There's another general protection fault in kmem_cache_alloc_trace later, called fr= om cgroup code. Looks like there might be a general memory management related issue.
Maybe you can try enabling KASAN and see if that catches anyth= ing earlier.
Created attachment 139154 [de=
tails]
journaltcl log file with KASAN enabled in kernel
I have compiled a new kernel with KASAN module enabled.
However, I am not sure I am getting any KASAN related output in the logs
(attached).
Is there any boot option I should be passing it? Alternatively, can you poi=
nt
me towards a good source of documentation on using KASAN (it is the first t=
ime
I am trying this).
Looks like KASAN isn't enabled yet =E2=80=94 the lines in dmes= g containing "PREEMPT SMP NOPTI" should contain "KASAN" as well. FWIW, I just enable CONFIG_KASAN and CONFIG_KASAN_INLINE in .config, I don't have to do anything at runtime to enable it.
I had previously made a mistake in loading the kernel's .confi= g file. I have now managed to compile the kernel with the options for KASAN set. However, booting this kernel results in an instant reboot after displaying = the message "Loading initial ramdisk...". I will try to compile another kernel using CONFIG_KASAN_OUTLINE this time a= nd see if I have better luck.
| What | Removed | Added |
|---|---|---|
| Attachment #139154 is obsolete | 1 |
Created attach=
ment 139178 [details]
journalctl log with KASAN_OUTLINE
Ok, so using CONFIG_KASAN_OUTLINE works.
I get a KASAN enabled Kernel, which boots.
Every
Single
Time
I cannot reproduce the error with KASAN enabled after more than 15 reboots =
(I
lost count after that).
Using the "regular" kernel keeps giving me the error ~1 out of 3 =
boots.
I have attached a KASAN enabled log, but I am not sure how useful it might =
be.
Getting closer, please try again with kasan_multi_shot on the = kernel command line, otherwise KASAN only reports the first thing it catches.
| What | Removed | Added |
|---|---|---|
| Attachment #139178 is obsolete | 1 |
Created attachment 139181 [details]
journalctl log with KASAN_OUTLINE and kasan_multi_shot
Bingo.
Now here is the thing...
With the KASAN enabled kernel and the multi_shot option set, I can **never**
bot successfully.
In fact, even mode-setting is not happening. I am uploading 2 log files, si=
nce
on one of the occasions, I got a black screen when I was expecting modesett=
ing
to happen. On the other log, I got a non-modeset text screen with the KASAN
dumps.
I hope this is what was needed.
Please provide the output of the following in your kernel buil= d tree: scripts/faddr2line drivers/gpu/drm/amd/amdgpu/amdgpu.ko firmware_parser_create+0xa70/0xd90
francisco@ZenBox [18:03:53] [/usr/lib/modules/4.16.5-1-kas=
an/build]=20
-> $ scripts/faddr2line drivers/gpu/drm/amd/amdgpu/amdgpu.ko
firmware_parser_create+0xa70/0xd90
ERROR: can't find objfile drivers/gpu/drm/amd/amdgpu/amdgpu.ko
the directory "drivers/gpu/drm/amd/amdgpu/" contains only a file =
named
"Kconfig", containing the following:
```
config DRM_AMDGPU_SI
bool "Enable amdgpu support for SI parts"
depends on DRM_AMDGPU
help
Choose this option if you want to enable experimental support
for SI asics.
SI is already supported in radeon. Experimental support for SI
in amdgpu will be disabled by default and is still provided by
radeon. Use module options to override this:
radeon.si_support=3D0 amdgpu.si_support=3D1
config DRM_AMDGPU_CIK
bool "Enable amdgpu support for CIK parts"
depends on DRM_AMDGPU
help
Choose this option if you want to enable support for CIK asics.
CIK is already supported in radeon. Support for CIK in amdgpu
will be disabled by default and is still provided by radeon.
Use module options to override this:
radeon.cik_support=3D0 amdgpu.cik_support=3D1
config DRM_AMDGPU_USERPTR
bool "Always enable userptr write support"
depends on DRM_AMDGPU
select MMU_NOTIFIER
help
This option selects CONFIG_MMU_NOTIFIER if it isn't already
selected to enabled full userptr support.
config DRM_AMDGPU_GART_DEBUGFS
bool "Allow GART access through debugfs"
depends on DRM_AMDGPU
depends on DEBUG_FS
default n
help
Selecting this option creates a debugfs file to inspect the mapped
pages. Uses more memory for housekeeping, enable only for debuggi=
ng.
source "drivers/gpu/drm/amd/acp/Kconfig"
source "drivers/gpu/drm/amd/display/Kconfig"
```
I did find amdgpu.ko.xz under
"/usr/lib/modules/4.16.5-1-kasan/kernel/drivers/gpu/drm/amd/amdgpu/amd=
gpu.ko.xz"
which I have decompressed using "xz -k -d".
Running the command you requested resulted in the following output:
francisco@ZenBox [18:12:53] [/usr/lib/modules/4.16.5-1-kasan/build]=20
-> $ scripts/faddr2line ../kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko
firmware_parser_create+0xa70/0xd90=20
firmware_parser_create+0xa70/0xd90:
firmware_parser_create at ??:?
PS - Thank you for your patience
(In reply to Francisco Pina Martins from comment #12) > francisco@ZenBox [18:12:53] [/usr/lib/module= s/4.16.5-1-kasan/build]=20 > -> $ scripts/faddr2line ../kernel/drivers/gpu/drm/amd/amdgpu/amdgpu= .ko > firmware_parser_create+0xa70/0xd90=20 > firmware_parser_create+0xa70/0xd90: > firmware_parser_create at ??:? What does file ../kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko say? If its output doesn't say "not stripped", look for an unstri= pped version of the module. If the file output for that doesn't say "with debug_inf= o", try enabling CONFIG_DEBUG_INFO (and maybe CONFIG_DEBUG_INFO_REDUCED). If that still doesn't result in better output than ??:? from faddr2line, the best guess so far is that there's an issue somewhere in drivers/gpu/drm/amd/display/dc/bios/bios_parser2.c:bios_parser_construct.
francisco@ZenBox [10:31:49] [~]=20 -> $ file /usr/lib/modules/4.16.5-1-kasan/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko /usr/lib/modules/4.16.5-1-kasan/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=3Dfa6433331b1f8048ce1c9b487d7b67d7a4aa4c31, not stripped It does not say "with debug_info", so I'm compiling a new kernel = with CONFIG_DEBUG_INFO and CONFIG_DEBUG_INFO_REDUCED activated. I will post the results as soon as I have them.
Here you go: francisco@ZenBox [11:13:37] [/tmp/build/linux-kasan/src/linux-4.16]=20 -> $ scripts/faddr2line drivers/gpu/drm/amd/amdgpu/amdgpu.ko firmware_parser_create+0xa70/0xd90 firmware_parser_create+0xa70/0xd90: get_integrated_info_v11 at /tmp/build/linux-kasan/src/linux-4.16/drivers/gpu/drm/amd/amdgpu/../display= /dc/bios/bios_parser2.c:1572 (inlined by) construct_integrated_info at /tmp/build/linux-kasan/src/linux-4.16/drivers/gpu/drm/amd/amdgpu/../display= /dc/bios/bios_parser2.c:1714 (inlined by) bios_parser_create_integrated_info at /tmp/build/linux-kasan/src/linux-4.16/drivers/gpu/drm/amd/amdgpu/../display= /dc/bios/bios_parser2.c:1755 (inlined by) bios_parser_construct at /tmp/build/linux-kasan/src/linux-4.16/drivers/gpu/drm/amd/amdgpu/../display= /dc/bios/bios_parser2.c:1912 (inlined by) firmware_parser_create at /tmp/build/linux-kasan/src/linux-4.16/drivers/gpu/drm/amd/amdgpu/../display= /dc/bios/bios_parser2.c:1927 Is this it?
That does look helpful, thanks! I'll leave it to the DC folks = to take it from here. Francisco, just one more thing: Can you double-check that KASAN still repor= ts firmware_parser_create+0xa70/0xd90 with the current amdgpu.ko, or otherwise pass the current values to faddr2line?
I don't think I'll be able to test that, since The kernel with= debug enabled weights in at 590Mb, which is larger than my 512Mb /boot partition. Will it work if I just replace the amdgpu.ko file with the debug-enabled version?
(In reply to Francisco Pina Martins from comment #17) > Will it work if I just replace the amdgpu.ko fil= e with the debug-enabled > version? Yeah, that could work. If it doesn't, then it's not a big deal, it's unlike= ly that the value before the slash has changed but the one after the slash has= n't (if the latter had changed, faddr2line should have complained). Just double-checking.
Created attachment 139239 [details]
journalctl log with KASAN_OUTLINE and kasan_multi_shot using amdgpu.ko comp=
iled
with debug info
Here you go. journalctl log after booting with kasan_multi_shot and using t=
he
`amdgpu.ko` file compiled with debug info. Now this is a Frankenkernel mons=
ter
if I ever saw one.
I have taken the liberty of "grepping" for the pattern, and it se=
ems that
"firmware_parser_create+0xa70/0xd90" is still there.
Thank you for passing this along to the DC folk. Please let me know when a
patch I can test is made available. Also, if you need any more information =
(or
try something), just ask away.
At the very least, with this bug report I have discovered that the KASAN
enabled kernel boots every time, at the expense of an extra second while
booting. I did not notice any other performance differences between the KAS=
AN
enabled and the stock kernel (was I supposed to?).
Once again, thank you, Michael for your patience guiding me through the tas=
ks.
Next time will be easier. _-)
Now I notice there's another report for firmware_parser_create= +0xa9b/0xd90. What does faddr2line say for that? Though it's weird that KASAN claims the memory written in firmware_parser_create was freed from rcu_cpu_kthread. If that's accurate[0= ], it might indicate a lower level issue. [0] There is some doubt about that due to the "Frankenkernel monster&q= uot;. :) Does enabling CONFIG_DEBUG_INFO_REDUCED as well allow you to keep debugging symb= ols for everything, or at least for the vmlinuz image?
Created attachment 139288 [details]
journalctl log with KASAN_OUTLINE and kasan_multi_shot using a kernel compi=
led
with debug info
Ok, so here's what I did.
Since I did not have space for the kernel with both KASAN and debug info
enabled (the 590Mb was already with CONFIG_DEBUG_INFO_REDUCED), I got my ha=
nds
dirty and used nconfig to strip a ton of drivers I was pretty sure I didn't
need from the build (stuff like nouveau, industrial controllers, game-pads,
etc..). I got the kernel down to ~390Mb which was enough to install.
I have attached the journalctl log file with this new kernel
(linux-kasan-debug-stripped).
As for faddr2line output, here is the original command with the new kernel:
```
francisco@ZenBox [23:09:51]
[/usr/lib/modules/4.16.5-1-kasan-debug-stripped/kernel/drivers/gpu/drm/amd/=
amdgpu]=20
-> $ /usr/lib/modules/4.16.5-1-kasan-debug-stripped/build/scripts/faddr2=
line
amdgpu.ko firmware_parser_create+0xa70/0xd90
firmware_parser_create+0xa70/0xd90:
get_integrated_info_v11 at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1572
(inlined by) construct_integrated_info at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1714
(inlined by) bios_parser_create_integrated_info at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1755
(inlined by) bios_parser_construct at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1912
(inlined by) firmware_parser_create at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1927
```
And here is the "new" command with the new kernel:
```
francisco@ZenBox [23:10:27]
[/usr/lib/modules/4.16.5-1-kasan-debug-stripped/kernel/drivers/gpu/drm/amd/=
amdgpu]=20
-> $ /usr/lib/modules/4.16.5-1-kasan-debug-stripped/build/scripts/faddr2=
line
amdgpu.ko firmware_parser_create+0xa9b/0xd90
firmware_parser_create+0xa9b/0xd90:
get_integrated_info_v11 at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1574
(inlined by) construct_integrated_info at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1714
(inlined by) bios_parser_create_integrated_info at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1755
(inlined by) bios_parser_construct at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1912
(inlined by) firmware_parser_create at
/tmp/build/linux-kasan-debug-stripped/src/linux-4.16/drivers/gpu/drm/amd/am=
dgpu/../display/dc/bios/bios_parser2.c:1927
```
This is no longer a "franken-kernel-mosnter", does this output re=
veal anything
new?