BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

[Mikhail Gavrilov]

[-- Attachment #1: Type: text/plain, Size: 5582 bytes --]

Hi,
after enabling KASAN literally I was bombarded with messages about
slab-use-after-free in drm_sched_get_cleanup_job.
All messages has similar backtrace:
[ 1138.492091] ==================================================================
[ 1138.492104] BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492120] Read of size 8 at addr ffff88815adf04c0 by task sdma1/749

[ 1138.492130] CPU: 29 PID: 749 Comm: sdma1 Tainted: G        W    L
 -------  ---  6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
#1
[ 1138.492136] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
[ 1138.492141] Call Trace:
[ 1138.492145]  <TASK>
[ 1138.492150]  dump_stack_lvl+0x72/0xc0
[ 1138.492159]  print_report+0xcf/0x670
[ 1138.492169]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492181]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492193]  kasan_report+0xa4/0xe0
[ 1138.492200]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492215]  drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492229]  drm_sched_main+0x643/0x990 [gpu_sched]
[ 1138.492245]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1138.492259]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 1138.492269]  ? __kthread_parkme+0xc1/0x1f0
[ 1138.492277]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1138.492288]  kthread+0x29e/0x340
[ 1138.492294]  ? __pfx_kthread+0x10/0x10
[ 1138.492301]  ret_from_fork+0x2c/0x50
[ 1138.492314]  </TASK>

[ 1138.492320] Allocated by task 10867:
[ 1138.492323]  kasan_save_stack+0x2f/0x50
[ 1138.492329]  kasan_set_track+0x21/0x30
[ 1138.492334]  __kasan_kmalloc+0x8b/0x90
[ 1138.492339]  amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
[ 1138.493112]  drm_file_alloc+0x46e/0x880
[ 1138.493120]  drm_open_helper+0x161/0x460
[ 1138.493126]  drm_open+0x1e7/0x5c0
[ 1138.493131]  drm_stub_open+0x24d/0x400
[ 1138.493138]  chrdev_open+0x215/0x620
[ 1138.493144]  do_dentry_open+0x5f1/0x1000
[ 1138.493149]  path_openat+0x1b3d/0x28a0
[ 1138.493156]  do_filp_open+0x1bd/0x400
[ 1138.493161]  do_sys_openat2+0x140/0x420
[ 1138.493167]  __x64_sys_openat+0x11f/0x1d0
[ 1138.493173]  do_syscall_64+0x5b/0x80
[ 1138.493179]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1138.493189] Freed by task 10867:
[ 1138.493193]  kasan_save_stack+0x2f/0x50
[ 1138.493199]  kasan_set_track+0x21/0x30
[ 1138.493205]  kasan_save_free_info+0x2a/0x50
[ 1138.493210]  __kasan_slab_free+0x107/0x1a0
[ 1138.493216]  slab_free_freelist_hook+0x11e/0x1d0
[ 1138.493221]  __kmem_cache_free+0xbc/0x2e0
[ 1138.493227]  amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
[ 1138.493963]  drm_file_free.part.0+0x638/0xb70
[ 1138.493969]  drm_release+0x1ea/0x470
[ 1138.493975]  __fput+0x213/0x9e0
[ 1138.493981]  task_work_run+0x11b/0x200
[ 1138.493987]  exit_to_user_mode_prepare+0x23a/0x260
[ 1138.493994]  syscall_exit_to_user_mode+0x16/0x50
[ 1138.494001]  do_syscall_64+0x67/0x80
[ 1138.494006]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1138.494015] The buggy address belongs to the object at ffff88815adf0000
                which belongs to the cache kmalloc-4k of size 4096
[ 1138.494021] The buggy address is located 1216 bytes inside of
                freed 4096-byte region [ffff88815adf0000, ffff88815adf1000)

[ 1138.494030] The buggy address belongs to the physical page:
[ 1138.494034] page:0000000017cd5a82 refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x15adf0
[ 1138.494041] head:0000000017cd5a82 order:3 entire_mapcount:0
nr_pages_mapped:0 pincount:0
[ 1138.494046] flags:
0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 1138.494055] raw: 0017ffffc0010200 ffff88810004d040 dead000000000122
0000000000000000
[ 1138.494061] raw: 0000000000000000 0000000000040004 00000001ffffffff
0000000000000000
[ 1138.494065] page dumped because: kasan: bad access detected

[ 1138.494071] Memory state around the buggy address:
[ 1138.494076]  ffff88815adf0380: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494080]  ffff88815adf0400: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494084] >ffff88815adf0480: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494088]                                            ^
[ 1138.494092]  ffff88815adf0500: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494095]  ffff88815adf0580: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494099] ==================================================================
[ 1138.494140] Disabling lock debugging due to kernel taint

First thing I checked RAM for sure that it is definitely not a
hardware problem. I started investigating which application triggered
this message. I started to notice that it usually happens when any
game starts in the steam client. So I came to the culprit of the
problem is the compilation of shaders. For reproduction we should have
a kernel with enabled KASAN and a steam client. After allowing
background processing of Vulkan shaders the issue happens.
Unfortunately it does not happen immediately as you may expect.
I record four screencasts and uploaded them on to youtube [1-4] for
proving that definitely processing of Vulkan shaders  culprit here but
this may happens after some time.

[1] https://youtu.be/hFuWlGTbetg
[2] https://youtu.be/TAOqoYUYA5Q
[3] https://youtu.be/9SZNXWhwAZs
[4] https://youtu.be/W9gtYyL-2pg

I do not know which information would be also useful. Please ask me.

I attached a full kernel log and config if someone would be interested
to see it.

-- 
Best Regards,
Mike Gavrilov.

[-- Attachment #2: dmesg-slab-use-after-free-bug.tar.xz --]
[-- Type: application/octet-stream, Size: 35912 bytes --]

[-- Attachment #3: dmesg-slab-use-after-free-bug-2.tar.xz --]
[-- Type: application/octet-stream, Size: 36132 bytes --]

[-- Attachment #4: dmesg-slab-use-after-free-bug-3.tar.xz --]
[-- Type: application/octet-stream, Size: 36196 bytes --]

[-- Attachment #5: config-6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.tar.xz --]
[-- Type: application/x-xz, Size: 53996 bytes --]

Re: BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

[Christian König]

Hi Mikhail,

Am 20.03.23 um 13:05 schrieb Mikhail Gavrilov:
> Hi,
> after enabling KASAN literally I was bombarded with messages about
> slab-use-after-free in drm_sched_get_cleanup_job.

mhm, interesting.

> All messages has similar backtrace:
> [ 1138.492091] ==================================================================
> [ 1138.492104] BUG: KASAN: slab-use-after-free in
> drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492120] Read of size 8 at addr ffff88815adf04c0 by task sdma1/749
>
> [ 1138.492130] CPU: 29 PID: 749 Comm: sdma1 Tainted: G        W    L
>   -------  ---  6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
> #1
> [ 1138.492136] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
> [ 1138.492141] Call Trace:
> [ 1138.492145]  <TASK>
> [ 1138.492150]  dump_stack_lvl+0x72/0xc0
> [ 1138.492159]  print_report+0xcf/0x670
> [ 1138.492169]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492181]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492193]  kasan_report+0xa4/0xe0
> [ 1138.492200]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492215]  drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492229]  drm_sched_main+0x643/0x990 [gpu_sched]
> [ 1138.492245]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492259]  ? __pfx_autoremove_wake_function+0x10/0x10
> [ 1138.492269]  ? __kthread_parkme+0xc1/0x1f0
> [ 1138.492277]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492288]  kthread+0x29e/0x340
> [ 1138.492294]  ? __pfx_kthread+0x10/0x10
> [ 1138.492301]  ret_from_fork+0x2c/0x50
> [ 1138.492314]  </TASK>
>
> [ 1138.492320] Allocated by task 10867:
> [ 1138.492323]  kasan_save_stack+0x2f/0x50
> [ 1138.492329]  kasan_set_track+0x21/0x30
> [ 1138.492334]  __kasan_kmalloc+0x8b/0x90
> [ 1138.492339]  amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
> [ 1138.493112]  drm_file_alloc+0x46e/0x880
> [ 1138.493120]  drm_open_helper+0x161/0x460
> [ 1138.493126]  drm_open+0x1e7/0x5c0
> [ 1138.493131]  drm_stub_open+0x24d/0x400
> [ 1138.493138]  chrdev_open+0x215/0x620
> [ 1138.493144]  do_dentry_open+0x5f1/0x1000
> [ 1138.493149]  path_openat+0x1b3d/0x28a0
> [ 1138.493156]  do_filp_open+0x1bd/0x400
> [ 1138.493161]  do_sys_openat2+0x140/0x420
> [ 1138.493167]  __x64_sys_openat+0x11f/0x1d0
> [ 1138.493173]  do_syscall_64+0x5b/0x80
> [ 1138.493179]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.493189] Freed by task 10867:
> [ 1138.493193]  kasan_save_stack+0x2f/0x50
> [ 1138.493199]  kasan_set_track+0x21/0x30
> [ 1138.493205]  kasan_save_free_info+0x2a/0x50
> [ 1138.493210]  __kasan_slab_free+0x107/0x1a0
> [ 1138.493216]  slab_free_freelist_hook+0x11e/0x1d0
> [ 1138.493221]  __kmem_cache_free+0xbc/0x2e0
> [ 1138.493227]  amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
> [ 1138.493963]  drm_file_free.part.0+0x638/0xb70
> [ 1138.493969]  drm_release+0x1ea/0x470
> [ 1138.493975]  __fput+0x213/0x9e0
> [ 1138.493981]  task_work_run+0x11b/0x200
> [ 1138.493987]  exit_to_user_mode_prepare+0x23a/0x260
> [ 1138.493994]  syscall_exit_to_user_mode+0x16/0x50
> [ 1138.494001]  do_syscall_64+0x67/0x80
> [ 1138.494006]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.494015] The buggy address belongs to the object at ffff88815adf0000
>                  which belongs to the cache kmalloc-4k of size 4096
> [ 1138.494021] The buggy address is located 1216 bytes inside of
>                  freed 4096-byte region [ffff88815adf0000, ffff88815adf1000)
>
> [ 1138.494030] The buggy address belongs to the physical page:
> [ 1138.494034] page:0000000017cd5a82 refcount:1 mapcount:0
> mapping:0000000000000000 index:0x0 pfn:0x15adf0
> [ 1138.494041] head:0000000017cd5a82 order:3 entire_mapcount:0
> nr_pages_mapped:0 pincount:0
> [ 1138.494046] flags:
> 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
> [ 1138.494055] raw: 0017ffffc0010200 ffff88810004d040 dead000000000122
> 0000000000000000
> [ 1138.494061] raw: 0000000000000000 0000000000040004 00000001ffffffff
> 0000000000000000
> [ 1138.494065] page dumped because: kasan: bad access detected
>
> [ 1138.494071] Memory state around the buggy address:
> [ 1138.494076]  ffff88815adf0380: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494080]  ffff88815adf0400: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494084] >ffff88815adf0480: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494088]                                            ^
> [ 1138.494092]  ffff88815adf0500: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494095]  ffff88815adf0580: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494099] ==================================================================
> [ 1138.494140] Disabling lock debugging due to kernel taint
>
> First thing I checked RAM for sure that it is definitely not a
> hardware problem. I started investigating which application triggered
> this message. I started to notice that it usually happens when any
> game starts in the steam client. So I came to the culprit of the
> problem is the compilation of shaders. For reproduction we should have
> a kernel with enabled KASAN and a steam client. After allowing
> background processing of Vulkan shaders the issue happens.
> Unfortunately it does not happen immediately as you may expect.
> I record four screencasts and uploaded them on to youtube [1-4] for
> proving that definitely processing of Vulkan shaders  culprit here but
> this may happens after some time.
>
> [1] https://youtu.be/hFuWlGTbetg
> [2] https://youtu.be/TAOqoYUYA5Q
> [3] https://youtu.be/9SZNXWhwAZs
> [4] https://youtu.be/W9gtYyL-2pg
>
> I do not know which information would be also useful. Please ask me.
>
> I attached a full kernel log and config if someone would be interested
> to see it.

That looks like a reference counting issue to me.

I'm going to take a look, but we have already fixed one of those recently.

Probably best that you try this on drm-fixes, just to double check that 
this isn't the same issue.

Thanks,
Christian.

Re: BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

[Mikhail Gavrilov]

[-- Attachment #1: Type: text/plain, Size: 6393 bytes --]

On Tue, Mar 21, 2023 at 11:47 PM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> Hi Mikhail,
>
> That looks like a reference counting issue to me.
>
> I'm going to take a look, but we have already fixed one of those recently.
>
> Probably best that you try this on drm-fixes, just to double check that
> this isn't the same issue.
>

Hi Christian,
you meant this branch?
$ git clone -b drm-fixes git://anongit.freedesktop.org/drm/drm linux-drm

If yes I just checked and unfortunately see this issue unfixed there.

[ 1984.295833] ==================================================================
[ 1984.295876] BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.295898] Read of size 8 at addr ffff88814cadc4c0 by task sdma1/764

[ 1984.295924] CPU: 12 PID: 764 Comm: sdma1 Tainted: G        W    L
  6.3.0-rc3-drm-fixes+ #1
[ 1984.295937] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
[ 1984.295951] Call Trace:
[ 1984.295963]  <TASK>
[ 1984.295975]  dump_stack_lvl+0x72/0xc0
[ 1984.295991]  print_report+0xcf/0x670
[ 1984.296007]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296030]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296047]  kasan_report+0xa4/0xe0
[ 1984.296118]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296149]  drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296175]  drm_sched_main+0x643/0x990 [gpu_sched]
[ 1984.296204]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1984.296222]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 1984.296290]  ? __kthread_parkme+0xc1/0x1f0
[ 1984.296304]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1984.296321]  kthread+0x29e/0x340
[ 1984.296334]  ? __pfx_kthread+0x10/0x10
[ 1984.296501]  ret_from_fork+0x2c/0x50
[ 1984.296518]  </TASK>

[ 1984.296539] Allocated by task 12194:
[ 1984.296552]  kasan_save_stack+0x2f/0x50
[ 1984.296566]  kasan_set_track+0x21/0x30
[ 1984.296578]  __kasan_kmalloc+0x8b/0x90
[ 1984.296590]  amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
[ 1984.297051]  drm_file_alloc+0x46e/0x880
[ 1984.297064]  drm_open_helper+0x161/0x460
[ 1984.297076]  drm_open+0x1e7/0x5c0
[ 1984.297089]  drm_stub_open+0x24d/0x400
[ 1984.297107]  chrdev_open+0x215/0x620
[ 1984.297125]  do_dentry_open+0x5f1/0x1000
[ 1984.297146]  path_openat+0x1b3d/0x28a0
[ 1984.297164]  do_filp_open+0x1bd/0x400
[ 1984.297180]  do_sys_openat2+0x140/0x420
[ 1984.297197]  __x64_sys_openat+0x11f/0x1d0
[ 1984.297213]  do_syscall_64+0x5b/0x80
[ 1984.297231]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1984.297266] Freed by task 12195:
[ 1984.297284]  kasan_save_stack+0x2f/0x50
[ 1984.297303]  kasan_set_track+0x21/0x30
[ 1984.297323]  kasan_save_free_info+0x2a/0x50
[ 1984.297343]  __kasan_slab_free+0x107/0x1a0
[ 1984.297361]  slab_free_freelist_hook+0x11e/0x1d0
[ 1984.297373]  __kmem_cache_free+0xbc/0x2e0
[ 1984.297385]  amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
[ 1984.297821]  drm_file_free.part.0+0x638/0xb70
[ 1984.297834]  drm_release+0x1ea/0x470
[ 1984.297845]  __fput+0x213/0x9e0
[ 1984.297857]  task_work_run+0x11b/0x200
[ 1984.297869]  exit_to_user_mode_prepare+0x23a/0x260
[ 1984.297883]  syscall_exit_to_user_mode+0x16/0x50
[ 1984.297896]  do_syscall_64+0x67/0x80
[ 1984.297907]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1984.298033] Last potentially related work creation:
[ 1984.298044]  kasan_save_stack+0x2f/0x50
[ 1984.298057]  __kasan_record_aux_stack+0x97/0xb0
[ 1984.298075]  __call_rcu_common.constprop.0+0xf8/0x1af0
[ 1984.298095]  amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
[ 1984.298557]  amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
[ 1984.299055]  amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
[ 1984.299624]  drm_ioctl_kernel+0x1f8/0x3d0
[ 1984.299637]  drm_ioctl+0x4c1/0xaa0
[ 1984.299649]  amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
[ 1984.300083]  __x64_sys_ioctl+0x12d/0x1a0
[ 1984.300097]  do_syscall_64+0x5b/0x80
[ 1984.300109]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1984.300135] Second to last potentially related work creation:
[ 1984.300149]  kasan_save_stack+0x2f/0x50
[ 1984.300167]  __kasan_record_aux_stack+0x97/0xb0
[ 1984.300185]  __call_rcu_common.constprop.0+0xf8/0x1af0
[ 1984.300203]  amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
[ 1984.300692]  amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
[ 1984.301133]  amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
[ 1984.301577]  drm_ioctl_kernel+0x1f8/0x3d0
[ 1984.301598]  drm_ioctl+0x4c1/0xaa0
[ 1984.301610]  amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
[ 1984.302043]  __x64_sys_ioctl+0x12d/0x1a0
[ 1984.302056]  do_syscall_64+0x5b/0x80
[ 1984.302068]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

[ 1984.302090] The buggy address belongs to the object at ffff88814cadc000
                which belongs to the cache kmalloc-4k of size 4096
[ 1984.302103] The buggy address is located 1216 bytes inside of
                freed 4096-byte region [ffff88814cadc000, ffff88814cadd000)

[ 1984.302129] The buggy address belongs to the physical page:
[ 1984.302141] page:00000000103c1864 refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x14cad8
[ 1984.302155] head:00000000103c1864 order:3 entire_mapcount:0
nr_pages_mapped:0 pincount:0
[ 1984.302168] flags:
0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 1984.302184] raw: 0017ffffc0010200 ffff88810004d040 ffffea00311f0200
dead000000000002
[ 1984.302196] raw: 0000000000000000 0000000000040004 00000001ffffffff
0000000000000000
[ 1984.302208] page dumped because: kasan: bad access detected

[ 1984.302230] Memory state around the buggy address:
[ 1984.302241]  ffff88814cadc380: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302252]  ffff88814cadc400: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302263] >ffff88814cadc480: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302274]                                            ^
[ 1984.302285]  ffff88814cadc500: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302296]  ffff88814cadc580: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302307] ==================================================================
[ 1984.302339] Disabling lock debugging due to kernel taint

-- 
Best Regards,
Mike Gavrilov.

[-- Attachment #2: dmesg.tar.xz --]
[-- Type: application/x-xz, Size: 36212 bytes --]

Re: BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

[Christian König]

Am 23.03.23 um 20:05 schrieb Mikhail Gavrilov:
> On Tue, Mar 21, 2023 at 11:47 PM Christian König
> <ckoenig.leichtzumerken@gmail.com> wrote:
>> Hi Mikhail,
>>
>> That looks like a reference counting issue to me.
>>
>> I'm going to take a look, but we have already fixed one of those recently.
>>
>> Probably best that you try this on drm-fixes, just to double check that
>> this isn't the same issue.
>>
> Hi Christian,
> you meant this branch?
> $ git clone -b drm-fixes git://anongit.freedesktop.org/drm/drm linux-drm

Yeah, that one

Thanks for the info, looks like this isn't fixed.

Christian.

>
> If yes I just checked and unfortunately see this issue unfixed there.
>
> [ 1984.295833] ==================================================================
> [ 1984.295876] BUG: KASAN: slab-use-after-free in
> drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.295898] Read of size 8 at addr ffff88814cadc4c0 by task sdma1/764
>
> [ 1984.295924] CPU: 12 PID: 764 Comm: sdma1 Tainted: G        W    L
>    6.3.0-rc3-drm-fixes+ #1
> [ 1984.295937] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
> [ 1984.295951] Call Trace:
> [ 1984.295963]  <TASK>
> [ 1984.295975]  dump_stack_lvl+0x72/0xc0
> [ 1984.295991]  print_report+0xcf/0x670
> [ 1984.296007]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296030]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296047]  kasan_report+0xa4/0xe0
> [ 1984.296118]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296149]  drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296175]  drm_sched_main+0x643/0x990 [gpu_sched]
> [ 1984.296204]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1984.296222]  ? __pfx_autoremove_wake_function+0x10/0x10
> [ 1984.296290]  ? __kthread_parkme+0xc1/0x1f0
> [ 1984.296304]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1984.296321]  kthread+0x29e/0x340
> [ 1984.296334]  ? __pfx_kthread+0x10/0x10
> [ 1984.296501]  ret_from_fork+0x2c/0x50
> [ 1984.296518]  </TASK>
>
> [ 1984.296539] Allocated by task 12194:
> [ 1984.296552]  kasan_save_stack+0x2f/0x50
> [ 1984.296566]  kasan_set_track+0x21/0x30
> [ 1984.296578]  __kasan_kmalloc+0x8b/0x90
> [ 1984.296590]  amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
> [ 1984.297051]  drm_file_alloc+0x46e/0x880
> [ 1984.297064]  drm_open_helper+0x161/0x460
> [ 1984.297076]  drm_open+0x1e7/0x5c0
> [ 1984.297089]  drm_stub_open+0x24d/0x400
> [ 1984.297107]  chrdev_open+0x215/0x620
> [ 1984.297125]  do_dentry_open+0x5f1/0x1000
> [ 1984.297146]  path_openat+0x1b3d/0x28a0
> [ 1984.297164]  do_filp_open+0x1bd/0x400
> [ 1984.297180]  do_sys_openat2+0x140/0x420
> [ 1984.297197]  __x64_sys_openat+0x11f/0x1d0
> [ 1984.297213]  do_syscall_64+0x5b/0x80
> [ 1984.297231]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.297266] Freed by task 12195:
> [ 1984.297284]  kasan_save_stack+0x2f/0x50
> [ 1984.297303]  kasan_set_track+0x21/0x30
> [ 1984.297323]  kasan_save_free_info+0x2a/0x50
> [ 1984.297343]  __kasan_slab_free+0x107/0x1a0
> [ 1984.297361]  slab_free_freelist_hook+0x11e/0x1d0
> [ 1984.297373]  __kmem_cache_free+0xbc/0x2e0
> [ 1984.297385]  amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
> [ 1984.297821]  drm_file_free.part.0+0x638/0xb70
> [ 1984.297834]  drm_release+0x1ea/0x470
> [ 1984.297845]  __fput+0x213/0x9e0
> [ 1984.297857]  task_work_run+0x11b/0x200
> [ 1984.297869]  exit_to_user_mode_prepare+0x23a/0x260
> [ 1984.297883]  syscall_exit_to_user_mode+0x16/0x50
> [ 1984.297896]  do_syscall_64+0x67/0x80
> [ 1984.297907]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.298033] Last potentially related work creation:
> [ 1984.298044]  kasan_save_stack+0x2f/0x50
> [ 1984.298057]  __kasan_record_aux_stack+0x97/0xb0
> [ 1984.298075]  __call_rcu_common.constprop.0+0xf8/0x1af0
> [ 1984.298095]  amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
> [ 1984.298557]  amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
> [ 1984.299055]  amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
> [ 1984.299624]  drm_ioctl_kernel+0x1f8/0x3d0
> [ 1984.299637]  drm_ioctl+0x4c1/0xaa0
> [ 1984.299649]  amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
> [ 1984.300083]  __x64_sys_ioctl+0x12d/0x1a0
> [ 1984.300097]  do_syscall_64+0x5b/0x80
> [ 1984.300109]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.300135] Second to last potentially related work creation:
> [ 1984.300149]  kasan_save_stack+0x2f/0x50
> [ 1984.300167]  __kasan_record_aux_stack+0x97/0xb0
> [ 1984.300185]  __call_rcu_common.constprop.0+0xf8/0x1af0
> [ 1984.300203]  amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
> [ 1984.300692]  amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
> [ 1984.301133]  amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
> [ 1984.301577]  drm_ioctl_kernel+0x1f8/0x3d0
> [ 1984.301598]  drm_ioctl+0x4c1/0xaa0
> [ 1984.301610]  amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
> [ 1984.302043]  __x64_sys_ioctl+0x12d/0x1a0
> [ 1984.302056]  do_syscall_64+0x5b/0x80
> [ 1984.302068]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.302090] The buggy address belongs to the object at ffff88814cadc000
>                  which belongs to the cache kmalloc-4k of size 4096
> [ 1984.302103] The buggy address is located 1216 bytes inside of
>                  freed 4096-byte region [ffff88814cadc000, ffff88814cadd000)
>
> [ 1984.302129] The buggy address belongs to the physical page:
> [ 1984.302141] page:00000000103c1864 refcount:1 mapcount:0
> mapping:0000000000000000 index:0x0 pfn:0x14cad8
> [ 1984.302155] head:00000000103c1864 order:3 entire_mapcount:0
> nr_pages_mapped:0 pincount:0
> [ 1984.302168] flags:
> 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
> [ 1984.302184] raw: 0017ffffc0010200 ffff88810004d040 ffffea00311f0200
> dead000000000002
> [ 1984.302196] raw: 0000000000000000 0000000000040004 00000001ffffffff
> 0000000000000000
> [ 1984.302208] page dumped because: kasan: bad access detected
>
> [ 1984.302230] Memory state around the buggy address:
> [ 1984.302241]  ffff88814cadc380: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302252]  ffff88814cadc400: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302263] >ffff88814cadc480: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302274]                                            ^
> [ 1984.302285]  ffff88814cadc500: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302296]  ffff88814cadc580: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302307] ==================================================================
> [ 1984.302339] Disabling lock debugging due to kernel taint
>

Re: BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

[Mikhail Gavrilov]

[-- Attachment #1: Type: text/plain, Size: 3228 bytes --]

On Fri, Mar 24, 2023 at 7:37 PM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> Yeah, that one
>
> Thanks for the info, looks like this isn't fixed.
>
> Christian.
>

Hi,
glad to see that "BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0" was fixed in 6.3-rc5.
For history it would be good to know the commit which fixes this issue.
I waited for this moment because I know other one issue which was also
found by KASAN santiniser.

BUG: KASAN: null-ptr-deref in drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
Read of size 4 at addr 0000000000000078 by task GameThread/23915

CPU: 10 PID: 23915 Comm: GameThread Tainted: G        W    L
-------  ---  6.3.0-0.rc5.42.fc39.x86_64+debug #1
Hardware name: System manufacturer System Product Name/ROG STRIX
X570-I GAMING, BIOS 4601 02/02/2023
Call Trace:
 <TASK>
 dump_stack_lvl+0x72/0xc0
 kasan_report+0xa4/0xe0
 ? drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
 kasan_check_range+0x104/0x1b0
 drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
 ? __pfx_drm_sched_job_cleanup+0x10/0x10 [gpu_sched]
 ? slab_free_freelist_hook+0x11e/0x1d0
 ? amdgpu_cs_parser_fini+0x363/0x5a0 [amdgpu]
 amdgpu_job_free+0x40/0x1b0 [amdgpu]
 amdgpu_cs_parser_fini+0x3c9/0x5a0 [amdgpu]
 ? __pfx_amdgpu_cs_parser_fini+0x10/0x10 [amdgpu]
 amdgpu_cs_ioctl+0x3d9/0x5630 [amdgpu]
 ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
 ? mark_lock+0x101/0x16e0
 ? __lock_acquire+0xe54/0x59f0
 ? __pfx_lock_release+0x10/0x10
 ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
 drm_ioctl_kernel+0x1f8/0x3d0
 ? __pfx_drm_ioctl_kernel+0x10/0x10
 drm_ioctl+0x4c1/0xaa0
 ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
 ? __pfx_drm_ioctl+0x10/0x10
 ? _raw_spin_unlock_irqrestore+0x62/0x80
 ? lockdep_hardirqs_on+0x7d/0x100
 ? _raw_spin_unlock_irqrestore+0x4b/0x80
 amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
 __x64_sys_ioctl+0x12d/0x1a0
 do_syscall_64+0x5c/0x90
 ? do_syscall_64+0x68/0x90
 ? lockdep_hardirqs_on+0x7d/0x100
 ? do_syscall_64+0x68/0x90
 ? do_syscall_64+0x68/0x90
 ? lockdep_hardirqs_on+0x7d/0x100
 ? do_syscall_64+0x68/0x90
 ? do_syscall_64+0x68/0x90
 ? lockdep_hardirqs_on+0x7d/0x100
 entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fe97a50881d
Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00
00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2
3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
RSP: 002b:000000007c35d3f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 000000007c35d6e8 RCX: 00007fe97a50881d
RDX: 000000007c35d4d0 RSI: 00000000c0186444 RDI: 00000000000000ae
RBP: 000000007c35d440 R08: 00007fe8fc0f0970 R09: 000000007c35d490
R10: 000000007fb79000 R11: 0000000000000246 R12: 000000007c35d4d0
R13: 00000000c0186444 R14: 00000000000000ae R15: 00007fe8fc0f0900
 </TASK>

I know at least 3 games which 100% triggering this bug:
- Cyberpunk 2077
- Forza Horizon 4
- Forza Horizon 5

We would continue to discuss it here or better create a new thread
(for someone who is also faced with this issue could easily find a
solution on the internet)?

A full kernel log as usual attached here.

-- 
Best Regards,
Mike Gavrilov.

[-- Attachment #2: dmesg.tar.xz --]
[-- Type: application/x-xz, Size: 40976 bytes --]