* [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() @ 2024-02-23 2:10 Tetsuo Handa 2024-02-23 2:27 ` Yosry Ahmed 2024-02-23 4:43 ` Sergey Senozhatsky 0 siblings, 2 replies; 17+ messages in thread From: Tetsuo Handa @ 2024-02-23 2:10 UTC (permalink / raw) To: Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, Sergey Senozhatsky Cc: linux-mm I can observe this bug during evict_folios() from 6.7.0 to 6.8.0-rc5-00163-gffd2cb6b718e. Since I haven't observed with 6.6.0, this bug might be introduced in 6.7 cycle. ---------------------------------------- [ 0.000000][ T0] Linux version 6.8.0-rc5-00163-gffd2cb6b718e (root@ubuntu) (Ubuntu clang version 14.0.0-1ubuntu1.1, Ubuntu LLD 14.0.0) #1094 SMP PREEMPT_DYNAMIC Fri Feb 23 01:45:21 UTC 2024 [ 50.026544][ T2974] ===================================================== [ 50.030627][ T2974] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 [ 50.034611][ T2974] obj_malloc+0x6cc/0x7b0 obj_malloc at mm/zsmalloc.c:0 [ 50.037250][ T2974] zs_malloc+0xdbd/0x1400 zs_malloc at mm/zsmalloc.c:0 [ 50.039852][ T2974] zs_zpool_malloc+0xa5/0x1b0 zs_zpool_malloc at mm/zsmalloc.c:372 [ 50.044707][ T2974] zpool_malloc+0x110/0x150 zpool_malloc at mm/zpool.c:258 [ 50.049607][ T2974] zswap_store+0x2bbb/0x3d30 zswap_store at mm/zswap.c:1637 [ 50.054463][ T2974] swap_writepage+0x15b/0x4f0 swap_writepage at mm/page_io.c:198 [ 50.059392][ T2974] pageout+0x41d/0xef0 pageout at mm/vmscan.c:654 [ 50.064057][ T2974] shrink_folio_list+0x4d7a/0x7480 shrink_folio_list at mm/vmscan.c:1316 [ 50.069176][ T2974] evict_folios+0x30f1/0x5170 evict_folios at mm/vmscan.c:4521 [ 50.074082][ T2974] try_to_shrink_lruvec+0x983/0xd20 [ 50.079352][ T2974] shrink_one+0x72d/0xeb0 [ 50.084061][ T2974] shrink_many+0x70d/0x10b0 [ 50.088859][ T2974] lru_gen_shrink_node+0x577/0x850 [ 50.094192][ T2974] shrink_node+0x13d/0x1de0 [ 50.099028][ T2974] shrink_zones+0x878/0x14a0 [ 50.103958][ T2974] do_try_to_free_pages+0x2ac/0x16a0 [ 50.109138][ T2974] try_to_free_pages+0xd9e/0x1910 [ 50.114190][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 [ 50.119555][ T2974] __alloc_pages+0xb8c/0x1050 [ 50.124472][ T2974] alloc_pages_mpol+0x8e0/0xc80 [ 50.129367][ T2974] alloc_pages+0x224/0x240 [ 50.134022][ T2974] pipe_write+0xabe/0x2ba0 [ 50.138632][ T2974] vfs_write+0xfb0/0x1b80 [ 50.143171][ T2974] ksys_write+0x275/0x500 [ 50.147723][ T2974] __x64_sys_write+0xdf/0x120 [ 50.152431][ T2974] do_syscall_64+0xd1/0x1b0 [ 50.157106][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b [ 50.162382][ T2974] [ 50.165956][ T2974] Uninit was stored to memory at: [ 50.170819][ T2974] obj_malloc+0x70a/0x7b0 set_freeobj at mm/zsmalloc.c:476 (inlined by) obj_malloc at mm/zsmalloc.c:1333 [ 50.175341][ T2974] zs_malloc+0xdbd/0x1400 zs_malloc at mm/zsmalloc.c:0 [ 50.179923][ T2974] zs_zpool_malloc+0xa5/0x1b0 zs_zpool_malloc at mm/zsmalloc.c:372 [ 50.184636][ T2974] zpool_malloc+0x110/0x150 zpool_malloc at mm/zpool.c:258 [ 50.189257][ T2974] zswap_store+0x2bbb/0x3d30 zswap_store at mm/zswap.c:1637 [ 50.193918][ T2974] swap_writepage+0x15b/0x4f0 swap_writepage at mm/page_io.c:198 [ 50.198615][ T2974] pageout+0x41d/0xef0 pageout at mm/vmscan.c:654 [ 50.203012][ T2974] shrink_folio_list+0x4d7a/0x7480 shrink_folio_list at mm/vmscan.c:1316 [ 50.207772][ T2974] evict_folios+0x30f1/0x5170 evict_folios at mm/vmscan.c:4521 [ 50.212321][ T2974] try_to_shrink_lruvec+0x983/0xd20 [ 50.217092][ T2974] shrink_one+0x72d/0xeb0 [ 50.221441][ T2974] shrink_many+0x70d/0x10b0 [ 50.225891][ T2974] lru_gen_shrink_node+0x577/0x850 [ 50.230614][ T2974] shrink_node+0x13d/0x1de0 [ 50.235128][ T2974] shrink_zones+0x878/0x14a0 [ 50.239646][ T2974] do_try_to_free_pages+0x2ac/0x16a0 [ 50.244461][ T2974] try_to_free_pages+0xd9e/0x1910 [ 50.249151][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 [ 50.254148][ T2974] __alloc_pages+0xb8c/0x1050 [ 50.258679][ T2974] alloc_pages_mpol+0x8e0/0xc80 [ 50.263289][ T2974] alloc_pages+0x224/0x240 [ 50.267767][ T2974] pipe_write+0xabe/0x2ba0 [ 50.272190][ T2974] vfs_write+0xfb0/0x1b80 [ 50.276543][ T2974] ksys_write+0x275/0x500 [ 50.280931][ T2974] __x64_sys_write+0xdf/0x120 [ 50.289451][ T2974] do_syscall_64+0xd1/0x1b0 [ 50.303402][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b [ 50.318721][ T2974] [ 50.328931][ T2974] Uninit was created at: [ 50.341845][ T2974] free_unref_page_prepare+0x130/0xfc0 arch_static_branch_jump at arch/x86/include/asm/jump_label.h:55 (inlined by) memcg_kmem_online at include/linux/memcontrol.h:1840 (inlined by) free_pages_prepare at mm/page_alloc.c:1096 (inlined by) free_unref_page_prepare at mm/page_alloc.c:2346 [ 50.356492][ T2974] free_unref_page_list+0x139/0x1050 free_unref_page_list at mm/page_alloc.c:2532 [ 50.370898][ T2974] shrink_folio_list+0x7139/0x7480 list_empty at include/linux/list.h:373 (inlined by) list_splice at include/linux/list.h:545 (inlined by) shrink_folio_list at mm/vmscan.c:1490 [ 50.385025][ T2974] evict_folios+0x30f1/0x5170 evict_folios at mm/vmscan.c:4521 [ 50.398448][ T2974] try_to_shrink_lruvec+0x983/0xd20 [ 50.412660][ T2974] shrink_one+0x72d/0xeb0 [ 50.425591][ T2974] shrink_many+0x70d/0x10b0 [ 50.438827][ T2974] lru_gen_shrink_node+0x577/0x850 [ 50.454390][ T2974] shrink_node+0x13d/0x1de0 [ 50.479401][ T2974] shrink_zones+0x878/0x14a0 [ 50.529610][ T2974] do_try_to_free_pages+0x2ac/0x16a0 [ 50.544397][ T2974] try_to_free_pages+0xd9e/0x1910 [ 50.559556][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 [ 50.574932][ T2974] __alloc_pages+0xb8c/0x1050 [ 50.589024][ T2974] alloc_pages_mpol+0x8e0/0xc80 [ 50.603421][ T2974] alloc_pages+0x224/0x240 [ 50.616483][ T2974] pipe_write+0xabe/0x2ba0 [ 50.629601][ T2974] vfs_write+0xfb0/0x1b80 [ 50.643009][ T2974] ksys_write+0x275/0x500 [ 50.656157][ T2974] __x64_sys_write+0xdf/0x120 [ 50.670080][ T2974] do_syscall_64+0xd1/0x1b0 [ 50.683405][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b [ 50.698626][ T2974] ---------------------------------------- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 2:10 [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Tetsuo Handa @ 2024-02-23 2:27 ` Yosry Ahmed 2024-02-23 4:48 ` Sergey Senozhatsky 2024-02-23 4:43 ` Sergey Senozhatsky 1 sibling, 1 reply; 17+ messages in thread From: Yosry Ahmed @ 2024-02-23 2:27 UTC (permalink / raw) To: Tetsuo Handa Cc: Johannes Weiner, Nhat Pham, Minchan Kim, Sergey Senozhatsky, linux-mm On Thu, Feb 22, 2024 at 6:10 PM Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote: > > I can observe this bug during evict_folios() from 6.7.0 to 6.8.0-rc5-00163-gffd2cb6b718e. > Since I haven't observed with 6.6.0, this bug might be introduced in 6.7 cycle. I am not familiar with KMSAN bug reports, but it seems like it's reporting a user-after-free for zspage->freeobj. The report says it was created in free_unref_page_prepare() during lruvec reclaim, and I am not sure how that's possible given that zspage is allocated from the slab allocator. Perhaps I am mis-interpreting the report. I also don't see any recent changes in mm/zsmalloc.c that modify this code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and Sergey, I don't think zswap is an active actor in this bug report. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 2:27 ` Yosry Ahmed @ 2024-02-23 4:48 ` Sergey Senozhatsky 2024-02-23 4:50 ` Yosry Ahmed 2024-02-23 5:23 ` Chengming Zhou 0 siblings, 2 replies; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 4:48 UTC (permalink / raw) To: Yosry Ahmed Cc: Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, Sergey Senozhatsky, linux-mm On (24/02/22 18:27), Yosry Ahmed wrote: > I also don't see any recent changes in mm/zsmalloc.c that modify this > code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > Sergey, I don't think zswap is an active actor in this bug report. Yeah. [1] are the only recent zsmalloc patches I can recall, and those patches touch zsmalloc locking (zspages migration/compaction). https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:48 ` Sergey Senozhatsky @ 2024-02-23 4:50 ` Yosry Ahmed 2024-02-23 4:56 ` Sergey Senozhatsky 2024-02-23 5:23 ` Chengming Zhou 1 sibling, 1 reply; 17+ messages in thread From: Yosry Ahmed @ 2024-02-23 4:50 UTC (permalink / raw) To: Sergey Senozhatsky Cc: Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On Thu, Feb 22, 2024 at 8:48 PM Sergey Senozhatsky <senozhatsky@chromium.org> wrote: > > On (24/02/22 18:27), Yosry Ahmed wrote: > > I also don't see any recent changes in mm/zsmalloc.c that modify this > > code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > > Sergey, I don't think zswap is an active actor in this bug report. > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > patches touch zsmalloc locking (zspages migration/compaction). > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ These are not in 6.8.0-rc5 anyway, right? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:50 ` Yosry Ahmed @ 2024-02-23 4:56 ` Sergey Senozhatsky 2024-02-23 4:58 ` Sergey Senozhatsky 0 siblings, 1 reply; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 4:56 UTC (permalink / raw) To: Yosry Ahmed Cc: Sergey Senozhatsky, Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On (24/02/22 20:50), Yosry Ahmed wrote: > On Thu, Feb 22, 2024 at 8:48 PM Sergey Senozhatsky > <senozhatsky@chromium.org> wrote: > > > > On (24/02/22 18:27), Yosry Ahmed wrote: > > > I also don't see any recent changes in mm/zsmalloc.c that modify this > > > code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > > > Sergey, I don't think zswap is an active actor in this bug report. > > > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > > patches touch zsmalloc locking (zspages migration/compaction). > > > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ > > These are not in 6.8.0-rc5 anyway, right? I see them in next-20240223, which seems to be 6.8-rc6 (according to Makefile) ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:56 ` Sergey Senozhatsky @ 2024-02-23 4:58 ` Sergey Senozhatsky 2024-02-23 5:05 ` Yosry Ahmed 0 siblings, 1 reply; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 4:58 UTC (permalink / raw) To: Sergey Senozhatsky Cc: Yosry Ahmed, Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On (24/02/23 13:56), Sergey Senozhatsky wrote: > On (24/02/22 20:50), Yosry Ahmed wrote: > > On Thu, Feb 22, 2024 at 8:48 PM Sergey Senozhatsky > > <senozhatsky@chromium.org> wrote: > > > > > > On (24/02/22 18:27), Yosry Ahmed wrote: > > > > I also don't see any recent changes in mm/zsmalloc.c that modify this > > > > code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > > > > Sergey, I don't think zswap is an active actor in this bug report. > > > > > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > > > patches touch zsmalloc locking (zspages migration/compaction). > > > > > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ > > > > These are not in 6.8.0-rc5 anyway, right? > > I see them in next-20240223, which seems to be 6.8-rc6 (according to ^ -rc5 But they look more or less correct to me, so I'm not blaming those patches. We should be protected by pool->look. Bisection would help us a lot, I think. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:58 ` Sergey Senozhatsky @ 2024-02-23 5:05 ` Yosry Ahmed 2024-02-23 5:19 ` Sergey Senozhatsky 0 siblings, 1 reply; 17+ messages in thread From: Yosry Ahmed @ 2024-02-23 5:05 UTC (permalink / raw) To: Sergey Senozhatsky Cc: Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On Thu, Feb 22, 2024 at 8:58 PM Sergey Senozhatsky <senozhatsky@chromium.org> wrote: > > On (24/02/23 13:56), Sergey Senozhatsky wrote: > > On (24/02/22 20:50), Yosry Ahmed wrote: > > > On Thu, Feb 22, 2024 at 8:48 PM Sergey Senozhatsky > > > <senozhatsky@chromium.org> wrote: > > > > > > > > On (24/02/22 18:27), Yosry Ahmed wrote: > > > > > I also don't see any recent changes in mm/zsmalloc.c that modify this > > > > > code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > > > > > Sergey, I don't think zswap is an active actor in this bug report. > > > > > > > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > > > > patches touch zsmalloc locking (zspages migration/compaction). > > > > > > > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ > > > > > > These are not in 6.8.0-rc5 anyway, right? > > > > I see them in next-20240223, which seems to be 6.8-rc6 (according to > ^ -rc5 > > But they look more or less correct to me, so I'm not blaming those > patches. We should be protected by pool->look. Bisection would help > us a lot, I think. Andrew picked up those patches in mm-unstable, which is included in linux-next at some point IIUC, but the patches there don't all end up in the next rc unless I am misunderstanding something here. These patches should be headed to v6.9 AFAICT. Actually, if I am not mistaken the patches were sent *after* v.6.8-rc5 was out, and it's not common for non-fixes to make it into rc releases anyway, right? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 5:05 ` Yosry Ahmed @ 2024-02-23 5:19 ` Sergey Senozhatsky 0 siblings, 0 replies; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 5:19 UTC (permalink / raw) To: Yosry Ahmed Cc: Sergey Senozhatsky, Tetsuo Handa, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On (24/02/22 21:05), Yosry Ahmed wrote: > > > > These are not in 6.8.0-rc5 anyway, right? > > > > > > I see them in next-20240223, which seems to be 6.8-rc6 (according to > > ^ -rc5 > > > > But they look more or less correct to me, so I'm not blaming those > > patches. We should be protected by pool->look. Bisection would help > > us a lot, I think. > > Andrew picked up those patches in mm-unstable, which is included in > linux-next at some point IIUC, but the patches there don't all end up > in the next rc unless I am misunderstanding something here. These > patches should be headed to v6.9 AFAICT. > > Actually, if I am not mistaken the patches were sent *after* v.6.8-rc5 > was out, and it's not common for non-fixes to make it into rc releases > anyway, right? Oh, sorry, I realized that we talked about different 6.8-rc5. I talked about linux-next, not Linus's tree. You are absolutely right, those patches are not in Linus's 6.8-rc5 and are headed to 6.9, yes. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:48 ` Sergey Senozhatsky 2024-02-23 4:50 ` Yosry Ahmed @ 2024-02-23 5:23 ` Chengming Zhou 2024-02-23 5:29 ` Sergey Senozhatsky 2024-02-23 9:26 ` Tetsuo Handa 1 sibling, 2 replies; 17+ messages in thread From: Chengming Zhou @ 2024-02-23 5:23 UTC (permalink / raw) To: Tetsuo Handa, Sergey Senozhatsky, Yosry Ahmed Cc: Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On 2024/2/23 12:48, Sergey Senozhatsky wrote: > On (24/02/22 18:27), Yosry Ahmed wrote: >> I also don't see any recent changes in mm/zsmalloc.c that modify this >> code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and >> Sergey, I don't think zswap is an active actor in this bug report. > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > patches touch zsmalloc locking (zspages migration/compaction). > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ > I think these patches can't go into 6.8.0-rc5, right? So it maybe a bug with the current code of zsmalloc (maybe zswap? I don't know). Tetsuo, could you please check if the config has CONFIG_COMPACTION enabled? Since the first patch of that series did fix a locking bug of migration: (mm/zsmalloc: fix migrate_write_lock() when !CONFIG_COMPACTION) Thanks. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 5:23 ` Chengming Zhou @ 2024-02-23 5:29 ` Sergey Senozhatsky 2024-02-23 9:26 ` Tetsuo Handa 1 sibling, 0 replies; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 5:29 UTC (permalink / raw) To: Chengming Zhou Cc: Tetsuo Handa, Sergey Senozhatsky, Yosry Ahmed, Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On (24/02/23 13:23), Chengming Zhou wrote: > On 2024/2/23 12:48, Sergey Senozhatsky wrote: > > On (24/02/22 18:27), Yosry Ahmed wrote: > >> I also don't see any recent changes in mm/zsmalloc.c that modify this > >> code, so maybe it wasn't introduce in 6.7. I will defer to Minchan and > >> Sergey, I don't think zswap is an active actor in this bug report. > > > > Yeah. [1] are the only recent zsmalloc patches I can recall, and those > > patches touch zsmalloc locking (zspages migration/compaction). > > > > https://lore.kernel.org/lkml/20240219-b4-szmalloc-migrate-v1-0-34cd49c6545b@bytedance.com/ > > > > I think these patches can't go into 6.8.0-rc5, right? Only if 6.8-rc5 is linux-next. But the report is (that was not immediately apparent to me, somehow) for Linus's tree, so those zsmalloc patches are out of any suspicions. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 5:23 ` Chengming Zhou 2024-02-23 5:29 ` Sergey Senozhatsky @ 2024-02-23 9:26 ` Tetsuo Handa 2024-02-23 10:10 ` Chengming Zhou 1 sibling, 1 reply; 17+ messages in thread From: Tetsuo Handa @ 2024-02-23 9:26 UTC (permalink / raw) To: Chengming Zhou, Sergey Senozhatsky, Yosry Ahmed Cc: Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On 2024/02/23 14:23, Chengming Zhou wrote: > Tetsuo, could you please check if the config has CONFIG_COMPACTION enabled? Yes, CONFIG_COMPACTION is enabled. Also, I can observe this problem with 6.8.0-rc5-next-20240223. ---------------------------------------- [ 54.589642][ T157] ===================================================== [ 54.603721][ T157] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 [ 54.608092][ T157] obj_malloc+0x6cc/0x7b0 [ 54.610904][ T157] zs_malloc+0xda2/0x12d0 [ 54.613688][ T157] zs_zpool_malloc+0xa5/0x1b0 [ 54.619163][ T157] zpool_malloc+0x113/0x150 [ 54.624449][ T157] zswap_compress+0x69b/0xbd0 [ 54.629904][ T157] zswap_store+0x1f24/0x2d00 [ 54.635026][ T157] swap_writepage+0x15b/0x4f0 [ 54.640023][ T157] pageout+0x3d4/0xeb0 [ 54.644699][ T157] shrink_folio_list+0x4d7f/0x7480 [ 54.649867][ T157] evict_folios+0x2160/0x52c0 [ 54.654872][ T157] try_to_shrink_lruvec+0x1cb/0x460 [ 54.660074][ T157] shrink_one+0x72d/0xeb0 [ 54.664922][ T157] shrink_many+0x70d/0x10c0 [ 54.669849][ T157] lru_gen_shrink_node+0x832/0xd10 [ 54.675110][ T157] shrink_node+0x13a/0x1dd0 [ 54.680026][ T157] balance_pgdat+0x1556/0x2740 [ 54.685032][ T157] kswapd+0x50d/0x870 [ 54.689643][ T157] kthread+0x485/0x600 [ 54.694432][ T157] ret_from_fork+0xfa/0x140 [ 54.699305][ T157] ret_from_fork_asm+0x11/0x20 [ 54.704295][ T157] [ 54.707905][ T157] Uninit was stored to memory at: [ 54.712837][ T157] obj_malloc+0x70a/0x7b0 [ 54.717434][ T157] zs_malloc+0xda2/0x12d0 [ 54.722009][ T157] zs_zpool_malloc+0xa5/0x1b0 [ 54.726806][ T157] zpool_malloc+0x113/0x150 [ 54.731507][ T157] zswap_compress+0x69b/0xbd0 [ 54.736299][ T157] zswap_store+0x1f24/0x2d00 [ 54.741081][ T157] swap_writepage+0x15b/0x4f0 [ 54.745880][ T157] pageout+0x3d4/0xeb0 [ 54.750386][ T157] shrink_folio_list+0x4d7f/0x7480 [ 54.755378][ T157] evict_folios+0x2160/0x52c0 [ 54.760153][ T157] try_to_shrink_lruvec+0x1cb/0x460 [ 54.765223][ T157] shrink_one+0x72d/0xeb0 [ 54.769870][ T157] shrink_many+0x70d/0x10c0 [ 54.774445][ T157] lru_gen_shrink_node+0x832/0xd10 [ 54.779221][ T157] shrink_node+0x13a/0x1dd0 [ 54.783965][ T157] balance_pgdat+0x1556/0x2740 [ 54.788702][ T157] kswapd+0x50d/0x870 [ 54.793073][ T157] kthread+0x485/0x600 [ 54.798253][ T157] ret_from_fork+0xfa/0x140 [ 54.804206][ T157] ret_from_fork_asm+0x11/0x20 [ 54.809016][ T157] [ 54.812652][ T157] Uninit was created at: [ 54.817314][ T157] free_unref_page_prepare+0x130/0xfc0 [ 54.822499][ T157] free_unref_page_list+0x13f/0x1130 [ 54.828207][ T157] shrink_folio_list+0x713e/0x7480 [ 54.834143][ T157] evict_folios+0x2160/0x52c0 [ 54.839358][ T157] try_to_shrink_lruvec+0x1cb/0x460 [ 54.844628][ T157] shrink_one+0x72d/0xeb0 [ 54.849436][ T157] shrink_many+0x70d/0x10c0 [ 54.854310][ T157] lru_gen_shrink_node+0x832/0xd10 [ 54.859337][ T157] shrink_node+0x13a/0x1dd0 [ 54.864076][ T157] shrink_zones+0x787/0x1530 [ 54.868808][ T157] do_try_to_free_pages+0x2ac/0x16a0 [ 54.873865][ T157] try_to_free_pages+0xddb/0x19b0 [ 54.878795][ T157] __alloc_pages_slowpath+0x1a05/0x2d00 [ 54.883978][ T157] __alloc_pages+0xc6c/0x1040 [ 54.888802][ T157] alloc_pages_mpol+0x477/0xc40 [ 54.893629][ T157] alloc_pages+0x224/0x240 [ 54.898092][ T157] pipe_write+0xae5/0x2bd0 [ 54.902702][ T157] vfs_write+0xfb9/0x1b90 [ 54.907117][ T157] ksys_write+0x275/0x500 [ 54.911612][ T157] __x64_sys_write+0xdf/0x120 [ 54.916287][ T157] do_syscall_64+0xd5/0x1c0 [ 54.920782][ T157] entry_SYSCALL_64_after_hwframe+0x62/0x6a [ 54.925972][ T157] [ 54.929436][ T157] CPU: 4 PID: 157 Comm: kswapd1 Not tainted 6.8.0-rc5-next-20240223 #1 [ 54.937592][ T157] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 54.946147][ T157] ===================================================== [ 54.951772][ T157] Disabling lock debugging due to kernel taint [ 54.957040][ T157] Kernel panic - not syncing: kmsan.panic set ... [ 54.962443][ T157] CPU: 4 PID: 157 Comm: kswapd1 Tainted: G B 6.8.0-rc5-next-20240223 #1 [ 54.971295][ T157] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 54.979856][ T157] Call Trace: [ 54.983760][ T157] <TASK> [ 54.987503][ T157] dump_stack_lvl+0x24b/0x300 [ 54.992068][ T157] dump_stack+0x29/0x30 [ 54.996373][ T157] panic+0x4ed/0xca0 [ 55.000656][ T157] kmsan_report+0x2d1/0x2e0 [ 55.005155][ T157] ? kmem_cache_alloc+0x707/0xf50 [ 55.009909][ T157] ? kmsan_internal_poison_memory+0x7d/0x90 [ 55.015056][ T157] ? kmsan_internal_poison_memory+0x49/0x90 [ 55.020253][ T157] ? kmsan_slab_alloc+0xdf/0x160 [ 55.024995][ T157] ? __msan_warning+0x91/0x120 [ 55.029604][ T157] ? obj_malloc+0x6cc/0x7b0 [ 55.034166][ T157] ? zs_malloc+0xda2/0x12d0 [ 55.038692][ T157] ? zs_zpool_malloc+0xa5/0x1b0 [ 55.043342][ T157] ? zpool_malloc+0x113/0x150 [ 55.047909][ T157] ? zswap_compress+0x69b/0xbd0 [ 55.052576][ T157] ? zswap_store+0x1f24/0x2d00 [ 55.057213][ T157] ? swap_writepage+0x15b/0x4f0 [ 55.061836][ T157] ? pageout+0x3d4/0xeb0 [ 55.066216][ T157] ? shrink_folio_list+0x4d7f/0x7480 [ 55.071083][ T157] ? evict_folios+0x2160/0x52c0 [ 55.075734][ T157] ? try_to_shrink_lruvec+0x1cb/0x460 [ 55.080625][ T157] ? shrink_one+0x72d/0xeb0 [ 55.085139][ T157] ? shrink_many+0x70d/0x10c0 [ 55.089752][ T157] ? lru_gen_shrink_node+0x832/0xd10 [ 55.094614][ T157] ? shrink_node+0x13a/0x1dd0 [ 55.099188][ T157] ? balance_pgdat+0x1556/0x2740 [ 55.103891][ T157] ? kswapd+0x50d/0x870 [ 55.108212][ T157] ? kthread+0x485/0x600 [ 55.112459][ T157] ? ret_from_fork+0xfa/0x140 [ 55.116849][ T157] ? ret_from_fork_asm+0x11/0x20 [ 55.121438][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 55.126388][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 55.131446][ T157] ? should_fail_ex+0x91/0xa20 [ 55.136530][ T157] ? kmsan_get_metadata+0x146/0x1c0 [ 55.141199][ T157] ? kmsan_get_metadata+0x146/0x1c0 [ 55.145956][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 55.150928][ T157] ? __should_failslab+0x24f/0x2e0 [ 55.155750][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 55.161123][ T157] ? __should_failslab+0x24f/0x2e0 [ 55.165918][ T157] ? kmsan_get_metadata+0x146/0x1c0 [ 55.170723][ T157] ? kmsan_get_metadata+0x146/0x1c0 [ 55.175568][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 55.180684][ T157] __msan_warning+0x91/0x120 [ 55.185066][ T157] obj_malloc+0x6cc/0x7b0 [ 55.189320][ T157] ? kmsan_get_metadata+0x146/0x1c0 [ 55.194051][ T157] zs_malloc+0xda2/0x12d0 [ 55.198333][ T157] zs_zpool_malloc+0xa5/0x1b0 [ 55.202886][ T157] ? zs_zpool_destroy+0x50/0x50 [ 55.207378][ T157] zpool_malloc+0x113/0x150 [ 55.211829][ T157] zswap_compress+0x69b/0xbd0 [ 55.216298][ T157] zswap_store+0x1f24/0x2d00 [ 55.220727][ T157] swap_writepage+0x15b/0x4f0 [ 55.225186][ T157] ? generic_swapfile_activate+0xed0/0xed0 [ 55.230120][ T157] pageout+0x3d4/0xeb0 [ 55.234272][ T157] shrink_folio_list+0x4d7f/0x7480 [ 55.239002][ T157] evict_folios+0x2160/0x52c0 [ 55.243455][ T157] try_to_shrink_lruvec+0x1cb/0x460 [ 55.248119][ T157] shrink_one+0x72d/0xeb0 [ 55.252389][ T157] shrink_many+0x70d/0x10c0 [ 55.257702][ T157] lru_gen_shrink_node+0x832/0xd10 [ 55.262478][ T157] shrink_node+0x13a/0x1dd0 [ 55.266848][ T157] ? mem_cgroup_soft_limit_reclaim+0x34/0x17b0 [ 55.271983][ T157] ? filter_irq_stacks+0xb9/0x230 [ 55.276677][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 55.281724][ T157] ? kswapd_age_node+0x63/0xb00 [ 55.286322][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 55.291458][ T157] balance_pgdat+0x1556/0x2740 [ 55.295936][ T157] ? finish_wait+0x2f1/0x4a0 [ 55.300332][ T157] kswapd+0x50d/0x870 [ 55.304457][ T157] kthread+0x485/0x600 [ 55.308674][ T157] ? shrink_all_memory+0x3a0/0x3a0 [ 55.313311][ T157] ? kthread_blkcg+0x120/0x120 [ 55.317805][ T157] ret_from_fork+0xfa/0x140 [ 55.322138][ T157] ? kthread_blkcg+0x120/0x120 [ 55.326615][ T157] ? kthread_blkcg+0x120/0x120 [ 55.331198][ T157] ret_from_fork_asm+0x11/0x20 [ 55.335679][ T157] </TASK> [ 56.470556][ T157] Shutting down cpus with NMI [ 56.474684][ T157] Kernel Offset: disabled [ 56.478285][ T157] Rebooting in 10 seconds.. ---------------------------------------- ---------------------------------------- ubuntu login: [ 42.392666][ T155] ===================================================== [ 42.398208][ T155] BUG: KMSAN: use-after-free in lzo1x_decompress_safe+0x433/0x3930 [ 42.408589][ T155] lzo1x_decompress_safe+0x433/0x3930 [ 42.416017][ T155] lzo_sdecompress+0x119/0x220 [ 42.427324][ T155] scomp_acomp_comp_decomp+0x65b/0xa10 [ 42.439258][ T155] scomp_acomp_decompress+0x4e/0x60 [ 42.449860][ T155] zswap_decompress+0x618/0xa50 [ 42.459372][ T155] zswap_writeback_entry+0x6c0/0xaa0 [ 42.468643][ T155] shrink_memcg_cb+0x3e8/0x870 [ 42.474589][ T155] __list_lru_walk_one+0x4ee/0xf00 [ 42.477891][ T155] list_lru_walk_one+0x1f6/0x250 [ 42.481171][ T155] zswap_shrinker_scan+0x46b/0x760 [ 42.484544][ T155] do_shrink_slab+0x958/0x1750 [ 42.487742][ T155] shrink_slab_memcg+0x6ae/0xea0 [ 42.491686][ T155] shrink_slab+0x119/0x7c0 [ 42.496077][ T155] shrink_one+0x835/0xeb0 [ 42.500477][ T155] shrink_many+0x70d/0x10c0 [ 42.504933][ T155] lru_gen_shrink_node+0x832/0xd10 [ 42.508651][ T155] shrink_node+0x13a/0x1dd0 [ 42.512056][ T155] balance_pgdat+0x1556/0x2740 [ 42.515294][ T155] kswapd+0x50d/0x870 [ 42.518245][ T155] kthread+0x485/0x600 [ 42.521178][ T155] ret_from_fork+0xfa/0x140 [ 42.524242][ T155] ret_from_fork_asm+0x11/0x20 [ 42.527444][ T155] [ 42.529916][ T155] Uninit was stored to memory at: [ 42.533147][ T155] scatterwalk_map_and_copy+0x8b5/0xb50 [ 42.536505][ T155] scomp_acomp_comp_decomp+0x45c/0xa10 [ 42.539860][ T155] scomp_acomp_decompress+0x4e/0x60 [ 42.543099][ T155] zswap_decompress+0x618/0xa50 [ 42.546244][ T155] zswap_writeback_entry+0x6c0/0xaa0 [ 42.549525][ T155] shrink_memcg_cb+0x3e8/0x870 [ 42.552652][ T155] __list_lru_walk_one+0x4ee/0xf00 [ 42.555890][ T155] list_lru_walk_one+0x1f6/0x250 [ 42.567920][ T155] zswap_shrinker_scan+0x46b/0x760 [ 42.578533][ T155] do_shrink_slab+0x958/0x1750 [ 42.587474][ T155] shrink_slab_memcg+0x6ae/0xea0 [ 42.591733][ T155] shrink_slab+0x119/0x7c0 [ 42.595698][ T155] shrink_one+0x835/0xeb0 [ 42.599604][ T155] shrink_many+0x70d/0x10c0 [ 42.603671][ T155] lru_gen_shrink_node+0x832/0xd10 [ 42.608028][ T155] shrink_node+0x13a/0x1dd0 [ 42.612164][ T155] balance_pgdat+0x1556/0x2740 [ 42.616458][ T155] kswapd+0x50d/0x870 [ 42.620420][ T155] kthread+0x485/0x600 [ 42.624380][ T155] ret_from_fork+0xfa/0x140 [ 42.628490][ T155] ret_from_fork_asm+0x11/0x20 [ 42.632693][ T155] [ 42.635854][ T155] Uninit was stored to memory at: [ 42.640219][ T155] zswap_decompress+0x299/0xa50 [ 42.644446][ T155] zswap_writeback_entry+0x6c0/0xaa0 [ 42.648922][ T155] shrink_memcg_cb+0x3e8/0x870 [ 42.653115][ T155] __list_lru_walk_one+0x4ee/0xf00 [ 42.657464][ T155] list_lru_walk_one+0x1f6/0x250 [ 42.661710][ T155] zswap_shrinker_scan+0x46b/0x760 [ 42.666078][ T155] do_shrink_slab+0x958/0x1750 [ 42.670389][ T155] shrink_slab_memcg+0x6ae/0xea0 [ 42.679819][ T155] shrink_slab+0x119/0x7c0 [ 42.688501][ T155] shrink_one+0x835/0xeb0 [ 42.697021][ T155] shrink_many+0x70d/0x10c0 [ 42.705719][ T155] lru_gen_shrink_node+0x832/0xd10 [ 42.715345][ T155] shrink_node+0x13a/0x1dd0 [ 42.724486][ T155] balance_pgdat+0x1556/0x2740 [ 42.733580][ T155] kswapd+0x50d/0x870 [ 42.742222][ T155] kthread+0x485/0x600 [ 42.751032][ T155] ret_from_fork+0xfa/0x140 [ 42.760274][ T155] ret_from_fork_asm+0x11/0x20 [ 42.769826][ T155] [ 42.776910][ T155] Uninit was created at: [ 42.785419][ T155] free_unref_page_prepare+0x130/0xfc0 [ 42.795890][ T155] free_unref_page_list+0x13f/0x1130 [ 42.806224][ T155] shrink_folio_list+0x713e/0x7480 [ 42.815480][ T155] evict_folios+0x2160/0x52c0 [ 42.819727][ T155] try_to_shrink_lruvec+0x1cb/0x460 [ 42.824366][ T155] shrink_one+0x72d/0xeb0 [ 42.834077][ T155] shrink_many+0x70d/0x10c0 [ 42.844079][ T155] lru_gen_shrink_node+0x832/0xd10 [ 42.854215][ T155] shrink_node+0x13a/0x1dd0 [ 42.863639][ T155] shrink_zones+0x787/0x1530 [ 42.873152][ T155] do_try_to_free_pages+0x2ac/0x16a0 [ 42.877447][ T155] try_to_free_pages+0xddb/0x19b0 [ 42.880694][ T155] __alloc_pages_slowpath+0x1a05/0x2d00 [ 42.884052][ T155] __alloc_pages+0xc6c/0x1040 [ 42.887180][ T155] alloc_pages_mpol+0x477/0xc40 [ 42.890362][ T155] alloc_pages+0x224/0x240 [ 42.893479][ T155] pipe_write+0xae5/0x2bd0 [ 42.896519][ T155] vfs_write+0xfb9/0x1b90 [ 42.899562][ T155] ksys_write+0x275/0x500 [ 42.902616][ T155] __x64_sys_write+0xdf/0x120 [ 42.906154][ T155] do_syscall_64+0xd5/0x1c0 [ 42.909855][ T155] entry_SYSCALL_64_after_hwframe+0x62/0x6a [ 42.913670][ T155] [ 42.916155][ T155] CPU: 5 PID: 155 Comm: kswapd1 Not tainted 6.8.0-rc5-next-20240223 #1 [ 42.921857][ T155] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 42.927627][ T155] ===================================================== [ 42.931409][ T155] Disabling lock debugging due to kernel taint [ 42.934961][ T155] Kernel panic - not syncing: kmsan.panic set ... [ 42.938569][ T155] CPU: 5 PID: 155 Comm: kswapd1 Tainted: G B 6.8.0-rc5-next-20240223 #1 [ 42.944533][ T155] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 42.950295][ T155] Call Trace: [ 42.952978][ T155] <TASK> [ 42.955503][ T155] dump_stack_lvl+0x24b/0x300 [ 42.960955][ T155] dump_stack+0x29/0x30 [ 42.969770][ T155] panic+0x4ed/0xca0 [ 42.978544][ T155] kmsan_report+0x2d1/0x2e0 [ 42.987874][ T155] ? __msan_warning+0x91/0x120 [ 42.997602][ T155] ? lzo1x_decompress_safe+0x433/0x3930 [ 43.004774][ T155] ? lzo_sdecompress+0x119/0x220 [ 43.008069][ T155] ? scomp_acomp_comp_decomp+0x65b/0xa10 [ 43.011551][ T155] ? scomp_acomp_decompress+0x4e/0x60 [ 43.014956][ T155] ? zswap_decompress+0x618/0xa50 [ 43.018249][ T155] ? zswap_writeback_entry+0x6c0/0xaa0 [ 43.021719][ T155] ? shrink_memcg_cb+0x3e8/0x870 [ 43.025111][ T155] ? __list_lru_walk_one+0x4ee/0xf00 [ 43.028534][ T155] ? list_lru_walk_one+0x1f6/0x250 [ 43.031877][ T155] ? zswap_shrinker_scan+0x46b/0x760 [ 43.035452][ T155] ? do_shrink_slab+0x958/0x1750 [ 43.038777][ T155] ? shrink_slab_memcg+0x6ae/0xea0 [ 43.042092][ T155] ? shrink_slab+0x119/0x7c0 [ 43.045255][ T155] ? shrink_one+0x835/0xeb0 [ 43.048375][ T155] ? shrink_many+0x70d/0x10c0 [ 43.051557][ T155] ? lru_gen_shrink_node+0x832/0xd10 [ 43.054932][ T155] ? shrink_node+0x13a/0x1dd0 [ 43.058106][ T155] ? balance_pgdat+0x1556/0x2740 [ 43.061387][ T155] ? kswapd+0x50d/0x870 [ 43.064386][ T155] ? kthread+0x485/0x600 [ 43.067433][ T155] ? ret_from_fork+0xfa/0x140 [ 43.075042][ T155] ? ret_from_fork_asm+0x11/0x20 [ 43.084182][ T155] ? shrink_one+0x835/0xeb0 [ 43.093150][ T155] ? shrink_many+0x70d/0x10c0 [ 43.102332][ T155] ? lru_gen_shrink_node+0x832/0xd10 [ 43.112061][ T155] ? shrink_node+0x13a/0x1dd0 [ 43.121464][ T155] ? balance_pgdat+0x1556/0x2740 [ 43.131127][ T155] ? kswapd+0x50d/0x870 [ 43.140045][ T155] ? kthread+0x485/0x600 [ 43.148993][ T155] ? ret_from_fork+0xfa/0x140 [ 43.158046][ T155] ? ret_from_fork_asm+0x11/0x20 [ 43.166985][ T155] ? kmsan_internal_set_shadow_origin+0x66/0xe0 [ 43.177320][ T155] ? kmsan_get_metadata+0x146/0x1c0 [ 43.187138][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 43.198259][ T155] ? scatterwalk_map_and_copy+0xaa/0xb50 [ 43.209263][ T155] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 43.220485][ T155] ? filter_irq_stacks+0xb9/0x230 [ 43.230534][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 43.240946][ T155] __msan_warning+0x91/0x120 [ 43.249737][ T155] lzo1x_decompress_safe+0x433/0x3930 [ 43.259433][ T155] ? filter_irq_stacks+0xb9/0x230 [ 43.268747][ T155] ? kmsan_internal_set_shadow_origin+0x66/0xe0 [ 43.278890][ T155] ? kmsan_get_metadata+0x146/0x1c0 [ 43.288092][ T155] lzo_sdecompress+0x119/0x220 [ 43.296809][ T155] ? lzo_scompress+0x250/0x250 [ 43.305573][ T155] scomp_acomp_comp_decomp+0x65b/0xa10 [ 43.315139][ T155] scomp_acomp_decompress+0x4e/0x60 [ 43.324453][ T155] ? scomp_acomp_compress+0x60/0x60 [ 43.334172][ T155] zswap_decompress+0x618/0xa50 [ 43.343444][ T155] zswap_writeback_entry+0x6c0/0xaa0 [ 43.353130][ T155] shrink_memcg_cb+0x3e8/0x870 [ 43.362321][ T155] __list_lru_walk_one+0x4ee/0xf00 [ 43.371873][ T155] ? zswap_shrinker_count+0x670/0x670 [ 43.381677][ T155] ? __msan_metadata_ptr_for_load_1+0x24/0x40 [ 43.392255][ T155] list_lru_walk_one+0x1f6/0x250 [ 43.401742][ T155] ? zswap_shrinker_count+0x670/0x670 [ 43.411756][ T155] zswap_shrinker_scan+0x46b/0x760 [ 43.421682][ T155] ? zswap_debugfs_init+0x420/0x420 [ 43.432130][ T155] do_shrink_slab+0x958/0x1750 [ 43.436685][ T155] shrink_slab_memcg+0x6ae/0xea0 [ 43.441009][ T155] shrink_slab+0x119/0x7c0 [ 43.446049][ T155] ? try_to_shrink_lruvec+0x42c/0x460 [ 43.451031][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 43.456008][ T155] shrink_one+0x835/0xeb0 [ 43.460383][ T155] shrink_many+0x70d/0x10c0 [ 43.464760][ T155] lru_gen_shrink_node+0x832/0xd10 [ 43.469331][ T155] shrink_node+0x13a/0x1dd0 [ 43.473657][ T155] ? mem_cgroup_soft_limit_reclaim+0x34/0x17b0 [ 43.478672][ T155] ? filter_irq_stacks+0xb9/0x230 [ 43.485985][ T155] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 43.497004][ T155] ? kswapd_age_node+0x63/0xb00 [ 43.506282][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 43.516209][ T155] balance_pgdat+0x1556/0x2740 [ 43.570854][ T155] ? finish_wait+0x2f1/0x4a0 [ 43.580532][ T155] kswapd+0x50d/0x870 [ 43.589396][ T155] kthread+0x485/0x600 [ 43.598259][ T155] ? shrink_all_memory+0x3a0/0x3a0 [ 43.608549][ T155] ? kthread_blkcg+0x120/0x120 [ 43.618041][ T155] ret_from_fork+0xfa/0x140 [ 43.627225][ T155] ? kthread_blkcg+0x120/0x120 [ 43.636721][ T155] ? kthread_blkcg+0x120/0x120 [ 43.646141][ T155] ret_from_fork_asm+0x11/0x20 [ 43.655612][ T155] </TASK> [ 44.788328][ T155] Shutting down cpus with NMI [ 44.792527][ T155] Kernel Offset: disabled [ 44.795640][ T155] Rebooting in 10 seconds.. ---------------------------------------- Maybe a different cause, but I feel that frequency of hitting "corrupted stack end detected inside scheduler" problem has increased in linux-next.git compared to linux.git . Too much stack usage? ---------------------------------------- ubuntu login: [ 53.757790][ T194] Kernel panic - not syncing: corrupted stack end detected inside scheduler [ 53.784397][ T194] CPU: 3 PID: 194 Comm: kworker/u39:3 Not tainted 6.8.0-rc5-next-20240223 #1 [ 53.810595][ T194] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 53.829184][ T194] Workqueue: writeback wb_workfn (flush-8:0) [ 53.835445][ T194] Call Trace: [ 53.839997][ T194] <TASK> [ 53.844176][ T194] dump_stack_lvl+0x24b/0x300 [ 53.849323][ T194] dump_stack+0x29/0x30 [ 53.854261][ T194] panic+0x4ed/0xca0 [ 53.858938][ T194] ? kmsan_get_metadata+0x50/0x1c0 [ 53.864326][ T194] __schedule+0x9e4/0x2770 [ 53.883521][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 53.910719][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 53.936505][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 53.964199][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 53.989716][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.015124][ T194] __cond_resched+0x50/0xc0 [ 54.038931][ T194] rmap_walk_file+0x382/0x8d0 [ 54.066110][ T194] folio_mkclean+0x34d/0x530 [ 54.089049][ T194] ? folio_mkclean+0x530/0x530 [ 54.117183][ T194] ? page_mkclean_one+0x3f0/0x3f0 [ 54.135476][ T194] folio_clear_dirty_for_io+0x22a/0xae0 [ 54.144905][ T194] ? filemap_get_folios_tag+0x64a/0x6c0 [ 54.155053][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.165436][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 54.175572][ T194] mpage_submit_folio+0x12a/0x5d0 [ 54.186797][ T194] ext4_do_writepages+0x3401/0x63d0 [ 54.193608][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.206517][ T194] ext4_writepages+0x338/0x870 [ 54.234367][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.243997][ T194] ? ext4_read_folio+0x440/0x440 [ 54.271561][ T194] do_writepages+0x5e5/0x15c0 [ 54.287149][ T194] ? wake_up_bit+0x9c/0x490 [ 54.297127][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 54.318153][ T194] ? filter_irq_stacks+0xb9/0x230 [ 54.326784][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.343015][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 54.349741][ T194] __writeback_single_inode+0x170/0x1090 [ 54.356296][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 54.364305][ T194] writeback_sb_inodes+0xd74/0x1e20 [ 54.371317][ T194] ? kmsan_internal_set_shadow_origin+0x66/0xe0 [ 54.378719][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.385263][ T194] __writeback_inodes_wb+0x1d6/0x510 [ 54.391720][ T194] wb_writeback+0x63e/0xff0 [ 54.399899][ T194] ? stack_depot_save_flags+0x2c/0x6f0 [ 54.408778][ T194] ? kmsan_internal_set_shadow_origin+0x60/0xe0 [ 54.439971][ T194] wb_do_writeback+0x120b/0x1510 [ 54.467029][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.494644][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.512469][ T194] wb_workfn+0x190/0x850 [ 54.537678][ T194] ? kmsan_get_metadata+0x146/0x1c0 [ 54.565645][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 [ 54.595256][ T194] ? inode_wait_for_writeback+0x320/0x320 [ 54.609201][ T194] process_one_work+0xa0c/0x1c60 [ 54.614993][ T194] worker_thread+0x11f2/0x1ba0 [ 54.620515][ T194] kthread+0x485/0x600 [ 54.625631][ T194] ? pr_cont_work+0xee0/0xee0 [ 54.630919][ T194] ? kthread_blkcg+0x120/0x120 [ 54.636291][ T194] ret_from_fork+0xfa/0x140 [ 54.641771][ T194] ? kthread_blkcg+0x120/0x120 [ 54.647279][ T194] ? kthread_blkcg+0x120/0x120 [ 54.652705][ T194] ret_from_fork_asm+0x11/0x20 [ 54.658127][ T194] </TASK> [ 54.683905][ T194] Kernel Offset: disabled [ 54.688874][ T194] Rebooting in 10 seconds.. ---------------------------------------- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 9:26 ` Tetsuo Handa @ 2024-02-23 10:10 ` Chengming Zhou 0 siblings, 0 replies; 17+ messages in thread From: Chengming Zhou @ 2024-02-23 10:10 UTC (permalink / raw) To: Tetsuo Handa, Sergey Senozhatsky, Yosry Ahmed Cc: Johannes Weiner, Nhat Pham, Minchan Kim, linux-mm On 2024/2/23 17:26, Tetsuo Handa wrote: > On 2024/02/23 14:23, Chengming Zhou wrote: >> Tetsuo, could you please check if the config has CONFIG_COMPACTION enabled? > > Yes, CONFIG_COMPACTION is enabled. > > Also, I can observe this problem with 6.8.0-rc5-next-20240223. Ok, from the report it seems UAF of the zspage? which is allocated from slab. I have no idea of the reason. Maybe it's better to run a bisect, as suggested by Sergey. Thanks. > > ---------------------------------------- > [ 54.589642][ T157] ===================================================== > [ 54.603721][ T157] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 > [ 54.608092][ T157] obj_malloc+0x6cc/0x7b0 > [ 54.610904][ T157] zs_malloc+0xda2/0x12d0 > [ 54.613688][ T157] zs_zpool_malloc+0xa5/0x1b0 > [ 54.619163][ T157] zpool_malloc+0x113/0x150 > [ 54.624449][ T157] zswap_compress+0x69b/0xbd0 > [ 54.629904][ T157] zswap_store+0x1f24/0x2d00 > [ 54.635026][ T157] swap_writepage+0x15b/0x4f0 > [ 54.640023][ T157] pageout+0x3d4/0xeb0 > [ 54.644699][ T157] shrink_folio_list+0x4d7f/0x7480 > [ 54.649867][ T157] evict_folios+0x2160/0x52c0 > [ 54.654872][ T157] try_to_shrink_lruvec+0x1cb/0x460 > [ 54.660074][ T157] shrink_one+0x72d/0xeb0 > [ 54.664922][ T157] shrink_many+0x70d/0x10c0 > [ 54.669849][ T157] lru_gen_shrink_node+0x832/0xd10 > [ 54.675110][ T157] shrink_node+0x13a/0x1dd0 > [ 54.680026][ T157] balance_pgdat+0x1556/0x2740 > [ 54.685032][ T157] kswapd+0x50d/0x870 > [ 54.689643][ T157] kthread+0x485/0x600 > [ 54.694432][ T157] ret_from_fork+0xfa/0x140 > [ 54.699305][ T157] ret_from_fork_asm+0x11/0x20 > [ 54.704295][ T157] > [ 54.707905][ T157] Uninit was stored to memory at: > [ 54.712837][ T157] obj_malloc+0x70a/0x7b0 > [ 54.717434][ T157] zs_malloc+0xda2/0x12d0 > [ 54.722009][ T157] zs_zpool_malloc+0xa5/0x1b0 > [ 54.726806][ T157] zpool_malloc+0x113/0x150 > [ 54.731507][ T157] zswap_compress+0x69b/0xbd0 > [ 54.736299][ T157] zswap_store+0x1f24/0x2d00 > [ 54.741081][ T157] swap_writepage+0x15b/0x4f0 > [ 54.745880][ T157] pageout+0x3d4/0xeb0 > [ 54.750386][ T157] shrink_folio_list+0x4d7f/0x7480 > [ 54.755378][ T157] evict_folios+0x2160/0x52c0 > [ 54.760153][ T157] try_to_shrink_lruvec+0x1cb/0x460 > [ 54.765223][ T157] shrink_one+0x72d/0xeb0 > [ 54.769870][ T157] shrink_many+0x70d/0x10c0 > [ 54.774445][ T157] lru_gen_shrink_node+0x832/0xd10 > [ 54.779221][ T157] shrink_node+0x13a/0x1dd0 > [ 54.783965][ T157] balance_pgdat+0x1556/0x2740 > [ 54.788702][ T157] kswapd+0x50d/0x870 > [ 54.793073][ T157] kthread+0x485/0x600 > [ 54.798253][ T157] ret_from_fork+0xfa/0x140 > [ 54.804206][ T157] ret_from_fork_asm+0x11/0x20 > [ 54.809016][ T157] > [ 54.812652][ T157] Uninit was created at: > [ 54.817314][ T157] free_unref_page_prepare+0x130/0xfc0 > [ 54.822499][ T157] free_unref_page_list+0x13f/0x1130 > [ 54.828207][ T157] shrink_folio_list+0x713e/0x7480 > [ 54.834143][ T157] evict_folios+0x2160/0x52c0 > [ 54.839358][ T157] try_to_shrink_lruvec+0x1cb/0x460 > [ 54.844628][ T157] shrink_one+0x72d/0xeb0 > [ 54.849436][ T157] shrink_many+0x70d/0x10c0 > [ 54.854310][ T157] lru_gen_shrink_node+0x832/0xd10 > [ 54.859337][ T157] shrink_node+0x13a/0x1dd0 > [ 54.864076][ T157] shrink_zones+0x787/0x1530 > [ 54.868808][ T157] do_try_to_free_pages+0x2ac/0x16a0 > [ 54.873865][ T157] try_to_free_pages+0xddb/0x19b0 > [ 54.878795][ T157] __alloc_pages_slowpath+0x1a05/0x2d00 > [ 54.883978][ T157] __alloc_pages+0xc6c/0x1040 > [ 54.888802][ T157] alloc_pages_mpol+0x477/0xc40 > [ 54.893629][ T157] alloc_pages+0x224/0x240 > [ 54.898092][ T157] pipe_write+0xae5/0x2bd0 > [ 54.902702][ T157] vfs_write+0xfb9/0x1b90 > [ 54.907117][ T157] ksys_write+0x275/0x500 > [ 54.911612][ T157] __x64_sys_write+0xdf/0x120 > [ 54.916287][ T157] do_syscall_64+0xd5/0x1c0 > [ 54.920782][ T157] entry_SYSCALL_64_after_hwframe+0x62/0x6a > [ 54.925972][ T157] > [ 54.929436][ T157] CPU: 4 PID: 157 Comm: kswapd1 Not tainted 6.8.0-rc5-next-20240223 #1 > [ 54.937592][ T157] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 54.946147][ T157] ===================================================== > [ 54.951772][ T157] Disabling lock debugging due to kernel taint > [ 54.957040][ T157] Kernel panic - not syncing: kmsan.panic set ... > [ 54.962443][ T157] CPU: 4 PID: 157 Comm: kswapd1 Tainted: G B 6.8.0-rc5-next-20240223 #1 > [ 54.971295][ T157] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 54.979856][ T157] Call Trace: > [ 54.983760][ T157] <TASK> > [ 54.987503][ T157] dump_stack_lvl+0x24b/0x300 > [ 54.992068][ T157] dump_stack+0x29/0x30 > [ 54.996373][ T157] panic+0x4ed/0xca0 > [ 55.000656][ T157] kmsan_report+0x2d1/0x2e0 > [ 55.005155][ T157] ? kmem_cache_alloc+0x707/0xf50 > [ 55.009909][ T157] ? kmsan_internal_poison_memory+0x7d/0x90 > [ 55.015056][ T157] ? kmsan_internal_poison_memory+0x49/0x90 > [ 55.020253][ T157] ? kmsan_slab_alloc+0xdf/0x160 > [ 55.024995][ T157] ? __msan_warning+0x91/0x120 > [ 55.029604][ T157] ? obj_malloc+0x6cc/0x7b0 > [ 55.034166][ T157] ? zs_malloc+0xda2/0x12d0 > [ 55.038692][ T157] ? zs_zpool_malloc+0xa5/0x1b0 > [ 55.043342][ T157] ? zpool_malloc+0x113/0x150 > [ 55.047909][ T157] ? zswap_compress+0x69b/0xbd0 > [ 55.052576][ T157] ? zswap_store+0x1f24/0x2d00 > [ 55.057213][ T157] ? swap_writepage+0x15b/0x4f0 > [ 55.061836][ T157] ? pageout+0x3d4/0xeb0 > [ 55.066216][ T157] ? shrink_folio_list+0x4d7f/0x7480 > [ 55.071083][ T157] ? evict_folios+0x2160/0x52c0 > [ 55.075734][ T157] ? try_to_shrink_lruvec+0x1cb/0x460 > [ 55.080625][ T157] ? shrink_one+0x72d/0xeb0 > [ 55.085139][ T157] ? shrink_many+0x70d/0x10c0 > [ 55.089752][ T157] ? lru_gen_shrink_node+0x832/0xd10 > [ 55.094614][ T157] ? shrink_node+0x13a/0x1dd0 > [ 55.099188][ T157] ? balance_pgdat+0x1556/0x2740 > [ 55.103891][ T157] ? kswapd+0x50d/0x870 > [ 55.108212][ T157] ? kthread+0x485/0x600 > [ 55.112459][ T157] ? ret_from_fork+0xfa/0x140 > [ 55.116849][ T157] ? ret_from_fork_asm+0x11/0x20 > [ 55.121438][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 55.126388][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 55.131446][ T157] ? should_fail_ex+0x91/0xa20 > [ 55.136530][ T157] ? kmsan_get_metadata+0x146/0x1c0 > [ 55.141199][ T157] ? kmsan_get_metadata+0x146/0x1c0 > [ 55.145956][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 55.150928][ T157] ? __should_failslab+0x24f/0x2e0 > [ 55.155750][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 55.161123][ T157] ? __should_failslab+0x24f/0x2e0 > [ 55.165918][ T157] ? kmsan_get_metadata+0x146/0x1c0 > [ 55.170723][ T157] ? kmsan_get_metadata+0x146/0x1c0 > [ 55.175568][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 55.180684][ T157] __msan_warning+0x91/0x120 > [ 55.185066][ T157] obj_malloc+0x6cc/0x7b0 > [ 55.189320][ T157] ? kmsan_get_metadata+0x146/0x1c0 > [ 55.194051][ T157] zs_malloc+0xda2/0x12d0 > [ 55.198333][ T157] zs_zpool_malloc+0xa5/0x1b0 > [ 55.202886][ T157] ? zs_zpool_destroy+0x50/0x50 > [ 55.207378][ T157] zpool_malloc+0x113/0x150 > [ 55.211829][ T157] zswap_compress+0x69b/0xbd0 > [ 55.216298][ T157] zswap_store+0x1f24/0x2d00 > [ 55.220727][ T157] swap_writepage+0x15b/0x4f0 > [ 55.225186][ T157] ? generic_swapfile_activate+0xed0/0xed0 > [ 55.230120][ T157] pageout+0x3d4/0xeb0 > [ 55.234272][ T157] shrink_folio_list+0x4d7f/0x7480 > [ 55.239002][ T157] evict_folios+0x2160/0x52c0 > [ 55.243455][ T157] try_to_shrink_lruvec+0x1cb/0x460 > [ 55.248119][ T157] shrink_one+0x72d/0xeb0 > [ 55.252389][ T157] shrink_many+0x70d/0x10c0 > [ 55.257702][ T157] lru_gen_shrink_node+0x832/0xd10 > [ 55.262478][ T157] shrink_node+0x13a/0x1dd0 > [ 55.266848][ T157] ? mem_cgroup_soft_limit_reclaim+0x34/0x17b0 > [ 55.271983][ T157] ? filter_irq_stacks+0xb9/0x230 > [ 55.276677][ T157] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 55.281724][ T157] ? kswapd_age_node+0x63/0xb00 > [ 55.286322][ T157] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 55.291458][ T157] balance_pgdat+0x1556/0x2740 > [ 55.295936][ T157] ? finish_wait+0x2f1/0x4a0 > [ 55.300332][ T157] kswapd+0x50d/0x870 > [ 55.304457][ T157] kthread+0x485/0x600 > [ 55.308674][ T157] ? shrink_all_memory+0x3a0/0x3a0 > [ 55.313311][ T157] ? kthread_blkcg+0x120/0x120 > [ 55.317805][ T157] ret_from_fork+0xfa/0x140 > [ 55.322138][ T157] ? kthread_blkcg+0x120/0x120 > [ 55.326615][ T157] ? kthread_blkcg+0x120/0x120 > [ 55.331198][ T157] ret_from_fork_asm+0x11/0x20 > [ 55.335679][ T157] </TASK> > [ 56.470556][ T157] Shutting down cpus with NMI > [ 56.474684][ T157] Kernel Offset: disabled > [ 56.478285][ T157] Rebooting in 10 seconds.. > ---------------------------------------- > > ---------------------------------------- > ubuntu login: [ 42.392666][ T155] ===================================================== > [ 42.398208][ T155] BUG: KMSAN: use-after-free in lzo1x_decompress_safe+0x433/0x3930 > [ 42.408589][ T155] lzo1x_decompress_safe+0x433/0x3930 > [ 42.416017][ T155] lzo_sdecompress+0x119/0x220 > [ 42.427324][ T155] scomp_acomp_comp_decomp+0x65b/0xa10 > [ 42.439258][ T155] scomp_acomp_decompress+0x4e/0x60 > [ 42.449860][ T155] zswap_decompress+0x618/0xa50 > [ 42.459372][ T155] zswap_writeback_entry+0x6c0/0xaa0 > [ 42.468643][ T155] shrink_memcg_cb+0x3e8/0x870 > [ 42.474589][ T155] __list_lru_walk_one+0x4ee/0xf00 > [ 42.477891][ T155] list_lru_walk_one+0x1f6/0x250 > [ 42.481171][ T155] zswap_shrinker_scan+0x46b/0x760 > [ 42.484544][ T155] do_shrink_slab+0x958/0x1750 > [ 42.487742][ T155] shrink_slab_memcg+0x6ae/0xea0 > [ 42.491686][ T155] shrink_slab+0x119/0x7c0 > [ 42.496077][ T155] shrink_one+0x835/0xeb0 > [ 42.500477][ T155] shrink_many+0x70d/0x10c0 > [ 42.504933][ T155] lru_gen_shrink_node+0x832/0xd10 > [ 42.508651][ T155] shrink_node+0x13a/0x1dd0 > [ 42.512056][ T155] balance_pgdat+0x1556/0x2740 > [ 42.515294][ T155] kswapd+0x50d/0x870 > [ 42.518245][ T155] kthread+0x485/0x600 > [ 42.521178][ T155] ret_from_fork+0xfa/0x140 > [ 42.524242][ T155] ret_from_fork_asm+0x11/0x20 > [ 42.527444][ T155] > [ 42.529916][ T155] Uninit was stored to memory at: > [ 42.533147][ T155] scatterwalk_map_and_copy+0x8b5/0xb50 > [ 42.536505][ T155] scomp_acomp_comp_decomp+0x45c/0xa10 > [ 42.539860][ T155] scomp_acomp_decompress+0x4e/0x60 > [ 42.543099][ T155] zswap_decompress+0x618/0xa50 > [ 42.546244][ T155] zswap_writeback_entry+0x6c0/0xaa0 > [ 42.549525][ T155] shrink_memcg_cb+0x3e8/0x870 > [ 42.552652][ T155] __list_lru_walk_one+0x4ee/0xf00 > [ 42.555890][ T155] list_lru_walk_one+0x1f6/0x250 > [ 42.567920][ T155] zswap_shrinker_scan+0x46b/0x760 > [ 42.578533][ T155] do_shrink_slab+0x958/0x1750 > [ 42.587474][ T155] shrink_slab_memcg+0x6ae/0xea0 > [ 42.591733][ T155] shrink_slab+0x119/0x7c0 > [ 42.595698][ T155] shrink_one+0x835/0xeb0 > [ 42.599604][ T155] shrink_many+0x70d/0x10c0 > [ 42.603671][ T155] lru_gen_shrink_node+0x832/0xd10 > [ 42.608028][ T155] shrink_node+0x13a/0x1dd0 > [ 42.612164][ T155] balance_pgdat+0x1556/0x2740 > [ 42.616458][ T155] kswapd+0x50d/0x870 > [ 42.620420][ T155] kthread+0x485/0x600 > [ 42.624380][ T155] ret_from_fork+0xfa/0x140 > [ 42.628490][ T155] ret_from_fork_asm+0x11/0x20 > [ 42.632693][ T155] > [ 42.635854][ T155] Uninit was stored to memory at: > [ 42.640219][ T155] zswap_decompress+0x299/0xa50 > [ 42.644446][ T155] zswap_writeback_entry+0x6c0/0xaa0 > [ 42.648922][ T155] shrink_memcg_cb+0x3e8/0x870 > [ 42.653115][ T155] __list_lru_walk_one+0x4ee/0xf00 > [ 42.657464][ T155] list_lru_walk_one+0x1f6/0x250 > [ 42.661710][ T155] zswap_shrinker_scan+0x46b/0x760 > [ 42.666078][ T155] do_shrink_slab+0x958/0x1750 > [ 42.670389][ T155] shrink_slab_memcg+0x6ae/0xea0 > [ 42.679819][ T155] shrink_slab+0x119/0x7c0 > [ 42.688501][ T155] shrink_one+0x835/0xeb0 > [ 42.697021][ T155] shrink_many+0x70d/0x10c0 > [ 42.705719][ T155] lru_gen_shrink_node+0x832/0xd10 > [ 42.715345][ T155] shrink_node+0x13a/0x1dd0 > [ 42.724486][ T155] balance_pgdat+0x1556/0x2740 > [ 42.733580][ T155] kswapd+0x50d/0x870 > [ 42.742222][ T155] kthread+0x485/0x600 > [ 42.751032][ T155] ret_from_fork+0xfa/0x140 > [ 42.760274][ T155] ret_from_fork_asm+0x11/0x20 > [ 42.769826][ T155] > [ 42.776910][ T155] Uninit was created at: > [ 42.785419][ T155] free_unref_page_prepare+0x130/0xfc0 > [ 42.795890][ T155] free_unref_page_list+0x13f/0x1130 > [ 42.806224][ T155] shrink_folio_list+0x713e/0x7480 > [ 42.815480][ T155] evict_folios+0x2160/0x52c0 > [ 42.819727][ T155] try_to_shrink_lruvec+0x1cb/0x460 > [ 42.824366][ T155] shrink_one+0x72d/0xeb0 > [ 42.834077][ T155] shrink_many+0x70d/0x10c0 > [ 42.844079][ T155] lru_gen_shrink_node+0x832/0xd10 > [ 42.854215][ T155] shrink_node+0x13a/0x1dd0 > [ 42.863639][ T155] shrink_zones+0x787/0x1530 > [ 42.873152][ T155] do_try_to_free_pages+0x2ac/0x16a0 > [ 42.877447][ T155] try_to_free_pages+0xddb/0x19b0 > [ 42.880694][ T155] __alloc_pages_slowpath+0x1a05/0x2d00 > [ 42.884052][ T155] __alloc_pages+0xc6c/0x1040 > [ 42.887180][ T155] alloc_pages_mpol+0x477/0xc40 > [ 42.890362][ T155] alloc_pages+0x224/0x240 > [ 42.893479][ T155] pipe_write+0xae5/0x2bd0 > [ 42.896519][ T155] vfs_write+0xfb9/0x1b90 > [ 42.899562][ T155] ksys_write+0x275/0x500 > [ 42.902616][ T155] __x64_sys_write+0xdf/0x120 > [ 42.906154][ T155] do_syscall_64+0xd5/0x1c0 > [ 42.909855][ T155] entry_SYSCALL_64_after_hwframe+0x62/0x6a > [ 42.913670][ T155] > [ 42.916155][ T155] CPU: 5 PID: 155 Comm: kswapd1 Not tainted 6.8.0-rc5-next-20240223 #1 > [ 42.921857][ T155] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 42.927627][ T155] ===================================================== > [ 42.931409][ T155] Disabling lock debugging due to kernel taint > [ 42.934961][ T155] Kernel panic - not syncing: kmsan.panic set ... > [ 42.938569][ T155] CPU: 5 PID: 155 Comm: kswapd1 Tainted: G B 6.8.0-rc5-next-20240223 #1 > [ 42.944533][ T155] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 42.950295][ T155] Call Trace: > [ 42.952978][ T155] <TASK> > [ 42.955503][ T155] dump_stack_lvl+0x24b/0x300 > [ 42.960955][ T155] dump_stack+0x29/0x30 > [ 42.969770][ T155] panic+0x4ed/0xca0 > [ 42.978544][ T155] kmsan_report+0x2d1/0x2e0 > [ 42.987874][ T155] ? __msan_warning+0x91/0x120 > [ 42.997602][ T155] ? lzo1x_decompress_safe+0x433/0x3930 > [ 43.004774][ T155] ? lzo_sdecompress+0x119/0x220 > [ 43.008069][ T155] ? scomp_acomp_comp_decomp+0x65b/0xa10 > [ 43.011551][ T155] ? scomp_acomp_decompress+0x4e/0x60 > [ 43.014956][ T155] ? zswap_decompress+0x618/0xa50 > [ 43.018249][ T155] ? zswap_writeback_entry+0x6c0/0xaa0 > [ 43.021719][ T155] ? shrink_memcg_cb+0x3e8/0x870 > [ 43.025111][ T155] ? __list_lru_walk_one+0x4ee/0xf00 > [ 43.028534][ T155] ? list_lru_walk_one+0x1f6/0x250 > [ 43.031877][ T155] ? zswap_shrinker_scan+0x46b/0x760 > [ 43.035452][ T155] ? do_shrink_slab+0x958/0x1750 > [ 43.038777][ T155] ? shrink_slab_memcg+0x6ae/0xea0 > [ 43.042092][ T155] ? shrink_slab+0x119/0x7c0 > [ 43.045255][ T155] ? shrink_one+0x835/0xeb0 > [ 43.048375][ T155] ? shrink_many+0x70d/0x10c0 > [ 43.051557][ T155] ? lru_gen_shrink_node+0x832/0xd10 > [ 43.054932][ T155] ? shrink_node+0x13a/0x1dd0 > [ 43.058106][ T155] ? balance_pgdat+0x1556/0x2740 > [ 43.061387][ T155] ? kswapd+0x50d/0x870 > [ 43.064386][ T155] ? kthread+0x485/0x600 > [ 43.067433][ T155] ? ret_from_fork+0xfa/0x140 > [ 43.075042][ T155] ? ret_from_fork_asm+0x11/0x20 > [ 43.084182][ T155] ? shrink_one+0x835/0xeb0 > [ 43.093150][ T155] ? shrink_many+0x70d/0x10c0 > [ 43.102332][ T155] ? lru_gen_shrink_node+0x832/0xd10 > [ 43.112061][ T155] ? shrink_node+0x13a/0x1dd0 > [ 43.121464][ T155] ? balance_pgdat+0x1556/0x2740 > [ 43.131127][ T155] ? kswapd+0x50d/0x870 > [ 43.140045][ T155] ? kthread+0x485/0x600 > [ 43.148993][ T155] ? ret_from_fork+0xfa/0x140 > [ 43.158046][ T155] ? ret_from_fork_asm+0x11/0x20 > [ 43.166985][ T155] ? kmsan_internal_set_shadow_origin+0x66/0xe0 > [ 43.177320][ T155] ? kmsan_get_metadata+0x146/0x1c0 > [ 43.187138][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 43.198259][ T155] ? scatterwalk_map_and_copy+0xaa/0xb50 > [ 43.209263][ T155] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 43.220485][ T155] ? filter_irq_stacks+0xb9/0x230 > [ 43.230534][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 43.240946][ T155] __msan_warning+0x91/0x120 > [ 43.249737][ T155] lzo1x_decompress_safe+0x433/0x3930 > [ 43.259433][ T155] ? filter_irq_stacks+0xb9/0x230 > [ 43.268747][ T155] ? kmsan_internal_set_shadow_origin+0x66/0xe0 > [ 43.278890][ T155] ? kmsan_get_metadata+0x146/0x1c0 > [ 43.288092][ T155] lzo_sdecompress+0x119/0x220 > [ 43.296809][ T155] ? lzo_scompress+0x250/0x250 > [ 43.305573][ T155] scomp_acomp_comp_decomp+0x65b/0xa10 > [ 43.315139][ T155] scomp_acomp_decompress+0x4e/0x60 > [ 43.324453][ T155] ? scomp_acomp_compress+0x60/0x60 > [ 43.334172][ T155] zswap_decompress+0x618/0xa50 > [ 43.343444][ T155] zswap_writeback_entry+0x6c0/0xaa0 > [ 43.353130][ T155] shrink_memcg_cb+0x3e8/0x870 > [ 43.362321][ T155] __list_lru_walk_one+0x4ee/0xf00 > [ 43.371873][ T155] ? zswap_shrinker_count+0x670/0x670 > [ 43.381677][ T155] ? __msan_metadata_ptr_for_load_1+0x24/0x40 > [ 43.392255][ T155] list_lru_walk_one+0x1f6/0x250 > [ 43.401742][ T155] ? zswap_shrinker_count+0x670/0x670 > [ 43.411756][ T155] zswap_shrinker_scan+0x46b/0x760 > [ 43.421682][ T155] ? zswap_debugfs_init+0x420/0x420 > [ 43.432130][ T155] do_shrink_slab+0x958/0x1750 > [ 43.436685][ T155] shrink_slab_memcg+0x6ae/0xea0 > [ 43.441009][ T155] shrink_slab+0x119/0x7c0 > [ 43.446049][ T155] ? try_to_shrink_lruvec+0x42c/0x460 > [ 43.451031][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 43.456008][ T155] shrink_one+0x835/0xeb0 > [ 43.460383][ T155] shrink_many+0x70d/0x10c0 > [ 43.464760][ T155] lru_gen_shrink_node+0x832/0xd10 > [ 43.469331][ T155] shrink_node+0x13a/0x1dd0 > [ 43.473657][ T155] ? mem_cgroup_soft_limit_reclaim+0x34/0x17b0 > [ 43.478672][ T155] ? filter_irq_stacks+0xb9/0x230 > [ 43.485985][ T155] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 43.497004][ T155] ? kswapd_age_node+0x63/0xb00 > [ 43.506282][ T155] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 43.516209][ T155] balance_pgdat+0x1556/0x2740 > [ 43.570854][ T155] ? finish_wait+0x2f1/0x4a0 > [ 43.580532][ T155] kswapd+0x50d/0x870 > [ 43.589396][ T155] kthread+0x485/0x600 > [ 43.598259][ T155] ? shrink_all_memory+0x3a0/0x3a0 > [ 43.608549][ T155] ? kthread_blkcg+0x120/0x120 > [ 43.618041][ T155] ret_from_fork+0xfa/0x140 > [ 43.627225][ T155] ? kthread_blkcg+0x120/0x120 > [ 43.636721][ T155] ? kthread_blkcg+0x120/0x120 > [ 43.646141][ T155] ret_from_fork_asm+0x11/0x20 > [ 43.655612][ T155] </TASK> > [ 44.788328][ T155] Shutting down cpus with NMI > [ 44.792527][ T155] Kernel Offset: disabled > [ 44.795640][ T155] Rebooting in 10 seconds.. > ---------------------------------------- > > > > Maybe a different cause, but I feel that frequency of hitting "corrupted stack end detected > inside scheduler" problem has increased in linux-next.git compared to linux.git . > Too much stack usage? > > ---------------------------------------- > ubuntu login: [ 53.757790][ T194] Kernel panic - not syncing: corrupted stack end detected inside scheduler > [ 53.784397][ T194] CPU: 3 PID: 194 Comm: kworker/u39:3 Not tainted 6.8.0-rc5-next-20240223 #1 > [ 53.810595][ T194] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 53.829184][ T194] Workqueue: writeback wb_workfn (flush-8:0) > [ 53.835445][ T194] Call Trace: > [ 53.839997][ T194] <TASK> > [ 53.844176][ T194] dump_stack_lvl+0x24b/0x300 > [ 53.849323][ T194] dump_stack+0x29/0x30 > [ 53.854261][ T194] panic+0x4ed/0xca0 > [ 53.858938][ T194] ? kmsan_get_metadata+0x50/0x1c0 > [ 53.864326][ T194] __schedule+0x9e4/0x2770 > [ 53.883521][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 53.910719][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 53.936505][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 53.964199][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 53.989716][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.015124][ T194] __cond_resched+0x50/0xc0 > [ 54.038931][ T194] rmap_walk_file+0x382/0x8d0 > [ 54.066110][ T194] folio_mkclean+0x34d/0x530 > [ 54.089049][ T194] ? folio_mkclean+0x530/0x530 > [ 54.117183][ T194] ? page_mkclean_one+0x3f0/0x3f0 > [ 54.135476][ T194] folio_clear_dirty_for_io+0x22a/0xae0 > [ 54.144905][ T194] ? filemap_get_folios_tag+0x64a/0x6c0 > [ 54.155053][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.165436][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 54.175572][ T194] mpage_submit_folio+0x12a/0x5d0 > [ 54.186797][ T194] ext4_do_writepages+0x3401/0x63d0 > [ 54.193608][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.206517][ T194] ext4_writepages+0x338/0x870 > [ 54.234367][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.243997][ T194] ? ext4_read_folio+0x440/0x440 > [ 54.271561][ T194] do_writepages+0x5e5/0x15c0 > [ 54.287149][ T194] ? wake_up_bit+0x9c/0x490 > [ 54.297127][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 54.318153][ T194] ? filter_irq_stacks+0xb9/0x230 > [ 54.326784][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.343015][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 54.349741][ T194] __writeback_single_inode+0x170/0x1090 > [ 54.356296][ T194] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 54.364305][ T194] writeback_sb_inodes+0xd74/0x1e20 > [ 54.371317][ T194] ? kmsan_internal_set_shadow_origin+0x66/0xe0 > [ 54.378719][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.385263][ T194] __writeback_inodes_wb+0x1d6/0x510 > [ 54.391720][ T194] wb_writeback+0x63e/0xff0 > [ 54.399899][ T194] ? stack_depot_save_flags+0x2c/0x6f0 > [ 54.408778][ T194] ? kmsan_internal_set_shadow_origin+0x60/0xe0 > [ 54.439971][ T194] wb_do_writeback+0x120b/0x1510 > [ 54.467029][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.494644][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.512469][ T194] wb_workfn+0x190/0x850 > [ 54.537678][ T194] ? kmsan_get_metadata+0x146/0x1c0 > [ 54.565645][ T194] ? kmsan_get_shadow_origin_ptr+0x4d/0xb0 > [ 54.595256][ T194] ? inode_wait_for_writeback+0x320/0x320 > [ 54.609201][ T194] process_one_work+0xa0c/0x1c60 > [ 54.614993][ T194] worker_thread+0x11f2/0x1ba0 > [ 54.620515][ T194] kthread+0x485/0x600 > [ 54.625631][ T194] ? pr_cont_work+0xee0/0xee0 > [ 54.630919][ T194] ? kthread_blkcg+0x120/0x120 > [ 54.636291][ T194] ret_from_fork+0xfa/0x140 > [ 54.641771][ T194] ? kthread_blkcg+0x120/0x120 > [ 54.647279][ T194] ? kthread_blkcg+0x120/0x120 > [ 54.652705][ T194] ret_from_fork_asm+0x11/0x20 > [ 54.658127][ T194] </TASK> > [ 54.683905][ T194] Kernel Offset: disabled > [ 54.688874][ T194] Rebooting in 10 seconds.. > ---------------------------------------- > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 2:10 [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Tetsuo Handa 2024-02-23 2:27 ` Yosry Ahmed @ 2024-02-23 4:43 ` Sergey Senozhatsky 2024-02-23 15:22 ` Tetsuo Handa 1 sibling, 1 reply; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-23 4:43 UTC (permalink / raw) To: Tetsuo Handa Cc: Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, Sergey Senozhatsky, linux-mm On (24/02/23 11:10), Tetsuo Handa wrote: > > I can observe this bug during evict_folios() from 6.7.0 to 6.8.0-rc5-00163-gffd2cb6b718e. > Since I haven't observed with 6.6.0, this bug might be introduced in 6.7 cycle. Can we please run a bisect? There are some zsmalloc patches for 6.8 (mm-unstable), I don't recall anything in 6.7. > ---------------------------------------- > [ 0.000000][ T0] Linux version 6.8.0-rc5-00163-gffd2cb6b718e (root@ubuntu) (Ubuntu clang version 14.0.0-1ubuntu1.1, Ubuntu LLD 14.0.0) #1094 SMP PREEMPT_DYNAMIC Fri Feb 23 01:45:21 UTC 2024 > [ 50.026544][ T2974] ===================================================== > [ 50.030627][ T2974] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 > [ 50.034611][ T2974] obj_malloc+0x6cc/0x7b0 > obj_malloc at mm/zsmalloc.c:0 > [ 50.037250][ T2974] zs_malloc+0xdbd/0x1400 > zs_malloc at mm/zsmalloc.c:0 > [ 50.039852][ T2974] zs_zpool_malloc+0xa5/0x1b0 > zs_zpool_malloc at mm/zsmalloc.c:372 > [ 50.044707][ T2974] zpool_malloc+0x110/0x150 > zpool_malloc at mm/zpool.c:258 > [ 50.049607][ T2974] zswap_store+0x2bbb/0x3d30 > zswap_store at mm/zswap.c:1637 > [ 50.054463][ T2974] swap_writepage+0x15b/0x4f0 > swap_writepage at mm/page_io.c:198 > [ 50.059392][ T2974] pageout+0x41d/0xef0 > pageout at mm/vmscan.c:654 > [ 50.064057][ T2974] shrink_folio_list+0x4d7a/0x7480 > shrink_folio_list at mm/vmscan.c:1316 > [ 50.069176][ T2974] evict_folios+0x30f1/0x5170 > evict_folios at mm/vmscan.c:4521 > [ 50.074082][ T2974] try_to_shrink_lruvec+0x983/0xd20 > [ 50.079352][ T2974] shrink_one+0x72d/0xeb0 > [ 50.084061][ T2974] shrink_many+0x70d/0x10b0 > [ 50.088859][ T2974] lru_gen_shrink_node+0x577/0x850 > [ 50.094192][ T2974] shrink_node+0x13d/0x1de0 > [ 50.099028][ T2974] shrink_zones+0x878/0x14a0 > [ 50.103958][ T2974] do_try_to_free_pages+0x2ac/0x16a0 > [ 50.109138][ T2974] try_to_free_pages+0xd9e/0x1910 > [ 50.114190][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 > [ 50.119555][ T2974] __alloc_pages+0xb8c/0x1050 > [ 50.124472][ T2974] alloc_pages_mpol+0x8e0/0xc80 > [ 50.129367][ T2974] alloc_pages+0x224/0x240 > [ 50.134022][ T2974] pipe_write+0xabe/0x2ba0 > [ 50.138632][ T2974] vfs_write+0xfb0/0x1b80 > [ 50.143171][ T2974] ksys_write+0x275/0x500 > [ 50.147723][ T2974] __x64_sys_write+0xdf/0x120 > [ 50.152431][ T2974] do_syscall_64+0xd1/0x1b0 > [ 50.157106][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b > [ 50.162382][ T2974] > [ 50.165956][ T2974] Uninit was stored to memory at: > [ 50.170819][ T2974] obj_malloc+0x70a/0x7b0 > set_freeobj at mm/zsmalloc.c:476 > (inlined by) obj_malloc at mm/zsmalloc.c:1333 > [ 50.175341][ T2974] zs_malloc+0xdbd/0x1400 > zs_malloc at mm/zsmalloc.c:0 > [ 50.179923][ T2974] zs_zpool_malloc+0xa5/0x1b0 > zs_zpool_malloc at mm/zsmalloc.c:372 > [ 50.184636][ T2974] zpool_malloc+0x110/0x150 > zpool_malloc at mm/zpool.c:258 > [ 50.189257][ T2974] zswap_store+0x2bbb/0x3d30 > zswap_store at mm/zswap.c:1637 > [ 50.193918][ T2974] swap_writepage+0x15b/0x4f0 > swap_writepage at mm/page_io.c:198 > [ 50.198615][ T2974] pageout+0x41d/0xef0 > pageout at mm/vmscan.c:654 > [ 50.203012][ T2974] shrink_folio_list+0x4d7a/0x7480 > shrink_folio_list at mm/vmscan.c:1316 > [ 50.207772][ T2974] evict_folios+0x30f1/0x5170 > evict_folios at mm/vmscan.c:4521 > [ 50.212321][ T2974] try_to_shrink_lruvec+0x983/0xd20 > [ 50.217092][ T2974] shrink_one+0x72d/0xeb0 > [ 50.221441][ T2974] shrink_many+0x70d/0x10b0 > [ 50.225891][ T2974] lru_gen_shrink_node+0x577/0x850 > [ 50.230614][ T2974] shrink_node+0x13d/0x1de0 > [ 50.235128][ T2974] shrink_zones+0x878/0x14a0 > [ 50.239646][ T2974] do_try_to_free_pages+0x2ac/0x16a0 > [ 50.244461][ T2974] try_to_free_pages+0xd9e/0x1910 > [ 50.249151][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 > [ 50.254148][ T2974] __alloc_pages+0xb8c/0x1050 > [ 50.258679][ T2974] alloc_pages_mpol+0x8e0/0xc80 > [ 50.263289][ T2974] alloc_pages+0x224/0x240 > [ 50.267767][ T2974] pipe_write+0xabe/0x2ba0 > [ 50.272190][ T2974] vfs_write+0xfb0/0x1b80 > [ 50.276543][ T2974] ksys_write+0x275/0x500 > [ 50.280931][ T2974] __x64_sys_write+0xdf/0x120 > [ 50.289451][ T2974] do_syscall_64+0xd1/0x1b0 > [ 50.303402][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b > [ 50.318721][ T2974] > [ 50.328931][ T2974] Uninit was created at: > [ 50.341845][ T2974] free_unref_page_prepare+0x130/0xfc0 > arch_static_branch_jump at arch/x86/include/asm/jump_label.h:55 > (inlined by) memcg_kmem_online at include/linux/memcontrol.h:1840 > (inlined by) free_pages_prepare at mm/page_alloc.c:1096 > (inlined by) free_unref_page_prepare at mm/page_alloc.c:2346 > [ 50.356492][ T2974] free_unref_page_list+0x139/0x1050 > free_unref_page_list at mm/page_alloc.c:2532 > [ 50.370898][ T2974] shrink_folio_list+0x7139/0x7480 > list_empty at include/linux/list.h:373 > (inlined by) list_splice at include/linux/list.h:545 > (inlined by) shrink_folio_list at mm/vmscan.c:1490 > [ 50.385025][ T2974] evict_folios+0x30f1/0x5170 > evict_folios at mm/vmscan.c:4521 > [ 50.398448][ T2974] try_to_shrink_lruvec+0x983/0xd20 > [ 50.412660][ T2974] shrink_one+0x72d/0xeb0 > [ 50.425591][ T2974] shrink_many+0x70d/0x10b0 > [ 50.438827][ T2974] lru_gen_shrink_node+0x577/0x850 > [ 50.454390][ T2974] shrink_node+0x13d/0x1de0 > [ 50.479401][ T2974] shrink_zones+0x878/0x14a0 > [ 50.529610][ T2974] do_try_to_free_pages+0x2ac/0x16a0 > [ 50.544397][ T2974] try_to_free_pages+0xd9e/0x1910 > [ 50.559556][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 > [ 50.574932][ T2974] __alloc_pages+0xb8c/0x1050 > [ 50.589024][ T2974] alloc_pages_mpol+0x8e0/0xc80 > [ 50.603421][ T2974] alloc_pages+0x224/0x240 > [ 50.616483][ T2974] pipe_write+0xabe/0x2ba0 > [ 50.629601][ T2974] vfs_write+0xfb0/0x1b80 > [ 50.643009][ T2974] ksys_write+0x275/0x500 > [ 50.656157][ T2974] __x64_sys_write+0xdf/0x120 > [ 50.670080][ T2974] do_syscall_64+0xd1/0x1b0 > [ 50.683405][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b > [ 50.698626][ T2974] > ---------------------------------------- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 4:43 ` Sergey Senozhatsky @ 2024-02-23 15:22 ` Tetsuo Handa 2024-02-23 23:54 ` [PATCH] x86: disable non-instrumented version of copy_page when KMSAN is enabled Tetsuo Handa 2024-02-24 14:23 ` [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Sergey Senozhatsky 0 siblings, 2 replies; 17+ messages in thread From: Tetsuo Handa @ 2024-02-23 15:22 UTC (permalink / raw) To: Sergey Senozhatsky, Alexander Potapenko Cc: Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, linux-mm, kasan-dev, Mark-PK Tsai On 2024/02/23 13:43, Sergey Senozhatsky wrote: > On (24/02/23 11:10), Tetsuo Handa wrote: >> >> I can observe this bug during evict_folios() from 6.7.0 to 6.8.0-rc5-00163-gffd2cb6b718e. >> Since I haven't observed with 6.6.0, this bug might be introduced in 6.7 cycle. > > Can we please run a bisect? Bisection pointed at commit afb2d666d025 ("zsmalloc: use copy_page for full page copy"), for copy_page() is implemented as non-instrumented code where KMSAN cannot handle. On x86_64, copy_page() is defined at arch/x86/lib/copy_page_64.S as below. ---------------------------------------- /* * Some CPUs run faster using the string copy instructions (sane microcode). * It is also a lot simpler. Use this when possible. But, don't use streaming * copy unless the CPU indicates X86_FEATURE_REP_GOOD. Could vary the * prefetch distance based on SMP/UP. */ ALIGN SYM_FUNC_START(copy_page) ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD movl $4096/8, %ecx rep movsq RET SYM_FUNC_END(copy_page) EXPORT_SYMBOL(copy_page) ---------------------------------------- To fix this problem, we need to implement copy_page() etc. in a way KMSAN can handle. Question to KASAN people: Is it possible to add annotation for KMSAN into assembly code? Do we need to disable assembly version and force use of C version when KMSAN is enabled? > > There are some zsmalloc patches for 6.8 (mm-unstable), I don't recall > anything in 6.7. > >> ---------------------------------------- >> [ 0.000000][ T0] Linux version 6.8.0-rc5-00163-gffd2cb6b718e (root@ubuntu) (Ubuntu clang version 14.0.0-1ubuntu1.1, Ubuntu LLD 14.0.0) #1094 SMP PREEMPT_DYNAMIC Fri Feb 23 01:45:21 UTC 2024 >> [ 50.026544][ T2974] ===================================================== >> [ 50.030627][ T2974] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 >> [ 50.034611][ T2974] obj_malloc+0x6cc/0x7b0 >> obj_malloc at mm/zsmalloc.c:0 >> [ 50.037250][ T2974] zs_malloc+0xdbd/0x1400 >> zs_malloc at mm/zsmalloc.c:0 >> [ 50.039852][ T2974] zs_zpool_malloc+0xa5/0x1b0 >> zs_zpool_malloc at mm/zsmalloc.c:372 >> [ 50.044707][ T2974] zpool_malloc+0x110/0x150 >> zpool_malloc at mm/zpool.c:258 >> [ 50.049607][ T2974] zswap_store+0x2bbb/0x3d30 >> zswap_store at mm/zswap.c:1637 >> [ 50.054463][ T2974] swap_writepage+0x15b/0x4f0 >> swap_writepage at mm/page_io.c:198 >> [ 50.059392][ T2974] pageout+0x41d/0xef0 >> pageout at mm/vmscan.c:654 >> [ 50.064057][ T2974] shrink_folio_list+0x4d7a/0x7480 >> shrink_folio_list at mm/vmscan.c:1316 >> [ 50.069176][ T2974] evict_folios+0x30f1/0x5170 >> evict_folios at mm/vmscan.c:4521 >> [ 50.074082][ T2974] try_to_shrink_lruvec+0x983/0xd20 >> [ 50.079352][ T2974] shrink_one+0x72d/0xeb0 >> [ 50.084061][ T2974] shrink_many+0x70d/0x10b0 >> [ 50.088859][ T2974] lru_gen_shrink_node+0x577/0x850 >> [ 50.094192][ T2974] shrink_node+0x13d/0x1de0 >> [ 50.099028][ T2974] shrink_zones+0x878/0x14a0 >> [ 50.103958][ T2974] do_try_to_free_pages+0x2ac/0x16a0 >> [ 50.109138][ T2974] try_to_free_pages+0xd9e/0x1910 >> [ 50.114190][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 >> [ 50.119555][ T2974] __alloc_pages+0xb8c/0x1050 >> [ 50.124472][ T2974] alloc_pages_mpol+0x8e0/0xc80 >> [ 50.129367][ T2974] alloc_pages+0x224/0x240 >> [ 50.134022][ T2974] pipe_write+0xabe/0x2ba0 >> [ 50.138632][ T2974] vfs_write+0xfb0/0x1b80 >> [ 50.143171][ T2974] ksys_write+0x275/0x500 >> [ 50.147723][ T2974] __x64_sys_write+0xdf/0x120 >> [ 50.152431][ T2974] do_syscall_64+0xd1/0x1b0 >> [ 50.157106][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b >> [ 50.162382][ T2974] >> [ 50.165956][ T2974] Uninit was stored to memory at: >> [ 50.170819][ T2974] obj_malloc+0x70a/0x7b0 >> set_freeobj at mm/zsmalloc.c:476 >> (inlined by) obj_malloc at mm/zsmalloc.c:1333 >> [ 50.175341][ T2974] zs_malloc+0xdbd/0x1400 >> zs_malloc at mm/zsmalloc.c:0 >> [ 50.179923][ T2974] zs_zpool_malloc+0xa5/0x1b0 >> zs_zpool_malloc at mm/zsmalloc.c:372 >> [ 50.184636][ T2974] zpool_malloc+0x110/0x150 >> zpool_malloc at mm/zpool.c:258 >> [ 50.189257][ T2974] zswap_store+0x2bbb/0x3d30 >> zswap_store at mm/zswap.c:1637 >> [ 50.193918][ T2974] swap_writepage+0x15b/0x4f0 >> swap_writepage at mm/page_io.c:198 >> [ 50.198615][ T2974] pageout+0x41d/0xef0 >> pageout at mm/vmscan.c:654 >> [ 50.203012][ T2974] shrink_folio_list+0x4d7a/0x7480 >> shrink_folio_list at mm/vmscan.c:1316 >> [ 50.207772][ T2974] evict_folios+0x30f1/0x5170 >> evict_folios at mm/vmscan.c:4521 >> [ 50.212321][ T2974] try_to_shrink_lruvec+0x983/0xd20 >> [ 50.217092][ T2974] shrink_one+0x72d/0xeb0 >> [ 50.221441][ T2974] shrink_many+0x70d/0x10b0 >> [ 50.225891][ T2974] lru_gen_shrink_node+0x577/0x850 >> [ 50.230614][ T2974] shrink_node+0x13d/0x1de0 >> [ 50.235128][ T2974] shrink_zones+0x878/0x14a0 >> [ 50.239646][ T2974] do_try_to_free_pages+0x2ac/0x16a0 >> [ 50.244461][ T2974] try_to_free_pages+0xd9e/0x1910 >> [ 50.249151][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 >> [ 50.254148][ T2974] __alloc_pages+0xb8c/0x1050 >> [ 50.258679][ T2974] alloc_pages_mpol+0x8e0/0xc80 >> [ 50.263289][ T2974] alloc_pages+0x224/0x240 >> [ 50.267767][ T2974] pipe_write+0xabe/0x2ba0 >> [ 50.272190][ T2974] vfs_write+0xfb0/0x1b80 >> [ 50.276543][ T2974] ksys_write+0x275/0x500 >> [ 50.280931][ T2974] __x64_sys_write+0xdf/0x120 >> [ 50.289451][ T2974] do_syscall_64+0xd1/0x1b0 >> [ 50.303402][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b >> [ 50.318721][ T2974] >> [ 50.328931][ T2974] Uninit was created at: >> [ 50.341845][ T2974] free_unref_page_prepare+0x130/0xfc0 >> arch_static_branch_jump at arch/x86/include/asm/jump_label.h:55 >> (inlined by) memcg_kmem_online at include/linux/memcontrol.h:1840 >> (inlined by) free_pages_prepare at mm/page_alloc.c:1096 >> (inlined by) free_unref_page_prepare at mm/page_alloc.c:2346 >> [ 50.356492][ T2974] free_unref_page_list+0x139/0x1050 >> free_unref_page_list at mm/page_alloc.c:2532 >> [ 50.370898][ T2974] shrink_folio_list+0x7139/0x7480 >> list_empty at include/linux/list.h:373 >> (inlined by) list_splice at include/linux/list.h:545 >> (inlined by) shrink_folio_list at mm/vmscan.c:1490 >> [ 50.385025][ T2974] evict_folios+0x30f1/0x5170 >> evict_folios at mm/vmscan.c:4521 >> [ 50.398448][ T2974] try_to_shrink_lruvec+0x983/0xd20 >> [ 50.412660][ T2974] shrink_one+0x72d/0xeb0 >> [ 50.425591][ T2974] shrink_many+0x70d/0x10b0 >> [ 50.438827][ T2974] lru_gen_shrink_node+0x577/0x850 >> [ 50.454390][ T2974] shrink_node+0x13d/0x1de0 >> [ 50.479401][ T2974] shrink_zones+0x878/0x14a0 >> [ 50.529610][ T2974] do_try_to_free_pages+0x2ac/0x16a0 >> [ 50.544397][ T2974] try_to_free_pages+0xd9e/0x1910 >> [ 50.559556][ T2974] __alloc_pages_slowpath+0x147a/0x2bd0 >> [ 50.574932][ T2974] __alloc_pages+0xb8c/0x1050 >> [ 50.589024][ T2974] alloc_pages_mpol+0x8e0/0xc80 >> [ 50.603421][ T2974] alloc_pages+0x224/0x240 >> [ 50.616483][ T2974] pipe_write+0xabe/0x2ba0 >> [ 50.629601][ T2974] vfs_write+0xfb0/0x1b80 >> [ 50.643009][ T2974] ksys_write+0x275/0x500 >> [ 50.656157][ T2974] __x64_sys_write+0xdf/0x120 >> [ 50.670080][ T2974] do_syscall_64+0xd1/0x1b0 >> [ 50.683405][ T2974] entry_SYSCALL_64_after_hwframe+0x63/0x6b >> [ 50.698626][ T2974] >> ---------------------------------------- ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH] x86: disable non-instrumented version of copy_page when KMSAN is enabled 2024-02-23 15:22 ` Tetsuo Handa @ 2024-02-23 23:54 ` Tetsuo Handa 2024-02-24 6:27 ` [PATCH v2] " Tetsuo Handa 2024-02-24 14:23 ` [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Sergey Senozhatsky 1 sibling, 1 reply; 17+ messages in thread From: Tetsuo Handa @ 2024-02-23 23:54 UTC (permalink / raw) To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, the arch/x86 maintainers, H. Peter Anvin Cc: Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, linux-mm, kasan-dev, Mark-PK Tsai, Sergey Senozhatsky, Alexander Potapenko I found that commit afb2d666d025 ("zsmalloc: use copy_page for full page copy") caused KMSAN warning. We need to fallback to instrumented version when KMSAN is enabled. [ 50.030627][ T2974] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 [ 50.165956][ T2974] Uninit was stored to memory at: [ 50.170819][ T2974] obj_malloc+0x70a/0x7b0 [ 50.328931][ T2974] Uninit was created at: [ 50.341845][ T2974] free_unref_page_prepare+0x130/0xfc0 Since the destination page likely already holds previously written value (i.e. KMSAN considers that the page was already initialized), whether to globally enforce an instrumented version when KMSAN is enabled might be questionable. But since finding why KMSAN considers that value is not initialized is difficult (developers tend to choose optimized version without knowing KMSAN), let's choose human-friendly version. That is, since arch/x86/include/asm/page_32.h implements copy_page() using memcpy(), let arch/x86/include/asm/page_64.h implement copy_page() using memcpy() when KMSAN is enabled. Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> --- arch/x86/include/asm/page_64.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..f13bba3a9dab 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -58,7 +58,16 @@ static inline void clear_page(void *page) : "cc", "memory", "rax", "rcx"); } +#ifdef CONFIG_KMSAN +/* Use of non-instrumented assembly version confuses KMSAN. */ +void *memcpy(void *to, const void *from, __kernel_size_t len); +static inline void copy_page(void *to, void *from) +{ + memcpy(to, from, PAGE_SIZE); +} +#else void copy_page(void *to, void *from); +#endif #ifdef CONFIG_X86_5LEVEL /* -- 2.34.1 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2] x86: disable non-instrumented version of copy_page when KMSAN is enabled 2024-02-23 23:54 ` [PATCH] x86: disable non-instrumented version of copy_page when KMSAN is enabled Tetsuo Handa @ 2024-02-24 6:27 ` Tetsuo Handa 0 siblings, 0 replies; 17+ messages in thread From: Tetsuo Handa @ 2024-02-24 6:27 UTC (permalink / raw) To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, the arch/x86 maintainers, H. Peter Anvin Cc: Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, linux-mm, kasan-dev, Mark-PK Tsai, Sergey Senozhatsky, Alexander Potapenko I found that commit afb2d666d025 ("zsmalloc: use copy_page for full page copy") caused a false-positive KMSAN warning. [ 50.030627][ T2974] BUG: KMSAN: use-after-free in obj_malloc+0x6cc/0x7b0 [ 50.165956][ T2974] Uninit was stored to memory at: [ 50.170819][ T2974] obj_malloc+0x70a/0x7b0 [ 50.328931][ T2974] Uninit was created at: [ 50.341845][ T2974] free_unref_page_prepare+0x130/0xfc0 We need to use instrumented version when KMSAN is enabled. Let arch/x86/include/asm/page_64.h implement copy_page() using memcpy() like arch/x86/include/asm/page_32.h does. Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> --- arch/x86/include/asm/page_64.h | 9 +++++++++ 1 file changed, 9 insertions(+) Changes in v2: Update explanation, for I misinterpreted source/destination direction. diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..f13bba3a9dab 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -58,7 +58,16 @@ static inline void clear_page(void *page) : "cc", "memory", "rax", "rcx"); } +#ifdef CONFIG_KMSAN +/* Use of non-instrumented assembly version confuses KMSAN. */ +void *memcpy(void *to, const void *from, __kernel_size_t len); +static inline void copy_page(void *to, void *from) +{ + memcpy(to, from, PAGE_SIZE); +} +#else void copy_page(void *to, void *from); +#endif #ifdef CONFIG_X86_5LEVEL /* -- 2.34.1 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() 2024-02-23 15:22 ` Tetsuo Handa 2024-02-23 23:54 ` [PATCH] x86: disable non-instrumented version of copy_page when KMSAN is enabled Tetsuo Handa @ 2024-02-24 14:23 ` Sergey Senozhatsky 1 sibling, 0 replies; 17+ messages in thread From: Sergey Senozhatsky @ 2024-02-24 14:23 UTC (permalink / raw) To: Tetsuo Handa Cc: Sergey Senozhatsky, Alexander Potapenko, Johannes Weiner, Yosry Ahmed, Nhat Pham, Minchan Kim, linux-mm, kasan-dev, Mark-PK Tsai On (24/02/24 00:22), Tetsuo Handa wrote: > On 2024/02/23 13:43, Sergey Senozhatsky wrote: > > On (24/02/23 11:10), Tetsuo Handa wrote: > >> > >> I can observe this bug during evict_folios() from 6.7.0 to 6.8.0-rc5-00163-gffd2cb6b718e. > >> Since I haven't observed with 6.6.0, this bug might be introduced in 6.7 cycle. > > > > Can we please run a bisect? > > Bisection pointed at commit afb2d666d025 ("zsmalloc: use copy_page for full page copy"), > for copy_page() is implemented as non-instrumented code where KMSAN cannot handle. > On x86_64, copy_page() is defined at arch/x86/lib/copy_page_64.S as below. Thank you so much. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-02-24 14:23 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-02-23 2:10 [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Tetsuo Handa 2024-02-23 2:27 ` Yosry Ahmed 2024-02-23 4:48 ` Sergey Senozhatsky 2024-02-23 4:50 ` Yosry Ahmed 2024-02-23 4:56 ` Sergey Senozhatsky 2024-02-23 4:58 ` Sergey Senozhatsky 2024-02-23 5:05 ` Yosry Ahmed 2024-02-23 5:19 ` Sergey Senozhatsky 2024-02-23 5:23 ` Chengming Zhou 2024-02-23 5:29 ` Sergey Senozhatsky 2024-02-23 9:26 ` Tetsuo Handa 2024-02-23 10:10 ` Chengming Zhou 2024-02-23 4:43 ` Sergey Senozhatsky 2024-02-23 15:22 ` Tetsuo Handa 2024-02-23 23:54 ` [PATCH] x86: disable non-instrumented version of copy_page when KMSAN is enabled Tetsuo Handa 2024-02-24 6:27 ` [PATCH v2] " Tetsuo Handa 2024-02-24 14:23 ` [mm/page_alloc or mm/vmscan or mm/zswap] use-after-free in obj_malloc() Sergey Senozhatsky
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).