Re: [PATCH v3 10/18] mm: Allow non-hugetlb large folios to be batch processed

[Ryan Roberts <ryan.roberts@arm.com>]

On 10/03/2024 11:11, Matthew Wilcox wrote:
> On Sun, Mar 10, 2024 at 11:01:06AM +0000, Ryan Roberts wrote:
>>> So after my patch, instead of calling (in order):
>>>
>>> 	page_cache_release(folio);
>>> 	folio_undo_large_rmappable(folio);
>>> 	mem_cgroup_uncharge(folio);
>>> 	free_unref_page()
>>>
>>> it calls:
>>>
>>> 	__page_cache_release(folio, &lruvec, &flags);
>>> 	mem_cgroup_uncharge_folios()
>>> 	folio_undo_large_rmappable(folio);
>>
>> I was just looking at this again, and something pops out...
>>
>> You have swapped the order of folio_undo_large_rmappable() and
>> mem_cgroup_uncharge(). But folio_undo_large_rmappable() calls
>> get_deferred_split_queue() which tries to get the split queue from
>> folio_memcg(folio) first and falls back to pgdat otherwise. If you are now
>> calling mem_cgroup_uncharge_folios() first, will that remove the folio from the
>> cgroup? Then we are operating on the wrong list? (just a guess based on the name
>> of the function...)
> 
> Oh my.  You've got it.  This explains everything.  Thank you!

I've just taken today's mm-unstable, added your official patch to fix the ordering and applied my large folio swap-out series on top (v4, which I haven't posted yet). In testing that, I'm seeing another oops :-( 

That's exactly how I discovered the original problem, and was hoping that with your fix, this would unblock me. Given I can only repro this when my changes are on top, I guess my code is most likely buggy, but perhaps you can take a quick look at the oops and tell me what you think?

[   96.372503] BUG: Bad page state in process usemem  pfn:be502
[   96.373336] page: refcount:0 mapcount:0 mapping:000000005abfa8d5 index:0x0 pfn:0xbe502
[   96.374341] aops:0x0 ino:fffffc0001f940c8
[   96.374893] flags: 0x7fff8000000000(node=0|zone=0|lastcpupid=0xffff)
[   96.375653] page_type: 0xffffffff()
[   96.376071] raw: 007fff8000000000 0000000000000000 fffffc0001f94090 ffff0000c99ee860
[   96.377055] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[   96.378650] page dumped because: non-NULL mapping
[   96.379828] Modules linked in: binfmt_misc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua sch_fq_codel drm efi_pstore ip_tables x_tables autofs4 xfs btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 crct10dif_ce ghash_ce sha2_ce virtio_net sha256_arm64 net_failover sha1_ce virtio_blk failover virtio_scsi virtio_rng aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[   96.386802] CPU: 13 PID: 4713 Comm: usemem Not tainted 6.8.0-rc5-ryarob01-swap-out-v4 #2
[   96.387691] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[   96.388887] Call trace:
[   96.389348]  dump_backtrace+0x9c/0x128
[   96.390213]  show_stack+0x20/0x38
[   96.390688]  dump_stack_lvl+0x78/0xc8
[   96.391163]  dump_stack+0x18/0x28
[   96.391545]  bad_page+0x88/0x128
[   96.391893]  get_page_from_freelist+0xa94/0x1bc0
[   96.392407]  __alloc_pages+0x194/0x10b0
[   96.392833]  alloc_pages_mpol+0x98/0x278
[   96.393278]  vma_alloc_folio+0x74/0xd8
[   96.393674]  __handle_mm_fault+0x7ac/0x1470
[   96.394146]  handle_mm_fault+0x70/0x2c8
[   96.394575]  do_page_fault+0x100/0x530
[   96.395013]  do_translation_fault+0xa4/0xd0
[   96.395476]  do_mem_abort+0x4c/0xa8
[   96.395869]  el0_da+0x30/0xa8
[   96.396229]  el0t_64_sync_handler+0xb4/0x130
[   96.396735]  el0t_64_sync+0x1a8/0x1b0
[   96.397133] Disabling lock debugging due to kernel taint
[  112.507052] Adding 36700156k swap on /dev/ram0.  Priority:-2 extents:1 across:36700156k SS
[  113.131515] ------------[ cut here ]------------
[  113.132190] UBSAN: array-index-out-of-bounds in mm/vmscan.c:1654:14
[  113.132892] index 7 is out of range for type 'long unsigned int [5]'
[  113.133617] CPU: 9 PID: 528 Comm: kswapd0 Tainted: G    B              6.8.0-rc5-ryarob01-swap-out-v4 #2
[  113.134705] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  113.135500] Call trace:
[  113.135776]  dump_backtrace+0x9c/0x128
[  113.136218]  show_stack+0x20/0x38
[  113.136574]  dump_stack_lvl+0x78/0xc8
[  113.136964]  dump_stack+0x18/0x28
[  113.137322]  __ubsan_handle_out_of_bounds+0xa0/0xd8
[  113.137885]  isolate_lru_folios+0x57c/0x658
[  113.138352]  shrink_lruvec+0x5b4/0xdf8
[  113.138751]  shrink_node+0x3f0/0x990
[  113.139152]  balance_pgdat+0x3d0/0x810
[  113.139579]  kswapd+0x268/0x568
[  113.139936]  kthread+0x118/0x128
[  113.140289]  ret_from_fork+0x10/0x20
[  113.140686] ---[ end trace ]---

The UBSAN issue reported for mm/vmscan.c:1654 is:

nr_skipped[folio_zonenum(folio)] += nr_pages;

nr_skipped is a stack array of 5 elements. So I guess folio_zonemem(folio) is returning 7. That comes from the flags. I guess this is most likely just a side effect of the corrupted folio due to someone writing to it while its on the free list?