From: Vasily Averin <vvs@virtuozzo.com>
To: Michal Hocko <mhocko@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Vladimir Davydov <vdavydov.dev@gmail.com>,
Andrew Morton <akpm@linux-foundation.org>
Cc: Roman Gushchin <guro@fb.com>, Uladzislau Rezki <urezki@gmail.com>,
Vlastimil Babka <vbabka@suse.cz>,
Shakeel Butt <shakeelb@google.com>,
Mel Gorman <mgorman@techsingularity.net>,
cgroups@vger.kernel.org, linux-mm@kvack.org,
linux-kernel@vger.kernel.org, kernel@openvz.org
Subject: [PATCH memcg 0/1] false global OOM triggered by memcg-limited task
Date: Mon, 18 Oct 2021 11:13:52 +0300 [thread overview]
Message-ID: <9d10df01-0127-fb40-81c3-cc53c9733c3e@virtuozzo.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]
While checking the patches fixed broken memcg accounting in vmalloc I found
another issue: a false global OOM triggered by memcg-limited user space task.
I executed vmalloc-eater inside a memcg limited LXC container in a loop, checked
that it does not consume host memory beyond the assigned limit, triggers memcg OOM
and generates "Memory cgroup out of memory" messages. Everything was as expected.
However I was surprised to find quite rare global OOM messages too.
I set sysctl vm.panic_on_oom to 1, repeated the test and successfully
crashed the node.
Dmesg showed that global OOM was detected on 16 GB node with ~10 GB of free memory.
syz-executor invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=1000
CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
Call Trace:
dump_stack_lvl+0x57/0x72
dump_header+0x4a/0x2c1
out_of_memory.cold+0xa/0x7e
pagefault_out_of_memory+0x46/0x60
exc_page_fault+0x79/0x2b0
asm_exc_page_fault+0x1e/0x30
...
Mem-Info:
Node 0 DMA: 0*4kB 0*8kB <...> = 13296kB
Node 0 DMA32: 705*4kB (UM) <...> = 2586964kB
Node 0 Normal: 2743*4kB (UME) <...> = 6904828kB
...
4095866 pages RAM
...
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Full dmesg can be found in attached file.
How could this happen?
User-space task inside the memcg-limited container generated a page fault,
its handler do_user_addr_fault() called handle_mm_fault which could not
allocate the page due to exceeding the memcg limit and returned VM_FAULT_OOM.
Then do_user_addr_fault() called pagefault_out_of_memory() which executed
out_of_memory() without set of memcg.
Partially this problem depends on one of my recent patches, disabled unlimited
memory allocation for dying tasks. However I think the problem can happen
on non-killed tasks too, for example because of kmem limit.
At present do_user_addr_fault() does not know why page allocation was failed,
i.e. was it global or memcg OOM. I propose to save this information in new flag
on task_struct. It can be set in case of memcg restrictons in
obj_cgroup_charge_pages() (for memory controller) and in try_charge_memcg()
(for kmem controller). Then it can be used in mem_cgroup_oom_synchronize()
called inside pagefault_out_of_memory():
in case of memcg-related restrictions it will not trigger fake global OOM and
returns to user space which will retry the fault or kill the process if it got
a fatal signal.
Thank you,
Vasily Averin
Vasily Averin (1):
memcg: prevent false global OOM trigggerd by memcg limited task.
include/linux/sched.h | 1 +
mm/memcontrol.c | 12 +++++++++---
2 files changed, 10 insertions(+), 3 deletions(-)
--
2.32.0
[-- Attachment #2: dmesg-oom.txt --]
[-- Type: text/plain, Size: 25650 bytes --]
[59622.176098] syz-executor invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=1000
[59622.178633] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.180840] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.182562] Call Trace:
[59622.183525] dump_stack_lvl+0x57/0x72
[59622.184782] dump_header+0x4a/0x2c1
[59622.185929] oom_kill_process.cold+0xb/0x10
[59622.187203] out_of_memory+0x229/0x5b0
[59622.188399] mem_cgroup_out_of_memory+0x111/0x130
[59622.189773] try_charge_memcg+0x693/0x720
[59622.191013] ? kvm_sched_clock_read+0x14/0x30
[59622.192318] charge_memcg+0x57/0x170
[59622.193482] mem_cgroup_swapin_charge_page+0x99/0x1d0
[59622.194932] do_swap_page+0x916/0xbf0
[59622.196110] ? __lock_acquire+0x3b3/0x1e00
[59622.197377] __handle_mm_fault+0xa5f/0x14e0
[59622.198649] ? lock_acquire+0xc4/0x2e0
[59622.199847] handle_mm_fault+0x149/0x3f0
[59622.201071] do_user_addr_fault+0x1f4/0x6c0
[59622.202346] exc_page_fault+0x79/0x2b0
[59622.203545] ? asm_exc_page_fault+0x8/0x30
[59622.204802] asm_exc_page_fault+0x1e/0x30
[59622.206043] RIP: 0033:0x41a948
[59622.207115] Code: 64 83 0c 25 08 03 00 00 10 64 48 8b 3c 25 00 03 00 00 e8 8b fe ff ff f4 66 2e 0f 1f 84 00 00 00 00 00 f7 c7 02 00 00 00 75 27 <64> 8b 04 25 08 03 00 00 41 89 c3 41 83 e3 fd f0 64 44 0f b1 1c 25
[59622.211557] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.213058] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.214919] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.216772] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.218623] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.220467] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
[59622.222367] memory: usage 524268kB, limit 524288kB, failcnt 28608
[59622.224112] memory+swap: usage 554776kB, limit 9007199254740988kB, failcnt 0
[59622.225982] kmem: usage 524168kB, limit 9007199254740988kB, failcnt 0
[59622.227740] Memory cgroup stats for /lxc.payload.test:
[59622.234613] anon 0
file 122880
kernel_stack 1294336
pagetables 3665920
percpu 490896
sock 0
shmem 0
file_mapped 12288
file_dirty 0
file_writeback 0
swapcached 49008640
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 0
active_anon 0
inactive_file 110592
active_file 0
unevictable 0
slab_reclaimable 1513744
slab_unreclaimable 14117008
slab 15630752
workingset_refault_anon 3800
workingset_refault_file 24974
workingset_activate_anon 923
workingset_activate_file 414
workingset_restore_anon 422
workingset_restore_file 190
workingset_nodereclaim 74
pgfault 120040
pgmajfault 4766
pgrefill 15999
pgscan 486263
pgsteal 44646
pgactivate 13066
pgdeactivate 14277
pglazyfree 60
pglazyfreed 34
thp_fault_alloc 0
thp_collapse_alloc 0
[59622.265206] Tasks state (memory values in pages):
[59622.266454] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[59622.268394] [ 14696] 0 14696 56226 690 90112 145 0 bash
[59622.270246] [ 14746] 0 14746 262198 0 266240 2900 0 syz-execprog
[59622.272217] [ 14764] 0 14764 12552 0 36864 16 0 syz-executor
[59622.274217] [ 14765] 0 14765 12552 0 36864 16 0 syz-executor
[59622.276165] [ 14769] 0 14769 12552 0 36864 16 0 syz-executor
[59622.278091] [ 14772] 0 14772 12552 0 36864 17 0 syz-executor
[59622.280023] [ 14775] 0 14775 12552 0 36864 17 0 syz-executor
[59622.281949] [ 14777] 0 14777 12552 0 36864 16 0 syz-executor
[59622.283847] [ 14781] 0 14781 12552 0 36864 16 0 syz-executor
[59622.285737] [ 14783] 0 14783 12552 0 36864 17 0 syz-executor
[59622.287673] [ 14787] 0 14787 12552 0 36864 16 0 syz-executor
[59622.287968] systemd-journald[648]: Compressed data object 720 -> 393 using ZSTD
[59622.289564] [ 14800] 0 14800 12552 0 36864 16 0 syz-executor
[59622.293144] [ 14803] 0 14803 12552 0 36864 16 0 syz-executor
[59622.295054] [ 14805] 0 14805 12552 0 36864 16 0 syz-executor
[59622.296928] [ 14812] 0 14812 12552 0 36864 17 0 syz-executor
[59622.298796] [ 14814] 0 14814 12552 0 36864 16 0 syz-executor
[59622.300663] [ 14817] 0 14817 12552 0 36864 17 0 syz-executor
[59622.302528] [ 14820] 0 14820 12552 0 36864 16 0 syz-executor
[59622.304399] [ 14779] 0 14779 12551 0 49152 37 0 syz-executor
[59622.306271] [ 15280] 0 15279 12584 0 69632 0 1000 syz-executor
[59622.308144] [ 14767] 0 14767 12551 0 49152 41 0 syz-executor
[59622.310046] [ 15282] 0 15281 12584 0 69632 0 1000 syz-executor
[59622.311926] [ 14766] 0 14766 12551 0 49152 44 0 syz-executor
[59622.313812] [ 15286] 0 15285 12584 0 65536 0 1000 syz-executor
[59622.315704] [ 14782] 0 14782 12551 0 49152 35 0 syz-executor
[59622.317621] [ 15284] 0 15283 12584 0 65536 0 1000 syz-executor
[59622.319570] [ 14784] 0 14784 12551 0 49152 36 0 syz-executor
[59622.321496] [ 15288] 0 15287 12617 0 65536 0 1000 syz-executor
[59622.323454] [ 14776] 0 14776 12551 0 49152 34 0 syz-executor
[59622.325374] [ 15291] 0 15289 12584 0 69632 4 1000 syz-executor
[59622.327293] [ 14813] 0 14813 12551 0 49152 33 0 syz-executor
[59622.329216] [ 15295] 0 15294 12584 0 65536 0 1000 syz-executor
[59622.331139] [ 14819] 0 14819 12551 0 49152 32 0 syz-executor
[59622.333068] [ 15293] 0 15292 12584 0 65536 0 1000 syz-executor
[59622.334991] [ 14821] 0 14821 12551 0 49152 31 0 syz-executor
[59622.336921] [ 15297] 0 15296 12584 0 65536 0 1000 syz-executor
[59622.338875] [ 14831] 0 14831 12551 0 49152 42 0 syz-executor
[59622.340811] [ 15299] 0 15298 12617 0 65536 0 1000 syz-executor
[59622.342752] [ 14804] 0 14804 12551 0 49152 38 0 syz-executor
[59622.344700] [ 15301] 0 15300 12584 0 69632 0 1000 syz-executor
[59622.346649] [ 14816] 0 14816 12551 0 49152 27 0 syz-executor
[59622.348604] [ 15306] 0 15305 12584 0 69632 0 1000 syz-executor
[59622.350615] [ 14774] 0 14774 12551 0 49152 30 0 syz-executor
[59622.352606] [ 15304] 0 15303 12584 0 65536 0 1000 syz-executor
[59622.354575] [ 14807] 0 14807 12551 0 49152 25 0 syz-executor
[59622.356531] [ 15307] 0 15307 12584 0 69632 37 1000 syz-executor
[59622.358481] [ 14801] 0 14801 12551 0 49152 23 0 syz-executor
[59622.360418] [ 15261] 0 15260 12618 0 65536 31 1000 syz-executor
[59622.362355] [ 14788] 0 14788 12551 0 49152 24 0 syz-executor
[59622.364291] [ 15263] 0 15262 12618 0 69632 0 1000 syz-executor
[59622.366269] [ 14503] 0 14503 25205 1712 241664 235 0 systemd-journal
[59622.368288] [ 14736] 0 14736 1637 398 61440 32 0 agetty
[59622.370161] [ 14737] 0 14737 1637 383 61440 31 0 agetty
[59622.372037] [ 14743] 0 14743 1637 398 61440 31 0 agetty
[59622.373939] [ 14744] 0 14744 1637 387 57344 31 0 agetty
[59622.375849] [ 14512] 0 14512 22648 1593 221184 233 0 systemd-logind
[59622.377858] [ 14513] 81 14513 13516 801 155648 138 -900 dbus-daemon
[59622.379867] [ 14734] 0 14734 52899 856 180224 440 0 rsyslogd
[59622.381813] [ 14735] 0 14735 1637 343 61440 31 0 agetty
[59622.383701] [ 14741] 0 14741 5724 572 94208 239 0 crond
[59622.385568] [ 14086] 0 14086 25345 2120 233472 520 0 systemd
[59622.387470] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.test,mems_allowed=0,oom_memcg=/lxc.payload.test,task_memcg=/lxc.payload.test,task=syz-executor,pid=15307,uid=0
[59622.391402] Memory cgroup out of memory: Killed process 15307 (syz-executor) total-vm:50336kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:68kB oom_score_adj:1000
[59622.395544] oom_reaper: reaped process 15307 (syz-executor), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[59622.411181] syz-executor invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=1000
[59622.414272] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.416247] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.417950] Call Trace:
[59622.418901] dump_stack_lvl+0x57/0x72
[59622.420067] dump_header+0x4a/0x2c1
[59622.421200] out_of_memory.cold+0xa/0x7e
[59622.422415] pagefault_out_of_memory+0x46/0x60
[59622.423784] exc_page_fault+0x79/0x2b0
[59622.424966] ? asm_exc_page_fault+0x8/0x30
[59622.426207] asm_exc_page_fault+0x1e/0x30
[59622.427426] RIP: 0033:0x41a948
[59622.428485] Code: Unable to access opcode bytes at RIP 0x41a91e.
[59622.430064] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.431513] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.433299] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.435090] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.436875] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.438691] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
[59622.440547] Mem-Info:
[59622.441516] active_anon:279 inactive_anon:24996 isolated_anon:0
active_file:395743 inactive_file:848017 isolated_file:0
unevictable:0 dirty:159 writeback:21
slab_reclaimable:40993 slab_unreclaimable:45236
mapped:66928 shmem:286 pagetables:2477 bounce:0
kernel_misc_reclaimable:0
free:2376272 free_pcp:7579 free_cma:0
[59622.451730] Node 0 active_anon:1116kB inactive_anon:99984kB active_file:1582972kB inactive_file:3392068kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:267712kB dirty:832kB writeback:84kB shmem:1144kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:5488kB pagetables:9908kB all_unreclaimable? no
[59622.458826] Node 0 DMA free:13296kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[59622.465131] lowmem_reserve[]: 0 2706 15265 15265 15265
[59622.466746] Node 0 DMA32 free:2586964kB min:11968kB low:14960kB high:17952kB reserved_highatomic:0KB active_anon:0kB inactive_anon:20kB active_file:27084kB inactive_file:172344kB unevictable:0kB writepending:0kB present:3129200kB managed:2801520kB mlocked:0kB bounce:0kB free_pcp:6120kB local_pcp:1004kB free_cma:0kB
[59622.473718] lowmem_reserve[]: 0 0 12559 12559 12559
[59622.475311] Node 0 Normal free:6904828kB min:55544kB low:69428kB high:83312kB reserved_highatomic:0KB active_anon:1116kB inactive_anon:99964kB active_file:1555888kB inactive_file:3219724kB unevictable:0kB writepending:916kB present:13238272kB managed:12871060kB mlocked:0kB bounce:0kB free_pcp:24068kB local_pcp:692kB free_cma:0kB
[59622.482647] lowmem_reserve[]: 0 0 0 0 0
[59622.484096] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 13296kB
[59622.488044] Node 0 DMA32: 705*4kB (UM) 624*8kB (UME) 497*16kB (UME) 594*32kB (UME) 532*64kB (UME) 49*128kB (UME) 36*256kB (UE) 36*512kB (UE) 20*1024kB (U) 17*2048kB (UE) 593*4096kB (U) = 2586964kB
[59622.492687] Node 0 Normal: 2743*4kB (UME) 2084*8kB (UME) 536*16kB (UME) 1496*32kB (UME) 1148*64kB (ME) 171*128kB (UME) 115*256kB (M) 94*512kB (M) 98*1024kB (UME) 163*2048kB (UME) 1517*4096kB (UM) = 6904828kB
[59622.497553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[59622.499898] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[59622.502197] 1244068 total pagecache pages
[59622.503688] 4 pages in swap cache
[59622.505046] Swap cache stats: add 54921, delete 54922, find 137/9740
[59622.506956] Free swap = 8357116kB
[59622.508352] Total swap = 8388604kB
[59622.510564] 4095866 pages RAM
[59622.512379] 0 pages HighMem/MovableOnly
[59622.513853] 173881 pages reserved
[59622.515220] 0 pages cma reserved
[59622.516581] 0 pages hwpoisoned
[59622.517897] Tasks state (memory values in pages):
[59622.519524] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[59622.521895] [ 648] 0 648 71491 45126 552960 0 -250 systemd-journal
[59622.524550] [ 666] 0 666 11976 3110 102400 0 -1000 systemd-udevd
[59622.526944] [ 797] 193 797 10732 5087 122880 0 0 systemd-resolve
[59622.529352] [ 798] 0 798 26617 605 69632 0 -1000 auditd
[59622.531635] [ 800] 0 800 2077 1056 49152 0 0 sedispatch
[59622.533963] [ 821] 0 821 19874 761 57344 0 0 irqbalance
[59622.536297] [ 823] 0 823 2921 533 45056 0 0 mcelog
[59622.538590] [ 824] 996 824 666787 5873 217088 0 0 polkitd
[59622.540882] [ 825] 0 825 20273 951 53248 0 0 qemu-ga
[59622.543158] [ 826] 0 826 205022 10950 1179648 0 0 rsyslogd
[59622.545449] [ 827] 0 827 12186 2965 102400 0 0 sssd
[59622.547678] [ 828] 0 828 4502 2175 73728 0 0 systemd-homed
[59622.550047] [ 829] 0 829 4466 2118 69632 0 0 systemd-machine
[59622.552466] [ 830] 81 830 2530 1083 69632 0 -900 dbus-broker-lau
[59622.554852] [ 835] 0 835 68167 3994 139264 0 0 abrtd
[59622.557060] [ 839] 982 839 23747 967 69632 0 0 chronyd
[59622.559289] [ 840] 81 840 1525 919 49152 0 -900 dbus-broker
[59622.561576] [ 843] 0 843 12718 3366 110592 0 0 sssd_be
[59622.563803] [ 847] 0 847 177006 5860 655360 0 0 abrt-dump-journ
[59622.566207] [ 849] 0 849 174956 6978 827392 0 0 abrt-dump-journ
[59622.568518] [ 850] 0 850 15570 9663 143360 0 0 sssd_nss
[59622.570712] [ 857] 0 857 4966 2724 81920 0 0 systemd-logind
[59622.572980] [ 860] 0 860 79033 2793 118784 0 0 ModemManager
[59622.575247] [ 861] 0 861 34022 10210 155648 0 0 firewalld
[59622.577428] [ 867] 0 867 66866 5087 147456 0 0 NetworkManager
[59622.579690] [ 881] 0 881 7636 2072 73728 0 -1000 sshd
[59622.581790] [ 887] 0 887 13477 801 77824 0 0 gssproxy
[59622.583916] [ 941] 0 941 5246 715 61440 0 0 atd
[59622.585957] [ 943] 0 943 4474 892 65536 0 0 crond
[59622.588007] [ 961] 0 961 2408 455 49152 0 0 agetty
[59622.590051] [ 962] 0 962 3050 458 57344 0 0 agetty
[59622.592069] [ 1018] 991 1018 6645 533 65536 0 0 dnsmasq
[59622.594114] [ 1060] 0 1060 11399 2828 90112 0 0 sshd
[59622.596075] [ 1459] 0 1459 4387 1924 73728 0 0 systemd-userdbd
[59622.598197] [ 1488] 1000 1488 7862 3570 90112 0 0 systemd
[59622.600204] [ 1490] 1000 1490 12058 1458 102400 0 0 (sd-pam)
[59622.602218] [ 1555] 1000 1555 11399 1367 81920 0 0 sshd
[59622.604170] [ 1561] 1000 1561 5188 1443 61440 0 0 bash
[59622.606168] [ 2078] 991 2078 6645 532 61440 0 0 dnsmasq
[59622.608154] [ 2080] 0 2080 6638 100 61440 0 0 dnsmasq
[59622.610161] [ 6747] 1000 6747 10102 2101 94208 0 0 su
[59622.612074] [ 6751] 0 6751 4504 1424 65536 0 0 bash
[59622.614022] [ 14077] 0 14077 4470 1927 73728 0 0 systemd-userwor
[59622.616135] [ 14078] 0 14078 4470 1908 77824 0 0 systemd-userwor
[59622.618252] [ 14080] 0 14080 4470 1929 69632 0 0 systemd-userwor
[59622.620358] [ 14085] 0 14085 2069 642 57344 0 0 lxc-start
[59622.622379] [ 14086] 0 14086 25345 2120 233472 520 0 systemd
[59622.624401] [ 14503] 0 14503 25205 1712 241664 235 0 systemd-journal
[59622.626610] [ 14512] 0 14512 22648 1593 221184 233 0 systemd-logind
[59622.628684] [ 14513] 81 14513 13516 801 155648 138 -900 dbus-daemon
[59622.630714] [ 14694] 0 14694 2072 1104 49152 0 0 3
[59622.632591] [ 14696] 0 14696 56226 690 90112 145 0 bash
[59622.634510] [ 14734] 0 14734 52899 856 180224 440 0 rsyslogd
[59622.636496] [ 14735] 0 14735 1637 343 61440 31 0 agetty
[59622.638468] [ 14736] 0 14736 1637 398 61440 32 0 agetty
[59622.640417] [ 14737] 0 14737 1637 383 61440 31 0 agetty
[59622.642356] [ 14741] 0 14741 5724 572 94208 239 0 crond
[59622.644286] [ 14743] 0 14743 1637 398 61440 31 0 agetty
[59622.646226] [ 14744] 0 14744 1637 387 57344 31 0 agetty
[59622.648164] [ 14746] 0 14746 262198 0 266240 2900 0 syz-execprog
[59622.650200] [ 14764] 0 14764 12552 0 36864 16 0 syz-executor
[59622.652255] [ 14765] 0 14765 12552 0 36864 16 0 syz-executor
[59622.654283] [ 14766] 0 14766 12551 0 49152 44 0 syz-executor
[59622.656306] [ 14767] 0 14767 12551 0 49152 41 0 syz-executor
[59622.658318] [ 14769] 0 14769 12552 0 36864 16 0 syz-executor
[59622.660328] [ 14772] 0 14772 12552 0 36864 17 0 syz-executor
[59622.662330] [ 14774] 0 14774 12551 0 49152 30 0 syz-executor
[59622.664337] [ 14775] 0 14775 12552 0 36864 17 0 syz-executor
[59622.666456] [ 14776] 0 14776 12551 0 49152 34 0 syz-executor
[59622.668449] [ 14777] 0 14777 12552 0 36864 16 0 syz-executor
[59622.670459] [ 14779] 0 14779 12551 0 49152 37 0 syz-executor
[59622.672445] [ 14781] 0 14781 12552 0 36864 16 0 syz-executor
[59622.674449] [ 14782] 0 14782 12551 0 49152 35 0 syz-executor
[59622.676423] [ 14783] 0 14783 12552 0 36864 17 0 syz-executor
[59622.678418] [ 14784] 0 14784 12551 0 49152 36 0 syz-executor
[59622.680430] [ 14787] 0 14787 12552 0 36864 16 0 syz-executor
[59622.682398] [ 14788] 0 14788 12551 0 49152 24 0 syz-executor
[59622.684359] [ 14800] 0 14800 12552 0 36864 16 0 syz-executor
[59622.686357] [ 14801] 0 14801 12551 0 49152 23 0 syz-executor
[59622.688449] [ 14803] 0 14803 12552 0 36864 16 0 syz-executor
[59622.690434] [ 14804] 0 14804 12551 0 49152 38 0 syz-executor
[59622.692466] [ 14805] 0 14805 12552 0 36864 16 0 syz-executor
[59622.694436] [ 14807] 0 14807 12551 0 49152 25 0 syz-executor
[59622.696378] [ 14812] 0 14812 12552 0 36864 17 0 syz-executor
[59622.698313] [ 14813] 0 14813 12551 0 49152 33 0 syz-executor
[59622.700252] [ 14814] 0 14814 12552 0 36864 16 0 syz-executor
[59622.702186] [ 14816] 0 14816 12551 0 49152 27 0 syz-executor
[59622.704126] [ 14817] 0 14817 12552 0 36864 17 0 syz-executor
[59622.706053] [ 14819] 0 14819 12551 0 49152 32 0 syz-executor
[59622.707978] [ 14820] 0 14820 12552 0 36864 16 0 syz-executor
[59622.709923] [ 14821] 0 14821 12551 0 49152 31 0 syz-executor
[59622.711847] [ 14831] 0 14831 12551 0 49152 42 0 syz-executor
[59622.713773] [ 15261] 0 15260 12618 0 65536 31 1000 syz-executor
[59622.715691] [ 15263] 0 15262 12618 0 69632 0 1000 syz-executor
[59622.717604] [ 15280] 0 15279 12584 0 69632 0 1000 syz-executor
[59622.719506] [ 15282] 0 15281 12584 0 69632 0 1000 syz-executor
[59622.721401] [ 15284] 0 15283 12584 0 65536 0 1000 syz-executor
[59622.723316] [ 15286] 0 15285 12584 0 65536 0 1000 syz-executor
[59622.725243] [ 15288] 0 15287 12617 0 65536 0 1000 syz-executor
[59622.727121] [ 15291] 0 15289 12584 0 69632 4 1000 syz-executor
[59622.728993] [ 15293] 0 15292 12584 0 65536 0 1000 syz-executor
[59622.730857] [ 15295] 0 15294 12584 0 65536 0 1000 syz-executor
[59622.732722] [ 15297] 0 15296 12584 0 65536 0 1000 syz-executor
[59622.734587] [ 15299] 0 15298 12617 0 65536 0 1000 syz-executor
[59622.736441] [ 15301] 0 15300 12584 0 69632 0 1000 syz-executor
[59622.738319] [ 15304] 0 15303 12584 0 65536 0 1000 syz-executor
[59622.740174] [ 15306] 0 15305 12584 0 69632 0 1000 syz-executor
[59622.742022] [ 15307] 0 15307 12584 0 69632 7 1000 syz-executor
[59622.743869] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
[59622.745603] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.747339] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.748817] Call Trace:
[59622.749557] dump_stack_lvl+0x57/0x72
[59622.750520] panic+0xff/0x2ea
[59622.751390] out_of_memory.cold+0x2f/0x7e
[59622.752413] pagefault_out_of_memory+0x46/0x60
[59622.753522] exc_page_fault+0x79/0x2b0
[59622.754504] ? asm_exc_page_fault+0x8/0x30
[59622.755549] asm_exc_page_fault+0x1e/0x30
[59622.756579] RIP: 0033:0x41a948
[59622.757441] Code: Unable to access opcode bytes at RIP 0x41a91e.
[59622.758831] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.760086] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.761687] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.763287] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.764888] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.766512] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
WARNING: multiple messages have this Message-ID (diff)
From: Vasily Averin <vvs-5HdwGun5lf+gSpxsJD1C4w@public.gmane.org>
To: Michal Hocko <mhocko-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>,
Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org>,
Vladimir Davydov
<vdavydov.dev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Andrew Morton
<akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
Cc: Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org>,
Uladzislau Rezki <urezki-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Vlastimil Babka <vbabka-AlSwsSmVLrQ@public.gmane.org>,
Shakeel Butt <shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
Mel Gorman
<mgorman-3eNAlZScCAx27rWaFMvyedHuzzzSOjJt@public.gmane.org>,
cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
kernel-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org
Subject: [PATCH memcg 0/1] false global OOM triggered by memcg-limited task
Date: Mon, 18 Oct 2021 11:13:52 +0300 [thread overview]
Message-ID: <9d10df01-0127-fb40-81c3-cc53c9733c3e@virtuozzo.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]
While checking the patches fixed broken memcg accounting in vmalloc I found
another issue: a false global OOM triggered by memcg-limited user space task.
I executed vmalloc-eater inside a memcg limited LXC container in a loop, checked
that it does not consume host memory beyond the assigned limit, triggers memcg OOM
and generates "Memory cgroup out of memory" messages. Everything was as expected.
However I was surprised to find quite rare global OOM messages too.
I set sysctl vm.panic_on_oom to 1, repeated the test and successfully
crashed the node.
Dmesg showed that global OOM was detected on 16 GB node with ~10 GB of free memory.
syz-executor invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=1000
CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
Call Trace:
dump_stack_lvl+0x57/0x72
dump_header+0x4a/0x2c1
out_of_memory.cold+0xa/0x7e
pagefault_out_of_memory+0x46/0x60
exc_page_fault+0x79/0x2b0
asm_exc_page_fault+0x1e/0x30
...
Mem-Info:
Node 0 DMA: 0*4kB 0*8kB <...> = 13296kB
Node 0 DMA32: 705*4kB (UM) <...> = 2586964kB
Node 0 Normal: 2743*4kB (UME) <...> = 6904828kB
...
4095866 pages RAM
...
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Full dmesg can be found in attached file.
How could this happen?
User-space task inside the memcg-limited container generated a page fault,
its handler do_user_addr_fault() called handle_mm_fault which could not
allocate the page due to exceeding the memcg limit and returned VM_FAULT_OOM.
Then do_user_addr_fault() called pagefault_out_of_memory() which executed
out_of_memory() without set of memcg.
Partially this problem depends on one of my recent patches, disabled unlimited
memory allocation for dying tasks. However I think the problem can happen
on non-killed tasks too, for example because of kmem limit.
At present do_user_addr_fault() does not know why page allocation was failed,
i.e. was it global or memcg OOM. I propose to save this information in new flag
on task_struct. It can be set in case of memcg restrictons in
obj_cgroup_charge_pages() (for memory controller) and in try_charge_memcg()
(for kmem controller). Then it can be used in mem_cgroup_oom_synchronize()
called inside pagefault_out_of_memory():
in case of memcg-related restrictions it will not trigger fake global OOM and
returns to user space which will retry the fault or kill the process if it got
a fatal signal.
Thank you,
Vasily Averin
Vasily Averin (1):
memcg: prevent false global OOM trigggerd by memcg limited task.
include/linux/sched.h | 1 +
mm/memcontrol.c | 12 +++++++++---
2 files changed, 10 insertions(+), 3 deletions(-)
--
2.32.0
[-- Attachment #2: dmesg-oom.txt --]
[-- Type: text/plain, Size: 25650 bytes --]
[59622.176098] syz-executor invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=1000
[59622.178633] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.180840] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.182562] Call Trace:
[59622.183525] dump_stack_lvl+0x57/0x72
[59622.184782] dump_header+0x4a/0x2c1
[59622.185929] oom_kill_process.cold+0xb/0x10
[59622.187203] out_of_memory+0x229/0x5b0
[59622.188399] mem_cgroup_out_of_memory+0x111/0x130
[59622.189773] try_charge_memcg+0x693/0x720
[59622.191013] ? kvm_sched_clock_read+0x14/0x30
[59622.192318] charge_memcg+0x57/0x170
[59622.193482] mem_cgroup_swapin_charge_page+0x99/0x1d0
[59622.194932] do_swap_page+0x916/0xbf0
[59622.196110] ? __lock_acquire+0x3b3/0x1e00
[59622.197377] __handle_mm_fault+0xa5f/0x14e0
[59622.198649] ? lock_acquire+0xc4/0x2e0
[59622.199847] handle_mm_fault+0x149/0x3f0
[59622.201071] do_user_addr_fault+0x1f4/0x6c0
[59622.202346] exc_page_fault+0x79/0x2b0
[59622.203545] ? asm_exc_page_fault+0x8/0x30
[59622.204802] asm_exc_page_fault+0x1e/0x30
[59622.206043] RIP: 0033:0x41a948
[59622.207115] Code: 64 83 0c 25 08 03 00 00 10 64 48 8b 3c 25 00 03 00 00 e8 8b fe ff ff f4 66 2e 0f 1f 84 00 00 00 00 00 f7 c7 02 00 00 00 75 27 <64> 8b 04 25 08 03 00 00 41 89 c3 41 83 e3 fd f0 64 44 0f b1 1c 25
[59622.211557] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.213058] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.214919] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.216772] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.218623] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.220467] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
[59622.222367] memory: usage 524268kB, limit 524288kB, failcnt 28608
[59622.224112] memory+swap: usage 554776kB, limit 9007199254740988kB, failcnt 0
[59622.225982] kmem: usage 524168kB, limit 9007199254740988kB, failcnt 0
[59622.227740] Memory cgroup stats for /lxc.payload.test:
[59622.234613] anon 0
file 122880
kernel_stack 1294336
pagetables 3665920
percpu 490896
sock 0
shmem 0
file_mapped 12288
file_dirty 0
file_writeback 0
swapcached 49008640
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 0
active_anon 0
inactive_file 110592
active_file 0
unevictable 0
slab_reclaimable 1513744
slab_unreclaimable 14117008
slab 15630752
workingset_refault_anon 3800
workingset_refault_file 24974
workingset_activate_anon 923
workingset_activate_file 414
workingset_restore_anon 422
workingset_restore_file 190
workingset_nodereclaim 74
pgfault 120040
pgmajfault 4766
pgrefill 15999
pgscan 486263
pgsteal 44646
pgactivate 13066
pgdeactivate 14277
pglazyfree 60
pglazyfreed 34
thp_fault_alloc 0
thp_collapse_alloc 0
[59622.265206] Tasks state (memory values in pages):
[59622.266454] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[59622.268394] [ 14696] 0 14696 56226 690 90112 145 0 bash
[59622.270246] [ 14746] 0 14746 262198 0 266240 2900 0 syz-execprog
[59622.272217] [ 14764] 0 14764 12552 0 36864 16 0 syz-executor
[59622.274217] [ 14765] 0 14765 12552 0 36864 16 0 syz-executor
[59622.276165] [ 14769] 0 14769 12552 0 36864 16 0 syz-executor
[59622.278091] [ 14772] 0 14772 12552 0 36864 17 0 syz-executor
[59622.280023] [ 14775] 0 14775 12552 0 36864 17 0 syz-executor
[59622.281949] [ 14777] 0 14777 12552 0 36864 16 0 syz-executor
[59622.283847] [ 14781] 0 14781 12552 0 36864 16 0 syz-executor
[59622.285737] [ 14783] 0 14783 12552 0 36864 17 0 syz-executor
[59622.287673] [ 14787] 0 14787 12552 0 36864 16 0 syz-executor
[59622.287968] systemd-journald[648]: Compressed data object 720 -> 393 using ZSTD
[59622.289564] [ 14800] 0 14800 12552 0 36864 16 0 syz-executor
[59622.293144] [ 14803] 0 14803 12552 0 36864 16 0 syz-executor
[59622.295054] [ 14805] 0 14805 12552 0 36864 16 0 syz-executor
[59622.296928] [ 14812] 0 14812 12552 0 36864 17 0 syz-executor
[59622.298796] [ 14814] 0 14814 12552 0 36864 16 0 syz-executor
[59622.300663] [ 14817] 0 14817 12552 0 36864 17 0 syz-executor
[59622.302528] [ 14820] 0 14820 12552 0 36864 16 0 syz-executor
[59622.304399] [ 14779] 0 14779 12551 0 49152 37 0 syz-executor
[59622.306271] [ 15280] 0 15279 12584 0 69632 0 1000 syz-executor
[59622.308144] [ 14767] 0 14767 12551 0 49152 41 0 syz-executor
[59622.310046] [ 15282] 0 15281 12584 0 69632 0 1000 syz-executor
[59622.311926] [ 14766] 0 14766 12551 0 49152 44 0 syz-executor
[59622.313812] [ 15286] 0 15285 12584 0 65536 0 1000 syz-executor
[59622.315704] [ 14782] 0 14782 12551 0 49152 35 0 syz-executor
[59622.317621] [ 15284] 0 15283 12584 0 65536 0 1000 syz-executor
[59622.319570] [ 14784] 0 14784 12551 0 49152 36 0 syz-executor
[59622.321496] [ 15288] 0 15287 12617 0 65536 0 1000 syz-executor
[59622.323454] [ 14776] 0 14776 12551 0 49152 34 0 syz-executor
[59622.325374] [ 15291] 0 15289 12584 0 69632 4 1000 syz-executor
[59622.327293] [ 14813] 0 14813 12551 0 49152 33 0 syz-executor
[59622.329216] [ 15295] 0 15294 12584 0 65536 0 1000 syz-executor
[59622.331139] [ 14819] 0 14819 12551 0 49152 32 0 syz-executor
[59622.333068] [ 15293] 0 15292 12584 0 65536 0 1000 syz-executor
[59622.334991] [ 14821] 0 14821 12551 0 49152 31 0 syz-executor
[59622.336921] [ 15297] 0 15296 12584 0 65536 0 1000 syz-executor
[59622.338875] [ 14831] 0 14831 12551 0 49152 42 0 syz-executor
[59622.340811] [ 15299] 0 15298 12617 0 65536 0 1000 syz-executor
[59622.342752] [ 14804] 0 14804 12551 0 49152 38 0 syz-executor
[59622.344700] [ 15301] 0 15300 12584 0 69632 0 1000 syz-executor
[59622.346649] [ 14816] 0 14816 12551 0 49152 27 0 syz-executor
[59622.348604] [ 15306] 0 15305 12584 0 69632 0 1000 syz-executor
[59622.350615] [ 14774] 0 14774 12551 0 49152 30 0 syz-executor
[59622.352606] [ 15304] 0 15303 12584 0 65536 0 1000 syz-executor
[59622.354575] [ 14807] 0 14807 12551 0 49152 25 0 syz-executor
[59622.356531] [ 15307] 0 15307 12584 0 69632 37 1000 syz-executor
[59622.358481] [ 14801] 0 14801 12551 0 49152 23 0 syz-executor
[59622.360418] [ 15261] 0 15260 12618 0 65536 31 1000 syz-executor
[59622.362355] [ 14788] 0 14788 12551 0 49152 24 0 syz-executor
[59622.364291] [ 15263] 0 15262 12618 0 69632 0 1000 syz-executor
[59622.366269] [ 14503] 0 14503 25205 1712 241664 235 0 systemd-journal
[59622.368288] [ 14736] 0 14736 1637 398 61440 32 0 agetty
[59622.370161] [ 14737] 0 14737 1637 383 61440 31 0 agetty
[59622.372037] [ 14743] 0 14743 1637 398 61440 31 0 agetty
[59622.373939] [ 14744] 0 14744 1637 387 57344 31 0 agetty
[59622.375849] [ 14512] 0 14512 22648 1593 221184 233 0 systemd-logind
[59622.377858] [ 14513] 81 14513 13516 801 155648 138 -900 dbus-daemon
[59622.379867] [ 14734] 0 14734 52899 856 180224 440 0 rsyslogd
[59622.381813] [ 14735] 0 14735 1637 343 61440 31 0 agetty
[59622.383701] [ 14741] 0 14741 5724 572 94208 239 0 crond
[59622.385568] [ 14086] 0 14086 25345 2120 233472 520 0 systemd
[59622.387470] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.test,mems_allowed=0,oom_memcg=/lxc.payload.test,task_memcg=/lxc.payload.test,task=syz-executor,pid=15307,uid=0
[59622.391402] Memory cgroup out of memory: Killed process 15307 (syz-executor) total-vm:50336kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:68kB oom_score_adj:1000
[59622.395544] oom_reaper: reaped process 15307 (syz-executor), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[59622.411181] syz-executor invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=1000
[59622.414272] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.416247] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.417950] Call Trace:
[59622.418901] dump_stack_lvl+0x57/0x72
[59622.420067] dump_header+0x4a/0x2c1
[59622.421200] out_of_memory.cold+0xa/0x7e
[59622.422415] pagefault_out_of_memory+0x46/0x60
[59622.423784] exc_page_fault+0x79/0x2b0
[59622.424966] ? asm_exc_page_fault+0x8/0x30
[59622.426207] asm_exc_page_fault+0x1e/0x30
[59622.427426] RIP: 0033:0x41a948
[59622.428485] Code: Unable to access opcode bytes at RIP 0x41a91e.
[59622.430064] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.431513] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.433299] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.435090] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.436875] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.438691] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
[59622.440547] Mem-Info:
[59622.441516] active_anon:279 inactive_anon:24996 isolated_anon:0
active_file:395743 inactive_file:848017 isolated_file:0
unevictable:0 dirty:159 writeback:21
slab_reclaimable:40993 slab_unreclaimable:45236
mapped:66928 shmem:286 pagetables:2477 bounce:0
kernel_misc_reclaimable:0
free:2376272 free_pcp:7579 free_cma:0
[59622.451730] Node 0 active_anon:1116kB inactive_anon:99984kB active_file:1582972kB inactive_file:3392068kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:267712kB dirty:832kB writeback:84kB shmem:1144kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:5488kB pagetables:9908kB all_unreclaimable? no
[59622.458826] Node 0 DMA free:13296kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[59622.465131] lowmem_reserve[]: 0 2706 15265 15265 15265
[59622.466746] Node 0 DMA32 free:2586964kB min:11968kB low:14960kB high:17952kB reserved_highatomic:0KB active_anon:0kB inactive_anon:20kB active_file:27084kB inactive_file:172344kB unevictable:0kB writepending:0kB present:3129200kB managed:2801520kB mlocked:0kB bounce:0kB free_pcp:6120kB local_pcp:1004kB free_cma:0kB
[59622.473718] lowmem_reserve[]: 0 0 12559 12559 12559
[59622.475311] Node 0 Normal free:6904828kB min:55544kB low:69428kB high:83312kB reserved_highatomic:0KB active_anon:1116kB inactive_anon:99964kB active_file:1555888kB inactive_file:3219724kB unevictable:0kB writepending:916kB present:13238272kB managed:12871060kB mlocked:0kB bounce:0kB free_pcp:24068kB local_pcp:692kB free_cma:0kB
[59622.482647] lowmem_reserve[]: 0 0 0 0 0
[59622.484096] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 2*2048kB (UM) 2*4096kB (M) = 13296kB
[59622.488044] Node 0 DMA32: 705*4kB (UM) 624*8kB (UME) 497*16kB (UME) 594*32kB (UME) 532*64kB (UME) 49*128kB (UME) 36*256kB (UE) 36*512kB (UE) 20*1024kB (U) 17*2048kB (UE) 593*4096kB (U) = 2586964kB
[59622.492687] Node 0 Normal: 2743*4kB (UME) 2084*8kB (UME) 536*16kB (UME) 1496*32kB (UME) 1148*64kB (ME) 171*128kB (UME) 115*256kB (M) 94*512kB (M) 98*1024kB (UME) 163*2048kB (UME) 1517*4096kB (UM) = 6904828kB
[59622.497553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[59622.499898] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[59622.502197] 1244068 total pagecache pages
[59622.503688] 4 pages in swap cache
[59622.505046] Swap cache stats: add 54921, delete 54922, find 137/9740
[59622.506956] Free swap = 8357116kB
[59622.508352] Total swap = 8388604kB
[59622.510564] 4095866 pages RAM
[59622.512379] 0 pages HighMem/MovableOnly
[59622.513853] 173881 pages reserved
[59622.515220] 0 pages cma reserved
[59622.516581] 0 pages hwpoisoned
[59622.517897] Tasks state (memory values in pages):
[59622.519524] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[59622.521895] [ 648] 0 648 71491 45126 552960 0 -250 systemd-journal
[59622.524550] [ 666] 0 666 11976 3110 102400 0 -1000 systemd-udevd
[59622.526944] [ 797] 193 797 10732 5087 122880 0 0 systemd-resolve
[59622.529352] [ 798] 0 798 26617 605 69632 0 -1000 auditd
[59622.531635] [ 800] 0 800 2077 1056 49152 0 0 sedispatch
[59622.533963] [ 821] 0 821 19874 761 57344 0 0 irqbalance
[59622.536297] [ 823] 0 823 2921 533 45056 0 0 mcelog
[59622.538590] [ 824] 996 824 666787 5873 217088 0 0 polkitd
[59622.540882] [ 825] 0 825 20273 951 53248 0 0 qemu-ga
[59622.543158] [ 826] 0 826 205022 10950 1179648 0 0 rsyslogd
[59622.545449] [ 827] 0 827 12186 2965 102400 0 0 sssd
[59622.547678] [ 828] 0 828 4502 2175 73728 0 0 systemd-homed
[59622.550047] [ 829] 0 829 4466 2118 69632 0 0 systemd-machine
[59622.552466] [ 830] 81 830 2530 1083 69632 0 -900 dbus-broker-lau
[59622.554852] [ 835] 0 835 68167 3994 139264 0 0 abrtd
[59622.557060] [ 839] 982 839 23747 967 69632 0 0 chronyd
[59622.559289] [ 840] 81 840 1525 919 49152 0 -900 dbus-broker
[59622.561576] [ 843] 0 843 12718 3366 110592 0 0 sssd_be
[59622.563803] [ 847] 0 847 177006 5860 655360 0 0 abrt-dump-journ
[59622.566207] [ 849] 0 849 174956 6978 827392 0 0 abrt-dump-journ
[59622.568518] [ 850] 0 850 15570 9663 143360 0 0 sssd_nss
[59622.570712] [ 857] 0 857 4966 2724 81920 0 0 systemd-logind
[59622.572980] [ 860] 0 860 79033 2793 118784 0 0 ModemManager
[59622.575247] [ 861] 0 861 34022 10210 155648 0 0 firewalld
[59622.577428] [ 867] 0 867 66866 5087 147456 0 0 NetworkManager
[59622.579690] [ 881] 0 881 7636 2072 73728 0 -1000 sshd
[59622.581790] [ 887] 0 887 13477 801 77824 0 0 gssproxy
[59622.583916] [ 941] 0 941 5246 715 61440 0 0 atd
[59622.585957] [ 943] 0 943 4474 892 65536 0 0 crond
[59622.588007] [ 961] 0 961 2408 455 49152 0 0 agetty
[59622.590051] [ 962] 0 962 3050 458 57344 0 0 agetty
[59622.592069] [ 1018] 991 1018 6645 533 65536 0 0 dnsmasq
[59622.594114] [ 1060] 0 1060 11399 2828 90112 0 0 sshd
[59622.596075] [ 1459] 0 1459 4387 1924 73728 0 0 systemd-userdbd
[59622.598197] [ 1488] 1000 1488 7862 3570 90112 0 0 systemd
[59622.600204] [ 1490] 1000 1490 12058 1458 102400 0 0 (sd-pam)
[59622.602218] [ 1555] 1000 1555 11399 1367 81920 0 0 sshd
[59622.604170] [ 1561] 1000 1561 5188 1443 61440 0 0 bash
[59622.606168] [ 2078] 991 2078 6645 532 61440 0 0 dnsmasq
[59622.608154] [ 2080] 0 2080 6638 100 61440 0 0 dnsmasq
[59622.610161] [ 6747] 1000 6747 10102 2101 94208 0 0 su
[59622.612074] [ 6751] 0 6751 4504 1424 65536 0 0 bash
[59622.614022] [ 14077] 0 14077 4470 1927 73728 0 0 systemd-userwor
[59622.616135] [ 14078] 0 14078 4470 1908 77824 0 0 systemd-userwor
[59622.618252] [ 14080] 0 14080 4470 1929 69632 0 0 systemd-userwor
[59622.620358] [ 14085] 0 14085 2069 642 57344 0 0 lxc-start
[59622.622379] [ 14086] 0 14086 25345 2120 233472 520 0 systemd
[59622.624401] [ 14503] 0 14503 25205 1712 241664 235 0 systemd-journal
[59622.626610] [ 14512] 0 14512 22648 1593 221184 233 0 systemd-logind
[59622.628684] [ 14513] 81 14513 13516 801 155648 138 -900 dbus-daemon
[59622.630714] [ 14694] 0 14694 2072 1104 49152 0 0 3
[59622.632591] [ 14696] 0 14696 56226 690 90112 145 0 bash
[59622.634510] [ 14734] 0 14734 52899 856 180224 440 0 rsyslogd
[59622.636496] [ 14735] 0 14735 1637 343 61440 31 0 agetty
[59622.638468] [ 14736] 0 14736 1637 398 61440 32 0 agetty
[59622.640417] [ 14737] 0 14737 1637 383 61440 31 0 agetty
[59622.642356] [ 14741] 0 14741 5724 572 94208 239 0 crond
[59622.644286] [ 14743] 0 14743 1637 398 61440 31 0 agetty
[59622.646226] [ 14744] 0 14744 1637 387 57344 31 0 agetty
[59622.648164] [ 14746] 0 14746 262198 0 266240 2900 0 syz-execprog
[59622.650200] [ 14764] 0 14764 12552 0 36864 16 0 syz-executor
[59622.652255] [ 14765] 0 14765 12552 0 36864 16 0 syz-executor
[59622.654283] [ 14766] 0 14766 12551 0 49152 44 0 syz-executor
[59622.656306] [ 14767] 0 14767 12551 0 49152 41 0 syz-executor
[59622.658318] [ 14769] 0 14769 12552 0 36864 16 0 syz-executor
[59622.660328] [ 14772] 0 14772 12552 0 36864 17 0 syz-executor
[59622.662330] [ 14774] 0 14774 12551 0 49152 30 0 syz-executor
[59622.664337] [ 14775] 0 14775 12552 0 36864 17 0 syz-executor
[59622.666456] [ 14776] 0 14776 12551 0 49152 34 0 syz-executor
[59622.668449] [ 14777] 0 14777 12552 0 36864 16 0 syz-executor
[59622.670459] [ 14779] 0 14779 12551 0 49152 37 0 syz-executor
[59622.672445] [ 14781] 0 14781 12552 0 36864 16 0 syz-executor
[59622.674449] [ 14782] 0 14782 12551 0 49152 35 0 syz-executor
[59622.676423] [ 14783] 0 14783 12552 0 36864 17 0 syz-executor
[59622.678418] [ 14784] 0 14784 12551 0 49152 36 0 syz-executor
[59622.680430] [ 14787] 0 14787 12552 0 36864 16 0 syz-executor
[59622.682398] [ 14788] 0 14788 12551 0 49152 24 0 syz-executor
[59622.684359] [ 14800] 0 14800 12552 0 36864 16 0 syz-executor
[59622.686357] [ 14801] 0 14801 12551 0 49152 23 0 syz-executor
[59622.688449] [ 14803] 0 14803 12552 0 36864 16 0 syz-executor
[59622.690434] [ 14804] 0 14804 12551 0 49152 38 0 syz-executor
[59622.692466] [ 14805] 0 14805 12552 0 36864 16 0 syz-executor
[59622.694436] [ 14807] 0 14807 12551 0 49152 25 0 syz-executor
[59622.696378] [ 14812] 0 14812 12552 0 36864 17 0 syz-executor
[59622.698313] [ 14813] 0 14813 12551 0 49152 33 0 syz-executor
[59622.700252] [ 14814] 0 14814 12552 0 36864 16 0 syz-executor
[59622.702186] [ 14816] 0 14816 12551 0 49152 27 0 syz-executor
[59622.704126] [ 14817] 0 14817 12552 0 36864 17 0 syz-executor
[59622.706053] [ 14819] 0 14819 12551 0 49152 32 0 syz-executor
[59622.707978] [ 14820] 0 14820 12552 0 36864 16 0 syz-executor
[59622.709923] [ 14821] 0 14821 12551 0 49152 31 0 syz-executor
[59622.711847] [ 14831] 0 14831 12551 0 49152 42 0 syz-executor
[59622.713773] [ 15261] 0 15260 12618 0 65536 31 1000 syz-executor
[59622.715691] [ 15263] 0 15262 12618 0 69632 0 1000 syz-executor
[59622.717604] [ 15280] 0 15279 12584 0 69632 0 1000 syz-executor
[59622.719506] [ 15282] 0 15281 12584 0 69632 0 1000 syz-executor
[59622.721401] [ 15284] 0 15283 12584 0 65536 0 1000 syz-executor
[59622.723316] [ 15286] 0 15285 12584 0 65536 0 1000 syz-executor
[59622.725243] [ 15288] 0 15287 12617 0 65536 0 1000 syz-executor
[59622.727121] [ 15291] 0 15289 12584 0 69632 4 1000 syz-executor
[59622.728993] [ 15293] 0 15292 12584 0 65536 0 1000 syz-executor
[59622.730857] [ 15295] 0 15294 12584 0 65536 0 1000 syz-executor
[59622.732722] [ 15297] 0 15296 12584 0 65536 0 1000 syz-executor
[59622.734587] [ 15299] 0 15298 12617 0 65536 0 1000 syz-executor
[59622.736441] [ 15301] 0 15300 12584 0 69632 0 1000 syz-executor
[59622.738319] [ 15304] 0 15303 12584 0 65536 0 1000 syz-executor
[59622.740174] [ 15306] 0 15305 12584 0 69632 0 1000 syz-executor
[59622.742022] [ 15307] 0 15307 12584 0 69632 7 1000 syz-executor
[59622.743869] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
[59622.745603] CPU: 2 PID: 15307 Comm: syz-executor Kdump: loaded Not tainted 5.15.0-rc4+ #55
[59622.747339] Hardware name: Virtuozzo KVM, BIOS 1.11.0-2.vz7.4 04/01/2014
[59622.748817] Call Trace:
[59622.749557] dump_stack_lvl+0x57/0x72
[59622.750520] panic+0xff/0x2ea
[59622.751390] out_of_memory.cold+0x2f/0x7e
[59622.752413] pagefault_out_of_memory+0x46/0x60
[59622.753522] exc_page_fault+0x79/0x2b0
[59622.754504] ? asm_exc_page_fault+0x8/0x30
[59622.755549] asm_exc_page_fault+0x1e/0x30
[59622.756579] RIP: 0033:0x41a948
[59622.757441] Code: Unable to access opcode bytes at RIP 0x41a91e.
[59622.758831] RSP: 002b:00007ffd10ebf6e8 EFLAGS: 00010246
[59622.760086] RAX: 0000000000000000 RBX: 000000000119cf40 RCX: 000000000041b331
[59622.761687] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[59622.763287] RBP: 000000000119d940 R08: 0000000000000000 R09: 0000000000000000
[59622.764888] R10: 00007ffd10ebf7c0 R11: 0000000000000293 R12: 00000000038dc0e1
[59622.766512] R13: 00000000038dbdc3 R14: 20c49ba5e353f7cf R15: ffffffffffffffff
next reply other threads:[~2021-10-18 8:14 UTC|newest]
Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-18 8:13 Vasily Averin [this message]
2021-10-18 8:13 ` [PATCH memcg 0/1] false global OOM triggered by memcg-limited task Vasily Averin
2021-10-18 9:04 ` Michal Hocko
2021-10-18 9:04 ` Michal Hocko
2021-10-18 10:05 ` Vasily Averin
2021-10-18 10:05 ` Vasily Averin
2021-10-18 10:12 ` Vasily Averin
2021-10-18 10:12 ` Vasily Averin
2021-10-18 11:53 ` Michal Hocko
2021-10-18 11:53 ` Michal Hocko
[not found] ` <27dc0c49-a0d6-875b-49c6-0ef5c0cc3ac8@virtuozzo.com>
2021-10-18 12:27 ` Michal Hocko
2021-10-18 12:27 ` Michal Hocko
2021-10-18 15:07 ` Shakeel Butt
2021-10-18 15:07 ` Shakeel Butt
2021-10-18 16:51 ` Michal Hocko
2021-10-18 16:51 ` Michal Hocko
2021-10-18 17:13 ` Shakeel Butt
2021-10-18 18:52 ` Vasily Averin
2021-10-18 18:52 ` Vasily Averin
2021-10-18 19:18 ` Vasily Averin
2021-10-18 19:18 ` Vasily Averin
2021-10-19 5:34 ` Shakeel Butt
2021-10-19 5:34 ` Shakeel Butt
2021-10-19 5:33 ` Shakeel Butt
2021-10-19 5:33 ` Shakeel Butt
2021-10-19 6:42 ` Vasily Averin
2021-10-19 6:42 ` Vasily Averin
2021-10-19 8:47 ` Michal Hocko
2021-10-19 8:47 ` Michal Hocko
2021-10-19 6:30 ` Vasily Averin
2021-10-19 6:30 ` Vasily Averin
2021-10-19 8:49 ` Michal Hocko
2021-10-19 8:49 ` Michal Hocko
2021-10-19 10:30 ` Vasily Averin
2021-10-19 10:30 ` Vasily Averin
2021-10-19 11:54 ` Michal Hocko
2021-10-19 11:54 ` Michal Hocko
2021-10-19 12:04 ` Michal Hocko
2021-10-19 12:04 ` Michal Hocko
2021-10-19 13:26 ` Vasily Averin
2021-10-19 13:26 ` Vasily Averin
2021-10-19 14:13 ` Michal Hocko
2021-10-19 14:13 ` Michal Hocko
2021-10-19 14:19 ` Michal Hocko
2021-10-19 14:19 ` Michal Hocko
2021-10-19 19:09 ` Vasily Averin
2021-10-19 19:09 ` Vasily Averin
2021-10-20 8:07 ` [PATCH memcg v4] memcg: prohibit unconditional exceeding the limit of dying tasks Vasily Averin
2021-10-20 8:07 ` Vasily Averin
2021-10-20 8:43 ` Michal Hocko
2021-10-20 8:43 ` Michal Hocko
2021-10-20 12:11 ` [PATCH memcg RFC 0/3] " Vasily Averin
2021-10-20 12:11 ` Vasily Averin
[not found] ` <cover.1634730787.git.vvs@virtuozzo.com>
2021-10-20 12:12 ` [PATCH memcg 1/3] mm: do not firce global OOM from inside " Vasily Averin
2021-10-20 12:12 ` Vasily Averin
2021-10-20 12:33 ` Michal Hocko
2021-10-20 12:33 ` Michal Hocko
2021-10-20 13:52 ` Vasily Averin
2021-10-20 13:52 ` Vasily Averin
2021-10-20 12:13 ` [PATCH memcg 2/3] memcg: remove charge forcinig for " Vasily Averin
2021-10-20 12:13 ` Vasily Averin
2021-10-20 12:41 ` Michal Hocko
2021-10-20 12:41 ` Michal Hocko
2021-10-20 14:21 ` Vasily Averin
2021-10-20 14:21 ` Vasily Averin
2021-10-20 14:57 ` Michal Hocko
2021-10-20 14:57 ` Michal Hocko
2021-10-20 15:20 ` Tetsuo Handa
2021-10-20 15:20 ` Tetsuo Handa
2021-10-21 10:03 ` Michal Hocko
2021-10-21 10:03 ` Michal Hocko
2021-10-20 12:14 ` [PATCH memcg 3/3] memcg: handle memcg oom failures Vasily Averin
2021-10-20 12:14 ` Vasily Averin
2021-10-20 13:02 ` Michal Hocko
2021-10-20 15:46 ` Vasily Averin
2021-10-20 15:46 ` Vasily Averin
2021-10-21 11:49 ` Michal Hocko
2021-10-21 11:49 ` Michal Hocko
2021-10-21 15:05 ` Vasily Averin
2021-10-21 15:05 ` Vasily Averin
2021-10-21 16:47 ` Michal Hocko
2021-10-21 16:47 ` Michal Hocko
2021-10-22 8:10 ` [PATCH memcg v2 0/2] memcg: prohibit unconditional exceeding the limit of dying tasks Vasily Averin
2021-10-22 8:10 ` Vasily Averin
[not found] ` <cover.1634889066.git.vvs@virtuozzo.com>
2021-10-22 8:11 ` [PATCH memcg v2 1/2] mm, oom: do not trigger out_of_memory from the #PF Vasily Averin
2021-10-22 8:11 ` Vasily Averin
2021-10-22 8:55 ` Michal Hocko
2021-10-22 8:55 ` Michal Hocko
2021-10-22 8:11 ` [PATCH memcg v2 2/2] memcg: prohibit unconditional exceeding the limit of dying tasks Vasily Averin
2021-10-22 8:11 ` Vasily Averin
2021-10-22 9:10 ` Michal Hocko
2021-10-22 9:10 ` Michal Hocko
2021-10-23 13:18 ` [PATCH memcg v3 0/3] " Vasily Averin
2021-10-23 13:18 ` Vasily Averin
[not found] ` <cover.1634994605.git.vvs@virtuozzo.com>
2021-10-23 13:19 ` [PATCH memcg v3 1/3] mm, oom: pagefault_out_of_memory: don't force global OOM for " Vasily Averin
2021-10-23 13:19 ` Vasily Averin
2021-10-25 9:27 ` Michal Hocko
2021-10-25 9:27 ` Michal Hocko
2021-10-23 13:20 ` [PATCH memcg v3 2/3] mm, oom: do not trigger out_of_memory from the #PF Vasily Averin
2021-10-23 13:20 ` Vasily Averin
2021-10-23 15:01 ` Tetsuo Handa
2021-10-23 15:01 ` Tetsuo Handa
2021-10-23 19:15 ` Vasily Averin
2021-10-25 8:04 ` Michal Hocko
2021-10-25 8:04 ` Michal Hocko
2021-10-26 13:56 ` Tetsuo Handa
2021-10-26 13:56 ` Tetsuo Handa
2021-10-26 14:07 ` Michal Hocko
2021-10-26 14:07 ` Michal Hocko
2021-10-25 9:34 ` Michal Hocko
2021-10-25 9:34 ` Michal Hocko
2021-10-23 13:20 ` [PATCH memcg v3 3/3] memcg: prohibit unconditional exceeding the limit of dying tasks Vasily Averin
2021-10-23 13:20 ` Vasily Averin
2021-10-25 9:36 ` Michal Hocko
2021-10-25 9:36 ` Michal Hocko
2021-10-27 22:36 ` Andrew Morton
2021-10-27 22:36 ` Andrew Morton
2021-10-28 7:22 ` Vasily Averin
2021-10-28 7:22 ` Vasily Averin
2021-10-29 7:46 ` Greg Kroah-Hartman
2021-10-29 7:46 ` Greg Kroah-Hartman
2021-10-29 7:58 ` Michal Hocko
2021-10-29 7:58 ` Michal Hocko
2021-11-12 23:48 ` [PATCH memcg v4] " kernel test robot
2021-11-26 4:32 ` kernel test robot
2021-10-21 8:03 ` [PATCH memcg 0/1] false global OOM triggered by memcg-limited task Vasily Averin
2021-10-21 8:03 ` Vasily Averin
2021-10-21 11:49 ` Michal Hocko
2021-10-21 11:49 ` Michal Hocko
2021-10-21 13:24 ` Vasily Averin
2021-10-21 13:24 ` Vasily Averin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9d10df01-0127-fb40-81c3-cc53c9733c3e@virtuozzo.com \
--to=vvs@virtuozzo.com \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=kernel@openvz.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=shakeelb@google.com \
--cc=urezki@gmail.com \
--cc=vbabka@suse.cz \
--cc=vdavydov.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.