* Understanding an OOM
@ 2023-11-28 9:46 Joakim Tjernlund
2023-11-29 19:44 ` David Rientjes
0 siblings, 1 reply; 3+ messages in thread
From: Joakim Tjernlund @ 2023-11-28 9:46 UTC (permalink / raw)
To: linux-mm
I got this OOM I am trying to make sense of:
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT)
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace:
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info:
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB
isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB
active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
4220kB
Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages
Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM
Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly
Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved
Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages):
Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev
Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd
Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd
Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear
Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd
Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog
Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker
Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main
Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent
Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset
Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent
Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent
Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent
Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent
Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi
Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate
Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh
Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear
Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh
Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install-
Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip
Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0
Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0
pgtables:604kB oom_score_adj:0
There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB
Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ?
This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP.
Any clues?
Jocke
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Understanding an OOM
2023-11-28 9:46 Understanding an OOM Joakim Tjernlund
@ 2023-11-29 19:44 ` David Rientjes
2023-11-29 20:41 ` Joakim Tjernlund
0 siblings, 1 reply; 3+ messages in thread
From: David Rientjes @ 2023-11-29 19:44 UTC (permalink / raw)
To: Joakim Tjernlund; +Cc: linux-mm
On Tue, 28 Nov 2023, Joakim Tjernlund wrote:
> I got this OOM I am trying to make sense of:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT)
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB
> isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB
> active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294
The reserved_highatomic looks strange because
"free" - "reserved_highatomic" brings us very close to "min".
So we'd have to look into whether reserved_highatomic is getting freed up
appropriately on such small systems. But I think the first order of
business would be to determine if you can reproduce this on a more recent
upstream kernel. v5.15 is over two years old now :/
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 4220kB
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages):
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install-
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0
> Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0
> pgtables:604kB oom_score_adj:0
>
> There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB
> Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ?
>
> This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP.
> Any clues?
>
> Jocke
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Understanding an OOM
2023-11-29 19:44 ` David Rientjes
@ 2023-11-29 20:41 ` Joakim Tjernlund
0 siblings, 0 replies; 3+ messages in thread
From: Joakim Tjernlund @ 2023-11-29 20:41 UTC (permalink / raw)
To: rientjes; +Cc: linux-mm
On Wed, 2023-11-29 at 11:44 -0800, David Rientjes wrote:
> On Tue, 28 Nov 2023, Joakim Tjernlund wrote:
>
> > I got this OOM I am trying to make sense of:
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT)
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace:
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info:
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB
> > isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB
> > active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294
>
> The reserved_highatomic looks strange because
> "free" - "reserved_highatomic" brings us very close to "min".
>
> So we'd have to look into whether reserved_highatomic is getting freed up
> appropriately on such small systems. But I think the first order of
Thanks for looking
Do I need reserved_highatomic at all on a small system? What would happen if I just forced it to 0 ?
> business would be to determine if you can reproduce this on a more recent
> upstream kernel. v5.15 is over two years old now :/
5.15.129 isn't that old, Aug. this year. I cannot update to latest and greatest on this system, will be a big effort.
Jocke
>
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> > 4220kB
> > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages
> > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM
> > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly
> > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages):
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install-
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip
> > Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0
> > Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0
> > pgtables:604kB oom_score_adj:0
> >
> > There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB
> > Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ?
> >
> > This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP.
> > Any clues?
> >
> > Jocke
> >
> >
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-11-29 20:41 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-28 9:46 Understanding an OOM Joakim Tjernlund
2023-11-29 19:44 ` David Rientjes
2023-11-29 20:41 ` Joakim Tjernlund
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.