From: David Rientjes <rientjes@google.com>
To: Joakim Tjernlund <Joakim.Tjernlund@infinera.com>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>
Subject: Re: Understanding an OOM
Date: Wed, 29 Nov 2023 11:44:11 -0800 (PST) [thread overview]
Message-ID: <a3ba0086-9645-a61c-b7de-dbbca894626e@google.com> (raw)
In-Reply-To: <fee8a25cff5256492b74a03f19e0dd3ab53a07f5.camel@infinera.com>
On Tue, 28 Nov 2023, Joakim Tjernlund wrote:
> I got this OOM I am trying to make sense of:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT)
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info:
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB
> isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB
> active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294
The reserved_highatomic looks strange because
"free" - "reserved_highatomic" brings us very close to "min".
So we'd have to look into whether reserved_highatomic is getting freed up
appropriately on such small systems. But I think the first order of
business would be to determine if you can reproduce this on a more recent
upstream kernel. v5.15 is over two years old now :/
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 4220kB
> Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly
> Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages):
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install-
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip
> Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0
> Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0
> pgtables:604kB oom_score_adj:0
>
> There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB
> Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ?
>
> This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP.
> Any clues?
>
> Jocke
>
>
next prev parent reply other threads:[~2023-11-29 19:44 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-28 9:46 Understanding an OOM Joakim Tjernlund
2023-11-29 19:44 ` David Rientjes [this message]
2023-11-29 20:41 ` Joakim Tjernlund
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a3ba0086-9645-a61c-b7de-dbbca894626e@google.com \
--to=rientjes@google.com \
--cc=Joakim.Tjernlund@infinera.com \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.