From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C128C4167B for ; Wed, 29 Nov 2023 19:44:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30D5D6B03B1; Wed, 29 Nov 2023 14:44:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BDBA6B03DE; Wed, 29 Nov 2023 14:44:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 185276B03DF; Wed, 29 Nov 2023 14:44:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 04DC06B03B1 for ; Wed, 29 Nov 2023 14:44:15 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D8A341A05E0 for ; Wed, 29 Nov 2023 19:44:14 +0000 (UTC) X-FDA: 81512018028.11.2FE27F1 Received: from mail-il1-f171.google.com (mail-il1-f171.google.com [209.85.166.171]) by imf05.hostedemail.com (Postfix) with ESMTP id 2111810001D for ; Wed, 29 Nov 2023 19:44:12 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JGmoxMs4; spf=pass (imf05.hostedemail.com: domain of rientjes@google.com designates 209.85.166.171 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701287053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FSe8ni27FEniIJaXrMNBOv/xw4C4hLVgJwTgLO68hB0=; b=K3oYTi25BWUJieeRgcU3TGMHEv70Ntp7IsM3F/aZoBDgt0t07n67P8AUb+cjjrgRCLOf2L 5NCoHK0fCYTjsMmuZKs9lqdCdYLKwrPQE81YmXNO0hb8qm8BcB3s5M45xCao+dkBDrpSRn BJJfVBNqnTWCezpIyv8TXqDIcvafHPQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701287053; a=rsa-sha256; cv=none; b=MmwMnj8gOKMxORu24HSJd5SxtNpKRSVE7uRXW6MDUhDFsbDiQg0TT5QPUOXHGuR0pWBu8I Ird8542GlIhe13JkCh+QIqVkDvG3ryKu0ngBviFosXB8lRT9HeKdbn1ZA/6zj04GiP0b5k F9lgoRnmHPQIBgBFAMkeLh9yw6SbAWs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JGmoxMs4; spf=pass (imf05.hostedemail.com: domain of rientjes@google.com designates 209.85.166.171 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-il1-f171.google.com with SMTP id e9e14a558f8ab-35c767a9e76so15885ab.1 for ; Wed, 29 Nov 2023 11:44:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701287052; x=1701891852; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=FSe8ni27FEniIJaXrMNBOv/xw4C4hLVgJwTgLO68hB0=; b=JGmoxMs4TuIEK1B9JK1xLZFbC8bfreKD274I/ydN4Pz2V+kYIgAIH/bOEc4N+Bmwiu cHdzTupZNHSIsI6+2+vFYCqCP6ilzmBmeGS24CAaZItOeDN/Rz7HQ47Xog/cELRNxCkc GGQiZWeyujy8HvmBaNhjrVNKmjM2mazW5nKJBOLoFCJ6IcKWLVS1BWAZTiTHdM2Hpo4t c7h1NvtqkMbEqXvXplDVnfjBQh1DlGNHbWZfakHp95V6/wpy00SF0YBWD5PT4unmhXug Plrmhb/b+o2ZTgG+mHNq5G5N+NCcETSBhMrzoMKpE7sp+QNV43OMpiQFo9mOOS/M1/P5 n3qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701287052; x=1701891852; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FSe8ni27FEniIJaXrMNBOv/xw4C4hLVgJwTgLO68hB0=; b=Gue9ExBIeU59f1DlOGdQuh7U/m63E374AdJmeDgJpv2zT9imZzS0nLoxlJaMqXY2t6 oQDVS2szBVluqOM7LQA8dRwcFH/96uuUBHaTnNG1yo0Cc6MqZ8WBObCei5NvNIth3V2G z/p3SHA6aZ74lICHgWngeG/eed/DwiHChJhUb5CZMSS9Q7L+IJkXD3Haq0m/XNiH30ry ZTmS6rsaoA923ClXJ5K8mZr9bibE4iXhYnGZ6kpQURZg+wN3fG5rS9luTrj8xnC9o1Ye p6qc0B5/Mr+uhhRJo4KcgmlWLIemMFzxhnpDNAY8nHFMEtXOS8ucAM7w6kiSTcONRNIf zDAw== X-Gm-Message-State: AOJu0Yx2Don0PsNbXrEeCT8rNloNlTXHXX0bbIwM6sxNqX82ykuTQm1d wp0z3Cwyz6mGFd3Sq8lX/e801w== X-Google-Smtp-Source: AGHT+IGl5/jI1bNC54jUcmhWn9887FTW3K6ASapjL6l72DekJfC3MvKCEEOjuQBLGm3vKZ047uU/YQ== X-Received: by 2002:a92:cd8f:0:b0:35c:abc4:f8c1 with SMTP id r15-20020a92cd8f000000b0035cabc4f8c1mr20570ilb.2.1701287052072; Wed, 29 Nov 2023 11:44:12 -0800 (PST) Received: from [2620:0:1008:15:9ddf:fd79:28e8:6290] ([2620:0:1008:15:9ddf:fd79:28e8:6290]) by smtp.gmail.com with ESMTPSA id a21-20020a63e855000000b00588e8421fa8sm11877451pgk.84.2023.11.29.11.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 11:44:11 -0800 (PST) Date: Wed, 29 Nov 2023 11:44:11 -0800 (PST) From: David Rientjes To: Joakim Tjernlund cc: "linux-mm@kvack.org" Subject: Re: Understanding an OOM In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 2111810001D X-Rspam-User: X-Stat-Signature: t7jsna8rbsegfoekaazg1s3zo9xkpqhp X-Rspamd-Server: rspam03 X-HE-Tag: 1701287052-412445 X-HE-Meta: U2FsdGVkX18JL7IVNtBx02vTTCbMYX31aWzlxkXnsuj3bBtY9AQGacATId+3mL9hxO5qMggOHDYZScvZPo19GqnUC1GVGFiXcqx4ydDCHBhMPRQJxIoujTuIaUF/3t9SSKL1km/F61Snidg8sZhHGCaiZi43m7fR2DcT6KUnVCnIqKZtT5AcyJ10BXbFZfeVvyrPihq3TqNJOLpN+A0ee5nJNvA1PfyRLmJzOKVsqAIAv5Vdze4O0GhhtiY5Ay6aoEQpx/C5lBWXXQNyjPQF1O/O6qUqKAowZb73bwKM9BQ5oyWbXzLzE/1T7a2eqoEoxLTwc4/ZbjYkP94mibJH4WpyDSwO1Bd5wio9VD2sjG0osrBvMK6S8eX5bZkvNBg7vQuB0VmQ5uuF/lbPR5eigmk171OeTCZ2feBqjywU00uK2BGmsE8QQ7zrdulReX4xQuRSSdzbD2bb9IVcnUUnoaAE6FuVYUaOe13SVihVcsF+n6fgUCxNTnNnF4BG2mkiXMAmEEQ4NUAb7DBf/3UXw6EPH8ud9H6IffqG26d3gUKyJHwFJZ7PPeOzGko9SBGZD+OI96eHhV4fzbHjeRQPM77WNxWrLRcDri77pGu80igDgUBTaWjDp0sOFCsT1l86XGQqhlrxxWTcovdCwBaNBDxV/YEuXWVxBk+NxJdroFLVLYcqgx+ncCKT0JzIYT7ffNNaIZk+6n3FsD0pSy0NFtgxbwZ1+mw3WUO7MR16PbG/Gm7oPZzf1rqxAkjqfhnKFi+W5eqN30zXWNItunM/H/nSgYzGCMefgi9L8xjY8hSkq0fIv2HrqPvXyyKsebeV9wdFMwvM9osc2ERHKKlmyo/tTx6VMlMbdXtnQods2YaAvLdcs90Nnsgnok71wC3i8cuu80aw5bZIqTkrGeprPgW6Nhql3fBmDZIs8BayWnZcptFNnQDbVPtbskqM9hKuRdS7mT2T4agE2bfJLaN Oj07JABi /D2AIAPhuCNnRqkpEc5q2PoQLkaVQ+f4opCB0H7ai0d8sh83x2RMe2nN9mepse5kBPnGAf+VHxIvqDip5B9HLm35f/jN1ER80dFXDUjYekRSHe1dZZ/e0mEoJwAeQSTtCXWWluQzIlwQ6eJqKlL5qme77IpTKCsk1QMu48FPZKqopeaLB1AIsxuZjo+mSOCfXEVx7v8gm0KaB2g52Ej7gNXSu3g0VPABDaKn+fgpvX7zF52sCA0m910GRcdy8fYW7ASGgEPn/AvpaEn4ROTlk+CpGafECSNWbUAjTXYfjK5N67tqezE8GO/9VZDS9vr1ivsYasIHyWIIQGsPTAsAns6CAYOkU7RNlP8tM27phJQbmxHnyVopFnHoxQwUheprpDKGhTcRTjqfxfex1uCCDBgWUeGEwOsdxtEaqTJW7tz9NwGvrxvpI8nUZ84gL8VaxH1IIOqJTnwir8IZgzK5J7pZxh9X67t1IyFqs2h6BxF5cbW8XvwNnPRd/ow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000568, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 28 Nov 2023, Joakim Tjernlund wrote: > I got this OOM I am trying to make sense of: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT) > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB > isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB > active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294 The reserved_highatomic looks strange because "free" - "reserved_highatomic" brings us very close to "min". So we'd have to look into whether reserved_highatomic is getting freed up appropriately on such small systems. But I think the first order of business would be to determine if you can reproduce this on a more recent upstream kernel. v5.15 is over two years old now :/ > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 4220kB > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved > Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages): > Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name > Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev > Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear > Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog > Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker > Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main > Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset > Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi > Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate > Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh > Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear > Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh > Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install- > Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip > Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0 > Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0 > pgtables:604kB oom_score_adj:0 > > There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB > Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ? > > This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP. > Any clues? > > Jocke > >