* Re: [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl
[not found] <bug-209245-27@https.bugzilla.kernel.org/>
@ 2020-09-13 0:38 ` Andrew Morton
2020-09-13 6:33 ` Christoph Hellwig
0 siblings, 1 reply; 3+ messages in thread
From: Andrew Morton @ 2020-09-13 0:38 UTC (permalink / raw)
To: Christoph Hellwig, Al Viro
Cc: bugzilla-daemon, gaetan, linux-mm, Alexey Dobriyan
(switched to email. Please respond via emailed reply-to-all, not via the
bugzilla web interface).
On Sat, 12 Sep 2020 09:12:18 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=209245
>
> Bug ID: 209245
> Summary: 5.8.x Cannot allocate memory requesting sysctl
> Product: Memory Management
> Version: 2.5
> Kernel Version: 5.8.x
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: high
> Priority: P1
> Component: Page Allocator
> Assignee: akpm@linux-foundation.org
> Reporter: gaetan@nexylan.com
> Regression: No
Actually, "Regression: yes".
> Created attachment 292483
> --> https://bugzilla.kernel.org/attachment.cgi?id=292483&action=edit
> dmesg
>
> Since we upgraded a few nodes to 5.8.x (Gentoo Linux), we are having random
> "Cannot allocate memory" when requesting infos from sysctl despite system has
> free memory available.
>
> Steps to reproduce :
>
> 1/ Have system with heavy workload under 5.8.2 (reproduced with 5.8.8)
> 2/ At some point, accessing sysctl is not possible anymore :
>
>
> ~# cat /proc/sys/net/netfilter/nf_conntrack_count
> cat: /proc/sys/net/netfilter/nf_conntrack_count: Cannot allocate memory
>
>
> 3/ Dmesg drops memory allocation errors (see attached)
>
>
> Memory usage :
>
> total used free shared buff/cache available
> Mem: 32009 22029 1547 0 8431 9506
> Swap: 7811 621 7190
>
> Node is running a single mongodb process.
>
The out-of-memory dump is below.
proc_sys_call_handler() is attempting an order-5 kmalloc for a read of
a simple int. Presumably because cat(1) is starting out with a 128kb
read().
AFAICT this is due to changes made in 32927393dc1cc ("sysctl: pass
kernel pointers to ->proc_handler")?
[Sat Sep 12 11:07:55 2020] cat: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
[Sat Sep 12 11:07:55 2020] CPU: 10 PID: 32410 Comm: cat Not tainted 5.8.8-gentoo #1
[Sat Sep 12 11:07:55 2020] Hardware name: Supermicro MBI-6119M-T2N/B2SC1-CPU, BIOS 1.1 06/25/2019
[Sat Sep 12 11:07:55 2020] Call Trace:
[Sat Sep 12 11:07:55 2020] dump_stack+0x6d/0x8b
[Sat Sep 12 11:07:55 2020] warn_alloc+0xf6/0x160
[Sat Sep 12 11:07:55 2020] __alloc_pages_slowpath.constprop.123+0xb28/0xb70
[Sat Sep 12 11:07:55 2020] ? __alloc_pages_nodemask+0x144/0x2e0
[Sat Sep 12 11:07:55 2020] __alloc_pages_nodemask+0x2af/0x2e0
[Sat Sep 12 11:07:55 2020] kmalloc_order+0x13/0x70
[Sat Sep 12 11:07:55 2020] kmalloc_order_trace+0x19/0xa0
[Sat Sep 12 11:07:55 2020] proc_sys_call_handler+0xf9/0x170
[Sat Sep 12 11:07:55 2020] vfs_read+0x96/0x170
[Sat Sep 12 11:07:55 2020] ksys_read+0x9c/0xd0
[Sat Sep 12 11:07:55 2020] do_syscall_64+0x44/0x70
[Sat Sep 12 11:07:55 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Sat Sep 12 11:07:55 2020] RIP: 0033:0x7f2c7a19ac7b
[Sat Sep 12 11:07:55 2020] Code: 00 3d 00 00 41 00 74 07 31 c0 e9 e9 fe ff ff 50 48 8d 3d 60 63 09 00 e8 3b c4 01 00 64 8b 04 25 18 00 00 00 85 c0 75 0b 0f 05 <48> 3d 00 f0 ff ff 77 4c c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[Sat Sep 12 11:07:55 2020] RSP: 002b:00007ffec0af8ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Sat Sep 12 11:07:55 2020] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f2c7a19ac7b
[Sat Sep 12 11:07:55 2020] RDX: 0000000000020000 RSI: 00007f2c7a097000 RDI: 0000000000000003
[Sat Sep 12 11:07:55 2020] RBP: 00007f2c7a097000 R08: 00007f2c7a096010 R09: 0000000000000000
[Sat Sep 12 11:07:55 2020] R10: 0000559b45d2aa00 R11: 0000000000000246 R12: 00007f2c7a097000
[Sat Sep 12 11:07:55 2020] R13: 0000000000000003 R14: 0000000000000fff R15: 0000000000020000
[Sat Sep 12 11:07:55 2020] Mem-Info:
[Sat Sep 12 11:07:55 2020] active_anon:5210006 inactive_anon:373749 isolated_anon:0
active_file:972354 inactive_file:990026 isolated_file:0
unevictable:1958 dirty:70 writeback:0
slab_reclaimable:198141 slab_unreclaimable:19557
mapped:8172 shmem:29 pagetables:11756 bounce:0
free:391832 free_pcp:0 free_cma:0
[Sat Sep 12 11:07:55 2020] Node 0 active_anon:20840024kB inactive_anon:1494996kB active_file:3889416kB inactive_file:3960104kB unevictable:7832kB isolated(anon):0kB isolated(file):0kB mapped:32688kB dirty:280kB writeback:0kB shmem:116kB writeback_tmp:0kB all_unreclaimable? no
[Sat Sep 12 11:07:55 2020] Node 0 DMA free:11800kB min:8kB low:20kB high:32kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 2108 31993 31993
[Sat Sep 12 11:07:55 2020] Node 0 DMA32 free:153020kB min:1504kB low:3656kB high:5808kB reserved_highatomic:0KB active_anon:1169844kB inactive_anon:206804kB active_file:180128kB inactive_file:209060kB unevictable:0kB writepending:0kB present:2257504kB managed:2159144kB mlocked:0kB kernel_stack:16kB pagetables:1848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 29884 29884
[Sat Sep 12 11:07:55 2020] Node 0 Normal free:1402508kB min:21380kB low:51980kB high:82580kB reserved_highatomic:0KB active_anon:19670180kB inactive_anon:1288192kB active_file:3709496kB inactive_file:3751000kB unevictable:7832kB writepending:280kB present:31178752kB managed:30602236kB mlocked:7832kB kernel_stack:4496kB pagetables:45176kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 0 0
[Sat Sep 12 11:07:55 2020] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11800kB
[Sat Sep 12 11:07:55 2020] Node 0 DMA32: 2120*4kB (UME) 9907*8kB (UME) 3974*16kB (UM) 56*32kB (UM) 3*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153304kB
[Sat Sep 12 11:07:55 2020] Node 0 Normal: 8036*4kB (UME) 62654*8kB (UME) 54156*16kB (UM) 155*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1404832kB
[Sat Sep 12 11:07:55 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sat Sep 12 11:07:55 2020] 1994278 total pagecache pages
[Sat Sep 12 11:07:55 2020] 33986 pages in swap cache
[Sat Sep 12 11:07:55 2020] Swap cache stats: add 205324, delete 171339, find 1160015/1183131
[Sat Sep 12 11:07:55 2020] Free swap = 7363576kB
[Sat Sep 12 11:07:55 2020] Total swap = 7999480kB
[Sat Sep 12 11:07:55 2020] 8363059 pages RAM
[Sat Sep 12 11:07:55 2020] 0 pages HighMem/MovableOnly
[Sat Sep 12 11:07:55 2020] 168740 pages reserved
[Sat Sep 12 11:07:55 2020] 0 pages hwpoisoned
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl
2020-09-13 0:38 ` [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl Andrew Morton
@ 2020-09-13 6:33 ` Christoph Hellwig
2020-09-13 23:57 ` Andrew Morton
0 siblings, 1 reply; 3+ messages in thread
From: Christoph Hellwig @ 2020-09-13 6:33 UTC (permalink / raw)
To: Andrew Morton
Cc: Christoph Hellwig, Al Viro, bugzilla-daemon, gaetan, linux-mm,
Alexey Dobriyan, Josef Bacik
On Sat, Sep 12, 2020 at 05:38:54PM -0700, Andrew Morton wrote:
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
This is probably fixed by the
"[PATCH 0/6] Some buffer management fixes for proc"
series from Josef.
>
> On Sat, 12 Sep 2020 09:12:18 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
> > https://bugzilla.kernel.org/show_bug.cgi?id=209245
> >
> > Bug ID: 209245
> > Summary: 5.8.x Cannot allocate memory requesting sysctl
> > Product: Memory Management
> > Version: 2.5
> > Kernel Version: 5.8.x
> > Hardware: All
> > OS: Linux
> > Tree: Mainline
> > Status: NEW
> > Severity: high
> > Priority: P1
> > Component: Page Allocator
> > Assignee: akpm@linux-foundation.org
> > Reporter: gaetan@nexylan.com
> > Regression: No
>
> Actually, "Regression: yes".
>
> > Created attachment 292483
> > --> https://bugzilla.kernel.org/attachment.cgi?id=292483&action=edit
> > dmesg
> >
> > Since we upgraded a few nodes to 5.8.x (Gentoo Linux), we are having random
> > "Cannot allocate memory" when requesting infos from sysctl despite system has
> > free memory available.
> >
> > Steps to reproduce :
> >
> > 1/ Have system with heavy workload under 5.8.2 (reproduced with 5.8.8)
> > 2/ At some point, accessing sysctl is not possible anymore :
> >
> >
> > ~# cat /proc/sys/net/netfilter/nf_conntrack_count
> > cat: /proc/sys/net/netfilter/nf_conntrack_count: Cannot allocate memory
> >
> >
> > 3/ Dmesg drops memory allocation errors (see attached)
> >
> >
> > Memory usage :
> >
> > total used free shared buff/cache available
> > Mem: 32009 22029 1547 0 8431 9506
> > Swap: 7811 621 7190
> >
> > Node is running a single mongodb process.
> >
>
> The out-of-memory dump is below.
>
> proc_sys_call_handler() is attempting an order-5 kmalloc for a read of
> a simple int. Presumably because cat(1) is starting out with a 128kb
> read().
>
> AFAICT this is due to changes made in 32927393dc1cc ("sysctl: pass
> kernel pointers to ->proc_handler")?
>
>
>
>
> [Sat Sep 12 11:07:55 2020] cat: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
> [Sat Sep 12 11:07:55 2020] CPU: 10 PID: 32410 Comm: cat Not tainted 5.8.8-gentoo #1
> [Sat Sep 12 11:07:55 2020] Hardware name: Supermicro MBI-6119M-T2N/B2SC1-CPU, BIOS 1.1 06/25/2019
> [Sat Sep 12 11:07:55 2020] Call Trace:
> [Sat Sep 12 11:07:55 2020] dump_stack+0x6d/0x8b
> [Sat Sep 12 11:07:55 2020] warn_alloc+0xf6/0x160
> [Sat Sep 12 11:07:55 2020] __alloc_pages_slowpath.constprop.123+0xb28/0xb70
> [Sat Sep 12 11:07:55 2020] ? __alloc_pages_nodemask+0x144/0x2e0
> [Sat Sep 12 11:07:55 2020] __alloc_pages_nodemask+0x2af/0x2e0
> [Sat Sep 12 11:07:55 2020] kmalloc_order+0x13/0x70
> [Sat Sep 12 11:07:55 2020] kmalloc_order_trace+0x19/0xa0
> [Sat Sep 12 11:07:55 2020] proc_sys_call_handler+0xf9/0x170
> [Sat Sep 12 11:07:55 2020] vfs_read+0x96/0x170
> [Sat Sep 12 11:07:55 2020] ksys_read+0x9c/0xd0
> [Sat Sep 12 11:07:55 2020] do_syscall_64+0x44/0x70
> [Sat Sep 12 11:07:55 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [Sat Sep 12 11:07:55 2020] RIP: 0033:0x7f2c7a19ac7b
> [Sat Sep 12 11:07:55 2020] Code: 00 3d 00 00 41 00 74 07 31 c0 e9 e9 fe ff ff 50 48 8d 3d 60 63 09 00 e8 3b c4 01 00 64 8b 04 25 18 00 00 00 85 c0 75 0b 0f 05 <48> 3d 00 f0 ff ff 77 4c c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> [Sat Sep 12 11:07:55 2020] RSP: 002b:00007ffec0af8ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> [Sat Sep 12 11:07:55 2020] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f2c7a19ac7b
> [Sat Sep 12 11:07:55 2020] RDX: 0000000000020000 RSI: 00007f2c7a097000 RDI: 0000000000000003
> [Sat Sep 12 11:07:55 2020] RBP: 00007f2c7a097000 R08: 00007f2c7a096010 R09: 0000000000000000
> [Sat Sep 12 11:07:55 2020] R10: 0000559b45d2aa00 R11: 0000000000000246 R12: 00007f2c7a097000
> [Sat Sep 12 11:07:55 2020] R13: 0000000000000003 R14: 0000000000000fff R15: 0000000000020000
> [Sat Sep 12 11:07:55 2020] Mem-Info:
> [Sat Sep 12 11:07:55 2020] active_anon:5210006 inactive_anon:373749 isolated_anon:0
> active_file:972354 inactive_file:990026 isolated_file:0
> unevictable:1958 dirty:70 writeback:0
> slab_reclaimable:198141 slab_unreclaimable:19557
> mapped:8172 shmem:29 pagetables:11756 bounce:0
> free:391832 free_pcp:0 free_cma:0
> [Sat Sep 12 11:07:55 2020] Node 0 active_anon:20840024kB inactive_anon:1494996kB active_file:3889416kB inactive_file:3960104kB unevictable:7832kB isolated(anon):0kB isolated(file):0kB mapped:32688kB dirty:280kB writeback:0kB shmem:116kB writeback_tmp:0kB all_unreclaimable? no
> [Sat Sep 12 11:07:55 2020] Node 0 DMA free:11800kB min:8kB low:20kB high:32kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 2108 31993 31993
> [Sat Sep 12 11:07:55 2020] Node 0 DMA32 free:153020kB min:1504kB low:3656kB high:5808kB reserved_highatomic:0KB active_anon:1169844kB inactive_anon:206804kB active_file:180128kB inactive_file:209060kB unevictable:0kB writepending:0kB present:2257504kB managed:2159144kB mlocked:0kB kernel_stack:16kB pagetables:1848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 29884 29884
> [Sat Sep 12 11:07:55 2020] Node 0 Normal free:1402508kB min:21380kB low:51980kB high:82580kB reserved_highatomic:0KB active_anon:19670180kB inactive_anon:1288192kB active_file:3709496kB inactive_file:3751000kB unevictable:7832kB writepending:280kB present:31178752kB managed:30602236kB mlocked:7832kB kernel_stack:4496kB pagetables:45176kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 0 0
> [Sat Sep 12 11:07:55 2020] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11800kB
> [Sat Sep 12 11:07:55 2020] Node 0 DMA32: 2120*4kB (UME) 9907*8kB (UME) 3974*16kB (UM) 56*32kB (UM) 3*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153304kB
> [Sat Sep 12 11:07:55 2020] Node 0 Normal: 8036*4kB (UME) 62654*8kB (UME) 54156*16kB (UM) 155*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1404832kB
> [Sat Sep 12 11:07:55 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [Sat Sep 12 11:07:55 2020] 1994278 total pagecache pages
> [Sat Sep 12 11:07:55 2020] 33986 pages in swap cache
> [Sat Sep 12 11:07:55 2020] Swap cache stats: add 205324, delete 171339, find 1160015/1183131
> [Sat Sep 12 11:07:55 2020] Free swap = 7363576kB
> [Sat Sep 12 11:07:55 2020] Total swap = 7999480kB
> [Sat Sep 12 11:07:55 2020] 8363059 pages RAM
> [Sat Sep 12 11:07:55 2020] 0 pages HighMem/MovableOnly
> [Sat Sep 12 11:07:55 2020] 168740 pages reserved
> [Sat Sep 12 11:07:55 2020] 0 pages hwpoisoned
---end quoted text---
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl
2020-09-13 6:33 ` Christoph Hellwig
@ 2020-09-13 23:57 ` Andrew Morton
0 siblings, 0 replies; 3+ messages in thread
From: Andrew Morton @ 2020-09-13 23:57 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Al Viro, bugzilla-daemon, gaetan, linux-mm, Alexey Dobriyan, Josef Bacik
On Sun, 13 Sep 2020 08:33:02 +0200 Christoph Hellwig <hch@lst.de> wrote:
> On Sat, Sep 12, 2020 at 05:38:54PM -0700, Andrew Morton wrote:
> > (switched to email. Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
>
> This is probably fixed by the
>
> "[PATCH 0/6] Some buffer management fixes for proc"
>
> series from Josef.
Something backportable to -stable might be needed?
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-09-13 23:58 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <bug-209245-27@https.bugzilla.kernel.org/>
2020-09-13 0:38 ` [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl Andrew Morton
2020-09-13 6:33 ` Christoph Hellwig
2020-09-13 23:57 ` Andrew Morton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.