* Stacktrace from 5.2.26 kernel.
@ 2020-04-02 15:35 jesper
2020-04-02 16:16 ` Stacktrace from 5.4.26 kernel Jeff Layton
0 siblings, 1 reply; 2+ messages in thread
From: jesper @ 2020-04-02 15:35 UTC (permalink / raw)
To: ceph-devel
Kernel. 5.4.26 and Ceph Luminous
Seem to be working still but I got this after putting load on it.
[ 785.581198] kworker/3:2: page allocation failure: order:0,
mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
[ 785.581211] CPU: 3 PID: 332 Comm: kworker/3:2 Not tainted 5.4.26 #4
[ 785.581212] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 785.581251] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[ 785.581252] Call Trace:
[ 785.581255] <IRQ>
[ 785.581271] dump_stack+0x6d/0x95
[ 785.581275] warn_alloc+0xfe/0x160
[ 785.581277] __alloc_pages_slowpath+0xe07/0xe40
[ 785.581282] ? dev_gro_receive+0x626/0x690
[ 785.581284] __alloc_pages_nodemask+0x2cd/0x320
[ 785.581287] alloc_pages_current+0x6a/0xe0
[ 785.581298] skb_page_frag_refill+0xd4/0x100
[ 785.581302] try_fill_recv+0x3ed/0x740 [virtio_net]
[ 785.581304] virtnet_poll+0x31f/0x349 [virtio_net]
[ 785.581306] net_rx_action+0x140/0x3c0
[ 785.581313] __do_softirq+0xe4/0x2da
[ 785.581318] irq_exit+0xae/0xb0
[ 785.581320] do_IRQ+0x59/0xe0
[ 785.581322] common_interrupt+0xf/0xf
[ 785.581322] </IRQ>
[ 785.581332] RIP: 0010:tcp_rcv_space_adjust+0x56/0x1c0
[ 785.581333] Code: 06 00 00 bf 00 00 00 00 89 d1 48 89 f0 c1 e9 03 48 2b
83 50 08 00 00 48 0f 48 c7 39 c1 77 04 85 d2 75 4a 5b 41 5c 41 5d 41 5e
<41> 5f 5d c3 65 8b 05 ff 40 09 5b 89 c0 48 0f a3 05 05 2f ec 00 73
[ 785.581334] RSP: 0018:ffffa47c40267bf0 EFLAGS: 00000202 ORIG_RAX:
ffffffffffffffdd
[ 785.581335] RAX: 0000000000000002 RBX: 0000000000005437 RCX:
00000000000004fd
[ 785.581336] RDX: 00000000000027eb RSI: 000000002ed30b4b RDI:
0000000000000000
[ 785.581336] RBP: ffffa47c40267bf8 R08: 0000011c5ab9d567 R09:
0000000000000001
[ 785.581337] R10: ffffa47c40267d38 R11: 0000000000000000 R12:
ffff94854d06d400
[ 785.581338] R13: ffff948711b64600 R14: ffff948711b64b74 R15:
0000000000000000
[ 785.581340] tcp_recvmsg+0x297/0xb70
[ 785.581344] ? _cond_resched+0x19/0x40
[ 785.581347] ? aa_sk_perm+0x43/0x190
[ 785.581351] inet_recvmsg+0x5e/0xe0
[ 785.581353] sock_recvmsg+0x66/0x70
[ 785.581357] ceph_tcp_recvpage+0x76/0xa0 [libceph]
[ 785.581363] ceph_con_workfn+0x1a33/0x2be0 [libceph]
[ 785.581365] process_one_work+0x20f/0x400
[ 785.581366] worker_thread+0x34/0x410
[ 785.581369] kthread+0x121/0x140
[ 785.581370] ? process_one_work+0x400/0x400
[ 785.581371] ? kthread_park+0x90/0x90
[ 785.581372] ret_from_fork+0x35/0x40
[ 785.581374] Mem-Info:
[ 785.581379] active_anon:5599 inactive_anon:7153 isolated_anon:0
[ 785.581382] Node 0 active_anon:22396kB inactive_anon:28612kB
active_file:31252kB inactive_file:7673984kB unevictable:0kB
isolated(anon):0kB isolated(file):64kB mapped:28112kB dirty:0kB
writeback:0kB shmem:2384kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 785.581383] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15992kB managed:15908kB
mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB
[ 785.581386] lowmem_reserve[]: 0 3444 7878 7878 7878
[ 785.581387] Node 0 DMA32 free:31476kB min:37680kB low:45052kB
high:52424kB active_anon:792kB inactive_anon:1216kB active_file:2316kB
inactive_file:3427096kB unevictable:0kB writepending:0kB present:3653608kB
managed:3588072kB mlocked:0kB kernel_stack:80kB pagetables:36kB bounce:0kB
free_pcp:808kB local_pcp:696kB free_cma:0kB
[ 785.581390] lowmem_reserve[]: 0 0 4433 4433 4433
[ 785.581391] Node 0 Normal free:20672kB min:56392kB low:65880kB
high:75368kB active_anon:21604kB inactive_anon:27396kB active_file:28936kB
inactive_file:4246864kB unevictable:0kB writepending:0kB present:4718592kB
managed:4548188kB mlocked:0kB kernel_stack:3280kB pagetables:4208kB
bounce:0kB free_pcp:1168kB local_pcp:1168kB free_cma:0kB
[ 785.581393] lowmem_reserve[]: 0 0 0 0 0
[ 785.581395] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
15908kB
[ 785.581399] Node 0 DMA32: 35*4kB (UMEH) 109*8kB (UMEH) 372*16kB (UMEH)
144*32kB (MEH) 28*64kB (MEH) 37*128kB (ME) 55*256kB (UME) 0*512kB 0*1024kB
0*2048kB 0*4096kB = 32180kB
[ 785.581404] Node 0 Normal: 347*4kB (UMEH) 250*8kB (UMEH) 655*16kB
(UMEH) 80*32kB (UMEH) 29*64kB (ME) 7*128kB (MEH) 6*256kB (MEH) 0*512kB
0*1024kB 0*2048kB 0*4096kB = 20716kB
[ 785.581409] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=1048576kB
[ 785.581409] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB
[ 785.581410] 1926939 total pagecache pages
[ 785.581413] 38 pages in swap cache
[ 785.581413] Swap cache stats: add 994, delete 956, find 81/89
[ 785.581414] Free swap = 4033776kB
[ 785.581414] Total swap = 4038652kB
[ 785.581415] 2097048 pages RAM
[ 785.581415] 0 pages HighMem/MovableOnly
[ 785.581416] 59006 pages reserved
[ 785.581416] 0 pages cma reserved
[ 785.581417] 0 pages hwpoisoned
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Stacktrace from 5.4.26 kernel.
2020-04-02 15:35 Stacktrace from 5.2.26 kernel jesper
@ 2020-04-02 16:16 ` Jeff Layton
0 siblings, 0 replies; 2+ messages in thread
From: Jeff Layton @ 2020-04-02 16:16 UTC (permalink / raw)
To: jesper, ceph-devel; +Cc: virtualization
On Thu, 2020-04-02 at 17:35 +0200, jesper@krogh.cc wrote:
> Kernel. 5.4.26 and Ceph Luminous
>
> Seem to be working still but I got this after putting load on it.
>
> [ 785.581198] kworker/3:2: page allocation failure: order:0,
> mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
(cc'ing virtualization mailing list)
Basically, looks like the kernel needed to allocate a page (4k) of
memory in order to handle a receive, and wasn't able to do so. This is a
GFP_ATOMIC allocation down inside the networking code, so it couldn't do
anything like reclaim memory to satisfy the allocation.
Looks like the box was just critically out of memory, but maybe the
problem is something else.
> [ 785.581211] CPU: 3 PID: 332 Comm: kworker/3:2 Not tainted 5.4.26 #4
> [ 785.581212] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [ 785.581251] Workqueue: ceph-msgr ceph_con_workfn [libceph]
> [ 785.581252] Call Trace:
> [ 785.581255] <IRQ>
> [ 785.581271] dump_stack+0x6d/0x95
> [ 785.581275] warn_alloc+0xfe/0x160
> [ 785.581277] __alloc_pages_slowpath+0xe07/0xe40
> [ 785.581282] ? dev_gro_receive+0x626/0x690
> [ 785.581284] __alloc_pages_nodemask+0x2cd/0x320
> [ 785.581287] alloc_pages_current+0x6a/0xe0
> [ 785.581298] skb_page_frag_refill+0xd4/0x100
> [ 785.581302] try_fill_recv+0x3ed/0x740 [virtio_net]
> [ 785.581304] virtnet_poll+0x31f/0x349 [virtio_net]
> [ 785.581306] net_rx_action+0x140/0x3c0
> [ 785.581313] __do_softirq+0xe4/0x2da
> [ 785.581318] irq_exit+0xae/0xb0
> [ 785.581320] do_IRQ+0x59/0xe0
> [ 785.581322] common_interrupt+0xf/0xf
> [ 785.581322] </IRQ>
> [ 785.581332] RIP: 0010:tcp_rcv_space_adjust+0x56/0x1c0
> [ 785.581333] Code: 06 00 00 bf 00 00 00 00 89 d1 48 89 f0 c1 e9 03 48 2b
> 83 50 08 00 00 48 0f 48 c7 39 c1 77 04 85 d2 75 4a 5b 41 5c 41 5d 41 5e
> <41> 5f 5d c3 65 8b 05 ff 40 09 5b 89 c0 48 0f a3 05 05 2f ec 00 73
> [ 785.581334] RSP: 0018:ffffa47c40267bf0 EFLAGS: 00000202 ORIG_RAX:
> ffffffffffffffdd
> [ 785.581335] RAX: 0000000000000002 RBX: 0000000000005437 RCX:
> 00000000000004fd
> [ 785.581336] RDX: 00000000000027eb RSI: 000000002ed30b4b RDI:
> 0000000000000000
> [ 785.581336] RBP: ffffa47c40267bf8 R08: 0000011c5ab9d567 R09:
> 0000000000000001
> [ 785.581337] R10: ffffa47c40267d38 R11: 0000000000000000 R12:
> ffff94854d06d400
> [ 785.581338] R13: ffff948711b64600 R14: ffff948711b64b74 R15:
> 0000000000000000
> [ 785.581340] tcp_recvmsg+0x297/0xb70
> [ 785.581344] ? _cond_resched+0x19/0x40
> [ 785.581347] ? aa_sk_perm+0x43/0x190
> [ 785.581351] inet_recvmsg+0x5e/0xe0
> [ 785.581353] sock_recvmsg+0x66/0x70
> [ 785.581357] ceph_tcp_recvpage+0x76/0xa0 [libceph]
> [ 785.581363] ceph_con_workfn+0x1a33/0x2be0 [libceph]
> [ 785.581365] process_one_work+0x20f/0x400
> [ 785.581366] worker_thread+0x34/0x410
> [ 785.581369] kthread+0x121/0x140
> [ 785.581370] ? process_one_work+0x400/0x400
> [ 785.581371] ? kthread_park+0x90/0x90
> [ 785.581372] ret_from_fork+0x35/0x40
> [ 785.581374] Mem-Info:
> [ 785.581379] active_anon:5599 inactive_anon:7153 isolated_anon:0
> [ 785.581382] Node 0 active_anon:22396kB inactive_anon:28612kB
> active_file:31252kB inactive_file:7673984kB unevictable:0kB
> isolated(anon):0kB isolated(file):64kB mapped:28112kB dirty:0kB
> writeback:0kB shmem:2384kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
> 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
> [ 785.581383] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB writepending:0kB present:15992kB managed:15908kB
> mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB
> local_pcp:0kB free_cma:0kB
> [ 785.581386] lowmem_reserve[]: 0 3444 7878 7878 7878
> [ 785.581387] Node 0 DMA32 free:31476kB min:37680kB low:45052kB
> high:52424kB active_anon:792kB inactive_anon:1216kB active_file:2316kB
> inactive_file:3427096kB unevictable:0kB writepending:0kB present:3653608kB
> managed:3588072kB mlocked:0kB kernel_stack:80kB pagetables:36kB bounce:0kB
> free_pcp:808kB local_pcp:696kB free_cma:0kB
> [ 785.581390] lowmem_reserve[]: 0 0 4433 4433 4433
> [ 785.581391] Node 0 Normal free:20672kB min:56392kB low:65880kB
> high:75368kB active_anon:21604kB inactive_anon:27396kB active_file:28936kB
> inactive_file:4246864kB unevictable:0kB writepending:0kB present:4718592kB
> managed:4548188kB mlocked:0kB kernel_stack:3280kB pagetables:4208kB
> bounce:0kB free_pcp:1168kB local_pcp:1168kB free_cma:0kB
> [ 785.581393] lowmem_reserve[]: 0 0 0 0 0
> [ 785.581395] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U)
> 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
> 15908kB
> [ 785.581399] Node 0 DMA32: 35*4kB (UMEH) 109*8kB (UMEH) 372*16kB (UMEH)
> 144*32kB (MEH) 28*64kB (MEH) 37*128kB (ME) 55*256kB (UME) 0*512kB 0*1024kB
> 0*2048kB 0*4096kB = 32180kB
> [ 785.581404] Node 0 Normal: 347*4kB (UMEH) 250*8kB (UMEH) 655*16kB
> (UMEH) 80*32kB (UMEH) 29*64kB (ME) 7*128kB (MEH) 6*256kB (MEH) 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 20716kB
> [ 785.581409] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
> hugepages_size=1048576kB
> [ 785.581409] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
> hugepages_size=2048kB
> [ 785.581410] 1926939 total pagecache pages
> [ 785.581413] 38 pages in swap cache
> [ 785.581413] Swap cache stats: add 994, delete 956, find 81/89
> [ 785.581414] Free swap = 4033776kB
> [ 785.581414] Total swap = 4038652kB
> [ 785.581415] 2097048 pages RAM
> [ 785.581415] 0 pages HighMem/MovableOnly
> [ 785.581416] 59006 pages reserved
> [ 785.581416] 0 pages cma reserved
> [ 785.581417] 0 pages hwpoisoned
>
>
--
Jeff Layton <jlayton@poochiereds.net>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-04-02 16:16 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-02 15:35 Stacktrace from 5.2.26 kernel jesper
2020-04-02 16:16 ` Stacktrace from 5.4.26 kernel Jeff Layton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.