* memory reclaim problems on fs usage
@ 2015-11-10 22:13 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-10 22:13 UTC (permalink / raw)
To: linux-mm, xfs
Hi.
I have a x86_64 system running 4.1.12 kernel on top of software raid array (raid 1 and 6)
on top of adaptec HBA card (ASR71605E) that provides connectivity to 16 sata
rotational disks. fs is XFS.
System has 8GB of ram and 111GB of swap on ssd disk (swap is barely used:
~7,4MB in use).
Usage scenario on this machine is 5-10 (sometimes more) rsnapshot/rsync processes
doing hardlinks and copying tons of files.
The usual (repeatable) problem is like this:
full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
partial log:
122365.832373] swapper/3: page allocation failure: order:0, mode:0x20
[122365.832382] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.1.12-3 #1
[122365.832384] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.2a 06/27/2012
[122365.832386] 0000000000000000 ab5d50b5f2ae9872 ffff88023fcc3b18 ffffffff8164b37a
[122365.832390] 0000000000000000 0000000000000020 ffff88023fcc3ba8 ffffffff8118f02e
[122365.832392] 0000000000000000 0000000000000001 ffff880200000030 ffff8800ba984400
[122365.832395] Call Trace:
[122365.832398] <IRQ> [<ffffffff8164b37a>] dump_stack+0x45/0x57
[122365.832409] [<ffffffff8118f02e>] warn_alloc_failed+0xfe/0x150
[122365.832415] [<ffffffffc0247658>] ? raid5_align_endio+0x148/0x160 [raid456]
[122365.832418] [<ffffffff81192c02>] __alloc_pages_nodemask+0x322/0xa90
[122365.832423] [<ffffffff815281bc>] __alloc_page_frag+0x12c/0x150
[122365.832426] [<ffffffff8152afd6>] __alloc_rx_skb+0x66/0x100
[122365.832430] [<ffffffff8131101c>] ? __blk_mq_complete_request+0x7c/0x110
[122365.832433] [<ffffffff8152b0d2>] __napi_alloc_skb+0x22/0x50
[122365.832440] [<ffffffffc0336f1e>] e1000_clean_rx_irq+0x33e/0x3f0 [e1000e]
[122365.832444] [<ffffffff810eaa10>] ? timer_cpu_notify+0x160/0x160
[122365.832449] [<ffffffffc033debc>] e1000e_poll+0xbc/0x2f0 [e1000e]
[122365.832457] [<ffffffffc00e244f>] ? aac_src_intr_message+0xaf/0x3e0 [aacraid]
[122365.832461] [<ffffffff8153a7c2>] net_rx_action+0x212/0x340
[122365.832465] [<ffffffff8107b2f3>] __do_softirq+0x103/0x280
[122365.832467] [<ffffffff8107b5ed>] irq_exit+0xad/0xb0
[122365.832471] [<ffffffff81653a58>] do_IRQ+0x58/0xf0
[122365.832474] [<ffffffff816518ae>] common_interrupt+0x6e/0x6e
[122365.832476] <EOI> [<ffffffff8101f34c>] ? mwait_idle+0x8c/0x150
[122365.832482] [<ffffffff8101fd4f>] arch_cpu_idle+0xf/0x20
[122365.832485] [<ffffffff810b92e0>] cpu_startup_entry+0x380/0x400
[122365.832488] [<ffffffff8104bf7d>] start_secondary+0x17d/0x1a0
[122365.832491] Mem-Info:
[122365.832496] active_anon:28246 inactive_anon:31593 isolated_anon:0
active_file:6641 inactive_file:1616279 isolated_file:0
unevictable:0 dirty:136960 writeback:0 unstable:0
slab_reclaimable:191482 slab_unreclaimable:34061
mapped:3744 shmem:0 pagetables:1015 bounce:0
free:5700 free_pcp:551 free_cma:0
[122365.832500] Node 0 DMA free:15884kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:15884kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[122365.832505] lowmem_reserve[]: 0 2968 7958 7958
[122365.832508] Node 0 DMA32 free:6916kB min:4224kB low:5280kB high:6336kB active_anon:34904kB inactive_anon:44024kB active_file:9076kB inactive_file:2313600kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:3120704kB managed:3043796kB mlocked:0kB dirty:199004kB writeback:0kB mapped:5488kB shmem:0kB slab_reclaimable:441924kB slab_unreclaimable:38440kB kernel_stack:960kB pagetables:1084kB unstable:0kB
bounce:0kB free_pcp:1132kB local_pcp:184kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[122365.832514] lowmem_reserve[]: 0 0 4990 4990
[122365.832517] Node 0 Normal free:0kB min:7104kB low:8880kB high:10656kB active_anon:78080kB inactive_anon:82348kB active_file:17488kB inactive_file:4151516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:5242880kB managed:5109980kB mlocked:0kB dirty:348836kB writeback:0kB mapped:9488kB shmem:0kB slab_reclaimable:324004kB slab_unreclaimable:97804kB kernel_stack:1760kB pagetables:2976kB unstable:0kB
bounce:0kB free_pcp:1072kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[122365.832522] lowmem_reserve[]: 0 0 0 0
[122365.832525] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15884kB
[122365.832536] Node 0 DMA32: 1487*4kB (UE) 0*8kB 7*16kB (R) 9*32kB (R) 5*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6668kB
[122365.832544] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[122365.832552] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[122365.832554] 1623035 total pagecache pages
[122365.832556] 96 pages in swap cache
[122365.832558] Swap cache stats: add 1941, delete 1845, find 1489/1529
[122365.832559] Free swap = 117213444kB
[122365.832561] Total swap = 117220820kB
[122365.832562] 2094888 pages RAM
[122365.832564] 0 pages HighMem/MovableOnly
[122365.832565] 48377 pages reserved
[122365.832567] 4096 pages cma reserved
[122365.832568] 0 pages hwpoisoned
[122377.888271] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122379.889804] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122381.891337] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122383.892871] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
Tried to ask on #xfs@freenode and #mm@oftc and did a bit of irc relay between channels and people.
Essential parts of discussion:
#xfs
22:00 < dchinner__> arekm: so teh machine has 8GB ram, and it has almost 6GB of inactive file pages?
22:01 < dchinner__> it seems like there is a lot of reclaimable memory in that machine when it starts having problems...
22:04 < dchinner__> indeed, the ethernet driver is having problems with an order 0 allocation, when there appears to be lots of reclaimable memory....
22:04 < arekm> dchinner__: 8GB of ram, 111GB of swap (ssd; looks unused - only ~7.4MB in use), 5x rsync, 1xmysqldump, raid1 and raid6 on sata disks
22:04 < dchinner__> ah:
22:05 < dchinner__> Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
22:05 < dchinner__> looks like there's a problem with a zone imbalance
22:05 < dchinner__> Node 0 DMA32: 1487*4kB (UE) 0*8kB 7*16kB (R) 9*32kB (R) 5*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6668kB
22:07 < dchinner__> given that the zones have heaps of clean, inactive file pages, the DMA32 and NORMAL zones are not marked as "all unreclaimable", and there's the free
pages in ZONE_NORMAL have been completely drained
22:07 < dchinner__> I'd be asking the mm folks what is going on
22:08 < dchinner__> I'd say XFS is backed up on the same issue
so normal zone is drained to 0
#mm
22:15 < arekmx> hi. I'm running backup system on 4.1.12 kernel. Machine mostly does rsnapshot/rsyncs (5-10 in parallel). Unfortunately it hits memory problems often ->
http://sprunge.us/VEiE . I've asked XFS people and the conclusion was that this is most likely mm problem -> http://sprunge.us/ggVG Any ideas what could be
going on ? (like normal zone is completly drained for example)
22:29 < sasha_> Wild guess: your xfs is on a rather slow storage device (network?)
22:33 < arekmx> sasha_: raid6 on local rotational sata disks... so could be slow, especially when 10x rsyncs start and hdd heads need to jump like crazy
22:33 < sasha_> Hm, shouldn't be *that* slow though
22:34 < sasha_> The scenario I see is that xfs can't run reclaim fast enough, so the system runs out of memory and it appears to have a lot of "unused cache" it should
have freed
22:34 < sasha_> Look at all those cpus stuck in xfs reclaim, while one of them is waiting for IO
22:38 < sasha_> I suppose the easiest one is just not caching on that filesystem
I don't think there is a way to do that.
22:41 < sasha_> Or maybe your RAID box/disks are dying?
Nope, good condition according to smart logs (but started long tests to retest)
#xfs:
22:40 < dchinner__> arekm: XFs is waiting on slab cache reclaim
22:41 < dchinner__> because there are already as many slab reclaimers as there are AGs, and reclaim can't progress any faster than that
22:41 < dchinner__> but slab reclaim does not prevent clean pages from being reclaimed by direct reclaim during memory allocation
22:41 < dchinner__> it's a completely different part of memory reclaim
22:42 < dchinner__> the fact that XFs is repeatedly saying "memory allocation failed" means it is not getting backed up on slab cache reclaim
22:42 < dchinner__> especially as it's a GFP_NOFS allocation which means the slab shrinkers are being skipped.
22:43 < dchinner__> direct page cache reclaim should be occurring on GFP_NOFS allocation because there are clean pages available to be reclaimed
22:44 < dchinner__> but that is not happening - the processes blocked in the shrinkers are not relevant to the XFS allocations that
Overall I was asked to post this to both mailing list to get better coverage and possibly solution to the problem.
kernel config:
http://sprunge.us/SRUi
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdg[0] sdi[5] sdh[4] sdd[3] sdf[2] sde[1]
11720540160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
bitmap: 1/22 pages [4KB], 65536KB chunk
md3 : active raid6 sdj[9] sdq[7] sdp[6] sdo[10] sdn[4] sdm[8] sdl[2] sdk[1]
5859781632 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 3/8 pages [12KB], 65536KB chunk
md1 : active raid1 sdb1[0] sdc1[1]
524224 blocks [2/2] [UU]
md2 : active raid1 sdb2[0] sdc2[1]
731918016 blocks super 1.2 [2/2] [UU]
rsync/rsnapshot processes operate on md3 and md4
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* memory reclaim problems on fs usage
@ 2015-11-10 22:13 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-10 22:13 UTC (permalink / raw)
To: linux-mm, xfs
Hi.
I have a x86_64 system running 4.1.12 kernel on top of software raid array (raid 1 and 6)
on top of adaptec HBA card (ASR71605E) that provides connectivity to 16 sata
rotational disks. fs is XFS.
System has 8GB of ram and 111GB of swap on ssd disk (swap is barely used:
~7,4MB in use).
Usage scenario on this machine is 5-10 (sometimes more) rsnapshot/rsync processes
doing hardlinks and copying tons of files.
The usual (repeatable) problem is like this:
full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
partial log:
122365.832373] swapper/3: page allocation failure: order:0, mode:0x20
[122365.832382] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.1.12-3 #1
[122365.832384] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.2a 06/27/2012
[122365.832386] 0000000000000000 ab5d50b5f2ae9872 ffff88023fcc3b18 ffffffff8164b37a
[122365.832390] 0000000000000000 0000000000000020 ffff88023fcc3ba8 ffffffff8118f02e
[122365.832392] 0000000000000000 0000000000000001 ffff880200000030 ffff8800ba984400
[122365.832395] Call Trace:
[122365.832398] <IRQ> [<ffffffff8164b37a>] dump_stack+0x45/0x57
[122365.832409] [<ffffffff8118f02e>] warn_alloc_failed+0xfe/0x150
[122365.832415] [<ffffffffc0247658>] ? raid5_align_endio+0x148/0x160 [raid456]
[122365.832418] [<ffffffff81192c02>] __alloc_pages_nodemask+0x322/0xa90
[122365.832423] [<ffffffff815281bc>] __alloc_page_frag+0x12c/0x150
[122365.832426] [<ffffffff8152afd6>] __alloc_rx_skb+0x66/0x100
[122365.832430] [<ffffffff8131101c>] ? __blk_mq_complete_request+0x7c/0x110
[122365.832433] [<ffffffff8152b0d2>] __napi_alloc_skb+0x22/0x50
[122365.832440] [<ffffffffc0336f1e>] e1000_clean_rx_irq+0x33e/0x3f0 [e1000e]
[122365.832444] [<ffffffff810eaa10>] ? timer_cpu_notify+0x160/0x160
[122365.832449] [<ffffffffc033debc>] e1000e_poll+0xbc/0x2f0 [e1000e]
[122365.832457] [<ffffffffc00e244f>] ? aac_src_intr_message+0xaf/0x3e0 [aacraid]
[122365.832461] [<ffffffff8153a7c2>] net_rx_action+0x212/0x340
[122365.832465] [<ffffffff8107b2f3>] __do_softirq+0x103/0x280
[122365.832467] [<ffffffff8107b5ed>] irq_exit+0xad/0xb0
[122365.832471] [<ffffffff81653a58>] do_IRQ+0x58/0xf0
[122365.832474] [<ffffffff816518ae>] common_interrupt+0x6e/0x6e
[122365.832476] <EOI> [<ffffffff8101f34c>] ? mwait_idle+0x8c/0x150
[122365.832482] [<ffffffff8101fd4f>] arch_cpu_idle+0xf/0x20
[122365.832485] [<ffffffff810b92e0>] cpu_startup_entry+0x380/0x400
[122365.832488] [<ffffffff8104bf7d>] start_secondary+0x17d/0x1a0
[122365.832491] Mem-Info:
[122365.832496] active_anon:28246 inactive_anon:31593 isolated_anon:0
active_file:6641 inactive_file:1616279 isolated_file:0
unevictable:0 dirty:136960 writeback:0 unstable:0
slab_reclaimable:191482 slab_unreclaimable:34061
mapped:3744 shmem:0 pagetables:1015 bounce:0
free:5700 free_pcp:551 free_cma:0
[122365.832500] Node 0 DMA free:15884kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:15884kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[122365.832505] lowmem_reserve[]: 0 2968 7958 7958
[122365.832508] Node 0 DMA32 free:6916kB min:4224kB low:5280kB high:6336kB active_anon:34904kB inactive_anon:44024kB active_file:9076kB inactive_file:2313600kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:3120704kB managed:3043796kB mlocked:0kB dirty:199004kB writeback:0kB mapped:5488kB shmem:0kB slab_reclaimable:441924kB slab_unreclaimable:38440kB kernel_stack:960kB pagetables:1084kB unstable:0kB
bounce:0kB free_pcp:1132kB local_pcp:184kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[122365.832514] lowmem_reserve[]: 0 0 4990 4990
[122365.832517] Node 0 Normal free:0kB min:7104kB low:8880kB high:10656kB active_anon:78080kB inactive_anon:82348kB active_file:17488kB inactive_file:4151516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:5242880kB managed:5109980kB mlocked:0kB dirty:348836kB writeback:0kB mapped:9488kB shmem:0kB slab_reclaimable:324004kB slab_unreclaimable:97804kB kernel_stack:1760kB pagetables:2976kB unstable:0kB
bounce:0kB free_pcp:1072kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[122365.832522] lowmem_reserve[]: 0 0 0 0
[122365.832525] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15884kB
[122365.832536] Node 0 DMA32: 1487*4kB (UE) 0*8kB 7*16kB (R) 9*32kB (R) 5*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6668kB
[122365.832544] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[122365.832552] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[122365.832554] 1623035 total pagecache pages
[122365.832556] 96 pages in swap cache
[122365.832558] Swap cache stats: add 1941, delete 1845, find 1489/1529
[122365.832559] Free swap = 117213444kB
[122365.832561] Total swap = 117220820kB
[122365.832562] 2094888 pages RAM
[122365.832564] 0 pages HighMem/MovableOnly
[122365.832565] 48377 pages reserved
[122365.832567] 4096 pages cma reserved
[122365.832568] 0 pages hwpoisoned
[122377.888271] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122379.889804] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122381.891337] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
[122383.892871] XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:0x250)
Tried to ask on #xfs@freenode and #mm@oftc and did a bit of irc relay between channels and people.
Essential parts of discussion:
#xfs
22:00 < dchinner__> arekm: so teh machine has 8GB ram, and it has almost 6GB of inactive file pages?
22:01 < dchinner__> it seems like there is a lot of reclaimable memory in that machine when it starts having problems...
22:04 < dchinner__> indeed, the ethernet driver is having problems with an order 0 allocation, when there appears to be lots of reclaimable memory....
22:04 < arekm> dchinner__: 8GB of ram, 111GB of swap (ssd; looks unused - only ~7.4MB in use), 5x rsync, 1xmysqldump, raid1 and raid6 on sata disks
22:04 < dchinner__> ah:
22:05 < dchinner__> Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
22:05 < dchinner__> looks like there's a problem with a zone imbalance
22:05 < dchinner__> Node 0 DMA32: 1487*4kB (UE) 0*8kB 7*16kB (R) 9*32kB (R) 5*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6668kB
22:07 < dchinner__> given that the zones have heaps of clean, inactive file pages, the DMA32 and NORMAL zones are not marked as "all unreclaimable", and there's the free
pages in ZONE_NORMAL have been completely drained
22:07 < dchinner__> I'd be asking the mm folks what is going on
22:08 < dchinner__> I'd say XFS is backed up on the same issue
so normal zone is drained to 0
#mm
22:15 < arekmx> hi. I'm running backup system on 4.1.12 kernel. Machine mostly does rsnapshot/rsyncs (5-10 in parallel). Unfortunately it hits memory problems often ->
http://sprunge.us/VEiE . I've asked XFS people and the conclusion was that this is most likely mm problem -> http://sprunge.us/ggVG Any ideas what could be
going on ? (like normal zone is completly drained for example)
22:29 < sasha_> Wild guess: your xfs is on a rather slow storage device (network?)
22:33 < arekmx> sasha_: raid6 on local rotational sata disks... so could be slow, especially when 10x rsyncs start and hdd heads need to jump like crazy
22:33 < sasha_> Hm, shouldn't be *that* slow though
22:34 < sasha_> The scenario I see is that xfs can't run reclaim fast enough, so the system runs out of memory and it appears to have a lot of "unused cache" it should
have freed
22:34 < sasha_> Look at all those cpus stuck in xfs reclaim, while one of them is waiting for IO
22:38 < sasha_> I suppose the easiest one is just not caching on that filesystem
I don't think there is a way to do that.
22:41 < sasha_> Or maybe your RAID box/disks are dying?
Nope, good condition according to smart logs (but started long tests to retest)
#xfs:
22:40 < dchinner__> arekm: XFs is waiting on slab cache reclaim
22:41 < dchinner__> because there are already as many slab reclaimers as there are AGs, and reclaim can't progress any faster than that
22:41 < dchinner__> but slab reclaim does not prevent clean pages from being reclaimed by direct reclaim during memory allocation
22:41 < dchinner__> it's a completely different part of memory reclaim
22:42 < dchinner__> the fact that XFs is repeatedly saying "memory allocation failed" means it is not getting backed up on slab cache reclaim
22:42 < dchinner__> especially as it's a GFP_NOFS allocation which means the slab shrinkers are being skipped.
22:43 < dchinner__> direct page cache reclaim should be occurring on GFP_NOFS allocation because there are clean pages available to be reclaimed
22:44 < dchinner__> but that is not happening - the processes blocked in the shrinkers are not relevant to the XFS allocations that
Overall I was asked to post this to both mailing list to get better coverage and possibly solution to the problem.
kernel config:
http://sprunge.us/SRUi
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdg[0] sdi[5] sdh[4] sdd[3] sdf[2] sde[1]
11720540160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
bitmap: 1/22 pages [4KB], 65536KB chunk
md3 : active raid6 sdj[9] sdq[7] sdp[6] sdo[10] sdn[4] sdm[8] sdl[2] sdk[1]
5859781632 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 3/8 pages [12KB], 65536KB chunk
md1 : active raid1 sdb1[0] sdc1[1]
524224 blocks [2/2] [UU]
md2 : active raid1 sdb2[0] sdc2[1]
731918016 blocks super 1.2 [2/2] [UU]
rsync/rsnapshot processes operate on md3 and md4
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-10 22:13 ` Arkadiusz Miśkiewicz
@ 2015-11-11 15:58 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-11 15:58 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz, linux-mm, xfs
On 2015/11/11 7:13, Arkadiusz Miśkiewicz wrote:
> The usual (repeatable) problem is like this:
>
> full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
Maybe somebody doing GFP_NOIO allocation which XFS driver doing GFP_NOFS
allocation is waiting for is stalling inside memory allocator. I think that
checking tasks which are stalling inside memory allocator would help.
Please try reproducing this problem with a debug printk() patch shown below
applied. This is a patch which I used for debugging silent lockup problem.
When memory allocation got stuck, lines with MemAlloc keyword will be
printed.
---
fs/xfs/kmem.c | 10 ++-
fs/xfs/xfs_buf.c | 3 +-
include/linux/mmzone.h | 1 +
include/linux/vmstat.h | 1 +
mm/page_alloc.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++
mm/vmscan.c | 22 +++++
6 files changed, 249 insertions(+), 5 deletions(-)
diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c
index a7a3a63..535c136 100644
--- a/fs/xfs/kmem.c
+++ b/fs/xfs/kmem.c
@@ -55,8 +55,9 @@ kmem_alloc(size_t size, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
@@ -120,8 +121,9 @@ kmem_zone_alloc(kmem_zone_t *zone, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..16322cb 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -354,7 +354,8 @@ retry:
*/
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
__func__, gfp_mask);
XFS_STATS_INC(xb_page_retries);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 54d74f6..932a6d6 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -527,6 +527,7 @@ struct zone {
ZONE_PADDING(_pad3_)
/* Zone statistics */
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
+ unsigned long stat_last_updated[NR_VM_ZONE_STAT_ITEMS];
} ____cacheline_internodealigned_in_smp;
enum zone_flags {
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 82e7db7..2488925 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -115,6 +115,7 @@ static inline void zone_page_state_add(long x, struct zone *zone,
{
atomic_long_add(x, &zone->vm_stat[item]);
atomic_long_add(x, &vm_stat[item]);
+ zone->stat_last_updated[item] = jiffies;
}
static inline unsigned long global_page_state(enum zone_stat_item item)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 18490f3..35a46b4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -61,6 +61,8 @@
#include <linux/hugetlb.h>
#include <linux/sched/rt.h>
#include <linux/page_owner.h>
+#include <linux/nmi.h>
+#include <linux/kthread.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -2496,6 +2498,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
}
#endif /* CONFIG_COMPACTION */
+pid_t dump_target_pid;
+
/* Perform direct synchronous page reclaim */
static int
__perform_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -2645,6 +2649,208 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_NO_WATERMARKS);
}
+static unsigned long kmallocwd_timeout = 10 * HZ; /* Scan interval. */
+static u8 memalloc_counter_active_index; /* Either 0 or 1. */
+static int memalloc_counter[2]; /* Number of tasks doing memory allocation. */
+
+struct memalloc {
+ struct list_head list; /* Connected to memalloc_list. */
+ struct task_struct *task; /* Iniatilized to current. */
+ unsigned long start; /* Initialized to jiffies. */
+ unsigned int order;
+ gfp_t gfp;
+ u8 index; /* Initialized to memalloc_counter_active_index. */
+ u8 dumped;
+};
+
+static LIST_HEAD(memalloc_list); /* List of "struct memalloc".*/
+static DEFINE_SPINLOCK(memalloc_list_lock); /* Lock for memalloc_list. */
+
+/*
+ * kmallocwd - A kernel thread for monitoring memory allocation stalls.
+ *
+ * @unused: Not used.
+ *
+ * This kernel thread does not terminate.
+ */
+static int kmallocwd(void *unused)
+{
+ struct memalloc *m;
+ struct task_struct *g, *p;
+ unsigned long now;
+ unsigned int sigkill_pending;
+ unsigned int memdie_pending;
+ unsigned int stalling_tasks;
+ u8 index;
+ pid_t pid;
+
+ not_stalling: /* Healty case. */
+ /* Switch active counter and wait for timeout duration. */
+ index = memalloc_counter_active_index;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter_active_index ^= 1;
+ spin_unlock(&memalloc_list_lock);
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ /*
+ * If memory allocations are working, the counter should remain 0
+ * because tasks will be able to call both start_memalloc_timer()
+ * and stop_memalloc_timer() within timeout duration.
+ */
+ if (likely(!memalloc_counter[index]))
+ goto not_stalling;
+ maybe_stalling: /* Maybe something is wrong. Let's check. */
+ now = jiffies;
+ /* Count stalling tasks, dying and victim tasks. */
+ sigkill_pending = 0;
+ memdie_pending = 0;
+ stalling_tasks = 0;
+ pid = 0;
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_after(now - m->start, kmallocwd_timeout))
+ stalling_tasks++;
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ memdie_pending++;
+ if (fatal_signal_pending(p))
+ sigkill_pending++;
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ pr_warn("MemAlloc-Info: %u stalling task, %u dying task, %u victim task.\n",
+ stalling_tasks, sigkill_pending, memdie_pending);
+ /* Report stalling tasks, dying and victim tasks. */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout))
+ continue;
+ p = m->task;
+ pr_warn("MemAlloc: %s(%u) gfp=0x%x order=%u delay=%lu\n",
+ p->comm, p->pid, m->gfp, m->order, now - m->start);
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ u8 type = 0;
+
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ type |= 1;
+ if (fatal_signal_pending(p))
+ type |= 2;
+ if (likely(!type))
+ continue;
+ if (p->state & TASK_UNINTERRUPTIBLE)
+ type |= 4;
+ pr_warn("MemAlloc: %s(%u)%s%s%s\n", p->comm, p->pid,
+ (type & 4) ? " uninterruptible" : "",
+ (type & 2) ? " dying" : "",
+ (type & 1) ? " victim" : "");
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ /*
+ * Show traces of newly reported (or too long) stalling tasks.
+ *
+ * Show traces only once per 256 timeouts because their traces
+ * will likely be the same (e.g. cond_sched() or congestion_wait())
+ * when they are stalling inside __alloc_pages_slowpath().
+ */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout) ||
+ m->dumped++)
+ continue;
+ p = m->task;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ if (!pid)
+ pid = p->pid;
+ }
+ spin_unlock(&memalloc_list_lock);
+ /*
+ * Show traces of dying tasks (including victim tasks).
+ *
+ * Only dying tasks which are in trouble (e.g. blocked at unkillable
+ * locks held by memory allocating tasks) will be repeatedly shown.
+ * Therefore, we need to pay attention to tasks repeatedly shown here.
+ */
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (likely(!fatal_signal_pending(p)))
+ continue;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ show_workqueue_state();
+ if (!dump_target_pid)
+ dump_target_pid = -pid;
+ /* Wait until next timeout duration. */
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ if (memalloc_counter[index])
+ goto maybe_stalling;
+ goto not_stalling;
+ return 0; /* To suppress "no return statement" compiler warning. */
+}
+
+static int __init start_kmallocwd(void)
+{
+ if (kmallocwd_timeout) {
+ struct task_struct *task = kthread_run(kmallocwd, NULL,
+ "kmallocwd");
+ BUG_ON(IS_ERR(task));
+ }
+ return 0;
+}
+late_initcall(start_kmallocwd);
+
+static int __init kmallocwd_config(char *str)
+{
+ if (kstrtoul(str, 10, &kmallocwd_timeout) == 0)
+ kmallocwd_timeout = min(kmallocwd_timeout * HZ,
+ (unsigned long) LONG_MAX);
+ return 0;
+}
+__setup("kmallocwd=", kmallocwd_config);
+
+static void start_memalloc_timer(struct memalloc *m, const gfp_t gfp_mask,
+ const int order)
+{
+ if (!kmallocwd_timeout || m->task)
+ return;
+ m->task = current;
+ m->start = jiffies;
+ m->gfp = gfp_mask;
+ m->order = order;
+ m->dumped = 0;
+ spin_lock(&memalloc_list_lock);
+ m->index = memalloc_counter_active_index;
+ memalloc_counter[m->index]++;
+ list_add_tail(&m->list, &memalloc_list);
+ spin_unlock(&memalloc_list_lock);
+}
+
+static void stop_memalloc_timer(struct memalloc *m)
+{
+ if (!m->task)
+ return;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter[m->index]--;
+ list_del(&m->list);
+ spin_unlock(&memalloc_list_lock);
+}
+
static inline struct page *
__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
struct alloc_context *ac)
@@ -2657,6 +2863,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
enum migrate_mode migration_mode = MIGRATE_ASYNC;
bool deferred_compaction = false;
int contended_compaction = COMPACT_CONTENDED_NONE;
+ struct memalloc m = { .task = NULL };
/*
* In the slowpath, we sanity check order to avoid ever trying to
@@ -2678,6 +2885,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
goto nopage;
retry:
+ if (dump_target_pid == -current->pid)
+ dump_target_pid = -dump_target_pid;
+
if (!(gfp_mask & __GFP_NO_KSWAPD))
wake_all_kswapds(order, ac);
@@ -2740,6 +2950,8 @@ retry:
if (test_thread_flag(TIF_MEMDIE) && !(gfp_mask & __GFP_NOFAIL))
goto nopage;
+ start_memalloc_timer(&m, gfp_mask, order);
+
/*
* Try direct compaction. The first pass is asynchronous. Subsequent
* attempts after direct reclaim are synchronous
@@ -2798,6 +3010,10 @@ retry:
goto got_pg;
/* Check if we should retry the allocation */
+ if (dump_target_pid == current->pid) {
+ printk(KERN_INFO "did_some_progress=%lu\n", did_some_progress);
+ dump_target_pid = 0;
+ }
pages_reclaimed += did_some_progress;
if (should_alloc_retry(gfp_mask, order, did_some_progress,
pages_reclaimed)) {
@@ -2834,6 +3050,7 @@ retry:
nopage:
warn_alloc_failed(gfp_mask, order, NULL);
got_pg:
+ stop_memalloc_timer(&m);
return page;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 1a17bd7..c449371 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2432,6 +2432,8 @@ static inline bool compaction_ready(struct zone *zone, int order)
return watermark_ok;
}
+extern pid_t dump_target_pid;
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
@@ -2533,7 +2535,27 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
if (global_reclaim(sc) &&
!reclaimable && zone_reclaimable(zone))
+ {
+ if (dump_target_pid == current->pid) {
+ unsigned long rec = zone_reclaimable_pages(zone);
+ unsigned long free = zone_page_state(zone, NR_FREE_PAGES);
+ unsigned long min = min_wmark_pages(zone);
+ unsigned long scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+ unsigned long now = jiffies;
+ unsigned long rec2 = zone_page_state_snapshot(zone, NR_ACTIVE_FILE) +
+ zone_page_state_snapshot(zone, NR_INACTIVE_FILE);
+ unsigned long free2 = zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ unsigned long scanned2 = zone_page_state_snapshot(zone, NR_PAGES_SCANNED);
+
+ printk(KERN_INFO "%s zone_reclaimable: reclaim:%lu(%lu,%lu,%ld) free:%lu(%lu,%ld) min:%lu pages_scanned:%lu(%lu,%ld) prio:%d\n",
+ zone->name, rec, now - zone->stat_last_updated[NR_ACTIVE_FILE],
+ now - zone->stat_last_updated[NR_INACTIVE_FILE], rec - rec2,
+ free, now - zone->stat_last_updated[NR_FREE_PAGES], free - free2,
+ min, scanned, now - zone->stat_last_updated[NR_PAGES_SCANNED],
+ scanned - scanned2, sc->priority);
+ }
reclaimable = true;
+ }
}
/*
--
1.8.3.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-11 15:58 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-11 15:58 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz, linux-mm, xfs
On 2015/11/11 7:13, Arkadiusz MiA?kiewicz wrote:
> The usual (repeatable) problem is like this:
>
> full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
Maybe somebody doing GFP_NOIO allocation which XFS driver doing GFP_NOFS
allocation is waiting for is stalling inside memory allocator. I think that
checking tasks which are stalling inside memory allocator would help.
Please try reproducing this problem with a debug printk() patch shown below
applied. This is a patch which I used for debugging silent lockup problem.
When memory allocation got stuck, lines with MemAlloc keyword will be
printed.
---
fs/xfs/kmem.c | 10 ++-
fs/xfs/xfs_buf.c | 3 +-
include/linux/mmzone.h | 1 +
include/linux/vmstat.h | 1 +
mm/page_alloc.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++
mm/vmscan.c | 22 +++++
6 files changed, 249 insertions(+), 5 deletions(-)
diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c
index a7a3a63..535c136 100644
--- a/fs/xfs/kmem.c
+++ b/fs/xfs/kmem.c
@@ -55,8 +55,9 @@ kmem_alloc(size_t size, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
@@ -120,8 +121,9 @@ kmem_zone_alloc(kmem_zone_t *zone, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..16322cb 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -354,7 +354,8 @@ retry:
*/
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
__func__, gfp_mask);
XFS_STATS_INC(xb_page_retries);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 54d74f6..932a6d6 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -527,6 +527,7 @@ struct zone {
ZONE_PADDING(_pad3_)
/* Zone statistics */
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
+ unsigned long stat_last_updated[NR_VM_ZONE_STAT_ITEMS];
} ____cacheline_internodealigned_in_smp;
enum zone_flags {
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 82e7db7..2488925 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -115,6 +115,7 @@ static inline void zone_page_state_add(long x, struct zone *zone,
{
atomic_long_add(x, &zone->vm_stat[item]);
atomic_long_add(x, &vm_stat[item]);
+ zone->stat_last_updated[item] = jiffies;
}
static inline unsigned long global_page_state(enum zone_stat_item item)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 18490f3..35a46b4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -61,6 +61,8 @@
#include <linux/hugetlb.h>
#include <linux/sched/rt.h>
#include <linux/page_owner.h>
+#include <linux/nmi.h>
+#include <linux/kthread.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -2496,6 +2498,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
}
#endif /* CONFIG_COMPACTION */
+pid_t dump_target_pid;
+
/* Perform direct synchronous page reclaim */
static int
__perform_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -2645,6 +2649,208 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_NO_WATERMARKS);
}
+static unsigned long kmallocwd_timeout = 10 * HZ; /* Scan interval. */
+static u8 memalloc_counter_active_index; /* Either 0 or 1. */
+static int memalloc_counter[2]; /* Number of tasks doing memory allocation. */
+
+struct memalloc {
+ struct list_head list; /* Connected to memalloc_list. */
+ struct task_struct *task; /* Iniatilized to current. */
+ unsigned long start; /* Initialized to jiffies. */
+ unsigned int order;
+ gfp_t gfp;
+ u8 index; /* Initialized to memalloc_counter_active_index. */
+ u8 dumped;
+};
+
+static LIST_HEAD(memalloc_list); /* List of "struct memalloc".*/
+static DEFINE_SPINLOCK(memalloc_list_lock); /* Lock for memalloc_list. */
+
+/*
+ * kmallocwd - A kernel thread for monitoring memory allocation stalls.
+ *
+ * @unused: Not used.
+ *
+ * This kernel thread does not terminate.
+ */
+static int kmallocwd(void *unused)
+{
+ struct memalloc *m;
+ struct task_struct *g, *p;
+ unsigned long now;
+ unsigned int sigkill_pending;
+ unsigned int memdie_pending;
+ unsigned int stalling_tasks;
+ u8 index;
+ pid_t pid;
+
+ not_stalling: /* Healty case. */
+ /* Switch active counter and wait for timeout duration. */
+ index = memalloc_counter_active_index;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter_active_index ^= 1;
+ spin_unlock(&memalloc_list_lock);
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ /*
+ * If memory allocations are working, the counter should remain 0
+ * because tasks will be able to call both start_memalloc_timer()
+ * and stop_memalloc_timer() within timeout duration.
+ */
+ if (likely(!memalloc_counter[index]))
+ goto not_stalling;
+ maybe_stalling: /* Maybe something is wrong. Let's check. */
+ now = jiffies;
+ /* Count stalling tasks, dying and victim tasks. */
+ sigkill_pending = 0;
+ memdie_pending = 0;
+ stalling_tasks = 0;
+ pid = 0;
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_after(now - m->start, kmallocwd_timeout))
+ stalling_tasks++;
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ memdie_pending++;
+ if (fatal_signal_pending(p))
+ sigkill_pending++;
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ pr_warn("MemAlloc-Info: %u stalling task, %u dying task, %u victim task.\n",
+ stalling_tasks, sigkill_pending, memdie_pending);
+ /* Report stalling tasks, dying and victim tasks. */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout))
+ continue;
+ p = m->task;
+ pr_warn("MemAlloc: %s(%u) gfp=0x%x order=%u delay=%lu\n",
+ p->comm, p->pid, m->gfp, m->order, now - m->start);
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ u8 type = 0;
+
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ type |= 1;
+ if (fatal_signal_pending(p))
+ type |= 2;
+ if (likely(!type))
+ continue;
+ if (p->state & TASK_UNINTERRUPTIBLE)
+ type |= 4;
+ pr_warn("MemAlloc: %s(%u)%s%s%s\n", p->comm, p->pid,
+ (type & 4) ? " uninterruptible" : "",
+ (type & 2) ? " dying" : "",
+ (type & 1) ? " victim" : "");
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ /*
+ * Show traces of newly reported (or too long) stalling tasks.
+ *
+ * Show traces only once per 256 timeouts because their traces
+ * will likely be the same (e.g. cond_sched() or congestion_wait())
+ * when they are stalling inside __alloc_pages_slowpath().
+ */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout) ||
+ m->dumped++)
+ continue;
+ p = m->task;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ if (!pid)
+ pid = p->pid;
+ }
+ spin_unlock(&memalloc_list_lock);
+ /*
+ * Show traces of dying tasks (including victim tasks).
+ *
+ * Only dying tasks which are in trouble (e.g. blocked at unkillable
+ * locks held by memory allocating tasks) will be repeatedly shown.
+ * Therefore, we need to pay attention to tasks repeatedly shown here.
+ */
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (likely(!fatal_signal_pending(p)))
+ continue;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ show_workqueue_state();
+ if (!dump_target_pid)
+ dump_target_pid = -pid;
+ /* Wait until next timeout duration. */
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ if (memalloc_counter[index])
+ goto maybe_stalling;
+ goto not_stalling;
+ return 0; /* To suppress "no return statement" compiler warning. */
+}
+
+static int __init start_kmallocwd(void)
+{
+ if (kmallocwd_timeout) {
+ struct task_struct *task = kthread_run(kmallocwd, NULL,
+ "kmallocwd");
+ BUG_ON(IS_ERR(task));
+ }
+ return 0;
+}
+late_initcall(start_kmallocwd);
+
+static int __init kmallocwd_config(char *str)
+{
+ if (kstrtoul(str, 10, &kmallocwd_timeout) == 0)
+ kmallocwd_timeout = min(kmallocwd_timeout * HZ,
+ (unsigned long) LONG_MAX);
+ return 0;
+}
+__setup("kmallocwd=", kmallocwd_config);
+
+static void start_memalloc_timer(struct memalloc *m, const gfp_t gfp_mask,
+ const int order)
+{
+ if (!kmallocwd_timeout || m->task)
+ return;
+ m->task = current;
+ m->start = jiffies;
+ m->gfp = gfp_mask;
+ m->order = order;
+ m->dumped = 0;
+ spin_lock(&memalloc_list_lock);
+ m->index = memalloc_counter_active_index;
+ memalloc_counter[m->index]++;
+ list_add_tail(&m->list, &memalloc_list);
+ spin_unlock(&memalloc_list_lock);
+}
+
+static void stop_memalloc_timer(struct memalloc *m)
+{
+ if (!m->task)
+ return;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter[m->index]--;
+ list_del(&m->list);
+ spin_unlock(&memalloc_list_lock);
+}
+
static inline struct page *
__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
struct alloc_context *ac)
@@ -2657,6 +2863,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
enum migrate_mode migration_mode = MIGRATE_ASYNC;
bool deferred_compaction = false;
int contended_compaction = COMPACT_CONTENDED_NONE;
+ struct memalloc m = { .task = NULL };
/*
* In the slowpath, we sanity check order to avoid ever trying to
@@ -2678,6 +2885,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
goto nopage;
retry:
+ if (dump_target_pid == -current->pid)
+ dump_target_pid = -dump_target_pid;
+
if (!(gfp_mask & __GFP_NO_KSWAPD))
wake_all_kswapds(order, ac);
@@ -2740,6 +2950,8 @@ retry:
if (test_thread_flag(TIF_MEMDIE) && !(gfp_mask & __GFP_NOFAIL))
goto nopage;
+ start_memalloc_timer(&m, gfp_mask, order);
+
/*
* Try direct compaction. The first pass is asynchronous. Subsequent
* attempts after direct reclaim are synchronous
@@ -2798,6 +3010,10 @@ retry:
goto got_pg;
/* Check if we should retry the allocation */
+ if (dump_target_pid == current->pid) {
+ printk(KERN_INFO "did_some_progress=%lu\n", did_some_progress);
+ dump_target_pid = 0;
+ }
pages_reclaimed += did_some_progress;
if (should_alloc_retry(gfp_mask, order, did_some_progress,
pages_reclaimed)) {
@@ -2834,6 +3050,7 @@ retry:
nopage:
warn_alloc_failed(gfp_mask, order, NULL);
got_pg:
+ stop_memalloc_timer(&m);
return page;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 1a17bd7..c449371 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2432,6 +2432,8 @@ static inline bool compaction_ready(struct zone *zone, int order)
return watermark_ok;
}
+extern pid_t dump_target_pid;
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
@@ -2533,7 +2535,27 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
if (global_reclaim(sc) &&
!reclaimable && zone_reclaimable(zone))
+ {
+ if (dump_target_pid == current->pid) {
+ unsigned long rec = zone_reclaimable_pages(zone);
+ unsigned long free = zone_page_state(zone, NR_FREE_PAGES);
+ unsigned long min = min_wmark_pages(zone);
+ unsigned long scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+ unsigned long now = jiffies;
+ unsigned long rec2 = zone_page_state_snapshot(zone, NR_ACTIVE_FILE) +
+ zone_page_state_snapshot(zone, NR_INACTIVE_FILE);
+ unsigned long free2 = zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ unsigned long scanned2 = zone_page_state_snapshot(zone, NR_PAGES_SCANNED);
+
+ printk(KERN_INFO "%s zone_reclaimable: reclaim:%lu(%lu,%lu,%ld) free:%lu(%lu,%ld) min:%lu pages_scanned:%lu(%lu,%ld) prio:%d\n",
+ zone->name, rec, now - zone->stat_last_updated[NR_ACTIVE_FILE],
+ now - zone->stat_last_updated[NR_INACTIVE_FILE], rec - rec2,
+ free, now - zone->stat_last_updated[NR_FREE_PAGES], free - free2,
+ min, scanned, now - zone->stat_last_updated[NR_PAGES_SCANNED],
+ scanned - scanned2, sc->priority);
+ }
reclaimable = true;
+ }
}
/*
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-11 15:58 ` Tetsuo Handa
@ 2015-11-11 16:19 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-11 16:19 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> On 2015/11/11 7:13, Arkadiusz Miśkiewicz wrote:
> > The usual (repeatable) problem is like this:
> >
> > full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
>
> Maybe somebody doing GFP_NOIO allocation which XFS driver doing GFP_NOFS
> allocation is waiting for is stalling inside memory allocator. I think that
> checking tasks which are stalling inside memory allocator would help.
>
> Please try reproducing this problem with a debug printk() patch shown below
> applied. This is a patch which I used for debugging silent lockup problem.
> When memory allocation got stuck, lines with MemAlloc keyword will be
> printed.
>
> ---
> fs/xfs/kmem.c | 10 ++-
> fs/xfs/xfs_buf.c | 3 +-
> include/linux/mmzone.h | 1 +
> include/linux/vmstat.h | 1 +
> mm/page_alloc.c | 217
> +++++++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c |
> 22 +++++
> 6 files changed, 249 insertions(+), 5 deletions(-)
This patch is against which tree? (tried 4.1, 4.2 and 4.3)
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-11 16:19 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-11 16:19 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> On 2015/11/11 7:13, Arkadiusz Miśkiewicz wrote:
> > The usual (repeatable) problem is like this:
> >
> > full dmesg: http://sprunge.us/VEiE (more in it then in partial log below)
>
> Maybe somebody doing GFP_NOIO allocation which XFS driver doing GFP_NOFS
> allocation is waiting for is stalling inside memory allocator. I think that
> checking tasks which are stalling inside memory allocator would help.
>
> Please try reproducing this problem with a debug printk() patch shown below
> applied. This is a patch which I used for debugging silent lockup problem.
> When memory allocation got stuck, lines with MemAlloc keyword will be
> printed.
>
> ---
> fs/xfs/kmem.c | 10 ++-
> fs/xfs/xfs_buf.c | 3 +-
> include/linux/mmzone.h | 1 +
> include/linux/vmstat.h | 1 +
> mm/page_alloc.c | 217
> +++++++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c |
> 22 +++++
> 6 files changed, 249 insertions(+), 5 deletions(-)
This patch is against which tree? (tried 4.1, 4.2 and 4.3)
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-11 16:19 ` Arkadiusz Miśkiewicz
@ 2015-11-11 22:19 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-11 22:19 UTC (permalink / raw)
To: arekm; +Cc: linux-mm, xfs
Arkadiusz Mi?kiewicz wrote:
> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
Reposting with one condition corrected.
if (!dump_target_pid) => if (dump_target_pid <= 0)
---
fs/xfs/kmem.c | 10 ++-
fs/xfs/xfs_buf.c | 3 +-
include/linux/mmzone.h | 1 +
include/linux/vmstat.h | 1 +
mm/page_alloc.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++
mm/vmscan.c | 22 +++++
6 files changed, 249 insertions(+), 5 deletions(-)
diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c
index a7a3a63..535c136 100644
--- a/fs/xfs/kmem.c
+++ b/fs/xfs/kmem.c
@@ -55,8 +55,9 @@ kmem_alloc(size_t size, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
@@ -120,8 +121,9 @@ kmem_zone_alloc(kmem_zone_t *zone, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..16322cb 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -354,7 +354,8 @@ retry:
*/
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
__func__, gfp_mask);
XFS_STATS_INC(xb_page_retries);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 54d74f6..932a6d6 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -527,6 +527,7 @@ struct zone {
ZONE_PADDING(_pad3_)
/* Zone statistics */
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
+ unsigned long stat_last_updated[NR_VM_ZONE_STAT_ITEMS];
} ____cacheline_internodealigned_in_smp;
enum zone_flags {
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 82e7db7..2488925 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -115,6 +115,7 @@ static inline void zone_page_state_add(long x, struct zone *zone,
{
atomic_long_add(x, &zone->vm_stat[item]);
atomic_long_add(x, &vm_stat[item]);
+ zone->stat_last_updated[item] = jiffies;
}
static inline unsigned long global_page_state(enum zone_stat_item item)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 18490f3..35a46b4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -61,6 +61,8 @@
#include <linux/hugetlb.h>
#include <linux/sched/rt.h>
#include <linux/page_owner.h>
+#include <linux/nmi.h>
+#include <linux/kthread.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -2496,6 +2498,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
}
#endif /* CONFIG_COMPACTION */
+pid_t dump_target_pid;
+
/* Perform direct synchronous page reclaim */
static int
__perform_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -2645,6 +2649,208 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_NO_WATERMARKS);
}
+static unsigned long kmallocwd_timeout = 10 * HZ; /* Scan interval. */
+static u8 memalloc_counter_active_index; /* Either 0 or 1. */
+static int memalloc_counter[2]; /* Number of tasks doing memory allocation. */
+
+struct memalloc {
+ struct list_head list; /* Connected to memalloc_list. */
+ struct task_struct *task; /* Iniatilized to current. */
+ unsigned long start; /* Initialized to jiffies. */
+ unsigned int order;
+ gfp_t gfp;
+ u8 index; /* Initialized to memalloc_counter_active_index. */
+ u8 dumped;
+};
+
+static LIST_HEAD(memalloc_list); /* List of "struct memalloc".*/
+static DEFINE_SPINLOCK(memalloc_list_lock); /* Lock for memalloc_list. */
+
+/*
+ * kmallocwd - A kernel thread for monitoring memory allocation stalls.
+ *
+ * @unused: Not used.
+ *
+ * This kernel thread does not terminate.
+ */
+static int kmallocwd(void *unused)
+{
+ struct memalloc *m;
+ struct task_struct *g, *p;
+ unsigned long now;
+ unsigned int sigkill_pending;
+ unsigned int memdie_pending;
+ unsigned int stalling_tasks;
+ u8 index;
+ pid_t pid;
+
+ not_stalling: /* Healty case. */
+ /* Switch active counter and wait for timeout duration. */
+ index = memalloc_counter_active_index;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter_active_index ^= 1;
+ spin_unlock(&memalloc_list_lock);
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ /*
+ * If memory allocations are working, the counter should remain 0
+ * because tasks will be able to call both start_memalloc_timer()
+ * and stop_memalloc_timer() within timeout duration.
+ */
+ if (likely(!memalloc_counter[index]))
+ goto not_stalling;
+ maybe_stalling: /* Maybe something is wrong. Let's check. */
+ now = jiffies;
+ /* Count stalling tasks, dying and victim tasks. */
+ sigkill_pending = 0;
+ memdie_pending = 0;
+ stalling_tasks = 0;
+ pid = 0;
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_after(now - m->start, kmallocwd_timeout))
+ stalling_tasks++;
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ memdie_pending++;
+ if (fatal_signal_pending(p))
+ sigkill_pending++;
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ pr_warn("MemAlloc-Info: %u stalling task, %u dying task, %u victim task.\n",
+ stalling_tasks, sigkill_pending, memdie_pending);
+ /* Report stalling tasks, dying and victim tasks. */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout))
+ continue;
+ p = m->task;
+ pr_warn("MemAlloc: %s(%u) gfp=0x%x order=%u delay=%lu\n",
+ p->comm, p->pid, m->gfp, m->order, now - m->start);
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ u8 type = 0;
+
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ type |= 1;
+ if (fatal_signal_pending(p))
+ type |= 2;
+ if (likely(!type))
+ continue;
+ if (p->state & TASK_UNINTERRUPTIBLE)
+ type |= 4;
+ pr_warn("MemAlloc: %s(%u)%s%s%s\n", p->comm, p->pid,
+ (type & 4) ? " uninterruptible" : "",
+ (type & 2) ? " dying" : "",
+ (type & 1) ? " victim" : "");
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ /*
+ * Show traces of newly reported (or too long) stalling tasks.
+ *
+ * Show traces only once per 256 timeouts because their traces
+ * will likely be the same (e.g. cond_sched() or congestion_wait())
+ * when they are stalling inside __alloc_pages_slowpath().
+ */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout) ||
+ m->dumped++)
+ continue;
+ p = m->task;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ if (!pid)
+ pid = p->pid;
+ }
+ spin_unlock(&memalloc_list_lock);
+ /*
+ * Show traces of dying tasks (including victim tasks).
+ *
+ * Only dying tasks which are in trouble (e.g. blocked at unkillable
+ * locks held by memory allocating tasks) will be repeatedly shown.
+ * Therefore, we need to pay attention to tasks repeatedly shown here.
+ */
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (likely(!fatal_signal_pending(p)))
+ continue;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ show_workqueue_state();
+ if (dump_target_pid <= 0)
+ dump_target_pid = -pid;
+ /* Wait until next timeout duration. */
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ if (memalloc_counter[index])
+ goto maybe_stalling;
+ goto not_stalling;
+ return 0; /* To suppress "no return statement" compiler warning. */
+}
+
+static int __init start_kmallocwd(void)
+{
+ if (kmallocwd_timeout) {
+ struct task_struct *task = kthread_run(kmallocwd, NULL,
+ "kmallocwd");
+ BUG_ON(IS_ERR(task));
+ }
+ return 0;
+}
+late_initcall(start_kmallocwd);
+
+static int __init kmallocwd_config(char *str)
+{
+ if (kstrtoul(str, 10, &kmallocwd_timeout) == 0)
+ kmallocwd_timeout = min(kmallocwd_timeout * HZ,
+ (unsigned long) LONG_MAX);
+ return 0;
+}
+__setup("kmallocwd=", kmallocwd_config);
+
+static void start_memalloc_timer(struct memalloc *m, const gfp_t gfp_mask,
+ const int order)
+{
+ if (!kmallocwd_timeout || m->task)
+ return;
+ m->task = current;
+ m->start = jiffies;
+ m->gfp = gfp_mask;
+ m->order = order;
+ m->dumped = 0;
+ spin_lock(&memalloc_list_lock);
+ m->index = memalloc_counter_active_index;
+ memalloc_counter[m->index]++;
+ list_add_tail(&m->list, &memalloc_list);
+ spin_unlock(&memalloc_list_lock);
+}
+
+static void stop_memalloc_timer(struct memalloc *m)
+{
+ if (!m->task)
+ return;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter[m->index]--;
+ list_del(&m->list);
+ spin_unlock(&memalloc_list_lock);
+}
+
static inline struct page *
__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
struct alloc_context *ac)
@@ -2657,6 +2863,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
enum migrate_mode migration_mode = MIGRATE_ASYNC;
bool deferred_compaction = false;
int contended_compaction = COMPACT_CONTENDED_NONE;
+ struct memalloc m = { .task = NULL };
/*
* In the slowpath, we sanity check order to avoid ever trying to
@@ -2678,6 +2885,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
goto nopage;
retry:
+ if (dump_target_pid == -current->pid)
+ dump_target_pid = -dump_target_pid;
+
if (!(gfp_mask & __GFP_NO_KSWAPD))
wake_all_kswapds(order, ac);
@@ -2740,6 +2950,8 @@ retry:
if (test_thread_flag(TIF_MEMDIE) && !(gfp_mask & __GFP_NOFAIL))
goto nopage;
+ start_memalloc_timer(&m, gfp_mask, order);
+
/*
* Try direct compaction. The first pass is asynchronous. Subsequent
* attempts after direct reclaim are synchronous
@@ -2798,6 +3010,10 @@ retry:
goto got_pg;
/* Check if we should retry the allocation */
+ if (dump_target_pid == current->pid) {
+ printk(KERN_INFO "did_some_progress=%lu\n", did_some_progress);
+ dump_target_pid = 0;
+ }
pages_reclaimed += did_some_progress;
if (should_alloc_retry(gfp_mask, order, did_some_progress,
pages_reclaimed)) {
@@ -2834,6 +3050,7 @@ retry:
nopage:
warn_alloc_failed(gfp_mask, order, NULL);
got_pg:
+ stop_memalloc_timer(&m);
return page;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 1a17bd7..c449371 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2432,6 +2432,8 @@ static inline bool compaction_ready(struct zone *zone, int order)
return watermark_ok;
}
+extern pid_t dump_target_pid;
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
@@ -2533,7 +2535,27 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
if (global_reclaim(sc) &&
!reclaimable && zone_reclaimable(zone))
+ {
+ if (dump_target_pid == current->pid) {
+ unsigned long rec = zone_reclaimable_pages(zone);
+ unsigned long free = zone_page_state(zone, NR_FREE_PAGES);
+ unsigned long min = min_wmark_pages(zone);
+ unsigned long scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+ unsigned long now = jiffies;
+ unsigned long rec2 = zone_page_state_snapshot(zone, NR_ACTIVE_FILE) +
+ zone_page_state_snapshot(zone, NR_INACTIVE_FILE);
+ unsigned long free2 = zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ unsigned long scanned2 = zone_page_state_snapshot(zone, NR_PAGES_SCANNED);
+
+ printk(KERN_INFO "%s zone_reclaimable: reclaim:%lu(%lu,%lu,%ld) free:%lu(%lu,%ld) min:%lu pages_scanned:%lu(%lu,%ld) prio:%d\n",
+ zone->name, rec, now - zone->stat_last_updated[NR_ACTIVE_FILE],
+ now - zone->stat_last_updated[NR_INACTIVE_FILE], rec - rec2,
+ free, now - zone->stat_last_updated[NR_FREE_PAGES], free - free2,
+ min, scanned, now - zone->stat_last_updated[NR_PAGES_SCANNED],
+ scanned - scanned2, sc->priority);
+ }
reclaimable = true;
+ }
}
/*
--
1.8.3.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-11 22:19 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-11 22:19 UTC (permalink / raw)
To: arekm; +Cc: linux-mm, xfs
Arkadiusz Mi?kiewicz wrote:
> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
Reposting with one condition corrected.
if (!dump_target_pid) => if (dump_target_pid <= 0)
---
fs/xfs/kmem.c | 10 ++-
fs/xfs/xfs_buf.c | 3 +-
include/linux/mmzone.h | 1 +
include/linux/vmstat.h | 1 +
mm/page_alloc.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++
mm/vmscan.c | 22 +++++
6 files changed, 249 insertions(+), 5 deletions(-)
diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c
index a7a3a63..535c136 100644
--- a/fs/xfs/kmem.c
+++ b/fs/xfs/kmem.c
@@ -55,8 +55,9 @@ kmem_alloc(size_t size, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
@@ -120,8 +121,9 @@ kmem_zone_alloc(kmem_zone_t *zone, xfs_km_flags_t flags)
return ptr;
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
- __func__, lflags);
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
+ __func__, lflags);
congestion_wait(BLK_RW_ASYNC, HZ/50);
} while (1);
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..16322cb 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -354,7 +354,8 @@ retry:
*/
if (!(++retries % 100))
xfs_err(NULL,
- "possible memory allocation deadlock in %s (mode:0x%x)",
+ "%s(%u) possible memory allocation deadlock in %s (mode:0x%x)",
+ current->comm, current->pid,
__func__, gfp_mask);
XFS_STATS_INC(xb_page_retries);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 54d74f6..932a6d6 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -527,6 +527,7 @@ struct zone {
ZONE_PADDING(_pad3_)
/* Zone statistics */
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
+ unsigned long stat_last_updated[NR_VM_ZONE_STAT_ITEMS];
} ____cacheline_internodealigned_in_smp;
enum zone_flags {
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 82e7db7..2488925 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -115,6 +115,7 @@ static inline void zone_page_state_add(long x, struct zone *zone,
{
atomic_long_add(x, &zone->vm_stat[item]);
atomic_long_add(x, &vm_stat[item]);
+ zone->stat_last_updated[item] = jiffies;
}
static inline unsigned long global_page_state(enum zone_stat_item item)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 18490f3..35a46b4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -61,6 +61,8 @@
#include <linux/hugetlb.h>
#include <linux/sched/rt.h>
#include <linux/page_owner.h>
+#include <linux/nmi.h>
+#include <linux/kthread.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -2496,6 +2498,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
}
#endif /* CONFIG_COMPACTION */
+pid_t dump_target_pid;
+
/* Perform direct synchronous page reclaim */
static int
__perform_reclaim(gfp_t gfp_mask, unsigned int order,
@@ -2645,6 +2649,208 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_NO_WATERMARKS);
}
+static unsigned long kmallocwd_timeout = 10 * HZ; /* Scan interval. */
+static u8 memalloc_counter_active_index; /* Either 0 or 1. */
+static int memalloc_counter[2]; /* Number of tasks doing memory allocation. */
+
+struct memalloc {
+ struct list_head list; /* Connected to memalloc_list. */
+ struct task_struct *task; /* Iniatilized to current. */
+ unsigned long start; /* Initialized to jiffies. */
+ unsigned int order;
+ gfp_t gfp;
+ u8 index; /* Initialized to memalloc_counter_active_index. */
+ u8 dumped;
+};
+
+static LIST_HEAD(memalloc_list); /* List of "struct memalloc".*/
+static DEFINE_SPINLOCK(memalloc_list_lock); /* Lock for memalloc_list. */
+
+/*
+ * kmallocwd - A kernel thread for monitoring memory allocation stalls.
+ *
+ * @unused: Not used.
+ *
+ * This kernel thread does not terminate.
+ */
+static int kmallocwd(void *unused)
+{
+ struct memalloc *m;
+ struct task_struct *g, *p;
+ unsigned long now;
+ unsigned int sigkill_pending;
+ unsigned int memdie_pending;
+ unsigned int stalling_tasks;
+ u8 index;
+ pid_t pid;
+
+ not_stalling: /* Healty case. */
+ /* Switch active counter and wait for timeout duration. */
+ index = memalloc_counter_active_index;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter_active_index ^= 1;
+ spin_unlock(&memalloc_list_lock);
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ /*
+ * If memory allocations are working, the counter should remain 0
+ * because tasks will be able to call both start_memalloc_timer()
+ * and stop_memalloc_timer() within timeout duration.
+ */
+ if (likely(!memalloc_counter[index]))
+ goto not_stalling;
+ maybe_stalling: /* Maybe something is wrong. Let's check. */
+ now = jiffies;
+ /* Count stalling tasks, dying and victim tasks. */
+ sigkill_pending = 0;
+ memdie_pending = 0;
+ stalling_tasks = 0;
+ pid = 0;
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_after(now - m->start, kmallocwd_timeout))
+ stalling_tasks++;
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ memdie_pending++;
+ if (fatal_signal_pending(p))
+ sigkill_pending++;
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ pr_warn("MemAlloc-Info: %u stalling task, %u dying task, %u victim task.\n",
+ stalling_tasks, sigkill_pending, memdie_pending);
+ /* Report stalling tasks, dying and victim tasks. */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout))
+ continue;
+ p = m->task;
+ pr_warn("MemAlloc: %s(%u) gfp=0x%x order=%u delay=%lu\n",
+ p->comm, p->pid, m->gfp, m->order, now - m->start);
+ }
+ spin_unlock(&memalloc_list_lock);
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ u8 type = 0;
+
+ if (test_tsk_thread_flag(p, TIF_MEMDIE))
+ type |= 1;
+ if (fatal_signal_pending(p))
+ type |= 2;
+ if (likely(!type))
+ continue;
+ if (p->state & TASK_UNINTERRUPTIBLE)
+ type |= 4;
+ pr_warn("MemAlloc: %s(%u)%s%s%s\n", p->comm, p->pid,
+ (type & 4) ? " uninterruptible" : "",
+ (type & 2) ? " dying" : "",
+ (type & 1) ? " victim" : "");
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ cond_resched();
+ /*
+ * Show traces of newly reported (or too long) stalling tasks.
+ *
+ * Show traces only once per 256 timeouts because their traces
+ * will likely be the same (e.g. cond_sched() or congestion_wait())
+ * when they are stalling inside __alloc_pages_slowpath().
+ */
+ spin_lock(&memalloc_list_lock);
+ list_for_each_entry(m, &memalloc_list, list) {
+ if (time_before(now - m->start, kmallocwd_timeout) ||
+ m->dumped++)
+ continue;
+ p = m->task;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ if (!pid)
+ pid = p->pid;
+ }
+ spin_unlock(&memalloc_list_lock);
+ /*
+ * Show traces of dying tasks (including victim tasks).
+ *
+ * Only dying tasks which are in trouble (e.g. blocked at unkillable
+ * locks held by memory allocating tasks) will be repeatedly shown.
+ * Therefore, we need to pay attention to tasks repeatedly shown here.
+ */
+ preempt_disable();
+ rcu_read_lock();
+ for_each_process_thread(g, p) {
+ if (likely(!fatal_signal_pending(p)))
+ continue;
+ sched_show_task(p);
+ debug_show_held_locks(p);
+ touch_nmi_watchdog();
+ }
+ rcu_read_unlock();
+ preempt_enable();
+ show_workqueue_state();
+ if (dump_target_pid <= 0)
+ dump_target_pid = -pid;
+ /* Wait until next timeout duration. */
+ schedule_timeout_interruptible(kmallocwd_timeout);
+ if (memalloc_counter[index])
+ goto maybe_stalling;
+ goto not_stalling;
+ return 0; /* To suppress "no return statement" compiler warning. */
+}
+
+static int __init start_kmallocwd(void)
+{
+ if (kmallocwd_timeout) {
+ struct task_struct *task = kthread_run(kmallocwd, NULL,
+ "kmallocwd");
+ BUG_ON(IS_ERR(task));
+ }
+ return 0;
+}
+late_initcall(start_kmallocwd);
+
+static int __init kmallocwd_config(char *str)
+{
+ if (kstrtoul(str, 10, &kmallocwd_timeout) == 0)
+ kmallocwd_timeout = min(kmallocwd_timeout * HZ,
+ (unsigned long) LONG_MAX);
+ return 0;
+}
+__setup("kmallocwd=", kmallocwd_config);
+
+static void start_memalloc_timer(struct memalloc *m, const gfp_t gfp_mask,
+ const int order)
+{
+ if (!kmallocwd_timeout || m->task)
+ return;
+ m->task = current;
+ m->start = jiffies;
+ m->gfp = gfp_mask;
+ m->order = order;
+ m->dumped = 0;
+ spin_lock(&memalloc_list_lock);
+ m->index = memalloc_counter_active_index;
+ memalloc_counter[m->index]++;
+ list_add_tail(&m->list, &memalloc_list);
+ spin_unlock(&memalloc_list_lock);
+}
+
+static void stop_memalloc_timer(struct memalloc *m)
+{
+ if (!m->task)
+ return;
+ spin_lock(&memalloc_list_lock);
+ memalloc_counter[m->index]--;
+ list_del(&m->list);
+ spin_unlock(&memalloc_list_lock);
+}
+
static inline struct page *
__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
struct alloc_context *ac)
@@ -2657,6 +2863,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
enum migrate_mode migration_mode = MIGRATE_ASYNC;
bool deferred_compaction = false;
int contended_compaction = COMPACT_CONTENDED_NONE;
+ struct memalloc m = { .task = NULL };
/*
* In the slowpath, we sanity check order to avoid ever trying to
@@ -2678,6 +2885,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
goto nopage;
retry:
+ if (dump_target_pid == -current->pid)
+ dump_target_pid = -dump_target_pid;
+
if (!(gfp_mask & __GFP_NO_KSWAPD))
wake_all_kswapds(order, ac);
@@ -2740,6 +2950,8 @@ retry:
if (test_thread_flag(TIF_MEMDIE) && !(gfp_mask & __GFP_NOFAIL))
goto nopage;
+ start_memalloc_timer(&m, gfp_mask, order);
+
/*
* Try direct compaction. The first pass is asynchronous. Subsequent
* attempts after direct reclaim are synchronous
@@ -2798,6 +3010,10 @@ retry:
goto got_pg;
/* Check if we should retry the allocation */
+ if (dump_target_pid == current->pid) {
+ printk(KERN_INFO "did_some_progress=%lu\n", did_some_progress);
+ dump_target_pid = 0;
+ }
pages_reclaimed += did_some_progress;
if (should_alloc_retry(gfp_mask, order, did_some_progress,
pages_reclaimed)) {
@@ -2834,6 +3050,7 @@ retry:
nopage:
warn_alloc_failed(gfp_mask, order, NULL);
got_pg:
+ stop_memalloc_timer(&m);
return page;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 1a17bd7..c449371 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2432,6 +2432,8 @@ static inline bool compaction_ready(struct zone *zone, int order)
return watermark_ok;
}
+extern pid_t dump_target_pid;
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
@@ -2533,7 +2535,27 @@ static bool shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
if (global_reclaim(sc) &&
!reclaimable && zone_reclaimable(zone))
+ {
+ if (dump_target_pid == current->pid) {
+ unsigned long rec = zone_reclaimable_pages(zone);
+ unsigned long free = zone_page_state(zone, NR_FREE_PAGES);
+ unsigned long min = min_wmark_pages(zone);
+ unsigned long scanned = zone_page_state(zone, NR_PAGES_SCANNED);
+ unsigned long now = jiffies;
+ unsigned long rec2 = zone_page_state_snapshot(zone, NR_ACTIVE_FILE) +
+ zone_page_state_snapshot(zone, NR_INACTIVE_FILE);
+ unsigned long free2 = zone_page_state_snapshot(zone, NR_FREE_PAGES);
+ unsigned long scanned2 = zone_page_state_snapshot(zone, NR_PAGES_SCANNED);
+
+ printk(KERN_INFO "%s zone_reclaimable: reclaim:%lu(%lu,%lu,%ld) free:%lu(%lu,%ld) min:%lu pages_scanned:%lu(%lu,%ld) prio:%d\n",
+ zone->name, rec, now - zone->stat_last_updated[NR_ACTIVE_FILE],
+ now - zone->stat_last_updated[NR_INACTIVE_FILE], rec - rec2,
+ free, now - zone->stat_last_updated[NR_FREE_PAGES], free - free2,
+ min, scanned, now - zone->stat_last_updated[NR_PAGES_SCANNED],
+ scanned - scanned2, sc->priority);
+ }
reclaimable = true;
+ }
}
/*
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-11 22:19 ` Tetsuo Handa
@ 2015-11-12 6:06 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-12 6:06 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Mi?kiewicz wrote:
> > This patch is against which tree? (tried 4.1, 4.2 and 4.3)
>
> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> Reposting with one condition corrected.
Here is log:
http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
Uncompresses is 1.4MB, so not posting here.
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-12 6:06 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-12 6:06 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Mi?kiewicz wrote:
> > This patch is against which tree? (tried 4.1, 4.2 and 4.3)
>
> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> Reposting with one condition corrected.
Here is log:
http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
Uncompresses is 1.4MB, so not posting here.
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-12 6:06 ` Arkadiusz Miśkiewicz
@ 2015-11-12 14:12 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-12 14:12 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz; +Cc: linux-mm, xfs
On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> On Wednesday 11 of November 2015, Tetsuo Handa wrote:
>> Arkadiusz Mi?kiewicz wrote:
>>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
>>
>> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
>> Reposting with one condition corrected.
>
> Here is log:
>
> http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
>
> Uncompresses is 1.4MB, so not posting here.
>
Thank you for the log. The result is unexpected for me.
What I feel strange is that free: remained below min: level.
While GFP_ATOMIC allocations can access memory reserves, I think that
these free: values are too small. Memory allocated by GFP_ATOMIC should
be released shortly, or any __GFP_WAIT allocations would stall for long.
[ 8633.753528] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:59008kB inactive_anon:75240kB active_file:14712kB inactive_file:3256960kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:20kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697520kB slab_unreclaimable:274524kB kernel_stack:2000kB
pagetables:2088kB unstable:0kB bounce:0kB free_pcp:1072kB local_pcp:672kB free_cma:128kB writeback_tmp:0kB pages_scanned:176 all_unreclaimable? no
[ 8678.467783] Node 0 Normal free:228kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3244288kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:4kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194712kB kernel_stack:1968kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:796kB local_pcp:64kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9460.400303] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9462.401840] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9464.403368] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9466.404900] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9468.406432] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9470.407964] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9472.409498] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9474.411031] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9476.412561] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9478.414094] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9480.415627] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9482.417161] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9484.418691] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9486.420224] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9488.421755] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9490.423290] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9492.424822] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9494.426353] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9496.427886] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9498.429419] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9500.430955] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9502.432503] Node 0 Normal free:40kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3240344kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194896kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9504.434015] Node 0 Normal free:404kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3238768kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194928kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9506.435566] Node 0 Normal free:280kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3237192kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194960kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:68kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9508.437089] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3235616kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194972kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9510.438640] Node 0 Normal free:200kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3234040kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194980kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9512.440155] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3232464kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194984kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9514.441700] Node 0 Normal free:616kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3230888kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194996kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9516.443219] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3229312kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:195040kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9518.444745] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3227736kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697160kB slab_unreclaimable:195076kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9520.446278] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3226164kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697160kB slab_unreclaimable:195104kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9522.447819] Node 0 Normal free:412kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3224596kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697156kB slab_unreclaimable:195132kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:68kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9524.449349] Node 0 Normal free:1544kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3223028kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697088kB slab_unreclaimable:195152kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9526.450875] Node 0 Normal free:404kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3221460kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195196kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9528.452403] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3219892kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195228kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9530.453942] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3218324kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195248kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9532.455470] Node 0 Normal free:348kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3216756kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697012kB slab_unreclaimable:195272kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:88kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9534.457005] Node 0 Normal free:1268kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3215188kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697012kB slab_unreclaimable:195300kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9536.458545] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3213620kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697008kB slab_unreclaimable:195316kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9538.460069] Node 0 Normal free:0kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3212052kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195352kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9540.461626] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3210484kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195356kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9542.463166] Node 0 Normal free:1696kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3208924kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465161] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465338] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465498] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465662] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465825] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465984] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.466143] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
There is no OOM victims thus ALLOC_NO_WATERMARKS is not by TIF_MEMDIE.
There are many stalls which lasted for finite period. Sometimes the system
slowed down, but it was not a livelock.
[ 1065.668820] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1075.676226] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1085.683607] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1095.690955] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1105.698273] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1115.705598] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1125.712926] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1145.727590] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1155.734973] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1165.742333] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1175.749707] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1185.757076] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1195.764440] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1205.771807] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1215.779177] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1235.793909] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1245.801276] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1255.808647] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1275.823377] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1285.830754] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1295.838130] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1305.845492] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1315.852860] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1325.860228] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1335.867595] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1345.874966] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1365.889706] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1375.897073] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1385.904443] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1405.919177] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1415.926283] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1425.933051] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1435.939906] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1445.947270] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1455.954635] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1465.961996] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1475.969365] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1485.976728] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1495.984092] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1505.991462] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1515.998823] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1526.006189] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1536.013556] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1556.028246] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1576.042951] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1586.050273] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1596.057617] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1606.064958] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1616.072324] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1626.079639] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1636.086982] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 1656.101667] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1666.109005] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1706.138381] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1716.145730] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1726.153060] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1736.160403] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1746.167741] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1756.175086] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1766.182427] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1776.189773] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1786.197115] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1796.204463] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1806.211968] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1816.219514] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1826.227075] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1856.249436] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1866.256788] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1876.264140] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1886.271494] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1906.286192] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1916.293546] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1936.308288] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1946.315699] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1966.330503] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1986.345304] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2006.360045] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2016.367414] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2026.374783] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2036.382152] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2046.389518] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2076.411626] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2086.419002] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2096.426372] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2106.433740] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2116.441108] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2126.448478] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2156.486445] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2176.501708] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2196.516972] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2206.524607] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2226.539866] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2236.547503] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2246.555129] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2276.578029] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2286.585663] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2306.600929] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2316.608562] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2326.616191] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2346.631457] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2356.639093] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2366.646733] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2376.654360] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2426.692537] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2436.700160] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2486.738327] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2496.745962] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2516.761238] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2526.768860] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2536.776489] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2546.784126] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2566.799391] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2576.807024] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2586.814659] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2626.845195] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2636.852824] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2706.907138] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2726.922671] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2736.930441] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2746.938204] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2826.999851] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3067.183816] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3087.199151] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3477.497792] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3567.566694] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3577.574349] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3587.582008] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3597.589662] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3607.597314] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 3617.604974] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3637.620285] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3647.627939] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3717.681530] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3797.742794] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4238.079941] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4338.156578] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 4468.256201] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4558.325170] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4618.371150] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4788.501423] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4828.532079] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4858.555065] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4898.585719] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5058.711672] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5469.025717] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5559.094621] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5649.163542] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5669.178855] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5679.186511] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5709.209479] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5759.247766] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5939.385591] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5989.423875] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6009.439191] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6019.446849] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6029.454508] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6079.492787] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6089.500450] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6099.508106] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6109.515761] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6119.523421] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6129.531076] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6199.584674] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6209.592332] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6229.607647] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6239.615305] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6249.622965] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6269.638288] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6559.860274] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6739.998076] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 8401.269303] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 8411.276966] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8421.284631] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8431.292299] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8441.299951] MemAlloc-Info: 9 stalling task, 0 dying task, 0 victim task.
[ 8451.307616] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8461.315273] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8481.330599] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8491.338257] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8501.345922] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8511.353584] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8521.361249] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8531.368911] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8541.376566] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8551.384226] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8561.391891] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8571.399551] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8581.407215] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8591.414878] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8601.422538] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8611.430199] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 8621.437860] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8631.445524] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8641.453189] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8651.460845] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8661.468509] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8671.476177] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 9462.081427] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9472.089089] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9482.096751] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9492.104413] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9502.112092] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9512.119764] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9522.127417] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9532.135091] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9542.142743] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9552.150406] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9562.158067] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9572.165732] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9612.196377] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 9622.204042] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9632.211698] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 9642.219358] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9652.227029] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
vmstat_update() and submit_flushes() remained pending for about 110 seconds.
If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
reported as MemAlloc: traces, but no such lines are recorded. I don't know why
xlog_cil_push_work() did not call schedule() for so long. Anyway, applying
http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should solve
vmstat_update() part.
[ 8491.338279] Showing busy workqueues and worker pools:
[ 8491.338281] workqueue events: flags=0x0
[ 8491.338283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256
[ 8491.338287] pending: vmstat_update, e1000_watchdog_task [e1000e]
[ 8491.338310] workqueue md: flags=0x8
[ 8491.338312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8491.338315] pending: submit_flushes [md_mod]
[ 8491.338333] workqueue xfs-cil/md3: flags=0xc
[ 8491.338334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8491.338336] in-flight: 8895:xlog_cil_push_work [xfs] BAR(576)
[ 8491.338368] pool 0: cpus=0 node=0 flags=0x0 nice=0 workers=3 manager: 12328 idle: 9632 8646
[ 8491.338372] pool 1: cpus=0 node=0 flags=0x0 nice=-20 workers=3 manager: 12327 idle: 4592 11916
[ 8491.338377] pool 4: cpus=2 node=0 flags=0x0 nice=0 workers=2 manager: 11509
(...snipped...)
[ 8601.422538] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8601.422542] MemAlloc: kthreadd(2) gfp=0x2000d0 order=2 delay=61314
[ 8601.422544] MemAlloc: ssh(9797) gfp=0xd0 order=0 delay=60227
[ 8601.422546] MemAlloc: cp(10054) gfp=0x2052d0 order=3 delay=57397
[ 8601.422547] MemAlloc: irqbalance(703) gfp=0x280da order=0 delay=54234
[ 8601.422557] Showing busy workqueues and worker pools:
[ 8601.422559] workqueue events: flags=0x0
[ 8601.422561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=5/256
[ 8601.422566] pending: vmstat_update, e1000_watchdog_task [e1000e], vmpressure_work_fn, kernfs_notify_workfn, key_garbage_collector
[ 8601.422597] workqueue md: flags=0x8
[ 8601.422598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8601.422601] pending: submit_flushes [md_mod]
[ 8601.422619] workqueue xfs-cil/md3: flags=0xc
[ 8601.422621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8601.422623] in-flight: 8895:xlog_cil_push_work [xfs] BAR(4592) BAR(576)
[ 8601.422653] workqueue xfs-log/md3: flags=0x1c
[ 8601.422655] pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256
[ 8601.422658] in-flight: 4592:xfs_log_worker [xfs]
[ 8601.422680] pool 0: cpus=0 node=0 flags=0x0 nice=0 workers=3 manager: 12328 idle: 9632 8646
[ 8601.422684] pool 1: cpus=0 node=0 flags=0x0 nice=-20 workers=3 manager: 12327 idle: 11916
[ 8601.422688] pool 4: cpus=2 node=0 flags=0x0 nice=0 workers=2 manager: 11509
Well, what steps should we try next for isolating the problem?
Swap is not used at all. Turning off swap might help.
[ 8633.753574] Free swap = 117220800kB
[ 8633.753576] Total swap = 117220820kB
Turning off perf might also help.
[ 5001.394085] perf interrupt took too long (2505 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-12 14:12 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-12 14:12 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz; +Cc: linux-mm, xfs
On 2015/11/12 15:06, Arkadiusz MiA?kiewicz wrote:
> On Wednesday 11 of November 2015, Tetsuo Handa wrote:
>> Arkadiusz Mi?kiewicz wrote:
>>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
>>
>> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
>> Reposting with one condition corrected.
>
> Here is log:
>
> http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
>
> Uncompresses is 1.4MB, so not posting here.
>
Thank you for the log. The result is unexpected for me.
What I feel strange is that free: remained below min: level.
While GFP_ATOMIC allocations can access memory reserves, I think that
these free: values are too small. Memory allocated by GFP_ATOMIC should
be released shortly, or any __GFP_WAIT allocations would stall for long.
[ 8633.753528] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:59008kB inactive_anon:75240kB active_file:14712kB inactive_file:3256960kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:20kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697520kB slab_unreclaimable:274524kB kernel_stack:2000kB
pagetables:2088kB unstable:0kB bounce:0kB free_pcp:1072kB local_pcp:672kB free_cma:128kB writeback_tmp:0kB pages_scanned:176 all_unreclaimable? no
[ 8678.467783] Node 0 Normal free:228kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3244288kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:4kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194712kB kernel_stack:1968kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:796kB local_pcp:64kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9460.400303] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9462.401840] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9464.403368] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9466.404900] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9468.406432] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9470.407964] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9472.409498] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9474.411031] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9476.412561] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9478.414094] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9480.415627] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9482.417161] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9484.418691] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9486.420224] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9488.421755] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9490.423290] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9492.424822] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9494.426353] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9496.427886] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9498.429419] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9500.430955] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3241920kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697376kB slab_unreclaimable:194860kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:584kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9502.432503] Node 0 Normal free:40kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3240344kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194896kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9504.434015] Node 0 Normal free:404kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3238768kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194928kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9506.435566] Node 0 Normal free:280kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3237192kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194960kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:68kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9508.437089] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3235616kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697304kB slab_unreclaimable:194972kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9510.438640] Node 0 Normal free:200kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3234040kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194980kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 9512.440155] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3232464kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194984kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9514.441700] Node 0 Normal free:616kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3230888kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:194996kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9516.443219] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3229312kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697232kB slab_unreclaimable:195040kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9518.444745] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3227736kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697160kB slab_unreclaimable:195076kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
[ 9520.446278] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3226164kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697160kB slab_unreclaimable:195104kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9522.447819] Node 0 Normal free:412kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3224596kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697156kB slab_unreclaimable:195132kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:68kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9524.449349] Node 0 Normal free:1544kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3223028kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697088kB slab_unreclaimable:195152kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9526.450875] Node 0 Normal free:404kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3221460kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195196kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9528.452403] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3219892kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195228kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9530.453942] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3218324kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697080kB slab_unreclaimable:195248kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9532.455470] Node 0 Normal free:348kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3216756kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697012kB slab_unreclaimable:195272kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:88kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9534.457005] Node 0 Normal free:1268kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3215188kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697012kB slab_unreclaimable:195300kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9536.458545] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3213620kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:697008kB slab_unreclaimable:195316kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9538.460069] Node 0 Normal free:0kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3212052kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195352kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9540.461626] Node 0 Normal free:128kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3210484kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195356kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
[ 9542.463166] Node 0 Normal free:1696kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3208924kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696952kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465161] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465338] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465498] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465662] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465825] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.465984] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
[ 9544.466143] Node 0 Normal free:2756kB min:7104kB low:8880kB high:10656kB active_anon:58992kB inactive_anon:75240kB active_file:14732kB inactive_file:3207364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:5242880kB managed:5109980kB mlocked:0kB dirty:0kB writeback:0kB mapped:7368kB shmem:0kB slab_reclaimable:696948kB slab_unreclaimable:195348kB kernel_stack:1920kB
pagetables:2084kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:112kB free_cma:128kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
There is no OOM victims thus ALLOC_NO_WATERMARKS is not by TIF_MEMDIE.
There are many stalls which lasted for finite period. Sometimes the system
slowed down, but it was not a livelock.
[ 1065.668820] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1075.676226] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1085.683607] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1095.690955] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1105.698273] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1115.705598] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1125.712926] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1145.727590] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1155.734973] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1165.742333] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1175.749707] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1185.757076] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1195.764440] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1205.771807] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1215.779177] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1235.793909] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1245.801276] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1255.808647] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1275.823377] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1285.830754] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1295.838130] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1305.845492] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1315.852860] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1325.860228] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1335.867595] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1345.874966] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1365.889706] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1375.897073] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1385.904443] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1405.919177] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1415.926283] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1425.933051] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1435.939906] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1445.947270] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1455.954635] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1465.961996] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1475.969365] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1485.976728] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1495.984092] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1505.991462] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1515.998823] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1526.006189] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1536.013556] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1556.028246] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1576.042951] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 1586.050273] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1596.057617] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1606.064958] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1616.072324] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1626.079639] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1636.086982] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 1656.101667] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1666.109005] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1706.138381] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1716.145730] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 1726.153060] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1736.160403] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1746.167741] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1756.175086] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1766.182427] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1776.189773] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1786.197115] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1796.204463] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1806.211968] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1816.219514] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1826.227075] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 1856.249436] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1866.256788] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1876.264140] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1886.271494] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 1906.286192] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1916.293546] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1936.308288] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1946.315699] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 1966.330503] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 1986.345304] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2006.360045] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2016.367414] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2026.374783] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2036.382152] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2046.389518] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2076.411626] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2086.419002] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2096.426372] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2106.433740] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2116.441108] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2126.448478] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2156.486445] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2176.501708] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2196.516972] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2206.524607] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2226.539866] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2236.547503] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2246.555129] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2276.578029] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2286.585663] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2306.600929] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2316.608562] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2326.616191] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2346.631457] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2356.639093] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2366.646733] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2376.654360] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2426.692537] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 2436.700160] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2486.738327] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2496.745962] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 2516.761238] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2526.768860] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2536.776489] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2546.784126] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 2566.799391] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2576.807024] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2586.814659] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2626.845195] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2636.852824] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2706.907138] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2726.922671] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2736.930441] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 2746.938204] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 2826.999851] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3067.183816] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3087.199151] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3477.497792] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3567.566694] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3577.574349] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3587.582008] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3597.589662] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3607.597314] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 3617.604974] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 3637.620285] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3647.627939] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3717.681530] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 3797.742794] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4238.079941] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4338.156578] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 4468.256201] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4558.325170] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4618.371150] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4788.501423] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4828.532079] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4858.555065] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 4898.585719] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5058.711672] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5469.025717] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5559.094621] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5649.163542] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5669.178855] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5679.186511] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5709.209479] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5759.247766] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5939.385591] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 5989.423875] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6009.439191] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6019.446849] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6029.454508] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6079.492787] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6089.500450] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6099.508106] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6109.515761] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 6119.523421] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6129.531076] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6199.584674] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6209.592332] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6229.607647] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6239.615305] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6249.622965] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6269.638288] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6559.860274] MemAlloc-Info: 1 stalling task, 0 dying task, 0 victim task.
[ 6739.998076] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 8401.269303] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 8411.276966] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8421.284631] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8431.292299] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8441.299951] MemAlloc-Info: 9 stalling task, 0 dying task, 0 victim task.
[ 8451.307616] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8461.315273] MemAlloc-Info: 7 stalling task, 0 dying task, 0 victim task.
[ 8481.330599] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8491.338257] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8501.345922] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 8511.353584] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8521.361249] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8531.368911] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8541.376566] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8551.384226] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8561.391891] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8571.399551] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8581.407215] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8591.414878] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8601.422538] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8611.430199] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
[ 8621.437860] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8631.445524] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 8641.453189] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8651.460845] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8661.468509] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 8671.476177] MemAlloc-Info: 8 stalling task, 0 dying task, 0 victim task.
[ 9462.081427] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9472.089089] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9482.096751] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9492.104413] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9502.112092] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9512.119764] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9522.127417] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9532.135091] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9542.142743] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9552.150406] MemAlloc-Info: 14 stalling task, 0 dying task, 0 victim task.
[ 9562.158067] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9572.165732] MemAlloc-Info: 15 stalling task, 0 dying task, 0 victim task.
[ 9612.196377] MemAlloc-Info: 2 stalling task, 0 dying task, 0 victim task.
[ 9622.204042] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9632.211698] MemAlloc-Info: 6 stalling task, 0 dying task, 0 victim task.
[ 9642.219358] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9652.227029] MemAlloc-Info: 5 stalling task, 0 dying task, 0 victim task.
[ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
vmstat_update() and submit_flushes() remained pending for about 110 seconds.
If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
reported as MemAlloc: traces, but no such lines are recorded. I don't know why
xlog_cil_push_work() did not call schedule() for so long. Anyway, applying
http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should solve
vmstat_update() part.
[ 8491.338279] Showing busy workqueues and worker pools:
[ 8491.338281] workqueue events: flags=0x0
[ 8491.338283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256
[ 8491.338287] pending: vmstat_update, e1000_watchdog_task [e1000e]
[ 8491.338310] workqueue md: flags=0x8
[ 8491.338312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8491.338315] pending: submit_flushes [md_mod]
[ 8491.338333] workqueue xfs-cil/md3: flags=0xc
[ 8491.338334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8491.338336] in-flight: 8895:xlog_cil_push_work [xfs] BAR(576)
[ 8491.338368] pool 0: cpus=0 node=0 flags=0x0 nice=0 workers=3 manager: 12328 idle: 9632 8646
[ 8491.338372] pool 1: cpus=0 node=0 flags=0x0 nice=-20 workers=3 manager: 12327 idle: 4592 11916
[ 8491.338377] pool 4: cpus=2 node=0 flags=0x0 nice=0 workers=2 manager: 11509
(...snipped...)
[ 8601.422538] MemAlloc-Info: 4 stalling task, 0 dying task, 0 victim task.
[ 8601.422542] MemAlloc: kthreadd(2) gfp=0x2000d0 order=2 delay=61314
[ 8601.422544] MemAlloc: ssh(9797) gfp=0xd0 order=0 delay=60227
[ 8601.422546] MemAlloc: cp(10054) gfp=0x2052d0 order=3 delay=57397
[ 8601.422547] MemAlloc: irqbalance(703) gfp=0x280da order=0 delay=54234
[ 8601.422557] Showing busy workqueues and worker pools:
[ 8601.422559] workqueue events: flags=0x0
[ 8601.422561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=5/256
[ 8601.422566] pending: vmstat_update, e1000_watchdog_task [e1000e], vmpressure_work_fn, kernfs_notify_workfn, key_garbage_collector
[ 8601.422597] workqueue md: flags=0x8
[ 8601.422598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8601.422601] pending: submit_flushes [md_mod]
[ 8601.422619] workqueue xfs-cil/md3: flags=0xc
[ 8601.422621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 8601.422623] in-flight: 8895:xlog_cil_push_work [xfs] BAR(4592) BAR(576)
[ 8601.422653] workqueue xfs-log/md3: flags=0x1c
[ 8601.422655] pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256
[ 8601.422658] in-flight: 4592:xfs_log_worker [xfs]
[ 8601.422680] pool 0: cpus=0 node=0 flags=0x0 nice=0 workers=3 manager: 12328 idle: 9632 8646
[ 8601.422684] pool 1: cpus=0 node=0 flags=0x0 nice=-20 workers=3 manager: 12327 idle: 11916
[ 8601.422688] pool 4: cpus=2 node=0 flags=0x0 nice=0 workers=2 manager: 11509
Well, what steps should we try next for isolating the problem?
Swap is not used at all. Turning off swap might help.
[ 8633.753574] Free swap = 117220800kB
[ 8633.753576] Total swap = 117220820kB
Turning off perf might also help.
[ 5001.394085] perf interrupt took too long (2505 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-12 14:12 ` Tetsuo Handa
@ 2015-11-12 20:06 ` Dave Chinner
-1 siblings, 0 replies; 34+ messages in thread
From: Dave Chinner @ 2015-11-12 20:06 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Thu, Nov 12, 2015 at 11:12:20PM +0900, Tetsuo Handa wrote:
> On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> >On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> >>Arkadiusz Mi?kiewicz wrote:
> >>>This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> >>
> >>Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> >>Reposting with one condition corrected.
> >
> >Here is log:
> >
> >http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> >
> >Uncompresses is 1.4MB, so not posting here.
> >
> Thank you for the log. The result is unexpected for me.
>
> What I feel strange is that free: remained below min: level.
> While GFP_ATOMIC allocations can access memory reserves, I think that
> these free: values are too small. Memory allocated by GFP_ATOMIC should
> be released shortly, or any __GFP_WAIT allocations would stall for long.
>
> [ 8633.753528] Node 0 Normal free:128kB min:7104kB low:8880kB
> high:10656kB active_anon:59008kB inactive_anon:75240kB
> active_file:14712kB inactive_file:3256960kB unevictable:0kB
....
> isolated(anon):0kB isolated(file):0kB present:5242880kB
> managed:5109980kB mlocked:0kB dirty:20kB writeback:0kB mapped:7368kB
....
> pages_scanned:176 all_unreclaimable? no
So: we have 3.2GB (800,000 pages) of immediately reclaimable page
cache in this zone (inactive_file) and a GFP_ATOMIC allocation
context so we can only really reclaim clean page cache pages
reliably.
So why have we only scanned *176* pages* during reclaim? On other
OOM reports in this trace it's as low as 12. Either that stat is
completely wrong, or we're not doing sufficient page LRU reclaim
scanning....
> [ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
>
> vmstat_update() and submit_flushes() remained pending for about 110 seconds.
> If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
> reported as MemAlloc: traces, but no such lines are recorded. I don't know why
> xlog_cil_push_work() did not call schedule() for so long.
I'd say it is repeatedly waiting for IO completion on log buffers to
write out the checkpoint. It's making progress, just if it's taking
multiple second per journal IO it will take a long time to write a
checkpoint. All the other blocked tasks in XFS inode reclaim are
either waiting directly on IO completion or waiting for the log to
complete a flush, so this really just looks like an overloaded IO
subsystem to me....
> Well, what steps should we try next for isolating the problem?
I think there's plenty that needs to be explained from this
information. We don't have a memory allocation livelock - slow
progress is being made on reclaiming inodes, but we
clearly have a zone imbalance and direct page reclaim is not freeing
clean inactive pages when there are huge numbers of them available
for reclaim. Somebody who understands the page reclaim code needs to
look closely at the code to explain this behaviour now, then we'll
know what information needs to be gathered next...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-12 20:06 ` Dave Chinner
0 siblings, 0 replies; 34+ messages in thread
From: Dave Chinner @ 2015-11-12 20:06 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: Arkadiusz Miśkiewicz, linux-mm, xfs
On Thu, Nov 12, 2015 at 11:12:20PM +0900, Tetsuo Handa wrote:
> On 2015/11/12 15:06, Arkadiusz MiA?kiewicz wrote:
> >On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> >>Arkadiusz Mi?kiewicz wrote:
> >>>This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> >>
> >>Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> >>Reposting with one condition corrected.
> >
> >Here is log:
> >
> >http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> >
> >Uncompresses is 1.4MB, so not posting here.
> >
> Thank you for the log. The result is unexpected for me.
>
> What I feel strange is that free: remained below min: level.
> While GFP_ATOMIC allocations can access memory reserves, I think that
> these free: values are too small. Memory allocated by GFP_ATOMIC should
> be released shortly, or any __GFP_WAIT allocations would stall for long.
>
> [ 8633.753528] Node 0 Normal free:128kB min:7104kB low:8880kB
> high:10656kB active_anon:59008kB inactive_anon:75240kB
> active_file:14712kB inactive_file:3256960kB unevictable:0kB
....
> isolated(anon):0kB isolated(file):0kB present:5242880kB
> managed:5109980kB mlocked:0kB dirty:20kB writeback:0kB mapped:7368kB
....
> pages_scanned:176 all_unreclaimable? no
So: we have 3.2GB (800,000 pages) of immediately reclaimable page
cache in this zone (inactive_file) and a GFP_ATOMIC allocation
context so we can only really reclaim clean page cache pages
reliably.
So why have we only scanned *176* pages* during reclaim? On other
OOM reports in this trace it's as low as 12. Either that stat is
completely wrong, or we're not doing sufficient page LRU reclaim
scanning....
> [ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
>
> vmstat_update() and submit_flushes() remained pending for about 110 seconds.
> If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
> reported as MemAlloc: traces, but no such lines are recorded. I don't know why
> xlog_cil_push_work() did not call schedule() for so long.
I'd say it is repeatedly waiting for IO completion on log buffers to
write out the checkpoint. It's making progress, just if it's taking
multiple second per journal IO it will take a long time to write a
checkpoint. All the other blocked tasks in XFS inode reclaim are
either waiting directly on IO completion or waiting for the log to
complete a flush, so this really just looks like an overloaded IO
subsystem to me....
> Well, what steps should we try next for isolating the problem?
I think there's plenty that needs to be explained from this
information. We don't have a memory allocation livelock - slow
progress is being made on reclaiming inodes, but we
clearly have a zone imbalance and direct page reclaim is not freeing
clean inactive pages when there are huge numbers of them available
for reclaim. Somebody who understands the page reclaim code needs to
look closely at the code to explain this behaviour now, then we'll
know what information needs to be gathered next...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-12 14:12 ` Tetsuo Handa
@ 2015-11-12 21:28 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-12 21:28 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Thursday 12 of November 2015, Tetsuo Handa wrote:
> On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> > On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> >> Arkadiusz Mi?kiewicz wrote:
> >>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> >>
> >> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> >> Reposting with one condition corrected.
> >
> > Here is log:
> >
> > http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> >
> > Uncompresses is 1.4MB, so not posting here.
>
> Thank you for the log. The result is unexpected for me.
[...]
>
> vmstat_update() and submit_flushes() remained pending for about 110
> seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS allocation,
> it should be reported as MemAlloc: traces, but no such lines are recorded.
> I don't know why xlog_cil_push_work() did not call schedule() for so long.
> Anyway, applying
> http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should solve
> vmstat_update() part.
To apply that patch on top of 4.1.13 I also had to apply patches listed below.
So in summary appllied:
http://sprunge.us/GYBb
http://sprunge.us/XWUX
http://sprunge.us/jZjV
(Could try http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz only
if there is version for 4.1 tree somewhere)
commit 0aaa29a56e4fb0fc9e24edb649e2733a672ca099
Author: Mel Gorman <mgorman@techsingularity.net>
Date: Fri Nov 6 16:28:37 2015 -0800
mm, page_alloc: reserve pageblocks for high-order atomic allocations on
demand
commit 974a786e63c96a2401a78ddba926f34c128474f1
Author: Mel Gorman <mgorman@techsingularity.net>
Date: Fri Nov 6 16:28:34 2015 -0800
mm, page_alloc: remove MIGRATE_RESERVE
commit c2d42c16ad83006a706d83e51a7268db04af733a
Author: Andrew Morton <akpm@linux-foundation.org>
Date: Thu Nov 5 18:48:43 2015 -0800
mm/vmstat.c: uninline node_page_state()
commit 176bed1de5bf977938cad26551969eca8f0883b1
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Oct 15 13:01:50 2015 -0700
vmstat: explicitly schedule per-cpu work on the CPU we need it to run on
[...]
>
> Well, what steps should we try next for isolating the problem?
>
> Swap is not used at all. Turning off swap might help.
Disabled swap.
>
> [ 8633.753574] Free swap = 117220800kB
> [ 8633.753576] Total swap = 117220820kB
>
> Turning off perf might also help.
>
> [ 5001.394085] perf interrupt took too long (2505 > 2495), lowering
> kernel.perf_event_max_sample_rate to 50100
Didn't find a way to disable perf. kernel .config option gets autoenabled by
some dependency. So left this untouched.
With mentioned patches I wasn't able to reproduce memory allocation problem
(still trying though).
Current debug log: http://ixion.pld-linux.org/~arekm/log-mm-2.txt.gz
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-12 21:28 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-12 21:28 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Thursday 12 of November 2015, Tetsuo Handa wrote:
> On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> > On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> >> Arkadiusz Mi?kiewicz wrote:
> >>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> >>
> >> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> >> Reposting with one condition corrected.
> >
> > Here is log:
> >
> > http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> >
> > Uncompresses is 1.4MB, so not posting here.
>
> Thank you for the log. The result is unexpected for me.
[...]
>
> vmstat_update() and submit_flushes() remained pending for about 110
> seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS allocation,
> it should be reported as MemAlloc: traces, but no such lines are recorded.
> I don't know why xlog_cil_push_work() did not call schedule() for so long.
> Anyway, applying
> http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should solve
> vmstat_update() part.
To apply that patch on top of 4.1.13 I also had to apply patches listed below.
So in summary appllied:
http://sprunge.us/GYBb
http://sprunge.us/XWUX
http://sprunge.us/jZjV
(Could try http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz only
if there is version for 4.1 tree somewhere)
commit 0aaa29a56e4fb0fc9e24edb649e2733a672ca099
Author: Mel Gorman <mgorman@techsingularity.net>
Date: Fri Nov 6 16:28:37 2015 -0800
mm, page_alloc: reserve pageblocks for high-order atomic allocations on
demand
commit 974a786e63c96a2401a78ddba926f34c128474f1
Author: Mel Gorman <mgorman@techsingularity.net>
Date: Fri Nov 6 16:28:34 2015 -0800
mm, page_alloc: remove MIGRATE_RESERVE
commit c2d42c16ad83006a706d83e51a7268db04af733a
Author: Andrew Morton <akpm@linux-foundation.org>
Date: Thu Nov 5 18:48:43 2015 -0800
mm/vmstat.c: uninline node_page_state()
commit 176bed1de5bf977938cad26551969eca8f0883b1
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Oct 15 13:01:50 2015 -0700
vmstat: explicitly schedule per-cpu work on the CPU we need it to run on
[...]
>
> Well, what steps should we try next for isolating the problem?
>
> Swap is not used at all. Turning off swap might help.
Disabled swap.
>
> [ 8633.753574] Free swap = 117220800kB
> [ 8633.753576] Total swap = 117220820kB
>
> Turning off perf might also help.
>
> [ 5001.394085] perf interrupt took too long (2505 > 2495), lowering
> kernel.perf_event_max_sample_rate to 50100
Didn't find a way to disable perf. kernel .config option gets autoenabled by
some dependency. So left this untouched.
With mentioned patches I wasn't able to reproduce memory allocation problem
(still trying though).
Current debug log: http://ixion.pld-linux.org/~arekm/log-mm-2.txt.gz
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-12 20:06 ` Dave Chinner
@ 2015-11-13 12:19 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-13 12:19 UTC (permalink / raw)
To: david; +Cc: htejun, xfs, linux-mm, mhocko, cl
Dave Chinner wrote:
> So why have we only scanned *176* pages* during reclaim? On other
> OOM reports in this trace it's as low as 12. Either that stat is
> completely wrong, or we're not doing sufficient page LRU reclaim
> scanning....
>
> > [ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
> >
> > vmstat_update() and submit_flushes() remained pending for about 110 seconds.
> > If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
> > reported as MemAlloc: traces, but no such lines are recorded. I don't know why
> > xlog_cil_push_work() did not call schedule() for so long.
>
> I'd say it is repeatedly waiting for IO completion on log buffers to
> write out the checkpoint. It's making progress, just if it's taking
> multiple second per journal IO it will take a long time to write a
> checkpoint. All the other blocked tasks in XFS inode reclaim are
> either waiting directly on IO completion or waiting for the log to
> complete a flush, so this really just looks like an overloaded IO
> subsystem to me....
The vmstat statistics can become wrong when vmstat_update() workqueue item
cannot be processed due to in-flight workqueue item not calling schedule().
If in-flight workqueue item (in this case xlog_cil_push_work()) called
schedule(), the pending vmstat_update() workqueue item will be processed
and the vmstat becomes up to dated. Like you expect that xlog_cil_push_work()
was waiting for IO completion on log buffers rather than spinning inside
GFP_NOFS allocation, what should happened is xlog_cil_push_work() called
schedule() and vmstat_update() was processed. But vmstat_update() remained
pending for about 110 seconds. That's strange...
Arkadiusz is trying http://marc.info/?l=linux-mm&m=144725782107096&w=2
which is for making sure that vmstat_update() workqueue item is processed
by changing wait_iff_congested() to call schedule(), and we are waiting
for test results.
Well, one of dependent patches "vmstat: explicitly schedule per-cpu work
on the CPU we need it to run on" might be relevant to this problem.
If http://sprunge.us/GYBb and http://sprunge.us/XWUX solve the problem
(for both with swap case and without swap case), the vmstat statistics
was wrong.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-13 12:19 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-13 12:19 UTC (permalink / raw)
To: david; +Cc: arekm, htejun, cl, mhocko, linux-mm, xfs
Dave Chinner wrote:
> So why have we only scanned *176* pages* during reclaim? On other
> OOM reports in this trace it's as low as 12. Either that stat is
> completely wrong, or we're not doing sufficient page LRU reclaim
> scanning....
>
> > [ 9662.234685] MemAlloc-Info: 3 stalling task, 0 dying task, 0 victim task.
> >
> > vmstat_update() and submit_flushes() remained pending for about 110 seconds.
> > If xlog_cil_push_work() were spinning inside GFP_NOFS allocation, it should be
> > reported as MemAlloc: traces, but no such lines are recorded. I don't know why
> > xlog_cil_push_work() did not call schedule() for so long.
>
> I'd say it is repeatedly waiting for IO completion on log buffers to
> write out the checkpoint. It's making progress, just if it's taking
> multiple second per journal IO it will take a long time to write a
> checkpoint. All the other blocked tasks in XFS inode reclaim are
> either waiting directly on IO completion or waiting for the log to
> complete a flush, so this really just looks like an overloaded IO
> subsystem to me....
The vmstat statistics can become wrong when vmstat_update() workqueue item
cannot be processed due to in-flight workqueue item not calling schedule().
If in-flight workqueue item (in this case xlog_cil_push_work()) called
schedule(), the pending vmstat_update() workqueue item will be processed
and the vmstat becomes up to dated. Like you expect that xlog_cil_push_work()
was waiting for IO completion on log buffers rather than spinning inside
GFP_NOFS allocation, what should happened is xlog_cil_push_work() called
schedule() and vmstat_update() was processed. But vmstat_update() remained
pending for about 110 seconds. That's strange...
Arkadiusz is trying http://marc.info/?l=linux-mm&m=144725782107096&w=2
which is for making sure that vmstat_update() workqueue item is processed
by changing wait_iff_congested() to call schedule(), and we are waiting
for test results.
Well, one of dependent patches "vmstat: explicitly schedule per-cpu work
on the CPU we need it to run on" might be relevant to this problem.
If http://sprunge.us/GYBb and http://sprunge.us/XWUX solve the problem
(for both with swap case and without swap case), the vmstat statistics
was wrong.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-12 21:28 ` Arkadiusz Miśkiewicz
@ 2015-11-14 20:40 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-14 20:40 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Thursday 12 of November 2015, Arkadiusz Miśkiewicz wrote:
> On Thursday 12 of November 2015, Tetsuo Handa wrote:
> > On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> > > On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> > >> Arkadiusz Mi?kiewicz wrote:
> > >>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> > >>
> > >> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> > >> Reposting with one condition corrected.
> > >
> > > Here is log:
> > >
> > > http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> > >
> > > Uncompresses is 1.4MB, so not posting here.
> >
> > Thank you for the log. The result is unexpected for me.
>
> [...]
>
> > vmstat_update() and submit_flushes() remained pending for about 110
> > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > allocation, it should be reported as MemAlloc: traces, but no such lines
> > are recorded. I don't know why xlog_cil_push_work() did not call
> > schedule() for so long. Anyway, applying
> > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > solve vmstat_update() part.
>
> To apply that patch on top of 4.1.13 I also had to apply patches listed
> below.
>
> So in summary appllied:
> http://sprunge.us/GYBb
> http://sprunge.us/XWUX
> http://sprunge.us/jZjV
I've tried more to trigger "page allocation failure" with usual actions that
triggered it previously but couldn't reproduce. With these patches applied it
doesn't happen.
Logs from my tests:
http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
But are these patches solving the problem or just hiding it?
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-14 20:40 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-14 20:40 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, xfs
On Thursday 12 of November 2015, Arkadiusz Miśkiewicz wrote:
> On Thursday 12 of November 2015, Tetsuo Handa wrote:
> > On 2015/11/12 15:06, Arkadiusz Miśkiewicz wrote:
> > > On Wednesday 11 of November 2015, Tetsuo Handa wrote:
> > >> Arkadiusz Mi?kiewicz wrote:
> > >>> This patch is against which tree? (tried 4.1, 4.2 and 4.3)
> > >>
> > >> Oops. Whitespace-damaged. This patch is for vanilla 4.1.2.
> > >> Reposting with one condition corrected.
> > >
> > > Here is log:
> > >
> > > http://ixion.pld-linux.org/~arekm/log-mm-1.txt.gz
> > >
> > > Uncompresses is 1.4MB, so not posting here.
> >
> > Thank you for the log. The result is unexpected for me.
>
> [...]
>
> > vmstat_update() and submit_flushes() remained pending for about 110
> > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > allocation, it should be reported as MemAlloc: traces, but no such lines
> > are recorded. I don't know why xlog_cil_push_work() did not call
> > schedule() for so long. Anyway, applying
> > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > solve vmstat_update() part.
>
> To apply that patch on top of 4.1.13 I also had to apply patches listed
> below.
>
> So in summary appllied:
> http://sprunge.us/GYBb
> http://sprunge.us/XWUX
> http://sprunge.us/jZjV
I've tried more to trigger "page allocation failure" with usual actions that
triggered it previously but couldn't reproduce. With these patches applied it
doesn't happen.
Logs from my tests:
http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
But are these patches solving the problem or just hiding it?
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-14 20:40 ` Arkadiusz Miśkiewicz
@ 2015-11-15 2:35 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-15 2:35 UTC (permalink / raw)
To: arekm; +Cc: linux-mm, cl, htejun, mhocko, xfs
Arkadiusz Miskiewicz wrote:
> > > vmstat_update() and submit_flushes() remained pending for about 110
> > > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > > allocation, it should be reported as MemAlloc: traces, but no such lines
> > > are recorded. I don't know why xlog_cil_push_work() did not call
> > > schedule() for so long. Anyway, applying
> > > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > > solve vmstat_update() part.
> >
> > To apply that patch on top of 4.1.13 I also had to apply patches listed
> > below.
> >
> > So in summary appllied:
> > http://sprunge.us/GYBb
> > http://sprunge.us/XWUX
> > http://sprunge.us/jZjV
>
> I've tried more to trigger "page allocation failure" with usual actions that
> triggered it previously but couldn't reproduce. With these patches applied it
> doesn't happen.
>
> Logs from my tests:
>
> http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
> http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
>
Good.
vmstat_update() and submit_flushes() are no longer pending for long.
log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393960] pending: vmstat_shepherd, vmpressure_work_fn
log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393984] pending: submit_flushes [md_mod]
log-mm-4.txt:Nov 14 16:41:08 srv kernel: [167813.439405] pending: submit_flushes [md_mod]
log-mm-4.txt:Nov 14 17:17:19 srv kernel: [169985.104806] pending: vmstat_shepherd
I think that the vmstat statistics now have correct values.
> But are these patches solving the problem or just hiding it?
>
Excuse me but I can't judge.
If you are interested in monitoring how vmstat statistics are changing
under stalled condition, you can try below patch.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 35a46b4..3de3a14 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2794,8 +2794,7 @@ static int kmallocwd(void *unused)
rcu_read_unlock();
preempt_enable();
show_workqueue_state();
- if (dump_target_pid <= 0)
- dump_target_pid = -pid;
+ show_mem(0);
/* Wait until next timeout duration. */
schedule_timeout_interruptible(kmallocwd_timeout);
if (memalloc_counter[index])
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-15 2:35 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-15 2:35 UTC (permalink / raw)
To: arekm; +Cc: htejun, cl, mhocko, linux-mm, xfs
Arkadiusz Miskiewicz wrote:
> > > vmstat_update() and submit_flushes() remained pending for about 110
> > > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > > allocation, it should be reported as MemAlloc: traces, but no such lines
> > > are recorded. I don't know why xlog_cil_push_work() did not call
> > > schedule() for so long. Anyway, applying
> > > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > > solve vmstat_update() part.
> >
> > To apply that patch on top of 4.1.13 I also had to apply patches listed
> > below.
> >
> > So in summary appllied:
> > http://sprunge.us/GYBb
> > http://sprunge.us/XWUX
> > http://sprunge.us/jZjV
>
> I've tried more to trigger "page allocation failure" with usual actions that
> triggered it previously but couldn't reproduce. With these patches applied it
> doesn't happen.
>
> Logs from my tests:
>
> http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
> http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
>
Good.
vmstat_update() and submit_flushes() are no longer pending for long.
log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393960] pending: vmstat_shepherd, vmpressure_work_fn
log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393984] pending: submit_flushes [md_mod]
log-mm-4.txt:Nov 14 16:41:08 srv kernel: [167813.439405] pending: submit_flushes [md_mod]
log-mm-4.txt:Nov 14 17:17:19 srv kernel: [169985.104806] pending: vmstat_shepherd
I think that the vmstat statistics now have correct values.
> But are these patches solving the problem or just hiding it?
>
Excuse me but I can't judge.
If you are interested in monitoring how vmstat statistics are changing
under stalled condition, you can try below patch.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 35a46b4..3de3a14 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2794,8 +2794,7 @@ static int kmallocwd(void *unused)
rcu_read_unlock();
preempt_enable();
show_workqueue_state();
- if (dump_target_pid <= 0)
- dump_target_pid = -pid;
+ show_mem(0);
/* Wait until next timeout duration. */
schedule_timeout_interruptible(kmallocwd_timeout);
if (memalloc_counter[index])
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-15 2:35 ` Tetsuo Handa
@ 2015-11-15 11:29 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-15 11:29 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, cl, htejun, mhocko, xfs
On Sunday 15 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Miskiewicz wrote:
> > > > vmstat_update() and submit_flushes() remained pending for about 110
> > > > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > > > allocation, it should be reported as MemAlloc: traces, but no such
> > > > lines are recorded. I don't know why xlog_cil_push_work() did not
> > > > call schedule() for so long. Anyway, applying
> > > > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > > > solve vmstat_update() part.
> > >
> > > To apply that patch on top of 4.1.13 I also had to apply patches listed
> > > below.
> > >
> > > So in summary appllied:
> > > http://sprunge.us/GYBb
> > > http://sprunge.us/XWUX
> > > http://sprunge.us/jZjV
> >
> > I've tried more to trigger "page allocation failure" with usual actions
> > that triggered it previously but couldn't reproduce. With these patches
> > applied it doesn't happen.
> >
> > Logs from my tests:
> >
> > http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
> > http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
>
> Good.
>
> vmstat_update() and submit_flushes() are no longer pending for long.
>
> log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393960] pending:
> vmstat_shepherd, vmpressure_work_fn log-mm-4.txt:Nov 14 16:40:08 srv
> kernel: [167753.393984] pending: submit_flushes [md_mod]
> log-mm-4.txt:Nov 14 16:41:08 srv kernel: [167813.439405] pending:
> submit_flushes [md_mod] log-mm-4.txt:Nov 14 17:17:19 srv kernel:
> [169985.104806] pending: vmstat_shepherd
>
> I think that the vmstat statistics now have correct values.
>
> > But are these patches solving the problem or just hiding it?
>
> Excuse me but I can't judge.
>
> If you are interested in monitoring how vmstat statistics are changing
> under stalled condition, you can try below patch.
Here is log with this and all previous patches applied:
http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 35a46b4..3de3a14 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2794,8 +2794,7 @@ static int kmallocwd(void *unused)
> rcu_read_unlock();
> preempt_enable();
> show_workqueue_state();
> - if (dump_target_pid <= 0)
> - dump_target_pid = -pid;
> + show_mem(0);
> /* Wait until next timeout duration. */
> schedule_timeout_interruptible(kmallocwd_timeout);
> if (memalloc_counter[index])
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-15 11:29 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-15 11:29 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: htejun, cl, mhocko, linux-mm, xfs
On Sunday 15 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Miskiewicz wrote:
> > > > vmstat_update() and submit_flushes() remained pending for about 110
> > > > seconds. If xlog_cil_push_work() were spinning inside GFP_NOFS
> > > > allocation, it should be reported as MemAlloc: traces, but no such
> > > > lines are recorded. I don't know why xlog_cil_push_work() did not
> > > > call schedule() for so long. Anyway, applying
> > > > http://lkml.kernel.org/r/20151111160336.GD1432@dhcp22.suse.cz should
> > > > solve vmstat_update() part.
> > >
> > > To apply that patch on top of 4.1.13 I also had to apply patches listed
> > > below.
> > >
> > > So in summary appllied:
> > > http://sprunge.us/GYBb
> > > http://sprunge.us/XWUX
> > > http://sprunge.us/jZjV
> >
> > I've tried more to trigger "page allocation failure" with usual actions
> > that triggered it previously but couldn't reproduce. With these patches
> > applied it doesn't happen.
> >
> > Logs from my tests:
> >
> > http://ixion.pld-linux.org/~arekm/log-mm-3.txt.gz
> > http://ixion.pld-linux.org/~arekm/log-mm-4.txt.gz (with swap added)
>
> Good.
>
> vmstat_update() and submit_flushes() are no longer pending for long.
>
> log-mm-4.txt:Nov 14 16:40:08 srv kernel: [167753.393960] pending:
> vmstat_shepherd, vmpressure_work_fn log-mm-4.txt:Nov 14 16:40:08 srv
> kernel: [167753.393984] pending: submit_flushes [md_mod]
> log-mm-4.txt:Nov 14 16:41:08 srv kernel: [167813.439405] pending:
> submit_flushes [md_mod] log-mm-4.txt:Nov 14 17:17:19 srv kernel:
> [169985.104806] pending: vmstat_shepherd
>
> I think that the vmstat statistics now have correct values.
>
> > But are these patches solving the problem or just hiding it?
>
> Excuse me but I can't judge.
>
> If you are interested in monitoring how vmstat statistics are changing
> under stalled condition, you can try below patch.
Here is log with this and all previous patches applied:
http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 35a46b4..3de3a14 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2794,8 +2794,7 @@ static int kmallocwd(void *unused)
> rcu_read_unlock();
> preempt_enable();
> show_workqueue_state();
> - if (dump_target_pid <= 0)
> - dump_target_pid = -pid;
> + show_mem(0);
> /* Wait until next timeout duration. */
> schedule_timeout_interruptible(kmallocwd_timeout);
> if (memalloc_counter[index])
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-15 11:29 ` Arkadiusz Miśkiewicz
@ 2015-11-15 14:13 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-15 14:13 UTC (permalink / raw)
To: arekm; +Cc: linux-mm, cl, htejun, mhocko, xfs
Arkadiusz Miskiewicz wrote:
> On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > I think that the vmstat statistics now have correct values.
> >
> > > But are these patches solving the problem or just hiding it?
> >
> > Excuse me but I can't judge.
> >
> > If you are interested in monitoring how vmstat statistics are changing
> > under stalled condition, you can try below patch.
>
>
> Here is log with this and all previous patches applied:
> http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
all free: values look sane to me. I think that your problem was solved.
$ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 } ' | cut -b 6- | sort -g
8608kB
8636kB
8920kB
8920kB
8952kB
8968kB
8980kB
(...snipped...)
215364kB
290068kB
290428kB
291176kB
292836kB
303992kB
306468kB
318080kB
319548kB
$ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 } ' | cut -b 6- | sort -g
0kB
40kB
128kB
128kB
128kB
128kB
128kB
128kB
128kB
128kB
(...snipped...)
412kB
616kB
1268kB
1544kB
1696kB
2756kB
2756kB
2756kB
2756kB
2756kB
2756kB
2756kB
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-15 14:13 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-15 14:13 UTC (permalink / raw)
To: arekm; +Cc: htejun, cl, mhocko, linux-mm, xfs
Arkadiusz Miskiewicz wrote:
> On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > I think that the vmstat statistics now have correct values.
> >
> > > But are these patches solving the problem or just hiding it?
> >
> > Excuse me but I can't judge.
> >
> > If you are interested in monitoring how vmstat statistics are changing
> > under stalled condition, you can try below patch.
>
>
> Here is log with this and all previous patches applied:
> http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
all free: values look sane to me. I think that your problem was solved.
$ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 } ' | cut -b 6- | sort -g
8608kB
8636kB
8920kB
8920kB
8952kB
8968kB
8980kB
(...snipped...)
215364kB
290068kB
290428kB
291176kB
292836kB
303992kB
306468kB
318080kB
319548kB
$ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 } ' | cut -b 6- | sort -g
0kB
40kB
128kB
128kB
128kB
128kB
128kB
128kB
128kB
128kB
(...snipped...)
412kB
616kB
1268kB
1544kB
1696kB
2756kB
2756kB
2756kB
2756kB
2756kB
2756kB
2756kB
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-15 14:13 ` Tetsuo Handa
@ 2015-11-15 14:49 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-15 14:49 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm, cl, htejun, mhocko, xfs
On Sunday 15 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Miskiewicz wrote:
> > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > I think that the vmstat statistics now have correct values.
> > >
> > > > But are these patches solving the problem or just hiding it?
> > >
> > > Excuse me but I can't judge.
> > >
> > > If you are interested in monitoring how vmstat statistics are changing
> > > under stalled condition, you can try below patch.
> >
> > Here is log with this and all previous patches applied:
> > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
>
> Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> all free: values look sane to me. I think that your problem was solved.
Great, thanks!
Will all (or part) of these patches
http://sprunge.us/GYBb
http://sprunge.us/XWUX (after it gets merged)
be pushed to stable@ or are too risky for stable ?
>
> $ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 }
> ' | cut -b 6- | sort -g 8608kB
> 8636kB
> 8920kB
> 8920kB
> 8952kB
> 8968kB
> 8980kB
> (...snipped...)
> 215364kB
> 290068kB
> 290428kB
> 291176kB
> 292836kB
> 303992kB
> 306468kB
> 318080kB
> 319548kB
>
> $ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 }
> ' | cut -b 6- | sort -g 0kB
> 40kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> (...snipped...)
> 412kB
> 616kB
> 1268kB
> 1544kB
> 1696kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-15 14:49 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-15 14:49 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: htejun, cl, mhocko, linux-mm, xfs
On Sunday 15 of November 2015, Tetsuo Handa wrote:
> Arkadiusz Miskiewicz wrote:
> > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > I think that the vmstat statistics now have correct values.
> > >
> > > > But are these patches solving the problem or just hiding it?
> > >
> > > Excuse me but I can't judge.
> > >
> > > If you are interested in monitoring how vmstat statistics are changing
> > > under stalled condition, you can try below patch.
> >
> > Here is log with this and all previous patches applied:
> > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
>
> Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> all free: values look sane to me. I think that your problem was solved.
Great, thanks!
Will all (or part) of these patches
http://sprunge.us/GYBb
http://sprunge.us/XWUX (after it gets merged)
be pushed to stable@ or are too risky for stable ?
>
> $ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 }
> ' | cut -b 6- | sort -g 8608kB
> 8636kB
> 8920kB
> 8920kB
> 8952kB
> 8968kB
> 8980kB
> (...snipped...)
> 215364kB
> 290068kB
> 290428kB
> 291176kB
> 292836kB
> 303992kB
> 306468kB
> 318080kB
> 319548kB
>
> $ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 }
> ' | cut -b 6- | sort -g 0kB
> 40kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> 128kB
> (...snipped...)
> 412kB
> 616kB
> 1268kB
> 1544kB
> 1696kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
> 2756kB
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-15 14:49 ` Arkadiusz Miśkiewicz
@ 2015-11-16 16:15 ` Michal Hocko
-1 siblings, 0 replies; 34+ messages in thread
From: Michal Hocko @ 2015-11-16 16:15 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz; +Cc: Tetsuo Handa, linux-mm, cl, htejun, xfs
On Sun 15-11-15 15:49:35, Arkadiusz Miśkiewicz wrote:
> On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > Arkadiusz Miskiewicz wrote:
> > > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > > I think that the vmstat statistics now have correct values.
> > > >
> > > > > But are these patches solving the problem or just hiding it?
> > > >
> > > > Excuse me but I can't judge.
> > > >
> > > > If you are interested in monitoring how vmstat statistics are changing
> > > > under stalled condition, you can try below patch.
> > >
> > > Here is log with this and all previous patches applied:
> > > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
> >
> > Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> > all free: values look sane to me. I think that your problem was solved.
>
> Great, thanks!
>
> Will all (or part) of these patches
>
> http://sprunge.us/GYBb
Migrate reserves are not a stable material I am afraid. "vmstat:
explicitly schedule per-cpu work on the CPU we need it to run on"
was not marked for stable either but I am not sure why it should make
any difference for your load. I understand that testing this is really
tedious but it would be better to know which of the patches actually
made a difference.
> http://sprunge.us/XWUX (after it gets merged)
Yes I plan to mark this one for stable.
> be pushed to stable@ or are too risky for stable ?
>
> >
> > $ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 }
> > ' | cut -b 6- | sort -g 8608kB
> > 8636kB
> > 8920kB
> > 8920kB
> > 8952kB
> > 8968kB
> > 8980kB
> > (...snipped...)
> > 215364kB
> > 290068kB
> > 290428kB
> > 291176kB
> > 292836kB
> > 303992kB
> > 306468kB
> > 318080kB
> > 319548kB
> >
> > $ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 }
> > ' | cut -b 6- | sort -g 0kB
> > 40kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > (...snipped...)
> > 412kB
> > 616kB
> > 1268kB
> > 1544kB
> > 1696kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
>
>
> --
> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
Michal Hocko
SUSE Labs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-16 16:15 ` Michal Hocko
0 siblings, 0 replies; 34+ messages in thread
From: Michal Hocko @ 2015-11-16 16:15 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz; +Cc: Tetsuo Handa, htejun, cl, linux-mm, xfs
On Sun 15-11-15 15:49:35, Arkadiusz MiA?kiewicz wrote:
> On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > Arkadiusz Miskiewicz wrote:
> > > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > > I think that the vmstat statistics now have correct values.
> > > >
> > > > > But are these patches solving the problem or just hiding it?
> > > >
> > > > Excuse me but I can't judge.
> > > >
> > > > If you are interested in monitoring how vmstat statistics are changing
> > > > under stalled condition, you can try below patch.
> > >
> > > Here is log with this and all previous patches applied:
> > > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
> >
> > Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> > all free: values look sane to me. I think that your problem was solved.
>
> Great, thanks!
>
> Will all (or part) of these patches
>
> http://sprunge.us/GYBb
Migrate reserves are not a stable material I am afraid. "vmstat:
explicitly schedule per-cpu work on the CPU we need it to run on"
was not marked for stable either but I am not sure why it should make
any difference for your load. I understand that testing this is really
tedious but it would be better to know which of the patches actually
made a difference.
> http://sprunge.us/XWUX (after it gets merged)
Yes I plan to mark this one for stable.
> be pushed to stable@ or are too risky for stable ?
>
> >
> > $ grep "Normal free:" log-mm-5.txt | cut -b 44- | awk -F' ' ' { print $4 }
> > ' | cut -b 6- | sort -g 8608kB
> > 8636kB
> > 8920kB
> > 8920kB
> > 8952kB
> > 8968kB
> > 8980kB
> > (...snipped...)
> > 215364kB
> > 290068kB
> > 290428kB
> > 291176kB
> > 292836kB
> > 303992kB
> > 306468kB
> > 318080kB
> > 319548kB
> >
> > $ grep "Normal free:" log-mm-1.txt | cut -b 44- | awk -F' ' ' { print $4 }
> > ' | cut -b 6- | sort -g 0kB
> > 40kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > 128kB
> > (...snipped...)
> > 412kB
> > 616kB
> > 1268kB
> > 1544kB
> > 1696kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
> > 2756kB
>
>
> --
> Arkadiusz MiA?kiewicz, arekm / ( maven.pl | pld-linux.org )
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-16 16:15 ` Michal Hocko
@ 2015-11-18 22:36 ` Arkadiusz Miśkiewicz
-1 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-18 22:36 UTC (permalink / raw)
To: Michal Hocko; +Cc: Tetsuo Handa, linux-mm, cl, htejun, xfs
On Monday 16 of November 2015, Michal Hocko wrote:
> On Sun 15-11-15 15:49:35, Arkadiusz Miśkiewicz wrote:
> > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > Arkadiusz Miskiewicz wrote:
> > > > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > > > I think that the vmstat statistics now have correct values.
> > > > >
> > > > > > But are these patches solving the problem or just hiding it?
> > > > >
> > > > > Excuse me but I can't judge.
> > > > >
> > > > > If you are interested in monitoring how vmstat statistics are
> > > > > changing under stalled condition, you can try below patch.
> > > >
> > > > Here is log with this and all previous patches applied:
> > > > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
> > >
> > > Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> > > all free: values look sane to me. I think that your problem was solved.
> >
> > Great, thanks!
> >
> > Will all (or part) of these patches
> >
> > http://sprunge.us/GYBb
>
> Migrate reserves are not a stable material I am afraid. "vmstat:
> explicitly schedule per-cpu work on the CPU we need it to run on"
> was not marked for stable either but I am not sure why it should make
> any difference for your load. I understand that testing this is really
> tedious but it would be better to know which of the patches actually
> made a difference.
Ok. In mean time I've tried 4.3.0 kernel + patches (the same as before + one
more) on second server which runs even more rsnapshot processes and also uses
xfs on md raid 6.
Patches:
http://sprunge.us/DfIQ (debug patch from Tetsuo)
http://sprunge.us/LQPF (backport of things from git + one from ml)
The problem is now with high order allocations probably:
http://ixion.pld-linux.org/~arekm/log-mm-2srv-1.txt.gz
System is doing very slow progress and for example depmod run took 2 hours
http://sprunge.us/HGbE
Sometimes I was able to ssh-in, dmesg took 10-15 minutes but sometimes it
worked fast for short period.
Ideas?
ps. I also had one problem with low order allocation but only once and wasn't
able to reproduce so far. I was running kernel with backport patches but no
debug patch, so got only this in logs:
http://sprunge.us/WPXi
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-18 22:36 ` Arkadiusz Miśkiewicz
0 siblings, 0 replies; 34+ messages in thread
From: Arkadiusz Miśkiewicz @ 2015-11-18 22:36 UTC (permalink / raw)
To: Michal Hocko; +Cc: Tetsuo Handa, htejun, cl, linux-mm, xfs
On Monday 16 of November 2015, Michal Hocko wrote:
> On Sun 15-11-15 15:49:35, Arkadiusz Miśkiewicz wrote:
> > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > Arkadiusz Miskiewicz wrote:
> > > > On Sunday 15 of November 2015, Tetsuo Handa wrote:
> > > > > I think that the vmstat statistics now have correct values.
> > > > >
> > > > > > But are these patches solving the problem or just hiding it?
> > > > >
> > > > > Excuse me but I can't judge.
> > > > >
> > > > > If you are interested in monitoring how vmstat statistics are
> > > > > changing under stalled condition, you can try below patch.
> > > >
> > > > Here is log with this and all previous patches applied:
> > > > http://ixion.pld-linux.org/~arekm/log-mm-5.txt.gz
> > >
> > > Regarding "Node 0 Normal" (min:7104kB low:8880kB high:10656kB),
> > > all free: values look sane to me. I think that your problem was solved.
> >
> > Great, thanks!
> >
> > Will all (or part) of these patches
> >
> > http://sprunge.us/GYBb
>
> Migrate reserves are not a stable material I am afraid. "vmstat:
> explicitly schedule per-cpu work on the CPU we need it to run on"
> was not marked for stable either but I am not sure why it should make
> any difference for your load. I understand that testing this is really
> tedious but it would be better to know which of the patches actually
> made a difference.
Ok. In mean time I've tried 4.3.0 kernel + patches (the same as before + one
more) on second server which runs even more rsnapshot processes and also uses
xfs on md raid 6.
Patches:
http://sprunge.us/DfIQ (debug patch from Tetsuo)
http://sprunge.us/LQPF (backport of things from git + one from ml)
The problem is now with high order allocations probably:
http://ixion.pld-linux.org/~arekm/log-mm-2srv-1.txt.gz
System is doing very slow progress and for example depmod run took 2 hours
http://sprunge.us/HGbE
Sometimes I was able to ssh-in, dmesg took 10-15 minutes but sometimes it
worked fast for short period.
Ideas?
ps. I also had one problem with low order allocation but only once and wasn't
able to reproduce so far. I was running kernel with backport patches but no
debug patch, so got only this in logs:
http://sprunge.us/WPXi
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
2015-11-18 22:36 ` Arkadiusz Miśkiewicz
@ 2015-11-19 15:59 ` Tetsuo Handa
-1 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-19 15:59 UTC (permalink / raw)
To: arekm, mhocko; +Cc: linux-mm, cl, htejun, xfs
Arkadiusz Miskiewicz wrote:
> Ok. In mean time I've tried 4.3.0 kernel + patches (the same as before + one
> more) on second server which runs even more rsnapshot processes and also uses
> xfs on md raid 6.
>
> Patches:
> http://sprunge.us/DfIQ (debug patch from Tetsuo)
> http://sprunge.us/LQPF (backport of things from git + one from ml)
>
> The problem is now with high order allocations probably:
> http://ixion.pld-linux.org/~arekm/log-mm-2srv-1.txt.gz
This seems to be stall upon allocating transparent huge pages.
(page fault -> try to allocate 2MB page -> reclaim -> waiting)
----------
[ 1166.110205] Node 0 Normal: 4801*4kB (UE) 1461*8kB (UMEC) 290*16kB (UMEC) 715*32kB (UM) 67*64kB (UME) 7*128kB (UMEC) 3*256kB (MEC) 6*512kB (UMEC) 3*1024kB (MEC) 10*2048kB (UME) 219*4096kB (M) = 988012kB
[ 1178.250751] Node 0 Normal: 4917*4kB (UME) 2622*8kB (UMEC) 530*16kB (UMC) 713*32kB (UME) 68*64kB (UM) 8*128kB (UMC) 5*256kB (MEC) 5*512kB (UMC) 8*1024kB (MEC) 4*2048kB (UME) 207*4096kB (M) = 945412kB
[ 1190.108587] Node 0 Normal: 4301*4kB (UME) 3132*8kB (UMEC) 670*16kB (UMEC) 766*32kB (UME) 74*64kB (UME) 11*128kB (UMC) 8*256kB (UMEC) 4*512kB (UMEC) 3*1024kB (MEC) 2*2048kB (UM) 206*4096kB (M) = 938676kB
[ 1202.014434] Node 0 Normal: 3971*4kB (UM) 2704*8kB (UMC) 650*16kB (UMEC) 754*32kB (UE) 68*64kB (UME) 6*128kB (UMC) 3*256kB (UEC) 3*512kB (UEC) 6*1024kB (MEC) 5*2048kB (UM) 206*4096kB (M) = 939628kB
[ 1212.438969] Node 0 Normal: 5307*4kB (UE) 4976*8kB (UEC) 1095*16kB (UC) 702*32kB (UM) 67*64kB (UM) 6*128kB (UMC) 2*256kB (UC) 37*512kB (UMC) 36*1024kB (MC) 29*2048kB (UME) 324*4096kB (M) = 1548892kB
[ 1222.840549] Node 0 Normal: 0*4kB 5*8kB (UMEC) 711*16kB (UC) 490*32kB (UME) 66*64kB (U) 6*128kB (UEC) 3*256kB (UMC) 3*512kB (UEC) 2*1024kB (MC) 1*2048kB (U) 303*4096kB (M) = 1279576kB
[ 1243.941537] Node 0 Normal: 28825*4kB (UE) 26405*8kB (UMEC) 12409*16kB (UMC) 3351*32kB (UME) 408*64kB (UM) 19*128kB (UMC) 3*256kB (UC) 4*512kB (UMC) 2*1024kB (UC) 3*2048kB (UME) 65*4096kB (M) = 938108kB
[ 1256.006097] Node 0 Normal: 40219*4kB (UME) 34873*8kB (UE) 17232*16kB (UME) 5633*32kB (UME) 598*64kB (UM) 1*128kB (M) 0*256kB 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 935252kB
[ 1268.951714] Node 0 Normal: 36607*4kB (UME) 37875*8kB (UME) 17450*16kB (UME) 5497*32kB (UME) 485*64kB (U) 0*128kB 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 936084kB
[ 1279.263718] Node 0 Normal: 39565*4kB (UME) 39229*8kB (UME) 17553*16kB (UME) 5174*32kB (UME) 279*64kB (UM) 2*128kB (ME) 4*256kB (ME) 3*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 939180kB
[ 1300.454484] Node 0 Normal: 37945*4kB (U) 52089*8kB (U) 24197*16kB (UE) 7420*32kB (UME) 493*64kB (UM) 10*128kB (UME) 1*256kB (M) 1*512kB (E) 2*1024kB (ME) 1*2048kB (E) 39*4096kB (M) = 1390524kB
[ 1310.761624] Node 0 Normal: 18034*4kB (U) 44842*8kB (UE) 19948*16kB (UE) 5383*32kB (UME) 162*64kB (UME) 2*128kB (ME) 3*256kB (ME) 5*512kB (ME) 1*1024kB (M) 0*2048kB 0*4096kB = 937272kB
[ 1320.848763] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=4574
[ 1321.988480] Node 0 Normal: 19294*4kB (UME) 45490*8kB (UE) 20125*16kB (UME) 5229*32kB (UM) 93*64kB (UM) 4*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 936888kB
[ 1342.042799] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=10944
[ 1352.791917] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=14172
[ 1364.143762] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=17581
[ 1374.499973] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=20691
[ 1384.842838] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=23797
[ 1395.192348] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=26905
[ 1405.525243] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=30008
[ 1415.831459] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=33103
[ 1427.113385] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=36491
[ 1437.416290] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=39585
[ 1437.428716] MemAlloc: rsnapshot(3639) gfp=0x4f52ca order=9 delay=3389
(...snipped...)
[30084.882928] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3654551
[30085.744771] Node 0 Normal: 212267*4kB (UME) 10914*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 936380kB
[30084.889563] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=324074
[30106.734724] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3698251
[30106.747497] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=330638
[30117.713596] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3701548
[30117.719721] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3664412
[30117.726356] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=333935
[30139.904472] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3708212
[30139.910594] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3671076
[30139.917237] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=340599
[30172.684553] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3718056
[30173.749694] Node 0 Normal: 234069*4kB (UE) 148*8kB (UE) 113*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 939268kB
----------
So many 4kB pages but no 2048kB pages.
> System is doing very slow progress and for example depmod run took 2 hours
> http://sprunge.us/HGbE
> Sometimes I was able to ssh-in, dmesg took 10-15 minutes but sometimes it
> worked fast for short period.
>
> Ideas?
Memory fragmentation is out of my understanding.
Maybe disabling THP can help.
>
> ps. I also had one problem with low order allocation but only once and wasn't
> able to reproduce so far. I was running kernel with backport patches but no
> debug patch, so got only this in logs:
> http://sprunge.us/WPXi
Unfortunately 16kB pages was not available at that moment.
But it was GFP_ATOMIC and infrequent failure should not become a problem.
----------
[ 8513.740326] swapper/3: page allocation failure: order:2, mode:0xc020
(...snipped...)
[ 8514.247719] Node 0 Normal: 207920*4kB (UE) 18052*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 976096kB
----------
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: memory reclaim problems on fs usage
@ 2015-11-19 15:59 ` Tetsuo Handa
0 siblings, 0 replies; 34+ messages in thread
From: Tetsuo Handa @ 2015-11-19 15:59 UTC (permalink / raw)
To: arekm, mhocko; +Cc: htejun, cl, linux-mm, xfs
Arkadiusz Miskiewicz wrote:
> Ok. In mean time I've tried 4.3.0 kernel + patches (the same as before + one
> more) on second server which runs even more rsnapshot processes and also uses
> xfs on md raid 6.
>
> Patches:
> http://sprunge.us/DfIQ (debug patch from Tetsuo)
> http://sprunge.us/LQPF (backport of things from git + one from ml)
>
> The problem is now with high order allocations probably:
> http://ixion.pld-linux.org/~arekm/log-mm-2srv-1.txt.gz
This seems to be stall upon allocating transparent huge pages.
(page fault -> try to allocate 2MB page -> reclaim -> waiting)
----------
[ 1166.110205] Node 0 Normal: 4801*4kB (UE) 1461*8kB (UMEC) 290*16kB (UMEC) 715*32kB (UM) 67*64kB (UME) 7*128kB (UMEC) 3*256kB (MEC) 6*512kB (UMEC) 3*1024kB (MEC) 10*2048kB (UME) 219*4096kB (M) = 988012kB
[ 1178.250751] Node 0 Normal: 4917*4kB (UME) 2622*8kB (UMEC) 530*16kB (UMC) 713*32kB (UME) 68*64kB (UM) 8*128kB (UMC) 5*256kB (MEC) 5*512kB (UMC) 8*1024kB (MEC) 4*2048kB (UME) 207*4096kB (M) = 945412kB
[ 1190.108587] Node 0 Normal: 4301*4kB (UME) 3132*8kB (UMEC) 670*16kB (UMEC) 766*32kB (UME) 74*64kB (UME) 11*128kB (UMC) 8*256kB (UMEC) 4*512kB (UMEC) 3*1024kB (MEC) 2*2048kB (UM) 206*4096kB (M) = 938676kB
[ 1202.014434] Node 0 Normal: 3971*4kB (UM) 2704*8kB (UMC) 650*16kB (UMEC) 754*32kB (UE) 68*64kB (UME) 6*128kB (UMC) 3*256kB (UEC) 3*512kB (UEC) 6*1024kB (MEC) 5*2048kB (UM) 206*4096kB (M) = 939628kB
[ 1212.438969] Node 0 Normal: 5307*4kB (UE) 4976*8kB (UEC) 1095*16kB (UC) 702*32kB (UM) 67*64kB (UM) 6*128kB (UMC) 2*256kB (UC) 37*512kB (UMC) 36*1024kB (MC) 29*2048kB (UME) 324*4096kB (M) = 1548892kB
[ 1222.840549] Node 0 Normal: 0*4kB 5*8kB (UMEC) 711*16kB (UC) 490*32kB (UME) 66*64kB (U) 6*128kB (UEC) 3*256kB (UMC) 3*512kB (UEC) 2*1024kB (MC) 1*2048kB (U) 303*4096kB (M) = 1279576kB
[ 1243.941537] Node 0 Normal: 28825*4kB (UE) 26405*8kB (UMEC) 12409*16kB (UMC) 3351*32kB (UME) 408*64kB (UM) 19*128kB (UMC) 3*256kB (UC) 4*512kB (UMC) 2*1024kB (UC) 3*2048kB (UME) 65*4096kB (M) = 938108kB
[ 1256.006097] Node 0 Normal: 40219*4kB (UME) 34873*8kB (UE) 17232*16kB (UME) 5633*32kB (UME) 598*64kB (UM) 1*128kB (M) 0*256kB 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 935252kB
[ 1268.951714] Node 0 Normal: 36607*4kB (UME) 37875*8kB (UME) 17450*16kB (UME) 5497*32kB (UME) 485*64kB (U) 0*128kB 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 936084kB
[ 1279.263718] Node 0 Normal: 39565*4kB (UME) 39229*8kB (UME) 17553*16kB (UME) 5174*32kB (UME) 279*64kB (UM) 2*128kB (ME) 4*256kB (ME) 3*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 939180kB
[ 1300.454484] Node 0 Normal: 37945*4kB (U) 52089*8kB (U) 24197*16kB (UE) 7420*32kB (UME) 493*64kB (UM) 10*128kB (UME) 1*256kB (M) 1*512kB (E) 2*1024kB (ME) 1*2048kB (E) 39*4096kB (M) = 1390524kB
[ 1310.761624] Node 0 Normal: 18034*4kB (U) 44842*8kB (UE) 19948*16kB (UE) 5383*32kB (UME) 162*64kB (UME) 2*128kB (ME) 3*256kB (ME) 5*512kB (ME) 1*1024kB (M) 0*2048kB 0*4096kB = 937272kB
[ 1320.848763] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=4574
[ 1321.988480] Node 0 Normal: 19294*4kB (UME) 45490*8kB (UE) 20125*16kB (UME) 5229*32kB (UM) 93*64kB (UM) 4*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 936888kB
[ 1342.042799] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=10944
[ 1352.791917] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=14172
[ 1364.143762] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=17581
[ 1374.499973] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=20691
[ 1384.842838] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=23797
[ 1395.192348] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=26905
[ 1405.525243] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=30008
[ 1415.831459] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=33103
[ 1427.113385] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=36491
[ 1437.416290] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=39585
[ 1437.428716] MemAlloc: rsnapshot(3639) gfp=0x4f52ca order=9 delay=3389
(...snipped...)
[30084.882928] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3654551
[30085.744771] Node 0 Normal: 212267*4kB (UME) 10914*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 936380kB
[30084.889563] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=324074
[30106.734724] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3698251
[30106.747497] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=330638
[30117.713596] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3701548
[30117.719721] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3664412
[30117.726356] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=333935
[30139.904472] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3708212
[30139.910594] MemAlloc: khugepaged(32) gfp=0xcf52ca order=9 delay=3671076
[30139.917237] MemAlloc: syslog-ng(1671) gfp=0x4f52ca order=9 delay=340599
[30172.684553] MemAlloc: cp(2795) gfp=0x4f52ca order=9 delay=3718056
[30173.749694] Node 0 Normal: 234069*4kB (UE) 148*8kB (UE) 113*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 939268kB
----------
So many 4kB pages but no 2048kB pages.
> System is doing very slow progress and for example depmod run took 2 hours
> http://sprunge.us/HGbE
> Sometimes I was able to ssh-in, dmesg took 10-15 minutes but sometimes it
> worked fast for short period.
>
> Ideas?
Memory fragmentation is out of my understanding.
Maybe disabling THP can help.
>
> ps. I also had one problem with low order allocation but only once and wasn't
> able to reproduce so far. I was running kernel with backport patches but no
> debug patch, so got only this in logs:
> http://sprunge.us/WPXi
Unfortunately 16kB pages was not available at that moment.
But it was GFP_ATOMIC and infrequent failure should not become a problem.
----------
[ 8513.740326] swapper/3: page allocation failure: order:2, mode:0xc020
(...snipped...)
[ 8514.247719] Node 0 Normal: 207920*4kB (UE) 18052*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 976096kB
----------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2015-11-19 16:00 UTC | newest]
Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-10 22:13 memory reclaim problems on fs usage Arkadiusz Miśkiewicz
2015-11-10 22:13 ` Arkadiusz Miśkiewicz
2015-11-11 15:58 ` Tetsuo Handa
2015-11-11 15:58 ` Tetsuo Handa
2015-11-11 16:19 ` Arkadiusz Miśkiewicz
2015-11-11 16:19 ` Arkadiusz Miśkiewicz
2015-11-11 22:19 ` Tetsuo Handa
2015-11-11 22:19 ` Tetsuo Handa
2015-11-12 6:06 ` Arkadiusz Miśkiewicz
2015-11-12 6:06 ` Arkadiusz Miśkiewicz
2015-11-12 14:12 ` Tetsuo Handa
2015-11-12 14:12 ` Tetsuo Handa
2015-11-12 20:06 ` Dave Chinner
2015-11-12 20:06 ` Dave Chinner
2015-11-13 12:19 ` Tetsuo Handa
2015-11-13 12:19 ` Tetsuo Handa
2015-11-12 21:28 ` Arkadiusz Miśkiewicz
2015-11-12 21:28 ` Arkadiusz Miśkiewicz
2015-11-14 20:40 ` Arkadiusz Miśkiewicz
2015-11-14 20:40 ` Arkadiusz Miśkiewicz
2015-11-15 2:35 ` Tetsuo Handa
2015-11-15 2:35 ` Tetsuo Handa
2015-11-15 11:29 ` Arkadiusz Miśkiewicz
2015-11-15 11:29 ` Arkadiusz Miśkiewicz
2015-11-15 14:13 ` Tetsuo Handa
2015-11-15 14:13 ` Tetsuo Handa
2015-11-15 14:49 ` Arkadiusz Miśkiewicz
2015-11-15 14:49 ` Arkadiusz Miśkiewicz
2015-11-16 16:15 ` Michal Hocko
2015-11-16 16:15 ` Michal Hocko
2015-11-18 22:36 ` Arkadiusz Miśkiewicz
2015-11-18 22:36 ` Arkadiusz Miśkiewicz
2015-11-19 15:59 ` Tetsuo Handa
2015-11-19 15:59 ` Tetsuo Handa
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.