* high kswapd CPU usage when executing binaries from NFS w/ CMA and COMPACTION
@ 2013-12-04 2:30 Florian Fainelli
2013-12-04 13:00 ` Mel Gorman
0 siblings, 1 reply; 3+ messages in thread
From: Florian Fainelli @ 2013-12-04 2:30 UTC (permalink / raw)
To: linux-kernel, mgorman, mhocko, hannes, riel, linux-mm,
m.szyprowski, marc.ceeeee
[-- Attachment #1: Type: text/plain, Size: 750 bytes --]
Hi all,
I am experiencing high kswapd CPU usage on an ARMv7 system running
3.8.13 when executing relatively large binaries from NFS. When this
happens kswapd consumes around 55-60% CPU usage and the applications
takes a huge time to load.
My system has both CONFIG_CMA and CONFIG_COMPACTION enabled and
turning off CONFIG_COMPACTION does seem to make things slightly better
since kswapd does not take much CPU usage if at all and the
application loads with its expected loading time.
Attached are the contents of /proc/zoneinfo, vmstat and meminfo when
the system works correctly and their counterparts when the system is
busy in kswapd.
Any pointers to patches I should eventually backport or try would be
much appreciated!
Thanks!
--
Florian
[-- Attachment #2: meminfo-working --]
[-- Type: application/octet-stream, Size: 1036 bytes --]
MemTotal: 1032460 kB
MemFree: 115332 kB
Buffers: 0 kB
Cached: 206260 kB
SwapCached: 0 kB
Active: 126616 kB
Inactive: 167564 kB
Active(anon): 95440 kB
Inactive(anon): 24552 kB
Active(file): 31176 kB
Inactive(file): 143012 kB
Unevictable: 7392 kB
Mlocked: 0 kB
HighTotal: 270336 kB
HighFree: 1164 kB
LowTotal: 762124 kB
LowFree: 114168 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 95272 kB
Mapped: 76248 kB
Shmem: 24680 kB
Slab: 6876 kB
SReclaimable: 1912 kB
SUnreclaim: 4964 kB
KernelStack: 3024 kB
PageTables: 5972 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 516228 kB
Committed_AS: 857028 kB
VmallocTotal: 245760 kB
VmallocUsed: 35416 kB
VmallocChunk: 193752 kB
[-- Attachment #3: vmstat-working --]
[-- Type: application/octet-stream, Size: 1571 bytes --]
nr_free_pages 28886
nr_inactive_anon 6138
nr_active_anon 23868
nr_inactive_file 35753
nr_active_file 7794
nr_unevictable 1848
nr_mlock 0
nr_anon_pages 23842
nr_mapped 19062
nr_file_pages 51565
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 478
nr_slab_unreclaimable 1250
nr_page_table_pages 1493
nr_kernel_stack 378
nr_unstable 0
nr_bounce 0
nr_vmscan_write 1
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 6170
nr_dirtied 6
nr_written 6
nr_anon_transparent_hugepages 0
nr_free_cma 25186
nr_dirty_threshold 11769
nr_dirty_background_threshold 5884
pgpgin 0
pgpgout 0
pswpin 0
pswpout 0
pgalloc_normal 663824
pgalloc_high 259115
pgalloc_movable 0
pgfree 1105572
pgactivate 25919
pgdeactivate 16972
pgfault 275613
pgmajfault 575
pgrefill_normal 0
pgrefill_high 23191
pgrefill_movable 0
pgsteal_kswapd_normal 56931
pgsteal_kswapd_high 159750
pgsteal_kswapd_movable 0
pgsteal_direct_normal 0
pgsteal_direct_high 0
pgsteal_direct_movable 0
pgscan_kswapd_normal 64349
pgscan_kswapd_high 374239
pgscan_kswapd_movable 0
pgscan_direct_normal 0
pgscan_direct_high 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
pginodesteal 0
slabs_scanned 128
kswapd_inodesteal 0
kswapd_low_wmark_hit_quickly 45
kswapd_high_wmark_hit_quickly 8
kswapd_skip_congestion_wait 5719
pageoutrun 4570
allocstall 0
pgrotated 1
pgmigrate_success 382
pgmigrate_fail 0
unevictable_pgs_culled 1848
unevictable_pgs_scanned 0
unevictable_pgs_rescued 0
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
[-- Attachment #4: zoneinfo-working --]
[-- Type: application/octet-stream, Size: 2386 bytes --]
Node 0, zone Normal
pages free 27099
min 878
low 1097
high 1317
scanned 0
spanned 194560
present 193040
managed 55756
nr_free_pages 27099
nr_inactive_anon 2771
nr_active_anon 12059
nr_inactive_file 28965
nr_active_file 6688
nr_unevictable 5
nr_mlock 0
nr_anon_pages 12054
nr_mapped 14374
nr_file_pages 38445
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 481
nr_slab_unreclaimable 1401
nr_page_table_pages 1488
nr_kernel_stack 380
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 2787
nr_dirtied 0
nr_written 0
nr_anon_transparent_hugepages 0
nr_free_cma 25139
protection: (0, 2095, 2095)
pagesets
cpu: 0
count: 152
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 177
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
start_pfn: 0
inactive_ratio: 1
Node 0, zone HighMem
pages free 186
min 65
low 141
high 217
scanned 2
spanned 67584
present 67056
managed 67584
nr_free_pages 186
nr_inactive_anon 3139
nr_active_anon 10560
nr_inactive_file 9935
nr_active_file 796
nr_unevictable 1842
nr_mlock 0
nr_anon_pages 10537
nr_mapped 4560
nr_file_pages 15736
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 1
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 3163
nr_dirtied 2
nr_written 2
nr_anon_transparent_hugepages 0
nr_free_cma 196
protection: (0, 0, 0)
pagesets
cpu: 0
count: 15
high: 90
batch: 15
vm stats threshold: 12
cpu: 1
count: 70
high: 90
batch: 15
vm stats threshold: 12
all_unreclaimable: 0
start_pfn: 194560
inactive_ratio: 1
[-- Attachment #5: meminfo-bogus --]
[-- Type: application/octet-stream, Size: 1036 bytes --]
MemTotal: 1032452 kB
MemFree: 203612 kB
Buffers: 0 kB
Cached: 195468 kB
SwapCached: 0 kB
Active: 23576 kB
Inactive: 183612 kB
Active(anon): 19196 kB
Inactive(anon): 12 kB
Active(file): 4380 kB
Inactive(file): 183600 kB
Unevictable: 7380 kB
Mlocked: 0 kB
HighTotal: 270336 kB
HighFree: 76524 kB
LowTotal: 762116 kB
LowFree: 127088 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 19188 kB
Mapped: 29184 kB
Shmem: 20 kB
Slab: 7380 kB
SReclaimable: 1692 kB
SUnreclaim: 5688 kB
KernelStack: 2704 kB
PageTables: 4832 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 516224 kB
Committed_AS: 459304 kB
VmallocTotal: 245760 kB
VmallocUsed: 35416 kB
VmallocChunk: 193752 kB
[-- Attachment #6: vmstat-bogus --]
[-- Type: application/octet-stream, Size: 1699 bytes --]
nr_free_pages 47589
nr_inactive_anon 3
nr_active_anon 4813
nr_inactive_file 48554
nr_active_file 1482
nr_unevictable 1845
nr_mlock 0
nr_anon_pages 4817
nr_mapped 7642
nr_file_pages 51929
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 423
nr_slab_unreclaimable 1506
nr_page_table_pages 1208
nr_kernel_stack 338
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 32
nr_shmem 5
nr_dirtied 2
nr_written 2
nr_anon_transparent_hugepages 0
nr_free_cma 40290
nr_dirty_threshold 14334
nr_dirty_background_threshold 7167
pgpgin 0
pgpgout 0
pswpin 0
pswpout 0
pgalloc_normal 2057267
pgalloc_high 1034733
pgalloc_movable 0
pgfree 3293327
pgactivate 10498
pgdeactivate 9001
pgfault 121174
pgmajfault 1611
pgrefill_normal 0
pgrefill_high 19017
pgrefill_movable 0
pgsteal_kswapd_normal 9056
pgsteal_kswapd_high 988279
pgsteal_kswapd_movable 0
pgsteal_direct_normal 0
pgsteal_direct_high 0
pgsteal_direct_movable 0
pgscan_kswapd_normal 10081
pgscan_kswapd_high 8227540
pgscan_kswapd_movable 0
pgscan_direct_normal 0
pgscan_direct_high 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
pginodesteal 0
slabs_scanned 128
kswapd_inodesteal 0
kswapd_low_wmark_hit_quickly 3705
kswapd_high_wmark_hit_quickly 449
kswapd_skip_congestion_wait 7935
pageoutrun 13170
allocstall 0
pgrotated 1
pgmigrate_success 283
pgmigrate_fail 0
compact_migrate_scanned 148723
compact_free_scanned 161
compact_isolated 165833
compact_stall 0
compact_fail 0
compact_success 0
unevictable_pgs_culled 1845
unevictable_pgs_scanned 0
unevictable_pgs_rescued 0
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
[-- Attachment #7: zoneinfo-bogus --]
[-- Type: application/octet-stream, Size: 2372 bytes --]
Node 0, zone Normal
pages free 31813
min 878
low 1097
high 1317
scanned 0
spanned 194560
present 193040
managed 55754
nr_free_pages 31813
nr_inactive_anon 0
nr_active_anon 2782
nr_inactive_file 42529
nr_active_file 1043
nr_unevictable 3
nr_mlock 0
nr_anon_pages 2781
nr_mapped 6621
nr_file_pages 43576
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 423
nr_slab_unreclaimable 1399
nr_page_table_pages 1208
nr_kernel_stack 337
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 1
nr_dirtied 0
nr_written 0
nr_anon_transparent_hugepages 0
nr_free_cma 24251
protection: (0, 2095, 2095)
pagesets
cpu: 0
count: 34
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 168
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
start_pfn: 0
inactive_ratio: 1
Node 0, zone HighMem
pages free 20799
min 65
low 141
high 217
scanned 0
spanned 67584
present 67056
managed 67584
nr_free_pages 20799
nr_inactive_anon 3
nr_active_anon 2027
nr_inactive_file 1233
nr_active_file 545
nr_unevictable 1842
nr_mlock 0
nr_anon_pages 2032
nr_mapped 1131
nr_file_pages 3641
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_vmscan_immediate_reclaim 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 4
nr_dirtied 0
nr_written 0
nr_anon_transparent_hugepages 0
nr_free_cma 20703
protection: (0, 0, 0)
pagesets
cpu: 0
count: 73
high: 90
batch: 15
vm stats threshold: 12
cpu: 1
count: 7
high: 90
batch: 15
vm stats threshold: 12
all_unreclaimable: 0
start_pfn: 194560
inactive_ratio: 1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: high kswapd CPU usage when executing binaries from NFS w/ CMA and COMPACTION
2013-12-04 2:30 high kswapd CPU usage when executing binaries from NFS w/ CMA and COMPACTION Florian Fainelli
@ 2013-12-04 13:00 ` Mel Gorman
2013-12-06 22:51 ` Florian Fainelli
0 siblings, 1 reply; 3+ messages in thread
From: Mel Gorman @ 2013-12-04 13:00 UTC (permalink / raw)
To: Florian Fainelli
Cc: linux-kernel, mhocko, hannes, riel, linux-mm, m.szyprowski, marc.ceeeee
On Tue, Dec 03, 2013 at 06:30:28PM -0800, Florian Fainelli wrote:
> Hi all,
>
> I am experiencing high kswapd CPU usage on an ARMv7 system running
> 3.8.13 when executing relatively large binaries from NFS. When this
> happens kswapd consumes around 55-60% CPU usage and the applications
> takes a huge time to load.
>
There were a number of changes made related to how and when kswapd
stalls, particularly when pages are dirty. Brief check confirms that
git log v3.8..v3.12 --pretty=one --author "Mel Gorman" mm/vmscan.c
NFS dirty pages are problematic for compaction as dirty pages cannot be
migrated until cleaned. I'd suggest checking if current mainline suffers
the same problem and if not, focus on patches related to dirty page
handling and kswapd throttling in mm/vmscan.c as backport candidates.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: high kswapd CPU usage when executing binaries from NFS w/ CMA and COMPACTION
2013-12-04 13:00 ` Mel Gorman
@ 2013-12-06 22:51 ` Florian Fainelli
0 siblings, 0 replies; 3+ messages in thread
From: Florian Fainelli @ 2013-12-06 22:51 UTC (permalink / raw)
To: Mel Gorman
Cc: linux-kernel, mhocko, hannes, riel, linux-mm, m.szyprowski, marc.ceeeee
2013/12/4 Mel Gorman <mgorman@suse.de>:
> On Tue, Dec 03, 2013 at 06:30:28PM -0800, Florian Fainelli wrote:
>> Hi all,
>>
>> I am experiencing high kswapd CPU usage on an ARMv7 system running
>> 3.8.13 when executing relatively large binaries from NFS. When this
>> happens kswapd consumes around 55-60% CPU usage and the applications
>> takes a huge time to load.
>>
>
> There were a number of changes made related to how and when kswapd
> stalls, particularly when pages are dirty. Brief check confirms that
>
> git log v3.8..v3.12 --pretty=one --author "Mel Gorman" mm/vmscan.c
>
> NFS dirty pages are problematic for compaction as dirty pages cannot be
> migrated until cleaned. I'd suggest checking if current mainline suffers
> the same problem and if not, focus on patches related to dirty page
> handling and kswapd throttling in mm/vmscan.c as backport candidates.
I have just backported these patches to 3.8.13 and am still seeing the
problem, although kswapd usage dropped considerably (by half
approximately). Will keep you updated once I have properly tested
current mainline on my platform. Thanks!
--
Florian
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-12-06 22:51 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-12-04 2:30 high kswapd CPU usage when executing binaries from NFS w/ CMA and COMPACTION Florian Fainelli
2013-12-04 13:00 ` Mel Gorman
2013-12-06 22:51 ` Florian Fainelli
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).