Re: lot of MemAvailable but falling cache and raising PSI

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Am 05.09.19 um 13:40 schrieb Michal Hocko:
> On Thu 05-09-19 13:27:10, Stefan Priebe - Profihost AG wrote:
>> Hello all,
>>
>> i hope you can help me again to understand the current MemAvailable
>> value in the linux kernel. I'm running a 4.19.52 kernel + psi patches in
>> this case.
>>
>> I'm seeing the following behaviour i don't understand and ask for help.
>>
>> While MemAvailable shows 5G the kernel starts to drop cache from 4G down
>> to 1G while the apache spawns some PHP processes. After that the PSI
>> mem.some value rises and the kernel tries to reclaim memory but
>> MemAvailable stays at 5G.
>>
>> Any ideas?
> 
> Can you collect /proc/vmstat (every second or so) and post it while this
> is the case please?

Yes sure.

But i don't know which event you mean exactly. Current situation is PSI
/ memory pressure is > 20 but:

This is the current status where MemAvailable show 5G but Cached is
already dropped to 1G coming from 4G:


meminfo:
MemTotal:       16423116 kB
MemFree:         5280736 kB
MemAvailable:    5332752 kB
Buffers:            2572 kB
Cached:          1225112 kB
SwapCached:            0 kB
Active:          8934976 kB
Inactive:        1026900 kB
Active(anon):    8740396 kB
Inactive(anon):   873448 kB
Active(file):     194580 kB
Inactive(file):   153452 kB
Unevictable:       19900 kB
Mlocked:           19900 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1980 kB
Writeback:             0 kB
AnonPages:       8423480 kB
Mapped:           978212 kB
Shmem:            875680 kB
Slab:             839868 kB
SReclaimable:     383396 kB
SUnreclaim:       456472 kB
KernelStack:       22576 kB
PageTables:        49824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8211556 kB
Committed_AS:   32060624 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:           118048 kB
HardwareCorrupted:     0 kB
AnonHugePages:   6406144 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     2580336 kB
DirectMap2M:    14196736 kB
DirectMap1G:     2097152 kB


vmstat shows:
nr_free_pages 1320053
nr_zone_inactive_anon 218362
nr_zone_active_anon 2185108
nr_zone_inactive_file 38363
nr_zone_active_file 48645
nr_zone_unevictable 4975
nr_zone_write_pending 495
nr_mlock 4975
nr_page_table_pages 12553
nr_kernel_stack 22576
nr_bounce 0
nr_zspages 0
nr_free_cma 0
numa_hit 13916119899
numa_miss 0
numa_foreign 0
numa_interleave 15629
numa_local 13916119899
numa_other 0
nr_inactive_anon 218362
nr_active_anon 2185164
nr_inactive_file 38363
nr_active_file 48645
nr_unevictable 4975
nr_slab_reclaimable 95849
nr_slab_unreclaimable 114118
nr_isolated_anon 0
nr_isolated_file 0
workingset_refault 71365357
workingset_activate 20281670
workingset_restore 8995665
workingset_nodereclaim 326085
nr_anon_pages 2105903
nr_mapped 244553
nr_file_pages 306921
nr_dirty 495
nr_writeback 0
nr_writeback_temp 0
nr_shmem 218920
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_anon_transparent_hugepages 3128
nr_unstable 0
nr_vmscan_write 0
nr_vmscan_immediate_reclaim 1833104
nr_dirtied 386544087
nr_written 259220036
nr_dirty_threshold 265636
nr_dirty_background_threshold 132656
pgpgin 1817628997
pgpgout 3730818029
pswpin 0
pswpout 0
pgalloc_dma 0
pgalloc_dma32 5790777997
pgalloc_normal 20003662520
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 39
allocstall_movable 1980089
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 0
pgskip_movable 0
pgfree 26637215947
pgactivate 316722654
pgdeactivate 261039211
pglazyfree 0
pgfault 17719356599
pgmajfault 30985544
pglazyfreed 0
pgrefill 286826568
pgsteal_kswapd 36740923
pgsteal_direct 349291470
pgscan_kswapd 36878966
pgscan_direct 395327492
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 49817087
slabs_scanned 597956834
kswapd_inodesteal 1412447
kswapd_low_wmark_hit_quickly 39
kswapd_high_wmark_hit_quickly 319
pageoutrun 3585
pgrotated 2873743
drop_pagecache 0
drop_slab 0
oom_kill 0
pgmigrate_success 839062285
pgmigrate_fail 507313
compact_migrate_scanned 9619077010
compact_free_scanned 67985619651
compact_isolated 1684537704
compact_stall 205761
compact_fail 182420
compact_success 23341
compact_daemon_wake 2
compact_daemon_migrate_scanned 811
compact_daemon_free_scanned 490241
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 1006521
unevictable_pgs_scanned 0
unevictable_pgs_rescued 997077
unevictable_pgs_mlocked 1319203
unevictable_pgs_munlocked 842471
unevictable_pgs_cleared 470531
unevictable_pgs_stranded 459613
thp_fault_alloc 20263113
thp_fault_fallback 3368635
thp_collapse_alloc 226476
thp_collapse_alloc_failed 17594
thp_file_alloc 0
thp_file_mapped 0
thp_split_page 1159
thp_split_page_failed 3927
thp_deferred_split_page 20348941
thp_split_pmd 53361
thp_split_pud 0
thp_zero_page_alloc 1
thp_zero_page_alloc_failed 0
thp_swpout 0
thp_swpout_fallback 0
balloon_inflate 0
balloon_deflate 0
balloon_migrate 0
swap_ra 0
swap_ra_hit 0

Greets,
Stefan