Re: lot of MemAvailable but falling cache and raising PSI

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Am 05.09.19 um 20:46 schrieb Yang Shi:
> On Thu, Sep 5, 2019 at 10:26 AM Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>>
>> Hi,
>> Am 05.09.19 um 18:28 schrieb Yang Shi:
>>> On Thu, Sep 5, 2019 at 4:56 AM Stefan Priebe - Profihost AG
>>> <s.priebe@profihost.ag> wrote:
>>>>
>>>>
>>>> Am 05.09.19 um 13:40 schrieb Michal Hocko:
>>>>> On Thu 05-09-19 13:27:10, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> i hope you can help me again to understand the current MemAvailable
>>>>>> value in the linux kernel. I'm running a 4.19.52 kernel + psi patches in
>>>>>> this case.
>>>>>>
>>>>>> I'm seeing the following behaviour i don't understand and ask for help.
>>>>>>
>>>>>> While MemAvailable shows 5G the kernel starts to drop cache from 4G down
>>>>>> to 1G while the apache spawns some PHP processes. After that the PSI
>>>>>> mem.some value rises and the kernel tries to reclaim memory but
>>>>>> MemAvailable stays at 5G.
>>>>>>
>>>>>> Any ideas?
>>>>>
>>>>> Can you collect /proc/vmstat (every second or so) and post it while this
>>>>> is the case please?
>>>>
>>>> Yes sure.
>>>>
>>>> But i don't know which event you mean exactly. Current situation is PSI
>>>> / memory pressure is > 20 but:
>>>>
>>>> This is the current status where MemAvailable show 5G but Cached is
>>>> already dropped to 1G coming from 4G:
>>>
>>> I don't get what problem you are running into. MemAvailable is *not*
>>> the indication for triggering memory reclaim.
>>
>> Yes it's not sure. But i don't get why:
>> * PSI is raising and Caches are dropped when MemAvail and MemFree show 5GB
> 
> You need check your water mark (/proc/min_free_kbytes,
> /proc/watermark_scale_factor and /proc/zoneinfo) setting why kswapd is
> launched when there is 5 GB free memory.

sure i did but can't find anything:
# cat /proc/sys/vm/min_free_kbytes
164231

# cat /proc/sys/vm/watermark_scale_factor
10


# cat /proc/zoneinfo
Node 0, zone      DMA
  per-node stats
      nr_inactive_anon 177046
      nr_active_anon 1718836
      nr_inactive_file 288146
      nr_active_file 121497
      nr_unevictable 5510
      nr_slab_reclaimable 301721
      nr_slab_unreclaimable 119276
      nr_isolated_anon 0
      nr_isolated_file 0
      workingset_refault 72376392
      workingset_activate 20641006
      workingset_restore 9149962
      workingset_nodereclaim 326469
      nr_anon_pages 1647524
      nr_mapped    211704
      nr_file_pages 587984
      nr_dirty     212
      nr_writeback 0
      nr_writeback_temp 0
      nr_shmem     177458
      nr_shmem_hugepages 0
      nr_shmem_pmdmapped 0
      nr_anon_transparent_hugepages 2480
      nr_unstable  0
      nr_vmscan_write 0
      nr_vmscan_immediate_reclaim 1843759
      nr_dirtied   388618149
      nr_written   260643754
  pages free     3977
        min      39
        low      48
        high     57
        spanned  4095
        present  3998
        managed  3977
        protection: (0, 2968, 16022, 16022, 16022)
      nr_free_pages 3977
      nr_zone_inactive_anon 0
      nr_zone_active_anon 0
      nr_zone_inactive_file 0
      nr_zone_active_file 0
      nr_zone_unevictable 0
      nr_zone_write_pending 0
      nr_mlock     0
      nr_page_table_pages 0
      nr_kernel_stack 0
      nr_bounce    0
      nr_zspages   0
      nr_free_cma  0
      numa_hit     0
      numa_miss    0
      numa_foreign 0
      numa_interleave 0
      numa_local   0
      numa_other   0
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 4
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 5
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 6
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 7
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
  node_unreclaimable:  0
  start_pfn:           1
Node 0, zone    DMA32
  pages free     439019
        min      7600
        low      9500
        high     11400
        spanned  1044480
        present  782300
        managed  760023
        protection: (0, 0, 13053, 13053, 13053)
      nr_free_pages 439019
      nr_zone_inactive_anon 0
      nr_zone_active_anon 309777
      nr_zone_inactive_file 809
      nr_zone_active_file 645
      nr_zone_unevictable 2048
      nr_zone_write_pending 1
      nr_mlock     2048
      nr_page_table_pages 8
      nr_kernel_stack 32
      nr_bounce    0
      nr_zspages   0
      nr_free_cma  0
      numa_hit     213697054
      numa_miss    0
      numa_foreign 0
      numa_interleave 0
      numa_local   213697054
      numa_other   0
  pagesets
    cpu: 0
              count: 0
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 1
              count: 1
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 2
              count: 338
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 3
              count: 10
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 4
              count: 0
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 5
              count: 324
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 6
              count: 136
              high:  378
              batch: 63
  vm stats threshold: 48
    cpu: 7
              count: 1
              high:  378
              batch: 63
  vm stats threshold: 48
  node_unreclaimable:  0
  start_pfn:           4096
Node 0, zone   Normal
  pages free     734519
        min      33417
        low      41771
        high     50125
        spanned  3407872
        present  3407872
        managed  3341779
        protection: (0, 0, 0, 0, 0)
      nr_free_pages 734519
      nr_zone_inactive_anon 177046
      nr_zone_active_anon 1409059
      nr_zone_inactive_file 287337
      nr_zone_active_file 120852
      nr_zone_unevictable 3462
      nr_zone_write_pending 211
      nr_mlock     3462
      nr_page_table_pages 10551
      nr_kernel_stack 22464
      nr_bounce    0
      nr_zspages   0
      nr_free_cma  0
      numa_hit     13801352577
      numa_miss    0
      numa_foreign 0
      numa_interleave 15629
      numa_local   13801352577
      numa_other   0
  pagesets
    cpu: 0
              count: 12
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 1
              count: 40
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 2
              count: 41
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 3
              count: 41
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 4
              count: 37
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 5
              count: 39
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 6
              count: 19
              high:  42
              batch: 7
  vm stats threshold: 64
    cpu: 7
              count: 9
              high:  42
              batch: 7
  vm stats threshold: 64
  node_unreclaimable:  0
  start_pfn:           1048576
Node 0, zone  Movable
  pages free     0
        min      0
        low      0
        high     0
        spanned  0
        present  0
        managed  0
        protection: (0, 0, 0, 0, 0)
Node 0, zone   Device
  pages free     0
        min      0
        low      0
        high     0
        spanned  0
        present  0
        managed  0
        protection: (0, 0, 0, 0, 0)

>>> Basically MemAvailable = MemFree + page cache (active file + inactive
>>> file) / 2 + SReclaimable / 2, which means that much memory could be
>>> reclaimed if memory pressure is hit.
>>
>> Yes but MemFree also shows 5G in this case see below and still file
>> cache gets dropped and PSI is rising.
>>
>>> But, memory pressure (tracked by PSI) is triggered by how much memory
>>> (aka watermark) is consumed.
>> What does this exactly mean?
> 
> cat /proc/zoneinfo, it would show something like:
> 
> pages free     4118641
>         min      12470
>         low      16598
>         high     20726
> 
> Here min/low/high are the so-called "water mark". When free memory is
> lower than low, kswapd would be launched.
> 
>>
>>> So, it looks page reclaim logic just reclaimed file cache (it looks
>>> sane since your VM doesn't have swap partition), so I'm supposed you
>>> would see MemFree increased along with dropping "Cached",
>>
>> No it does not. MemFree and MemAvail stay constant at 5G.
>>
>>> but
>>> MemAvailable basically is not changed. It looks sane to me. Am I
>>> missing something else?
>>
>> I ever thought the kerne would not free the cache nor PSI gets rising
>> when there are 5GB in MemFree and in MemAvail. This makes still no sense
>> to me. Why drop the cache when you have 5G free. This results currently
>> in I/O waits as the page was dropped.
>>
>> Greets,
>> Stefan
>>
>>>>
>>>> meminfo:
>>>> MemTotal:       16423116 kB
>>>> MemFree:         5280736 kB
>>>> MemAvailable:    5332752 kB
>>>> Buffers:            2572 kB
>>>> Cached:          1225112 kB
>>>> SwapCached:            0 kB
>>>> Active:          8934976 kB
>>>> Inactive:        1026900 kB
>>>> Active(anon):    8740396 kB
>>>> Inactive(anon):   873448 kB
>>>> Active(file):     194580 kB
>>>> Inactive(file):   153452 kB
>>>> Unevictable:       19900 kB
>>>> Mlocked:           19900 kB
>>>> SwapTotal:             0 kB
>>>> SwapFree:              0 kB
>>>> Dirty:              1980 kB
>>>> Writeback:             0 kB
>>>> AnonPages:       8423480 kB
>>>> Mapped:           978212 kB
>>>> Shmem:            875680 kB
>>>> Slab:             839868 kB
>>>> SReclaimable:     383396 kB
>>>> SUnreclaim:       456472 kB
>>>> KernelStack:       22576 kB
>>>> PageTables:        49824 kB
>>>> NFS_Unstable:          0 kB
>>>> Bounce:                0 kB
>>>> WritebackTmp:          0 kB
>>>> CommitLimit:     8211556 kB
>>>> Committed_AS:   32060624 kB
>>>> VmallocTotal:   34359738367 kB
>>>> VmallocUsed:           0 kB
>>>> VmallocChunk:          0 kB
>>>> Percpu:           118048 kB
>>>> HardwareCorrupted:     0 kB
>>>> AnonHugePages:   6406144 kB
>>>> ShmemHugePages:        0 kB
>>>> ShmemPmdMapped:        0 kB
>>>> HugePages_Total:       0
>>>> HugePages_Free:        0
>>>> HugePages_Rsvd:        0
>>>> HugePages_Surp:        0
>>>> Hugepagesize:       2048 kB
>>>> Hugetlb:               0 kB
>>>> DirectMap4k:     2580336 kB
>>>> DirectMap2M:    14196736 kB
>>>> DirectMap1G:     2097152 kB
>>>>
>>>>
>>>> vmstat shows:
>>>> nr_free_pages 1320053
>>>> nr_zone_inactive_anon 218362
>>>> nr_zone_active_anon 2185108
>>>> nr_zone_inactive_file 38363
>>>> nr_zone_active_file 48645
>>>> nr_zone_unevictable 4975
>>>> nr_zone_write_pending 495
>>>> nr_mlock 4975
>>>> nr_page_table_pages 12553
>>>> nr_kernel_stack 22576
>>>> nr_bounce 0
>>>> nr_zspages 0
>>>> nr_free_cma 0
>>>> numa_hit 13916119899
>>>> numa_miss 0
>>>> numa_foreign 0
>>>> numa_interleave 15629
>>>> numa_local 13916119899
>>>> numa_other 0
>>>> nr_inactive_anon 218362
>>>> nr_active_anon 2185164
>>>> nr_inactive_file 38363
>>>> nr_active_file 48645
>>>> nr_unevictable 4975
>>>> nr_slab_reclaimable 95849
>>>> nr_slab_unreclaimable 114118
>>>> nr_isolated_anon 0
>>>> nr_isolated_file 0
>>>> workingset_refault 71365357
>>>> workingset_activate 20281670
>>>> workingset_restore 8995665
>>>> workingset_nodereclaim 326085
>>>> nr_anon_pages 2105903
>>>> nr_mapped 244553
>>>> nr_file_pages 306921
>>>> nr_dirty 495
>>>> nr_writeback 0
>>>> nr_writeback_temp 0
>>>> nr_shmem 218920
>>>> nr_shmem_hugepages 0
>>>> nr_shmem_pmdmapped 0
>>>> nr_anon_transparent_hugepages 3128
>>>> nr_unstable 0
>>>> nr_vmscan_write 0
>>>> nr_vmscan_immediate_reclaim 1833104
>>>> nr_dirtied 386544087
>>>> nr_written 259220036
>>>> nr_dirty_threshold 265636
>>>> nr_dirty_background_threshold 132656
>>>> pgpgin 1817628997
>>>> pgpgout 3730818029
>>>> pswpin 0
>>>> pswpout 0
>>>> pgalloc_dma 0
>>>> pgalloc_dma32 5790777997
>>>> pgalloc_normal 20003662520
>>>> pgalloc_movable 0
>>>> allocstall_dma 0
>>>> allocstall_dma32 0
>>>> allocstall_normal 39
>>>> allocstall_movable 1980089
>>>> pgskip_dma 0
>>>> pgskip_dma32 0
>>>> pgskip_normal 0
>>>> pgskip_movable 0
>>>> pgfree 26637215947
>>>> pgactivate 316722654
>>>> pgdeactivate 261039211
>>>> pglazyfree 0
>>>> pgfault 17719356599
>>>> pgmajfault 30985544
>>>> pglazyfreed 0
>>>> pgrefill 286826568
>>>> pgsteal_kswapd 36740923
>>>> pgsteal_direct 349291470
>>>> pgscan_kswapd 36878966
>>>> pgscan_direct 395327492
>>>> pgscan_direct_throttle 0
>>>> zone_reclaim_failed 0
>>>> pginodesteal 49817087
>>>> slabs_scanned 597956834
>>>> kswapd_inodesteal 1412447
>>>> kswapd_low_wmark_hit_quickly 39
>>>> kswapd_high_wmark_hit_quickly 319
>>>> pageoutrun 3585
>>>> pgrotated 2873743
>>>> drop_pagecache 0
>>>> drop_slab 0
>>>> oom_kill 0
>>>> pgmigrate_success 839062285
>>>> pgmigrate_fail 507313
>>>> compact_migrate_scanned 9619077010
>>>> compact_free_scanned 67985619651
>>>> compact_isolated 1684537704
>>>> compact_stall 205761
>>>> compact_fail 182420
>>>> compact_success 23341
>>>> compact_daemon_wake 2
>>>> compact_daemon_migrate_scanned 811
>>>> compact_daemon_free_scanned 490241
>>>> htlb_buddy_alloc_success 0
>>>> htlb_buddy_alloc_fail 0
>>>> unevictable_pgs_culled 1006521
>>>> unevictable_pgs_scanned 0
>>>> unevictable_pgs_rescued 997077
>>>> unevictable_pgs_mlocked 1319203
>>>> unevictable_pgs_munlocked 842471
>>>> unevictable_pgs_cleared 470531
>>>> unevictable_pgs_stranded 459613
>>>> thp_fault_alloc 20263113
>>>> thp_fault_fallback 3368635
>>>> thp_collapse_alloc 226476
>>>> thp_collapse_alloc_failed 17594
>>>> thp_file_alloc 0
>>>> thp_file_mapped 0
>>>> thp_split_page 1159
>>>> thp_split_page_failed 3927
>>>> thp_deferred_split_page 20348941
>>>> thp_split_pmd 53361
>>>> thp_split_pud 0
>>>> thp_zero_page_alloc 1
>>>> thp_zero_page_alloc_failed 0
>>>> thp_swpout 0
>>>> thp_swpout_fallback 0
>>>> balloon_inflate 0
>>>> balloon_deflate 0
>>>> balloon_migrate 0
>>>> swap_ra 0
>>>> swap_ra_hit 0
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>>