From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C6CC43331 for ; Fri, 6 Sep 2019 10:08:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 978FF2082C for ; Fri, 6 Sep 2019 10:08:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 978FF2082C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=profihost.ag Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 453726B0007; Fri, 6 Sep 2019 06:08:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 402DF6B0008; Fri, 6 Sep 2019 06:08:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 319DC6B000A; Fri, 6 Sep 2019 06:08:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 134646B0007 for ; Fri, 6 Sep 2019 06:08:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id A6AC5824CA3A for ; Fri, 6 Sep 2019 10:08:35 +0000 (UTC) X-FDA: 75904071390.17.dolls38_4aef796450c22 X-HE-Tag: dolls38_4aef796450c22 X-Filterd-Recvd-Size: 7890 Received: from cloud1-vm154.de-nserver.de (cloud1-vm154.de-nserver.de [178.250.10.56]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 6 Sep 2019 10:08:34 +0000 (UTC) Received: (qmail 17846 invoked from network); 6 Sep 2019 12:08:32 +0200 X-Fcrdns: No Received: from phoffice.de-nserver.de (HELO [10.242.2.9]) (185.39.223.5) (smtp-auth username hostmaster@profihost.com, mechanism plain) by cloud1-vm154.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPSA; Fri, 06 Sep 2019 12:08:32 +0200 Subject: Re: lot of MemAvailable but falling cache and raising PSI From: Stefan Priebe - Profihost AG To: Michal Hocko Cc: "linux-mm@kvack.org" , l.roehrs@profihost.ag, cgroups@vger.kernel.org, Johannes Weiner References: <4b4ba042-3741-7b16-2292-198c569da2aa@profihost.ag> <20190905114022.GH3838@dhcp22.suse.cz> <7a3d23f2-b5fe-b4c0-41cd-e79070637bd9@profihost.ag> Message-ID: Date: Fri, 6 Sep 2019 12:08:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <7a3d23f2-b5fe-b4c0-41cd-e79070637bd9@profihost.ag> Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 7bit X-User-Auth: Auth by hostmaster@profihost.com through 185.39.223.5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are the biggest differences in meminfo before and after cached starts to drop. I didn't expect cached end up in MemFree. Before: MemTotal: 16423116 kB MemFree: 374572 kB MemAvailable: 5633816 kB Cached: 5550972 kB Inactive: 4696580 kB Inactive(file): 3624776 kB After: MemTotal: 16423116 kB MemFree: 3477168 kB MemAvailable: 6066916 kB Cached: 2724504 kB Inactive: 1854740 kB Inactive(file): 950680 kB Any explanation? Greets, Stefan Am 05.09.19 um 13:56 schrieb Stefan Priebe - Profihost AG: > > Am 05.09.19 um 13:40 schrieb Michal Hocko: >> On Thu 05-09-19 13:27:10, Stefan Priebe - Profihost AG wrote: >>> Hello all, >>> >>> i hope you can help me again to understand the current MemAvailable >>> value in the linux kernel. I'm running a 4.19.52 kernel + psi patches in >>> this case. >>> >>> I'm seeing the following behaviour i don't understand and ask for help. >>> >>> While MemAvailable shows 5G the kernel starts to drop cache from 4G down >>> to 1G while the apache spawns some PHP processes. After that the PSI >>> mem.some value rises and the kernel tries to reclaim memory but >>> MemAvailable stays at 5G. >>> >>> Any ideas? >> >> Can you collect /proc/vmstat (every second or so) and post it while this >> is the case please? > > Yes sure. > > But i don't know which event you mean exactly. Current situation is PSI > / memory pressure is > 20 but: > > This is the current status where MemAvailable show 5G but Cached is > already dropped to 1G coming from 4G: > > > meminfo: > MemTotal: 16423116 kB > MemFree: 5280736 kB > MemAvailable: 5332752 kB > Buffers: 2572 kB > Cached: 1225112 kB > SwapCached: 0 kB > Active: 8934976 kB > Inactive: 1026900 kB > Active(anon): 8740396 kB > Inactive(anon): 873448 kB > Active(file): 194580 kB > Inactive(file): 153452 kB > Unevictable: 19900 kB > Mlocked: 19900 kB > SwapTotal: 0 kB > SwapFree: 0 kB > Dirty: 1980 kB > Writeback: 0 kB > AnonPages: 8423480 kB > Mapped: 978212 kB > Shmem: 875680 kB > Slab: 839868 kB > SReclaimable: 383396 kB > SUnreclaim: 456472 kB > KernelStack: 22576 kB > PageTables: 49824 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 8211556 kB > Committed_AS: 32060624 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > Percpu: 118048 kB > HardwareCorrupted: 0 kB > AnonHugePages: 6406144 kB > ShmemHugePages: 0 kB > ShmemPmdMapped: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > Hugetlb: 0 kB > DirectMap4k: 2580336 kB > DirectMap2M: 14196736 kB > DirectMap1G: 2097152 kB > > > vmstat shows: > nr_free_pages 1320053 > nr_zone_inactive_anon 218362 > nr_zone_active_anon 2185108 > nr_zone_inactive_file 38363 > nr_zone_active_file 48645 > nr_zone_unevictable 4975 > nr_zone_write_pending 495 > nr_mlock 4975 > nr_page_table_pages 12553 > nr_kernel_stack 22576 > nr_bounce 0 > nr_zspages 0 > nr_free_cma 0 > numa_hit 13916119899 > numa_miss 0 > numa_foreign 0 > numa_interleave 15629 > numa_local 13916119899 > numa_other 0 > nr_inactive_anon 218362 > nr_active_anon 2185164 > nr_inactive_file 38363 > nr_active_file 48645 > nr_unevictable 4975 > nr_slab_reclaimable 95849 > nr_slab_unreclaimable 114118 > nr_isolated_anon 0 > nr_isolated_file 0 > workingset_refault 71365357 > workingset_activate 20281670 > workingset_restore 8995665 > workingset_nodereclaim 326085 > nr_anon_pages 2105903 > nr_mapped 244553 > nr_file_pages 306921 > nr_dirty 495 > nr_writeback 0 > nr_writeback_temp 0 > nr_shmem 218920 > nr_shmem_hugepages 0 > nr_shmem_pmdmapped 0 > nr_anon_transparent_hugepages 3128 > nr_unstable 0 > nr_vmscan_write 0 > nr_vmscan_immediate_reclaim 1833104 > nr_dirtied 386544087 > nr_written 259220036 > nr_dirty_threshold 265636 > nr_dirty_background_threshold 132656 > pgpgin 1817628997 > pgpgout 3730818029 > pswpin 0 > pswpout 0 > pgalloc_dma 0 > pgalloc_dma32 5790777997 > pgalloc_normal 20003662520 > pgalloc_movable 0 > allocstall_dma 0 > allocstall_dma32 0 > allocstall_normal 39 > allocstall_movable 1980089 > pgskip_dma 0 > pgskip_dma32 0 > pgskip_normal 0 > pgskip_movable 0 > pgfree 26637215947 > pgactivate 316722654 > pgdeactivate 261039211 > pglazyfree 0 > pgfault 17719356599 > pgmajfault 30985544 > pglazyfreed 0 > pgrefill 286826568 > pgsteal_kswapd 36740923 > pgsteal_direct 349291470 > pgscan_kswapd 36878966 > pgscan_direct 395327492 > pgscan_direct_throttle 0 > zone_reclaim_failed 0 > pginodesteal 49817087 > slabs_scanned 597956834 > kswapd_inodesteal 1412447 > kswapd_low_wmark_hit_quickly 39 > kswapd_high_wmark_hit_quickly 319 > pageoutrun 3585 > pgrotated 2873743 > drop_pagecache 0 > drop_slab 0 > oom_kill 0 > pgmigrate_success 839062285 > pgmigrate_fail 507313 > compact_migrate_scanned 9619077010 > compact_free_scanned 67985619651 > compact_isolated 1684537704 > compact_stall 205761 > compact_fail 182420 > compact_success 23341 > compact_daemon_wake 2 > compact_daemon_migrate_scanned 811 > compact_daemon_free_scanned 490241 > htlb_buddy_alloc_success 0 > htlb_buddy_alloc_fail 0 > unevictable_pgs_culled 1006521 > unevictable_pgs_scanned 0 > unevictable_pgs_rescued 997077 > unevictable_pgs_mlocked 1319203 > unevictable_pgs_munlocked 842471 > unevictable_pgs_cleared 470531 > unevictable_pgs_stranded 459613 > thp_fault_alloc 20263113 > thp_fault_fallback 3368635 > thp_collapse_alloc 226476 > thp_collapse_alloc_failed 17594 > thp_file_alloc 0 > thp_file_mapped 0 > thp_split_page 1159 > thp_split_page_failed 3927 > thp_deferred_split_page 20348941 > thp_split_pmd 53361 > thp_split_pud 0 > thp_zero_page_alloc 1 > thp_zero_page_alloc_failed 0 > thp_swpout 0 > thp_swpout_fallback 0 > balloon_inflate 0 > balloon_deflate 0 > balloon_migrate 0 > swap_ra 0 > swap_ra_hit 0 > > Greets, > Stefan >