Re: lot of MemAvailable but falling cache and raising PSI - Stefan Priebe

From: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
To: Michal Hocko <mhocko@kernel.org>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
	l.roehrs@profihost.ag, cgroups@vger.kernel.org,
	Johannes Weiner <hannes@cmpxchg.org>,
	Vlastimil Babka <vbabka@suse.cz>
Subject: Re: lot of MemAvailable but falling cache and raising PSI
Date: Thu, 19 Sep 2019 12:21:15 +0200	[thread overview]
Message-ID: <2fe81a9e-5d29-79cf-f747-c66ae35defd0@profihost.ag> (raw)
In-Reply-To: <d07620d9-4967-40fe-fa0f-be51f2459dc5@profihost.ag>

Dear Michal,

Am 11.09.19 um 08:12 schrieb Stefan Priebe - Profihost AG:
> Hi Michal,
> Am 10.09.19 um 15:24 schrieb Michal Hocko:
>> On Tue 10-09-19 15:14:45, Stefan Priebe - Profihost AG wrote:
>>> Am 10.09.19 um 15:05 schrieb Stefan Priebe - Profihost AG:
>>>>
>>>> Am 10.09.19 um 14:57 schrieb Michal Hocko:
>>>>> On Tue 10-09-19 14:45:37, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello Michal,
>>>>>>
>>>>>> ok this might take a long time. Attached you'll find a graph from a
>>>>>> fresh boot what happens over time (here 17 August to 30 August). Memory
>>>>>> Usage decreases as well as cache but slowly and only over time and days.
>>>>>>
>>>>>> So it might take 2-3 weeks running Kernel 5.3 to see what happens.
>>>>>
>>>>> No problem. Just make sure to collect the requested data from the time
>>>>> you see the actual problem. Btw. you try my very dumb scriplets to get
>>>>> an idea of how much memory gets reclaimed due to THP.
>>>>
>>>> You mean your sed and sort on top of the trace file? No i did not with
>>>> the current 5.3 kernel do you think it will show anything interesting?
>>>> Which line shows me how much memory gets reclaimed due to THP?
>>
>> Please re-read http://lkml.kernel.org/r/20190910082919.GL2063@dhcp22.suse.cz
>> Each command has a commented output. If you see nunmber of reclaimed
>> pages to be large for GFP_TRANSHUGE then you are seeing a similar
>> problem.
>>
>>> Is something like a kernel memory leak possible? Or wouldn't this end up
>>> in having a lot of free memory which doesn't seem usable.
>>
>> I would be really surprised if this was the case.
>>
>>> I also wonder why a reclaim takes place when there is enough memory.
>>
>> This is not clear yet and it might be a bug that has been fixed since
>> 4.18. That's why we need to see whether the same is pattern is happening
>> with 5.3 as well.

Kernel 5.2.14 is now running since exactly 7 days and now we can easaly
view a trend i', not sure if i should post graphs.

Cache size is continuously shrinking while memfree is rising.

While there were 4,5GB free in avg in the beginnen we now have an avg of
8GB free memory.

Cache has shrinked from avg 24G to avg 18G.

Memory pressure has rised from avg 0% to avg 0.1% - not much but if you
look at the graphs it's continuously rising while cache is shrinking and
memfree is rising.

Which values should i collect now?

Greets,
Stefan