From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by kanga.kvack.org (Postfix) with ESMTP id 382176B000A for ; Fri, 13 Jul 2018 11:48:11 -0400 (EDT) Received: by mail-wr1-f72.google.com with SMTP id u1-v6so5590750wrs.18 for ; Fri, 13 Jul 2018 08:48:11 -0700 (PDT) Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id y2-v6sor1843996wmg.19.2018.07.13.08.48.09 for (Google Transport Security); Fri, 13 Jul 2018 08:48:09 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20180712113411.GB328@dhcp22.suse.cz> References: <20180712113411.GB328@dhcp22.suse.cz> From: Marinko Catovic Date: Fri, 13 Jul 2018 17:48:08 +0200 Message-ID: Subject: Re: Caching/buffers become useless after some time Content-Type: multipart/alternative; boundary="0000000000000b6ac30570e36833" Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org --0000000000000b6ac30570e36833 Content-Type: text/plain; charset="UTF-8" hello Michal well these hints were just ideas mentioned by some people, it took me weeks just to figure out that 2>drop_caches helps, still not knowing why this happens. Right now I am observing ~18GB of unused RAM, since yesterday, so this is not always about 100MB/3.5GB, but right now it may be in the process of shrinking. I really can not tell for sure, this is so nondeterministic - I just wish I could reproduce it for better testing. Right now top shows: KiB Mem : 65892044 total, 18169232 free, 11879604 used, 35843208 buff/cache Where 1GB goes to buffers, the rest to cache, the host *is* busy and the buff/cache consumed all RAM yesterday, where I did 2>drop_caches about one day before. Another host (still) shows full usage. That other one is 1:1 the same by software and config, but with different data/users; the use-cases and load are pretty much similar. Affected host at this time: https://pastebin.com/fRQMPuwb https://pastebin.com/tagXJRi1 .. 3 minutes later https://pastebin.com/8YNFfKXf .. 3 minutes later https://pastebin.com/UEq7NKR4 .. 3 minutes later To compare - this is the other host, that is still showing full buffers/cache usage by now: https://pastebin.com/Jraux2gy Usually both show this more or less at the same time, sometimes it is the one, sometimes the other. Other hosts I have are currently not under similar high load, making it even harder to compare. However, right now I can not observe this dropping towards really low values, but I am sure it will come. fs is ext4, mount options are auto,rw,data=writeback,noatime ,nodiratime,nodev,nosuid,async previous mount options with same behavior also had max_dir_size_kb, quotas and defaults for data= so I also played around with these, but that made no difference. --------- follow up (sorry, messed up with reply-to this mailing list): https://pastebin.com/0v4ZFNCv .. one hour later, right after my last report, 22GB free https://pastebin.com/rReWnHtE .. one day later, 28GB free It is interesting to see however, that this did not get that low as mentioned before. So not sure where this is going right now, but nevertheless, the RAM is not occupied fully, there should be no reason to allow 28GB to be free at all. Still lots I/O, and I am 100% positive that if I'd echo 2 > drop_caches, this would fill up the entire RAM again. What I can see is that buffers are around 500-700MB, the values increase and decrease all the time, really "oscillating" around 600. afaik this should get as high as possible, as long there is free ram - the other host that is still healthy has about 2GB/48GB fully occupying RAM. Currently I have set vm.dirty_ratio = 15, vm.dirty_background_ratio = 3, vm.vfs_cache_pressure = 1 and the low usage occurred 3 days before, other values like the defaults or when I was playing around with vm.dirty_ratio = 90, vm.dirty_background_ratio = 80 and whatever cache_pressure showed similar results. 2018-07-12 13:34 GMT+02:00 Michal Hocko : > On Wed 11-07-18 15:18:30, Marinko Catovic wrote: > > hello guys > > > > > > I tried in a few IRC, people told me to ask here, so I'll give it a try. > > > > > > I have a very weird issue with mm on several hosts. > > The systems are for shared hosting, so lots of users there with lots of > > files. > > Maybe 5TB of files per host, several million at least, there is lots of > I/O > > which can be handled perfectly fine with buffers/cache > > > > The kernel version is the latest stable, 4.17.4, I had 3.x before and did > > not notice any issues until now. the same is for 4.16 which was in use > > before: > > > > The hosts altogether have 64G of RAM and operate with SSD+HDD. > > HDDs are the issue here, since those 5TB of data are stored there, there > > goes the high I/O. > > Running applications need about 15GB, so say 40GB of RAM are left for > > buffers/caching. > > > > Usually this works perfectly fine. The buffers take about 1-3G of RAM, > the > > cache the rest, say 35GB as an example. > > But every now and then, maybe every 2 days it happens that both drop to > > really low values, say 100MB buffers, 3GB caches and the rest of the RAM > is > > not in use, so there are about 35GB+ of totally free RAM. > > > > The performance of the host goes down significantly then, as it becomes > > unusable at some point, since it behaves as if the buffers/cache were > > totally useless. > > After lots and lots of playing around I noticed that when shutting down > all > > services that access the HDDs on the system and restarting them, that > this > > does *not* make any difference. > > > > But what did make a difference was stopping and umounting the fs, > mounting > > it again and starting the services. > > Then the buffers+cache built up to 5GB/35GB as usual after a while and > > everything was perfectly fine again! > > > > I noticed that what happens when umount is called, that the caches are > > being dropped. So I gave it a try: > > > > sync; echo 2 > /proc/sys/vm/drop_caches > > > > has the exactly same effect. Note that echo 1 > .. does not. > > > > So if that low usage like 100MB/3GB occurs I'd have to drop the caches by > > echoing 2 to drop_caches. The 3GB then become even lower, which is > > expected, but then at least the buffers/cache built up again to ordinary > > values and the usual performance is restored after a few minutes. > > I have never seen this before, this happened after I switched the systems > > to newer ones, where the old ones had kernel 3.x, this behavior was never > > observed before. > > > > Do you have *any idea* at all what could be causing this? that issue is > > bugging me since over a month and seriously really disturbs everything > I'm > > doing since lot of people access that data and all of them start to > > complain at some point where I see that the caches became useless at that > > time, having to drop them to rebuild again. > > > > Some guys in IRC suggested that his could be a fragmentation problem or > > something, or about slab shrinking. > > Well, the page cache shouldn't really care about fragmentation because > single pages are used. Btw. what is the filesystem that you are using? > > > The problem is that I can not reproduce this, I have to wait a while, > maybe > > 2 days to observe that, until that the buffers/caches are fully in use > and > > at some point they decrease within a few hours to those useless values. > > Sadly this is a production system and I can not play that much around, > > already causing downtime when dropping caches (populating caches needs > > maybe 5-10 minutes until the performance is ok again). > > This doesn't really ring bells for me. > > > Please tell me whatever info you need me to pastebin and when > (before/after > > what event). > > Any hints are appreciated a lot, it really gives me lots of headache, > since > > I am really busy with other things. Thank you very much! > > Could you collect /proc/vmstat every few seconds over that time period? > Maybe it will tell us more. > -- > Michal Hocko > SUSE Labs > --0000000000000b6ac30570e36833 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
hello Michal


= well these hints were just ideas mentioned by some people, it took me weeks= just to figure
out that 2>drop_caches helps, still not knowin= g why this happens.

Right now I am observing ~18GB= of unused RAM, since yesterday, so this is not always
about 100M= B/3.5GB, but right now it may be in the process of shrinking.
I r= eally can not tell for sure, this is so nondeterministic - I just wish I co= uld reproduce it for better testing.

Right now top= shows:
KiB Mem : 65892044 total, 18169232 free, 11879604 used, 3= 5843208 buff/cache
Where 1GB goes to buffers, the rest to cac= he, the host *is* busy and the buff/cache consumed
all RAM yester= day, where I did 2>drop_caches about one day before.

<= /div>
Another host (still) shows full usage. That other one is 1:1 the = same by software and config,
but with different data/users; the u= se-cases and load are pretty much similar.

Affecte= d host at this time:
https://pastebin.com/tagX= JRi1=C2=A0 .. 3 minutes later
https://pastebin.com/8YNFfKXf=C2=A0 ..= 3 minutes later
<= div>
To compare - this is the other host, that is still showi= ng full buffers/cache usage by now:
=
Usually both show this more or less at the same time, someti= mes it is the one, sometimes
the other. Other hosts I have are cu= rrently not under similar high load, making it even harder
to com= pare.

However, right now I can not observe this dr= opping towards really low values, but I am sure it will come.
fs is ext4, mount options are auto,rw,data=3Dwriteback,noatime<= wbr>,nodiratime,nodev,nosuid,async
previous mount opti= ons with same behavior also had max_dir_size_kb, quotas and defaults for da= ta=3D
so I also played around with these, but that made no differ= ence.

---------

follow up= (sorry, messed up with reply-to this mailing list):


https://pastebin.com/0v4ZF= NCv .. one hour later, right after my last report, 22GB free
https://pastebin.com/r= ReWnHtE .. one day later, 28GB free

It is interesting to see ho= wever, that this did not get that low as mentioned before.
So not sure w= here this is going right now, but nevertheless, the RAM is not occupied ful= ly,
there should be no reason to allow 28GB to be free at all.

St= ill lots I/O, and I am 100% positive that if I'd echo 2 > drop_cache= s, this would fill up the
entire RAM again.


What I can see is= that buffers are around 500-700MB, the values increase and decrease
all= the time, really "oscillating" around 600. afaik this should get= as high as possible, as long
there is free ram - the other host that is= still healthy has about 2GB/48GB fully occupying RAM.

Currently I h= ave set vm.dirty_ratio =3D 15, vm.dirty_background_ratio =3D 3, vm.vfs_cach= e_pressure =3D 1
and the low usage occurred 3 days before, other values = like the defaults or when I was playing
around with vm.dirty_ratio =3D 9= 0, vm.dirty_background_ratio =3D 80 and whatever cache_pressure
showed s= imilar results.


2018-07-12 13:34 GMT+02:00 Michal Hocko <mhocko@ker= nel.org>:
On Wed = 11-07-18 15:18:30, Marinko Catovic wrote:
> hello guys
>
>
> I tried in a few IRC, people told me to ask here, so I'll give it = a try.
>
>
> I have a very weird issue with mm on several hosts.
> The systems are for shared hosting, so lots of users there with lots o= f
> files.
> Maybe 5TB of files per host, several million at least, there is lots o= f I/O
> which can be handled perfectly fine with buffers/cache
>
> The kernel version is the latest stable, 4.17.4, I had 3.x before and = did
> not notice any issues until now. the same is for 4.16 which was in use=
> before:
>
> The hosts altogether have 64G of RAM and operate with SSD+HDD.
> HDDs are the issue here, since those 5TB of data are stored there, the= re
> goes the high I/O.
> Running applications need about 15GB, so say 40GB of RAM are left for<= br> > buffers/caching.
>
> Usually this works perfectly fine. The buffers take about 1-3G of RAM,= the
> cache the rest, say 35GB as an example.
> But every now and then, maybe every 2 days it happens that both drop t= o
> really low values, say 100MB buffers, 3GB caches and the rest of the R= AM is
> not in use, so there are about 35GB+ of totally free RAM.
>
> The performance of the host goes down significantly then, as it become= s
> unusable at some point, since it behaves as if the buffers/cache were<= br> > totally useless.
> After lots and lots of playing around I noticed that when shutting dow= n all
> services that access the HDDs on the system and restarting them, that = this
> does *not* make any difference.
>
> But what did make a difference was stopping and umounting the fs, moun= ting
> it again and starting the services.
> Then the buffers+cache built up to 5GB/35GB as usual after a while and=
> everything was perfectly fine again!
>
> I noticed that what happens when umount is called, that the caches are=
> being dropped. So I gave it a try:
>
> sync; echo 2 > /proc/sys/vm/drop_caches
>
> has the exactly same effect. Note that echo 1 > .. does not.
>
> So if that low usage like 100MB/3GB occurs I'd have to drop the ca= ches by
> echoing 2 to drop_caches. The 3GB then become even lower, which is
> expected, but then at least the buffers/cache built up again to ordina= ry
> values and the usual performance is restored after a few minutes.
> I have never seen this before, this happened after I switched the syst= ems
> to newer ones, where the old ones had kernel 3.x, this behavior was ne= ver
> observed before.
>
> Do you have *any idea* at all what could be causing this? that issue i= s
> bugging me since over a month and seriously really disturbs everything= I'm
> doing since lot of people access that data and all of them start to > complain at some point where I see that the caches became useless at t= hat
> time, having to drop them to rebuild again.
>
> Some guys in IRC suggested that his could be a fragmentation problem o= r
> something, or about slab shrinking.

Well, the page cache shouldn't really care about fragmentat= ion because
single pages are used. Btw. what is the filesystem that you are using?

> The problem is that I can not reproduce this, I have to wait a while, = maybe
> 2 days to observe that, until that the buffers/caches are fully in use= and
> at some point they decrease within a few hours to those useless values= .
> Sadly this is a production system and I can not play that much around,=
> already causing downtime when dropping caches (populating caches needs=
> maybe 5-10 minutes until the performance is ok again).

This doesn't really ring bells for me.

> Please tell me whatever info you need me to pastebin and when (before/= after
> what event).
> Any hints are appreciated a lot, it really gives me lots of headache, = since
> I am really busy with other things. Thank you very much!

Could you collect /proc/vmstat every few seconds over that time peri= od?
Maybe it will tell us more.
--
Michal Hocko
SUSE Labs

--0000000000000b6ac30570e36833--