how periodically do you want them? I assumed this some-hours and days
snapshots would be sufficient.
any particular command with or without grep perhaps?

I just had to drop caches, right before your response, the performance was
simply too bad.

this is for your information, how it was right after dropping and 0+5+25
minutes later

https://pastebin.com/LcjKgQkg .. this is what it looks like just after
sync; echo 2 > /proc/sys/vm/drop_caches
https://pastebin.com/ZCeFCKrb .. 5 minutes later, when performance is
starting to get better again
https://pastebin.com/8hij8Lid .. 20 minutes after that, you can expect this
to consume all the available ram within 1-2 hours


2018-07-16 18:23 GMT+02:00 Michal Hocko <mhocko@kernel.org>:

> On Mon 16-07-18 17:53:42, Marinko Catovic wrote:
> > I can provide further data now, monitoring vmstat:
> >
> > https://pastebin.com/j0dMGBe4 .. 1 day later, 600MB/13GB in use, 35GB
> free
> > https://pastebin.com/N011kYyd .. 1 day later, 300MB/10GB in use, 40GB
> free,
> > performance becomes even worse
> >
> > the issue raises up again, I would have to drop caches by now to restore
> > normal usage for another day or two.
> >
> > Afaik there should be no reason at all to not have the buffers/cache fill
> > up the entire memory, isn't that true?
> > There is to my knowledge almost no O_DIRECT involved, also as mentioned
> > before: when dropping caches
> > the buffers/cache usage would eat up all RAM within the hour as usual for
> > 1-2 days until it starts to go crazy again.
> >
> > As mentioned, the usage oscillates up and down instead of up until all
> RAM
> > is consumed.
> >
> > Please tell me if there is anything else I can do to help investigate
> this.
>
> Do you have periodic /proc/vmstat snapshots I have asked before?
> --
> Michal Hocko
> SUSE Labs
>