From: Marinko Catovic <marinko.catovic@gmail.com>
To: Vlastimil Babka <vbabka@suse.cz>
Cc: linux-mm@kvack.org, Michal Hocko <mhocko@kernel.org>
Subject: Re: Caching/buffers become useless after some time
Date: Mon, 6 Aug 2018 12:29:43 +0200 [thread overview]
Message-ID: <CADF2uSocjT5Oz=1Wohahjf5-58YpT2Jm2vTQKuqA=8ywBFwCaQ@mail.gmail.com> (raw)
In-Reply-To: <30f7ec9a-e090-06f1-1851-b18b3214f5e3@suse.cz>
[-- Attachment #1: Type: text/plain, Size: 5362 bytes --]
> Maybe a memcg with kmemcg limit? Michal could know more.
Could you/Michael explain this perhaps?
The hardware is pretty much high end datacenter grade, I really would
not know how this is to be related with the hardware :(
I do not understand why apparently the caching is working very much
fine for the beginning after a drop_caches, then degrades to low usage
somewhat later. I can not possibly drop caches automatically, since
this requires monitoring for overload with temporary dropping traffic
on specific ports until the writes/reads cool down.
2018-08-06 11:40 GMT+02:00 Vlastimil Babka <vbabka@suse.cz>:
> On 08/03/2018 04:13 PM, Marinko Catovic wrote:
> > Thanks for the analysis.
> >
> > So since I am no mem management dev, what exactly does this mean?
> > Is there any way of workaround or quickfix or something that can/will
> > be fixed at some point in time?
>
> Workaround would be the manual / periodic cache flushing, unfortunately.
>
> Maybe a memcg with kmemcg limit? Michal could know more.
>
> A long-term generic solution will be much harder to find :(
>
> > I can not imagine that I am the only one who is affected by this, nor do
> I
> > know why my use case would be so much different from any other.
> > Most 'cloud' services should be affected as well.
>
> Hmm, either your workload is specific in being hungry for fs metadata
> and not much data (page cache). And/Or there's some source of the
> high-order allocations that others don't have, possibly related to some
> piece of hardware?
>
> > Tell me if you need any other snapshots or whatever info.
> >
> > 2018-08-02 18:15 GMT+02:00 Vlastimil Babka <vbabka@suse.cz
> > <mailto:vbabka@suse.cz>>:
> >
> > On 07/31/2018 12:08 AM, Marinko Catovic wrote:
> > >
> > >> Can you provide (a single snapshot) /proc/pagetypeinfo and
> > >> /proc/slabinfo from a system that's currently experiencing the
> issue,
> > >> also with /proc/vmstat and /proc/zoneinfo to verify? Thanks.
> > >
> > > your request came in just one day after I 2>drop_caches again when
> the
> > > ram usage
> > > was really really low again. Up until now it did not reoccur on
> any of
> > > the 2 hosts,
> > > where one shows 550MB/11G with 37G of totally free ram for now -
> so not
> > > that low
> > > like last time when I dropped it, I think it was like 300M/8G or
> so, but
> > > I hope it helps:
> >
> > Thanks.
> >
> > > /proc/pagetypeinfo https://pastebin.com/6QWEZagL
> >
> > Yep, looks like fragmented by reclaimable slabs:
> >
> > Node 0, zone Normal, type Unmovable 29101 32754 8372
> > 2790 1334 354 23 3 4 0 0
> > Node 0, zone Normal, type Movable 142449 83386 99426
> > 69177 36761 12931 1378 24 0 0 0
> > Node 0, zone Normal, type Reclaimable 467195 530638 355045
> > 192638 80358 15627 2029 231 18 0 0
> >
> > Number of blocks type Unmovable Movable Reclaimable
> > HighAtomic Isolate
> > Node 0, zone DMA 1 7 0
> > 0 0
> > Node 0, zone DMA32 34 703 375
> > 0 0
> > Node 0, zone Normal 1672 14276 15659
> > 1 0
> >
> > Half of the memory is marked as reclaimable (2 megabyte) pageblocks.
> > zoneinfo has nr_slab_reclaimable 1679817 so the reclaimable slabs
> occupy
> > only 3280 (6G) pageblocks, yet they are spread over 5 times as much.
> > It's also possible they pollute the Movable pageblocks as well, but
> the
> > stats can't tell us. Either the page grouping mobility heuristics are
> > broken here, or the worst case scenario happened - memory was at
> > some point
> > really wholly filled with reclaimable slabs, and the rather random
> > reclaim
> > did not result in whole pageblocks being freed.
> >
> > > /proc/slabinfo https://pastebin.com/81QAFgke
> >
> > Largest caches seem to be:
> > # name <active_objs> <num_objs> <objsize> <objperslab>
> > <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> :
> > slabdata <active_slabs> <num_slabs> <sharedavail>
> > ext4_inode_cache 3107754 3759573 1080 3 1 : tunables 24
> > 12 8 : slabdata 1253191 1253191 0
> > dentry 2840237 7328181 192 21 1 : tunables 120
> > 60 8 : slabdata 348961 348961 120
> >
> > The internal framentation of dentry cache is significant as well.
> > Dunno if some of those objects pin movable pages as well...
> >
> > So looks like there's insufficient slab reclaim (shrinker activity),
> and
> > possibly problems with page grouping by mobility heuristics as
> well...
> >
> > > /proc/vmstat https://pastebin.com/S7mrQx1s
> > > /proc/zoneinfo https://pastebin.com/csGeqNyX
> > >
> > > also please note - whether this makes any difference: there is no
> swap
> > > file/partition
> > > I am using this without swap space. imho this should not be
> > necessary since
> > > applications running on the hosts would not consume more than
> > 20GB, the rest
> > > should be used by buffers/cache.
> > >
> >
> >
>
>
[-- Attachment #2: Type: text/html, Size: 7521 bytes --]
next prev parent reply other threads:[~2018-08-06 10:29 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-11 13:18 Caching/buffers become useless after some time Marinko Catovic
2018-07-12 11:34 ` Michal Hocko
2018-07-13 15:48 ` Marinko Catovic
2018-07-16 15:53 ` Marinko Catovic
2018-07-16 16:23 ` Michal Hocko
2018-07-16 16:33 ` Marinko Catovic
2018-07-16 16:45 ` Michal Hocko
2018-07-20 22:03 ` Marinko Catovic
2018-07-27 11:15 ` Vlastimil Babka
2018-07-30 14:40 ` Michal Hocko
2018-07-30 22:08 ` Marinko Catovic
2018-08-02 16:15 ` Vlastimil Babka
2018-08-03 14:13 ` Marinko Catovic
2018-08-06 9:40 ` Vlastimil Babka
2018-08-06 10:29 ` Marinko Catovic [this message]
2018-08-06 12:00 ` Michal Hocko
2018-08-06 15:37 ` Christopher Lameter
2018-08-06 18:16 ` Michal Hocko
2018-08-09 8:29 ` Marinko Catovic
2018-08-21 0:36 ` Marinko Catovic
2018-08-21 6:49 ` Michal Hocko
2018-08-21 7:19 ` Vlastimil Babka
2018-08-22 20:02 ` Marinko Catovic
2018-08-23 12:10 ` Vlastimil Babka
2018-08-23 12:21 ` Michal Hocko
2018-08-24 0:11 ` Marinko Catovic
2018-08-24 6:34 ` Vlastimil Babka
2018-08-24 8:11 ` Marinko Catovic
2018-08-24 8:36 ` Vlastimil Babka
2018-08-29 14:54 ` Marinko Catovic
2018-08-29 15:01 ` Michal Hocko
2018-08-29 15:13 ` Marinko Catovic
2018-08-29 15:27 ` Michal Hocko
2018-08-29 16:44 ` Marinko Catovic
2018-10-22 1:19 ` Marinko Catovic
2018-10-23 17:41 ` Marinko Catovic
2018-10-26 5:48 ` Marinko Catovic
2018-10-26 8:01 ` Michal Hocko
2018-10-26 23:31 ` Marinko Catovic
2018-10-27 6:42 ` Michal Hocko
[not found] ` <6e3a9434-32f2-0388-e0c7-2bd1c2ebc8b1@suse.cz>
2018-10-30 15:30 ` Michal Hocko
2018-10-30 16:08 ` Marinko Catovic
2018-10-30 17:00 ` Vlastimil Babka
2018-10-30 18:26 ` Marinko Catovic
2018-10-31 7:34 ` Michal Hocko
2018-10-31 7:32 ` Michal Hocko
2018-10-31 13:40 ` Vlastimil Babka
2018-10-31 14:53 ` Marinko Catovic
2018-10-31 17:01 ` Michal Hocko
2018-10-31 19:21 ` Marinko Catovic
2018-11-01 13:23 ` Michal Hocko
2018-11-01 22:46 ` Marinko Catovic
2018-11-02 8:05 ` Michal Hocko
2018-11-02 11:31 ` Marinko Catovic
2018-11-02 11:49 ` Michal Hocko
2018-11-02 12:22 ` Vlastimil Babka
2018-11-02 12:41 ` Marinko Catovic
2018-11-02 13:13 ` Vlastimil Babka
2018-11-02 13:50 ` Marinko Catovic
2018-11-02 14:49 ` Vlastimil Babka
2018-11-02 14:59 ` Vlastimil Babka
2018-11-30 12:01 ` Marinko Catovic
2018-12-10 21:30 ` Marinko Catovic
2018-12-10 21:47 ` Michal Hocko
2018-10-31 13:12 ` Vlastimil Babka
2018-08-24 6:24 ` Vlastimil Babka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CADF2uSocjT5Oz=1Wohahjf5-58YpT2Jm2vTQKuqA=8ywBFwCaQ@mail.gmail.com' \
--to=marinko.catovic@gmail.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.