From: Taras Kondratiuk <takondra@cisco.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-mm@kvack.org, xe-linux-external@cisco.com,
Ruslan Ruslichenko <rruslich@cisco.com>,
linux-kernel@vger.kernel.org
Subject: Re: Detecting page cache trashing state
Date: Fri, 15 Sep 2017 10:28:30 -0700 [thread overview]
Message-ID: <150549651001.4512.15084374619358055097@takondra-t460s> (raw)
In-Reply-To: <20170915143619.2ifgex2jxck2xt5u@dhcp22.suse.cz>
Quoting Michal Hocko (2017-09-15 07:36:19)
> On Thu 14-09-17 17:16:27, Taras Kondratiuk wrote:
> > Hi
> >
> > In our devices under low memory conditions we often get into a trashing
> > state when system spends most of the time re-reading pages of .text
> > sections from a file system (squashfs in our case). Working set doesn't
> > fit into available page cache, so it is expected. The issue is that
> > OOM killer doesn't get triggered because there is still memory for
> > reclaiming. System may stuck in this state for a quite some time and
> > usually dies because of watchdogs.
> >
> > We are trying to detect such trashing state early to take some
> > preventive actions. It should be a pretty common issue, but for now we
> > haven't find any existing VM/IO statistics that can reliably detect such
> > state.
> >
> > Most of metrics provide absolute values: number/rate of page faults,
> > rate of IO operations, number of stolen pages, etc. For a specific
> > device configuration we can determine threshold values for those
> > parameters that will detect trashing state, but it is not feasible for
> > hundreds of device configurations.
> >
> > We are looking for some relative metric like "percent of CPU time spent
> > handling major page faults". With such relative metric we could use a
> > common threshold across all devices. For now we have added such metric
> > to /proc/stat in our kernel, but we would like to find some mechanism
> > available in upstream kernel.
> >
> > Has somebody faced similar issue? How are you solving it?
>
> Yes this is a pain point for a _long_ time. And we still do not have a
> good answer upstream. Johannes has been playing in this area [1].
> The main problem is that our OOM detection logic is based on the ability
> to reclaim memory to allocate new memory. And that is pretty much true
> for the pagecache when you are trashing. So we do not know that
> basically whole time is spent refaulting the memory back and forth.
> We do have some refault stats for the page cache but that is not
> integrated to the oom detection logic because this is really a
> non-trivial problem to solve without triggering early oom killer
> invocations.
>
> [1] http://lkml.kernel.org/r/20170727153010.23347-1-hannes@cmpxchg.org
Thanks Michal. memdelay looks promising. We will check it.
next prev parent reply other threads:[~2017-09-15 17:28 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-15 0:16 Detecting page cache trashing state Taras Kondratiuk
2017-09-15 11:55 ` Zdenek Kabelac
2017-09-15 14:22 ` Daniel Walker
2017-09-15 16:38 ` Taras Kondratiuk
2017-09-15 17:31 ` Daniel Walker
2017-09-15 14:36 ` Michal Hocko
2017-09-15 17:28 ` Taras Kondratiuk [this message]
2017-09-18 16:34 ` Johannes Weiner
2017-09-19 10:55 ` [PATCH 1/3] sched/loadavg: consolidate LOAD_INT, LOAD_FRAC macros kbuild test robot
2017-09-19 11:02 ` kbuild test robot
2017-09-28 15:49 ` Detecting page cache trashing state Ruslan Ruslichenko -X (rruslich - GLOBALLOGIC INC at Cisco)
2017-10-25 16:53 ` Daniel Walker
2017-10-25 17:54 ` Johannes Weiner
2017-10-27 20:19 ` Ruslan Ruslichenko -X (rruslich - GLOBALLOGIC INC at Cisco)
2017-11-20 19:40 ` Ruslan Ruslichenko -X (rruslich - GLOBALLOGIC INC at Cisco)
2017-11-27 2:18 ` Minchan Kim
2017-10-26 3:53 ` vinayak menon
2017-10-27 20:29 ` Ruslan Ruslichenko -X (rruslich - GLOBALLOGIC INC at Cisco)
2017-09-15 21:20 ` vcaputo
2017-09-15 23:40 ` Taras Kondratiuk
2017-09-18 5:55 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=150549651001.4512.15084374619358055097@takondra-t460s \
--to=takondra@cisco.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=rruslich@cisco.com \
--cc=xe-linux-external@cisco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).