linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Vlastimil Babka <vbabka@suse.cz>
To: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>,
	Suren Baghdasaryan <surenb@google.com>,
	"Artem S. Tashkinov" <aros@gmx.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-mm <linux-mm@kvack.org>
Subject: Re: Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure
Date: Tue, 13 Aug 2019 15:47:44 +0200	[thread overview]
Message-ID: <71773a98-803f-7731-e47c-d51cf262bbf7@suse.cz> (raw)
In-Reply-To: <20190809173108.GA21089@cmpxchg.org>

On 8/9/19 7:31 PM, Johannes Weiner wrote:
>> It made a difference, but not enough, it seems. Before the patch I could
>> observe "io:full avg10" around 75% and "memory:full avg10" around 20%,
>> after the patch, "memory:full avg10" went to around 45%, while io stayed
>> the same (BTW should the refaults be discounted from the io counters, so
>> that the sum is still <=100%?)
>>
>> As a result I could change the knobs to recover successfully with
>> thrashing detected for 10s of 40% memory pressure.
>>
>> Perhaps being low on memory we can't detect refaults so well due to
>> limited number of shadow entries, or there was genuine non-refault I/O
>> in the mix. The detection would then probably have to look at both I/O
>> and memory?
> 
> Thanks for testing it. It's possible that there is legitimate
> non-refault IO, and there can be interaction of course between that
> and the refault IO. But to be sure that all genuine refaults are
> captured, can you record the workingset_* values from /proc/vmstat
> before/after the thrash storm? In particular, workingset_nodereclaim
> would indicate whether we are losing refault information.

Let's see... after a ~45 second stall that I ended up by alt-sysrq-f, I
see the following pressure info:

cpu:some avg10=1.04 avg60=2.22 avg300=2.01 total=147402828
io:some avg10=97.13 avg60=65.48 avg300=28.86 total=240442256
io:full avg10=83.93 avg60=57.05 avg300=24.56 total=212125506
memory:some avg10=54.62 avg60=33.69 avg300=15.89 total=67989547
memory:full avg10=44.48 avg60=28.17 avg300=13.17 total=55963961

Captured vmstat workingset values

before:
workingset_nodes 15756
workingset_refault 6111959
workingset_activate 1805063
workingset_restore 919138
workingset_nodereclaim 40796
pgpgin 33889644

after:
workingset_nodes 14842
workingset_refault 9248248
workingset_activate 1966317
workingset_restore 961179
workingset_nodereclaim 41060
pgpgin 46488352

Doesn't seem like losing too much refault info, and it's indeed a mix of
refaults and other I/O? (difference is 3M for refaults and 12.5M for
pgpgin).

> [ The different resource pressures are not meant to be summed
>   up. Refaults truly are both IO events and memory events: they
>   indicate memory contention, but they also contribute to the IO
>   load. So both metrics need to include them, or it would skew the
>   picture when you only look at one of them. ]

Understood, makes sense.

  reply	other threads:[~2019-08-13 13:47 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-04  9:23 Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure Artem S. Tashkinov
2019-08-05 12:13 ` Vlastimil Babka
2019-08-05 13:31   ` Michal Hocko
2019-08-05 16:47     ` Suren Baghdasaryan
2019-08-05 18:55     ` Johannes Weiner
2019-08-06  9:29       ` Michal Hocko
2019-08-05 19:31   ` Johannes Weiner
2019-08-06  1:08     ` Suren Baghdasaryan
2019-08-06  9:36       ` Vlastimil Babka
2019-08-06 14:27         ` Johannes Weiner
2019-08-06 14:36           ` Michal Hocko
2019-08-06 16:27             ` Suren Baghdasaryan
2019-08-06 22:01               ` Johannes Weiner
2019-08-07  7:59                 ` Michal Hocko
2019-08-07 20:51                   ` Johannes Weiner
2019-08-07 21:01                     ` Andrew Morton
2019-08-07 21:34                       ` Johannes Weiner
2019-08-07 21:12                     ` Johannes Weiner
2019-08-08 11:48                     ` Michal Hocko
2019-08-08 15:10                       ` ndrw.xf
2019-08-08 16:32                         ` Michal Hocko
2019-08-08 17:57                           ` ndrw.xf
2019-08-08 18:59                             ` Michal Hocko
2019-08-08 21:59                               ` ndrw
2019-08-09  8:57                                 ` Michal Hocko
2019-08-09 10:09                                   ` ndrw
2019-08-09 10:50                                     ` Michal Hocko
2019-08-09 14:18                                       ` Pintu Agarwal
2019-08-10 12:34                                       ` ndrw
2019-08-12  8:24                                         ` Michal Hocko
2019-08-10 21:07                                   ` ndrw
2021-07-24 17:32                         ` Alexey Avramov
2019-08-08 14:47                     ` Vlastimil Babka
2019-08-08 17:27                       ` Johannes Weiner
2019-08-09 14:56                         ` Vlastimil Babka
2019-08-09 17:31                           ` Johannes Weiner
2019-08-13 13:47                             ` Vlastimil Babka [this message]
2019-08-06 21:43       ` James Courtier-Dutton
2019-08-06 19:00 ` Florian Weimer
2019-08-20  6:46 ` Daniel Drake
2019-08-21 21:42   ` James Courtier-Dutton
2019-08-29 12:29     ` Michal Hocko
2019-09-02 20:15     ` Pavel Machek
2019-08-23  1:54   ` ndrw
2019-08-23  2:14     ` Daniel Drake
     [not found] <20190805090514.5992-1-hdanton@sina.com>
2019-08-05 12:01 ` Artem S. Tashkinov
2019-08-06  8:57 Johannes Buchner
2019-08-06 19:43 Remi Gauvin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=71773a98-803f-7731-e47c-d51cf262bbf7@suse.cz \
    --to=vbabka@suse.cz \
    --cc=akpm@linux-foundation.org \
    --cc=aros@gmx.com \
    --cc=hannes@cmpxchg.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=surenb@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).