From: ndrw <ndrw.xf@redhazel.co.uk>
To: Michal Hocko <mhocko@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>,
Suren Baghdasaryan <surenb@google.com>,
Vlastimil Babka <vbabka@suse.cz>,
"Artem S. Tashkinov" <aros@gmx.com>,
Andrew Morton <akpm@linux-foundation.org>,
LKML <linux-kernel@vger.kernel.org>,
linux-mm <linux-mm@kvack.org>
Subject: Re: Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure
Date: Thu, 8 Aug 2019 22:59:32 +0100 [thread overview]
Message-ID: <08e5d007-a41a-e322-5631-b89978b9cc20@redhazel.co.uk> (raw)
In-Reply-To: <20190808185925.GH18351@dhcp22.suse.cz>
On 08/08/2019 19:59, Michal Hocko wrote:
> Well, I am afraid that implementing anything like that in the kernel
> will lead to many regressions and bug reports. People tend to have very
> different opinions on when it is suitable to kill a potentially
> important part of a workload just because memory gets low.
Are you proposing having a zero memory reserve or not having such option
at all? I'm fine with the current default (zero reserve/margin).
I strongly prefer forcing OOM killer when the system is still running
normally. Not just for preventing stalls: in my limited testing I found
the OOM killer on a stalled system rather inaccurate, occasionally
killing system services etc. I had much better experience with earlyoom.
> LRU aspect doesn't help much, really. If we are reclaiming the same set
> of pages becuase they are needed for the workload to operate then we are
> effectivelly treshing no matter what kind of replacement policy you are
> going to use.
In my case it would work fine (my system already works well with
earlyoom, and without it it remains responsive until last couple hundred
MB of RAM).
>>> PSI is giving you a matric that tells you how much time you
>>> spend on the memory reclaim. So you can start watching the system from
>>> lower utilization already.
I've tested it on a system with 45GB of RAM, SSD, swap disabled (my
intention was to approximate a worst-case scenario) and it didn't really
detect stall before it happened. I can see some activity after reaching
~42GB, the system remains fully responsive until it suddenly freezes and
requires sysrq-f. PSI appears to increase a bit when the system is about
to run out of memory but the change is so small it would be difficult to
set a reliable threshold. I expect the PSI numbers to increase
significantly after the stall (I wasn't able to capture them) but, as
mentioned above, I was hoping for a solution that would work before the
stall.
$ while true; do sleep 1; cat /proc/pressure/memory ; done
[starting a test script and waiting for several minutes to fill up memory]
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.00 avg60=0.00 avg300=0.00 total=10389
full avg10=0.00 avg60=0.00 avg300=0.00 total=6442
some avg10=0.00 avg60=0.00 avg300=0.00 total=18950
full avg10=0.00 avg60=0.00 avg300=0.00 total=11576
some avg10=0.00 avg60=0.00 avg300=0.00 total=25655
full avg10=0.00 avg60=0.00 avg300=0.00 total=16159
some avg10=0.00 avg60=0.00 avg300=0.00 total=31438
full avg10=0.00 avg60=0.00 avg300=0.00 total=19552
some avg10=0.00 avg60=0.00 avg300=0.00 total=44549
full avg10=0.00 avg60=0.00 avg300=0.00 total=27772
some avg10=0.00 avg60=0.00 avg300=0.00 total=52520
full avg10=0.00 avg60=0.00 avg300=0.00 total=32580
some avg10=0.00 avg60=0.00 avg300=0.00 total=60451
full avg10=0.00 avg60=0.00 avg300=0.00 total=37704
some avg10=0.00 avg60=0.00 avg300=0.00 total=68986
full avg10=0.00 avg60=0.00 avg300=0.00 total=42859
some avg10=0.00 avg60=0.00 avg300=0.00 total=76598
full avg10=0.00 avg60=0.00 avg300=0.00 total=48370
some avg10=0.00 avg60=0.00 avg300=0.00 total=83080
full avg10=0.00 avg60=0.00 avg300=0.00 total=52930
some avg10=0.00 avg60=0.00 avg300=0.00 total=89384
full avg10=0.00 avg60=0.00 avg300=0.00 total=56350
some avg10=0.00 avg60=0.00 avg300=0.00 total=95293
full avg10=0.00 avg60=0.00 avg300=0.00 total=60260
some avg10=0.00 avg60=0.00 avg300=0.00 total=101566
full avg10=0.00 avg60=0.00 avg300=0.00 total=64408
some avg10=0.00 avg60=0.00 avg300=0.00 total=108131
full avg10=0.00 avg60=0.00 avg300=0.00 total=68412
some avg10=0.00 avg60=0.00 avg300=0.00 total=121932
full avg10=0.00 avg60=0.00 avg300=0.00 total=77413
some avg10=0.00 avg60=0.00 avg300=0.00 total=140807
full avg10=0.00 avg60=0.00 avg300=0.00 total=91269
some avg10=0.00 avg60=0.00 avg300=0.00 total=170494
full avg10=0.00 avg60=0.00 avg300=0.00 total=110611
[stall, sysrq-f]
Best regards,
ndrw
next prev parent reply other threads:[~2019-08-08 21:59 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-04 9:23 Let's talk about the elephant in the room - the Linux kernel's inability to gracefully handle low memory pressure Artem S. Tashkinov
2019-08-05 12:13 ` Vlastimil Babka
2019-08-05 13:31 ` Michal Hocko
2019-08-05 16:47 ` Suren Baghdasaryan
2019-08-05 18:55 ` Johannes Weiner
2019-08-06 9:29 ` Michal Hocko
2019-08-05 19:31 ` Johannes Weiner
2019-08-06 1:08 ` Suren Baghdasaryan
2019-08-06 9:36 ` Vlastimil Babka
2019-08-06 14:27 ` Johannes Weiner
2019-08-06 14:36 ` Michal Hocko
2019-08-06 16:27 ` Suren Baghdasaryan
2019-08-06 22:01 ` Johannes Weiner
2019-08-07 7:59 ` Michal Hocko
2019-08-07 20:51 ` Johannes Weiner
2019-08-07 21:01 ` Andrew Morton
2019-08-07 21:34 ` Johannes Weiner
2019-08-07 21:12 ` Johannes Weiner
2019-08-08 11:48 ` Michal Hocko
2019-08-08 15:10 ` ndrw.xf
2019-08-08 16:32 ` Michal Hocko
2019-08-08 17:57 ` ndrw.xf
2019-08-08 18:59 ` Michal Hocko
2019-08-08 21:59 ` ndrw [this message]
2019-08-09 8:57 ` Michal Hocko
2019-08-09 10:09 ` ndrw
2019-08-09 10:50 ` Michal Hocko
2019-08-09 14:18 ` Pintu Agarwal
2019-08-10 12:34 ` ndrw
2019-08-12 8:24 ` Michal Hocko
2019-08-10 21:07 ` ndrw
2021-07-24 17:32 ` Alexey Avramov
2019-08-08 14:47 ` Vlastimil Babka
2019-08-08 17:27 ` Johannes Weiner
2019-08-09 14:56 ` Vlastimil Babka
2019-08-09 17:31 ` Johannes Weiner
2019-08-13 13:47 ` Vlastimil Babka
2019-08-06 21:43 ` James Courtier-Dutton
2019-08-06 19:00 ` Florian Weimer
2019-08-20 6:46 ` Daniel Drake
2019-08-21 21:42 ` James Courtier-Dutton
2019-08-29 12:29 ` Michal Hocko
2019-09-02 20:15 ` Pavel Machek
2019-08-23 1:54 ` ndrw
2019-08-23 2:14 ` Daniel Drake
[not found] <20190805090514.5992-1-hdanton@sina.com>
2019-08-05 12:01 ` Artem S. Tashkinov
2019-08-06 8:57 Johannes Buchner
2019-08-06 19:43 Remi Gauvin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=08e5d007-a41a-e322-5631-b89978b9cc20@redhazel.co.uk \
--to=ndrw.xf@redhazel.co.uk \
--cc=akpm@linux-foundation.org \
--cc=aros@gmx.com \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=surenb@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).