All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alexey Avramov <hakavlad@inbox.lv>
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, mgorman@techsingularity.net,
	mhocko@suse.com, vbabka@suse.cz, neilb@suse.de,
	akpm@linux-foundation.org, corbet@lwn.net, riel@surriel.com,
	hannes@cmpxchg.org, david@fromorbit.com, willy@infradead.org,
	hdanton@sina.com, penguin-kernel@i-love.sakura.ne.jp,
	oleksandr@natalenko.name, kernel@xanmod.org,
	michael@michaellarabel.com, aros@gmx.com, hakavlad@inbox.lv,
	hakavlad@gmail.com
Subject: mm: 5.16 regression: reclaim_throttle leads to stall in near-OOM conditions
Date: Wed, 24 Nov 2021 01:19:54 +0900	[thread overview]
Message-ID: <20211124011954.7cab9bb4@mail.inbox.lv> (raw)

I found stalls in near-OOM conditions with Linux 5.16. This is not the
hang-up that was reported by Artem S. Tashkinov in 2019 [1]. It's a *new* 
regression. I will demonstrate this with one simple experiment, which I
will reproduce with different kernels or settings.

With older versions of the kernel, running the `tail /dev/zero` command
usually quickly leads to OOM condition.

I will run the command `for i in {1...3}; do tail /dev/zero; done` and log
PSI metrics (using psi2log script from nohang v0.2.0 [2]) and some values
from `/proc/meminfo` (using mem2log v0.1.0 [3]) while this command is
running. During the experiment a single tab browser will be kept opened in
which some video will be playing.

The OS is Debian 9 x86_64 on HDD with ext4+LUKS, MemTotal: 11.5 GiB.


1. 5.15.0 and expected behavior
===============================

The expected behavior for a fast memory leak: when MemFree and
Active(file)+Inactive(file) drop to the minimum, OOM is called and the
available memory is kept at the minimum for only a short period of time,
usually not more than 1 second. This happens with kernel 5.15 and earlier.

This is mem2log log 
(mem2log started as `mem2log --inerval 0.5 --log /tmpfs/n`):

2021-11-21 23:10:42,923: Starting mem2log with interval 0.5s, mode: 1
2021-11-21 23:10:42,923: Log file: /tmpfs/27
2021-11-21 23:10:42,924: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:10:42,924: All values are in mebibytes
2021-11-21 23:10:42,924: MemTotal: 11798.7, SwapTotal: 0.0
2021-11-21 23:10:42,924: --
2021-11-21 23:10:42,924: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-21 23:10:42,925: F is File, AF is Active(file), IF is Inactive(file)
2021-11-21 23:10:42,925: D is Dirty, C is Clean file (File - Dirty)
2021-11-21 23:10:42,925: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-21 23:10:42,925: --
2021-11-21 23:10:42,925: MA=10010=85% MF=9977 A=1226 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:43,426: MA=8942=76% MF=8910 A=2292 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:43,927: MA=7868=67% MF=7836 A=3363 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:44,428: MA=6802=58% MF=6770 A=4426 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:44,928: MA=5745=49% MF=5713 A=5481 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:45,429: MA=4688=40% MF=4656 A=6536 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:45,930: MA=3625=31% MF=3593 A=7597 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:46,431: MA=2563=22% MF=2531 A=8657 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:46,932: MA=1497=13% MF=1465 A=9719 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:47,433: MA=442=4% MF=410 A=10772 F=251 AF=66 IF=185 D=0 C=251 SF=0 SU=0
2021-11-21 23:10:47,933: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0  <----- OOM
2021-11-21 23:10:48,434: MA=9699=82% MF=9835 A=1640 F=4 AF=0 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:10:48,935: MA=10508=89% MF=10641 A=850 F=9 AF=0 IF=9 D=0 C=9 SF=0 SU=0
2021-11-21 23:10:49,436: MA=10506=89% MF=10636 A=850 F=15 AF=0 IF=15 D=0 C=15 SF=0 SU=0
2021-11-21 23:10:49,938: MA=10504=89% MF=10632 A=850 F=19 AF=0 IF=19 D=0 C=19 SF=0 SU=0
2021-11-21 23:10:50,439: MA=10502=89% MF=10627 A=850 F=25 AF=0 IF=25 D=0 C=25 SF=0 SU=0
2021-11-21 23:10:50,940: MA=10501=89% MF=10623 A=850 F=31 AF=1 IF=30 D=0 C=31 SF=0 SU=0
2021-11-21 23:10:51,441: MA=10499=89% MF=10619 A=850 F=35 AF=1 IF=35 D=0 C=35 SF=0 SU=0
2021-11-21 23:10:51,943: MA=10496=89% MF=10614 A=850 F=40 AF=1 IF=39 D=0 C=40 SF=0 SU=0
2021-11-21 23:10:52,444: MA=10493=89% MF=10608 A=850 F=45 AF=1 IF=45 D=0 C=45 SF=0 SU=0
2021-11-21 23:10:52,945: MA=10492=89% MF=10604 A=850 F=50 AF=1 IF=49 D=0 C=50 SF=0 SU=0
2021-11-21 23:10:53,446: MA=10489=89% MF=10599 A=850 F=55 AF=2 IF=53 D=0 C=55 SF=0 SU=0
2021-11-21 23:10:53,947: MA=10487=89% MF=10594 A=850 F=59 AF=2 IF=57 D=0 C=59 SF=0 SU=0
2021-11-21 23:10:54,448: MA=9928=84% MF=10034 A=1405 F=64 AF=2 IF=61 D=0 C=64 SF=0 SU=0
2021-11-21 23:10:54,949: MA=8901=75% MF=9004 A=2428 F=68 AF=3 IF=66 D=0 C=68 SF=0 SU=0
2021-11-21 23:10:55,450: MA=7819=66% MF=7921 A=3507 F=72 AF=3 IF=69 D=0 C=72 SF=0 SU=0
2021-11-21 23:10:55,951: MA=6749=57% MF=6848 A=4569 F=76 AF=3 IF=73 D=0 C=76 SF=0 SU=0
2021-11-21 23:10:56,452: MA=5675=48% MF=5772 A=5639 F=82 AF=3 IF=78 D=0 C=82 SF=0 SU=0
2021-11-21 23:10:56,953: MA=4595=39% MF=4689 A=6716 F=87 AF=3 IF=83 D=0 C=87 SF=0 SU=0
2021-11-21 23:10:57,453: MA=3510=30% MF=3602 A=7797 F=92 AF=3 IF=88 D=0 C=92 SF=0 SU=0
2021-11-21 23:10:57,954: MA=2432=21% MF=2522 A=8872 F=95 AF=4 IF=92 D=0 C=95 SF=0 SU=0
2021-11-21 23:10:58,455: MA=1363=12% MF=1450 A=9935 F=100 AF=4 IF=97 D=0 C=100 SF=0 SU=0
2021-11-21 23:10:58,956: MA=293=2% MF=378 A=10999 F=104 AF=4 IF=101 D=0 C=104 SF=0 SU=0
2021-11-21 23:10:59,457: MA=4639=39% MF=4776 A=6711 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0  <----- OOM
2021-11-21 23:10:59,957: MA=10524=89% MF=10659 A=845 F=6 AF=1 IF=5 D=0 C=6 SF=0 SU=0
2021-11-21 23:11:00,459: MA=10522=89% MF=10654 A=845 F=11 AF=1 IF=11 D=0 C=11 SF=0 SU=0
2021-11-21 23:11:00,960: MA=10520=89% MF=10649 A=845 F=18 AF=1 IF=17 D=0 C=18 SF=0 SU=0
2021-11-21 23:11:01,461: MA=10516=89% MF=10643 A=845 F=23 AF=1 IF=22 D=0 C=23 SF=0 SU=0
2021-11-21 23:11:01,962: MA=10514=89% MF=10638 A=845 F=27 AF=1 IF=26 D=0 C=27 SF=0 SU=0
2021-11-21 23:11:02,464: MA=10512=89% MF=10633 A=845 F=32 AF=1 IF=31 D=0 C=32 SF=0 SU=0
2021-11-21 23:11:02,965: MA=10509=89% MF=10628 A=845 F=36 AF=1 IF=35 D=0 C=36 SF=0 SU=0
2021-11-21 23:11:03,466: MA=10506=89% MF=10623 A=845 F=43 AF=1 IF=41 D=0 C=43 SF=0 SU=0
2021-11-21 23:11:03,967: MA=10503=89% MF=10617 A=845 F=49 AF=2 IF=47 D=0 C=49 SF=0 SU=0
2021-11-21 23:11:04,469: MA=10500=89% MF=10610 A=845 F=55 AF=2 IF=53 D=0 C=55 SF=0 SU=0
2021-11-21 23:11:04,969: MA=10479=89% MF=10587 A=865 F=59 AF=2 IF=57 D=0 C=59 SF=0 SU=0
2021-11-21 23:11:05,470: MA=9479=80% MF=9584 A=1861 F=64 AF=2 IF=61 D=0 C=64 SF=0 SU=0
2021-11-21 23:11:05,971: MA=8413=71% MF=8516 A=2921 F=70 AF=2 IF=67 D=0 C=70 SF=0 SU=0
2021-11-21 23:11:06,472: MA=7340=62% MF=7441 A=3990 F=73 AF=2 IF=71 D=0 C=73 SF=0 SU=0
2021-11-21 23:11:06,973: MA=6273=53% MF=6372 A=5053 F=77 AF=2 IF=75 D=0 C=77 SF=0 SU=0
2021-11-21 23:11:07,474: MA=5207=44% MF=5303 A=6113 F=82 AF=2 IF=80 D=0 C=82 SF=0 SU=0
2021-11-21 23:11:07,975: MA=4140=35% MF=4234 A=7175 F=87 AF=2 IF=85 D=0 C=87 SF=0 SU=0
2021-11-21 23:11:08,476: MA=3074=26% MF=3166 A=8237 F=90 AF=2 IF=88 D=0 C=90 SF=0 SU=0
2021-11-21 23:11:08,977: MA=2012=17% MF=2103 A=9294 F=93 AF=2 IF=91 D=1 C=92 SF=0 SU=0
2021-11-21 23:11:09,477: MA=1002=8% MF=1091 A=10292 F=98 AF=2 IF=95 D=2 C=96 SF=0 SU=0
2021-11-21 23:11:09,978: MA=20=0% MF=118 A=11301 F=78 AF=39 IF=39 D=0 C=78 SF=0 SU=0
2021-11-21 23:11:10,479: MA=0=0% MF=118 A=11380 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-21 23:11:10,980: MA=5941=50% MF=6077 A=5410 F=4 AF=0 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:11:11,481: MA=10532=89% MF=10665 A=843 F=9 AF=1 IF=9 D=0 C=9 SF=0 SU=0
2021-11-21 23:11:11,983: MA=10530=89% MF=10660 A=843 F=15 AF=1 IF=14 D=0 C=15 SF=0 SU=0
2021-11-21 23:11:12,484: MA=10527=89% MF=10655 A=843 F=20 AF=1 IF=19 D=0 C=20 SF=0 SU=0
2021-11-21 23:11:12,985: MA=10526=89% MF=10651 A=843 F=24 AF=1 IF=24 D=0 C=24 SF=0 SU=0
2021-11-21 23:11:13,486: MA=10524=89% MF=10647 A=843 F=29 AF=1 IF=29 D=0 C=29 SF=0 SU=0
2021-11-21 23:11:13,988: MA=10521=89% MF=10642 A=843 F=34 AF=1 IF=34 D=0 C=34 SF=0 SU=0
2021-11-21 23:11:14,489: MA=10519=89% MF=10637 A=843 F=40 AF=1 IF=39 D=0 C=40 SF=0 SU=0

and PSI metrics at the same time
(psi2log started as `psi2log --mode 2 --log /tmpfs/n`):

2021-11-21 23:10:38,826: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-21 23:10:38,826: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-21 23:10:38,828: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:10:38,828: ======|=============|=============|
2021-11-21 23:10:38,828:  cpu  |      io     |    memory   |
2021-11-21 23:10:38,828: ----- | ----------- | ----------- |
2021-11-21 23:10:38,828:  some |  some  full |  some  full | interval
2021-11-21 23:10:38,828: ----- | ----- ----- | ----- ----- | --------
2021-11-21 23:10:40,829:   1.4 |   0.0   0.0 |   0.0   0.0 | 2.001
2021-11-21 23:10:42,832:   1.3 |   2.6   2.6 |   0.0   0.0 | 2.002
2021-11-21 23:10:44,834:   0.3 |   0.2   0.2 |   0.0   0.0 | 2.002
2021-11-21 23:10:46,836:   0.2 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-21 23:10:48,839:   2.6 |  38.0  33.2 |  14.6  14.5 | 2.003
2021-11-21 23:10:50,842:   0.2 |  94.6  93.3 |   0.0   0.0 | 2.003
2021-11-21 23:10:52,844:   1.1 |  93.1  88.6 |   0.0   0.0 | 2.003
2021-11-21 23:10:54,845:   0.8 |  94.2  85.9 |   0.0   0.0 | 2.001
2021-11-21 23:10:56,848:   0.5 |  75.6  69.0 |   0.0   0.0 | 2.002
2021-11-21 23:10:58,849:   0.5 |  66.2  63.0 |   0.0   0.0 | 2.002
2021-11-21 23:11:00,852:   2.3 |  86.8  78.6 |   7.8   7.6 | 2.003
2021-11-21 23:11:02,855:   0.2 |  99.9  98.7 |   0.0   0.0 | 2.003
2021-11-21 23:11:04,857:   0.4 |  97.7  95.7 |   0.0   0.0 | 2.003
2021-11-21 23:11:06,860:   0.5 |  77.7  73.9 |   0.0   0.0 | 2.003
2021-11-21 23:11:08,862:   0.6 |  60.1  57.4 |   0.0   0.0 | 2.002
2021-11-21 23:11:10,864:   3.4 |  68.7  58.1 |  21.5  21.1 | 2.002
2021-11-21 23:11:12,867:   0.2 |  97.0  94.4 |   0.0   0.0 | 2.003
2021-11-21 23:11:14,869:   0.4 |  98.1  96.3 |   0.0   0.0 | 2.002
2021-11-21 23:11:16,872:   1.2 |  87.9  83.8 |   0.0   0.0 | 2.003
2021-11-21 23:11:18,873:   1.2 |  82.4  76.6 |   0.0   0.0 | 2.001
2021-11-21 23:11:20,876:   1.1 |  83.8  79.6 |   0.0   0.0 | 2.003
2021-11-21 23:11:22,877:   1.4 |  61.3  57.8 |   0.0   0.0 | 2.001
2021-11-21 23:11:24,880:   1.5 |  54.4  50.4 |   0.0   0.0 | 2.003
2021-11-21 23:11:26,881:   1.5 |  34.3  32.5 |   0.0   0.0 | 2.001
2021-11-21 23:11:28,884:   1.3 |  39.8  38.1 |   0.0   0.0 | 2.003
2021-11-21 23:11:30,887:   1.5 |  25.3  23.5 |   0.0   0.0 | 2.003
2021-11-21 23:11:32,889:   1.6 |   1.3   1.2 |   0.0   0.0 | 2.003
2021-11-21 23:11:34,892:   1.3 |   0.0   0.0 |   0.0   0.0 | 2.003

In the mem2log log, we see OOM arrivals for less than 1 second:
23:10:47,933, 23:10:59,457 and 23:11:10,479.
At the same time, memory pressure metrics do not reach large values.


2. Vanilla 5.16-rc1 and stall
=============================

Freezes occur for tens of seconds.

2021-11-21 22:53:11,393: Starting mem2log with interval 0.5s, mode: 1
2021-11-21 22:53:11,393: Log file: /tmpfs/27
2021-11-21 22:53:11,395: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 22:53:11,395: All values are in mebibytes
2021-11-21 22:53:11,395: MemTotal: 11798.5, SwapTotal: 0.0
2021-11-21 22:53:11,395: --
2021-11-21 22:53:11,395: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-21 22:53:11,395: F is File, AF is Active(file), IF is Inactive(file)
2021-11-21 22:53:11,395: D is Dirty, C is Clean file (File - Dirty)
2021-11-21 22:53:11,395: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-21 22:53:11,395: --
2021-11-21 22:53:11,395: MA=10309=87% MF=10307 A=914 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:11,896: MA=9434=80% MF=9432 A=1789 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:12,399: MA=8438=72% MF=8436 A=2783 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:12,900: MA=7431=63% MF=7428 A=3788 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:13,401: MA=6429=54% MF=6427 A=4779 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:13,902: MA=5453=46% MF=5450 A=5757 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:14,403: MA=4463=38% MF=4461 A=6754 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:14,904: MA=3458=29% MF=3456 A=7754 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:15,405: MA=2453=21% MF=2451 A=8755 F=218 AF=42 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:15,906: MA=1444=12% MF=1442 A=9761 F=218 AF=43 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:16,407: MA=428=4% MF=426 A=10773 F=218 AF=43 IF=176 D=0 C=218 SF=0 SU=0
2021-11-21 22:53:16,908: MA=104=1% MF=140 A=11117 F=183 AF=103 IF=80 D=0 C=183 SF=0 SU=0
2021-11-21 22:53:17,408: MA=0=0% MF=131 A=11346 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0  <----- stall started
2021-11-21 22:53:17,909: MA=0=0% MF=131 A=11346 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:18,410: MA=0=0% MF=130 A=11346 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:18,911: MA=0=0% MF=131 A=11346 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:19,411: MA=0=0% MF=130 A=11346 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:19,912: MA=0=0% MF=131 A=11346 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:20,413: MA=0=0% MF=131 A=11346 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:20,914: MA=0=0% MF=131 A=11346 F=3 AF=1 IF=1 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:21,415: MA=0=0% MF=130 A=11346 F=4 AF=2 IF=2 D=0 C=4 SF=0 SU=0
2021-11-21 22:53:21,915: MA=0=0% MF=130 A=11346 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:22,416: MA=0=0% MF=121 A=11355 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:22,917: MA=0=0% MF=121 A=11355 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:23,418: MA=0=0% MF=121 A=11355 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:23,919: MA=0=0% MF=122 A=11355 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:24,419: MA=0=0% MF=122 A=11355 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:24,920: MA=0=0% MF=122 A=11355 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:25,421: MA=0=0% MF=121 A=11355 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:25,922: MA=0=0% MF=119 A=11358 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:26,423: MA=0=0% MF=119 A=11358 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:26,923: MA=0=0% MF=121 A=11358 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:27,424: MA=0=0% MF=121 A=11358 F=3 AF=1 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:27,925: MA=0=0% MF=119 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:28,426: MA=0=0% MF=120 A=11359 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:28,926: MA=0=0% MF=120 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:29,427: MA=0=0% MF=120 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:29,928: MA=0=0% MF=120 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:30,429: MA=0=0% MF=119 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:30,930: MA=0=0% MF=119 A=11359 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:31,431: MA=0=0% MF=119 A=11361 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:31,931: MA=0=0% MF=119 A=11361 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:32,432: MA=0=0% MF=119 A=11361 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:32,933: MA=0=0% MF=119 A=11361 F=3 AF=1 IF=1 D=0 C=3 SF=0 SU=0
2021-11-21 22:53:33,434: MA=0=0% MF=118 A=11361 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:33,935: MA=0=0% MF=117 A=11362 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:34,435: MA=0=0% MF=118 A=11362 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:34,936: MA=0=0% MF=118 A=11362 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:35,437: MA=0=0% MF=119 A=11362 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:35,938: MA=0=0% MF=119 A=11362 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:36,439: MA=0=0% MF=118 A=11362 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:36,939: MA=0=0% MF=119 A=11362 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:37,440: MA=0=0% MF=118 A=11362 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:37,941: MA=0=0% MF=119 A=11362 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:38,442: MA=0=0% MF=119 A=11362 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:38,943: MA=0=0% MF=118 A=11362 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:39,443: MA=0=0% MF=118 A=11362 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:39,944: MA=0=0% MF=118 A=11363 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:40,445: MA=0=0% MF=119 A=11363 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:40,946: MA=0=0% MF=118 A=11363 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:41,447: MA=0=0% MF=119 A=11363 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:41,947: MA=0=0% MF=118 A=11363 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:42,448: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:42,949: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:43,450: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:43,951: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:44,451: MA=0=0% MF=118 A=11364 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:44,952: MA=0=0% MF=118 A=11364 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:45,453: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:45,954: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:46,455: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:46,956: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:47,456: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:47,957: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:48,458: MA=0=0% MF=118 A=11364 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:48,959: MA=0=0% MF=117 A=11364 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:49,459: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:49,960: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:50,461: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:50,962: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:51,463: MA=0=0% MF=118 A=11364 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:51,964: MA=0=0% MF=119 A=11364 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:52,464: MA=0=0% MF=118 A=11364 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:52,965: MA=0=0% MF=118 A=11364 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:53,466: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:53,968: MA=0=0% MF=118 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:54,469: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:54,970: MA=0=0% MF=119 A=11365 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:53:55,471: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:55,972: MA=0=0% MF=119 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:53:56,473: MA=0=0% MF=118 A=11365 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:53:56,974: MA=0=0% MF=117 A=11365 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:53:57,475: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:04,997: MA=0=0% MF=118 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:05,498: MA=0=0% MF=117 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:06,000: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:06,501: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:07,002: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:07,503: MA=0=0% MF=119 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:08,004: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:08,505: MA=0=0% MF=117 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:09,007: MA=0=0% MF=117 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:09,508: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:10,009: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:10,510: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:11,012: MA=0=0% MF=117 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:11,513: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:12,014: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:12,516: MA=0=0% MF=118 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:13,017: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:13,518: MA=0=0% MF=118 A=11365 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:54:14,019: MA=0=0% MF=118 A=11365 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:54:14,520: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:15,022: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:15,523: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:16,024: MA=0=0% MF=117 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:16,525: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:17,025: MA=0=0% MF=117 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:17,527: MA=0=0% MF=118 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:18,028: MA=0=0% MF=118 A=11365 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:18,529: MA=0=0% MF=118 A=11365 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:19,030: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:19,531: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:20,032: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:20,533: MA=0=0% MF=117 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:21,033: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:21,535: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:22,036: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:22,537: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:23,038: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:23,540: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:24,041: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:24,542: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:25,043: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:25,545: MA=0=0% MF=117 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:26,046: MA=0=0% MF=118 A=11366 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:54:26,547: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:27,048: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:27,549: MA=0=0% MF=117 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:28,051: MA=0=0% MF=118 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:28,552: MA=0=0% MF=117 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:29,053: MA=0=0% MF=118 A=11366 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:29,555: MA=0=0% MF=117 A=11366 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:30,056: MA=0=0% MF=117 A=11366 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0   <----- 66s stall is over, OOM
2021-11-21 22:54:37,893: MA=10536=89% MF=10669 A=814 F=9 AF=1 IF=8 D=0 C=9 SF=0 SU=0
2021-11-21 22:54:38,395: MA=10535=89% MF=10664 A=814 F=15 AF=1 IF=14 D=0 C=15 SF=0 SU=0
2021-11-21 22:54:38,896: MA=10532=89% MF=10659 A=814 F=22 AF=1 IF=21 D=0 C=22 SF=0 SU=0
2021-11-21 22:54:39,397: MA=10531=89% MF=10654 A=814 F=27 AF=1 IF=26 D=0 C=27 SF=0 SU=0
2021-11-21 22:54:39,898: MA=10528=89% MF=10649 A=814 F=32 AF=1 IF=31 D=0 C=32 SF=0 SU=0
2021-11-21 22:54:40,400: MA=10526=89% MF=10644 A=814 F=38 AF=2 IF=36 D=0 C=38 SF=0 SU=0
2021-11-21 22:54:40,901: MA=10523=89% MF=10639 A=814 F=43 AF=2 IF=41 D=0 C=43 SF=0 SU=0
2021-11-21 22:54:41,402: MA=10522=89% MF=10636 A=814 F=46 AF=2 IF=44 D=0 C=46 SF=0 SU=0
2021-11-21 22:54:41,903: MA=10519=89% MF=10632 A=814 F=50 AF=2 IF=48 D=0 C=50 SF=0 SU=0
2021-11-21 22:54:42,404: MA=10516=89% MF=10625 A=814 F=56 AF=2 IF=54 D=0 C=56 SF=0 SU=0
2021-11-21 22:54:42,905: MA=10514=89% MF=10621 A=812 F=61 AF=2 IF=59 D=0 C=61 SF=0 SU=0
2021-11-21 22:54:43,406: MA=10512=89% MF=10616 A=804 F=67 AF=3 IF=64 D=0 C=67 SF=0 SU=0
2021-11-21 22:54:43,908: MA=10511=89% MF=10612 A=804 F=71 AF=3 IF=69 D=0 C=71 SF=0 SU=0
2021-11-21 22:54:44,409: MA=10508=89% MF=10607 A=804 F=77 AF=3 IF=74 D=0 C=77 SF=0 SU=0
2021-11-21 22:54:44,910: MA=10504=89% MF=10600 A=807 F=82 AF=3 IF=80 D=0 C=82 SF=0 SU=0
2021-11-21 22:54:45,411: MA=10503=89% MF=10597 A=804 F=86 AF=3 IF=83 D=0 C=86 SF=0 SU=0
2021-11-21 22:54:45,912: MA=9578=81% MF=9670 A=1728 F=91 AF=3 IF=88 D=0 C=91 SF=0 SU=0
2021-11-21 22:54:46,413: MA=8489=72% MF=8578 A=2814 F=96 AF=3 IF=94 D=0 C=96 SF=0 SU=0
2021-11-21 22:54:46,914: MA=7399=63% MF=7486 A=3899 F=100 AF=3 IF=97 D=0 C=100 SF=0 SU=0
2021-11-21 22:54:47,415: MA=6309=53% MF=6394 A=4988 F=104 AF=3 IF=101 D=0 C=104 SF=0 SU=0
2021-11-21 22:54:47,915: MA=5229=44% MF=5313 A=6062 F=108 AF=3 IF=104 D=0 C=107 SF=0 SU=0
2021-11-21 22:54:48,416: MA=4146=35% MF=4228 A=7144 F=111 AF=4 IF=107 D=1 C=110 SF=0 SU=0
2021-11-21 22:54:48,917: MA=3065=26% MF=3145 A=8216 F=114 AF=4 IF=110 D=1 C=114 SF=0 SU=0
2021-11-21 22:54:49,418: MA=1980=17% MF=2059 A=9273 F=118 AF=6 IF=112 D=1 C=117 SF=0 SU=0
2021-11-21 22:54:49,919: MA=1001=8% MF=1078 A=10244 F=122 AF=7 IF=114 D=1 C=120 SF=0 SU=0
2021-11-21 22:54:50,420: MA=42=0% MF=118 A=11235 F=122 AF=29 IF=93 D=1 C=120 SF=0 SU=0
2021-11-21 22:54:50,921: MA=55=0% MF=150 A=11259 F=84 AF=33 IF=50 D=0 C=84 SF=0 SU=0
2021-11-21 22:54:51,422: MA=0=0% MF=122 A=11366 F=3 AF=1 IF=1 D=0 C=3 SF=0 SU=0  <----- stall started
2021-11-21 22:54:51,923: MA=0=0% MF=122 A=11366 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:54:52,423: MA=0=0% MF=123 A=11366 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:54:52,924: MA=0=0% MF=122 A=11366 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:54:53,425: MA=0=0% MF=123 A=11366 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:53,926: MA=0=0% MF=122 A=11366 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:54:54,427: MA=0=0% MF=121 A=11366 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:54:54,927: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:55,428: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:55,929: MA=0=0% MF=119 A=11372 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:56,430: MA=0=0% MF=118 A=11372 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:56,931: MA=0=0% MF=119 A=11372 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:54:57,432: MA=0=0% MF=119 A=11372 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:57,932: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:58,433: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:58,934: MA=0=0% MF=117 A=11372 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:54:59,435: MA=0=0% MF=119 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:54:59,936: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:00,436: MA=0=0% MF=118 A=11372 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:00,937: MA=0=0% MF=118 A=11372 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:01,438: MA=0=0% MF=120 A=11372 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:01,938: MA=0=0% MF=118 A=11372 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:02,439: MA=0=0% MF=119 A=11372 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:02,940: MA=0=0% MF=119 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:03,441: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:03,942: MA=0=0% MF=119 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:04,443: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:04,944: MA=0=0% MF=118 A=11373 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:05,445: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:05,947: MA=0=0% MF=120 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:06,448: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:06,949: MA=0=0% MF=117 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:07,450: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:07,951: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:08,452: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:08,953: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:09,454: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:09,955: MA=0=0% MF=117 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:10,455: MA=0=0% MF=119 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:10,956: MA=0=0% MF=118 A=11373 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:11,458: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:11,959: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:12,460: MA=0=0% MF=119 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:12,961: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:13,463: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:13,964: MA=0=0% MF=119 A=11373 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:14,465: MA=0=0% MF=119 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:14,967: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:15,468: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:15,969: MA=0=0% MF=118 A=11373 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:16,470: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:16,970: MA=0=0% MF=118 A=11373 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:17,471: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:17,972: MA=0=0% MF=119 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:18,473: MA=0=0% MF=119 A=11373 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:18,974: MA=0=0% MF=117 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:19,475: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:19,976: MA=0=0% MF=117 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:20,477: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:20,978: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:21,479: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:21,980: MA=0=0% MF=119 A=11373 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:22,481: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:22,981: MA=0=0% MF=118 A=11373 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:23,482: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:23,984: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:24,486: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:24,987: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:25,488: MA=0=0% MF=117 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:25,989: MA=0=0% MF=118 A=11373 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:26,491: MA=0=0% MF=118 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:26,992: MA=0=0% MF=117 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:27,493: MA=0=0% MF=119 A=11373 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:27,995: MA=0=0% MF=120 A=11373 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:28,496: MA=0=0% MF=117 A=11374 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:28,997: MA=0=0% MF=117 A=11374 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:29,498: MA=0=0% MF=118 A=11374 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:29,999: MA=0=0% MF=118 A=11374 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:30,501: MA=0=0% MF=118 A=11374 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:31,001: MA=0=0% MF=118 A=11374 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:31,503: MA=0=0% MF=118 A=11374 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:32,004: MA=0=0% MF=117 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:32,505: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:33,006: MA=0=0% MF=117 A=11375 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:33,507: MA=0=0% MF=117 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:34,008: MA=0=0% MF=118 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:34,509: MA=0=0% MF=117 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:35,010: MA=0=0% MF=118 A=11375 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:55:35,511: MA=0=0% MF=117 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:36,012: MA=0=0% MF=117 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:36,514: MA=0=0% MF=117 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:37,015: MA=0=0% MF=117 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:37,516: MA=0=0% MF=117 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:38,017: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:38,518: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:39,020: MA=0=0% MF=118 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:39,521: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:40,022: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:40,523: MA=0=0% MF=118 A=11375 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:41,025: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:41,526: MA=0=0% MF=118 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:42,026: MA=0=0% MF=118 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:42,528: MA=0=0% MF=119 A=11375 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:43,028: MA=0=0% MF=117 A=11375 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:55:43,530: MA=0=0% MF=119 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:44,031: MA=0=0% MF=119 A=11375 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:55:44,532: MA=0=0% MF=117 A=11375 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0   <----- 53.5s stall is over, OOM
2021-11-21 22:55:45,033: MA=2820=24% MF=2956 A=8534 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:55:45,534: MA=10537=89% MF=10672 A=809 F=4 AF=0 IF=4 D=0 C=4 SF=0 SU=0
2021-11-21 22:55:46,035: MA=10553=89% MF=10686 A=809 F=9 AF=2 IF=7 D=0 C=9 SF=0 SU=0
2021-11-21 22:55:46,537: MA=10551=89% MF=10681 A=809 F=14 AF=2 IF=12 D=0 C=14 SF=0 SU=0
2021-11-21 22:55:47,038: MA=10548=89% MF=10676 A=809 F=19 AF=2 IF=16 D=0 C=19 SF=0 SU=0
2021-11-21 22:55:47,539: MA=10545=89% MF=10670 A=809 F=25 AF=2 IF=22 D=0 C=25 SF=0 SU=0
2021-11-21 22:55:48,040: MA=10543=89% MF=10665 A=809 F=29 AF=2 IF=27 D=0 C=29 SF=0 SU=0
2021-11-21 22:55:48,541: MA=10541=89% MF=10660 A=809 F=36 AF=3 IF=33 D=0 C=36 SF=0 SU=0
2021-11-21 22:55:49,043: MA=10539=89% MF=10656 A=810 F=40 AF=3 IF=37 D=0 C=40 SF=0 SU=0
2021-11-21 22:55:49,544: MA=10537=89% MF=10652 A=809 F=45 AF=3 IF=43 D=0 C=45 SF=0 SU=0
2021-11-21 22:55:50,045: MA=10534=89% MF=10645 A=809 F=52 AF=3 IF=49 D=0 C=52 SF=0 SU=0
2021-11-21 22:55:50,546: MA=10532=89% MF=10641 A=809 F=56 AF=3 IF=53 D=0 C=56 SF=0 SU=0
2021-11-21 22:55:51,048: MA=10530=89% MF=10637 A=807 F=61 AF=3 IF=57 D=0 C=61 SF=0 SU=0
2021-11-21 22:55:51,549: MA=10528=89% MF=10633 A=807 F=66 AF=3 IF=62 D=0 C=66 SF=0 SU=0
2021-11-21 22:55:52,050: MA=10526=89% MF=10628 A=807 F=70 AF=3 IF=67 D=0 C=70 SF=0 SU=0
2021-11-21 22:55:52,551: MA=10524=89% MF=10624 A=807 F=74 AF=4 IF=71 D=0 C=74 SF=0 SU=0
2021-11-21 22:55:53,053: MA=10521=89% MF=10619 A=807 F=79 AF=4 IF=76 D=0 C=79 SF=0 SU=0
2021-11-21 22:55:53,554: MA=10163=86% MF=10258 A=1163 F=84 AF=4 IF=80 D=0 C=84 SF=0 SU=0
2021-11-21 22:55:54,055: MA=9078=77% MF=9172 A=2244 F=88 AF=4 IF=84 D=0 C=88 SF=0 SU=0
2021-11-21 22:55:54,555: MA=7991=68% MF=8082 A=3327 F=92 AF=4 IF=88 D=0 C=92 SF=0 SU=0
2021-11-21 22:55:55,056: MA=6901=58% MF=6990 A=4414 F=95 AF=4 IF=91 D=0 C=95 SF=0 SU=0
2021-11-21 22:55:55,557: MA=5815=49% MF=5903 A=5495 F=98 AF=4 IF=94 D=1 C=98 SF=0 SU=0
2021-11-21 22:55:56,058: MA=4734=40% MF=4821 A=6571 F=101 AF=5 IF=96 D=1 C=100 SF=0 SU=0
2021-11-21 22:55:56,559: MA=3644=31% MF=3729 A=7638 F=105 AF=6 IF=99 D=1 C=104 SF=0 SU=0
2021-11-21 22:55:57,060: MA=2645=22% MF=2728 A=8615 F=108 AF=7 IF=101 D=1 C=107 SF=0 SU=0
2021-11-21 22:55:57,561: MA=1647=14% MF=1729 A=9606 F=110 AF=7 IF=103 D=1 C=109 SF=0 SU=0
2021-11-21 22:55:58,062: MA=655=6% MF=736 A=10593 F=113 AF=8 IF=105 D=1 C=112 SF=0 SU=0
2021-11-21 22:55:58,563: MA=0=0% MF=120 A=11365 F=14 AF=8 IF=6 D=0 C=14 SF=0 SU=0
2021-11-21 22:55:59,063: MA=0=0% MF=127 A=11368 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0  <----- stall started
2021-11-21 22:55:59,564: MA=0=0% MF=128 A=11368 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:00,065: MA=0=0% MF=128 A=11368 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:00,566: MA=0=0% MF=127 A=11368 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:01,067: MA=0=0% MF=126 A=11368 F=3 AF=1 IF=1 D=0 C=3 SF=0 SU=0
2021-11-21 22:56:01,567: MA=0=0% MF=117 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:02,068: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:02,569: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:03,070: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:03,571: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:04,071: MA=0=0% MF=120 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:04,572: MA=0=0% MF=119 A=11376 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:05,073: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:05,574: MA=0=0% MF=118 A=11378 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:06,074: MA=0=0% MF=119 A=11378 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:06,575: MA=0=0% MF=118 A=11378 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:07,076: MA=0=0% MF=118 A=11378 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:07,577: MA=0=0% MF=118 A=11378 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:08,078: MA=0=0% MF=118 A=11378 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:08,579: MA=0=0% MF=117 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:09,080: MA=0=0% MF=119 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:09,581: MA=0=0% MF=119 A=11379 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:10,081: MA=0=0% MF=117 A=11379 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:10,582: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:11,083: MA=0=0% MF=120 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:11,584: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:12,085: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:12,585: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:13,086: MA=0=0% MF=117 A=11379 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:13,587: MA=0=0% MF=117 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:14,088: MA=0=0% MF=121 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:14,589: MA=0=0% MF=120 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:15,090: MA=0=0% MF=117 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:15,590: MA=0=0% MF=119 A=11379 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:16,091: MA=0=0% MF=119 A=11379 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:16,594: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:17,096: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:17,597: MA=0=0% MF=117 A=11379 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:24,260: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:24,761: MA=0=0% MF=121 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:25,262: MA=0=0% MF=119 A=11380 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:25,763: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:26,264: MA=0=0% MF=117 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:26,766: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:27,267: MA=0=0% MF=117 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:27,768: MA=0=0% MF=119 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:28,269: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:28,770: MA=0=0% MF=120 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:29,271: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:29,772: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:30,272: MA=0=0% MF=120 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:30,773: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:31,274: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:31,775: MA=0=0% MF=119 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:32,276: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:32,777: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:33,277: MA=0=0% MF=118 A=11380 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:33,778: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:34,279: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:34,780: MA=0=0% MF=118 A=11380 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:35,281: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:35,782: MA=0=0% MF=118 A=11380 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:36,283: MA=0=0% MF=117 A=11380 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:36,784: MA=0=0% MF=118 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:37,285: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:37,786: MA=0=0% MF=120 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:38,287: MA=0=0% MF=119 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:38,788: MA=0=0% MF=120 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:39,289: MA=0=0% MF=120 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:39,790: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:40,292: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:40,793: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:41,294: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:41,795: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:42,296: MA=0=0% MF=119 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:42,797: MA=0=0% MF=118 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:43,298: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:43,799: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:44,300: MA=0=0% MF=119 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:44,801: MA=0=0% MF=118 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:45,302: MA=0=0% MF=120 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:45,803: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:46,304: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:46,806: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:47,307: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:47,808: MA=0=0% MF=118 A=11381 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:48,309: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:48,811: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:49,312: MA=0=0% MF=119 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:49,813: MA=0=0% MF=118 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:50,315: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:50,816: MA=0=0% MF=118 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:51,317: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:51,818: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:52,320: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:52,821: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:53,322: MA=0=0% MF=119 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:53,824: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:54,325: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:54,826: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:55,327: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:55,829: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:56,330: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:56,831: MA=0=0% MF=118 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:57,332: MA=0=0% MF=120 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:56:57,834: MA=0=0% MF=118 A=11381 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 22:56:58,335: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:58,836: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:59,337: MA=0=0% MF=116 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:56:59,839: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:00,340: MA=0=0% MF=119 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:00,841: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:01,342: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:01,843: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:02,344: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:02,845: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:03,345: MA=0=0% MF=117 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:57:03,846: MA=0=0% MF=117 A=11381 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:04,347: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:04,848: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:05,349: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:05,850: MA=0=0% MF=118 A=11381 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:06,351: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:06,852: MA=0=0% MF=117 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:07,353: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:07,854: MA=0=0% MF=117 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:57:08,355: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:08,856: MA=0=0% MF=118 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 22:57:09,358: MA=0=0% MF=119 A=11381 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 22:57:09,859: MA=0=0% MF=117 A=11381 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0   <----- 65s stall is over, OOM
2021-11-21 22:57:10,360: MA=9028=77% MF=9164 A=2328 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 22:57:10,861: MA=10561=90% MF=10695 A=809 F=5 AF=0 IF=5 D=0 C=5 SF=0 SU=0
2021-11-21 22:57:11,362: MA=10559=89% MF=10693 A=808 F=8 AF=0 IF=8 D=0 C=8 SF=0 SU=0
2021-11-21 22:57:11,863: MA=10558=89% MF=10689 A=808 F=12 AF=2 IF=11 D=0 C=12 SF=0 SU=0
2021-11-21 22:57:12,365: MA=10556=89% MF=10685 A=808 F=18 AF=2 IF=16 D=0 C=18 SF=0 SU=0
2021-11-21 22:57:12,866: MA=10556=89% MF=10681 A=808 F=24 AF=2 IF=21 D=0 C=24 SF=0 SU=0
2021-11-21 22:57:13,367: MA=10553=89% MF=10676 A=808 F=30 AF=2 IF=27 D=0 C=30 SF=0 SU=0
2021-11-21 22:57:13,868: MA=10552=89% MF=10671 A=808 F=35 AF=2 IF=33 D=0 C=35 SF=0 SU=0
2021-11-21 22:57:14,369: MA=10549=89% MF=10666 A=808 F=40 AF=2 IF=38 D=0 C=40 SF=0 SU=0
2021-11-21 22:57:14,871: MA=10546=89% MF=10661 A=808 F=45 AF=2 IF=43 D=0 C=45 SF=0 SU=0
2021-11-21 22:57:15,372: MA=10544=89% MF=10656 A=803 F=51 AF=3 IF=48 D=0 C=51 SF=0 SU=0

and PSI metrics at the same time:

2021-11-21 22:53:08,298: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-21 22:53:08,298: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-21 22:53:08,300: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 22:53:08,300: ======|=============|=============|
2021-11-21 22:53:08,300:  cpu  |      io     |    memory   |
2021-11-21 22:53:08,300: ----- | ----------- | ----------- |
2021-11-21 22:53:08,300:  some |  some  full |  some  full | interval
2021-11-21 22:53:08,300: ----- | ----- ----- | ----- ----- | --------
2021-11-21 22:53:10,303:   0.9 |   0.6   0.6 |   0.0   0.0 | 2.002
2021-11-21 22:53:12,305:   0.8 |   2.9   2.6 |   0.0   0.0 | 2.003
2021-11-21 22:53:14,307:   0.6 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-21 22:53:16,309:   0.4 |   0.3   0.2 |   0.0   0.0 | 2.001
2021-11-21 22:53:18,311:   0.7 |  45.1  42.9 |  38.3  36.7 | 2.002
2021-11-21 22:53:20,313:   0.1 |  91.7  86.9 |  43.4  43.3 | 2.002
2021-11-21 22:53:22,315:   0.2 |  86.1  85.4 |  40.5  40.3 | 2.002
2021-11-21 22:53:24,317:   0.1 |  87.3  86.7 |  39.0  38.9 | 2.002
2021-11-21 22:53:26,320:   0.1 |  87.2  86.7 |  40.8  40.6 | 2.002
2021-11-21 22:53:28,322:   0.1 |  87.5  87.0 |  42.9  42.8 | 2.002
2021-11-21 22:53:30,324:   0.2 |  87.5  86.8 |  93.7  93.2 | 2.002
2021-11-21 22:53:32,327:   0.2 |  89.3  86.5 |  66.9  66.5 | 2.002
2021-11-21 22:53:34,329:   0.2 |  76.7  73.4 |  99.2  98.8 | 2.002
2021-11-21 22:53:36,331:   0.4 |  75.4  74.6 | 100.0  99.4 | 2.002
2021-11-21 22:53:38,333:   0.3 |  61.0  60.4 | 100.0  99.6 | 2.002
2021-11-21 22:53:40,336:   0.5 |  55.3  54.6 | 100.0  99.5 | 2.002
2021-11-21 22:53:42,338:   0.5 |  67.7  67.0 | 100.0  99.5 | 2.002
2021-11-21 22:53:44,340:   0.4 |  50.9  50.5 | 100.0  99.6 | 2.002
2021-11-21 22:53:46,343:   0.6 |  72.9  71.9 | 100.0  99.4 | 2.002
2021-11-21 22:53:48,345:   0.3 |  51.8  51.4 |  99.5  99.1 | 2.002
2021-11-21 22:53:50,347:   0.4 |  61.2  60.4 |  99.3  98.9 | 2.002
2021-11-21 22:53:52,349:   0.8 |  34.9  33.6 |  98.8  98.5 | 2.002
2021-11-21 22:53:54,352:   0.4 |  29.1  28.8 |  98.4  98.1 | 2.002
2021-11-21 22:53:56,354:   0.7 |  33.1  31.6 | 100.0  99.8 | 2.002
2021-11-21 22:53:58,357:   1.1 |  34.4  33.8 | 100.0  99.8 | 2.003
2021-11-21 22:54:00,359:   0.4 |  29.4  28.8 | 100.0  99.8 | 2.002
2021-11-21 22:54:02,362:   0.6 |  48.1  44.9 | 100.0  99.9 | 2.003
2021-11-21 22:54:04,365:   0.7 |  68.4  62.3 |  99.0  98.8 | 2.003
2021-11-21 22:54:06,368:   0.7 |  55.5  53.7 |  99.2  99.0 | 2.003
2021-11-21 22:54:08,370:   1.3 |  22.1  19.5 |  96.9  96.7 | 2.003
2021-11-21 22:54:10,373:   0.4 |   1.5   1.5 | 100.0  99.9 | 2.003
2021-11-21 22:54:12,375:   0.5 |   4.3   3.3 | 100.0  99.8 | 2.003
2021-11-21 22:54:14,378:   0.5 |   3.9   3.3 | 100.0  99.8 | 2.003
2021-11-21 22:54:16,379:   0.6 |  11.8  11.5 |  98.4  98.3 | 2.001
2021-11-21 22:54:18,381:   0.6 |  42.2  41.3 | 100.0  99.9 | 2.002
2021-11-21 22:54:20,384:   0.7 |  51.2  48.7 |  98.3  98.2 | 2.003
2021-11-21 22:54:22,386:   0.6 |  10.9  10.6 | 100.0  99.9 | 2.003
2021-11-21 22:54:24,389:   0.7 |  25.1  21.5 | 100.0  99.9 | 2.003
2021-11-21 22:54:26,392:   0.7 |  36.6  36.2 | 100.0  99.9 | 2.003
2021-11-21 22:54:28,394:   0.5 |  73.8  73.2 | 100.0  99.9 | 2.003
2021-11-21 22:54:30,397:   0.5 |  79.3  77.0 | 100.0  99.9 | 2.003
2021-11-21 22:54:32,400:   0.7 |  16.4  16.2 | 100.0  99.9 | 2.003
2021-11-21 22:54:34,403:   0.5 |   0.4   0.4 | 100.0  99.9 | 2.003
2021-11-21 22:54:36,405:   0.7 |   8.5   8.0 |  99.8  96.4 | 2.003
2021-11-21 22:54:38,407:   0.9 | 100.0  97.9 |  63.8  63.0 | 2.001
2021-11-21 22:54:40,409:   0.2 |  97.2  96.1 |   0.2   0.2 | 2.003
2021-11-21 22:54:42,412:   0.2 |  97.5  96.0 |   0.0   0.0 | 2.003
2021-11-21 22:54:44,414:   0.9 |  94.9  90.5 |   0.0   0.0 | 2.003
2021-11-21 22:54:46,417:   0.7 |  90.3  86.8 |   0.0   0.0 | 2.002
2021-11-21 22:54:48,418:   0.6 |  68.2  65.5 |   0.0   0.0 | 2.002
2021-11-21 22:54:50,421:   0.7 |  43.0  39.7 |   0.2   0.2 | 2.002
2021-11-21 22:54:52,423:   0.6 |  64.7  62.0 |  51.3  50.2 | 2.002
2021-11-21 22:54:54,426:   0.2 |  81.1  80.2 |  36.0  35.7 | 2.002
2021-11-21 22:54:56,427:   0.4 |  41.9  41.4 |  88.1  87.4 | 2.001
2021-11-21 22:54:58,429:   0.5 |  46.7  46.2 |  98.7  98.3 | 2.002
2021-11-21 22:55:00,431:   0.3 |  52.2  45.2 |  98.3  98.0 | 2.002
2021-11-21 22:55:02,433:   0.4 |  65.1  62.3 |  98.8  98.6 | 2.002
2021-11-21 22:55:04,437:   0.5 |  57.3  50.7 |  99.2  99.0 | 2.001
2021-11-21 22:55:06,440:   0.6 |  22.1  20.9 |  98.5  98.4 | 2.005
2021-11-21 22:55:08,442:   0.8 |  15.2  14.3 |  99.8  99.5 | 2.003
2021-11-21 22:55:10,443:   0.5 |  22.8  22.5 |  96.6  96.3 | 2.001
2021-11-21 22:55:12,445:   0.7 |  18.2  15.6 |  98.6  98.4 | 2.002
2021-11-21 22:55:14,448:   0.5 |   6.6   5.9 | 100.0  99.8 | 2.003
2021-11-21 22:55:16,450:   0.5 |  15.2  14.8 |  97.9  97.6 | 2.003
2021-11-21 22:55:18,453:   0.5 |  34.5  32.9 |  99.0  98.7 | 2.002
2021-11-21 22:55:20,455:   0.6 |  42.6  40.4 | 100.0  99.8 | 2.003
2021-11-21 22:55:22,458:   0.3 |  62.6  52.5 |  99.4  99.1 | 2.002
2021-11-21 22:55:24,460:   0.4 |  11.3   9.8 |  98.5  98.3 | 2.002
2021-11-21 22:55:26,463:   0.4 |   3.6   3.6 | 100.0  99.8 | 2.003
2021-11-21 22:55:28,465:   0.7 |  18.9  18.1 | 100.0  99.8 | 2.003
2021-11-21 22:55:30,468:   0.3 |  28.2  26.4 |  99.7  99.6 | 2.002
2021-11-21 22:55:32,470:   0.5 |  13.0  12.4 |  97.6  97.4 | 2.003
2021-11-21 22:55:34,473:   0.7 |   4.5   4.2 | 100.0  99.8 | 2.002
2021-11-21 22:55:36,475:   0.7 |  13.9  11.8 | 100.0  99.8 | 2.002
2021-11-21 22:55:38,477:   0.4 |   1.6   1.6 | 100.0  99.9 | 2.002
2021-11-21 22:55:40,480:   0.4 |   0.0   0.0 | 100.0  99.9 | 2.003
2021-11-21 22:55:42,483:   0.3 |   0.1   0.1 | 100.0  99.9 | 2.003
2021-11-21 22:55:44,485:   0.6 |   4.1   4.0 | 100.0  99.9 | 2.003
2021-11-21 22:55:46,487:   0.5 |  74.7  73.5 |  80.5  79.9 | 2.001
2021-11-21 22:55:48,489:   0.2 |  97.4  96.3 |  31.4  31.1 | 2.003
2021-11-21 22:55:50,492:   0.6 |  94.5  92.1 |   2.1   2.1 | 2.002
2021-11-21 22:55:52,494:   0.9 |  87.5  83.3 |   2.9   2.8 | 2.003
2021-11-21 22:55:54,497:   0.8 |  81.7  72.9 |   0.5   0.4 | 2.003
2021-11-21 22:55:56,499:   0.6 |  62.4  60.2 |   0.0   0.0 | 2.002
2021-11-21 22:55:58,501:   0.6 |  22.7  20.7 |   2.9   2.9 | 2.002
2021-11-21 22:56:00,503:   0.1 |  78.8  78.4 |  41.4  41.2 | 2.002
2021-11-21 22:56:02,506:   0.3 |  82.3  80.7 |  45.0  44.9 | 2.002
2021-11-21 22:56:04,508:   0.1 |  87.5  86.9 |  61.4  61.0 | 2.002
2021-11-21 22:56:06,510:   0.2 |  55.6  55.3 |  88.3  87.9 | 2.002
2021-11-21 22:56:08,513:   0.5 |  41.4  39.3 |  98.7  98.4 | 2.002
2021-11-21 22:56:10,515:   0.2 |  39.0  38.8 |  98.7  98.4 | 2.002
2021-11-21 22:56:12,517:   0.3 |  27.3  27.0 |  99.3  99.1 | 2.002
2021-11-21 22:56:14,520:   0.4 |  11.2  10.3 |  97.2  97.0 | 2.003
2021-11-21 22:56:16,523:   0.6 |  32.0  31.4 |  98.2  97.8 | 2.003
2021-11-21 22:56:18,525:   0.4 |  18.2  18.1 |  97.9  97.7 | 2.003
2021-11-21 22:56:20,528:   0.3 |  17.9  16.0 |  99.2  98.9 | 2.002
2021-11-21 22:56:22,530:   0.4 |  31.4  29.7 | 100.0  99.7 | 2.003
2021-11-21 22:56:24,531:   0.2 |  14.1  14.0 |  97.9  97.8 | 2.001
2021-11-21 22:56:26,533:   0.4 |  23.5  22.6 |  99.8  99.4 | 2.003
2021-11-21 22:56:28,536:   0.4 |   7.2   7.0 | 100.0  99.7 | 2.003
2021-11-21 22:56:30,538:   0.5 |  12.1  12.0 |  98.4  98.2 | 2.003
2021-11-21 22:56:32,541:   0.5 |  18.4  18.0 | 100.0  99.8 | 2.002
2021-11-21 22:56:34,543:   0.6 |  12.7  12.2 |  98.5  98.3 | 2.002
2021-11-21 22:56:36,546:   0.7 |  30.2  29.1 |  99.4  99.0 | 2.003
2021-11-21 22:56:38,549:   0.5 |  26.6  26.2 |  97.3  97.0 | 2.003
2021-11-21 22:56:40,551:   0.3 |   6.4   5.5 |  98.4  98.2 | 2.003
2021-11-21 22:56:42,553:   0.5 |  14.8  14.6 | 100.0  99.8 | 2.002
2021-11-21 22:56:44,556:   0.3 |  48.8  48.2 |  98.4  98.1 | 2.002
2021-11-21 22:56:46,558:   0.3 |  50.1  49.8 |  98.2  98.0 | 2.002
2021-11-21 22:56:48,560:   0.4 |  14.1  13.8 | 100.0  99.8 | 2.002
2021-11-21 22:56:50,563:   0.6 |   8.1   6.4 | 100.0  99.9 | 2.002
2021-11-21 22:56:52,565:   0.6 |  15.0  14.8 |  97.6  97.5 | 2.002
2021-11-21 22:56:54,568:   0.4 |  15.3  14.6 |  97.1  97.0 | 2.002
2021-11-21 22:56:56,570:   0.8 |   9.5   8.3 |  98.8  98.7 | 2.003
2021-11-21 22:56:58,573:   0.4 |   7.6   6.5 | 100.0  99.8 | 2.002
2021-11-21 22:57:00,575:   0.5 |  13.7  13.5 | 100.0  99.9 | 2.003
2021-11-21 22:57:02,578:   0.6 |  37.9  36.6 | 100.0  99.9 | 2.003
2021-11-21 22:57:04,581:   0.5 |  32.6  30.8 | 100.0  99.8 | 2.003
2021-11-21 22:57:06,583:   0.4 |  17.0  15.5 |  98.4  98.3 | 2.003
2021-11-21 22:57:08,586:   0.5 |   4.4   3.8 | 100.0  99.9 | 2.003
2021-11-21 22:57:10,589:   0.7 |  15.9  15.4 |  97.9  94.3 | 2.003
2021-11-21 22:57:12,591:   0.6 |  99.5  97.5 |  36.8  36.4 | 2.003
2021-11-21 22:57:14,594:   0.2 |  99.5  98.4 |   0.0   0.0 | 2.003
2021-11-21 22:57:16,597:   0.6 |  96.1  93.3 |   0.0   0.0 | 2.003
2021-11-21 22:57:18,599:   1.1 |  91.2  86.6 |   0.0   0.0 | 2.002
2021-11-21 22:57:20,601:   1.2 |  77.3  73.1 |   0.0   0.0 | 2.003
2021-11-21 22:57:22,603:   1.2 |  56.5  53.3 |   0.0   0.0 | 2.001
2021-11-21 22:57:24,605:   1.1 |  40.4  37.8 |   0.0   0.0 | 2.003
2021-11-21 22:57:26,607:   1.0 |  33.7  30.8 |   0.0   0.0 | 2.001
2021-11-21 22:57:28,609:   1.2 |  30.6  27.9 |   0.0   0.0 | 2.003
2021-11-21 22:57:30,611:   1.1 |  16.1  15.0 |   0.0   0.0 | 2.001
2021-11-21 22:57:32,613:   0.8 |   0.2   0.2 |   0.0   0.0 | 2.002
2021-11-21 22:57:34,615:   0.7 |   0.4   0.3 |   0.0   0.0 | 2.002
2021-11-21 22:57:36,617:   0.9 |   0.0   0.0 |   0.0   0.0 | 2.002

During the stalls, the memory pressure is kept at 99-100.


3. 5.16-rc1, timeouts are set to 0
==================================

This is the modified kernel: timeout in reclaim_throttle() is set to 0.
Code like this (in vmscan.c):
```
	switch(reason) {
	case VMSCAN_THROTTLE_WRITEBACK:
		timeout = HZ/10;

		if (atomic_inc_return(&pgdat->nr_writeback_throttled) == 1) {
			WRITE_ONCE(pgdat->nr_reclaim_start,
				node_page_state(pgdat, NR_THROTTLED_WRITTEN));
		}

		break;
	case VMSCAN_THROTTLE_NOPROGRESS:
		timeout = HZ/2;
		break;
	case VMSCAN_THROTTLE_ISOLATED:
		timeout = HZ/50;
		break;
	default:
		WARN_ON_ONCE(1);
		timeout = HZ;
		break;
	}

	timeout = 0;
```
and the result of running `for i in {1...3}; do tail /dev/zero; done`:

2021-11-21 23:36:44,448: Starting mem2log with interval 0.5s, mode: 1
2021-11-21 23:36:44,449: Log file: /tmpfs/27
2021-11-21 23:36:44,450: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:36:44,450: All values are in mebibytes
2021-11-21 23:36:44,450: MemTotal: 11798.5, SwapTotal: 0.0
2021-11-21 23:36:44,450: --
2021-11-21 23:36:44,450: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-21 23:36:44,450: F is File, AF is Active(file), IF is Inactive(file)
2021-11-21 23:36:44,450: D is Dirty, C is Clean file (File - Dirty)
2021-11-21 23:36:44,450: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-21 23:36:44,450: --
2021-11-21 23:36:44,450: MA=10215=87% MF=10186 A=1012 F=244 AF=52 IF=193 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:44,951: MA=9153=78% MF=9125 A=2071 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:45,452: MA=8088=69% MF=8060 A=3131 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:45,953: MA=7018=59% MF=6990 A=4198 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:46,454: MA=5958=51% MF=5930 A=5256 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:46,954: MA=4895=41% MF=4867 A=6318 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:47,455: MA=3830=32% MF=3802 A=7380 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:47,956: MA=2768=23% MF=2739 A=8447 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:48,457: MA=1695=14% MF=1666 A=9518 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:48,958: MA=625=5% MF=597 A=10584 F=244 AF=52 IF=192 D=0 C=244 SF=0 SU=0
2021-11-21 23:36:49,458: MA=0=0% MF=135 A=11350 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0  <----- stall started
2021-11-21 23:36:49,959: MA=0=0% MF=118 A=11368 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:36:50,460: MA=0=0% MF=118 A=11369 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:36:50,961: MA=0=0% MF=117 A=11369 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:36:51,462: MA=0=0% MF=118 A=11369 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0   <----- 2.5s stall stall is over, OOM
2021-11-21 23:36:51,963: MA=5222=44% MF=5357 A=6106 F=3 AF=0 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 23:36:52,464: MA=10479=89% MF=10612 A=873 F=9 AF=1 IF=8 D=0 C=9 SF=0 SU=0
2021-11-21 23:36:52,965: MA=10479=89% MF=10609 A=873 F=14 AF=1 IF=13 D=0 C=14 SF=0 SU=0
2021-11-21 23:36:53,466: MA=10477=89% MF=10605 A=873 F=19 AF=1 IF=18 D=0 C=19 SF=0 SU=0
2021-11-21 23:36:53,968: MA=10475=89% MF=10600 A=873 F=23 AF=1 IF=23 D=0 C=23 SF=0 SU=0
2021-11-21 23:36:54,469: MA=10473=89% MF=10595 A=873 F=29 AF=1 IF=29 D=0 C=29 SF=0 SU=0
2021-11-21 23:36:54,970: MA=10470=89% MF=10590 A=873 F=34 AF=1 IF=34 D=0 C=34 SF=0 SU=0
2021-11-21 23:36:55,471: MA=10468=89% MF=10585 A=873 F=39 AF=1 IF=38 D=0 C=39 SF=0 SU=0
2021-11-21 23:36:55,972: MA=10466=89% MF=10581 A=873 F=44 AF=1 IF=43 D=0 C=44 SF=0 SU=0
2021-11-21 23:36:56,474: MA=10464=89% MF=10576 A=873 F=49 AF=1 IF=48 D=0 C=49 SF=0 SU=0
2021-11-21 23:36:56,975: MA=10462=89% MF=10571 A=872 F=56 AF=1 IF=55 D=0 C=56 SF=0 SU=0
2021-11-21 23:36:57,476: MA=10461=89% MF=10568 A=872 F=60 AF=1 IF=59 D=0 C=60 SF=0 SU=0
2021-11-21 23:36:57,977: MA=9843=83% MF=9948 A=1488 F=65 AF=1 IF=64 D=0 C=65 SF=0 SU=0
2021-11-21 23:36:58,478: MA=8758=74% MF=8861 A=2568 F=69 AF=1 IF=68 D=0 C=69 SF=0 SU=0
2021-11-21 23:36:58,978: MA=7675=65% MF=7776 A=3647 F=73 AF=1 IF=71 D=0 C=73 SF=0 SU=0
2021-11-21 23:36:59,479: MA=6589=56% MF=6687 A=4730 F=78 AF=1 IF=76 D=1 C=77 SF=0 SU=0
2021-11-21 23:36:59,980: MA=5504=47% MF=5600 A=5812 F=83 AF=1 IF=81 D=1 C=82 SF=0 SU=0
2021-11-21 23:37:00,481: MA=4430=38% MF=4524 A=6879 F=86 AF=2 IF=84 D=1 C=85 SF=0 SU=0
2021-11-21 23:37:00,982: MA=3338=28% MF=3431 A=7968 F=88 AF=2 IF=86 D=1 C=87 SF=0 SU=0
2021-11-21 23:37:01,483: MA=2252=19% MF=2345 A=9051 F=88 AF=2 IF=86 D=1 C=87 SF=0 SU=0
2021-11-21 23:37:01,983: MA=1171=10% MF=1262 A=10131 F=91 AF=2 IF=88 D=1 C=90 SF=0 SU=0
2021-11-21 23:37:02,484: MA=98=1% MF=188 A=11197 F=95 AF=2 IF=92 D=1 C=94 SF=0 SU=0
2021-11-21 23:37:02,985: MA=0=0% MF=121 A=11376 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0  <----- stall started
2021-11-21 23:37:03,486: MA=0=0% MF=119 A=11376 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:03,987: MA=0=0% MF=118 A=11377 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:04,553: MA=0=0% MF=117 A=11378 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:05,054: MA=0=0% MF=118 A=11378 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:05,555: MA=0=0% MF=118 A=11378 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:06,056: MA=0=0% MF=117 A=11378 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:06,556: MA=0=0% MF=118 A=11378 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:07,057: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:07,558: MA=0=0% MF=118 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:08,059: MA=0=0% MF=118 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:08,560: MA=0=0% MF=117 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:09,061: MA=0=0% MF=117 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:09,562: MA=0=0% MF=118 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:10,062: MA=0=0% MF=117 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:10,563: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:11,064: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:11,565: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:12,066: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:12,567: MA=0=0% MF=118 A=11379 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:13,068: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:13,569: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:14,069: MA=0=0% MF=118 A=11379 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:14,570: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:15,071: MA=0=0% MF=118 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:15,572: MA=0=0% MF=118 A=11380 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:16,073: MA=0=0% MF=117 A=11380 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:16,769: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:17,270: MA=0=0% MF=117 A=11380 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:17,771: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0   <----- 15s stall is over, OOM
2021-11-21 23:37:18,271: MA=2205=19% MF=2341 A=9142 F=3 AF=0 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:18,773: MA=10489=89% MF=10622 A=874 F=10 AF=0 IF=9 D=0 C=10 SF=0 SU=0
2021-11-21 23:37:19,274: MA=10487=89% MF=10617 A=874 F=15 AF=0 IF=15 D=0 C=15 SF=0 SU=0
2021-11-21 23:37:19,775: MA=10485=89% MF=10612 A=874 F=21 AF=0 IF=20 D=0 C=21 SF=0 SU=0
2021-11-21 23:37:20,276: MA=10484=89% MF=10609 A=874 F=25 AF=0 IF=25 D=0 C=25 SF=0 SU=0
2021-11-21 23:37:20,778: MA=10482=89% MF=10604 A=874 F=31 AF=1 IF=30 D=0 C=31 SF=0 SU=0
2021-11-21 23:37:21,279: MA=10479=89% MF=10597 A=874 F=37 AF=1 IF=36 D=0 C=37 SF=0 SU=0
2021-11-21 23:37:21,780: MA=10476=89% MF=10593 A=874 F=42 AF=1 IF=41 D=0 C=42 SF=0 SU=0
2021-11-21 23:37:22,281: MA=10473=89% MF=10586 A=874 F=48 AF=1 IF=47 D=0 C=48 SF=0 SU=0
2021-11-21 23:37:22,782: MA=10470=89% MF=10580 A=874 F=54 AF=1 IF=53 D=0 C=54 SF=0 SU=0
2021-11-21 23:37:23,283: MA=10468=89% MF=10576 A=874 F=59 AF=1 IF=58 D=0 C=59 SF=0 SU=0
2021-11-21 23:37:23,784: MA=10465=89% MF=10570 A=874 F=63 AF=1 IF=63 D=0 C=63 SF=0 SU=0
2021-11-21 23:37:24,285: MA=10463=89% MF=10566 A=874 F=68 AF=1 IF=67 D=0 C=67 SF=0 SU=0
2021-11-21 23:37:24,787: MA=10459=89% MF=10559 A=875 F=74 AF=1 IF=72 D=0 C=74 SF=0 SU=0
2021-11-21 23:37:25,288: MA=10456=89% MF=10553 A=875 F=79 AF=2 IF=78 D=0 C=79 SF=0 SU=0
2021-11-21 23:37:25,789: MA=10455=89% MF=10551 A=875 F=82 AF=2 IF=80 D=0 C=82 SF=0 SU=0
2021-11-21 23:37:26,290: MA=9757=83% MF=9851 A=1570 F=85 AF=3 IF=82 D=0 C=85 SF=0 SU=0
2021-11-21 23:37:26,791: MA=8684=74% MF=8778 A=2642 F=87 AF=3 IF=84 D=0 C=87 SF=0 SU=0
2021-11-21 23:37:27,292: MA=7620=65% MF=7712 A=3703 F=91 AF=4 IF=87 D=0 C=90 SF=0 SU=0
2021-11-21 23:37:27,792: MA=6545=55% MF=6635 A=4774 F=94 AF=4 IF=91 D=1 C=94 SF=0 SU=0
2021-11-21 23:37:28,293: MA=5475=46% MF=5563 A=5839 F=99 AF=4 IF=95 D=1 C=98 SF=0 SU=0
2021-11-21 23:37:28,794: MA=4407=37% MF=4493 A=6904 F=103 AF=4 IF=99 D=1 C=102 SF=0 SU=0
2021-11-21 23:37:29,295: MA=3332=28% MF=3416 A=7973 F=106 AF=4 IF=102 D=1 C=105 SF=0 SU=0
2021-11-21 23:37:29,796: MA=2270=19% MF=2353 A=9030 F=108 AF=4 IF=104 D=1 C=107 SF=0 SU=0
2021-11-21 23:37:30,297: MA=1203=10% MF=1284 A=10093 F=112 AF=4 IF=108 D=1 C=111 SF=0 SU=0
2021-11-21 23:37:30,798: MA=133=1% MF=212 A=11159 F=115 AF=4 IF=111 D=1 C=114 SF=0 SU=0
2021-11-21 23:37:31,299: MA=0=0% MF=133 A=11360 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0  <----- stall started
2021-11-21 23:37:31,800: MA=0=0% MF=118 A=11377 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:32,301: MA=0=0% MF=118 A=11377 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:32,801: MA=0=0% MF=117 A=11377 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:33,302: MA=0=0% MF=118 A=11378 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:33,803: MA=0=0% MF=117 A=11379 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:34,304: MA=0=0% MF=118 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:34,805: MA=0=0% MF=118 A=11379 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:35,306: MA=0=0% MF=117 A=11379 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:35,806: MA=0=0% MF=127 A=11370 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:36,307: MA=0=0% MF=126 A=11369 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:36,808: MA=0=0% MF=127 A=11369 F=4 AF=1 IF=2 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:37,309: MA=0=0% MF=125 A=11369 F=3 AF=1 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:37,810: MA=0=0% MF=118 A=11377 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:38,310: MA=0=0% MF=118 A=11378 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:38,811: MA=0=0% MF=118 A=11378 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:39,312: MA=0=0% MF=118 A=11379 F=3 AF=0 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:39,813: MA=0=0% MF=117 A=11379 F=3 AF=0 IF=3 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:40,314: MA=0=0% MF=118 A=11379 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:40,814: MA=0=0% MF=118 A=11379 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:41,315: MA=0=0% MF=118 A=11380 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:41,816: MA=0=0% MF=119 A=11380 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:42,317: MA=0=0% MF=118 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:42,817: MA=0=0% MF=117 A=11380 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:43,318: MA=0=0% MF=118 A=11380 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:43,819: MA=0=0% MF=118 A=11380 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:44,320: MA=0=0% MF=117 A=11380 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:44,821: MA=0=0% MF=117 A=11380 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:45,321: MA=0=0% MF=118 A=11380 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:45,822: MA=0=0% MF=118 A=11380 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:46,323: MA=0=0% MF=117 A=11380 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:46,824: MA=0=0% MF=117 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:47,325: MA=0=0% MF=118 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:47,825: MA=0=0% MF=117 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:48,326: MA=0=0% MF=118 A=11380 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:48,827: MA=0=0% MF=117 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:49,328: MA=0=0% MF=118 A=11380 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:37:49,829: MA=0=0% MF=117 A=11380 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:50,329: MA=0=0% MF=118 A=11381 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:50,830: MA=0=0% MF=118 A=11381 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:51,331: MA=0=0% MF=118 A=11381 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:51,832: MA=0=0% MF=117 A=11381 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:52,333: MA=0=0% MF=117 A=11381 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:52,833: MA=0=0% MF=118 A=11381 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:53,334: MA=0=0% MF=117 A=11381 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:53,835: MA=0=0% MF=117 A=11381 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:54,336: MA=0=0% MF=118 A=11381 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:54,837: MA=0=0% MF=117 A=11382 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:55,338: MA=0=0% MF=117 A=11382 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:55,838: MA=0=0% MF=117 A=11382 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:56,339: MA=0=0% MF=118 A=11382 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:56,840: MA=0=0% MF=119 A=11382 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:37:57,341: MA=0=0% MF=118 A=11382 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:37:57,842: MA=0=0% MF=118 A=11382 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:58,342: MA=0=0% MF=117 A=11382 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:58,843: MA=0=0% MF=118 A=11382 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-21 23:37:59,344: MA=0=0% MF=118 A=11382 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:37:59,845: MA=0=0% MF=118 A=11382 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:38:00,346: MA=0=0% MF=118 A=11382 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:38:00,846: MA=0=0% MF=117 A=11382 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:38:01,347: MA=0=0% MF=117 A=11382 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:38:01,848: MA=0=0% MF=118 A=11382 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-21 23:38:02,349: MA=0=0% MF=118 A=11382 F=1 AF=1 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:38:02,850: MA=0=0% MF=117 A=11382 F=1 AF=1 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:38:03,351: MA=0=0% MF=118 A=11382 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:38:03,851: MA=0=0% MF=117 A=11383 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:38:04,352: MA=0=0% MF=117 A=11383 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:38:04,854: MA=0=0% MF=117 A=11383 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0
2021-11-21 23:38:05,354: MA=0=0% MF=117 A=11383 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0
2021-11-21 23:38:05,855: MA=0=0% MF=117 A=11383 F=1 AF=0 IF=0 D=0 C=1 SF=0 SU=0
2021-11-21 23:38:06,773: MA=0=0% MF=119 A=11383 F=0 AF=0 IF=0 D=0 C=0 SF=0 SU=0   <----- 35.5s stall is over, OOM
2021-11-21 23:38:07,274: MA=10504=89% MF=10639 A=869 F=6 AF=1 IF=6 D=0 C=6 SF=0 SU=0
2021-11-21 23:38:07,775: MA=10501=89% MF=10632 A=869 F=13 AF=1 IF=12 D=0 C=13 SF=0 SU=0
2021-11-21 23:38:08,276: MA=10499=89% MF=10627 A=869 F=18 AF=1 IF=17 D=0 C=18 SF=0 SU=0
2021-11-21 23:38:08,777: MA=10496=89% MF=10623 A=869 F=22 AF=1 IF=21 D=0 C=22 SF=0 SU=0
2021-11-21 23:38:09,278: MA=10493=89% MF=10616 A=869 F=28 AF=1 IF=27 D=0 C=28 SF=0 SU=0
2021-11-21 23:38:09,779: MA=10491=89% MF=10612 A=869 F=33 AF=1 IF=32 D=0 C=33 SF=0 SU=0
2021-11-21 23:38:10,281: MA=10488=89% MF=10605 A=869 F=40 AF=1 IF=39 D=0 C=40 SF=0 SU=0
2021-11-21 23:38:10,782: MA=10486=89% MF=10601 A=869 F=44 AF=1 IF=43 D=0 C=44 SF=0 SU=0
2021-11-21 23:38:11,283: MA=10482=89% MF=10594 A=869 F=50 AF=1 IF=49 D=0 C=50 SF=0 SU=0
2021-11-21 23:38:11,784: MA=10484=89% MF=10593 A=859 F=56 AF=1 IF=55 D=0 C=56 SF=0 SU=0

and PSI metrics at the same time:

2021-11-21 23:36:40,262: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-21 23:36:40,263: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-21 23:36:40,266: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:36:40,266: ======|=============|=============|
2021-11-21 23:36:40,266:  cpu  |      io     |    memory   |
2021-11-21 23:36:40,267: ----- | ----------- | ----------- |
2021-11-21 23:36:40,267:  some |  some  full |  some  full | interval
2021-11-21 23:36:40,267: ----- | ----- ----- | ----- ----- | --------
2021-11-21 23:36:42,269:   1.3 |   0.6   0.5 |   0.0   0.0 | 2.001
2021-11-21 23:36:44,272:   1.3 |   0.2   0.2 |   0.0   0.0 | 2.003
2021-11-21 23:36:46,273:   0.3 |   0.2   0.1 |   0.0   0.0 | 2.001
2021-11-21 23:36:48,275:   0.2 |   0.2   0.1 |   0.0   0.0 | 2.002
2021-11-21 23:36:50,277:   0.5 |  24.4  23.7 |  30.8  30.5 | 2.002
2021-11-21 23:36:52,279:   1.6 |  42.4  38.3 |  72.1  71.6 | 2.003
2021-11-21 23:36:54,282:   0.2 |  97.2  96.3 |   0.0   0.0 | 2.003
2021-11-21 23:36:56,285:   0.3 |  95.4  94.1 |   0.0   0.0 | 2.003
2021-11-21 23:36:58,287:   0.9 |  77.5  74.6 |   0.0   0.0 | 2.003
2021-11-21 23:37:00,290:   0.6 |  48.5  46.6 |   0.0   0.0 | 2.002
2021-11-21 23:37:02,292:   0.5 |  13.6  13.0 |   0.0   0.0 | 2.002
2021-11-21 23:37:04,294:   0.5 |  66.8  65.2 |  47.5  47.3 | 2.002
2021-11-21 23:37:06,297:   0.9 |  90.6  82.7 |  87.5  86.9 | 2.002
2021-11-21 23:37:08,299:   3.6 |  49.7  44.7 |  95.6  94.7 | 2.002
2021-11-21 23:37:10,301:   3.1 |  66.8  58.3 |  97.2  96.4 | 2.002
2021-11-21 23:37:12,303:   3.4 |  59.5  51.5 |  95.2  94.5 | 2.002
2021-11-21 23:37:14,305:   4.2 |  59.0  50.0 |  97.9  97.0 | 2.002
2021-11-21 23:37:16,307:   4.4 |  60.0  53.6 |  99.3  98.5 | 2.002
2021-11-21 23:37:18,309:   5.5 |  40.9  34.3 |  95.4  94.3 | 2.002
2021-11-21 23:37:20,312:   0.2 | 100.0  96.4 |   0.0   0.0 | 2.002
2021-11-21 23:37:22,315:   0.3 |  97.9  96.2 |   0.0   0.0 | 2.003
2021-11-21 23:37:24,317:   1.2 |  89.7  84.3 |   0.0   0.0 | 2.002
2021-11-21 23:37:26,319:   1.1 |  85.3  78.5 |   0.0   0.0 | 2.003
2021-11-21 23:37:28,321:   0.6 |  76.5  61.4 |   0.0   0.0 | 2.001
2021-11-21 23:37:30,323:   0.7 |  71.1  66.6 |   0.0   0.0 | 2.002
2021-11-21 23:37:32,325:   0.6 |  83.2  72.8 |  46.5  46.1 | 2.002
2021-11-21 23:37:34,327:   2.1 |  78.6  71.8 |  93.2  92.4 | 2.002
2021-11-21 23:37:36,329:   2.5 |  68.7  62.7 |  89.5  88.8 | 2.002
2021-11-21 23:37:38,331:   0.1 |  92.0  85.9 |  41.2  41.0 | 2.002
2021-11-21 23:37:40,333:   1.0 |  90.5  84.4 |  77.1  76.7 | 2.001
2021-11-21 23:37:42,335:   2.2 |  94.4  81.3 |  94.5  93.9 | 2.002
2021-11-21 23:37:44,337:   2.6 |  89.2  78.6 |  95.4  94.8 | 2.002
2021-11-21 23:37:46,339:   2.3 |  94.4  81.3 |  98.6  98.0 | 2.002
2021-11-21 23:37:48,341:   1.6 |  93.4  82.3 |  80.9  80.5 | 2.002
2021-11-21 23:37:50,343:   1.9 |  91.9  81.6 |  91.8  91.3 | 2.002
2021-11-21 23:37:52,345:   2.9 |  92.8  80.8 |  96.0  95.4 | 2.002
2021-11-21 23:37:54,347:   2.9 |  88.7  79.6 |  98.8  98.1 | 2.002
2021-11-21 23:37:56,349:   3.9 |  82.7  74.9 |  98.7  98.0 | 2.002
2021-11-21 23:37:58,351:   3.5 |  80.3  72.0 |  87.9  87.2 | 2.002
2021-11-21 23:38:00,353:   3.8 |  83.1  74.4 |  97.4  96.7 | 2.002
2021-11-21 23:38:02,355:   4.5 |  80.1  67.1 |  99.2  98.5 | 2.002
2021-11-21 23:38:04,357:   6.0 |  47.9  42.7 |  99.8  98.9 | 2.002
2021-11-21 23:38:06,359:   6.2 |  29.8  26.5 |  99.8  98.7 | 2.002
2021-11-21 23:38:08,362:   2.6 |  78.5  72.5 |  36.4  35.6 | 2.002
2021-11-21 23:38:10,364:   0.2 |  99.6  98.4 |   0.0   0.0 | 2.003
2021-11-21 23:38:12,367:   0.3 |  98.6  97.2 |   0.0   0.0 | 2.003
2021-11-21 23:38:14,370:   0.4 | 100.0  97.3 |   0.0   0.0 | 2.003
2021-11-21 23:38:16,372:   0.8 |  96.9  92.5 |   0.0   0.0 | 2.003
2021-11-21 23:38:18,373:   1.1 |  83.4  77.5 |   0.0   0.0 | 2.001
2021-11-21 23:38:20,375:   1.3 |  63.4  58.6 |   0.0   0.0 | 2.003
2021-11-21 23:38:22,377:   1.4 |  30.5  28.3 |   0.0   0.0 | 2.002
2021-11-21 23:38:24,380:   1.5 |   0.2   0.2 |   0.0   0.0 | 2.003

After the timeouts are set to 0, there is a decrease in freezing periods.
During freezing, the memory pressure remains high.


4. 5.16-rc1, reclaim_throttle() is disabled
===========================================

Code like this:
```
void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)
{
	return;
```
The result:

2021-11-21 23:41:17,252: Starting mem2log with interval 0.5s, mode: 1
2021-11-21 23:41:17,252: Log file: /tmpfs/27
2021-11-21 23:41:17,253: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:41:17,253: All values are in mebibytes
2021-11-21 23:41:17,253: MemTotal: 11798.5, SwapTotal: 0.0
2021-11-21 23:41:17,254: --
2021-11-21 23:41:17,254: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-21 23:41:17,254: F is File, AF is Active(file), IF is Inactive(file)
2021-11-21 23:41:17,254: D is Dirty, C is Clean file (File - Dirty)
2021-11-21 23:41:17,254: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-21 23:41:17,254: --
2021-11-21 23:41:17,254: MA=10262=87% MF=10199 A=972 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:17,755: MA=9258=78% MF=9196 A=1974 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:18,256: MA=8251=70% MF=8188 A=2987 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:18,757: MA=7206=61% MF=7144 A=4030 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:19,258: MA=6206=53% MF=6143 A=5031 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:19,759: MA=5156=44% MF=5093 A=6078 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:20,260: MA=4119=35% MF=4056 A=7115 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:20,761: MA=3067=26% MF=3004 A=8165 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:21,262: MA=2093=18% MF=2030 A=9136 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:21,763: MA=1042=9% MF=980 A=10180 F=281 AF=107 IF=174 D=0 C=280 SF=0 SU=0
2021-11-21 23:41:22,263: MA=110=1% MF=125 A=11153 F=207 AF=100 IF=107 D=0 C=207 SF=0 SU=0
2021-11-21 23:41:22,764: MA=1789=15% MF=1924 A=9553 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-21 23:41:23,265: MA=10503=89% MF=10636 A=855 F=7 AF=1 IF=7 D=0 C=7 SF=0 SU=0
2021-11-21 23:41:23,767: MA=10501=89% MF=10631 A=855 F=13 AF=1 IF=12 D=0 C=13 SF=0 SU=0
2021-11-21 23:41:24,268: MA=10499=89% MF=10627 A=855 F=17 AF=1 IF=16 D=0 C=17 SF=0 SU=0
2021-11-21 23:41:24,769: MA=10497=89% MF=10623 A=855 F=21 AF=1 IF=21 D=0 C=21 SF=0 SU=0
2021-11-21 23:41:25,270: MA=10496=89% MF=10619 A=855 F=26 AF=1 IF=26 D=0 C=26 SF=0 SU=0
2021-11-21 23:41:25,772: MA=10493=89% MF=10614 A=855 F=32 AF=1 IF=31 D=0 C=32 SF=0 SU=0
2021-11-21 23:41:26,273: MA=10490=89% MF=10607 A=855 F=38 AF=1 IF=37 D=0 C=38 SF=0 SU=0
2021-11-21 23:41:26,774: MA=10488=89% MF=10602 A=855 F=43 AF=1 IF=42 D=0 C=43 SF=0 SU=0
2021-11-21 23:41:27,275: MA=10483=89% MF=10594 A=856 F=50 AF=1 IF=49 D=0 C=50 SF=0 SU=0
2021-11-21 23:41:27,776: MA=10468=89% MF=10577 A=853 F=54 AF=1 IF=53 D=0 C=54 SF=0 SU=0
2021-11-21 23:41:28,278: MA=10462=89% MF=10570 A=858 F=57 AF=1 IF=56 D=0 C=57 SF=0 SU=0
2021-11-21 23:41:28,779: MA=10456=89% MF=10562 A=864 F=62 AF=1 IF=60 D=0 C=62 SF=0 SU=0
2021-11-21 23:41:29,280: MA=10375=88% MF=10478 A=945 F=66 AF=2 IF=65 D=0 C=66 SF=0 SU=0
2021-11-21 23:41:29,781: MA=9336=79% MF=9437 A=1982 F=71 AF=2 IF=69 D=0 C=71 SF=0 SU=0
2021-11-21 23:41:30,281: MA=8259=70% MF=8358 A=3055 F=76 AF=2 IF=74 D=0 C=76 SF=0 SU=0
2021-11-21 23:41:30,782: MA=7192=61% MF=7288 A=4118 F=81 AF=2 IF=78 D=0 C=80 SF=0 SU=0
2021-11-21 23:41:31,283: MA=6155=52% MF=6249 A=5153 F=84 AF=3 IF=81 D=0 C=84 SF=0 SU=0
2021-11-21 23:41:31,784: MA=5096=43% MF=5188 A=6207 F=89 AF=3 IF=86 D=1 C=88 SF=0 SU=0
2021-11-21 23:41:32,285: MA=4056=34% MF=4145 A=7243 F=94 AF=4 IF=90 D=1 C=93 SF=0 SU=0
2021-11-21 23:41:32,786: MA=3061=26% MF=3147 A=8223 F=99 AF=4 IF=95 D=1 C=98 SF=0 SU=0
2021-11-21 23:41:33,286: MA=2017=17% MF=2103 A=9261 F=102 AF=4 IF=97 D=1 C=100 SF=0 SU=0
2021-11-21 23:41:33,787: MA=974=8% MF=1058 A=10300 F=105 AF=4 IF=101 D=1 C=104 SF=0 SU=0
2021-11-21 23:41:34,288: MA=35=0% MF=119 A=11250 F=105 AF=37 IF=69 D=1 C=104 SF=0 SU=0
2021-11-21 23:41:34,789: MA=0=0% MF=119 A=11376 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-21 23:41:35,290: MA=7250=61% MF=7384 A=4103 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-21 23:41:35,791: MA=10509=89% MF=10639 A=859 F=12 AF=1 IF=11 D=0 C=12 SF=0 SU=0
2021-11-21 23:41:36,292: MA=10505=89% MF=10632 A=859 F=19 AF=0 IF=18 D=0 C=19 SF=0 SU=0
2021-11-21 23:41:36,794: MA=10503=89% MF=10628 A=859 F=23 AF=0 IF=23 D=0 C=23 SF=0 SU=0
2021-11-21 23:41:37,295: MA=10501=89% MF=10623 A=859 F=29 AF=0 IF=28 D=0 C=28 SF=0 SU=0
2021-11-21 23:41:37,796: MA=10498=89% MF=10618 A=859 F=33 AF=0 IF=32 D=0 C=33 SF=0 SU=0
2021-11-21 23:41:38,297: MA=10495=89% MF=10612 A=859 F=39 AF=1 IF=39 D=0 C=39 SF=0 SU=0
2021-11-21 23:41:38,798: MA=10492=89% MF=10607 A=859 F=44 AF=1 IF=43 D=0 C=43 SF=0 SU=0
2021-11-21 23:41:39,299: MA=10490=89% MF=10602 A=858 F=49 AF=1 IF=48 D=0 C=48 SF=0 SU=0
2021-11-21 23:41:39,801: MA=10481=89% MF=10590 A=859 F=54 AF=1 IF=53 D=0 C=54 SF=0 SU=0
2021-11-21 23:41:40,302: MA=10480=89% MF=10586 A=858 F=60 AF=1 IF=59 D=0 C=59 SF=0 SU=0
2021-11-21 23:41:40,803: MA=10470=89% MF=10574 A=863 F=65 AF=1 IF=64 D=0 C=64 SF=0 SU=0
2021-11-21 23:41:41,304: MA=9867=84% MF=9969 A=1461 F=69 AF=1 IF=68 D=0 C=69 SF=0 SU=0
2021-11-21 23:41:41,805: MA=8795=75% MF=8894 A=2528 F=74 AF=1 IF=73 D=0 C=74 SF=0 SU=0
2021-11-21 23:41:42,306: MA=7718=65% MF=7815 A=3604 F=79 AF=1 IF=78 D=1 C=78 SF=0 SU=0
2021-11-21 23:41:42,807: MA=6651=56% MF=6744 A=4663 F=86 AF=1 IF=85 D=1 C=85 SF=0 SU=0
2021-11-21 23:41:43,308: MA=5578=47% MF=5670 A=5731 F=90 AF=1 IF=89 D=1 C=89 SF=0 SU=0
2021-11-21 23:41:43,808: MA=4507=38% MF=4597 A=6797 F=94 AF=1 IF=93 D=1 C=93 SF=0 SU=0
2021-11-21 23:41:44,309: MA=3427=29% MF=3515 A=7872 F=98 AF=1 IF=97 D=1 C=97 SF=0 SU=0
2021-11-21 23:41:44,810: MA=2363=20% MF=2448 A=8932 F=103 AF=1 IF=102 D=1 C=102 SF=0 SU=0
2021-11-21 23:41:45,311: MA=1340=11% MF=1423 A=9939 F=107 AF=1 IF=106 D=1 C=106 SF=0 SU=0
2021-11-21 23:41:45,812: MA=282=2% MF=363 A=10989 F=110 AF=1 IF=109 D=1 C=109 SF=0 SU=0
2021-11-21 23:41:46,313: MA=0=0% MF=118 A=11377 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-21 23:41:46,814: MA=8614=73% MF=8747 A=2733 F=5 AF=1 IF=5 D=0 C=5 SF=0 SU=0
2021-11-21 23:41:47,315: MA=10501=89% MF=10632 A=860 F=12 AF=1 IF=11 D=0 C=12 SF=0 SU=0
2021-11-21 23:41:47,816: MA=10499=89% MF=10626 A=860 F=17 AF=1 IF=17 D=0 C=17 SF=0 SU=0
2021-11-21 23:41:48,317: MA=10496=89% MF=10620 A=860 F=25 AF=1 IF=24 D=0 C=25 SF=0 SU=0
2021-11-21 23:41:48,818: MA=10492=89% MF=10613 A=860 F=31 AF=1 IF=30 D=0 C=31 SF=0 SU=0
2021-11-21 23:41:49,320: MA=10490=89% MF=10608 A=860 F=37 AF=1 IF=36 D=0 C=37 SF=0 SU=0

and PSI metrics at the same time:

2021-11-21 23:41:13,285: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-21 23:41:13,286: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-21 23:41:13,290: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-21 23:41:13,290: ======|=============|=============|
2021-11-21 23:41:13,290:  cpu  |      io     |    memory   |
2021-11-21 23:41:13,290: ----- | ----------- | ----------- |
2021-11-21 23:41:13,290:  some |  some  full |  some  full | interval
2021-11-21 23:41:13,290: ----- | ----- ----- | ----- ----- | --------
2021-11-21 23:41:15,293:   1.6 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-21 23:41:17,295:   1.5 |   1.0   0.9 |   0.0   0.0 | 2.003
2021-11-21 23:41:19,297:   0.4 |   0.0   0.0 |   0.0   0.0 | 2.001
2021-11-21 23:41:21,299:   0.4 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-21 23:41:23,301:   0.6 |  33.8  29.4 |   8.7   8.6 | 2.002
2021-11-21 23:41:25,304:   0.2 |  96.5  95.7 |   0.0   0.0 | 2.003
2021-11-21 23:41:27,306:   0.6 |  94.5  91.5 |   0.0   0.0 | 2.003
2021-11-21 23:41:29,309:   1.3 |  92.2  83.2 |   0.0   0.0 | 2.003
2021-11-21 23:41:31,311:   0.6 |  80.2  68.9 |   0.0   0.0 | 2.002
2021-11-21 23:41:33,313:   0.7 |  62.6  57.9 |   0.0   0.0 | 2.002
2021-11-21 23:41:35,315:   3.8 |  56.7  50.7 |  16.2  15.9 | 2.002
2021-11-21 23:41:37,317:   0.3 |  95.8  94.2 |   0.0   0.0 | 2.002
2021-11-21 23:41:39,319:   0.6 |  98.8  96.1 |   0.0   0.0 | 2.003
2021-11-21 23:41:41,321:   1.2 |  97.8  88.5 |   0.0   0.0 | 2.001
2021-11-21 23:41:43,323:   0.5 |  75.7  69.2 |   0.0   0.0 | 2.002
2021-11-21 23:41:45,325:   0.6 |  65.5  59.8 |   0.0   0.0 | 2.002
2021-11-21 23:41:47,327:   1.0 |  72.7  67.1 |  14.3  13.8 | 2.002
2021-11-21 23:41:49,330:   0.2 |  96.7  95.7 |   0.0   0.0 | 2.003
2021-11-21 23:41:51,333:   0.4 |  96.8  94.8 |   0.0   0.0 | 2.003
2021-11-21 23:41:53,335:   1.1 |  89.6  82.5 |   0.0   0.0 | 2.003
2021-11-21 23:41:55,337:   1.3 |  76.8  70.9 |   0.0   0.0 | 2.002
2021-11-21 23:41:57,340:   1.4 |  69.2  63.1 |   0.0   0.0 | 2.003
2021-11-21 23:41:59,342:   1.5 |  35.5  31.3 |   0.0   0.0 | 2.003
2021-11-21 23:42:01,345:   1.6 |   6.1   5.4 |   0.0   0.0 | 2.003

After disabling the reclaim_throttle() function, the behavior is restored
as in old kernels - there is no delay, the killer comes quickly. Memory
pressure does not reach high values. 


5. 5.16-rc1, MAX_RECLAIM_RETRIES=0, default timeouts
====================================================

Setting MAX_RECLAIM_RETRIES to 0 greatly reduces the stall intervals:

2021-11-22 00:08:46,259: Starting mem2log with interval 0.5s, mode: 1
2021-11-22 00:08:46,259: Log file: /tmpfs/27
2021-11-22 00:08:46,260: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 00:08:46,260: All values are in mebibytes
2021-11-22 00:08:46,260: MemTotal: 11798.5, SwapTotal: 0.0
2021-11-22 00:08:46,260: --
2021-11-22 00:08:46,260: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-22 00:08:46,260: F is File, AF is Active(file), IF is Inactive(file)
2021-11-22 00:08:46,261: D is Dirty, C is Clean file (File - Dirty)
2021-11-22 00:08:46,261: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-22 00:08:46,261: --
2021-11-22 00:08:46,261: MA=10239=87% MF=10198 A=911 F=353 AF=68 IF=285 D=0 C=353 SF=0 SU=0
2021-11-22 00:08:46,762: MA=9207=78% MF=9166 A=1942 F=353 AF=68 IF=285 D=0 C=353 SF=0 SU=0
2021-11-22 00:08:47,263: MA=8153=69% MF=8112 A=2992 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:47,764: MA=7092=60% MF=7051 A=4051 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:48,265: MA=6044=51% MF=6003 A=5096 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:48,766: MA=5013=42% MF=4972 A=6125 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:49,267: MA=4001=34% MF=3959 A=7136 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:49,768: MA=2934=25% MF=2893 A=8200 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:50,269: MA=1884=16% MF=1843 A=9247 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:50,770: MA=841=7% MF=799 A=10288 F=354 AF=68 IF=286 D=0 C=354 SF=0 SU=0
2021-11-22 00:08:51,271: MA=220=2% MF=240 A=11014 F=230 AF=115 IF=115 D=0 C=230 SF=0 SU=0
2021-11-22 00:08:51,772: MA=117=1% MF=240 A=11220 F=25 AF=13 IF=12 D=0 C=25 SF=0 SU=0
2021-11-22 00:08:52,273: MA=115=1% MF=240 A=11221 F=21 AF=12 IF=9 D=0 C=21 SF=0 SU=0
2021-11-22 00:08:52,774: MA=114=1% MF=242 A=11227 F=15 AF=8 IF=7 D=0 C=15 SF=0 SU=0
2021-11-22 00:08:53,276: MA=109=1% MF=240 A=11235 F=10 AF=5 IF=5 D=0 C=10 SF=0 SU=0
2021-11-22 00:08:53,777: MA=108=1% MF=240 A=11238 F=7 AF=4 IF=4 D=0 C=7 SF=0 SU=0  <----- stall started
2021-11-22 00:08:54,278: MA=107=1% MF=240 A=11239 F=5 AF=3 IF=3 D=0 C=5 SF=0 SU=0
2021-11-22 00:08:54,780: MA=108=1% MF=242 A=11239 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-22 00:08:55,281: MA=105=1% MF=239 A=11241 F=6 AF=1 IF=4 D=0 C=6 SF=0 SU=0
2021-11-22 00:08:55,782: MA=106=1% MF=240 A=11241 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-22 00:08:56,283: MA=104=1% MF=239 A=11242 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0   <----- 3s stall is over, OOM
2021-11-22 00:08:56,784: MA=9696=82% MF=9829 A=1636 F=4 AF=1 IF=4 D=0 C=4 SF=0 SU=0
2021-11-22 00:08:57,286: MA=10466=89% MF=10597 A=889 F=11 AF=1 IF=11 D=0 C=11 SF=0 SU=0
2021-11-22 00:08:57,787: MA=10465=89% MF=10593 A=889 F=16 AF=1 IF=16 D=0 C=16 SF=0 SU=0
2021-11-22 00:08:58,288: MA=10464=89% MF=10590 A=889 F=20 AF=1 IF=19 D=0 C=20 SF=0 SU=0
2021-11-22 00:08:58,789: MA=10461=89% MF=10585 A=889 F=25 AF=1 IF=24 D=0 C=25 SF=0 SU=0
2021-11-22 00:08:59,290: MA=10459=89% MF=10580 A=889 F=30 AF=1 IF=29 D=0 C=30 SF=0 SU=0
2021-11-22 00:08:59,792: MA=10457=89% MF=10575 A=889 F=35 AF=1 IF=34 D=0 C=35 SF=0 SU=0
2021-11-22 00:09:00,293: MA=10454=89% MF=10569 A=889 F=41 AF=1 IF=40 D=0 C=41 SF=0 SU=0
2021-11-22 00:09:00,794: MA=10451=89% MF=10564 A=889 F=46 AF=1 IF=45 D=0 C=46 SF=0 SU=0
2021-11-22 00:09:01,295: MA=10449=89% MF=10560 A=889 F=51 AF=1 IF=49 D=0 C=51 SF=0 SU=0
2021-11-22 00:09:01,796: MA=10447=89% MF=10555 A=889 F=55 AF=1 IF=54 D=0 C=55 SF=0 SU=0
2021-11-22 00:09:02,297: MA=10444=89% MF=10550 A=889 F=61 AF=2 IF=59 D=0 C=61 SF=0 SU=0
2021-11-22 00:09:02,799: MA=10442=88% MF=10545 A=889 F=66 AF=2 IF=64 D=0 C=66 SF=0 SU=0
2021-11-22 00:09:03,300: MA=10439=88% MF=10540 A=889 F=70 AF=2 IF=68 D=0 C=70 SF=0 SU=0
2021-11-22 00:09:03,801: MA=10103=86% MF=10202 A=1223 F=74 AF=3 IF=71 D=0 C=74 SF=0 SU=0
2021-11-22 00:09:04,301: MA=9029=77% MF=9126 A=2291 F=80 AF=3 IF=77 D=0 C=79 SF=0 SU=0
2021-11-22 00:09:04,802: MA=7957=67% MF=8051 A=3360 F=83 AF=3 IF=79 D=0 C=83 SF=0 SU=0
2021-11-22 00:09:05,303: MA=6882=58% MF=6975 A=4429 F=86 AF=4 IF=82 D=0 C=86 SF=0 SU=0
2021-11-22 00:09:05,804: MA=5814=49% MF=5905 A=5493 F=89 AF=4 IF=85 D=0 C=89 SF=0 SU=0
2021-11-22 00:09:06,305: MA=4742=40% MF=4832 A=6559 F=93 AF=4 IF=88 D=0 C=93 SF=0 SU=0
2021-11-22 00:09:06,806: MA=3663=31% MF=3751 A=7633 F=96 AF=4 IF=91 D=0 C=95 SF=0 SU=0
2021-11-22 00:09:07,307: MA=2587=22% MF=2674 A=8706 F=98 AF=5 IF=93 D=0 C=98 SF=0 SU=0
2021-11-22 00:09:07,808: MA=1516=13% MF=1602 A=9770 F=102 AF=5 IF=96 D=0 C=101 SF=0 SU=0
2021-11-22 00:09:08,308: MA=462=4% MF=545 A=10823 F=106 AF=5 IF=101 D=0 C=106 SF=0 SU=0
2021-11-22 00:09:08,809: MA=154=1% MF=241 A=11156 F=98 AF=49 IF=49 D=0 C=98 SF=0 SU=0
2021-11-22 00:09:09,311: MA=111=1% MF=240 A=11238 F=15 AF=7 IF=8 D=0 C=15 SF=0 SU=0
2021-11-22 00:09:09,812: MA=111=1% MF=240 A=11241 F=13 AF=6 IF=7 D=0 C=13 SF=0 SU=0
2021-11-22 00:09:10,313: MA=109=1% MF=240 A=11243 F=10 AF=5 IF=5 D=0 C=10 SF=0 SU=0
2021-11-22 00:09:10,815: MA=108=1% MF=240 A=11246 F=8 AF=4 IF=4 D=0 C=8 SF=0 SU=0  <----- stall started
2021-11-22 00:09:11,316: MA=107=1% MF=240 A=11248 F=6 AF=3 IF=3 D=0 C=6 SF=0 SU=0
2021-11-22 00:09:11,817: MA=108=1% MF=242 A=11248 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-22 00:09:12,318: MA=106=1% MF=240 A=11250 F=4 AF=2 IF=2 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:12,820: MA=107=1% MF=242 A=11250 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:09:13,321: MA=790=7% MF=925 A=10558 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0   <----- 3s stall is over, OOM
2021-11-22 00:09:13,822: MA=10470=89% MF=10602 A=890 F=7 AF=2 IF=5 D=0 C=7 SF=0 SU=0
2021-11-22 00:09:14,323: MA=10468=89% MF=10597 A=890 F=13 AF=2 IF=11 D=0 C=13 SF=0 SU=0
2021-11-22 00:09:14,824: MA=10465=89% MF=10593 A=890 F=17 AF=2 IF=15 D=0 C=17 SF=0 SU=0
2021-11-22 00:09:15,325: MA=10463=89% MF=10587 A=890 F=24 AF=3 IF=21 D=0 C=24 SF=0 SU=0
2021-11-22 00:09:15,827: MA=10460=89% MF=10582 A=890 F=30 AF=3 IF=26 D=0 C=30 SF=0 SU=0
2021-11-22 00:09:16,328: MA=10459=89% MF=10577 A=890 F=36 AF=3 IF=32 D=0 C=36 SF=0 SU=0
2021-11-22 00:09:16,829: MA=10457=89% MF=10572 A=890 F=41 AF=3 IF=38 D=0 C=41 SF=0 SU=0
2021-11-22 00:09:17,330: MA=10454=89% MF=10568 A=890 F=45 AF=3 IF=42 D=0 C=45 SF=0 SU=0
2021-11-22 00:09:17,832: MA=10452=89% MF=10562 A=890 F=52 AF=3 IF=48 D=0 C=52 SF=0 SU=0
2021-11-22 00:09:18,333: MA=10450=89% MF=10557 A=890 F=57 AF=3 IF=54 D=0 C=57 SF=0 SU=0
2021-11-22 00:09:18,834: MA=10448=89% MF=10553 A=890 F=61 AF=4 IF=58 D=0 C=61 SF=0 SU=0
2021-11-22 00:09:19,335: MA=10424=88% MF=10527 A=912 F=66 AF=4 IF=63 D=0 C=66 SF=0 SU=0
2021-11-22 00:09:19,836: MA=9450=80% MF=9550 A=1883 F=72 AF=4 IF=68 D=0 C=72 SF=0 SU=0
2021-11-22 00:09:20,337: MA=8477=72% MF=8574 A=2853 F=77 AF=4 IF=73 D=0 C=77 SF=0 SU=0
2021-11-22 00:09:20,838: MA=7495=64% MF=7590 A=3830 F=82 AF=4 IF=78 D=0 C=82 SF=0 SU=0
2021-11-22 00:09:21,339: MA=6480=55% MF=6573 A=4841 F=86 AF=4 IF=82 D=0 C=86 SF=0 SU=0
2021-11-22 00:09:21,840: MA=5433=46% MF=5524 A=5883 F=91 AF=4 IF=86 D=0 C=91 SF=0 SU=0
2021-11-22 00:09:22,340: MA=4382=37% MF=4470 A=6931 F=95 AF=4 IF=90 D=0 C=95 SF=0 SU=0
2021-11-22 00:09:22,841: MA=3323=28% MF=3410 A=7983 F=100 AF=4 IF=95 D=0 C=100 SF=0 SU=0
2021-11-22 00:09:23,342: MA=2272=19% MF=2355 A=9027 F=106 AF=4 IF=102 D=0 C=106 SF=0 SU=0
2021-11-22 00:09:23,843: MA=1224=10% MF=1305 A=10072 F=110 AF=5 IF=106 D=0 C=110 SF=0 SU=0
2021-11-22 00:09:24,344: MA=178=2% MF=258 A=11110 F=113 AF=5 IF=108 D=0 C=113 SF=0 SU=0
2021-11-22 00:09:24,844: MA=135=1% MF=241 A=11194 F=61 AF=30 IF=30 D=0 C=61 SF=0 SU=0
2021-11-22 00:09:25,346: MA=111=1% MF=240 A=11241 F=13 AF=6 IF=7 D=0 C=13 SF=0 SU=0
2021-11-22 00:09:25,847: MA=110=1% MF=240 A=11244 F=11 AF=5 IF=5 D=0 C=11 SF=0 SU=0
2021-11-22 00:09:26,348: MA=108=1% MF=240 A=11246 F=9 AF=5 IF=4 D=0 C=9 SF=0 SU=0  <----- stall started
2021-11-22 00:09:26,849: MA=106=1% MF=239 A=11249 F=6 AF=2 IF=3 D=0 C=6 SF=0 SU=0
2021-11-22 00:09:27,351: MA=106=1% MF=240 A=11250 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-22 00:09:27,852: MA=106=1% MF=240 A=11250 F=4 AF=1 IF=2 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:28,353: MA=106=1% MF=241 A=11251 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:09:28,854: MA=105=1% MF=239 A=11251 F=4 AF=1 IF=4 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:29,454: MA=106=1% MF=242 A=11251 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:09:29,955: MA=105=1% MF=240 A=11251 F=3 AF=0 IF=3 D=0 C=3 SF=0 SU=0
2021-11-22 00:09:30,489: MA=105=1% MF=238 A=11251 F=5 AF=0 IF=4 D=0 C=5 SF=0 SU=0
2021-11-22 00:09:30,991: MA=105=1% MF=239 A=11251 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:31,492: MA=106=1% MF=241 A=11252 F=2 AF=1 IF=1 D=0 C=2 SF=0 SU=0
2021-11-22 00:09:31,993: MA=104=1% MF=239 A=11253 F=4 AF=1 IF=3 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:32,494: MA=104=1% MF=239 A=11253 F=4 AF=0 IF=3 D=0 C=4 SF=0 SU=0
2021-11-22 00:09:32,996: MA=105=1% MF=240 A=11253 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0
2021-11-22 00:09:33,497: MA=105=1% MF=240 A=11253 F=3 AF=0 IF=2 D=0 C=3 SF=0 SU=0   <----- 7.5s stall is over, OOM
2021-11-22 00:09:33,998: MA=3284=28% MF=3418 A=8065 F=5 AF=0 IF=5 D=0 C=5 SF=0 SU=0
2021-11-22 00:09:34,499: MA=10468=89% MF=10600 A=892 F=8 AF=1 IF=7 D=0 C=8 SF=0 SU=0
2021-11-22 00:09:35,000: MA=10466=89% MF=10596 A=892 F=13 AF=1 IF=12 D=0 C=13 SF=0 SU=0
2021-11-22 00:09:35,501: MA=10458=89% MF=10585 A=894 F=20 AF=1 IF=18 D=0 C=20 SF=0 SU=0
2021-11-22 00:09:36,003: MA=10458=89% MF=10582 A=894 F=24 AF=1 IF=23 D=0 C=24 SF=0 SU=0
2021-11-22 00:09:36,504: MA=10456=89% MF=10578 A=894 F=28 AF=2 IF=26 D=0 C=28 SF=0 SU=0

and PSI metrics at the same time:

2021-11-22 00:08:43,631: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-22 00:08:43,631: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-22 00:08:43,633: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 00:08:43,633: ======|=============|=============|
2021-11-22 00:08:43,633:  cpu  |      io     |    memory   |
2021-11-22 00:08:43,633: ----- | ----------- | ----------- |
2021-11-22 00:08:43,633:  some |  some  full |  some  full | interval
2021-11-22 00:08:43,633: ----- | ----- ----- | ----- ----- | --------
2021-11-22 00:08:45,636:   1.9 |   0.5   0.5 |   0.0   0.0 | 2.002
2021-11-22 00:08:47,637:   1.1 |   0.8   0.7 |   0.0   0.0 | 2.002
2021-11-22 00:08:49,640:   0.7 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-22 00:08:51,642:   1.1 |   1.3   1.2 |   8.3   8.2 | 2.002
2021-11-22 00:08:53,645:   0.9 |  23.6  21.7 |  75.3  73.9 | 2.003
2021-11-22 00:08:55,647:   1.0 |  42.8  40.9 |  74.5  74.1 | 2.003
2021-11-22 00:08:57,650:   1.2 |  69.8  65.2 |  42.6  42.0 | 2.003
2021-11-22 00:08:59,653:   0.2 |  97.4  96.3 |   0.0   0.0 | 2.003
2021-11-22 00:09:01,656:   1.0 |  94.0  90.5 |   0.0   0.0 | 2.003
2021-11-22 00:09:03,658:   1.3 |  90.0  84.1 |   0.0   0.0 | 2.003
2021-11-22 00:09:05,660:   1.0 |  80.0  66.5 |   0.0   0.0 | 2.001
2021-11-22 00:09:07,662:   0.9 |  76.1  69.4 |   0.0   0.0 | 2.002
2021-11-22 00:09:09,665:   1.4 |  66.6  62.1 |  26.8  26.4 | 2.003
2021-11-22 00:09:11,667:   1.0 |  25.0  23.9 |  86.7  86.4 | 2.003
2021-11-22 00:09:13,670:   1.0 |  59.4  54.5 |  75.8  73.3 | 2.003
2021-11-22 00:09:15,673:   0.2 |  93.9  92.7 |   6.3   6.2 | 2.003
2021-11-22 00:09:17,676:   0.3 |  96.0  94.7 |   0.0   0.0 | 2.003
2021-11-22 00:09:19,678:   1.3 |  86.0  81.4 |   0.0   0.0 | 2.003
2021-11-22 00:09:21,681:   0.9 |  72.3  68.6 |   0.0   0.0 | 2.002
2021-11-22 00:09:23,683:   0.9 |  70.6  63.2 |   0.0   0.0 | 2.002
2021-11-22 00:09:25,684:   1.3 |  59.3  53.5 |  30.9  30.3 | 2.001
2021-11-22 00:09:27,686:   1.1 |  50.9  48.3 |  81.9  81.4 | 2.002
2021-11-22 00:09:29,689:   1.6 |  54.1  51.8 |  93.7  93.1 | 2.003
2021-11-22 00:09:31,691:   2.5 |  58.9  55.8 |  96.0  95.3 | 2.003
2021-11-22 00:09:33,694:   2.5 |  37.2  34.9 | 100.0  99.2 | 2.003
2021-11-22 00:09:35,697:   1.1 |  94.1  89.8 |  51.7  51.2 | 2.003
2021-11-22 00:09:37,700:   0.3 |  99.4  98.0 |  13.5  13.3 | 2.003
2021-11-22 00:09:39,702:   1.2 |  94.4  89.9 |   3.4   3.2 | 2.003
2021-11-22 00:09:41,705:   1.2 |  81.0  77.3 |   0.0   0.0 | 2.002
2021-11-22 00:09:43,708:   1.2 |  78.4  73.4 |   0.0   0.0 | 2.003
2021-11-22 00:09:45,710:   1.5 |  86.1  80.4 |   0.0   0.0 | 2.003
2021-11-22 00:09:47,712:   1.6 |  80.0  73.2 |   0.0   0.0 | 2.001
2021-11-22 00:09:49,713:   1.8 |  52.4  47.6 |   0.0   0.0 | 2.001
2021-11-22 00:09:51,715:   2.0 |  50.0  46.3 |   0.0   0.0 | 2.003
2021-11-22 00:09:53,718:   1.8 |  24.4  22.6 |   0.0   0.0 | 2.003
2021-11-22 00:09:55,720:   1.7 |   0.4   0.4 |   0.0   0.0 | 2.002

Zeroing the MAX_RECLAIM_RETRIES gives a better result than zeroing the
timeouts.


6. 5.16-rc1, MAX_RECLAIM_RETRIES=0, timeout=0
=============================================

Combination of MAX_RECLAIM_RETRIES=0 and timeout=0 gives almost
satisfactory result:

2021-11-22 00:42:21,651: Starting mem2log with interval 0.5s, mode: 1
2021-11-22 00:42:21,651: Log file: /tmpfs/27
2021-11-22 00:42:21,652: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 00:42:21,652: All values are in mebibytes
2021-11-22 00:42:21,652: MemTotal: 11798.5, SwapTotal: 0.0
2021-11-22 00:42:21,652: --
2021-11-22 00:42:21,652: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-22 00:42:21,652: F is File, AF is Active(file), IF is Inactive(file)
2021-11-22 00:42:21,652: D is Dirty, C is Clean file (File - Dirty)
2021-11-22 00:42:21,652: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-22 00:42:21,653: --
2021-11-22 00:42:21,653: MA=10264=87% MF=10288 A=949 F=225 AF=40 IF=185 D=0 C=225 SF=0 SU=0
2021-11-22 00:42:22,153: MA=9192=78% MF=9216 A=2021 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:22,654: MA=8121=69% MF=8145 A=3088 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:23,155: MA=7043=60% MF=7067 A=4164 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:23,656: MA=5961=51% MF=5984 A=5244 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:24,157: MA=4893=41% MF=4916 A=6303 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:24,658: MA=3811=32% MF=3835 A=7383 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:25,159: MA=2726=23% MF=2750 A=8466 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:25,660: MA=1659=14% MF=1682 A=9530 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:26,160: MA=587=5% MF=610 A=10600 F=225 AF=40 IF=184 D=0 C=224 SF=0 SU=0
2021-11-22 00:42:26,661: MA=188=2% MF=241 A=11078 F=166 AF=83 IF=83 D=0 C=166 SF=0 SU=0
2021-11-22 00:42:27,162: MA=107=1% MF=240 A=11239 F=6 AF=3 IF=3 D=0 C=6 SF=0 SU=0
2021-11-22 00:42:27,663: MA=105=1% MF=240 A=11243 F=2 AF=1 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:42:28,164: MA=106=1% MF=241 A=11243 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:42:28,666: MA=105=1% MF=240 A=11243 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0
2021-11-22 00:42:29,167: MA=105=1% MF=241 A=11243 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-22 00:42:29,668: MA=2009=17% MF=2144 A=9321 F=2 AF=0 IF=2 D=0 C=2 SF=0 SU=0
2021-11-22 00:42:30,169: MA=10476=89% MF=10608 A=876 F=9 AF=0 IF=9 D=0 C=9 SF=0 SU=0
2021-11-22 00:42:30,670: MA=10474=89% MF=10602 A=876 F=15 AF=0 IF=15 D=0 C=15 SF=0 SU=0
2021-11-22 00:42:31,171: MA=10472=89% MF=10599 A=876 F=19 AF=0 IF=18 D=0 C=19 SF=0 SU=0
2021-11-22 00:42:31,672: MA=10469=89% MF=10592 A=876 F=27 AF=1 IF=26 D=0 C=26 SF=0 SU=0
2021-11-22 00:42:32,174: MA=10467=89% MF=10587 A=876 F=32 AF=1 IF=32 D=0 C=32 SF=0 SU=0
2021-11-22 00:42:32,675: MA=10464=89% MF=10582 A=876 F=37 AF=1 IF=37 D=0 C=37 SF=0 SU=0
2021-11-22 00:42:33,176: MA=10461=89% MF=10576 A=876 F=42 AF=1 IF=41 D=0 C=42 SF=0 SU=0
2021-11-22 00:42:33,677: MA=10459=89% MF=10572 A=876 F=46 AF=1 IF=45 D=0 C=46 SF=0 SU=0
2021-11-22 00:42:34,179: MA=10457=89% MF=10567 A=876 F=51 AF=1 IF=50 D=0 C=51 SF=0 SU=0
2021-11-22 00:42:34,680: MA=10454=89% MF=10563 A=876 F=56 AF=2 IF=54 D=0 C=56 SF=0 SU=0
2021-11-22 00:42:35,181: MA=10363=88% MF=10469 A=966 F=60 AF=2 IF=58 D=0 C=60 SF=0 SU=0
2021-11-22 00:42:35,682: MA=9366=79% MF=9470 A=1959 F=64 AF=2 IF=62 D=0 C=64 SF=0 SU=0
2021-11-22 00:42:36,183: MA=8378=71% MF=8480 A=2942 F=69 AF=3 IF=66 D=0 C=69 SF=0 SU=0
2021-11-22 00:42:36,684: MA=7386=63% MF=7486 A=3931 F=73 AF=3 IF=70 D=0 C=73 SF=0 SU=0
2021-11-22 00:42:37,184: MA=6395=54% MF=6493 A=4916 F=77 AF=3 IF=74 D=0 C=77 SF=0 SU=0
2021-11-22 00:42:37,685: MA=5317=45% MF=5413 A=5990 F=81 AF=3 IF=78 D=0 C=81 SF=0 SU=0
2021-11-22 00:42:38,186: MA=4254=36% MF=4348 A=7046 F=85 AF=3 IF=82 D=0 C=85 SF=0 SU=0
2021-11-22 00:42:38,687: MA=3174=27% MF=3265 A=8125 F=90 AF=3 IF=87 D=1 C=89 SF=0 SU=0
2021-11-22 00:42:39,188: MA=2099=18% MF=2189 A=9193 F=94 AF=4 IF=90 D=1 C=93 SF=0 SU=0
2021-11-22 00:42:39,689: MA=1022=9% MF=1110 A=10266 F=97 AF=5 IF=93 D=1 C=96 SF=0 SU=0
2021-11-22 00:42:40,190: MA=146=1% MF=240 A=11170 F=84 AF=42 IF=42 D=0 C=84 SF=0 SU=0
2021-11-22 00:42:40,691: MA=106=1% MF=240 A=11250 F=5 AF=3 IF=2 D=0 C=5 SF=0 SU=0
2021-11-22 00:42:41,192: MA=105=1% MF=240 A=11253 F=2 AF=0 IF=1 D=0 C=2 SF=0 SU=0  <----- OOM
2021-11-22 00:42:41,717: MA=6800=58% MF=6934 A=4550 F=5 AF=1 IF=5 D=0 C=5 SF=0 SU=0
2021-11-22 00:42:42,219: MA=10485=89% MF=10615 A=876 F=12 AF=1 IF=11 D=0 C=12 SF=0 SU=0
2021-11-22 00:42:42,720: MA=10482=89% MF=10611 A=876 F=16 AF=1 IF=16 D=0 C=16 SF=0 SU=0
2021-11-22 00:42:43,221: MA=10479=89% MF=10604 A=876 F=23 AF=1 IF=22 D=0 C=23 SF=0 SU=0
2021-11-22 00:42:43,722: MA=10476=89% MF=10599 A=876 F=27 AF=1 IF=26 D=0 C=27 SF=0 SU=0
2021-11-22 00:42:44,224: MA=10473=89% MF=10593 A=876 F=33 AF=1 IF=32 D=0 C=33 SF=0 SU=0
2021-11-22 00:42:44,724: MA=10471=89% MF=10588 A=876 F=38 AF=1 IF=38 D=0 C=38 SF=0 SU=0
2021-11-22 00:42:45,226: MA=10468=89% MF=10582 A=876 F=44 AF=1 IF=43 D=0 C=44 SF=0 SU=0
2021-11-22 00:42:45,727: MA=10466=89% MF=10578 A=876 F=49 AF=1 IF=48 D=0 C=49 SF=0 SU=0
2021-11-22 00:42:46,228: MA=10463=89% MF=10573 A=876 F=54 AF=1 IF=54 D=0 C=54 SF=0 SU=0
2021-11-22 00:42:46,729: MA=10462=89% MF=10569 A=876 F=59 AF=1 IF=58 D=0 C=59 SF=0 SU=0
2021-11-22 00:42:47,230: MA=10460=89% MF=10565 A=876 F=63 AF=1 IF=62 D=0 C=63 SF=0 SU=0
2021-11-22 00:42:47,732: MA=10458=89% MF=10561 A=877 F=68 AF=1 IF=68 D=0 C=68 SF=0 SU=0
2021-11-22 00:42:48,232: MA=10455=89% MF=10555 A=877 F=74 AF=1 IF=73 D=0 C=74 SF=0 SU=0
2021-11-22 00:42:48,734: MA=10454=89% MF=10552 A=877 F=77 AF=1 IF=76 D=0 C=77 SF=0 SU=0
2021-11-22 00:42:49,235: MA=10450=89% MF=10545 A=877 F=84 AF=1 IF=83 D=0 C=84 SF=0 SU=0
2021-11-22 00:42:49,736: MA=10450=89% MF=10543 A=868 F=89 AF=1 IF=88 D=0 C=89 SF=0 SU=0
2021-11-22 00:42:50,237: MA=9653=82% MF=9742 A=1661 F=94 AF=1 IF=92 D=0 C=94 SF=0 SU=0
2021-11-22 00:42:50,738: MA=8581=73% MF=8669 A=2724 F=98 AF=1 IF=97 D=0 C=98 SF=0 SU=0
2021-11-22 00:42:51,239: MA=7544=64% MF=7629 A=3759 F=104 AF=2 IF=103 D=0 C=104 SF=0 SU=0
2021-11-22 00:42:51,739: MA=6470=55% MF=6552 A=4831 F=109 AF=2 IF=107 D=0 C=109 SF=0 SU=0
2021-11-22 00:42:52,240: MA=5395=46% MF=5476 A=5901 F=113 AF=2 IF=111 D=0 C=112 SF=0 SU=0
2021-11-22 00:42:52,741: MA=4319=37% MF=4398 A=6973 F=116 AF=2 IF=114 D=1 C=115 SF=0 SU=0
2021-11-22 00:42:53,242: MA=3247=28% MF=3324 A=8042 F=119 AF=2 IF=117 D=1 C=118 SF=0 SU=0
2021-11-22 00:42:53,743: MA=2177=18% MF=2254 A=9106 F=120 AF=2 IF=118 D=1 C=118 SF=0 SU=0
2021-11-22 00:42:54,244: MA=1121=10% MF=1198 A=10157 F=120 AF=2 IF=119 D=1 C=119 SF=0 SU=0
2021-11-22 00:42:54,744: MA=163=1% MF=240 A=11138 F=119 AF=17 IF=102 D=1 C=118 SF=0 SU=0
2021-11-22 00:42:55,245: MA=105=1% MF=240 A=11254 F=3 AF=1 IF=2 D=0 C=3 SF=0 SU=0
2021-11-22 00:42:55,747: MA=105=1% MF=241 A=11255 F=1 AF=0 IF=1 D=0 C=1 SF=0 SU=0  <----- OOM
2021-11-22 00:42:56,248: MA=9986=85% MF=10119 A=1353 F=5 AF=1 IF=5 D=0 C=5 SF=0 SU=0
2021-11-22 00:42:56,749: MA=10523=89% MF=10653 A=843 F=13 AF=1 IF=12 D=0 C=13 SF=0 SU=0
2021-11-22 00:42:57,250: MA=10521=89% MF=10648 A=843 F=19 AF=1 IF=19 D=0 C=19 SF=0 SU=0
2021-11-22 00:42:57,751: MA=10519=89% MF=10644 A=843 F=23 AF=1 IF=23 D=0 C=23 SF=0 SU=0
2021-11-22 00:42:58,253: MA=10516=89% MF=10638 A=842 F=29 AF=1 IF=28 D=0 C=29 SF=0 SU=0

and PSI metrics at the same time:

2021-11-22 00:42:20,020: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2 sec, log file: /tmpfs/1, suppress output: False
2021-11-22 00:42:20,020: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-22 00:42:20,024: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 00:42:20,024: ======|=============|=============|
2021-11-22 00:42:20,024:  cpu  |      io     |    memory   |
2021-11-22 00:42:20,024: ----- | ----------- | ----------- |
2021-11-22 00:42:20,025:  some |  some  full |  some  full | interval
2021-11-22 00:42:20,025: ----- | ----- ----- | ----- ----- | --------
2021-11-22 00:42:22,027:   1.3 |   0.3   0.3 |   0.0   0.0 | 2.002
2021-11-22 00:42:24,028:   0.3 |   0.0   0.0 |   0.0   0.0 | 2.001
2021-11-22 00:42:26,031:   0.3 |   0.1   0.1 |   0.0   0.0 | 2.002
2021-11-22 00:42:28,032:   1.1 |  45.5  40.1 |  29.5  29.1 | 2.002
2021-11-22 00:42:30,035:   1.4 |  92.9  79.0 |  39.9  39.4 | 2.003
2021-11-22 00:42:32,038:   0.2 |  99.0  98.0 |   0.0   0.0 | 2.003
2021-11-22 00:42:34,040:   0.7 |  96.0  93.4 |   0.0   0.0 | 2.003
2021-11-22 00:42:36,043:   0.9 |  81.9  75.7 |   0.0   0.0 | 2.003
2021-11-22 00:42:38,044:   0.6 |  62.0  58.0 |   0.0   0.0 | 2.001
2021-11-22 00:42:40,047:   0.5 |  38.3  35.0 |   0.0   0.0 | 2.002
2021-11-22 00:42:42,048:   1.1 |  79.9  69.0 |  27.8  27.4 | 2.002
2021-11-22 00:42:44,051:   0.2 |  94.9  94.2 |   0.0   0.0 | 2.003
2021-11-22 00:42:46,054:   0.3 |  96.0  94.7 |   0.0   0.0 | 2.003
2021-11-22 00:42:48,056:   1.0 |  89.9  84.7 |   0.0   0.0 | 2.003
2021-11-22 00:42:50,058:   1.1 |  84.9  78.4 |   0.0   0.0 | 2.002
2021-11-22 00:42:52,060:   0.6 |  70.0  65.6 |   0.0   0.0 | 2.002
2021-11-22 00:42:54,063:   0.6 |  25.7  23.2 |   0.0   0.0 | 2.002
2021-11-22 00:42:56,065:   1.5 |  43.3  37.1 |  31.2  30.6 | 2.002
2021-11-22 00:42:58,068:   0.2 |  96.5  94.2 |   0.0   0.0 | 2.002
2021-11-22 00:43:00,068:   0.8 |  95.6  92.7 |   0.0   0.0 | 2.001
2021-11-22 00:43:02,071:   1.3 |  90.5  85.8 |   0.0   0.0 | 2.003
2021-11-22 00:43:04,072:   1.2 |  88.2  82.2 |   0.0   0.0 | 2.001
2021-11-22 00:43:06,075:   1.1 |  74.7  70.2 |   0.0   0.0 | 2.003
2021-11-22 00:43:08,076:   1.3 |  75.3  69.6 |   0.0   0.0 | 2.001
2021-11-22 00:43:10,078:   1.5 |  31.5  29.4 |   0.0   0.0 | 2.002
2021-11-22 00:43:12,080:   1.3 |  14.4  13.7 |   0.0   0.0 | 2.002
2021-11-22 00:43:14,083:   1.5 |   2.3   2.2 |   0.0   0.0 | 2.003


7. 5.15.0, swappiness=0, swap on zram
=====================================

Just demo of expected behavior with swappiness=0 (running tail /dev/zero):

2021-11-23 21:20:07,162: Starting mem2log with interval 0.5s, mode: 1
2021-11-23 21:20:07,162: Log file: /tmpfs/1
2021-11-23 21:20:07,163: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-23 21:20:07,164: All values are in mebibytes
2021-11-23 21:20:07,164: MemTotal: 11798.7, SwapTotal: 11798.6
2021-11-23 21:20:07,164: --
2021-11-23 21:20:07,164: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-23 21:20:07,164: F is File, AF is Active(file), IF is Inactive(file)
2021-11-23 21:20:07,164: D is Dirty, C is Clean file (File - Dirty)
2021-11-23 21:20:07,164: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-23 21:20:07,164: --
2021-11-23 21:20:07,164: MA=10315=87% MF=10208 A=923 F=326 AF=171 IF=156 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:07,665: MA=10314=87% MF=10207 A=923 F=326 AF=171 IF=156 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:08,166: MA=10313=87% MF=10206 A=923 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:08,667: MA=9363=79% MF=9256 A=1870 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:09,168: MA=8289=70% MF=8182 A=2944 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:09,669: MA=7191=61% MF=7084 A=4040 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:10,170: MA=6099=52% MF=5991 A=5131 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:10,671: MA=5000=42% MF=4893 A=6228 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:11,172: MA=3908=33% MF=3801 A=7318 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:11,673: MA=2821=24% MF=2713 A=8403 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:12,174: MA=1757=15% MF=1650 A=9476 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:12,674: MA=666=6% MF=559 A=10565 F=326 AF=171 IF=155 D=0 C=326 SF=11799 SU=0
2021-11-23 21:20:13,175: MA=0=0% MF=118 A=11399 F=2 AF=1 IF=1 D=0 C=2 SF=11799 SU=0
2021-11-23 21:20:13,676: MA=0=0% MF=119 A=11399 F=2 AF=0 IF=1 D=0 C=2 SF=11799 SU=0
2021-11-23 21:20:14,177: MA=0=0% MF=119 A=11399 F=3 AF=1 IF=1 D=0 C=3 SF=11799 SU=0
2021-11-23 21:20:14,678: MA=0=0% MF=117 A=11401 F=2 AF=1 IF=2 D=0 C=2 SF=11799 SU=0
2021-11-23 21:20:15,179: MA=0=0% MF=118 A=11402 F=2 AF=1 IF=1 D=0 C=2 SF=11799 SU=0  <----- 2-2.5s stall, OOM
2021-11-23 21:20:15,680: MA=9598=81% MF=9733 A=1757 F=6 AF=5 IF=1 D=0 C=6 SF=11799 SU=0
2021-11-23 21:20:16,182: MA=10424=88% MF=10557 A=946 F=11 AF=10 IF=1 D=0 C=11 SF=11799 SU=0
2021-11-23 21:20:16,683: MA=10423=88% MF=10554 A=946 F=16 AF=15 IF=1 D=0 C=16 SF=11799 SU=0
2021-11-23 21:20:17,184: MA=10425=88% MF=10552 A=946 F=22 AF=21 IF=1 D=0 C=22 SF=11799 SU=0
2021-11-23 21:20:17,686: MA=10425=88% MF=10549 A=946 F=28 AF=26 IF=1 D=0 C=27 SF=11799 SU=0
2021-11-23 21:20:18,187: MA=10424=88% MF=10546 A=946 F=32 AF=30 IF=1 D=0 C=32 SF=11799 SU=0
2021-11-23 21:20:18,688: MA=10422=88% MF=10543 A=946 F=35 AF=34 IF=2 D=0 C=35 SF=11799 SU=0
2021-11-23 21:20:19,189: MA=10421=88% MF=10539 A=946 F=41 AF=38 IF=3 D=0 C=41 SF=11799 SU=0
2021-11-23 21:20:19,690: MA=10418=88% MF=10533 A=946 F=46 AF=43 IF=3 D=0 C=46 SF=11799 SU=0

and PSI metrics at the same time:

2021-11-23 21:20:06,522: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 2.0 sec, log file: /tmpfs/2, suppress output: False
2021-11-23 21:20:06,523: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-23 21:20:06,526: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-23 21:20:06,526: ======|=============|=============|
2021-11-23 21:20:06,526:  cpu  |      io     |    memory   |
2021-11-23 21:20:06,526: ----- | ----------- | ----------- |
2021-11-23 21:20:06,526:  some |  some  full |  some  full | interval
2021-11-23 21:20:06,526: ----- | ----- ----- | ----- ----- | --------
2021-11-23 21:20:08,529:   1.3 |   0.4   0.4 |   0.2   0.2 | 2.002
2021-11-23 21:20:10,530:   0.2 |   0.6   0.6 |   0.0   0.0 | 2.001
2021-11-23 21:20:12,532:   0.2 |   0.0   0.0 |   0.0   0.0 | 2.002
2021-11-23 21:20:14,534:   0.5 |  69.8  64.6 |  72.8  71.8 | 2.002
2021-11-23 21:20:16,537:   3.1 |  95.6  82.2 |  97.6  92.4 | 2.002
2021-11-23 21:20:18,538:   0.3 |  97.6  95.9 |  92.1  90.6 | 2.002
2021-11-23 21:20:20,541:   1.1 |  91.8  87.7 |  80.5  76.7 | 2.003
2021-11-23 21:20:22,544:   1.0 |  79.2  75.7 |  59.7  56.7 | 2.003
2021-11-23 21:20:24,546:   1.2 |  78.1  73.9 |  60.4  56.9 | 2.002
2021-11-23 21:20:26,549:   1.1 |  55.2  51.7 |  40.8  38.3 | 2.003
2021-11-23 21:20:28,550:   1.3 |  63.8  59.3 |  35.7  33.4 | 2.001
2021-11-23 21:20:30,553:   1.3 |  32.2  30.4 |  28.0  26.4 | 2.003
2021-11-23 21:20:32,556:   1.2 |   5.2   5.0 |   0.0   0.0 | 2.003
2021-11-23 21:20:34,558:   1.3 |   3.6   3.5 |   1.4   1.3 | 2.003

There are no significant stalls. No progress leads to OOM.


8. 5.16-rc2, swappiness=0, swap on zram
=======================================

Note that the scripts started with interval=5, and both scripts hang in
this experience in spite mlockall().

Running tail /dev/zero with 5.16-rc2 and swappiness=0:

2021-11-22 21:23:07,259: Starting mem2log with interval 5.0s, mode: 1
2021-11-22 21:23:07,259: Log file: /tmpfs/1
2021-11-22 21:23:07,265: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 21:23:07,265: All values are in mebibytes
2021-11-22 21:23:07,265: MemTotal: 11798.5, SwapTotal: 11798.4
2021-11-22 21:23:07,266: --
2021-11-22 21:23:07,266: MA is MemAvailable, MF is MemFree, A is Anon
2021-11-22 21:23:07,266: F is File, AF is Active(file), IF is Inactive(file)
2021-11-22 21:23:07,266: D is Dirty, C is Clean file (File - Dirty)
2021-11-22 21:23:07,266: SF is SwapFree, SU is SwapUsed (SwapTotal - SwapFree)
2021-11-22 21:23:07,267: --
2021-11-22 21:23:07,267: MA=10205=86% MF=10245 A=1054 F=181 AF=42 IF=139 D=0 C=181 SF=11798 SU=0
2021-11-22 21:23:12,272: MA=10201=86% MF=10241 A=1054 F=181 AF=43 IF=139 D=0 C=181 SF=11798 SU=0
2021-11-22 21:23:17,277: MA=1539=13% MF=1578 A=9704 F=182 AF=43 IF=139 D=0 C=182 SF=11798 SU=0
2021-11-22 21:23:22,282: MA=0=0% MF=120 A=11401 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:23:27,288: MA=0=0% MF=118 A=11405 F=2 AF=0 IF=1 D=0 C=2 SF=11798 SU=0
2021-11-22 21:23:32,294: MA=0=0% MF=118 A=11406 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:23:37,300: MA=0=0% MF=118 A=11407 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:23:42,306: MA=0=0% MF=119 A=11407 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:23:47,311: MA=0=0% MF=118 A=11407 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:23:52,317: MA=0=0% MF=118 A=11407 F=1 AF=1 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:23:57,323: MA=0=0% MF=119 A=11407 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:24:02,329: MA=0=0% MF=117 A=11407 F=1 AF=1 IF=1 D=0 C=1 SF=11798 SU=0
2021-11-22 21:24:07,334: MA=0=0% MF=119 A=11407 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:24:13,807: MA=0=0% MF=118 A=11407 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:24:18,812: MA=0=0% MF=118 A=11407 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:24:46,511: MA=0=0% MF=118 A=11407 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:24:51,513: MA=0=0% MF=118 A=11407 F=1 AF=1 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:24:56,519: MA=0=0% MF=117 A=11408 F=1 AF=0 IF=1 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:01,525: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:06,530: MA=0=0% MF=118 A=11408 F=0 AF=0 IF=0 D=0 C=0 SF=11798 SU=0
2021-11-22 21:25:11,533: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:16,539: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=1 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:21,545: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=1 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:26,550: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=1 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:31,556: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:36,562: MA=0=0% MF=118 A=11408 F=1 AF=1 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:41,565: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0
2021-11-22 21:25:46,571: MA=0=0% MF=118 A=11408 F=1 AF=0 IF=0 D=0 C=1 SF=11798 SU=0 <-- 3.5min mem2log stall before OOM
2021-11-22 21:29:16,359: MA=10298=87% MF=10436 A=1075 F=2 AF=1 IF=1 D=0 C=2 SF=11798 SU=0
2021-11-22 21:29:21,365: MA=10295=87% MF=10411 A=1074 F=46 AF=7 IF=39 D=0 C=46 SF=11798 SU=0
2021-11-22 21:29:26,369: MA=10272=87% MF=10367 A=1075 F=88 AF=11 IF=77 D=0 C=88 SF=11798 SU=0
2021-11-22 21:29:31,373: MA=10254=87% MF=10334 A=1071 F=119 AF=14 IF=105 D=0 C=119 SF=11798 SU=0

and PSI metrics at the same time:

2021-11-22 21:23:07,259: Starting psi2log, target: SYSTEM_WIDE, mode: 2, interval: 5.0 sec, log file: /tmpfs/2, suppress output: False
2021-11-22 21:23:07,259: PSI source dir: /proc/pressure/, source files: cpu, io, memory
2021-11-22 21:23:07,265: Process memory locked with MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT
2021-11-22 21:23:07,265: ======|=============|=============|
2021-11-22 21:23:07,265:  cpu  |      io     |    memory   |
2021-11-22 21:23:07,265: ----- | ----------- | ----------- |
2021-11-22 21:23:07,265:  some |  some  full |  some  full | interval
2021-11-22 21:23:07,265: ----- | ----- ----- | ----- ----- | --------
2021-11-22 21:23:12,269:   1.4 |   1.3   1.3 |   0.0   0.0 | 5.003
2021-11-22 21:23:17,273:   0.4 |   0.6   0.6 |   0.0   0.0 | 5.004
2021-11-22 21:23:22,277:   0.4 |  17.8  17.6 |  67.5  66.8 | 5.004
2021-11-22 21:23:27,281:   0.5 |  28.5  27.4 |  98.2  97.6 | 5.004
2021-11-22 21:23:32,286:   0.5 |  12.1  11.4 |  99.6  99.2 | 5.005
2021-11-22 21:23:37,289:   0.5 |   3.7   3.7 |  99.1  98.9 | 5.003
2021-11-22 21:23:42,295:   0.3 |  21.7  21.4 |  99.8  99.7 | 5.006
2021-11-22 21:23:47,301:   0.8 |  21.8  21.3 |  99.8  99.5 | 5.006
2021-11-22 21:23:52,306:   0.7 |   9.2   8.9 |  99.7  99.4 | 5.006
2021-11-22 21:23:57,312:   0.6 |   5.5   5.4 |  99.1  98.8 | 5.006
2021-11-22 21:24:02,318:   0.2 |   0.8   0.8 |  99.2  99.1 | 5.006
2021-11-22 21:24:07,321:   0.7 |   5.9   5.7 | 100.0  99.7 | 5.003
2021-11-22 21:24:12,327:   0.7 |   7.2   6.6 |  99.6  99.2 | 5.006
2021-11-22 21:24:17,332:   1.0 |  21.5  20.4 |  99.5  99.2 | 5.006
2021-11-22 21:24:22,338:   1.4 |  10.9  10.2 |  99.5  99.0 | 5.006
2021-11-22 21:24:27,344:   1.0 |  13.2  12.9 |  99.8  99.5 | 5.006
2021-11-22 21:24:32,349:   0.6 |  32.5  32.2 |  99.8  99.6 | 5.006
2021-11-22 21:24:37,355:   1.1 |  18.1  17.4 | 100.0  99.7 | 5.006
2021-11-22 21:24:42,361:   1.2 |  16.0  15.2 |  99.6  99.3 | 5.006
2021-11-22 21:24:47,365:   0.8 |   8.0   7.2 |  99.4  99.2 | 5.004
2021-11-22 21:24:52,371:   0.6 |  16.4  16.2 | 100.0  99.8 | 5.006
2021-11-22 21:24:57,376:   0.8 |  12.5  12.3 |  99.6  99.4 | 5.006
2021-11-22 21:25:02,382:   0.4 |   4.3   4.3 |  99.0  98.8 | 5.006
2021-11-22 21:25:07,388:   0.3 |  11.8  11.5 | 100.0  99.9 | 5.006
2021-11-22 21:25:12,393:   0.5 |   0.9   0.9 |  99.8  99.6 | 5.005
2021-11-22 21:25:17,399:   0.5 |  18.3  18.0 |  99.8  99.7 | 5.006
2021-11-22 21:25:22,404:   0.3 |  25.1  23.2 | 100.0  99.9 | 5.006
2021-11-22 21:25:27,410:   0.4 |   1.6   1.3 | 100.0  99.9 | 5.006
2021-11-22 21:25:32,416:   0.6 |   1.2   1.2 |  99.8  99.6 | 5.006
2021-11-22 21:25:37,422:   0.4 |   1.0   1.0 | 100.0  99.9 | 5.006
2021-11-22 21:25:42,427:   0.8 |   1.4   1.4 | 100.0  99.8 | 5.006
2021-11-22 21:25:47,433:   0.3 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:25:52,439:   0.5 |   0.9   0.9 |  99.6  99.4 | 5.006
2021-11-22 21:25:57,444:   0.4 |   0.0   0.0 |  99.8  99.6 | 5.006
2021-11-22 21:26:02,450:   0.4 |   1.0   1.0 | 100.0  99.8 | 5.006
2021-11-22 21:26:07,456:   0.7 |   1.0   1.0 | 100.0  99.8 | 5.006
2021-11-22 21:26:12,461:   0.6 |   0.8   0.8 | 100.0  99.9 | 5.006
2021-11-22 21:26:17,467:   0.4 |   0.1   0.0 |  98.9  98.8 | 5.006
2021-11-22 21:26:22,473:   0.4 |   0.1   0.1 | 100.0  99.9 | 5.006
2021-11-22 21:26:27,478:   0.4 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:26:32,484:   0.4 |   0.1   0.1 | 100.0  99.9 | 5.006
2021-11-22 21:26:37,490:   0.3 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:26:42,496:   0.3 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:26:47,501:   0.3 |   0.0   0.0 | 100.0  99.8 | 5.006
2021-11-22 21:26:52,507:   0.3 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:26:57,513:   0.3 |   0.0   0.0 | 100.0  99.8 | 5.006
2021-11-22 21:27:02,518:   0.3 |   0.0   0.0 | 100.0  99.8 | 5.006
2021-11-22 21:27:07,524:   0.3 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:27:12,530:   0.4 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:27:17,535:   0.4 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:27:22,541:   0.4 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:27:27,547:   0.4 |   0.0   0.0 | 100.0  99.9 | 5.006
2021-11-22 21:29:21,883: WARNING: abnormal interval (114.336 sec), metrics may be provided incorrect
2021-11-22 21:29:21,883:   0.4 |   5.0   4.9 |  96.5  96.3 | 114.336
2021-11-22 21:29:26,885:   1.1 |  88.6  82.5 |   2.1   1.7 | 5.002
2021-11-22 21:29:31,891:   1.3 |  78.8  73.0 |   0.0   0.0 | 5.006
2021-11-22 21:29:36,897:   1.4 |  77.3  71.4 |   0.1   0.1 | 5.006
2021-11-22 21:29:41,901:   1.6 |  39.3  36.1 |   0.0   0.0 | 5.004
2021-11-22 21:29:46,905:   1.4 |   1.6   1.5 |   0.0   0.0 | 5.004

23:22 - 29:16: almost 6 min stall with extremely high memory pressure and
with no progress (no progress for the userspace, of course).


Resume
======

- It's easy to reproduce freezes in near-OOM conditions with 5.16-rc1/rc2,
  and it's a new regression.
- The easiest and most effective way to return the expected behavior is to
  disable the reclaim_throttle() function completely. 
- Setting timeout to 0 in reclaim_throttle() leads to reducing stall
  intervals.
- Setting MAX_RECLAIM_RETRIES to 0 reduces stall intervals even more
  effective then setting timeouts to 0.


Questions
=========

- Should a new knob like vm.reclaim_throttle_factor be added (perhaps a
  single timeout value will not suit everyone)?
- Is it time to new "mm, oom: rework oom detection"?


Special thanks to Oleksandr Natalenko for suspecting [4] the regression and
confirming [5] it. 


[1] https://lore.kernel.org/lkml/d9802b6a-949b-b327-c4a6-3dbca485ec20@gmx.com/
[2] https://github.com/hakavlad/nohang
[3] https://github.com/hakavlad/mem2log
[4] https://www.linux.org.ru/forum/general/16321096?cid=16648804
[5] https://www.linux.org.ru/forum/general/16321096?cid=16650121

             reply	other threads:[~2021-11-23 16:26 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-23 16:19 Alexey Avramov [this message]
2021-11-24  7:40 ` mm: 5.16 regression: reclaim_throttle leads to stall in near-OOM conditions Thorsten Leemhuis
2021-11-28  8:06   ` mm: 5.16 regression: reclaim_throttle leads to stall in near-OOM conditions #justforregzbot Thorsten Leemhuis
2021-11-24 10:35 ` mm: 5.16 regression: reclaim_throttle leads to stall in near-OOM conditions Mel Gorman
2021-11-24 10:54   ` Alexey Avramov
2021-11-24 11:50     ` Mel Gorman
2021-11-24 12:03       ` Alexey Avramov
2021-11-24 12:44       ` Alexey Avramov
2021-11-24 14:33         ` Mel Gorman
2021-11-26 16:06           ` Alexey Avramov
2021-11-26 16:24             ` Mel Gorman
2021-12-20  8:50               ` Sultan Alsawaf
2021-11-27 19:37           ` Alexey Avramov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211124011954.7cab9bb4@mail.inbox.lv \
    --to=hakavlad@inbox.lv \
    --cc=akpm@linux-foundation.org \
    --cc=aros@gmx.com \
    --cc=corbet@lwn.net \
    --cc=david@fromorbit.com \
    --cc=hakavlad@gmail.com \
    --cc=hannes@cmpxchg.org \
    --cc=hdanton@sina.com \
    --cc=kernel@xanmod.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@techsingularity.net \
    --cc=mhocko@suse.com \
    --cc=michael@michaellarabel.com \
    --cc=neilb@suse.de \
    --cc=oleksandr@natalenko.name \
    --cc=penguin-kernel@i-love.sakura.ne.jp \
    --cc=riel@surriel.com \
    --cc=vbabka@suse.cz \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.