From: Alexey Avramov <hakavlad@inbox.lv>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Michal Hocko <mhocko@suse.com>, Vlastimil Babka <vbabka@suse.cz>,
Rik van Riel <riel@surriel.com>, Mike Galbraith <efault@gmx.de>,
Darrick Wong <djwong@kernel.org>,
regressions@lists.linux.dev,
Linux-fsdevel <linux-fsdevel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/1] mm: vmscan: Reduce throttling due to a failure to make progress
Date: Sun, 28 Nov 2021 04:26:35 +0900 [thread overview]
Message-ID: <20211128042635.543a2d04@mail.inbox.lv> (raw)
In-Reply-To: <20211126165211.GL3366@techsingularity.net>
I will present the results of the new tests here.
TLDR;
=====
No one Mel's patch doesn't prevent stalls in my tests.
Test case:
Running firefox with youtube tab (basic desktop workload) and starting
$ for i in {1..10}; do tail /dev/zero; done
-- 1. with noswap and 2. with SwapTotal > MemTotal and swappiness=0.
I will log PSI metrics using latest psi2log (it's part of nohang [1]
package, src: [2]) and log some meminfo metrics using mem2log [3].
psi2log cmd:
$ psi2log -m 2 -i 1 -l /tmpfs/psi
mem2log cmd:
$ mem2log -i 1 -l /tmpfs/mem
The OS is Debian 9 x86_64 on HDD with ext4+LUKS, MemTotal: 11.5 GiB.
Repo with logs:
https://github.com/hakavlad/cache-tests/tree/main/516-reclaim-throttle
5.15
====
- With noswap, psi2log summary:
2021-11-27 20:51:14,018: Stall times for the last 141.6s:
2021-11-27 20:51:14,018: -----------
2021-11-27 20:51:14,018: some cpu 1.3s, avg 0.9%
2021-11-27 20:51:14,018: -----------
2021-11-27 20:51:14,018: some io 99.1s, avg 70.0%
2021-11-27 20:51:14,018: full io 93.4s, avg 66.0%
2021-11-27 20:51:14,018: -----------
2021-11-27 20:51:14,019: some memory 4.3s, avg 3.1%
2021-11-27 20:51:14,019: full memory 4.2s, avg 3.0%
-- average full memory stall is 0.4s per one `tail /dev/zero`.
full psi2log log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/515/noswap/psi
full mem2log log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/515/noswap/mem
- With swappiness=0 and SwapTotal > 0:
2021-11-27 20:57:30,217: Stall times for the last 141.6s:
2021-11-27 20:57:30,217: -----------
2021-11-27 20:57:30,217: some cpu 1.2s, avg 0.8%
2021-11-27 20:57:30,217: -----------
2021-11-27 20:57:30,217: some io 100.7s, avg 71.1%
2021-11-27 20:57:30,218: full io 94.8s, avg 67.0%
2021-11-27 20:57:30,218: -----------
2021-11-27 20:57:30,218: some memory 3.9s, avg 2.8%
2021-11-27 20:57:30,218: full memory 3.9s, avg 2.7%
-- 0.4s memory stall per one `tail /dev/zero`.
full psi2log log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/515/swappiness0/psi
full mem2log log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/515/swappiness0/mem
5.16-rc2 vanilla
================
- with noswap
psi2log summary:
2021-11-27 21:45:09,307: Stall times for the last 1086.5s:
2021-11-27 21:45:09,307: -----------
2021-11-27 21:45:09,307: some cpu 4.2s, avg 0.4%
2021-11-27 21:45:09,307: -----------
2021-11-27 21:45:09,307: some io 337.6s, avg 31.1%
2021-11-27 21:45:09,307: full io 325.2s, avg 29.9%
2021-11-27 21:45:09,307: -----------
2021-11-27 21:45:09,307: some memory 888.7s, avg 81.8%
2021-11-27 21:45:09,307: full memory 886.2s, avg 81.6%
average full memory stall is 87s per one `tail /dev/zero`.
full psi2log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/vanilla/noswap/psi
full mem2log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/vanilla/noswap/mem
- And even worst with swappiness=0:
2021-11-27 22:33:19,207: Stall times for the last 2677.7s:
2021-11-27 22:33:19,208: -----------
2021-11-27 22:33:19,208: some cpu 14.0s, avg 0.5%
2021-11-27 22:33:19,208: -----------
2021-11-27 22:33:19,208: some io 306.7s, avg 11.5%
2021-11-27 22:33:19,208: full io 292.4s, avg 10.9%
2021-11-27 22:33:19,208: -----------
2021-11-27 22:33:19,208: some memory 2504.8s, avg 93.5%
2021-11-27 22:33:19,209: full memory 2498.2s, avg 93.3%
What a horror!
In dmesg: a lot of
[] Purging GPU memory, 0 pages freed, 0 pages still pinned, 2112 pages left available.
(and dying journald)
psi2log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/vanilla/swappiness0/psi
mem2log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/vanilla/swappiness0/mem
dmesg:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/vanilla/dmesg
5.16-rc2 and 1st Mel's patch [4]
================================
- with noswap
Summary:
2021-11-27 23:28:23,707: Stall times for the last 1035.1s:
2021-11-27 23:28:23,707: -----------
2021-11-27 23:28:23,707: some cpu 5.6s, avg 0.5%
2021-11-27 23:28:23,708: -----------
2021-11-27 23:28:23,708: some io 336.5s, avg 32.5%
2021-11-27 23:28:23,708: full io 322.9s, avg 31.2%
2021-11-27 23:28:23,708: -----------
2021-11-27 23:28:23,708: some memory 851.1s, avg 82.2%
2021-11-27 23:28:23,708: full memory 848.3s, avg 82.0%
full psi2log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch1/noswap/psi
mem2log log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch1/noswap/mem
- with swappiness=0
Summary:
2021-11-27 23:50:25,722: Stall times for the last 1076.9s:
2021-11-27 23:50:25,722: -----------
2021-11-27 23:50:25,722: some cpu 12.2s, avg 1.1%
2021-11-27 23:50:25,722: -----------
2021-11-27 23:50:25,723: some io 232.1s, avg 21.6%
2021-11-27 23:50:25,723: full io 218.5s, avg 20.3%
2021-11-27 23:50:25,723: -----------
2021-11-27 23:50:25,723: some memory 957.5s, avg 88.9%
2021-11-27 23:50:25,723: full memory 951.2s, avg 88.3%
It's better than vanilla.
5.16-rc2 and 2nd Mel's patch [5]
================================
- with noswap
Summary:
2021-11-28 00:29:38,558: Stall times for the last 598.7s:
2021-11-28 00:29:38,558: -----------
2021-11-28 00:29:38,558: some cpu 5.2s, avg 0.9%
2021-11-28 00:29:38,558: -----------
2021-11-28 00:29:38,559: some io 235.0s, avg 39.3%
2021-11-28 00:29:38,559: full io 220.2s, avg 36.8%
2021-11-28 00:29:38,559: -----------
2021-11-28 00:29:38,559: some memory 439.0s, avg 73.3%
2021-11-28 00:29:38,559: full memory 437.5s, avg 73.1%
It's significant improvement, but far from the expected result.
mem:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch2/noswap/mem
psi:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch2/noswap/psi
In some cases the stall was quite short.
- with swappiness=0
Summary:
2021-11-28 00:53:41,659: Stall times for the last 1239.0s:
2021-11-28 00:53:41,659: -----------
2021-11-28 00:53:41,660: some cpu 9.4s, avg 0.8%
2021-11-28 00:53:41,660: -----------
2021-11-28 00:53:41,660: some io 206.5s, avg 16.7%
2021-11-28 00:53:41,660: full io 195.0s, avg 15.7%
2021-11-28 00:53:41,660: -----------
2021-11-28 00:53:41,660: some memory 1113.0s, avg 89.8%
2021-11-28 00:53:41,661: full memory 1107.1s, avg 89.3%
mem:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch2/swappiness0/mem
psi:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch2/swappiness0/psi
5.16-rc2 and 3rd Mel's patch [6] on top of 2nd
==============================================
- with noswap
Summary:
2021-11-28 01:35:26,077: Stall times for the last 949.4s:
2021-11-28 01:35:26,077: -----------
2021-11-28 01:35:26,078: some cpu 5.4s, avg 0.6%
2021-11-28 01:35:26,078: -----------
2021-11-28 01:35:26,078: some io 269.0s, avg 28.3%
2021-11-28 01:35:26,078: full io 259.4s, avg 27.3%
2021-11-28 01:35:26,078: -----------
2021-11-28 01:35:26,079: some memory 759.9s, avg 80.0%
2021-11-28 01:35:26,079: full memory 757.5s, avg 79.8%
mem:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch3/noswap/mem
psi:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch3/noswap/psi
- with swappiness=0
Summary:
2021-11-28 01:59:40,253: Stall times for the last 1127.8s:
2021-11-28 01:59:40,253: -----------
2021-11-28 01:59:40,253: some cpu 6.8s, avg 0.6%
2021-11-28 01:59:40,253: -----------
2021-11-28 01:59:40,253: some io 228.3s, avg 20.2%
2021-11-28 01:59:40,253: full io 218.7s, avg 19.4%
2021-11-28 01:59:40,253: -----------
2021-11-28 01:59:40,253: some memory 987.4s, avg 87.6%
2021-11-28 01:59:40,253: full memory 982.3s, avg 87.1%
mem log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch3/swappiness0/mem
psi log:
https://raw.githubusercontent.com/hakavlad/cache-tests/main/516-reclaim-throttle/516-rc2/patch3/swappiness0/psi
¯\_(ツ)_/¯
Should I test anything else?
[1] https://github.com/hakavlad/nohang
[2] https://raw.githubusercontent.com/hakavlad/nohang/master/src/psi2log
[3] https://github.com/hakavlad/mem2log
[4] https://lore.kernel.org/lkml/20211124115007.GG3366@techsingularity.net/
[5] https://lore.kernel.org/lkml/20211124143303.GH3366@techsingularity.net/
[6] https://lore.kernel.org/lkml/20211126162416.GK3366@techsingularity.net/
next prev parent reply other threads:[~2021-11-27 19:35 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-25 15:18 [PATCH 1/1] mm: vmscan: Reduce throttling due to a failure to make progress Mel Gorman
2021-11-26 10:14 ` Vlastimil Babka
2021-11-26 10:26 ` Mike Galbraith
2021-11-26 16:12 ` Alexey Avramov
2021-11-26 16:52 ` Mel Gorman
2021-11-27 19:26 ` Alexey Avramov [this message]
2021-11-28 10:00 ` Mike Galbraith
2021-11-28 11:39 ` Mike Galbraith
2021-11-28 12:35 ` Mike Galbraith
2021-11-28 18:38 ` Mike Galbraith
2021-11-28 11:58 ` Alexey Avramov
2021-11-29 8:26 ` Mel Gorman
2021-11-29 15:01 ` Mel Gorman
2021-11-30 10:14 ` Mike Galbraith
2021-11-30 11:22 ` Mel Gorman
2021-11-30 12:00 ` Mike Galbraith
2021-11-30 12:51 ` Mike Galbraith
2021-11-30 13:09 ` Mel Gorman
2021-12-01 4:32 ` Mike Galbraith
2021-11-30 16:03 ` Alexey Avramov
2021-11-30 17:27 ` Mel Gorman
2021-11-30 17:59 ` Mike Galbraith
2021-12-01 13:01 ` Mel Gorman
2021-12-01 13:52 ` Mike Galbraith
2021-12-01 15:06 ` Mel Gorman
2021-11-30 18:38 ` Alexey Avramov
2021-12-01 14:00 ` Mel Gorman
2021-12-01 17:29 ` Darrick J. Wong
2021-12-02 9:43 ` Mel Gorman
2021-12-02 16:09 ` Darrick J. Wong
2021-12-02 3:11 ` Mike Galbraith
2021-12-02 10:13 ` Mel Gorman
2021-12-02 10:51 ` Mike Galbraith
2021-12-02 11:42 ` Alexey Avramov
2021-12-02 12:14 ` Alexey Avramov
2021-12-02 12:15 ` Mel Gorman
2021-12-02 12:22 ` Alexey Avramov
2021-12-02 13:18 Mel Gorman
2021-12-02 14:51 ` kernel test robot
2021-12-02 14:51 ` kernel test robot
2021-12-02 16:02 ` kernel test robot
2021-12-02 16:02 ` kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211128042635.543a2d04@mail.inbox.lv \
--to=hakavlad@inbox.lv \
--cc=akpm@linux-foundation.org \
--cc=djwong@kernel.org \
--cc=efault@gmx.de \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=regressions@lists.linux.dev \
--cc=riel@surriel.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.