* [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory
@ 2021-10-20 6:32 Krzysztof Kozlowski
2021-10-20 6:59 ` liuxp11
0 siblings, 1 reply; 4+ messages in thread
From: Krzysztof Kozlowski @ 2021-10-20 6:32 UTC (permalink / raw)
To: ltp
Previous fix for an out-of-memory killer killing ioctl_sg01 process
in commit 4d2e3d44fad5 ("lib: memutils: don't pollute
entire system memory to avoid OoM") was not fully effective. While it
covers most of the cases, an ARM64 machine with 128 GB of memory, 64 kB
page size and v5.11 kernel hit it again. Polluting the memory fails
with OoM:
LTP: starting ioctl_sg01
ioctl_sg01 invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
...
Mem-Info:
active_anon:309 inactive_anon:1964781 isolated_anon:0
active_file:94 inactive_file:0 isolated_file:0
unevictable:305 dirty:0 writeback:0
slab_reclaimable:1510 slab_unreclaimable:5012
mapped:115 shmem:339 pagetables:463 bounce:0
free:112043 free_pcp:1 free_cma:3159
Node 0 active_anon:19776kB inactive_anon:125745984kB active_file:6016kB inactive_file:0kB unevictable:19520kB ...
Node 0 DMA free:710656kB min:205120kB low:256384kB high:307648kB reserved_highatomic:0KB active_anon:0kB inactive_anon:3332032kB ...
lowmem_reserve[]: 0 0 7908 7908 7908
Node 0 Normal free:6460096kB min:6463168kB low:8078912kB high:9694656kB reserved_highatomic:0KB active_anon:19776kB inactive_anon:122413952kB ...
lowmem_reserve[]: 0 0 0 0 0
The important part are details of memory on Node 0 in Normal zone:
1. free memory: 6460096 kB
2. min (minimum watermark): 6463168 kB
Parse the /proc/zoneinfo and include the "min" data when counting safety
(free memory which should not be polluted). This way we also include
minimum memory for DMA zones and all nodes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
Reviewed-by: Li Wang <liwang@redhat.com>
Reviewed-by: Liu Xinpeng <liuxp11@chinatelecom.cn>
---
Changes since v1:
1. Add static and rename to count_min_pages().
---
lib/tst_memutils.c | 31 +++++++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/lib/tst_memutils.c b/lib/tst_memutils.c
index af132bcc6c24..97f3b8275a42 100644
--- a/lib/tst_memutils.c
+++ b/lib/tst_memutils.c
@@ -6,22 +6,49 @@
#include <unistd.h>
#include <limits.h>
#include <sys/sysinfo.h>
+#include <stdio.h>
#include <stdlib.h>
#define TST_NO_DEFAULT_MAIN
#include "tst_test.h"
+#include "tst_safe_stdio.h"
#define BLOCKSIZE (16 * 1024 * 1024)
+static unsigned long count_min_pages(void)
+{
+ FILE *fp;
+ int ret;
+ unsigned long total_pages = 0;
+ unsigned long pages;
+ char line[BUFSIZ];
+
+ fp = SAFE_FOPEN("/proc/zoneinfo", "r");
+
+ while (fgets(line, BUFSIZ, fp) != NULL) {
+ ret = sscanf(line, " min %lu", &pages);
+ if (ret == 1)
+ total_pages += pages;
+ }
+
+ SAFE_FCLOSE(fp);
+
+ /* Apply a margin because we cannot get below "min" watermark */
+ total_pages += (total_pages / 10);
+
+ return total_pages;
+}
+
void tst_pollute_memory(size_t maxsize, int fillchar)
{
size_t i, map_count = 0, safety = 0, blocksize = BLOCKSIZE;
+ long pagesize = SAFE_SYSCONF(_SC_PAGESIZE);
void **map_blocks;
struct sysinfo info;
SAFE_SYSINFO(&info);
- safety = MAX(4096 * SAFE_SYSCONF(_SC_PAGESIZE), 128 * 1024 * 1024);
- safety = MAX(safety, (info.freeram / 64));
+ safety = MAX(4096 * pagesize, 128 * 1024 * 1024);
+ safety = MAX(safety, count_min_pages() * pagesize);
safety /= info.mem_unit;
if (info.freeswap > safety)
--
2.30.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory
2021-10-20 6:32 [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory Krzysztof Kozlowski
@ 2021-10-20 6:59 ` liuxp11
2021-10-20 7:07 ` Krzysztof Kozlowski
0 siblings, 1 reply; 4+ messages in thread
From: liuxp11 @ 2021-10-20 6:59 UTC (permalink / raw)
To: Krzysztof Kozlowski, ltp
Hi,
Why not to directly read minimum memory watermark from /proc/sys/vm/min_free_kbytes?
-----
Thanks!
From: Krzysztof Kozlowski
Date: 2021-10-20 14:32
To: ltp
Subject: [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory
Previous fix for an out-of-memory killer killing ioctl_sg01 process
in commit 4d2e3d44fad5 ("lib: memutils: don't pollute
entire system memory to avoid OoM") was not fully effective. While it
covers most of the cases, an ARM64 machine with 128 GB of memory, 64 kB
page size and v5.11 kernel hit it again. Polluting the memory fails
with OoM:
LTP: starting ioctl_sg01
ioctl_sg01 invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
...
Mem-Info:
active_anon:309 inactive_anon:1964781 isolated_anon:0
active_file:94 inactive_file:0 isolated_file:0
unevictable:305 dirty:0 writeback:0
slab_reclaimable:1510 slab_unreclaimable:5012
mapped:115 shmem:339 pagetables:463 bounce:0
free:112043 free_pcp:1 free_cma:3159
Node 0 active_anon:19776kB inactive_anon:125745984kB active_file:6016kB inactive_file:0kB unevictable:19520kB ...
Node 0 DMA free:710656kB min:205120kB low:256384kB high:307648kB reserved_highatomic:0KB active_anon:0kB inactive_anon:3332032kB ...
lowmem_reserve[]: 0 0 7908 7908 7908
Node 0 Normal free:6460096kB min:6463168kB low:8078912kB high:9694656kB reserved_highatomic:0KB active_anon:19776kB inactive_anon:122413952kB ...
lowmem_reserve[]: 0 0 0 0 0
The important part are details of memory on Node 0 in Normal zone:
1. free memory: 6460096 kB
2. min (minimum watermark): 6463168 kB
Parse the /proc/zoneinfo and include the "min" data when counting safety
(free memory which should not be polluted). This way we also include
minimum memory for DMA zones and all nodes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
Reviewed-by: Li Wang <liwang@redhat.com>
Reviewed-by: Liu Xinpeng <liuxp11@chinatelecom.cn>
---
Changes since v1:
1. Add static and rename to count_min_pages().
---
lib/tst_memutils.c | 31 +++++++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/lib/tst_memutils.c b/lib/tst_memutils.c
index af132bcc6c24..97f3b8275a42 100644
--- a/lib/tst_memutils.c
+++ b/lib/tst_memutils.c
@@ -6,22 +6,49 @@
#include <unistd.h>
#include <limits.h>
#include <sys/sysinfo.h>
+#include <stdio.h>
#include <stdlib.h>
#define TST_NO_DEFAULT_MAIN
#include "tst_test.h"
+#include "tst_safe_stdio.h"
#define BLOCKSIZE (16 * 1024 * 1024)
+static unsigned long count_min_pages(void)
+{
+ FILE *fp;
+ int ret;
+ unsigned long total_pages = 0;
+ unsigned long pages;
+ char line[BUFSIZ];
+
+ fp = SAFE_FOPEN("/proc/zoneinfo", "r");
+
+ while (fgets(line, BUFSIZ, fp) != NULL) {
+ ret = sscanf(line, " min %lu", &pages);
+ if (ret == 1)
+ total_pages += pages;
+ }
+
+ SAFE_FCLOSE(fp);
+
+ /* Apply a margin because we cannot get below "min" watermark */
+ total_pages += (total_pages / 10);
+
+ return total_pages;
+}
+
void tst_pollute_memory(size_t maxsize, int fillchar)
{
size_t i, map_count = 0, safety = 0, blocksize = BLOCKSIZE;
+ long pagesize = SAFE_SYSCONF(_SC_PAGESIZE);
void **map_blocks;
struct sysinfo info;
SAFE_SYSINFO(&info);
- safety = MAX(4096 * SAFE_SYSCONF(_SC_PAGESIZE), 128 * 1024 * 1024);
- safety = MAX(safety, (info.freeram / 64));
+ safety = MAX(4096 * pagesize, 128 * 1024 * 1024);
+ safety = MAX(safety, count_min_pages() * pagesize);
safety /= info.mem_unit;
if (info.freeswap > safety)
--
2.30.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory
2021-10-20 6:59 ` liuxp11
@ 2021-10-20 7:07 ` Krzysztof Kozlowski
2021-10-20 7:19 ` Krzysztof Kozlowski
0 siblings, 1 reply; 4+ messages in thread
From: Krzysztof Kozlowski @ 2021-10-20 7:07 UTC (permalink / raw)
To: liuxp11, ltp
On 20/10/2021 08:59, liuxp11@chinatelecom.cn wrote:
> Hi,
> Why not to directly read minimum memory watermark from /proc/sys/vm/min_free_kbytes?
>
Because it has different purpose, different values and won't achieve the
same result as we want to achieve here (pollute maximum allowed memory)?
P.S. Could you not top-post? Also you broke quoted email to some
huge-multi-spaced-something:
https://lists.linux.it/pipermail/ltp/2021-October/025524.html
Best regards,
Krzysztof
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory
2021-10-20 7:07 ` Krzysztof Kozlowski
@ 2021-10-20 7:19 ` Krzysztof Kozlowski
0 siblings, 0 replies; 4+ messages in thread
From: Krzysztof Kozlowski @ 2021-10-20 7:19 UTC (permalink / raw)
To: liuxp11, ltp
On 20/10/2021 09:07, Krzysztof Kozlowski wrote:
> On 20/10/2021 08:59, liuxp11@chinatelecom.cn wrote:
>> Hi,
>> Why not to directly read minimum memory watermark from /proc/sys/vm/min_free_kbytes?
>>
>
> Because it has different purpose, different values and won't achieve the
> same result as we want to achieve here (pollute maximum allowed memory)?
Hm, it actually looks it's the aggregate of all low watermark, so I
misinterpreted the documentation. It indeed solves the problem in easier
way, thanks!
Best regards,
Krzysztof
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-10-20 7:20 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-20 6:32 [LTP] [PATCH v2] lib: memutils: respect minimum memory watermark when polluting memory Krzysztof Kozlowski
2021-10-20 6:59 ` liuxp11
2021-10-20 7:07 ` Krzysztof Kozlowski
2021-10-20 7:19 ` Krzysztof Kozlowski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).