* [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels
@ 2021-07-13 9:22 Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
` (4 more replies)
0 siblings, 5 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Hi,
Changes since v3:
1. Patch 4/5: Include per-CPU memory as well in group limits.
Changes since v2:
1. Patch 1/5: Define MEM_USAGE_RANGE and explain it.
2. Patch 3/5: Use PAGESIZES to explain it is the batch accounting.
3. Patch 4/5: Add to this patchset (previously sent separately, no
dependencies on others).
4. Patch 5/5: New patch, fixes additional difference, usually of 2-3
pages coming from the kernel.
See also reply from Michal Hocko:
https://lore.kernel.org/linux-mm/85b8a4f9-b9e9-a6ca-5d0c-c1ecb8c11ef3@canonical.com/T/#m6459b3be3a86f5eaf2cfc48dd586b6faf949e440
v2: https://lists.linux.it/pipermail/ltp/2021-June/023259.html
Best regards,
Krzysztof
Krzysztof Kozlowski (5):
controllers/memcg: accept range of max_usage_in_bytes
controllers/memcg: accept range of usage_in_bytes
controllers/memcg: accept non-zero max_usage_in_bytes after reset
controllers/memcg: increase memory limit in subgroup charge
controllers/memcg: offset kernel memory
.../controllers/memcg/functional/memcg_lib.sh | 41 ++++++++++++++++---
.../memcg_max_usage_in_bytes_test.sh | 18 +++++++-
.../memcg/functional/memcg_stat_rss.sh | 20 ++++-----
.../memcg/functional/memcg_stat_test.sh | 8 ++--
.../memcg/functional/memcg_subgroup_charge.sh | 33 +++++++++++----
.../functional/memcg_usage_in_bytes_test.sh | 7 +++-
6 files changed, 96 insertions(+), 31 deletions(-)
--
2.27.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes Krzysztof Kozlowski
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report higher max_usage_in_bytes than
expected size of 1024 pages. For example v5.4, v5.8, v5.10
and 5.13.0-rc5:
memcg_max_usage_in_bytes_test 1 TINFO: Test memory.max_usage_in_bytes
memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 1393215
memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1393215
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 20 +++++++++++++++++--
.../memcg_max_usage_in_bytes_test.sh | 3 ++-
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index d9bb6d94324d..4c47e63745ff 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -27,6 +27,15 @@ fi
# Post 4.16 kernel updates stat in batch (> 32 pages) every time
PAGESIZES=$(($PAGESIZE * 33))
+# On recent Linux kernels (at least v5.4) updating stats happens in batches
+# (PAGESIZES) and also might depend on workload and number of CPUs. The kernel
+# caches the data and does not prioritize stats precision. This is especially
+# visible for max_usage_in_bytes where it usually exceeds
+# actual memory allocation.
+# When checking for usage_in_bytes and max_usage_in_bytes accept also higher values
+# from given range:
+MEM_USAGE_RANGE=$((PAGESIZES))
+
HUGEPAGESIZE=$(awk '/Hugepagesize/ {print $2}' /proc/meminfo)
[ -z $HUGEPAGESIZE ] && HUGEPAGESIZE=0
HUGEPAGESIZE=$(($HUGEPAGESIZE * 1024))
@@ -140,7 +149,8 @@ shmmax_cleanup()
# Check size in memcg
# $1 - Item name
-# $2 - Expected size
+# $2 - Expected size lower bound
+# $3 - Expected size upper bound (optional)
check_mem_stat()
{
local item_size
@@ -151,7 +161,13 @@ check_mem_stat()
item_size=$(grep -w $1 memory.stat | cut -d " " -f 2)
fi
- if [ "$2" = "$item_size" ]; then
+ if [ "$3" ]; then
+ if [ $item_size -ge $2 ] && [ $item_size -le $3 ]; then
+ tst_res TPASS "$1 is ${2}-${3} as expected"
+ else
+ tst_res TFAIL "$1 is $item_size, ${2}-${3} expected"
+ fi
+ elif [ "$2" = "$item_size" ]; then
tst_res TPASS "$1 is $2 as expected"
else
tst_res TFAIL "$1 is $item_size, $2 expected"
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 14daa4651798..8831f1937070 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -12,6 +12,7 @@ TST_CNT=4
. memcg_lib.sh
MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
MEM_LIMIT=$((MEM_TO_ALLOC * 2))
# Run test cases which checks memory.[memsw.]max_usage_in_bytes after make
@@ -32,7 +33,7 @@ test_max_usage_in_bytes()
signal_memcg_process $MEM_TO_ALLOC
signal_memcg_process $MEM_TO_ALLOC
- check_mem_stat $item $MEM_TO_ALLOC
+ check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset Krzysztof Kozlowski
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report higher usage_in_bytes than
expected size of 1024 pages. For example v5.4, v5.8, v5.10
and 5.13.0-rc5:
memcg_usage_in_bytes_test 1 TINFO: Test memory.usage_in_bytes
memcg_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_usage_in_bytes_test 1 TINFO: Warming up pid: 1160
memcg_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1160
memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 11 +++++++---
| 20 +++++++++----------
.../memcg/functional/memcg_stat_test.sh | 8 ++++----
.../functional/memcg_usage_in_bytes_test.sh | 7 +++++--
4 files changed, 27 insertions(+), 19 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index 4c47e63745ff..9634eb7e9f42 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -248,8 +248,9 @@ test_mem_stat()
local size=$2
local total_size=$3
local stat_name=$4
- local exp_stat_size=$5
- local check_after_free=$6
+ local exp_stat_size_low=$5
+ local exp_stat_size_up=$6
+ local check_after_free=$7
start_memcg_process $memtypes -s $size
@@ -260,7 +261,11 @@ test_mem_stat()
echo $MEMCG_PROCESS_PID > tasks
signal_memcg_process $size
- check_mem_stat $stat_name $exp_stat_size
+ if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
+ check_mem_stat $stat_name $exp_stat_size_low
+ else
+ check_mem_stat $stat_name $exp_stat_size_low $exp_stat_size_up
+ fi
signal_memcg_process $size
if $check_after_free; then
--git a/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh b/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
index 1a6128a6dba8..d9b4ec287b5f 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
@@ -18,54 +18,54 @@ TST_CNT=10
# Test the management and counting of memory
test1()
{
- test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+ test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
}
test2()
{
- test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 false
+ test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 false
}
test3()
{
- test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 false
+ test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 0 false
}
test4()
{
test_mem_stat "--mmap-anon --mmap-file --shm" \
- $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES false
+ $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES false
}
test5()
{
- test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+ test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
}
test6()
{
- test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+ test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
}
test7()
{
- test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 true
+ test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 true
}
test8()
{
- test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 true
+ test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 0 true
}
test9()
{
test_mem_stat "--mmap-anon --mmap-file --shm" \
- $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES true
+ $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES true
}
test10()
{
- test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+ test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
}
tst_run
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
index 925c4ecf87bc..e5eb7e5d0001 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
@@ -14,28 +14,28 @@ TST_CNT=8
test1()
{
tst_res TINFO "Test cache"
- test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES false
+ test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES $PAGESIZES false
}
test2()
{
tst_res TINFO "Test mapped_file"
test_mem_stat "--mmap-file" $PAGESIZES $PAGESIZES \
- "mapped_file" $PAGESIZES false
+ "mapped_file" $PAGESIZES $PAGESIZES false
}
test3()
{
tst_res TINFO "Test unevictable with MAP_LOCKED"
test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES \
- "unevictable" $PAGESIZES false
+ "unevictable" $PAGESIZES $PAGESIZES false
}
test4()
{
tst_res TINFO "Test unevictable with mlock"
test_mem_stat "--mmap-lock2" $PAGESIZES $PAGESIZES \
- "unevictable" $PAGESIZES false
+ "unevictable" $PAGESIZES $PAGESIZES false
}
test5()
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
index e77d6bf2ef23..9140fd9d1fd7 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
@@ -12,13 +12,15 @@ TST_CNT=2
. memcg_lib.sh
MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
MEM_LIMIT=$((MEM_TO_ALLOC * 2))
test1()
{
tst_res TINFO "Test memory.usage_in_bytes"
test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
- "memory.usage_in_bytes" $MEM_TO_ALLOC false
+ "memory.usage_in_bytes" $MEM_TO_ALLOC \
+ $MEM_EXPECTED_UPPER false
}
test2()
@@ -29,7 +31,8 @@ test2()
echo $MEM_LIMIT > memory.limit_in_bytes
echo $MEM_LIMIT > memory.memsw.limit_in_bytes
test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
- "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC false
+ "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \
+ $MEM_EXPECTED_UPPER false
}
tst_run
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory Krzysztof Kozlowski
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report a non-zero max_usage_in_bytes after
resetting the counter. For example v5.4, v5.8, v5.10, v5.11, v5.12 and
5.13.0-rc5:
memcg_max_usage_in_bytes_test 4 TINFO: Test reset memory.memsw.max_usage_in_bytes
memcg_max_usage_in_bytes_test 4 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 4 TINFO: Warming up pid: 1416
memcg_max_usage_in_bytes_test 4 TINFO: Process is still here after warm up: 1416
memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected
memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 122880, 0 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../memcg/functional/memcg_max_usage_in_bytes_test.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 8831f1937070..2d494ac3a78f 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -37,7 +37,7 @@ test_max_usage_in_bytes()
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
- check_mem_stat $item 0
+ check_mem_stat $item 0 $PAGESIZES
fi
stop_memcg_process
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
` (2 preceding siblings ...)
2021-07-13 9:22 ` [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:40 ` Richard Palethorpe
2021-07-13 9:22 ` [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory Krzysztof Kozlowski
4 siblings, 1 reply; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
with:
memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
memcg_subgroup_charge 1 TPASS: rss is 0 as expected
In dmesg one could see that OOM killer killed the process even though
group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
...
memory: usage 132kB, limit 132kB, failcnt 9
memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
...
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 19289] 0 19289 669 389 40960 0 0 memcg_process
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
It seems actual kernel memory usage by a given group depends on number
of CPUs, where at least one page is allocated per-cpu beside regular
(expected) allocation. Fix the test on machines with more CPUs by
including this per-CPU memory in group limits, plus some slack of 4
PAGES. Increase also memory allocation from 32 to 64 pages to be more
distinctive from kernel per-CPU memory.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
1 file changed, 25 insertions(+), 8 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
index 9b23177a4dc5..0d2b235aff7c 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
@@ -14,16 +14,33 @@ TST_CNT=3
. memcg_lib.sh
+# Allocate memory bigger than per-cpu kernel memory
+MEM_TO_ALLOC=$((PAGESIZES * 2))
+
# Test the memory charge won't move to subgroup
# $1 - memory.limit_in_bytes in parent group
# $2 - memory.limit_in_bytes in sub group
test_subgroup()
{
+ local limit_parent=$1
+ local limit_subgroup=$2
+ local total_cpus=`tst_ncpus`
+
+ # Kernel memory allocated for the process is also charged.
+ # It might depend on the number of CPUs. For example on kernel v5.11
+ # additionally total_cpus plus 1-2 pages are charged to the group.
+ if [ $limit_parent -ne 0 ]; then
+ limit_parent=$((limit_parent + 4 * PAGESIZE + total_cpus * PAGESIZE))
+ fi
+ if [ $limit_subgroup -ne 0 ]; then
+ limit_subgroup=$((limit_subgroup + 4 * PAGESIZE + total_cpus * PAGESIZE))
+ fi
+
mkdir subgroup
- echo $1 > memory.limit_in_bytes
- echo $2 > subgroup/memory.limit_in_bytes
+ echo $limit_parent > memory.limit_in_bytes
+ echo $limit_subgroup > subgroup/memory.limit_in_bytes
- start_memcg_process --mmap-anon -s $PAGESIZES
+ start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
warmup
if [ $? -ne 0 ]; then
@@ -31,8 +48,8 @@ test_subgroup()
fi
echo $MEMCG_PROCESS_PID > tasks
- signal_memcg_process $PAGESIZES
- check_mem_stat "rss" $PAGESIZES
+ signal_memcg_process $MEM_TO_ALLOC
+ check_mem_stat "rss" $MEM_TO_ALLOC
cd subgroup
echo $MEMCG_PROCESS_PID > tasks
@@ -47,17 +64,17 @@ test_subgroup()
test1()
{
tst_res TINFO "Test that group and subgroup have no relationship"
- test_subgroup $PAGESIZES $((2 * PAGESIZES))
+ test_subgroup $MEM_TO_ALLOC $((2 * MEM_TO_ALLOC))
}
test2()
{
- test_subgroup $PAGESIZES $PAGESIZES
+ test_subgroup $MEM_TO_ALLOC $MEM_TO_ALLOC
}
test3()
{
- test_subgroup $PAGESIZES 0
+ test_subgroup $MEM_TO_ALLOC 0
}
tst_run
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
` (3 preceding siblings ...)
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
The memory allocated by kernel with __GFP_ACCOUNT is counted as well in
memory group usage limits. Add it to fix failures like:
memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 925811
memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 925811
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4333568, 4194304-4325376 expected
(where memory.kmem.max_usage_in_bytes:8192)
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 10 ++++++++++
.../functional/memcg_max_usage_in_bytes_test.sh | 15 ++++++++++++++-
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index 9634eb7e9f42..dad66c798e19 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -251,6 +251,7 @@ test_mem_stat()
local exp_stat_size_low=$5
local exp_stat_size_up=$6
local check_after_free=$7
+ local kmem_stat_name="${stat_name##*.}"
start_memcg_process $memtypes -s $size
@@ -261,6 +262,15 @@ test_mem_stat()
echo $MEMCG_PROCESS_PID > tasks
signal_memcg_process $size
+ if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+ [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+ local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+ if [ $? -eq 0 ]; then
+ exp_stat_size_low=$((exp_stat_size_low + kmem))
+ exp_stat_size_up=$((exp_stat_size_up + kmem))
+ fi
+ fi
+
if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
check_mem_stat $stat_name $exp_stat_size_low
else
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 2d494ac3a78f..24e9d115c19e 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -22,6 +22,10 @@ test_max_usage_in_bytes()
local item="memory.max_usage_in_bytes"
[ $1 -eq 1 ] && item="memory.memsw.max_usage_in_bytes"
local check_after_reset=$2
+ local exp_stat_size_low=$MEM_TO_ALLOC
+ local exp_stat_size_up=$MEM_EXPECTED_UPPER
+ local kmem_stat_name="${item##*.}"
+
start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
warmup
@@ -33,7 +37,16 @@ test_max_usage_in_bytes()
signal_memcg_process $MEM_TO_ALLOC
signal_memcg_process $MEM_TO_ALLOC
- check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
+ if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+ [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+ local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+ if [ $? -eq 0 ]; then
+ exp_stat_size_low=$((exp_stat_size_low + kmem))
+ exp_stat_size_up=$((exp_stat_size_up + kmem))
+ fi
+ fi
+
+ check_mem_stat $item $exp_stat_size_low $exp_stat_size_up
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
@ 2021-07-13 9:40 ` Richard Palethorpe
2021-07-13 9:48 ` Krzysztof Kozlowski
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2021-07-13 9:40 UTC (permalink / raw)
To: ltp
Hello Krzysztof,
Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
> with:
>
> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>
> In dmesg one could see that OOM killer killed the process even though
> group memory limit was matching the usage:
>
> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
> ...
> memory: usage 132kB, limit 132kB, failcnt 9
> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
> ...
> Tasks state (memory values in pages):
> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>
> It seems actual kernel memory usage by a given group depends on number
> of CPUs, where at least one page is allocated per-cpu beside regular
> (expected) allocation. Fix the test on machines with more CPUs by
> including this per-CPU memory in group limits, plus some slack of 4
> PAGES. Increase also memory allocation from 32 to 64 pages to be more
> distinctive from kernel per-CPU memory.
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>
> Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
> ---
> .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
> 1 file changed, 25 insertions(+), 8 deletions(-)
>
> diff --git a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> index 9b23177a4dc5..0d2b235aff7c 100755
> --- a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> +++ b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> @@ -14,16 +14,33 @@ TST_CNT=3
>
> . memcg_lib.sh
>
> +# Allocate memory bigger than per-cpu kernel memory
> +MEM_TO_ALLOC=$((PAGESIZES * 2))
> +
--
Thank you,
Richard.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:40 ` Richard Palethorpe
@ 2021-07-13 9:48 ` Krzysztof Kozlowski
2021-07-13 14:17 ` Richard Palethorpe
0 siblings, 1 reply; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:48 UTC (permalink / raw)
To: ltp
On 13/07/2021 11:40, Richard Palethorpe wrote:
> Hello Krzysztof,
>
> Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
>
>> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
>> with:
>>
>> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
>> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
>> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
>> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
>> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>>
>> In dmesg one could see that OOM killer killed the process even though
>> group memory limit was matching the usage:
>>
>> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
>> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
>> ...
>> memory: usage 132kB, limit 132kB, failcnt 9
>> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
>> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
>> ...
>> Tasks state (memory values in pages):
>> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
>> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
>> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
>> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
>> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>>
>> It seems actual kernel memory usage by a given group depends on number
>> of CPUs, where at least one page is allocated per-cpu beside regular
>> (expected) allocation. Fix the test on machines with more CPUs by
>> including this per-CPU memory in group limits, plus some slack of 4
>> PAGES. Increase also memory allocation from 32 to 64 pages to be more
>> distinctive from kernel per-CPU memory.
>
> Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>
Yes, right. Maybe this could be fixed when applying - no need for resend.
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:48 ` Krzysztof Kozlowski
@ 2021-07-13 14:17 ` Richard Palethorpe
2021-07-14 6:05 ` Li Wang
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2021-07-13 14:17 UTC (permalink / raw)
To: ltp
Hello,
Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
> On 13/07/2021 11:40, Richard Palethorpe wrote:
>> Hello Krzysztof,
>>
>> Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
>>
>>> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
>>> with:
>>>
>>> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
>>> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
>>> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
>>> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
>>> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>>>
>>> In dmesg one could see that OOM killer killed the process even though
>>> group memory limit was matching the usage:
>>>
>>> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
>>> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
>>> ...
>>> memory: usage 132kB, limit 132kB, failcnt 9
>>> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
>>> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
>>> ...
>>> Tasks state (memory values in pages):
>>> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
>>> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
>>> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
>>> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
>>> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>>>
>>> It seems actual kernel memory usage by a given group depends on number
>>> of CPUs, where at least one page is allocated per-cpu beside regular
>>> (expected) allocation. Fix the test on machines with more CPUs by
>>> including this per-CPU memory in group limits, plus some slack of 4
>>> PAGES. Increase also memory allocation from 32 to 64 pages to be more
>>> distinctive from kernel per-CPU memory.
>>
>> Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>>
>
> Yes, right. Maybe this could be fixed when applying - no need for resend.
>
>
> Best regards,
> Krzysztof
Well I am more than happy with the patchset.
Acked-by: Richard Palethorpe <rpalethorpe@suse.com>
--
Thank you,
Richard.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 14:17 ` Richard Palethorpe
@ 2021-07-14 6:05 ` Li Wang
0 siblings, 0 replies; 10+ messages in thread
From: Li Wang @ 2021-07-14 6:05 UTC (permalink / raw)
To: ltp
Hi Krzysztof, Richard,
Richard Palethorpe <rpalethorpe@suse.de> wrote:
>
> >> Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
> >>
> >
> > Yes, right. Maybe this could be fixed when applying - no need for resend.
>
I help correct that and applied patchset, thanks for all your work on this.
--
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210714/3f229093/attachment.htm>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-07-14 6:05 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
2021-07-13 9:40 ` Richard Palethorpe
2021-07-13 9:48 ` Krzysztof Kozlowski
2021-07-13 14:17 ` Richard Palethorpe
2021-07-14 6:05 ` Li Wang
2021-07-13 9:22 ` [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory Krzysztof Kozlowski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.