All of lore.kernel.org
 help / color / mirror / Atom feed
From: Roman Gushchin <roman.gushchin@linux.dev>
To: "Michal Koutný" <mkoutny@suse.com>
Cc: void@manifault.com, akpm@linux-foundation.org,
	cgroups@vger.kernel.org, hannes@cmpxchg.org, kernel-team@fb.com,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	mhocko@kernel.org, shakeelb@google.com, tj@kernel.org,
	Richard Palethorpe <rpalethorpe@suse.de>
Subject: Re: [PATCH 3/4] selftests: memcg: Adjust expected reclaim values of protected cgroups
Date: Fri, 13 May 2022 11:52:52 -0700	[thread overview]
Message-ID: <Yn6pBPq+lAXm9NG8@carbon> (raw)
In-Reply-To: <20220513171811.730-4-mkoutny@suse.com>

On Fri, May 13, 2022 at 07:18:10PM +0200, Michal Koutny wrote:
> The numbers are not easy to derive in a closed form (certainly mere
> protections ratios do not apply), therefore use a simulation to obtain
> expected numbers.
> 
> The new values make the protection tests succeed more precisely.
> 
> 	% run as: octave-cli script
> 	%
> 	% Input configurations
> 	% -------------------
> 	% E parent effective protection
> 	% n nominal protection of siblings set at the givel level
> 	% c current consumption -,,-
> 
> 	% example from testcase (values in GB)
> 	E = 50 / 1024;
> 	n = [75 25 0 500 ] / 1024;
> 	c = [50 50 50 0] / 1024;
> 
> 	% Reclaim parameters
> 	% ------------------
> 
> 	% Minimal reclaim amount (GB)
> 	cluster = 32*4 / 2**20;
> 
> 	% Reclaim coefficient (think as 0.5^sc->priority)
> 	alpha = .1
> 
> 	% Simulation parameters
> 	% ---------------------
> 	epsilon = 1e-7;
> 	timeout = 1000;
> 
> 	% Simulation loop
> 	% ---------------------
> 	% Simulation assumes siblings consumed the initial amount of memory (w/out
> 	% reclaim) and then the reclaim starts, all memory is reclaimable, i.e. treated
> 	% same. It simulates only non-low reclaim and assumes all memory.min = 0.
> 
> 	ch = [];
> 	eh = [];
> 	rh = [];
> 
> 	for t = 1:timeout
> 		% low_usage
> 		u = min(c, n);
> 		siblings = sum(u);
> 
> 		% effective_protection()
> 		protected = min(n, c);                % start with nominal
> 		e = protected * min(1, E / siblings); % normalize overcommit
> 
> 		% recursive protection
> 		unclaimed = max(0, E - siblings);
> 		parent_overuse = sum(c) - siblings;
> 		if (unclaimed > 0 && parent_overuse > 0)
> 			overuse = max(0, c - protected);
> 			e += unclaimed * (overuse / parent_overuse);
> 		endif
> 
> 		% get_scan_count()
> 		r = alpha * c;             % assume all memory is in a single LRU list
> 
> 		% commit 1bc63fb1272b ("mm, memcg: make scan aggression always exclude protection")
> 		sz = max(e, c);
> 		r .*= (1 - (e+epsilon) ./ (sz+epsilon));
> 
> 		% uncomment to debug prints
> 		% e, c, r
> 
> 		% nothing to reclaim, reached equilibrium
> 		if max(r) < epsilon
> 			break;
> 		endif
> 
> 		% SWAP_CLUSTER_MAX
> 		r = max(r, (r > epsilon) .* cluster);
> 		% XXX here I do parallel reclaim of all siblings
> 		% in reality reclaim is serialized and each sibling recalculates own residual
> 		c = max(c - r, 0);
> 
> 		ch = [ch ; c];
> 		eh = [eh ; e];
> 		rh = [rh ; r];
> 	endfor
> 
> 	t
> 	c, e

This is a cool stuff!

How about to place it into a separate file and add a comment into the code
with a reference?

Thanks!

WARNING: multiple messages have this Message-ID (diff)
From: Roman Gushchin <roman.gushchin-fxUVXftIFDnyG1zEObXtfA@public.gmane.org>
To: "Michal Koutný" <mkoutny-IBi9RG/b67k@public.gmane.org>
Cc: void-gq6j2QGBifHby3iVrkZq2A@public.gmane.org,
	akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org,
	cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org,
	kernel-team-b10kYP2dOMg@public.gmane.org,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org,
	mhocko-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
	shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org,
	tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
	Richard Palethorpe <rpalethorpe-l3A5Bk7waGM@public.gmane.org>
Subject: Re: [PATCH 3/4] selftests: memcg: Adjust expected reclaim values of protected cgroups
Date: Fri, 13 May 2022 11:52:52 -0700	[thread overview]
Message-ID: <Yn6pBPq+lAXm9NG8@carbon> (raw)
In-Reply-To: <20220513171811.730-4-mkoutny-IBi9RG/b67k@public.gmane.org>

On Fri, May 13, 2022 at 07:18:10PM +0200, Michal Koutny wrote:
> The numbers are not easy to derive in a closed form (certainly mere
> protections ratios do not apply), therefore use a simulation to obtain
> expected numbers.
> 
> The new values make the protection tests succeed more precisely.
> 
> 	% run as: octave-cli script
> 	%
> 	% Input configurations
> 	% -------------------
> 	% E parent effective protection
> 	% n nominal protection of siblings set at the givel level
> 	% c current consumption -,,-
> 
> 	% example from testcase (values in GB)
> 	E = 50 / 1024;
> 	n = [75 25 0 500 ] / 1024;
> 	c = [50 50 50 0] / 1024;
> 
> 	% Reclaim parameters
> 	% ------------------
> 
> 	% Minimal reclaim amount (GB)
> 	cluster = 32*4 / 2**20;
> 
> 	% Reclaim coefficient (think as 0.5^sc->priority)
> 	alpha = .1
> 
> 	% Simulation parameters
> 	% ---------------------
> 	epsilon = 1e-7;
> 	timeout = 1000;
> 
> 	% Simulation loop
> 	% ---------------------
> 	% Simulation assumes siblings consumed the initial amount of memory (w/out
> 	% reclaim) and then the reclaim starts, all memory is reclaimable, i.e. treated
> 	% same. It simulates only non-low reclaim and assumes all memory.min = 0.
> 
> 	ch = [];
> 	eh = [];
> 	rh = [];
> 
> 	for t = 1:timeout
> 		% low_usage
> 		u = min(c, n);
> 		siblings = sum(u);
> 
> 		% effective_protection()
> 		protected = min(n, c);                % start with nominal
> 		e = protected * min(1, E / siblings); % normalize overcommit
> 
> 		% recursive protection
> 		unclaimed = max(0, E - siblings);
> 		parent_overuse = sum(c) - siblings;
> 		if (unclaimed > 0 && parent_overuse > 0)
> 			overuse = max(0, c - protected);
> 			e += unclaimed * (overuse / parent_overuse);
> 		endif
> 
> 		% get_scan_count()
> 		r = alpha * c;             % assume all memory is in a single LRU list
> 
> 		% commit 1bc63fb1272b ("mm, memcg: make scan aggression always exclude protection")
> 		sz = max(e, c);
> 		r .*= (1 - (e+epsilon) ./ (sz+epsilon));
> 
> 		% uncomment to debug prints
> 		% e, c, r
> 
> 		% nothing to reclaim, reached equilibrium
> 		if max(r) < epsilon
> 			break;
> 		endif
> 
> 		% SWAP_CLUSTER_MAX
> 		r = max(r, (r > epsilon) .* cluster);
> 		% XXX here I do parallel reclaim of all siblings
> 		% in reality reclaim is serialized and each sibling recalculates own residual
> 		c = max(c - r, 0);
> 
> 		ch = [ch ; c];
> 		eh = [eh ; e];
> 		rh = [rh ; r];
> 	endfor
> 
> 	t
> 	c, e

This is a cool stuff!

How about to place it into a separate file and add a comment into the code
with a reference?

Thanks!

  reply	other threads:[~2022-05-13 18:53 UTC|newest]

Thread overview: 72+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-23 15:56 [PATCH v2 0/5] Fix bugs in memcontroller cgroup tests David Vernet
2022-04-23 15:56 ` David Vernet
2022-04-23 15:56 ` [PATCH v2 1/5] cgroups: Refactor children cgroups in memcg tests David Vernet
2022-04-26  1:56   ` Roman Gushchin
2022-04-26  1:56     ` Roman Gushchin
2022-04-23 15:56 ` [PATCH v2 2/5] cgroup: Account for memory_recursiveprot in test_memcg_low() David Vernet
2022-04-23 15:56   ` David Vernet
2022-04-27 14:09   ` Michal Koutný
2022-04-27 14:09     ` Michal Koutný
2022-04-29  1:03     ` David Vernet
2022-04-29  1:03       ` David Vernet
2022-04-29  9:26       ` Michal Koutný
2022-04-29  9:26         ` Michal Koutný
2022-05-06 16:40         ` David Vernet
2022-05-06 16:40           ` David Vernet
2022-05-09 15:09           ` Johannes Weiner
2022-05-09 15:09             ` Johannes Weiner
2022-05-10  0:44             ` Andrew Morton
2022-05-10  0:44               ` Andrew Morton
2022-05-10 17:43               ` Michal Koutný
2022-05-10 17:43                 ` Michal Koutný
2022-05-11 17:53                 ` Johannes Weiner
2022-05-11 17:53                   ` Johannes Weiner
2022-05-12 17:27                   ` Michal Koutný
2022-05-12 17:27                     ` Michal Koutný
2022-04-23 15:56 ` [PATCH v2 3/5] cgroup: Account for memory_localevents in test_memcg_oom_group_leaf_events() David Vernet
2022-04-23 15:56   ` David Vernet
2022-04-23 15:56 ` [PATCH v2 4/5] cgroup: Removing racy check in test_memcg_sock() David Vernet
2022-04-23 15:56   ` David Vernet
2022-04-23 15:56 ` [PATCH v2 5/5] cgroup: Fix racy check in alloc_pagecache_max_30M() helper function David Vernet
2022-04-23 15:56   ` David Vernet
2022-05-12 17:04 ` [PATCH v2 0/5] Fix bugs in memcontroller cgroup tests Michal Koutný
2022-05-12 17:04   ` Michal Koutný
2022-05-12 17:30   ` David Vernet
2022-05-12 17:30     ` David Vernet
2022-05-12 17:44     ` David Vernet
2022-05-12 17:44       ` David Vernet
2022-05-13 17:18       ` [PATCH 0/4] memcontrol selftests fixups Michal Koutný
2022-05-13 17:18         ` Michal Koutný
2022-05-13 17:18         ` [PATCH 1/4] selftests: memcg: Fix compilation Michal Koutný
2022-05-13 17:18           ` Michal Koutný
2022-05-13 17:40           ` David Vernet
2022-05-13 17:40             ` David Vernet
2022-05-13 18:53           ` Roman Gushchin
2022-05-13 18:53             ` Roman Gushchin
2022-05-13 19:09             ` Roman Gushchin
2022-05-13 19:09               ` Roman Gushchin
2022-05-13 17:18         ` [PATCH 2/4] selftests: memcg: Expect no low events in unprotected sibling Michal Koutný
2022-05-13 17:18           ` Michal Koutný
2022-05-13 17:42           ` David Vernet
2022-05-13 17:42             ` David Vernet
2022-05-13 18:54           ` Roman Gushchin
2022-05-18 15:54             ` Michal Koutný
2022-05-18 15:54               ` Michal Koutný
2022-05-13 17:18         ` [PATCH 3/4] selftests: memcg: Adjust expected reclaim values of protected cgroups Michal Koutný
2022-05-13 17:18           ` Michal Koutný
2022-05-13 18:52           ` Roman Gushchin [this message]
2022-05-13 18:52             ` Roman Gushchin
2022-05-13 17:18         ` [PATCH 4/4] selftests: memcg: Remove protection from top level memcg Michal Koutný
2022-05-13 17:18           ` Michal Koutný
2022-05-13 18:59           ` Roman Gushchin
2022-05-13 18:59             ` Roman Gushchin
2022-05-18  0:24             ` Andrew Morton
2022-05-18  0:24               ` Andrew Morton
2022-05-18  0:52               ` Roman Gushchin
2022-05-18  0:52                 ` Roman Gushchin
2022-05-18 15:44                 ` Michal Koutný
2022-05-18 15:44                   ` Michal Koutný
2022-05-13 19:14           ` David Vernet
2022-05-13 19:14             ` David Vernet
2022-05-18 15:40 [PATCH 0/4] memcontrol selftests fixups Michal Koutný
2022-05-18 15:40 ` [PATCH 3/4] selftests: memcg: Adjust expected reclaim values of protected cgroups Michal Koutný
2022-05-18 15:40   ` Michal Koutný

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Yn6pBPq+lAXm9NG8@carbon \
    --to=roman.gushchin@linux.dev \
    --cc=akpm@linux-foundation.org \
    --cc=cgroups@vger.kernel.org \
    --cc=hannes@cmpxchg.org \
    --cc=kernel-team@fb.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=mkoutny@suse.com \
    --cc=rpalethorpe@suse.de \
    --cc=shakeelb@google.com \
    --cc=tj@kernel.org \
    --cc=void@manifault.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.