Re: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs

From: Peter Zijlstra <peterz@infradead.org>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: Ingo Molnar <mingo@kernel.org>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Aubrey Li <aubrey.li@linux.intel.com>,
	Barry Song <song.bao.hua@hisilicon.com>,
	Mike Galbraith <efault@gmx.de>,
	Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs
Date: Sat, 4 Dec 2021 11:40:56 +0100	[thread overview]
Message-ID: <20211204104056.GR16608@worktop.programming.kicks-ass.net> (raw)
In-Reply-To: <20211201151844.20488-3-mgorman@techsingularity.net>

On Wed, Dec 01, 2021 at 03:18:44PM +0000, Mel Gorman wrote:
> +	/* Calculate allowed NUMA imbalance */
> +	for_each_cpu(i, cpu_map) {
> +		int imb_numa_nr = 0;
> +
> +		for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
> +			struct sched_domain *child = sd->child;
> +
> +			if (!(sd->flags & SD_SHARE_PKG_RESOURCES) && child &&
> +			    (child->flags & SD_SHARE_PKG_RESOURCES)) {
> +				int nr_groups;
> +
> +				nr_groups = sd->span_weight / child->span_weight;
> +				imb_numa_nr = max(1U, ((child->span_weight) >> 1) /
> +						(nr_groups * num_online_nodes()));
> +			}
> +
> +			sd->imb_numa_nr = imb_numa_nr;
> +		}

OK, so let's see. All domains with SHARE_PKG_RESOURCES set will have
imb_numa_nr = 0, all domains above it will have the same value
calculated here.

So far so good I suppose :-)

Then nr_groups is what it says on the tin; we could've equally well
iterated sd->groups and gotten the same number, but this is simpler.

Now, imb_numa_nr is where the magic happens, the way it's written
doesn't help, but it's something like:

	(child->span_weight / 2) / (nr_groups * num_online_nodes())

With a minimum value of 1. So the larger the system is, or the smaller
the LLCs, the smaller this number gets, right?

So my ivb-ep that has 20 cpus in a LLC and 2 nodes, will get: (20 / 2)
/ (1 * 2) = 10, while the ivb-ex will get: (20/2) / (1*4) = 5.

But a Zen box that has only like 4 CPUs per LLC will have 1, regardless
of how many nodes it has.

Now, I'm thinking this assumes (fairly reasonable) that the level above
LLC is a node, but I don't think we need to assume this, while also not
assuming the balance domain spans the whole machine (yay paritions!).

	for (top = sd; top->parent; top = top->parent)
		;

	nr_llcs = top->span_weight / child->span_weight;
	imb_numa_nr = max(1, child->span_weight / nr_llcs);

which for my ivb-ep gets me:  20 / (40 / 20) = 10
and the Zen system will have:  4 / (huge number) = 1

Now, the exp: a / (b / a) is equivalent to a * (a / b) or a^2/b, so we
can also write the above as:

	(child->span_weight * child->span_weight) / top->span_weight;

Hmm?

> +	}
> +
>  	/* Calculate CPU capacity for physical packages and nodes */
>  	for (i = nr_cpumask_bits-1; i >= 0; i--) {
>  		if (!cpumask_test_cpu(i, cpu_map))
> -- 
> 2.31.1
> 