Re: [PATCH] sched, fair: Allow a small degree of load imbalance between SD_NUMA domains v2

From: Mel Gorman <mgorman@techsingularity.net>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Phil Auld <pauld@redhat.com>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
	Quentin Perret <quentin.perret@arm.com>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Morten Rasmussen <Morten.Rasmussen@arm.com>,
	Hillf Danton <hdanton@sina.com>, Parth Shah <parth@linux.ibm.com>,
	Rik van Riel <riel@surriel.com>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] sched, fair: Allow a small degree of load imbalance between SD_NUMA domains v2
Date: Tue, 7 Jan 2020 20:24:06 +0000	[thread overview]
Message-ID: <20200107202406.GJ3466@techsingularity.net> (raw)
In-Reply-To: <CAKfTPtAacdmxniM9yZUrQPW39LAvhpBQt6ZiGiwk5qpEx7zicA@mail.gmail.com>

On Tue, Jan 07, 2020 at 05:00:29PM +0100, Vincent Guittot wrote:
> > > Taking into account child domain makes sense to me, but shouldn't we
> > > take into account the number of child group instead ? This should
> > > reflect the number of different LLC caches.
> >
> > I guess it would but why is it inherently better? The number of domains
> > would yield a similar result if we assume that all the lower domains
> > have equal weight so it simply because the weight of the SD_NUMA domain
> > divided by the number of child domains.
> 
> but that's not what you are doing in your proposal. You are using
> directly child->span_weight which reflects the number of CPUs in the
> child and not the number of group
> 
> you should do something like  sds->busiest->span_weight /
> sds->busiest->child->span_weight which gives you an approximation of
> the number of independent group inside the busiest numa node from a
> shared resource pov
> 

Now I get you, but unfortunately it also would not work out. The number
of groups is not related to the LLC except in some specific cases.
It's possible to use the first CPU to find the size of an LLC but now I
worry that it would lead to unpredictable behaviour. AMD has different
numbers of LLCs per node depending on the CPU family and while Intel
generally has one LLC per node, I imagine there are counter examples.
This means that load balancing on different machines with similar core
counts will behave differently due to the LLC size. It might be possible
to infer it if the intermediate domain was DIE instead of MC but I doubt
that's guaranteed and it would still be unpredictable. It may be the type
of complexity that should only be introduced with a separate patch with
clear rationale as to why it's necessary and we are not at that threshold
so I withdraw the suggestion.

-- 
Mel Gorman
SUSE Labs