Re: [PATCH v4 04/11] sched/fair: rework load_balance

From: Mel Gorman <mgorman@techsingularity.net>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
	Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Phil Auld <pauld@redhat.com>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
	Quentin Perret <quentin.perret@arm.com>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Morten Rasmussen <Morten.Rasmussen@arm.com>,
	Hillf Danton <hdanton@sina.com>, Parth Shah <parth@linux.ibm.com>,
	Rik van Riel <riel@surriel.com>
Subject: Re: [PATCH v4 04/11] sched/fair: rework load_balance
Date: Tue, 12 Nov 2019 17:45:08 +0000	[thread overview]
Message-ID: <20191112174508.GY3016@techsingularity.net> (raw)
In-Reply-To: <CAKfTPtCVdG1zcd4kyU4d+K_+VdW7TZn+RSDKt4Hk28B366NPOQ@mail.gmail.com>

On Tue, Nov 12, 2019 at 04:40:20PM +0100, Vincent Guittot wrote:
> On Tue, 12 Nov 2019 at 16:06, Mel Gorman <mgorman@techsingularity.net> wrote:
> >
> > On Tue, Nov 12, 2019 at 11:58:30AM +0100, Vincent Guittot wrote:
> > > > This roughly matches what I've seen. The interesting part to me for
> > > > netperf is the next section of the report that reports the locality of
> > > > numa hints. With netperf on a 2-socket machine, it's generally around
> > > > 50% as the client/server are pulled apart. Because netperf is not
> > > > heavily memory bound, it doesn't have much impact on the overall
> > > > performance but it's good at catching the cross-node migrations.
> > >
> > > Ok. I didn't want to make my reply too long. I have put them below for
> > > the netperf-tcp results:
> > >                                         5.3-rc2        5.3-rc2
> > >                                             tip      +rwk+fix
> > > Ops NUMA alloc hit                  60077762.00    60387907.00
> > > Ops NUMA alloc miss                        0.00           0.00
> > > Ops NUMA interleave hit                    0.00           0.00
> > > Ops NUMA alloc local                60077571.00    60387798.00
> > > Ops NUMA base-page range updates        5948.00       17223.00
> > > Ops NUMA PTE updates                    5948.00       17223.00
> > > Ops NUMA PMD updates                       0.00           0.00
> > > Ops NUMA hint faults                    4639.00       14050.00
> > > Ops NUMA hint local faults %            2073.00        6515.00
> > > Ops NUMA hint local percent               44.69          46.37
> > > Ops NUMA pages migrated                 1528.00        4306.00
> > > Ops AutoNUMA cost                         23.27          70.45
> > >
> >
> > Thanks -- it was "NUMA hint local percent" I was interested in and the
> > 46.37% local hinting faults is likely indicative of the client/server
> > being load balanced across SD_NUMA domains without NUMA Balancing being
> > aggressive enough to fix it. At least I know I am not just seriously
> > unlucky or testing magical machines!
> 
> I agree that the collaboration between load balanced across SD_NUMA
> level and NUMA balancing should be improved
> 
> It's also interesting to notice that the patchset doesn't seem to do
> worse than the baseline: 46.37% vs 44.69%
> 

Yes, I should have highlighted that. The series appears to improve a
number of areas while being performance neutral with respect to SD_NUMA.
If this turns out to be wrong in some case, it should be semi-obvious even
if the locality looks ok. It'll be a headline regression with increased
NUMA pte scanning and increased frequency of migrations indicating that
NUMA balancing is taken excessive corrective action. I'll know it when
I see it :P

-- 
Mel Gorman
SUSE Labs