linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Valentin Schneider <valentin.schneider@arm.com>
To: Oliver Sang <oliver.sang@intel.com>
Cc: 0day robot <lkp@intel.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	LKML <linux-kernel@vger.kernel.org>,
	lkp@lists.01.org, ying.huang@intel.com, feng.tang@intel.com,
	zhengjun.xing@intel.com,
	Lingutla Chandrasekhar <clingutla@codeaurora.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@kernel.org>,
	Morten Rasmussen <morten.rasmussen@arm.com>,
	Qais Yousef <qais.yousef@arm.com>,
	Quentin Perret <qperret@google.com>,
	Pavan Kondeti <pkondeti@codeaurora.org>,
	Rik van Riel <riel@surriel.com>,
	aubrey.li@linux.intel.com, yu.c.chen@intel.com,
	Mel Gorman <mgorman@suse.de>
Subject: Re: [sched/fair]  38ac256d1c:  stress-ng.vm-segv.ops_per_sec -13.8% regression
Date: Wed, 28 Apr 2021 23:00:07 +0100	[thread overview]
Message-ID: <87k0omxe6w.mognet@arm.com> (raw)
In-Reply-To: <87mttqt5jc.mognet@arm.com>

On 22/04/21 21:42, Valentin Schneider wrote:
> On 22/04/21 10:55, Valentin Schneider wrote:
>> I'll go find myself some other x86 box and dig into it;
>> I'd rather not leave this hanging for too long.
>
> So I found myself a dual-socket Xeon Gold 5120 @ 2.20GHz (64 CPUs) and
> *there* I get a somewhat consistent ~-6% regression. As I'm suspecting
> cacheline shenanigans, I also ran that with Peter's recent
> kthread_is_per_cpu() change, and that brings it down to ~-3%
>

Ha ha ho ho, so that was a red herring. My statistical paranoia somewhat
paid off, and the kthread_is_per_cpu() thing doesn't really change anything
when you stare at 20+ iterations of that vm-segv thing.

As far as I can tell, the culprit is the loss of LBF_SOME_PINNED. By some
happy accident, the load balancer repeatedly iterates over PCPU kthreads,
sets LBF_SOME_PINNED and causes a group to be classified as group_imbalanced
in a later load-balance. This, in turn, forces a 1-task pull, and repeating
this pattern ~25 times a sec ends up increasing CPU utilization by ~5% over the
span of the benchmark.

schedstats are somewhat noisy but seem to indicate the baseline had many
more migrations at the NUMA level (test machine has SMT, MC, NUMA). Because
of that I suspected

  b396f52326de ("sched/fair: Allow a small load imbalance between low utilisation SD_NUMA domains")

but reverting that actually makes things worse. I'm still digging, though
I'm slowly heading towards:

  https://www.youtube.com/watch?v=3L6i5AwVAbs

  reply	other threads:[~2021-04-28 22:00 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-07 22:06 [PATCH v5 0/3] sched/fair: load-balance vs capacity margins Valentin Schneider
2021-04-07 22:06 ` [PATCH v5 1/3] sched/fair: Ignore percpu threads for imbalance pulls Valentin Schneider
2021-04-09 11:24   ` [tip: sched/core] " tip-bot2 for Lingutla Chandrasekhar
2021-04-09 12:05   ` tip-bot2 for Lingutla Chandrasekhar
2021-04-09 16:14   ` tip-bot2 for Lingutla Chandrasekhar
2021-04-14  5:21   ` [sched/fair] 38ac256d1c: stress-ng.vm-segv.ops_per_sec -13.8% regression kernel test robot
2021-04-14 17:17     ` Valentin Schneider
2021-04-21  3:20       ` Oliver Sang
2021-04-21 10:27         ` Valentin Schneider
2021-04-21 12:03           ` Peter Zijlstra
2021-04-22  7:47           ` Oliver Sang
2021-04-22  9:55             ` Valentin Schneider
2021-04-22 20:42               ` Valentin Schneider
2021-04-28 22:00                 ` Valentin Schneider [this message]
2021-05-06 16:11                   ` Valentin Schneider
2021-04-07 22:06 ` [PATCH v5 2/3] sched/fair: Clean up active balance nr_balance_failed trickery Valentin Schneider
2021-04-09 11:24   ` [tip: sched/core] " tip-bot2 for Valentin Schneider
2021-04-09 12:05   ` tip-bot2 for Valentin Schneider
2021-04-09 16:14   ` tip-bot2 for Valentin Schneider
2021-04-07 22:06 ` [PATCH v5 3/3] sched/fair: Introduce a CPU capacity comparison helper Valentin Schneider
2021-04-09 11:24   ` [tip: sched/core] " tip-bot2 for Valentin Schneider
2021-04-09 12:05   ` tip-bot2 for Valentin Schneider
2021-04-09 16:14   ` tip-bot2 for Valentin Schneider

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87k0omxe6w.mognet@arm.com \
    --to=valentin.schneider@arm.com \
    --cc=aubrey.li@linux.intel.com \
    --cc=clingutla@codeaurora.org \
    --cc=dietmar.eggemann@arm.com \
    --cc=feng.tang@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lkp@intel.com \
    --cc=lkp@lists.01.org \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=morten.rasmussen@arm.com \
    --cc=oliver.sang@intel.com \
    --cc=peterz@infradead.org \
    --cc=pkondeti@codeaurora.org \
    --cc=qais.yousef@arm.com \
    --cc=qperret@google.com \
    --cc=riel@surriel.com \
    --cc=vincent.guittot@linaro.org \
    --cc=ying.huang@intel.com \
    --cc=yu.c.chen@intel.com \
    --cc=zhengjun.xing@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).