From: Peter Zijlstra <peterz@infradead.org>
To: Barry Song <song.bao.hua@hisilicon.com>
Cc: valentin.schneider@arm.com, vincent.guittot@linaro.org,
mgorman@suse.de, mingo@kernel.org, dietmar.eggemann@arm.com,
morten.rasmussen@arm.com, linux-kernel@vger.kernel.org,
linuxarm@openeuler.org, xuwei5@huawei.com,
liguozhu@hisilicon.com, tiantao6@hisilicon.com,
wanghuiqiang@huawei.com, prime.zeng@hisilicon.com,
jonathan.cameron@huawei.com, guodong.xu@linaro.org,
yangyicong@huawei.com, Meelis Roos <mroos@linux.ee>
Subject: Re: [PATCH v4] sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
Date: Tue, 2 Mar 2021 11:56:20 +0100 [thread overview]
Message-ID: <YD4Z1K6Vb0+u0JQz@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <20210224030944.15232-1-song.bao.hua@hisilicon.com>
On Wed, Feb 24, 2021 at 04:09:44PM +1300, Barry Song wrote:
> As long as NUMA diameter > 2, building sched_domain by sibling's child
> domain will definitely create a sched_domain with sched_group which will
> span out of the sched_domain:
>
> +------+ +------+ +-------+ +------+
> | node | 12 |node | 20 | node | 12 |node |
> | 0 +---------+1 +--------+ 2 +-------+3 |
> +------+ +------+ +-------+ +------+
>
> domain0 node0 node1 node2 node3
>
> domain1 node0+1 node0+1 node2+3 node2+3
> +
> domain2 node0+1+2 |
> group: node0+1 |
> group:node2+3 <-------------------+
>
> when node2 is added into the domain2 of node0, kernel is using the child
> domain of node2's domain2, which is domain1(node2+3). Node 3 is outside
> the span of the domain including node0+1+2.
>
> This will make load_balance() run based on screwed avg_load and group_type
> in the sched_group spanning out of the sched_domain, and it also makes
> select_task_rq_fair() pick an idle CPU outside the sched_domain.
>
> Real servers which suffer from this problem include Kunpeng920 and 8-node
> Sun Fire X4600-M2, at least.
>
> Here we move to use the *child* domain of the *child* domain of node2's
> domain2 as the new added sched_group. At the same, we re-use the lower
> level sgc directly.
> +------+ +------+ +-------+ +------+
> | node | 12 |node | 20 | node | 12 |node |
> | 0 +---------+1 +--------+ 2 +-------+3 |
> +------+ +------+ +-------+ +------+
>
> domain0 node0 node1 +- node2 node3
> |
> domain1 node0+1 node0+1 | node2+3 node2+3
> |
> domain2 node0+1+2 |
> group: node0+1 |
> group:node2 <-------------------+
>
> While the lower level sgc is re-used, this patch only changes the remote
> sched_groups for those sched_domains playing grandchild trick, therefore,
> sgc->next_update is still safe since it's only touched by CPUs that have
> the group span as local group. And sgc->imbalance is also safe because
> sd_parent remains the same in load_balance and LB only tries other CPUs
> from the local group.
> Moreover, since local groups are not touched, they are still getting
> roughly equal size in a TL. And should_we_balance() only matters with
> local groups, so the pull probability of those groups are still roughly
> equal.
>
> Reported-by: Valentin Schneider <valentin.schneider@arm.com>
> Tested-by: Meelis Roos <mroos@linux.ee>
> Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
Thanks!
next prev parent reply other threads:[~2021-03-02 11:22 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-24 3:09 [PATCH v4] sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2 Barry Song
2021-03-02 10:56 ` Peter Zijlstra [this message]
2021-03-04 9:09 ` [tip: sched/core] " tip-bot2 for Barry Song
2021-03-06 11:42 ` tip-bot2 for Barry Song
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YD4Z1K6Vb0+u0JQz@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=dietmar.eggemann@arm.com \
--cc=guodong.xu@linaro.org \
--cc=jonathan.cameron@huawei.com \
--cc=liguozhu@hisilicon.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxarm@openeuler.org \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=morten.rasmussen@arm.com \
--cc=mroos@linux.ee \
--cc=prime.zeng@hisilicon.com \
--cc=song.bao.hua@hisilicon.com \
--cc=tiantao6@hisilicon.com \
--cc=valentin.schneider@arm.com \
--cc=vincent.guittot@linaro.org \
--cc=wanghuiqiang@huawei.com \
--cc=xuwei5@huawei.com \
--cc=yangyicong@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).