From: "Gautham R. Shenoy" <gautham.shenoy@amd.com>
To: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Yicong Yang <yangyicong@hisilicon.com>,
peterz@infradead.org, mingo@redhat.com, juri.lelli@redhat.com,
vincent.guittot@linaro.org, tim.c.chen@linux.intel.com,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, dietmar.eggemann@arm.com,
rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com,
prime.zeng@huawei.com, jonathan.cameron@huawei.com,
ego@linux.vnet.ibm.com, srikar@linux.vnet.ibm.com,
linuxarm@huawei.com, 21cnbao@gmail.com, guodong.xu@linaro.org,
hesham.almatary@huawei.com, john.garry@huawei.com,
shenyang39@huawei.com
Subject: Re: [PATCH v4 1/2] sched: Add per_cpu cluster domain info and cpus_share_resources API
Date: Wed, 15 Jun 2022 21:13:39 +0530 [thread overview]
Message-ID: <Yqn+K08JFRgtR3eY@BLR-5CG11610CF.amd.com> (raw)
In-Reply-To: <e000b124-afd4-28e1-fde2-393b0e38ce19@amd.com>
On Wed, Jun 15, 2022 at 07:49:22PM +0530, K Prateek Nayak wrote:
[..snip..]
>
> - Bisecting:
>
> When we ran the tests with only Patch 1 of the series, the
> regression was visible and the numbers were worse.
>
> Clients: tip cluster Patch 1 Only
> 8 3263.81 (0.00 pct) 3086.81 (-5.42 pct) 3018.63 (-7.51 pct)
> 16 6011.19 (0.00 pct) 5360.28 (-10.82 pct) 4869.26 (-18.99 pct)
> 32 12058.31 (0.00 pct) 8769.08 (-27.27 pct) 8159.60 (-32.33 pct)
> 64 21258.21 (0.00 pct) 19021.09 (-10.52 pct) 13161.92 (-38.08 pct)
>
> We further bisected the hunks to narrow down the cause to the per CPU
> variable declarations.
>
>
> >
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 01259611beb9..b9bcfcf8d14d 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -1753,7 +1753,9 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag)
> > DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc);
> > DECLARE_PER_CPU(int, sd_llc_size);
> > DECLARE_PER_CPU(int, sd_llc_id);
> > +DECLARE_PER_CPU(int, sd_share_id);
> > DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared);
> > +DECLARE_PER_CPU(struct sched_domain __rcu *, sd_cluster);
>
> The main reason for the regression seems to be the above declarations.
I think you meant that the regressions are due to the DEFINE_PER_CPU()
instances from the following hunk:
> > @@ -664,6 +664,8 @@ static void destroy_sched_domains(struct sched_domain *sd)
> > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);
> > DEFINE_PER_CPU(int, sd_llc_size);
> > DEFINE_PER_CPU(int, sd_llc_id);
> > +DEFINE_PER_CPU(int, sd_share_id);
> > +DEFINE_PER_CPU(struct sched_domain __rcu *, sd_cluster);
> > DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared);
> > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);
> > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
> >
The System.map diff for these variables between tip vs tip +
cluster-sched-v4 on your test system looks as follows:
0000000000020520 D sd_asym_packing
0000000000020528 D sd_numa
-0000000000020530 D sd_llc_shared
-0000000000020538 D sd_llc_id
-000000000002053c D sd_llc_size
-0000000000020540 D sd_llc
+0000000000020530 D sd_cluster
+0000000000020538 D sd_llc_shared
+0000000000020540 D sd_share_id
+0000000000020544 D sd_llc_id
+0000000000020548 D sd_llc_size
+0000000000020550 D sd_llc
The allocations are in the reverse-order of the definitions.
That perhaps explains why you no longer see the regression when you
define the sd_share_id and sd_cluster per-cpu definitions at the
beginning as indicated by the following
> - Move the declarations of sd_share_id and sd_cluster to the top
>
> Clients: tip Patch 1 Patch 1 (Declarion on Top)
> 8 3255.69 (0.00 pct) 3018.63 (-7.28 pct) 3072.30 (-5.63 pct)
> 16 6092.67 (0.00 pct) 4869.26 (-20.08 pct) 5586.59 (-8.30 pct)
> 32 11156.56 (0.00 pct) 8159.60 (-26.86 pct) 11184.17 (0.24 pct)
> 64 21019.97 (0.00 pct) 13161.92 (-37.38 pct) 20289.70 (-3.47 pct)
--
Thanks and Regards
gautham.
next prev parent reply other threads:[~2022-06-15 15:44 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-09 12:06 [PATCH v4 0/2] sched/fair: Wake task within the cluster when possible Yicong Yang
2022-06-09 12:06 ` [PATCH v4 1/2] sched: Add per_cpu cluster domain info and cpus_share_resources API Yicong Yang
2022-06-09 22:28 ` Tim Chen
2022-06-10 6:54 ` Yicong Yang
2022-06-15 14:19 ` K Prateek Nayak
2022-06-15 15:43 ` Gautham R. Shenoy [this message]
2022-06-16 8:10 ` Yicong Yang
2022-06-16 7:55 ` Yicong Yang
2022-06-17 12:20 ` K Prateek Nayak
2022-06-17 16:50 ` Tim Chen
2022-06-20 11:20 ` K Prateek Nayak
2022-06-20 13:37 ` Abel Wu
2022-06-28 11:55 ` K Prateek Nayak
2022-06-29 3:22 ` Yicong Yang
2022-06-09 12:06 ` [PATCH v4 2/2] sched/fair: Scan cluster before scanning LLC in wake-up path Yicong Yang
2022-06-09 22:47 ` Tim Chen
2022-06-10 4:01 ` Barry Song
2022-06-10 6:39 ` Yicong Yang
2022-06-10 21:19 ` Tim Chen
2022-06-11 3:03 ` Chen Yu
2022-06-11 7:40 ` Yicong Yang
2022-06-11 9:04 ` Chen Yu
2022-06-26 12:13 ` Abel Wu
2022-06-27 8:16 ` Yicong Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Yqn+K08JFRgtR3eY@BLR-5CG11610CF.amd.com \
--to=gautham.shenoy@amd.com \
--cc=21cnbao@gmail.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=ego@linux.vnet.ibm.com \
--cc=guodong.xu@linaro.org \
--cc=hesham.almatary@huawei.com \
--cc=john.garry@huawei.com \
--cc=jonathan.cameron@huawei.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxarm@huawei.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=prime.zeng@huawei.com \
--cc=rostedt@goodmis.org \
--cc=shenyang39@huawei.com \
--cc=srikar@linux.vnet.ibm.com \
--cc=tim.c.chen@linux.intel.com \
--cc=vincent.guittot@linaro.org \
--cc=yangyicong@hisilicon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).