linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Valentin Schneider <valentin.schneider@arm.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	"Cc: Len Brown" <lenb@kernel.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	Jonathan Cameron <jonathan.cameron@huawei.com>,
	Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Juri Lelli <juri.lelli@redhat.com>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Mark Rutland <mark.rutland@arm.com>,
	LAK <linux-arm-kernel@lists.infradead.org>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	ACPI Devel Maling List <linux-acpi@vger.kernel.org>,
	Linuxarm <linuxarm@huawei.com>, "xuwei (O)" <xuwei5@huawei.com>,
	"Zengtao (B)" <prime.zeng@hisilicon.com>
Subject: RE: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters
Date: Thu, 3 Dec 2020 09:11:15 +0000	[thread overview]
Message-ID: <d81006facd444d8a83bd7f1e24ccf6d9@hisilicon.com> (raw)
In-Reply-To: <CAKfTPtALPjSvOZ2xf9cka9R-1uqi3AHQ+GYy7asT3wfvmLqaXw@mail.gmail.com>



> -----Original Message-----
> From: Vincent Guittot [mailto:vincent.guittot@linaro.org]
> Sent: Thursday, December 3, 2020 10:04 PM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> Cc: Valentin Schneider <valentin.schneider@arm.com>; Catalin Marinas
> <catalin.marinas@arm.com>; Will Deacon <will@kernel.org>; Rafael J. Wysocki
> <rjw@rjwysocki.net>; Cc: Len Brown <lenb@kernel.org>;
> gregkh@linuxfoundation.org; Jonathan Cameron <jonathan.cameron@huawei.com>;
> Ingo Molnar <mingo@redhat.com>; Peter Zijlstra <peterz@infradead.org>; Juri
> Lelli <juri.lelli@redhat.com>; Dietmar Eggemann <dietmar.eggemann@arm.com>;
> Steven Rostedt <rostedt@goodmis.org>; Ben Segall <bsegall@google.com>; Mel
> Gorman <mgorman@suse.de>; Mark Rutland <mark.rutland@arm.com>; LAK
> <linux-arm-kernel@lists.infradead.org>; linux-kernel
> <linux-kernel@vger.kernel.org>; ACPI Devel Maling List
> <linux-acpi@vger.kernel.org>; Linuxarm <linuxarm@huawei.com>; xuwei (O)
> <xuwei5@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>
> Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters
> 
> On Wed, 2 Dec 2020 at 21:58, Song Bao Hua (Barry Song)
> <song.bao.hua@hisilicon.com> wrote:
> >
> > >
> > > Sorry. Please ignore this. I added some printk here while testing
> > > one numa. Will update you the data in another email.
> >
> > Re-tested in one NUMA node(cpu0-cpu23):
> >
> > g=1
> > Running in threaded mode with 1 groups using 40 file descriptors
> > Each sender will pass 100000 messages of 100 bytes
> > w/o: 7.689 7.485 7.485 7.458 7.524 7.539 7.738 7.693 7.568 7.674=7.5853
> > w/ : 7.516 7.941 7.374 7.963 7.881 7.910 7.420 7.556 7.695 7.441=7.6697
> > w/ but dropped select_idle_cluster:
> >      7.752 7.739 7.739 7.571 7.545 7.685 7.407 7.580 7.605 7.487=7.611
> >
> > g=2
> > Running in threaded mode with 2 groups using 40 file descriptors
> > Each sender will pass 100000 messages of 100 bytes
> > w/o: 10.127 10.119 10.070 10.196 10.057 10.111 10.045 10.164 10.162
> > 9.955=10.1006
> > w/ : 9.694 9.654 9.612 9.649 9.686 9.734 9.607 9.842 9.690 9.710=9.6878
> > w/ but dropped select_idle_cluster:
> >      9.877 10.069 9.951 9.918 9.947 9.790 9.906 9.820 9.863 9.906=9.9047
> >
> > g=3
> > Running in threaded mode with 3 groups using 40 file descriptors
> > Each sender will pass 100000 messages of 100 bytes
> > w/o: 15.885 15.254 15.932 15.647 16.120 15.878 15.857 15.759 15.674
> > 15.721=15.7727
> > w/ : 14.974 14.657 13.969 14.985 14.728 15.665 15.191 14.995 14.946
> > 14.895=14.9005
> > w/ but dropped select_idle_cluster:
> >      15.405 15.177 15.373 15.187 15.450 15.540 15.278 15.628 15.228
> 15.325=15.3591
> >
> > g=4
> > Running in threaded mode with 4 groups using 40 file descriptors
> > Each sender will pass 100000 messages of 100 bytes
> > w/o: 20.014 21.025 21.119 21.235 19.767 20.971 20.962 20.914 21.090
> 21.090=20.8187
> > w/ : 20.331 20.608 20.338 20.445 20.456 20.146 20.693 20.797 21.381
> 20.452=20.5647
> > w/ but dropped select_idle_cluster:
> >      19.814 20.126 20.229 20.350 20.750 20.404 19.957 19.888 20.226
> 20.562=20.2306
> >
> 
> I assume that you have run this on v5.9 as previous tests.

Yep

> The results don't show any real benefit of select_idle_cluster()
> inside a node whereas this is where we could expect most of the
> benefit. We have to understand why we have such an impact on numa
> tests only.

There is a 4-5.5% increase while g=2 and g=3.

Regarding the huge increase in NUMA case,  at the first beginning, I suspect
we have wrong llc domain. For example, if cpu0's llc domain span
cpu0-cpu47, then select_idle_cpu() is running in wrong range while
it should run in cpu0-cpu23.

But after printing the llc domain's span, I find it is completely right.
Cpu0's llc span: cpu0-cpu23
Cpu24's llc span: cpu24-cpu47

Maybe I need more trace data to figure out if select_idle_cpu() is running
correctly. For example, maybe I can figure out if it is always returning -1,
or it returns -1 very often?

Or do you have any idea?


> 
> > Thanks
> > Barry

Thanks
Barry


  reply	other threads:[~2020-12-03  9:12 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-01  2:59 [RFC PATCH v2 0/2] scheduler: expose the topology of clusters and add cluster scheduler Barry Song
2020-12-01  2:59 ` [RFC PATCH v2 1/2] topology: Represent clusters of CPUs within a die Barry Song
2020-12-01 16:03   ` Valentin Schneider
2020-12-02  9:55     ` Sudeep Holla
2020-12-01  2:59 ` [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters Barry Song
2020-12-01 16:04   ` Valentin Schneider
2020-12-03  9:28     ` Peter Zijlstra
2020-12-03  9:49       ` Mel Gorman
2020-12-03  9:57       ` Song Bao Hua (Barry Song)
2020-12-03 10:07         ` Peter Zijlstra
2020-12-02  8:27   ` Vincent Guittot
2020-12-02  9:20     ` Song Bao Hua (Barry Song)
2020-12-02 10:16       ` Vincent Guittot
2020-12-02 10:45         ` Song Bao Hua (Barry Song)
2020-12-02 10:48         ` Song Bao Hua (Barry Song)
2020-12-02 20:58         ` Song Bao Hua (Barry Song)
2020-12-03  9:03           ` Vincent Guittot
2020-12-03  9:11             ` Song Bao Hua (Barry Song) [this message]
2020-12-03  9:39               ` Vincent Guittot
2020-12-03  9:54                 ` Vincent Guittot
2020-12-07  9:59                 ` Song Bao Hua (Barry Song)
2020-12-07 15:29                   ` Vincent Guittot
2020-12-09 11:35                     ` Song Bao Hua (Barry Song)
2020-12-01 10:46 ` [RFC PATCH v2 0/2] scheduler: expose the topology of clusters and add cluster scheduler Dietmar Eggemann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d81006facd444d8a83bd7f1e24ccf6d9@hisilicon.com \
    --to=song.bao.hua@hisilicon.com \
    --cc=bsegall@google.com \
    --cc=catalin.marinas@arm.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jonathan.cameron@huawei.com \
    --cc=juri.lelli@redhat.com \
    --cc=lenb@kernel.org \
    --cc=linux-acpi@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linuxarm@huawei.com \
    --cc=mark.rutland@arm.com \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=prime.zeng@hisilicon.com \
    --cc=rjw@rjwysocki.net \
    --cc=rostedt@goodmis.org \
    --cc=valentin.schneider@arm.com \
    --cc=vincent.guittot@linaro.org \
    --cc=will@kernel.org \
    --cc=xuwei5@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).