From: Josef Bacik <jbacik@fb.com>
To: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>, <riel@redhat.com>,
<mingo@redhat.com>, <linux-kernel@vger.kernel.org>,
<morten.rasmussen@arm.com>, kernel-team <Kernel-team@fb.com>
Subject: Re: [PATCH RESEND] sched: prefer an idle cpu vs an idle sibling for BALANCE_WAKE
Date: Mon, 6 Jul 2015 15:41:02 -0400 [thread overview]
Message-ID: <559AD9CE.4090309@fb.com> (raw)
In-Reply-To: <1436207790.2940.30.camel@gmail.com>
On 07/06/2015 02:36 PM, Mike Galbraith wrote:
> On Mon, 2015-07-06 at 10:34 -0400, Josef Bacik wrote:
>> On 07/06/2015 01:13 AM, Mike Galbraith wrote:
>>> Hm. Piddling with pgbench, which doesn't seem to collapse into a
>>> quivering heap when load exceeds cores these days, deltas weren't all
>>> that impressive, but it does appreciate the extra effort a bit, and a
>>> bit more when clients receive it as well.
>>>
>>> If you test, and have time to piddle, you could try letting wake_wide()
>>> return 1 + sched_feat(WAKE_WIDE_IDLE) instead of adding only if wakee is
>>> the dispatcher.
>>>
>>> Numbers from my little desktop box.
>>>
>>> NO_WAKE_WIDE_IDLE
>>> postgres@homer:~> pgbench.sh
>>> clients 8 tps = 116697.697662
>>> clients 12 tps = 115160.230523
>>> clients 16 tps = 115569.804548
>>> clients 20 tps = 117879.230514
>>> clients 24 tps = 118281.753040
>>> clients 28 tps = 116974.796627
>>> clients 32 tps = 119082.163998 avg 117092.239 1.000
>>>
>>> WAKE_WIDE_IDLE
>>> postgres@homer:~> pgbench.sh
>>> clients 8 tps = 124351.735754
>>> clients 12 tps = 124419.673135
>>> clients 16 tps = 125050.716498
>>> clients 20 tps = 124813.042352
>>> clients 24 tps = 126047.442307
>>> clients 28 tps = 125373.719401
>>> clients 32 tps = 126711.243383 avg 125252.510 1.069 1.000
>>>
>>> WAKE_WIDE_IDLE (clients as well as server)
>>> postgres@homer:~> pgbench.sh
>>> clients 8 tps = 130539.795246
>>> clients 12 tps = 128984.648554
>>> clients 16 tps = 130564.386447
>>> clients 20 tps = 129149.693118
>>> clients 24 tps = 130211.119780
>>> clients 28 tps = 130325.355433
>>> clients 32 tps = 129585.656963 avg 129908.665 1.109 1.037
>
> I had a typo in my script, so those desktop box numbers were all doing
> the same number of clients. It doesn't invalidate anything, but the
> individual deltas are just run to run variance.. not to mention that
> single cache box is not all that interesting for this anyway. That
> happens when interconnect becomes a player.
>
>> I have time for twiddling, we're carrying ye olde WAKE_IDLE until we get
>> this solved upstream and then I'll rip out the old and put in the new,
>> I'm happy to screw around until we're all happy. I'll throw this in a
>> kernel this morning and run stuff today. Barring any issues with the
>> testing infrastructure I should have results today. Thanks,
>
> I'll be interested in your results. Taking pgbench to a little NUMA
> box, I'm seeing _nada_ outside of variance with master (crap). I have a
> way to win significantly for _older_ kernels, and that win over master
> _may_ provide some useful insight, but I don't trust postgres/pgbench as
> far as I can toss the planet, so don't have a warm fuzzy about trying to
> use it to approximate your real world load.
>
> BTW, what's your topology look like (numactl --hardware).
>
So the NO_WAKE_WIDE_IDLE results are very good, almost the same as the
baseline with a slight regression at lower RPS and a slight improvement
at high RPS. I'm running with WAKE_WIDE_IDLE set now, that should be
done soonish and then I'll do the 1 + sched_feat(WAKE_WIDE_IDLE) thing
next and those results should come in the morning. Here is the numa
information from one of the boxes in the test cluster
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 15890 MB
node 0 free: 2651 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 16125 MB
node 1 free: 2063 MB
node distances:
node 0 1
0: 10 20
1: 20 10
Thanks,
Josef
next prev parent reply other threads:[~2015-07-06 19:41 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-27 21:22 [PATCH RESEND] sched: prefer an idle cpu vs an idle sibling for BALANCE_WAKE Josef Bacik
2015-05-28 3:46 ` Mike Galbraith
2015-05-28 9:49 ` Morten Rasmussen
2015-05-28 10:57 ` Mike Galbraith
2015-05-28 11:48 ` Morten Rasmussen
2015-05-28 11:49 ` Mike Galbraith
2015-05-28 10:21 ` Peter Zijlstra
2015-05-28 11:05 ` Peter Zijlstra
2015-05-28 14:27 ` Josef Bacik
2015-05-29 21:03 ` Josef Bacik
2015-05-30 3:55 ` Mike Galbraith
2015-06-01 19:38 ` Josef Bacik
2015-06-01 20:42 ` Peter Zijlstra
2015-06-01 21:03 ` Josef Bacik
2015-06-02 17:12 ` Josef Bacik
2015-06-03 14:12 ` Rik van Riel
2015-06-03 14:24 ` Peter Zijlstra
2015-06-03 14:49 ` Josef Bacik
2015-06-03 15:30 ` Mike Galbraith
2015-06-03 15:57 ` Josef Bacik
2015-06-03 16:53 ` Mike Galbraith
2015-06-03 17:16 ` Josef Bacik
2015-06-03 17:43 ` Mike Galbraith
2015-06-03 20:34 ` Josef Bacik
2015-06-04 4:52 ` Mike Galbraith
2015-06-01 22:15 ` Rik van Riel
2015-06-11 20:33 ` Josef Bacik
2015-06-12 3:42 ` Rik van Riel
2015-06-12 5:35 ` Mike Galbraith
2015-06-17 18:06 ` Josef Bacik
2015-06-18 0:55 ` Mike Galbraith
2015-06-18 3:46 ` Josef Bacik
2015-06-18 4:12 ` Mike Galbraith
2015-07-02 17:44 ` Josef Bacik
2015-07-03 6:40 ` Mike Galbraith
2015-07-03 9:29 ` Mike Galbraith
2015-07-04 15:57 ` Mike Galbraith
2015-07-05 7:17 ` Mike Galbraith
2015-07-06 5:13 ` Mike Galbraith
2015-07-06 14:34 ` Josef Bacik
2015-07-06 18:36 ` Mike Galbraith
2015-07-06 19:41 ` Josef Bacik [this message]
2015-07-07 4:01 ` Mike Galbraith
2015-07-07 9:43 ` [patch] " Mike Galbraith
2015-07-07 13:40 ` Josef Bacik
2015-07-07 15:24 ` Mike Galbraith
2015-07-07 17:06 ` Josef Bacik
2015-07-08 6:13 ` [patch] sched: beef up wake_wide() Mike Galbraith
2015-07-09 13:26 ` Peter Zijlstra
2015-07-09 14:07 ` Mike Galbraith
2015-07-09 14:46 ` Mike Galbraith
2015-07-10 5:19 ` Mike Galbraith
2015-07-10 13:41 ` Josef Bacik
2015-07-10 20:59 ` Josef Bacik
2015-07-11 3:11 ` Mike Galbraith
2015-07-13 13:53 ` Josef Bacik
2015-07-14 11:19 ` Peter Zijlstra
2015-07-14 13:49 ` Mike Galbraith
2015-07-14 14:07 ` Peter Zijlstra
2015-07-14 14:17 ` Mike Galbraith
2015-07-14 15:04 ` Peter Zijlstra
2015-07-14 15:39 ` Mike Galbraith
2015-07-14 16:01 ` Josef Bacik
2015-07-14 17:59 ` Mike Galbraith
2015-07-15 17:11 ` Josef Bacik
2015-08-03 17:07 ` [tip:sched/core] sched/fair: Beef " tip-bot for Mike Galbraith
2015-05-28 11:16 ` [PATCH RESEND] sched: prefer an idle cpu vs an idle sibling for BALANCE_WAKE Mike Galbraith
2015-05-28 11:49 ` Ingo Molnar
2015-05-28 12:15 ` Mike Galbraith
2015-05-28 12:19 ` Peter Zijlstra
2015-05-28 12:29 ` Ingo Molnar
2015-05-28 15:22 ` David Ahern
2015-05-28 11:55 ` Srikar Dronamraju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=559AD9CE.4090309@fb.com \
--to=jbacik@fb.com \
--cc=Kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=morten.rasmussen@arm.com \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=umgwanakikbuti@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).