From: Ingo Molnar <mingo@kernel.org>
To: Aubrey Li <aubrey.intel@gmail.com>
Cc: "Julien Desfossez" <jdesfossez@digitalocean.com>,
"Vineeth Remanan Pillai" <vpillai@digitalocean.com>,
"Nishanth Aravamudan" <naravamudan@digitalocean.com>,
"Peter Zijlstra" <peterz@infradead.org>,
"Tim Chen" <tim.c.chen@linux.intel.com>,
"Thomas Gleixner" <tglx@linutronix.de>,
"Paul Turner" <pjt@google.com>,
"Linus Torvalds" <torvalds@linux-foundation.org>,
"Linux List Kernel Mailing" <linux-kernel@vger.kernel.org>,
"Subhra Mazumdar" <subhra.mazumdar@oracle.com>,
"Frédéric Weisbecker" <fweisbec@gmail.com>,
"Kees Cook" <keescook@chromium.org>,
"Greg Kerr" <kerrnel@google.com>, "Phil Auld" <pauld@redhat.com>,
"Aaron Lu" <aaron.lwe@gmail.com>,
"Valentin Schneider" <valentin.schneider@arm.com>,
"Mel Gorman" <mgorman@techsingularity.net>,
"Pawan Gupta" <pawan.kumar.gupta@linux.intel.com>,
"Paolo Bonzini" <pbonzini@redhat.com>
Subject: Re: [RFC PATCH v2 00/17] Core scheduling v2
Date: Sun, 28 Apr 2019 11:33:04 +0200 [thread overview]
Message-ID: <20190428093304.GA7393@gmail.com> (raw)
In-Reply-To: <CAERHkrtaU=Y-Lxypu_7uBbe-mJtG-3friz=ZLhV53X4FXHcEyA@mail.gmail.com>
* Aubrey Li <aubrey.intel@gmail.com> wrote:
> > But what we are really interested in are throughput numbers under
> > these three kernel variants, right?
>
> These are sysbench events per second number, higher is better.
>
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 1/1 508.5( 0.2%) 504.7( 1.1%) -0.8% 509.0( 0.2%) 0.1%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 2/2 1000.2( 1.4%) 1004.1( 1.6%) 0.4% 997.6( 1.2%) -0.3%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 4/4 1912.1( 1.0%) 1904.2( 1.1%) -0.4% 1914.9( 1.3%) 0.1%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 8/8 3753.5( 0.3%) 3748.2( 0.3%) -0.1% 3751.3( 0.4%) -0.1%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 16/16 7139.3( 2.4%) 7137.9( 1.8%) -0.0% 7049.2( 2.4%) -1.3%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 32/32 10899.0( 4.2%) 10780.3( 4.4%) -1.1% 10339.2( 9.6%) -5.1%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 64/64 15086.1(11.5%) 14262.0( 8.2%) -5.5% 11168.7(22.2%) -26.0%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 128/128 15371.9(22.0%) 14675.8(14.4%) -4.5% 10963.9(18.5%) -28.7%
> NA/AVX baseline(std%) coresched(std%) +/- nosmt(std%) +/-
> 256/256 15990.8(22.0%) 12227.9(10.3%) -23.5% 10469.9(19.6%) -34.5%
So because I'm a big fan of presenting data in a readable fashion, here
are your results, tabulated:
#
# Sysbench throughput comparison of 3 different kernels at different
# load levels, higher numbers are better:
#
.--------------------------------------|----------------------------------------------------------------.
| NA/AVX vanilla-SMT [stddev%] |coresched-SMT [stddev%] +/- | no-SMT [stddev%] +/- |
|--------------------------------------|----------------------------------------------------------------|
| 1/1 508.5 [ 0.2% ] | 504.7 [ 1.1% ] 0.8% | 509.0 [ 0.2% ] 0.1% |
| 2/2 1000.2 [ 1.4% ] | 1004.1 [ 1.6% ] 0.4% | 997.6 [ 1.2% ] 0.3% |
| 4/4 1912.1 [ 1.0% ] | 1904.2 [ 1.1% ] 0.4% | 1914.9 [ 1.3% ] 0.1% |
| 8/8 3753.5 [ 0.3% ] | 3748.2 [ 0.3% ] 0.1% | 3751.3 [ 0.4% ] 0.1% |
| 16/16 7139.3 [ 2.4% ] | 7137.9 [ 1.8% ] 0.0% | 7049.2 [ 2.4% ] 1.3% |
| 32/32 10899.0 [ 4.2% ] | 10780.3 [ 4.4% ] -1.1% | 10339.2 [ 9.6% ] -5.1% |
| 64/64 15086.1 [ 11.5% ] | 14262.0 [ 8.2% ] -5.5% | 11168.7 [ 22.2% ] -26.0% |
| 128/128 15371.9 [ 22.0% ] | 14675.8 [ 14.4% ] -4.5% | 10963.9 [ 18.5% ] -28.7% |
| 256/256 15990.8 [ 22.0% ] | 12227.9 [ 10.3% ] -23.5% | 10469.9 [ 19.6% ] -34.5% |
'--------------------------------------|----------------------------------------------------------------'
One major thing that sticks out is that if we compare the stddev numbers
to the +/- comparisons then it's pretty clear that the benchmarks are
very noisy: in all but the last row stddev is actually higher than the
measured effect.
So what does 'stddev' mean here, exactly? The stddev of multipe runs,
i.e. measured run-to-run variance? Or is it some internal metric of the
benchmark?
Thanks,
Ingo
next prev parent reply other threads:[~2019-04-28 9:33 UTC|newest]
Thread overview: 109+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-23 16:18 [RFC PATCH v2 00/17] Core scheduling v2 Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 01/17] stop_machine: Fix stop_cpus_in_progress ordering Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 02/17] sched: Fix kerneldoc comment for ia64_set_curr_task Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 03/17] sched: Wrap rq::lock access Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 04/17] sched/{rt,deadline}: Fix set_next_task vs pick_next_task Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 05/17] sched: Add task_struct pointer to sched_class::set_curr_task Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 06/17] sched/fair: Export newidle_balance() Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 07/17] sched: Allow put_prev_task() to drop rq->lock Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 08/17] sched: Rework pick_next_task() slow-path Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 09/17] sched: Introduce sched_class::pick_task() Vineeth Remanan Pillai
2019-04-26 14:02 ` Peter Zijlstra
2019-04-26 16:10 ` Vineeth Remanan Pillai
2019-04-29 5:38 ` Aaron Lu
2019-04-23 16:18 ` [RFC PATCH v2 10/17] sched: Core-wide rq->lock Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 11/17] sched: Basic tracking of matching tasks Vineeth Remanan Pillai
2019-04-24 0:08 ` Tim Chen
2019-04-24 20:43 ` Vineeth Remanan Pillai
2019-04-24 22:12 ` Tim Chen
2019-04-25 14:35 ` Phil Auld
2019-05-22 19:52 ` Vineeth Remanan Pillai
2019-04-24 0:17 ` Tim Chen
2019-04-24 20:43 ` Vineeth Remanan Pillai
2019-04-29 3:36 ` Aaron Lu
2019-05-10 13:06 ` Peter Zijlstra
2019-04-29 6:15 ` Aaron Lu
2019-05-01 23:27 ` Tim Chen
2019-05-03 0:06 ` Tim Chen
2019-05-08 15:49 ` Aubrey Li
2019-05-08 18:19 ` Subhra Mazumdar
2019-05-08 18:37 ` Subhra Mazumdar
2019-05-09 0:01 ` Aubrey Li
2019-05-09 0:25 ` Subhra Mazumdar
2019-05-09 1:38 ` Aubrey Li
2019-05-09 2:14 ` Subhra Mazumdar
2019-05-09 15:10 ` Aubrey Li
2019-05-09 17:50 ` Subhra Mazumdar
2019-05-10 0:09 ` Tim Chen
2019-04-23 16:18 ` [RFC PATCH v2 12/17] sched: A quick and dirty cgroup tagging interface Vineeth Remanan Pillai
2019-04-25 14:26 ` Phil Auld
2019-04-26 14:13 ` Peter Zijlstra
2019-04-26 14:19 ` Phil Auld
2019-05-10 15:12 ` Julien Desfossez
2019-04-23 16:18 ` [RFC PATCH v2 13/17] sched: Add core wide task selection and scheduling Vineeth Remanan Pillai
2019-04-29 7:13 ` Aaron Lu
2019-05-18 15:37 ` Aubrey Li
2019-05-20 13:04 ` Phil Auld
2019-05-20 14:04 ` Vineeth Pillai
2019-05-21 8:19 ` Aubrey Li
2019-05-21 13:24 ` Vineeth Pillai
2019-04-23 16:18 ` [RFC PATCH v2 14/17] sched/fair: Add a few assertions Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 15/17] sched: Trivial forced-newidle balancer Vineeth Remanan Pillai
2019-04-23 23:46 ` Aubrey Li
2019-04-24 14:03 ` Vineeth Remanan Pillai
2019-04-24 14:05 ` Vineeth Remanan Pillai
2019-04-23 16:18 ` [RFC PATCH v2 16/17] sched: Wake up sibling if it has something to run Vineeth Remanan Pillai
2019-04-26 15:03 ` Peter Zijlstra
2019-04-29 12:36 ` Julien Desfossez
2019-04-23 16:18 ` [RFC PATCH v2 17/17] sched: Debug bits Vineeth Remanan Pillai
2019-05-17 17:18 ` Aubrey Li
2019-04-23 18:02 ` [RFC PATCH v2 00/17] Core scheduling v2 Phil Auld
2019-04-23 18:45 ` Vineeth Remanan Pillai
2019-04-29 3:53 ` Aaron Lu
2019-05-06 19:39 ` Julien Desfossez
2019-05-08 2:30 ` Aaron Lu
2019-05-08 17:49 ` Julien Desfossez
2019-05-09 2:11 ` Aaron Lu
2019-05-15 21:36 ` Vineeth Remanan Pillai
2019-04-23 23:25 ` Aubrey Li
2019-04-24 11:19 ` Vineeth Remanan Pillai
2019-05-15 21:39 ` Vineeth Remanan Pillai
2019-04-24 13:13 ` Aubrey Li
2019-04-24 14:00 ` Julien Desfossez
2019-04-25 3:15 ` Aubrey Li
2019-04-25 9:55 ` Ingo Molnar
2019-04-25 14:46 ` Mel Gorman
2019-04-25 18:53 ` Ingo Molnar
2019-04-25 18:59 ` Thomas Gleixner
2019-04-25 19:34 ` Ingo Molnar
2019-04-25 21:31 ` Mel Gorman
2019-04-26 8:42 ` Ingo Molnar
2019-04-26 10:43 ` Mel Gorman
2019-04-26 18:37 ` Subhra Mazumdar
2019-04-26 19:49 ` Mel Gorman
2019-04-26 9:45 ` Ingo Molnar
2019-04-26 10:19 ` Mel Gorman
2019-04-27 9:06 ` Ingo Molnar
2019-04-26 9:51 ` Ingo Molnar
2019-04-26 14:15 ` Phil Auld
2019-04-26 2:18 ` Aubrey Li
2019-04-26 9:51 ` Ingo Molnar
2019-04-27 3:51 ` Aubrey Li
2019-04-27 9:17 ` Ingo Molnar
2019-04-27 14:04 ` Aubrey Li
2019-04-27 14:21 ` Ingo Molnar
2019-04-27 15:54 ` Aubrey Li
2019-04-28 9:33 ` Ingo Molnar [this message]
2019-04-28 10:29 ` Aubrey Li
2019-04-28 12:17 ` Ingo Molnar
2019-04-29 2:17 ` Li, Aubrey
2019-04-29 6:14 ` Ingo Molnar
2019-04-29 13:25 ` Li, Aubrey
2019-04-29 15:39 ` Phil Auld
2019-04-30 1:24 ` Aubrey Li
2019-04-29 16:00 ` Ingo Molnar
2019-04-30 1:34 ` Aubrey Li
2019-04-30 4:42 ` Ingo Molnar
2019-05-18 0:58 ` Li, Aubrey
2019-05-18 1:08 ` Li, Aubrey
2019-04-25 14:36 ` Julien Desfossez
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190428093304.GA7393@gmail.com \
--to=mingo@kernel.org \
--cc=aaron.lwe@gmail.com \
--cc=aubrey.intel@gmail.com \
--cc=fweisbec@gmail.com \
--cc=jdesfossez@digitalocean.com \
--cc=keescook@chromium.org \
--cc=kerrnel@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=naravamudan@digitalocean.com \
--cc=pauld@redhat.com \
--cc=pawan.kumar.gupta@linux.intel.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=pjt@google.com \
--cc=subhra.mazumdar@oracle.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=torvalds@linux-foundation.org \
--cc=valentin.schneider@arm.com \
--cc=vpillai@digitalocean.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.