From: Max Reitz <mreitz@redhat.com>
To: qemu-block@nongnu.org
Cc: Kevin Wolf <kwolf@redhat.com>,
Peter Maydell <peter.maydell@linaro.org>,
qemu-devel@nongnu.org, Max Reitz <mreitz@redhat.com>
Subject: [PULL 19/30] scripts/simplebench: use standard deviation for +- error
Date: Fri, 18 Dec 2020 16:12:38 +0100 [thread overview]
Message-ID: <20201218151249.715731-20-mreitz@redhat.com> (raw)
In-Reply-To: <20201218151249.715731-1-mreitz@redhat.com>
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Standard deviation is more usual to see after +- than current maximum
of deviations.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-16-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
scripts/simplebench/simplebench.py | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
index 2251cd34ea..55ec1ad5db 100644
--- a/scripts/simplebench/simplebench.py
+++ b/scripts/simplebench/simplebench.py
@@ -18,6 +18,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
+import statistics
+
def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
"""Benchmark one test-case
@@ -40,7 +42,7 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
'dimension': dimension of results, may be 'seconds' or 'iops'
'average': average value (iops or seconds) per run (exists only if at
least one run succeeded)
- 'delta': maximum delta between test_func result and the average
+ 'stdev': standard deviation of results
(exists only if at least one run succeeded)
'n-failed': number of failed runs (exists only if at least one run
failed)
@@ -67,10 +69,9 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
assert all('seconds' in r for r in succeeded)
assert all('iops' not in r for r in succeeded)
dim = 'seconds'
- avg = sum(r[dim] for r in succeeded) / len(succeeded)
result['dimension'] = dim
- result['average'] = avg
- result['delta'] = max(abs(r[dim] - avg) for r in succeeded)
+ result['average'] = statistics.mean(r[dim] for r in succeeded)
+ result['stdev'] = statistics.stdev(r[dim] for r in succeeded)
if len(succeeded) < count:
result['n-failed'] = count - len(succeeded)
@@ -81,7 +82,7 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
def ascii_one(result):
"""Return ASCII representation of bench_one() returned dict."""
if 'average' in result:
- s = '{:.2f} +- {:.2f}'.format(result['average'], result['delta'])
+ s = '{:.2f} +- {:.2f}'.format(result['average'], result['stdev'])
if 'n-failed' in result:
s += '\n({} failed)'.format(result['n-failed'])
return s
--
2.29.2
next prev parent reply other threads:[~2020-12-18 15:26 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-18 15:12 [PULL 00/30] Block patches Max Reitz
2020-12-18 15:12 ` [PULL 01/30] block: add bdrv_refresh_perms() helper Max Reitz
2020-12-18 15:12 ` [PULL 02/30] block: bdrv_set_perm() drop redundant parameters Max Reitz
2020-12-18 15:12 ` [PULL 03/30] block: bdrv_child_set_perm() " Max Reitz
2020-12-18 15:12 ` [PULL 04/30] block: drop tighten_restrictions Max Reitz
2020-12-18 15:12 ` [PULL 05/30] block: simplify comment to BDRV_REQ_SERIALISING Max Reitz
2020-12-18 15:12 ` [PULL 06/30] block/io.c: drop assertion on double waiting for request serialisation Max Reitz
2020-12-18 15:12 ` [PULL 07/30] block/io: split out bdrv_find_conflicting_request Max Reitz
2020-12-18 15:12 ` [PULL 08/30] block/io: bdrv_wait_serialising_requests_locked: drop extra bs arg Max Reitz
2020-12-18 15:12 ` [PULL 09/30] block: bdrv_mark_request_serialising: split non-waiting function Max Reitz
2020-12-18 15:12 ` [PULL 10/30] block: introduce BDRV_REQ_NO_WAIT flag Max Reitz
2020-12-18 15:12 ` [PULL 11/30] block: bdrv_check_perm(): process children anyway Max Reitz
2020-12-18 15:12 ` [PULL 12/30] block: introduce preallocate filter Max Reitz
2020-12-18 15:12 ` [PULL 13/30] qemu-io: add preallocate mode parameter for truncate command Max Reitz
2020-12-18 15:12 ` [PULL 14/30] iotests: qemu_io_silent: support --image-opts Max Reitz
2020-12-18 15:12 ` [PULL 15/30] iotests.py: execute_setup_common(): add required_fmts argument Max Reitz
2020-12-18 15:12 ` [PULL 16/30] iotests: add 298 to test new preallocate filter driver Max Reitz
2020-12-18 15:12 ` [PULL 17/30] scripts/simplebench: fix grammar: s/successed/succeeded/ Max Reitz
2020-12-18 15:12 ` [PULL 18/30] scripts/simplebench: support iops Max Reitz
2020-12-18 15:12 ` Max Reitz [this message]
2020-12-18 15:12 ` [PULL 20/30] simplebench: rename ascii() to results_to_text() Max Reitz
2020-12-18 15:12 ` [PULL 21/30] simplebench: move results_to_text() into separate file Max Reitz
2020-12-18 15:12 ` [PULL 22/30] simplebench/results_to_text: improve view of the table Max Reitz
2020-12-18 15:12 ` [PULL 23/30] simplebench/results_to_text: add difference line to " Max Reitz
2020-12-18 15:12 ` [PULL 24/30] simplebench/results_to_text: make executable Max Reitz
2020-12-18 15:12 ` [PULL 25/30] scripts/simplebench: add bench_prealloc.py Max Reitz
2020-12-18 15:12 ` [PULL 26/30] quorum: Implement bdrv_co_block_status() Max Reitz
2020-12-18 15:12 ` [PULL 27/30] quorum: Implement bdrv_co_pwrite_zeroes() Max Reitz
2020-12-18 15:12 ` [PULL 28/30] block/nvme: Implement fake truncate() coroutine Max Reitz
2020-12-18 15:12 ` [PULL 29/30] iotests/102: Pass $QEMU_HANDLE to _send_qemu_cmd Max Reitz
2020-12-18 15:12 ` [PULL 30/30] iotests: Fix _send_qemu_cmd with bash 5.1 Max Reitz
2021-01-01 12:53 ` [PULL 00/30] Block patches Peter Maydell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201218151249.715731-20-mreitz@redhat.com \
--to=mreitz@redhat.com \
--cc=kwolf@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).