* [PULL 0/9] scripts/simplebench patches
@ 2021-05-04 9:01 Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 1/9] simplebench: bench_one(): add slow_limit argument Vladimir Sementsov-Ogievskiy
` (11 more replies)
0 siblings, 12 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
are available in the Git repository at:
https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
----------------------------------------------------------------
scripts/simplebench improvements for 2021-05-04
----------------------------------------------------------------
Vladimir Sementsov-Ogievskiy (9):
simplebench: bench_one(): add slow_limit argument
simplebench: bench_one(): support count=1
simplebench/bench-backup: add --compressed option
simplebench/bench-backup: add target-cache argument
simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED
simplebench/bench-backup: support qcow2 source files
simplebench/bench-backup: add --count and --no-initial-run
simplebench/bench-backup: add --drop-caches argument
MAINTAINERS: update Benchmark util: add git tree
MAINTAINERS | 1 +
scripts/simplebench/bench-backup.py | 95 ++++++++++++++++++++++++++++------
scripts/simplebench/bench_block_job.py | 42 +++++++++++++--
scripts/simplebench/simplebench.py | 28 +++++++++-
4 files changed, 144 insertions(+), 22 deletions(-)
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PULL 1/9] simplebench: bench_one(): add slow_limit argument
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 2/9] simplebench: bench_one(): support count=1 Vladimir Sementsov-Ogievskiy
` (10 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
Sometimes one of cells in a testing table runs too slow. And we really
don't want to wait so long. Limit number of runs in this case.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
scripts/simplebench/simplebench.py | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
index f61513af90..0a3035732c 100644
--- a/scripts/simplebench/simplebench.py
+++ b/scripts/simplebench/simplebench.py
@@ -19,9 +19,11 @@
#
import statistics
+import time
-def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
+def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
+ slow_limit=100):
"""Benchmark one test-case
test_func -- benchmarking function with prototype
@@ -36,6 +38,8 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
test_case -- test case - opaque second argument for test_func
count -- how many times to call test_func, to calculate average
initial_run -- do initial run of test_func, which don't get into result
+ slow_limit -- stop at slow run (that exceedes the slow_limit by seconds).
+ (initial run is not measured)
Returns dict with the following fields:
'runs': list of test_func results
@@ -53,11 +57,19 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
runs = []
for i in range(count):
+ t = time.time()
+
print(' #run {}'.format(i+1))
res = test_func(test_env, test_case)
print(' ', res)
runs.append(res)
+ if time.time() - t > slow_limit:
+ print(' - run is too slow, stop here')
+ break
+
+ count = len(runs)
+
result = {'runs': runs}
succeeded = [r for r in runs if ('seconds' in r or 'iops' in r)]
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 2/9] simplebench: bench_one(): support count=1
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 1/9] simplebench: bench_one(): add slow_limit argument Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 3/9] simplebench/bench-backup: add --compressed option Vladimir Sementsov-Ogievskiy
` (9 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell, John Snow
statistics.stdev raises if sequence length is less than two. Support
that case by hand.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
scripts/simplebench/simplebench.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
index 0a3035732c..27bc4d4715 100644
--- a/scripts/simplebench/simplebench.py
+++ b/scripts/simplebench/simplebench.py
@@ -83,7 +83,10 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
dim = 'seconds'
result['dimension'] = dim
result['average'] = statistics.mean(r[dim] for r in succeeded)
- result['stdev'] = statistics.stdev(r[dim] for r in succeeded)
+ if len(succeeded) == 1:
+ result['stdev'] = 0
+ else:
+ result['stdev'] = statistics.stdev(r[dim] for r in succeeded)
if len(succeeded) < count:
result['n-failed'] = count - len(succeeded)
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 3/9] simplebench/bench-backup: add --compressed option
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 1/9] simplebench: bench_one(): add slow_limit argument Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 2/9] simplebench: bench_one(): support count=1 Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 4/9] simplebench/bench-backup: add target-cache argument Vladimir Sementsov-Ogievskiy
` (8 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
Allow bench compressed backup.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
scripts/simplebench/bench-backup.py | 55 ++++++++++++++++++--------
scripts/simplebench/bench_block_job.py | 23 +++++++++++
2 files changed, 62 insertions(+), 16 deletions(-)
diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py
index 33a1ecfefa..72eae85bb1 100755
--- a/scripts/simplebench/bench-backup.py
+++ b/scripts/simplebench/bench-backup.py
@@ -23,7 +23,7 @@
import simplebench
from results_to_text import results_to_text
-from bench_block_job import bench_block_copy, drv_file, drv_nbd
+from bench_block_job import bench_block_copy, drv_file, drv_nbd, drv_qcow2
def bench_func(env, case):
@@ -37,29 +37,41 @@ def bench_func(env, case):
def bench(args):
test_cases = []
- sources = {}
- targets = {}
- for d in args.dir:
- label, path = d.split(':') # paths with colon not supported
- sources[label] = drv_file(path + '/test-source')
- targets[label] = drv_file(path + '/test-target')
+ # paths with colon not supported, so we just split by ':'
+ dirs = dict(d.split(':') for d in args.dir)
+ nbd_drv = None
if args.nbd:
nbd = args.nbd.split(':')
host = nbd[0]
port = '10809' if len(nbd) == 1 else nbd[1]
- drv = drv_nbd(host, port)
- sources['nbd'] = drv
- targets['nbd'] = drv
+ nbd_drv = drv_nbd(host, port)
for t in args.test:
src, dst = t.split(':')
- test_cases.append({
- 'id': t,
- 'source': sources[src],
- 'target': targets[dst]
- })
+ if src == 'nbd' and dst == 'nbd':
+ raise ValueError("Can't use 'nbd' label for both src and dst")
+
+ if (src == 'nbd' or dst == 'nbd') and not nbd_drv:
+ raise ValueError("'nbd' label used but --nbd is not given")
+
+ if src == 'nbd':
+ source = nbd_drv
+ else:
+ source = drv_file(dirs[src] + '/test-source')
+
+ if dst == 'nbd':
+ test_cases.append({'id': t, 'source': source, 'target': nbd_drv})
+ continue
+
+ fname = dirs[dst] + '/test-target'
+ if args.compressed:
+ fname += '.qcow2'
+ target = drv_file(fname)
+ if args.compressed:
+ target = drv_qcow2(target)
+ test_cases.append({'id': t, 'source': source, 'target': target})
binaries = [] # list of (<label>, <path>, [<options>])
for i, q in enumerate(args.env):
@@ -106,6 +118,13 @@ def bench(args):
elif opt.startswith('max-workers='):
x_perf['max-workers'] = int(opt.split('=')[1])
+ backup_options = {}
+ if x_perf:
+ backup_options['x-perf'] = x_perf
+
+ if args.compressed:
+ backup_options['compress'] = True
+
if is_mirror:
assert not x_perf
test_envs.append({
@@ -117,7 +136,7 @@ def bench(args):
test_envs.append({
'id': f'backup({label})\n' + '\n'.join(opts),
'cmd': 'blockdev-backup',
- 'cmd-options': {'x-perf': x_perf} if x_perf else {},
+ 'cmd-options': backup_options,
'qemu-binary': path
})
@@ -163,5 +182,9 @@ def __call__(self, parser, namespace, values, option_string=None):
p.add_argument('--test', nargs='+', help='''\
Tests, in form source-dir-label:target-dir-label''',
action=ExtendAction)
+ p.add_argument('--compressed', help='''\
+Use compressed backup. It automatically means
+automatically creating qcow2 target with
+lazy_refcounts for each test run''', action='store_true')
bench(p.parse_args())
diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py
index 7332845c1c..08f86ed9c1 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -21,6 +21,7 @@
import sys
import os
+import subprocess
import socket
import json
@@ -77,11 +78,29 @@ def bench_block_job(cmd, cmd_args, qemu_args):
return {'seconds': (end_ms - start_ms) / 1000000.0}
+def get_image_size(path):
+ out = subprocess.run(['qemu-img', 'info', '--out=json', path],
+ stdout=subprocess.PIPE, check=True).stdout
+ return json.loads(out)['virtual-size']
+
+
# Bench backup or mirror
def bench_block_copy(qemu_binary, cmd, cmd_options, source, target):
"""Helper to run bench_block_job() for mirror or backup"""
assert cmd in ('blockdev-backup', 'blockdev-mirror')
+ if target['driver'] == 'qcow2':
+ try:
+ os.remove(target['file']['filename'])
+ except OSError:
+ pass
+
+ subprocess.run(['qemu-img', 'create', '-f', 'qcow2',
+ target['file']['filename'],
+ str(get_image_size(source['filename']))],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL, check=True)
+
source['node-name'] = 'source'
target['node-name'] = 'target'
@@ -106,6 +125,10 @@ def drv_nbd(host, port):
'server': {'type': 'inet', 'host': host, 'port': port}}
+def drv_qcow2(file):
+ return {'driver': 'qcow2', 'file': file}
+
+
if __name__ == '__main__':
import sys
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 4/9] simplebench/bench-backup: add target-cache argument
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2021-05-04 9:01 ` [PULL 3/9] simplebench/bench-backup: add --compressed option Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 5/9] simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell, John Snow
Allow benchmark with different kinds of target cache.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
scripts/simplebench/bench-backup.py | 33 ++++++++++++++++++++------
scripts/simplebench/bench_block_job.py | 10 +++++---
2 files changed, 33 insertions(+), 10 deletions(-)
diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py
index 72eae85bb1..fbc85f266f 100755
--- a/scripts/simplebench/bench-backup.py
+++ b/scripts/simplebench/bench-backup.py
@@ -65,13 +65,26 @@ def bench(args):
test_cases.append({'id': t, 'source': source, 'target': nbd_drv})
continue
- fname = dirs[dst] + '/test-target'
- if args.compressed:
- fname += '.qcow2'
- target = drv_file(fname)
- if args.compressed:
- target = drv_qcow2(target)
- test_cases.append({'id': t, 'source': source, 'target': target})
+ if args.target_cache == 'both':
+ target_caches = ['direct', 'cached']
+ else:
+ target_caches = [args.target_cache]
+
+ for c in target_caches:
+ o_direct = c == 'direct'
+ fname = dirs[dst] + '/test-target'
+ if args.compressed:
+ fname += '.qcow2'
+ target = drv_file(fname, o_direct=o_direct)
+ if args.compressed:
+ target = drv_qcow2(target)
+
+ test_id = t
+ if args.target_cache == 'both':
+ test_id += f'({c})'
+
+ test_cases.append({'id': test_id, 'source': source,
+ 'target': target})
binaries = [] # list of (<label>, <path>, [<options>])
for i, q in enumerate(args.env):
@@ -186,5 +199,11 @@ def __call__(self, parser, namespace, values, option_string=None):
Use compressed backup. It automatically means
automatically creating qcow2 target with
lazy_refcounts for each test run''', action='store_true')
+ p.add_argument('--target-cache', help='''\
+Setup cache for target nodes. Options:
+ direct: default, use O_DIRECT and aio=native
+ cached: use system cache (Qemu default) and aio=threads (Qemu default)
+ both: generate two test cases for each src:dst pair''',
+ default='direct', choices=('direct', 'cached', 'both'))
bench(p.parse_args())
diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py
index 08f86ed9c1..8f8385ccce 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -115,9 +115,13 @@ def bench_block_copy(qemu_binary, cmd, cmd_options, source, target):
'-blockdev', json.dumps(target)])
-def drv_file(filename):
- return {'driver': 'file', 'filename': filename,
- 'cache': {'direct': True}, 'aio': 'native'}
+def drv_file(filename, o_direct=True):
+ node = {'driver': 'file', 'filename': filename}
+ if o_direct:
+ node['cache'] = {'direct': True}
+ node['aio'] = 'native'
+
+ return node
def drv_nbd(host, port):
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 5/9] simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2021-05-04 9:01 ` [PULL 4/9] simplebench/bench-backup: add target-cache argument Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 6/9] simplebench/bench-backup: support qcow2 source files Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell, John Snow
We should not report success if there is an error in final event.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
scripts/simplebench/bench_block_job.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py
index 8f8385ccce..71d2e489c8 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -70,6 +70,10 @@ def bench_block_job(cmd, cmd_args, qemu_args):
vm.shutdown()
return {'error': 'block-job failed: ' + str(e),
'vm-log': vm.get_log()}
+ if 'error' in e['data']:
+ vm.shutdown()
+ return {'error': 'block-job failed: ' + e['data']['error'],
+ 'vm-log': vm.get_log()}
end_ms = e['timestamp']['seconds'] * 1000000 + \
e['timestamp']['microseconds']
finally:
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 6/9] simplebench/bench-backup: support qcow2 source files
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2021-05-04 9:01 ` [PULL 5/9] simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 7/9] simplebench/bench-backup: add --count and --no-initial-run Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell, John Snow
Add support for qcow2 source. New option says to use test-source.qcow2
instead of test-source. Of course, test-source.qcow2 should be
precreated.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
scripts/simplebench/bench-backup.py | 5 +++++
scripts/simplebench/bench_block_job.py | 7 ++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py
index fbc85f266f..a2120fcbf0 100755
--- a/scripts/simplebench/bench-backup.py
+++ b/scripts/simplebench/bench-backup.py
@@ -58,6 +58,8 @@ def bench(args):
if src == 'nbd':
source = nbd_drv
+ elif args.qcow2_sources:
+ source = drv_qcow2(drv_file(dirs[src] + '/test-source.qcow2'))
else:
source = drv_file(dirs[src] + '/test-source')
@@ -199,6 +201,9 @@ def __call__(self, parser, namespace, values, option_string=None):
Use compressed backup. It automatically means
automatically creating qcow2 target with
lazy_refcounts for each test run''', action='store_true')
+ p.add_argument('--qcow2-sources', help='''\
+Use test-source.qcow2 images as sources instead of
+test-source raw images''', action='store_true')
p.add_argument('--target-cache', help='''\
Setup cache for target nodes. Options:
direct: default, use O_DIRECT and aio=native
diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py
index 71d2e489c8..4f03c12169 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -88,6 +88,11 @@ def get_image_size(path):
return json.loads(out)['virtual-size']
+def get_blockdev_size(obj):
+ img = obj['filename'] if 'filename' in obj else obj['file']['filename']
+ return get_image_size(img)
+
+
# Bench backup or mirror
def bench_block_copy(qemu_binary, cmd, cmd_options, source, target):
"""Helper to run bench_block_job() for mirror or backup"""
@@ -101,7 +106,7 @@ def bench_block_copy(qemu_binary, cmd, cmd_options, source, target):
subprocess.run(['qemu-img', 'create', '-f', 'qcow2',
target['file']['filename'],
- str(get_image_size(source['filename']))],
+ str(get_blockdev_size(source))],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, check=True)
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 7/9] simplebench/bench-backup: add --count and --no-initial-run
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2021-05-04 9:01 ` [PULL 6/9] simplebench/bench-backup: support qcow2 source files Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 8/9] simplebench/bench-backup: add --drop-caches argument Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
Add arguments to set number of test runs per table cell and to disable
initial run that is not counted in results.
It's convenient to set --count 1 --no-initial-run to fast run test
onece, and to set --count to some large enough number for good
precision of the results.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
scripts/simplebench/bench-backup.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py
index a2120fcbf0..092fed5816 100755
--- a/scripts/simplebench/bench-backup.py
+++ b/scripts/simplebench/bench-backup.py
@@ -155,7 +155,8 @@ def bench(args):
'qemu-binary': path
})
- result = simplebench.bench(bench_func, test_envs, test_cases, count=3)
+ result = simplebench.bench(bench_func, test_envs, test_cases,
+ count=args.count, initial_run=args.initial_run)
with open('results.json', 'w') as f:
json.dump(result, f, indent=4)
print(results_to_text(result))
@@ -211,4 +212,13 @@ def __call__(self, parser, namespace, values, option_string=None):
both: generate two test cases for each src:dst pair''',
default='direct', choices=('direct', 'cached', 'both'))
+ p.add_argument('--count', type=int, default=3, help='''\
+Number of test runs per table cell''')
+
+ # BooleanOptionalAction helps to support --no-initial-run option
+ p.add_argument('--initial-run', action=argparse.BooleanOptionalAction,
+ help='''\
+Do additional initial run per cell which doesn't count in result,
+default true''')
+
bench(p.parse_args())
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 8/9] simplebench/bench-backup: add --drop-caches argument
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2021-05-04 9:01 ` [PULL 7/9] simplebench/bench-backup: add --count and --no-initial-run Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 9/9] MAINTAINERS: update Benchmark util: add git tree Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
Add an option to drop caches before each test run. It may probably
improve reliability of results when testing in cached mode.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
scripts/simplebench/bench-backup.py | 6 +++++-
scripts/simplebench/simplebench.py | 11 ++++++++++-
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py
index 092fed5816..5a0675c593 100755
--- a/scripts/simplebench/bench-backup.py
+++ b/scripts/simplebench/bench-backup.py
@@ -156,7 +156,8 @@ def bench(args):
})
result = simplebench.bench(bench_func, test_envs, test_cases,
- count=args.count, initial_run=args.initial_run)
+ count=args.count, initial_run=args.initial_run,
+ drop_caches=args.drop_caches)
with open('results.json', 'w') as f:
json.dump(result, f, indent=4)
print(results_to_text(result))
@@ -221,4 +222,7 @@ def __call__(self, parser, namespace, values, option_string=None):
Do additional initial run per cell which doesn't count in result,
default true''')
+ p.add_argument('--drop-caches', action='store_true', help='''\
+Do "sync; echo 3 > /proc/sys/vm/drop_caches" before each test run''')
+
bench(p.parse_args())
diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
index 27bc4d4715..8efca2af98 100644
--- a/scripts/simplebench/simplebench.py
+++ b/scripts/simplebench/simplebench.py
@@ -19,11 +19,17 @@
#
import statistics
+import subprocess
import time
+def do_drop_caches():
+ subprocess.run('sync; echo 3 > /proc/sys/vm/drop_caches', shell=True,
+ check=True)
+
+
def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
- slow_limit=100):
+ slow_limit=100, drop_caches=False):
"""Benchmark one test-case
test_func -- benchmarking function with prototype
@@ -40,6 +46,7 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
initial_run -- do initial run of test_func, which don't get into result
slow_limit -- stop at slow run (that exceedes the slow_limit by seconds).
(initial run is not measured)
+ drop_caches -- drop caches before each run
Returns dict with the following fields:
'runs': list of test_func results
@@ -53,6 +60,7 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
"""
if initial_run:
print(' #initial run:')
+ do_drop_caches()
print(' ', test_func(test_env, test_case))
runs = []
@@ -60,6 +68,7 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
t = time.time()
print(' #run {}'.format(i+1))
+ do_drop_caches()
res = test_func(test_env, test_case)
print(' ', res)
runs.append(res)
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PULL 9/9] MAINTAINERS: update Benchmark util: add git tree
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2021-05-04 9:01 ` [PULL 8/9] simplebench/bench-backup: add --drop-caches argument Vladimir Sementsov-Ogievskiy
@ 2021-05-04 9:01 ` Vladimir Sementsov-Ogievskiy
2021-05-11 5:59 ` [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
11 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-04 9:01 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, vsementsov, peter.maydell
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 4c05ff8bba..f9f2acea8f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2531,6 +2531,7 @@ Benchmark util
M: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
S: Maintained
F: scripts/simplebench/
+T: git https://src.openvz.org/scm/~vsementsov/qemu.git simplebench
Transactions helper
M: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
--
2.29.2
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (8 preceding siblings ...)
2021-05-04 9:01 ` [PULL 9/9] MAINTAINERS: update Benchmark util: add git tree Vladimir Sementsov-Ogievskiy
@ 2021-05-11 5:59 ` Vladimir Sementsov-Ogievskiy
2021-05-11 7:39 ` Peter Maydell
2021-05-20 19:17 ` Peter Maydell
2021-05-21 12:45 ` Peter Maydell
11 siblings, 1 reply; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-11 5:59 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, peter.maydell
Hi!
Kindly ping, or what's wrong with it?
04.05.2021 12:01, Vladimir Sementsov-Ogievskiy wrote:
> The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
>
> Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
>
> are available in the Git repository at:
>
> https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
>
> for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
>
> MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
>
> ----------------------------------------------------------------
> scripts/simplebench improvements for 2021-05-04
>
> ----------------------------------------------------------------
> Vladimir Sementsov-Ogievskiy (9):
> simplebench: bench_one(): add slow_limit argument
> simplebench: bench_one(): support count=1
> simplebench/bench-backup: add --compressed option
> simplebench/bench-backup: add target-cache argument
> simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED
> simplebench/bench-backup: support qcow2 source files
> simplebench/bench-backup: add --count and --no-initial-run
> simplebench/bench-backup: add --drop-caches argument
> MAINTAINERS: update Benchmark util: add git tree
>
> MAINTAINERS | 1 +
> scripts/simplebench/bench-backup.py | 95 ++++++++++++++++++++++++++++------
> scripts/simplebench/bench_block_job.py | 42 +++++++++++++--
> scripts/simplebench/simplebench.py | 28 +++++++++-
> 4 files changed, 144 insertions(+), 22 deletions(-)
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-11 5:59 ` [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
@ 2021-05-11 7:39 ` Peter Maydell
2021-05-11 10:02 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 17+ messages in thread
From: Peter Maydell @ 2021-05-11 7:39 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: QEMU Developers, Qemu-block
On Tue, 11 May 2021 at 06:59, Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
>
> Hi!
>
> Kindly ping, or what's wrong with it?
You've never sent me a pullreq before. Pull requests from new
people are a pain and take more time to deal with, so I only
look at them when I have the time to do that. Pull requests
from established submaintainers are easy because I know they
know the process and they have their gpg key set up and so on.
-- PMM
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-11 7:39 ` Peter Maydell
@ 2021-05-11 10:02 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-11 10:02 UTC (permalink / raw)
To: Peter Maydell; +Cc: Qemu-block, QEMU Developers
11.05.2021 10:39, Peter Maydell wrote:
> On Tue, 11 May 2021 at 06:59, Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
>>
>> Hi!
>>
>> Kindly ping, or what's wrong with it?
>
> You've never sent me a pullreq before. Pull requests from new
> people are a pain and take more time to deal with, so I only
> look at them when I have the time to do that. Pull requests
> from established submaintainers are easy because I know they
> know the process and they have their gpg key set up and so on.
>
OK, I'll wait)
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (9 preceding siblings ...)
2021-05-11 5:59 ` [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
@ 2021-05-20 19:17 ` Peter Maydell
2021-05-20 19:23 ` Vladimir Sementsov-Ogievskiy
2021-05-21 12:45 ` Peter Maydell
11 siblings, 1 reply; 17+ messages in thread
From: Peter Maydell @ 2021-05-20 19:17 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: QEMU Developers, Qemu-block
On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
>
> The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
>
> Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
>
> are available in the Git repository at:
>
> https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
>
> for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
>
> MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
>
> ----------------------------------------------------------------
> scripts/simplebench improvements for 2021-05-04
>
I couldn't find the gpg key you signed this with on the public
keyserver. Could you point me at where you uploaded it, please?
thanks
-- PMM
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-20 19:17 ` Peter Maydell
@ 2021-05-20 19:23 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-20 19:23 UTC (permalink / raw)
To: Peter Maydell; +Cc: Qemu-block, QEMU Developers
20.05.2021 22:17, Peter Maydell wrote:
> On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
>>
>> The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
>>
>> Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
>>
>> are available in the Git repository at:
>>
>> https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
>>
>> for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
>>
>> MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
>>
>> ----------------------------------------------------------------
>> scripts/simplebench improvements for 2021-05-04
>>
>
> I couldn't find the gpg key you signed this with on the public
> keyserver. Could you point me at where you uploaded it, please?
>
Here it is: http://keys.gnupg.net/pks/lookup?op=vindex&fingerprint=on&search=0x561F24C1F19F79FB
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
` (10 preceding siblings ...)
2021-05-20 19:17 ` Peter Maydell
@ 2021-05-21 12:45 ` Peter Maydell
2021-05-21 13:04 ` Vladimir Sementsov-Ogievskiy
11 siblings, 1 reply; 17+ messages in thread
From: Peter Maydell @ 2021-05-21 12:45 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: QEMU Developers, Qemu-block
On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
>
> The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
>
> Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
>
> are available in the Git repository at:
>
> https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
>
> for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
>
> MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
>
> ----------------------------------------------------------------
> scripts/simplebench improvements for 2021-05-04
>
> ----------------------------------------------------------------
> Vladimir Sementsov-Ogievskiy (9):
> simplebench: bench_one(): add slow_limit argument
> simplebench: bench_one(): support count=1
> simplebench/bench-backup: add --compressed option
> simplebench/bench-backup: add target-cache argument
> simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED
> simplebench/bench-backup: support qcow2 source files
> simplebench/bench-backup: add --count and --no-initial-run
> simplebench/bench-backup: add --drop-caches argument
> MAINTAINERS: update Benchmark util: add git tree
Applied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/6.1
for any user-visible changes.
-- PMM
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PULL 0/9] scripts/simplebench patches
2021-05-21 12:45 ` Peter Maydell
@ 2021-05-21 13:04 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 17+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 13:04 UTC (permalink / raw)
To: Peter Maydell; +Cc: Qemu-block, QEMU Developers
21.05.2021 15:45, Peter Maydell wrote:
> On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
>>
>> The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf:
>>
>> Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46 +0100)
>>
>> are available in the Git repository at:
>>
>> https://src.openvz.org/scm/~vsementsov/qemu.git tags/pull-simplebench-2021-05-04
>>
>> for you to fetch changes up to e34bd02694026722410b80cee02ab7f33f893e9b:
>>
>> MAINTAINERS: update Benchmark util: add git tree (2021-05-04 11:37:26 +0300)
>>
>> ----------------------------------------------------------------
>> scripts/simplebench improvements for 2021-05-04
>>
>> ----------------------------------------------------------------
>> Vladimir Sementsov-Ogievskiy (9):
>> simplebench: bench_one(): add slow_limit argument
>> simplebench: bench_one(): support count=1
>> simplebench/bench-backup: add --compressed option
>> simplebench/bench-backup: add target-cache argument
>> simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED
>> simplebench/bench-backup: support qcow2 source files
>> simplebench/bench-backup: add --count and --no-initial-run
>> simplebench/bench-backup: add --drop-caches argument
>> MAINTAINERS: update Benchmark util: add git tree
>
>
>
> Applied, thanks.
>
Thanks a lot for your time!
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2021-05-21 13:08 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-04 9:01 [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 1/9] simplebench: bench_one(): add slow_limit argument Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 2/9] simplebench: bench_one(): support count=1 Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 3/9] simplebench/bench-backup: add --compressed option Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 4/9] simplebench/bench-backup: add target-cache argument Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 5/9] simplebench/bench_block_job: handle error in BLOCK_JOB_COMPLETED Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 6/9] simplebench/bench-backup: support qcow2 source files Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 7/9] simplebench/bench-backup: add --count and --no-initial-run Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 8/9] simplebench/bench-backup: add --drop-caches argument Vladimir Sementsov-Ogievskiy
2021-05-04 9:01 ` [PULL 9/9] MAINTAINERS: update Benchmark util: add git tree Vladimir Sementsov-Ogievskiy
2021-05-11 5:59 ` [PULL 0/9] scripts/simplebench patches Vladimir Sementsov-Ogievskiy
2021-05-11 7:39 ` Peter Maydell
2021-05-11 10:02 ` Vladimir Sementsov-Ogievskiy
2021-05-20 19:17 ` Peter Maydell
2021-05-20 19:23 ` Vladimir Sementsov-Ogievskiy
2021-05-21 12:45 ` Peter Maydell
2021-05-21 13:04 ` Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.