* [PATCH 0/3] zonemode=strided fixes
@ 2019-10-15 14:29 vincentfu
2019-10-15 14:29 ` [PATCH 1/3] io_u: skip to the next zone when zoneskip is set to zero vincentfu
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: vincentfu @ 2019-10-15 14:29 UTC (permalink / raw)
To: axboe, fio; +Cc: Vincent Fu
From: Vincent Fu <vincent.fu@wdc.com>
Jens, please consider this series of patches.
The patches fix two different zonemode=strided issues and also include a
test script.
Vincent
Vincent Fu (3):
io_u: skip to the next zone when zoneskip is set to zero
filesetup: use zonerange for map and LFSR with zonemode=strided
testing: add test script for zonemode=strided
filesetup.c | 3 +
io_u.c | 3 +-
t/strided.py | 350 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 355 insertions(+), 1 deletion(-)
create mode 100755 t/strided.py
--
2.17.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/3] io_u: skip to the next zone when zoneskip is set to zero
2019-10-15 14:29 [PATCH 0/3] zonemode=strided fixes vincentfu
@ 2019-10-15 14:29 ` vincentfu
2019-10-15 14:29 ` [PATCH 2/3] filesetup: use zonerange for map and LFSR with zonemode=strided vincentfu
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: vincentfu @ 2019-10-15 14:29 UTC (permalink / raw)
To: axboe, fio; +Cc: Vincent Fu
From: Vincent Fu <vincent.fu@wdc.com>
If zoneskip is not set or set to 0 in zonemode=strided mode, all IO
occurs in a single zone. If zoneskip is non-zero, the next zone begins
zoneskip bytes beyond the end of the current zone. Thus, it's not
possible to access contiguous zones since zoneskip bytes will separate
the end of one zone and the beginning of the next zone.
This patch allows fio to move to the next zone when zoneskip is
explicitly set to 0, making it possible to issue IO to contiguous zones.
---
io_u.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/io_u.c b/io_u.c
index 94899552..5cbbe85a 100644
--- a/io_u.c
+++ b/io_u.c
@@ -850,7 +850,8 @@ static void setup_strided_zone_mode(struct thread_data *td, struct io_u *io_u)
/*
* See if it's time to switch to a new zone
*/
- if (td->zone_bytes >= td->o.zone_size && td->o.zone_skip) {
+ if (td->zone_bytes >= td->o.zone_size &&
+ fio_option_is_set(&td->o, zone_skip)) {
td->zone_bytes = 0;
f->file_offset += td->o.zone_range + td->o.zone_skip;
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/3] filesetup: use zonerange for map and LFSR with zonemode=strided
2019-10-15 14:29 [PATCH 0/3] zonemode=strided fixes vincentfu
2019-10-15 14:29 ` [PATCH 1/3] io_u: skip to the next zone when zoneskip is set to zero vincentfu
@ 2019-10-15 14:29 ` vincentfu
2019-10-15 14:29 ` [PATCH 3/3] testing: add test script for zonemode=strided vincentfu
2019-10-15 15:15 ` [PATCH 0/3] zonemode=strided fixes Jens Axboe
3 siblings, 0 replies; 5+ messages in thread
From: vincentfu @ 2019-10-15 14:29 UTC (permalink / raw)
To: axboe, fio; +Cc: Vincent Fu
From: Vincent Fu <vincent.fu@wdc.com>
When a random map is enabled, cover only the current zone when run with
zonemode=strided. This ensures that when we reach the end of the zone we
wrap around back into the current zone instead of accessing blocks
outside of the zone.
In addition, when an LFSR is used, constrain the blocks generated to the
current zone. Previously, the LFSR random_generator would ignore the
strided zonemode setting.
Fixes: https://github.com/axboe/fio/issues/809
---
filesetup.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/filesetup.c b/filesetup.c
index a439b6d6..1d3094c1 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -1338,6 +1338,9 @@ bool init_random_map(struct thread_data *td)
for_each_file(td, f, i) {
uint64_t fsize = min(f->real_file_size, f->io_size);
+ if (td->o.zone_mode == ZONE_MODE_STRIDED)
+ fsize = td->o.zone_range;
+
blocks = fsize / (unsigned long long) td->o.rw_min_bs;
if (check_rand_gen_limits(td, f, blocks))
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/3] testing: add test script for zonemode=strided
2019-10-15 14:29 [PATCH 0/3] zonemode=strided fixes vincentfu
2019-10-15 14:29 ` [PATCH 1/3] io_u: skip to the next zone when zoneskip is set to zero vincentfu
2019-10-15 14:29 ` [PATCH 2/3] filesetup: use zonerange for map and LFSR with zonemode=strided vincentfu
@ 2019-10-15 14:29 ` vincentfu
2019-10-15 15:15 ` [PATCH 0/3] zonemode=strided fixes Jens Axboe
3 siblings, 0 replies; 5+ messages in thread
From: vincentfu @ 2019-10-15 14:29 UTC (permalink / raw)
To: axboe, fio; +Cc: Vincent Fu
From: Vincent Fu <vincent.fu@wdc.com>
Python script to run jobs with randommap, with LFSR, and with
nonrandommap. Uses null ioengine or a real file.
---
t/strided.py | 350 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 350 insertions(+)
create mode 100755 t/strided.py
diff --git a/t/strided.py b/t/strided.py
new file mode 100755
index 00000000..47ce5523
--- /dev/null
+++ b/t/strided.py
@@ -0,0 +1,350 @@
+#!/usr/bin/python
+# Note: this script is python2 and python3 compatible.
+#
+# strided.py
+#
+# Test zonemode=strided. This uses the null ioengine when no file is
+# specified. If a file is specified, use it for randdom read testing.
+# Some of the zoneranges in the tests are 16MiB. So when using a file
+# a minimum size of 32MiB is recommended.
+#
+# USAGE
+# python strided.py fio-executable [-f file/device]
+#
+# EXAMPLES
+# python t/strided.py ./fio
+# python t/strided.py ./fio -f /dev/sda
+# dd if=/dev/zero of=temp bs=1M count=32
+# python t/strided.py ./fio -f temp
+#
+# REQUIREMENTS
+# Python 2.6+
+#
+# ===TEST MATRIX===
+#
+# --zonemode=strided, zoneskip >= 0
+# w/ randommap and LFSR
+# zonesize=zonerange all blocks in zonerange touched
+# zonesize>zonerange all blocks touched and roll-over back into zone
+# zonesize<zonerange all blocks inside zone
+#
+# w/o randommap all blocks inside zone
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+import os
+import sys
+import argparse
+import subprocess
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('fio',
+ help='path to fio executable (e.g., ./fio)')
+ parser.add_argument('-f', '--filename', help="file/device to test")
+ args = parser.parse_args()
+
+ return args
+
+
+def run_fio(fio, test, index):
+ filename = "strided"
+ fio_args = [
+ "--name=strided",
+ "--zonemode=strided",
+ "--log_offset=1",
+ "--randrepeat=0",
+ "--rw=randread",
+ "--zoneskip=0",
+ "--write_iops_log={0}{1:03d}".format(filename, index),
+ "--output={0}{1:03d}.out".format(filename, index),
+ "--zonerange={zonerange}".format(**test),
+ "--zonesize={zonesize}".format(**test),
+ "--bs={bs}".format(**test),
+ ]
+ if 'norandommap' in test:
+ fio_args.append('--norandommap')
+ if 'random_generator' in test:
+ fio_args.append('--random_generator={random_generator}'.format(**test))
+ if 'offset' in test:
+ fio_args.append('--offset={offset}'.format(**test))
+ if 'filename' in test:
+ fio_args.append('--filename={filename}'.format(**test))
+ fio_args.append('--filesize={filesize})'.format(**test))
+ else:
+ fio_args.append('--ioengine=null')
+ fio_args.append('--size={size}'.format(**test))
+ fio_args.append('--io_size={io_size}'.format(**test))
+ fio_args.append('--filesize={size})'.format(**test))
+
+ output = subprocess.check_output([fio] + fio_args, universal_newlines=True)
+
+ f = open("{0}{1:03d}_iops.1.log".format(filename, index), "r")
+ log = f.read()
+ f.close()
+
+ return log
+
+
+def check_output(iops_log, test):
+ zonestart = 0 if 'offset' not in test else test['offset']
+ iospersize = test['zonesize'] / test['bs']
+ iosperrange = test['zonerange'] / test['bs']
+ iosperzone = 0
+ lines = iops_log.split('\n')
+ zoneset = set()
+
+ for line in lines:
+ if len(line) == 0:
+ continue
+
+ if iosperzone == iospersize:
+ # time to move to a new zone
+ iosperzone = 0
+ zoneset = set()
+ zonestart += test['zonerange']
+ if zonestart >= test['filesize']:
+ zonestart = 0 if 'offset' not in test else test['offset']
+
+ iosperzone = iosperzone + 1
+ tokens = line.split(',')
+ offset = int(tokens[4])
+ if offset < zonestart or offset >= zonestart + test['zonerange']:
+ print("Offset {0} outside of zone starting at {1}".format(
+ offset, zonestart))
+ return False
+
+ # skip next section if norandommap is enabled with no
+ # random_generator or with a random_generator != lfsr
+ if 'norandommap' in test:
+ if 'random_generator' in test:
+ if test['random_generator'] != 'lfsr':
+ continue
+ else:
+ continue
+
+ # we either have a random map enabled or we
+ # are using an LFSR
+ # so all blocks should be unique and we should have
+ # covered the entire zone when iosperzone % iosperrange == 0
+ block = (offset - zonestart) / test['bs']
+ if block in zoneset:
+ print("Offset {0} in zone already touched".format(offset))
+ return False
+
+ zoneset.add(block)
+ if iosperzone % iosperrange == 0:
+ if len(zoneset) != iosperrange:
+ print("Expected {0} blocks in zone but only saw {1}".format(
+ iosperrange, len(zoneset)))
+ return False
+ zoneset = set()
+
+ return True
+
+
+if __name__ == '__main__':
+ args = parse_args()
+
+ tests = [ # randommap enabled
+ {
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "offset": 8*4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "zonerange": 16*1024*1024,
+ "zonesize": 16*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "zonerange": 4096,
+ "zonesize": 4*4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "zonerange": 16*1024*1024,
+ "zonesize": 32*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "zonerange": 8192,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "zonerange": 16*1024*1024,
+ "zonesize": 8*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ # lfsr
+ {
+ "random_generator": "lfsr",
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "offset": 8*4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 16*1024*1024,
+ "zonesize": 16*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 4096,
+ "zonesize": 4*4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 16*1024*1024,
+ "zonesize": 32*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 8192,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "random_generator": "lfsr",
+ "zonerange": 16*1024*1024,
+ "zonesize": 8*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ # norandommap
+ {
+ "norandommap": 1,
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "offset": 8*4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 4096,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 16*1024*1024,
+ "zonesize": 16*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 4096,
+ "zonesize": 8192,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 16*1024*1024,
+ "zonesize": 32*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 8192,
+ "zonesize": 4096,
+ "bs": 4096,
+ "size": 16*4096,
+ "io_size": 16*4096,
+ },
+ {
+ "norandommap": 1,
+ "zonerange": 16*1024*1024,
+ "zonesize": 8*1024*1024,
+ "bs": 4096,
+ "size": 256*1024*1024,
+ "io_size": 256*1024*204,
+ },
+
+ ]
+
+ index = 1
+ passed = 0
+ failed = 0
+
+ if args.filename:
+ statinfo = os.stat(args.filename)
+ filesize = statinfo.st_size
+ if filesize == 0:
+ f = os.open(args.filename, os.O_RDONLY)
+ filesize = os.lseek(f, 0, os.SEEK_END)
+ os.close(f)
+
+ for test in tests:
+ if args.filename:
+ test['filename'] = args.filename
+ test['filesize'] = filesize
+ else:
+ test['filesize'] = test['size']
+ iops_log = run_fio(args.fio, test, index)
+ status = check_output(iops_log, test)
+ print("Test {0} {1}".format(index, ("PASSED" if status else "FAILED")))
+ if status:
+ passed = passed + 1
+ else:
+ failed = failed + 1
+ index = index + 1
+
+ print("{0} tests passed, {1} failed".format(passed, failed))
+
+ sys.exit(failed)
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] zonemode=strided fixes
2019-10-15 14:29 [PATCH 0/3] zonemode=strided fixes vincentfu
` (2 preceding siblings ...)
2019-10-15 14:29 ` [PATCH 3/3] testing: add test script for zonemode=strided vincentfu
@ 2019-10-15 15:15 ` Jens Axboe
3 siblings, 0 replies; 5+ messages in thread
From: Jens Axboe @ 2019-10-15 15:15 UTC (permalink / raw)
To: vincentfu, fio; +Cc: Vincent Fu
On 10/15/19 8:29 AM, vincentfu@gmail.com wrote:
> From: Vincent Fu <vincent.fu@wdc.com>
>
> Jens, please consider this series of patches.
>
> The patches fix two different zonemode=strided issues and also include a
> test script.
Applied, thanks Vincent.
--
Jens Axboe
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-10-15 15:15 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-15 14:29 [PATCH 0/3] zonemode=strided fixes vincentfu
2019-10-15 14:29 ` [PATCH 1/3] io_u: skip to the next zone when zoneskip is set to zero vincentfu
2019-10-15 14:29 ` [PATCH 2/3] filesetup: use zonerange for map and LFSR with zonemode=strided vincentfu
2019-10-15 14:29 ` [PATCH 3/3] testing: add test script for zonemode=strided vincentfu
2019-10-15 15:15 ` [PATCH 0/3] zonemode=strided fixes Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.