[PATCH 0/9] testing patches

[PATCH 0/9] testing patches

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

Jens, please consider this series of patches related to testing.

The patches improve t/run-fio-tests.py in various ways, most prominently
adding support for Windows and macOS.

Also included are travis and appveyor patches that add run-fio-tests.py
as a step.  Currently both the travis and appveyor build processes
complete in less than four minutes.  Adding run-fio-tests.py increases
this to about 20 minutes for travis and 14 minutes for appveyor. 

Also worth noting is that sometimes the VMs these builds run on appear
to become overloaded and produce false failures. For instance, builds on
the same hash fail here

https://ci.appveyor.com/project/vincentkfu/fio/builds/29342164

but succeed here

https://ci.appveyor.com/project/vincentkfu/fio/builds/29344271

The failure occured because of a timeout running a null ioengine job that
reads 10G of data.

It's certainly nice to have these tests automatically run with every
commit, but the downside is that the build process is significantly
lengthened and some noise is introduced.


Vincent Fu (9):
  .gitignore: ignore zbd test output files
  t/run-fio-tests: a few small improvements
  t/run-fio-tests: detect requirements and skip tests accordingly
  t/run-fio-tests: improve Windows support
  t/run-fio-tests: identify test id for debug messages
  t/steadystate_tests: use null ioengine for tests
  .travis.yml: run t/run-fio.tests.py as part of build
  .appveyor.yml: run run-fio-tests.py
  t/run-fio-tests: relax acceptance criterion for t0011

 .appveyor.yml          |   6 +-
 .gitignore             |   1 +
 .travis.yml            |  26 ++---
 t/run-fio-tests.py     | 219 ++++++++++++++++++++++++++++++++++++-----
 t/steadystate_tests.py |  21 +---
 5 files changed, 219 insertions(+), 54 deletions(-)

-- 
2.17.1

[PATCH 1/9] .gitignore: ignore zbd test output files

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index f86bec64..b228938d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,3 +25,4 @@ lex.yy.c
 doc/output
 /tags
 /TAGS
+/t/zbd/test-zbd-support.log.*
-- 
2.17.1

[PATCH 2/9] t/run-fio-tests: a few small improvements

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

- add debug option that changes logging level
- improve error handling
- more output on test failures
- Python 3.5 compatibility with pathlib.Path and os.path.join
- warn if fio executable not found but don't abort because some tests
can run without the fio executable
---
 t/run-fio-tests.py | 43 +++++++++++++++++++++++++++++++------------
 1 file changed, 31 insertions(+), 12 deletions(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index cf8de093..72314505 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -46,6 +46,7 @@ import os
 import sys
 import json
 import time
+import shutil
 import logging
 import argparse
 import subprocess
@@ -177,8 +178,8 @@ class FioExeTest(FioTest):
                     self.failure_reason = "{0} zero return code,".format(self.failure_reason)
                     self.passed = False
 
+        stderr_size = os.path.getsize(self.stderr_file)
         if 'stderr_empty' in self.success:
-            stderr_size = os.path.getsize(self.stderr_file)
             if self.success['stderr_empty']:
                 if stderr_size != 0:
                     self.failure_reason = "{0} stderr not empty,".format(self.failure_reason)
@@ -262,15 +263,22 @@ class FioJobTest(FioExeTest):
 
         super(FioJobTest, self).check_result()
 
+        if not self.passed:
+            return
+
         if 'json' in self.output_format:
-            output_file = open(os.path.join(self.test_dir, self.fio_output), "r")
-            file_data = output_file.read()
-            output_file.close()
             try:
-                self.json_data = json.loads(file_data)
-            except json.JSONDecodeError:
-                self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
+                with open(os.path.join(self.test_dir, self.fio_output), "r") as output_file:
+                    file_data = output_file.read()
+            except EnvironmentError:
+                self.failure_reason = "{0} unable to open output file,".format(self.failure_reason)
                 self.passed = False
+            else:
+                try:
+                    self.json_data = json.loads(file_data)
+                except json.JSONDecodeError:
+                    self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
+                    self.passed = False
 
 
 class FioJobTest_t0005(FioJobTest):
@@ -587,26 +595,33 @@ def parse_args():
                         help='list of test(s) to skip')
     parser.add_argument('-o', '--run-only', nargs='+', type=int,
                         help='list of test(s) to run, skipping all others')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='provide debug output')
     args = parser.parse_args()
 
     return args
 
 
 def main():
-    logging.basicConfig(level=logging.INFO)
-
     args = parse_args()
+    if args.debug:
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.INFO)
+
     if args.fio_root:
         fio_root = args.fio_root
     else:
-        fio_root = Path(__file__).absolute().parent.parent
-    logging.debug("fio_root: %s" % fio_root)
+        fio_root = str(Path(__file__).absolute().parent.parent)
+    print("fio root is %s" % fio_root)
 
     if args.fio:
         fio_path = args.fio
     else:
         fio_path = os.path.join(fio_root, "fio")
-    logging.debug("fio_path: %s" % fio_path)
+    print("fio path is %s" % fio_path)
+    if not shutil.which(fio_path):
+        print("Warning: fio executable not found")
 
     artifact_root = args.artifact_root if args.artifact_root else \
         "fio-test-{0}".format(time.strftime("%Y%m%d-%H%M%S"))
@@ -667,6 +682,10 @@ def main():
         else:
             result = "FAILED: {0}".format(test.failure_reason)
             failed = failed + 1
+            with open(test.stderr_file, "r") as stderr_file:
+                logging.debug("stderr:\n%s" % stderr_file.read())
+            with open(test.stdout_file, "r") as stdout_file:
+                logging.debug("stdout:\n%s" % stdout_file.read())
         print("Test {0} {1}".format(config['test_id'], result))
 
     print("{0} test(s) passed, {1} failed, {2} skipped".format(passed, failed, skipped))
-- 
2.17.1

[PATCH 3/9] t/run-fio-tests: detect requirements and skip tests accordingly

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

Also provide an option to skip requirement checking

Suggested-by: <shinichiro.kawasaki@wdc.com>
---
 t/run-fio-tests.py | 129 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 126 insertions(+), 3 deletions(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index 72314505..40b85310 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -21,7 +21,7 @@
 #
 #
 # REQUIREMENTS
-# - Python 3
+# - Python 3.5 (subprocess.run)
 # - Linux (libaio ioengine, zbd tests, etc)
 # - The artifact directory must be on a file system that accepts 512-byte IO
 #   (t0002, t0003, t0004).
@@ -39,7 +39,6 @@
 #
 # TODO  run multiple tests simultaneously
 # TODO  Add sgunmap tests (requires SAS SSD)
-# TODO  automatically detect dependencies and skip tests accordingly
 #
 
 import os
@@ -49,7 +48,9 @@ import time
 import shutil
 import logging
 import argparse
+import platform
 import subprocess
+import multiprocessing
 from pathlib import Path
 
 
@@ -400,6 +401,85 @@ class FioJobTest_t0011(FioJobTest):
             self.passed = False
 
 
+class Requirements(object):
+    """Requirements consists of multiple run environment characteristics.
+    These are to determine if a particular test can be run"""
+
+    _linux = False
+    _libaio = False
+    _zbd = False
+    _root = False
+    _zoned_nullb = False
+    _not_macos = False
+    _unittests = False
+    _cpucount4 = False
+
+    def __init__(self, fio_root):
+        Requirements._not_macos = platform.system() != "Darwin"
+        Requirements._linux = platform.system() == "Linux"
+
+        if Requirements._linux:
+            try:
+                config_file = os.path.join(fio_root, "config-host.h")
+                with open(config_file, "r") as config:
+                    contents = config.read()
+            except Exception:
+                print("Unable to open {0} to check requirements".format(config_file))
+                Requirements._zbd = True
+            else:
+                Requirements._zbd = "CONFIG_LINUX_BLKZONED" in contents
+                Requirements._libaio = "CONFIG_LIBAIO" in contents
+
+            Requirements._root = (os.geteuid() == 0)
+            if Requirements._zbd and Requirements._root:
+                    subprocess.run(["modprobe", "null_blk"],
+                                   stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE)
+                    if os.path.exists("/sys/module/null_blk/parameters/zoned"):
+                        Requirements._zoned_nullb = True
+
+        unittest_path = os.path.join(fio_root, "unittests", "unittest")
+        Requirements._unittests = os.path.exists(unittest_path)
+
+        Requirements._cpucount4 = multiprocessing.cpu_count() >= 4
+
+        req_list = [Requirements.linux,
+                    Requirements.libaio,
+                    Requirements.zbd,
+                    Requirements.root,
+                    Requirements.zoned_nullb,
+                    Requirements.not_macos,
+                    Requirements.unittests,
+                    Requirements.cpucount4]
+        for req in req_list:
+            value, desc = req()
+            logging.debug("Requirement '%s' met? %s" % (desc, value))
+
+    def linux():
+        return Requirements._linux, "Linux required"
+
+    def libaio():
+        return Requirements._libaio, "libaio required"
+
+    def zbd():
+        return Requirements._zbd, "Zoned block device support required"
+
+    def root():
+        return Requirements._root, "root required"
+
+    def zoned_nullb():
+        return Requirements._zoned_nullb, "Zoned null block device support required"
+
+    def not_macos():
+        return Requirements._not_macos, "platform other than macOS required"
+
+    def unittests():
+        return Requirements._unittests, "Unittests support required"
+
+    def cpucount4():
+        return Requirements._cpucount4, "4+ CPUs required"
+
+
 SUCCESS_DEFAULT = {
         'zero_return': True,
         'stderr_empty': True,
@@ -423,6 +503,7 @@ TEST_LIST = [
             'success':          SUCCESS_DEFAULT,
             'pre_job':          None,
             'pre_success':      None,
+            'requirements':     [],
         },
         {
             'test_id':          2,
@@ -431,6 +512,7 @@ TEST_LIST = [
             'success':          SUCCESS_DEFAULT,
             'pre_job':          't0002-13af05ae-pre.fio',
             'pre_success':      None,
+            'requirements':     [Requirements.linux, Requirements.libaio],
         },
         {
             'test_id':          3,
@@ -439,6 +521,7 @@ TEST_LIST = [
             'success':          SUCCESS_NONZERO,
             'pre_job':          't0003-0ae2c6e1-pre.fio',
             'pre_success':      SUCCESS_DEFAULT,
+            'requirements':     [Requirements.linux, Requirements.libaio],
         },
         {
             'test_id':          4,
@@ -447,6 +530,7 @@ TEST_LIST = [
             'success':          SUCCESS_DEFAULT,
             'pre_job':          None,
             'pre_success':      None,
+            'requirements':     [Requirements.linux, Requirements.libaio],
         },
         {
             'test_id':          5,
@@ -456,6 +540,7 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [],
         },
         {
             'test_id':          6,
@@ -465,6 +550,7 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [Requirements.linux, Requirements.libaio],
         },
         {
             'test_id':          7,
@@ -474,6 +560,7 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [],
         },
         {
             'test_id':          8,
@@ -483,6 +570,7 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [],
         },
         {
             'test_id':          9,
@@ -492,6 +580,9 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [Requirements.not_macos,
+                                 Requirements.cpucount4],
+                                # mac os does not support CPU affinity
         },
         {
             'test_id':          10,
@@ -500,6 +591,7 @@ TEST_LIST = [
             'success':          SUCCESS_DEFAULT,
             'pre_job':          None,
             'pre_success':      None,
+            'requirements':     [],
         },
         {
             'test_id':          11,
@@ -509,6 +601,7 @@ TEST_LIST = [
             'pre_job':          None,
             'pre_success':      None,
             'output_format':    'json',
+            'requirements':     [],
         },
         {
             'test_id':          1000,
@@ -516,6 +609,7 @@ TEST_LIST = [
             'exe':              't/axmap',
             'parameters':       None,
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [],
         },
         {
             'test_id':          1001,
@@ -523,6 +617,7 @@ TEST_LIST = [
             'exe':              't/ieee754',
             'parameters':       None,
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [],
         },
         {
             'test_id':          1002,
@@ -530,6 +625,7 @@ TEST_LIST = [
             'exe':              't/lfsr-test',
             'parameters':       ['0xFFFFFF', '0', '0', 'verify'],
             'success':          SUCCESS_STDERR,
+            'requirements':     [],
         },
         {
             'test_id':          1003,
@@ -537,6 +633,7 @@ TEST_LIST = [
             'exe':              't/readonly.py',
             'parameters':       ['-f', '{fio_path}'],
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [],
         },
         {
             'test_id':          1004,
@@ -544,6 +641,7 @@ TEST_LIST = [
             'exe':              't/steadystate_tests.py',
             'parameters':       ['{fio_path}'],
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [],
         },
         {
             'test_id':          1005,
@@ -551,6 +649,7 @@ TEST_LIST = [
             'exe':              't/stest',
             'parameters':       None,
             'success':          SUCCESS_STDERR,
+            'requirements':     [],
         },
         {
             'test_id':          1006,
@@ -558,6 +657,7 @@ TEST_LIST = [
             'exe':              't/strided.py',
             'parameters':       ['{fio_path}'],
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [],
         },
         {
             'test_id':          1007,
@@ -565,6 +665,8 @@ TEST_LIST = [
             'exe':              't/zbd/run-tests-against-regular-nullb',
             'parameters':       None,
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [Requirements.linux, Requirements.zbd,
+                                 Requirements.root],
         },
         {
             'test_id':          1008,
@@ -572,6 +674,8 @@ TEST_LIST = [
             'exe':              't/zbd/run-tests-against-zoned-nullb',
             'parameters':       None,
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [Requirements.linux, Requirements.zbd,
+                                 Requirements.root, Requirements.zoned_nullb],
         },
         {
             'test_id':          1009,
@@ -579,6 +683,7 @@ TEST_LIST = [
             'exe':              'unittests/unittest',
             'parameters':       None,
             'success':          SUCCESS_DEFAULT,
+            'requirements':     [Requirements.unittests],
         },
 ]
 
@@ -597,6 +702,8 @@ def parse_args():
                         help='list of test(s) to run, skipping all others')
     parser.add_argument('-d', '--debug', action='store_true',
                         help='provide debug output')
+    parser.add_argument('-k', '--skip-req', action='store_true',
+                        help='skip requirements checking')
     args = parser.parse_args()
 
     return args
@@ -628,6 +735,9 @@ def main():
     os.mkdir(artifact_root)
     print("Artifact directory is %s" % artifact_root)
 
+    if not args.skip_req:
+        req = Requirements(fio_root)
+
     passed = 0
     failed = 0
     skipped = 0
@@ -636,7 +746,7 @@ def main():
         if (args.skip and config['test_id'] in args.skip) or \
            (args.run_only and config['test_id'] not in args.run_only):
             skipped = skipped + 1
-            print("Test {0} SKIPPED".format(config['test_id']))
+            print("Test {0} SKIPPED (User request)".format(config['test_id']))
             continue
 
         if issubclass(config['test_class'], FioJobTest):
@@ -673,6 +783,19 @@ def main():
             failed = failed + 1
             continue
 
+        if not args.skip_req:
+            skip = False
+            for req in config['requirements']:
+                ok, reason = req()
+                skip = not ok
+                logging.debug("Requirement '%s' met? %s" % (reason, ok))
+                if skip:
+                    break
+            if skip:
+                print("Test {0} SKIPPED ({1})".format(config['test_id'], reason))
+                skipped = skipped + 1
+                continue
+
         test.setup(artifact_root, config['test_id'])
         test.run()
         test.check_result()
-- 
2.17.1

[PATCH 4/9] t/run-fio-tests: improve Windows support

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

- add .exe extension for fio and unittest
- for python scripts use 'python.exe script.py' instead of ./script.py
- make JSON decoding more resilient by lopping off up to four of the
first lines when encountering decoding errors. This helps Windows
because for some jobs fio prints an informational message about
requiring threads
---
 t/run-fio-tests.py | 56 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 43 insertions(+), 13 deletions(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index 40b85310..7fc3e4cd 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -267,19 +267,35 @@ class FioJobTest(FioExeTest):
         if not self.passed:
             return
 
-        if 'json' in self.output_format:
+        if not 'json' in self.output_format:
+            return
+
+        try:
+            with open(os.path.join(self.test_dir, self.fio_output), "r") as output_file:
+                file_data = output_file.read()
+        except EnvironmentError:
+            self.failure_reason = "{0} unable to open output file,".format(self.failure_reason)
+            self.passed = False
+            return
+
+        #
+        # Sometimes fio informational messages are included at the top of the
+        # JSON output, especially under Windows. Try to decode output as JSON
+        # data, lopping off up to the first four lines
+        #
+        lines = file_data.splitlines()
+        for i in range(5):
+            file_data = '\n'.join(lines[i:])
             try:
-                with open(os.path.join(self.test_dir, self.fio_output), "r") as output_file:
-                    file_data = output_file.read()
-            except EnvironmentError:
-                self.failure_reason = "{0} unable to open output file,".format(self.failure_reason)
-                self.passed = False
+                self.json_data = json.loads(file_data)
+            except json.JSONDecodeError:
+                continue
             else:
-                try:
-                    self.json_data = json.loads(file_data)
-                except json.JSONDecodeError:
-                    self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
-                    self.passed = False
+                logging.debug("skipped %d lines decoding JSON data" % i)
+                return
+
+        self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
+        self.passed = False
 
 
 class FioJobTest_t0005(FioJobTest):
@@ -438,7 +454,11 @@ class Requirements(object):
                     if os.path.exists("/sys/module/null_blk/parameters/zoned"):
                         Requirements._zoned_nullb = True
 
-        unittest_path = os.path.join(fio_root, "unittests", "unittest")
+        if platform.system() == "Windows":
+            utest_exe = "unittest.exe"
+        else:
+            utest_exe = "unittest"
+        unittest_path = os.path.join(fio_root, "unittests", utest_exe)
         Requirements._unittests = os.path.exists(unittest_path)
 
         Requirements._cpucount4 = multiprocessing.cpu_count() >= 4
@@ -725,7 +745,11 @@ def main():
     if args.fio:
         fio_path = args.fio
     else:
-        fio_path = os.path.join(fio_root, "fio")
+        if platform.system() == "Windows":
+            fio_exe = "fio.exe"
+        else:
+            fio_exe = "fio"
+        fio_path = os.path.join(fio_root, fio_exe)
     print("fio path is %s" % fio_path)
     if not shutil.which(fio_path):
         print("Warning: fio executable not found")
@@ -776,6 +800,12 @@ def main():
                 parameters = [p.format(fio_path=fio_path) for p in config['parameters']]
             else:
                 parameters = None
+            if Path(exe_path).suffix == '.py' and platform.system() == "Windows":
+                if parameters:
+                    parameters.insert(0, exe_path)
+                else:
+                    parameters = [exe_path]
+                exe_path = "python.exe"
             test = config['test_class'](exe_path, parameters,
                                         config['success'])
         else:
-- 
2.17.1

[PATCH 5/9] t/run-fio-tests: identify test id for debug messages

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

---
 t/run-fio-tests.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index 7fc3e4cd..4c7cc3a4 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -137,7 +137,7 @@ class FioExeTest(FioTest):
                                     universal_newlines=True)
             proc.communicate(timeout=self.success['timeout'])
             exticode_file.write('{0}\n'.format(proc.returncode))
-            logging.debug("return code: %d" % proc.returncode)
+            logging.debug("Test %d: return code: %d" % (self.testnum, proc.returncode))
             self.output['proc'] = proc
         except subprocess.TimeoutExpired:
             proc.terminate()
@@ -254,7 +254,7 @@ class FioJobTest(FioExeTest):
         if not self.precon_failed:
             super(FioJobTest, self).run()
         else:
-            logging.debug("precondition step failed")
+            logging.debug("Test %d: precondition step failed" % self.testnum)
 
     def check_result(self):
         if self.precon_failed:
@@ -291,7 +291,7 @@ class FioJobTest(FioExeTest):
             except json.JSONDecodeError:
                 continue
             else:
-                logging.debug("skipped %d lines decoding JSON data" % i)
+                logging.debug("Test %d: skipped %d lines decoding JSON data" % (self.testnum, i))
                 return
 
         self.failure_reason = "{0} unable to decode JSON data,".format(self.failure_reason)
@@ -328,7 +328,7 @@ class FioJobTest_t0006(FioJobTest):
 
         ratio = self.json_data['jobs'][0]['read']['io_kbytes'] \
             / self.json_data['jobs'][0]['write']['io_kbytes']
-        logging.debug("ratio: %f" % ratio)
+        logging.debug("Test %d: ratio: %f" % (self.testnum, ratio))
         if ratio < 1.99 or ratio > 2.01:
             self.failure_reason = "{0} read/write ratio mismatch,".format(self.failure_reason)
             self.passed = False
@@ -364,7 +364,7 @@ class FioJobTest_t0008(FioJobTest):
             return
 
         ratio = self.json_data['jobs'][0]['write']['io_kbytes'] / 16568
-        logging.debug("ratio: %f" % ratio)
+        logging.debug("Test %d: ratio: %f" % (self.testnum, ratio))
 
         if ratio < 0.99 or ratio > 1.01:
             self.failure_reason = "{0} bytes written mismatch,".format(self.failure_reason)
@@ -384,7 +384,7 @@ class FioJobTest_t0009(FioJobTest):
         if not self.passed:
             return
 
-        logging.debug('elapsed: %d' % self.json_data['jobs'][0]['elapsed'])
+        logging.debug('Test %d: elapsed: %d' % (self.testnum, self.json_data['jobs'][0]['elapsed']))
 
         if self.json_data['jobs'][0]['elapsed'] < 60:
             self.failure_reason = "{0} elapsed time mismatch,".format(self.failure_reason)
@@ -406,7 +406,7 @@ class FioJobTest_t0011(FioJobTest):
         iops1 = self.json_data['jobs'][0]['read']['iops']
         iops2 = self.json_data['jobs'][1]['read']['iops']
         ratio = iops2 / iops1
-        logging.debug("ratio: %f" % ratio)
+        logging.debug("Test %d: ratio: %f" % (self.testnum, ratio))
 
         if iops1 < 999 or iops1 > 1001:
             self.failure_reason = "{0} iops value mismatch,".format(self.failure_reason)
@@ -473,7 +473,7 @@ class Requirements(object):
                     Requirements.cpucount4]
         for req in req_list:
             value, desc = req()
-            logging.debug("Requirement '%s' met? %s" % (desc, value))
+            logging.debug("Requirements: Requirement '%s' met? %s" % (desc, value))
 
     def linux():
         return Requirements._linux, "Linux required"
@@ -818,7 +818,7 @@ def main():
             for req in config['requirements']:
                 ok, reason = req()
                 skip = not ok
-                logging.debug("Requirement '%s' met? %s" % (reason, ok))
+                logging.debug("Test %d: Requirement '%s' met? %s" % (config['test_id'], reason, ok))
                 if skip:
                     break
             if skip:
@@ -836,9 +836,9 @@ def main():
             result = "FAILED: {0}".format(test.failure_reason)
             failed = failed + 1
             with open(test.stderr_file, "r") as stderr_file:
-                logging.debug("stderr:\n%s" % stderr_file.read())
+                logging.debug("Test %d: stderr:\n%s" % (config['test_id'], stderr_file.read()))
             with open(test.stdout_file, "r") as stdout_file:
-                logging.debug("stdout:\n%s" % stdout_file.read())
+                logging.debug("Test %d: stdout:\n%s" % (config['test_id'], stdout_file.read()))
         print("Test {0} {1}".format(config['test_id'], result))
 
     print("{0} test(s) passed, {1} failed, {2} skipped".format(passed, failed, skipped))
-- 
2.17.1

[PATCH 6/9] t/steadystate_tests: use null ioengine for tests

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

Instead of reading from /dev/zero just use the null ioengine. This
enables the script to run on Windows.
---
 t/steadystate_tests.py | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/t/steadystate_tests.py b/t/steadystate_tests.py
index 53b0f35e..9122a60f 100755
--- a/t/steadystate_tests.py
+++ b/t/steadystate_tests.py
@@ -31,12 +31,7 @@ from scipy import stats
 
 def parse_args():
     parser = argparse.ArgumentParser()
-    parser.add_argument('fio',
-                        help='path to fio executable')
-    parser.add_argument('--read',
-                        help='target for read testing')
-    parser.add_argument('--write',
-                        help='target for write testing')
+    parser.add_argument('fio', help='path to fio executable')
     args = parser.parse_args()
 
     return args
@@ -123,26 +118,16 @@ if __name__ == '__main__':
               {'s': True, 'timeout': 10, 'numjobs': 3, 'ss_dur': 10, 'ss_ramp': 500, 'iops': False, 'slope': True, 'ss_limit': 0.1, 'pct': True},
             ]
 
-    if args.read == None:
-        if os.name == 'posix':
-            args.read = '/dev/zero'
-            extra = [ "--size=128M" ]
-        else:
-            print("ERROR: file for read testing must be specified on non-posix systems")
-            sys.exit(1)
-    else:
-        extra = []
-
     jobnum = 0
     for job in reads:
 
         tf = "steadystate_job{0}.json".format(jobnum)
         parameters = [ "--name=job{0}".format(jobnum) ]
-        parameters.extend(extra)
         parameters.extend([ "--thread",
                             "--output-format=json",
                             "--output={0}".format(tf),
-                            "--filename={0}".format(args.read),
+                            "--ioengine=null",
+                            "--size=1G",
                             "--rw=randrw",
                             "--rwmixread=100",
                             "--stonewall",
-- 
2.17.1

[PATCH 7/9] .travis.yml: run t/run-fio.tests.py as part of build

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

- install SciPy and CUnit support
- skip test 6 because of timeout
- skip zbd tests 1007, 1008 because installing kernel modules is not supported
- replace xcode8.3 (lacks python3) with latest xcode11.2
---
 .travis.yml | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 4a87fe6c..0017db56 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,19 +15,13 @@ matrix:
     - os: osx
       compiler: clang # Workaround travis setting CC=["clang", "gcc"]
       env: BUILD_ARCH="x86_64"
-    # Build using the 10.12 SDK but target and run on OSX 10.11
-#   - os: osx
-#     compiler: clang
-#     osx_image: xcode8
-#     env: SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk MACOSX_DEPLOYMENT_TARGET=10.11
-    # Build on the latest OSX version (will eventually become obsolete)
     - os: osx
       compiler: clang
-      osx_image: xcode8.3
+      osx_image: xcode9.4
       env: BUILD_ARCH="x86_64"
     - os: osx
       compiler: clang
-      osx_image: xcode9.4
+      osx_image: xcode11.2
       env: BUILD_ARCH="x86_64"
   exclude:
     - os: osx
@@ -39,17 +33,27 @@ matrix:
 before_install:
   - EXTRA_CFLAGS="-Werror"
   - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
-        pkgs=(libaio-dev libnuma-dev libz-dev librbd-dev libibverbs-dev librdmacm-dev);
+        pkgs=(libaio-dev libnuma-dev libz-dev librbd-dev libibverbs-dev librdmacm-dev libcunit1 libcunit1-dev);
         if [[ "$BUILD_ARCH" == "x86" ]]; then
             pkgs=("${pkgs[@]/%/:i386}");
-            pkgs+=(gcc-multilib);
+            pkgs+=(gcc-multilib python-scipy);
             EXTRA_CFLAGS="${EXTRA_CFLAGS} -m32";
         else
-            pkgs+=(glusterfs-common);
+            pkgs+=(glusterfs-common python-scipy);
         fi;
         sudo apt-get -qq update;
         sudo apt-get install --no-install-recommends -qq -y "${pkgs[@]}";
     fi
+  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
+        brew update;
+        brew install cunit;
+        if [[ "$TRAVIS_OSX_IMAGE" == "xcode11.2" ]]; then
+            pip3 install scipy;
+        else
+            pip install scipy;
+        fi;
+    fi
 script:
   - ./configure --extra-cflags="${EXTRA_CFLAGS}" && make
   - make test
+  - sudo python3 t/run-fio-tests.py --skip 6 1007 1008
-- 
2.17.1

[PATCH 8/9] .appveyor.yml: run run-fio-tests.py

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

- Add CUnit support
- install SciPy
- skip test 5 because Windows does not support direct I/O with sync
ioengines
---
 .appveyor.yml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/.appveyor.yml b/.appveyor.yml
index ca8b2ab1..f6934096 100644
--- a/.appveyor.yml
+++ b/.appveyor.yml
@@ -13,8 +13,9 @@ environment:
       CONFIGURE_OPTIONS: --build-32bit-win --target-win-ver=xp
 
 install:
-  - '%CYG_ROOT%\setup-x86_64.exe --quiet-mode --no-shortcuts --only-site --site "%CYG_MIRROR%" --packages "mingw64-%PACKAGE_ARCH%-zlib" > NUL'
-  - SET PATH=%CYG_ROOT%\bin;%PATH% # NB: Changed env variables persist to later sections
+  - '%CYG_ROOT%\setup-x86_64.exe --quiet-mode --no-shortcuts --only-site --site "%CYG_MIRROR%" --packages "mingw64-%PACKAGE_ARCH%-zlib,mingw64-%PACKAGE_ARCH%-CUnit" > NUL'
+  - SET PATH=C:\Python38-x64;%CYG_ROOT%\bin;%PATH% # NB: Changed env variables persist to later sections
+  - python.exe -m pip install scipy
 
 build_script:
   - 'bash.exe -lc "cd \"${APPVEYOR_BUILD_FOLDER}\" && ./configure --disable-native --extra-cflags=\"-Werror\" ${CONFIGURE_OPTIONS} && make.exe'
@@ -24,6 +25,7 @@ after_build:
 
 test_script:
   - 'bash.exe -lc "cd \"${APPVEYOR_BUILD_FOLDER}\" && file.exe fio.exe && make.exe test'
+  - 'bash.exe -lc "cd \"${APPVEYOR_BUILD_FOLDER}\" && python.exe t/run-fio-tests.py --skip 5 --debug'
 
 artifacts:
   - path: os\windows\*.msi
-- 
2.17.1

[PATCH 9/9] t/run-fio-tests: relax acceptance criterion for t0011

[vincentfu]

From: Vincent Fu <vincent.fu@wdc.com>

I saw a handful of benign failures for t0011 when run on
travis/appveyor. This patch allows the test to still pass with a
measured IOPS of 998 to 1002 and adds a debug print for the value
actually observed.
---
 t/run-fio-tests.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index 4c7cc3a4..a0a1e8fa 100755
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -406,9 +406,10 @@ class FioJobTest_t0011(FioJobTest):
         iops1 = self.json_data['jobs'][0]['read']['iops']
         iops2 = self.json_data['jobs'][1]['read']['iops']
         ratio = iops2 / iops1
+        logging.debug("Test %d: iops1: %f" % (self.testnum, iops1))
         logging.debug("Test %d: ratio: %f" % (self.testnum, ratio))
 
-        if iops1 < 999 or iops1 > 1001:
+        if iops1 < 998 or iops1 > 1002:
             self.failure_reason = "{0} iops value mismatch,".format(self.failure_reason)
             self.passed = False
 
-- 
2.17.1

Re: [PATCH 0/9] testing patches

[Sitsofe Wheeler]

On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>
> From: Vincent Fu <vincent.fu@wdc.com>
>
> Jens, please consider this series of patches related to testing.
>
> The patches improve t/run-fio-tests.py in various ways, most prominently
> adding support for Windows and macOS.
>
> Also included are travis and appveyor patches that add run-fio-tests.py
> as a step.  Currently both the travis and appveyor build processes
> complete in less than four minutes.  Adding run-fio-tests.py increases
> this to about 20 minutes for travis and 14 minutes for appveyor.

In general I think this work is fantastic and much needed (you can
search through the fio commit logs using "git log --grep 'size='" to
find jobs files that have caused issues in the past and may be worth
turning into tests at some point). However, I think making the builds
so slow may be a disadvantage rather than a benefit. I agree with
nearly all the patch set bar running this by default with
travis/appveyor...

-- 
Sitsofe | http://sucs.org/~sits/

Re: [PATCH 0/9] testing patches

[Vincent Fu]

On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>
>> From: Vincent Fu <vincent.fu@wdc.com>
>>
>> Jens, please consider this series of patches related to testing.
>>
>> The patches improve t/run-fio-tests.py in various ways, most prominently
>> adding support for Windows and macOS.
>>
>> Also included are travis and appveyor patches that add run-fio-tests.py
>> as a step.  Currently both the travis and appveyor build processes
>> complete in less than four minutes.  Adding run-fio-tests.py increases
>> this to about 20 minutes for travis and 14 minutes for appveyor.
> 
> In general I think this work is fantastic and much needed (you can
> search through the fio commit logs using "git log --grep 'size='" to
> find jobs files that have caused issues in the past and may be worth
> turning into tests at some point). However, I think making the builds
> so slow may be a disadvantage rather than a benefit. I agree with
> nearly all the patch set bar running this by default with
> travis/appveyor...
> 

Many thanks for the feedback, Sitsofe. I agree that the build times are 
uncomfortably long, but since I had done the work I thought I would 
offer the patches to Jens.

Jens, what do you think? Would you like me to re-send the patch series 
without the appveyor and travis changes?

Re: [PATCH 0/9] testing patches

[Jens Axboe]

On 12/11/19 12:53 PM, Vincent Fu wrote:
> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>>
>>> From: Vincent Fu <vincent.fu@wdc.com>
>>>
>>> Jens, please consider this series of patches related to testing.
>>>
>>> The patches improve t/run-fio-tests.py in various ways, most prominently
>>> adding support for Windows and macOS.
>>>
>>> Also included are travis and appveyor patches that add run-fio-tests.py
>>> as a step.  Currently both the travis and appveyor build processes
>>> complete in less than four minutes.  Adding run-fio-tests.py increases
>>> this to about 20 minutes for travis and 14 minutes for appveyor.
>>
>> In general I think this work is fantastic and much needed (you can
>> search through the fio commit logs using "git log --grep 'size='" to
>> find jobs files that have caused issues in the past and may be worth
>> turning into tests at some point). However, I think making the builds
>> so slow may be a disadvantage rather than a benefit. I agree with
>> nearly all the patch set bar running this by default with
>> travis/appveyor...
>>
> 
> Many thanks for the feedback, Sitsofe. I agree that the build times are 
> uncomfortably long, but since I had done the work I thought I would 
> offer the patches to Jens.
> 
> Jens, what do you think? Would you like me to re-send the patch series 
> without the appveyor and travis changes?

Looks like it's about 20 min, I'm not so sure we need faster turn-around
than that. I would personally much rather see a pull request with 20 min
delay on whether it passes or not, rather than have to run it manually.
That's how things get missed.

So I'd be leaning towards just making it run it by default. Is 20 min
really that big of an issue? It's not like it's holding anyone up.

-- 
Jens Axboe

Re: [PATCH 0/9] testing patches

[Vincent Fu]

On 12/11/19 2:57 PM, Jens Axboe wrote:
> On 12/11/19 12:53 PM, Vincent Fu wrote:
>> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
>>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>>>
>>>> From: Vincent Fu <vincent.fu@wdc.com>
>>>>
>>>> Jens, please consider this series of patches related to testing.
>>>>
>>>> The patches improve t/run-fio-tests.py in various ways, most prominently
>>>> adding support for Windows and macOS.
>>>>
>>>> Also included are travis and appveyor patches that add run-fio-tests.py
>>>> as a step.  Currently both the travis and appveyor build processes
>>>> complete in less than four minutes.  Adding run-fio-tests.py increases
>>>> this to about 20 minutes for travis and 14 minutes for appveyor.
>>>
>>> In general I think this work is fantastic and much needed (you can
>>> search through the fio commit logs using "git log --grep 'size='" to
>>> find jobs files that have caused issues in the past and may be worth
>>> turning into tests at some point). However, I think making the builds
>>> so slow may be a disadvantage rather than a benefit. I agree with
>>> nearly all the patch set bar running this by default with
>>> travis/appveyor...
>>>
>>
>> Many thanks for the feedback, Sitsofe. I agree that the build times are
>> uncomfortably long, but since I had done the work I thought I would
>> offer the patches to Jens.
>>
>> Jens, what do you think? Would you like me to re-send the patch series
>> without the appveyor and travis changes?
> 
> Looks like it's about 20 min, I'm not so sure we need faster turn-around
> than that. I would personally much rather see a pull request with 20 min
> delay on whether it passes or not, rather than have to run it manually.
> That's how things get missed.
> 
> So I'd be leaning towards just making it run it by default. Is 20 min
> really that big of an issue? It's not like it's holding anyone up.
> 

I think the main difference is that at 4 minutes, it's easy enough to 
wait around to check that the build has succeeded, whereas at 20 minutes 
one has moved onto another task and will have to make a context switch 
to come back to check on the build. Also, if people add more tests the 
build times will become even longer.

All that said, it's easy enough to revert the patch if adding the tests 
hurts the workflow too much.

Re: [PATCH 0/9] testing patches

[Jens Axboe]

On 12/11/19 1:10 PM, Vincent Fu wrote:
> On 12/11/19 2:57 PM, Jens Axboe wrote:
>> On 12/11/19 12:53 PM, Vincent Fu wrote:
>>> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
>>>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>>>>
>>>>> From: Vincent Fu <vincent.fu@wdc.com>
>>>>>
>>>>> Jens, please consider this series of patches related to testing.
>>>>>
>>>>> The patches improve t/run-fio-tests.py in various ways, most prominently
>>>>> adding support for Windows and macOS.
>>>>>
>>>>> Also included are travis and appveyor patches that add run-fio-tests.py
>>>>> as a step.  Currently both the travis and appveyor build processes
>>>>> complete in less than four minutes.  Adding run-fio-tests.py increases
>>>>> this to about 20 minutes for travis and 14 minutes for appveyor.
>>>>
>>>> In general I think this work is fantastic and much needed (you can
>>>> search through the fio commit logs using "git log --grep 'size='" to
>>>> find jobs files that have caused issues in the past and may be worth
>>>> turning into tests at some point). However, I think making the builds
>>>> so slow may be a disadvantage rather than a benefit. I agree with
>>>> nearly all the patch set bar running this by default with
>>>> travis/appveyor...
>>>>
>>>
>>> Many thanks for the feedback, Sitsofe. I agree that the build times are
>>> uncomfortably long, but since I had done the work I thought I would
>>> offer the patches to Jens.
>>>
>>> Jens, what do you think? Would you like me to re-send the patch series
>>> without the appveyor and travis changes?
>>
>> Looks like it's about 20 min, I'm not so sure we need faster turn-around
>> than that. I would personally much rather see a pull request with 20 min
>> delay on whether it passes or not, rather than have to run it manually.
>> That's how things get missed.
>>
>> So I'd be leaning towards just making it run it by default. Is 20 min
>> really that big of an issue? It's not like it's holding anyone up.
>>
> 
> I think the main difference is that at 4 minutes, it's easy enough to 
> wait around to check that the build has succeeded, whereas at 20 minutes 
> one has moved onto another task and will have to make a context switch 
> to come back to check on the build. Also, if people add more tests the 
> build times will become even longer.
> 
> All that said, it's easy enough to revert the patch if adding the tests 
> hurts the workflow too much.

I don't disagree that it's a long time. But the idea is that you push
it out to a branch, and then you get a notification some time later
if it worked or failed. As long as the notification works for both
cases, then I don't think it's much of an issue. Even 4 minutes will not
have people sitting around waiting for it to finish, at least not me.
I'd be on to other stuff the second I push it out.

Let's try this out and see how it goes. If it doesn't work so well, then
we always have the option of slimming down the default run, or whatever
else we can do to make it faster.

-- 
Jens Axboe

Re: [PATCH 0/9] testing patches

[Sitsofe Wheeler]

On Thu, 12 Dec 2019 at 03:54, Jens Axboe <axboe@kernel.dk> wrote:
>
> On 12/11/19 1:10 PM, Vincent Fu wrote:
> > On 12/11/19 2:57 PM, Jens Axboe wrote:
> >> On 12/11/19 12:53 PM, Vincent Fu wrote:
> >>> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
> >>>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
> >>>>>
> >>>>> From: Vincent Fu <vincent.fu@wdc.com>
> >>>>>
> >>>>> Jens, please consider this series of patches related to testing.
> >>>>>
> >>>>> The patches improve t/run-fio-tests.py in various ways, most prominently
> >>>>> adding support for Windows and macOS.
> >>>>>
> >>>>> Also included are travis and appveyor patches that add run-fio-tests.py
> >>>>> as a step.  Currently both the travis and appveyor build processes
> >>>>> complete in less than four minutes.  Adding run-fio-tests.py increases
> >>>>> this to about 20 minutes for travis and 14 minutes for appveyor.

<snip>

> Let's try this out and see how it goes. If it doesn't work so well, then
> we always have the option of slimming down the default run, or whatever
> else we can do to make it faster.

OK fair enough :-)

-- 
Sitsofe | http://sucs.org/~sits/

Re: [PATCH 0/9] testing patches

[Jens Axboe]

On 12/11/19 8:54 PM, Jens Axboe wrote:
> On 12/11/19 1:10 PM, Vincent Fu wrote:
>> On 12/11/19 2:57 PM, Jens Axboe wrote:
>>> On 12/11/19 12:53 PM, Vincent Fu wrote:
>>>> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
>>>>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>>>>>
>>>>>> From: Vincent Fu <vincent.fu@wdc.com>
>>>>>>
>>>>>> Jens, please consider this series of patches related to testing.
>>>>>>
>>>>>> The patches improve t/run-fio-tests.py in various ways, most prominently
>>>>>> adding support for Windows and macOS.
>>>>>>
>>>>>> Also included are travis and appveyor patches that add run-fio-tests.py
>>>>>> as a step.  Currently both the travis and appveyor build processes
>>>>>> complete in less than four minutes.  Adding run-fio-tests.py increases
>>>>>> this to about 20 minutes for travis and 14 minutes for appveyor.
>>>>>
>>>>> In general I think this work is fantastic and much needed (you can
>>>>> search through the fio commit logs using "git log --grep 'size='" to
>>>>> find jobs files that have caused issues in the past and may be worth
>>>>> turning into tests at some point). However, I think making the builds
>>>>> so slow may be a disadvantage rather than a benefit. I agree with
>>>>> nearly all the patch set bar running this by default with
>>>>> travis/appveyor...
>>>>>
>>>>
>>>> Many thanks for the feedback, Sitsofe. I agree that the build times are
>>>> uncomfortably long, but since I had done the work I thought I would
>>>> offer the patches to Jens.
>>>>
>>>> Jens, what do you think? Would you like me to re-send the patch series
>>>> without the appveyor and travis changes?
>>>
>>> Looks like it's about 20 min, I'm not so sure we need faster turn-around
>>> than that. I would personally much rather see a pull request with 20 min
>>> delay on whether it passes or not, rather than have to run it manually.
>>> That's how things get missed.
>>>
>>> So I'd be leaning towards just making it run it by default. Is 20 min
>>> really that big of an issue? It's not like it's holding anyone up.
>>>
>>
>> I think the main difference is that at 4 minutes, it's easy enough to 
>> wait around to check that the build has succeeded, whereas at 20 minutes 
>> one has moved onto another task and will have to make a context switch 
>> to come back to check on the build. Also, if people add more tests the 
>> build times will become even longer.
>>
>> All that said, it's easy enough to revert the patch if adding the tests 
>> hurts the workflow too much.
> 
> I don't disagree that it's a long time. But the idea is that you push
> it out to a branch, and then you get a notification some time later
> if it worked or failed. As long as the notification works for both
> cases, then I don't think it's much of an issue. Even 4 minutes will not
> have people sitting around waiting for it to finish, at least not me.
> I'd be on to other stuff the second I push it out.
> 
> Let's try this out and see how it goes. If it doesn't work so well, then
> we always have the option of slimming down the default run, or whatever
> else we can do to make it faster.

It seems to fail, though:

https://travis-ci.org/axboe/fio/jobs/625901116?utm_medium=notification&utm_source=email

-- 
Jens Axboe

Re: [PATCH 0/9] testing patches

[Vincent Fu]

On 12/16/19 5:13 PM, Jens Axboe wrote:
> On 12/11/19 8:54 PM, Jens Axboe wrote:
>> On 12/11/19 1:10 PM, Vincent Fu wrote:
>>> On 12/11/19 2:57 PM, Jens Axboe wrote:
>>>> On 12/11/19 12:53 PM, Vincent Fu wrote:
>>>>> On 12/10/19 4:32 PM, Sitsofe Wheeler wrote:
>>>>>> On Tue, 10 Dec 2019 at 17:56, <vincentfu@gmail.com> wrote:
>>>>>>>
>>>>>>> From: Vincent Fu <vincent.fu@wdc.com>
>>>>>>>
>>>>>>> Jens, please consider this series of patches related to testing.
>>>>>>>
>>>>>>> The patches improve t/run-fio-tests.py in various ways, most prominently
>>>>>>> adding support for Windows and macOS.
>>>>>>>
>>>>>>> Also included are travis and appveyor patches that add run-fio-tests.py
>>>>>>> as a step.  Currently both the travis and appveyor build processes
>>>>>>> complete in less than four minutes.  Adding run-fio-tests.py increases
>>>>>>> this to about 20 minutes for travis and 14 minutes for appveyor.
>>>>>>
>>>>>> In general I think this work is fantastic and much needed (you can
>>>>>> search through the fio commit logs using "git log --grep 'size='" to
>>>>>> find jobs files that have caused issues in the past and may be worth
>>>>>> turning into tests at some point). However, I think making the builds
>>>>>> so slow may be a disadvantage rather than a benefit. I agree with
>>>>>> nearly all the patch set bar running this by default with
>>>>>> travis/appveyor...
>>>>>>
>>>>>
>>>>> Many thanks for the feedback, Sitsofe. I agree that the build times are
>>>>> uncomfortably long, but since I had done the work I thought I would
>>>>> offer the patches to Jens.
>>>>>
>>>>> Jens, what do you think? Would you like me to re-send the patch series
>>>>> without the appveyor and travis changes?
>>>>
>>>> Looks like it's about 20 min, I'm not so sure we need faster turn-around
>>>> than that. I would personally much rather see a pull request with 20 min
>>>> delay on whether it passes or not, rather than have to run it manually.
>>>> That's how things get missed.
>>>>
>>>> So I'd be leaning towards just making it run it by default. Is 20 min
>>>> really that big of an issue? It's not like it's holding anyone up.
>>>>
>>>
>>> I think the main difference is that at 4 minutes, it's easy enough to
>>> wait around to check that the build has succeeded, whereas at 20 minutes
>>> one has moved onto another task and will have to make a context switch
>>> to come back to check on the build. Also, if people add more tests the
>>> build times will become even longer.
>>>
>>> All that said, it's easy enough to revert the patch if adding the tests
>>> hurts the workflow too much.
>>
>> I don't disagree that it's a long time. But the idea is that you push
>> it out to a branch, and then you get a notification some time later
>> if it worked or failed. As long as the notification works for both
>> cases, then I don't think it's much of an issue. Even 4 minutes will not
>> have people sitting around waiting for it to finish, at least not me.
>> I'd be on to other stuff the second I push it out.
>>
>> Let's try this out and see how it goes. If it doesn't work so well, then
>> we always have the option of slimming down the default run, or whatever
>> else we can do to make it faster.
> 
> It seems to fail, though:
> 
> https://travis-ci.org/axboe/fio/jobs/625901116?utm_medium=notification&utm_source=email
> 
Fixed here: https://github.com/axboe/fio/pull/880