[Qemu-devel] [PATCH V4 0/5] tests: Add the image fuzzer with qcow2 support

[Qemu-devel] [PATCH V4 0/5] tests: Add the image fuzzer with qcow2 support

[Maria Kustova]

This patch series introduces the image fuzzer, a tool for stability and
reliability testing.
Its approach is to run large amount of tests in background. During every test a
program (e.g. qemu-img) is called to read or modify an invalid test image.
A test image has valid inner structure defined by its format specification with
some fields having random invalid values.

Patch 1 contains documentation for the image fuzzer, patch 2 is the test runner
and remaining ones relate to the image generator for qcow2 format.

This patch series was created for the 'block-next' branch.

v3 -> v4:
All changes are based on the review of Stefan Hajnoczi:
Runner:
 * list of test results replaced by a single boolean value
 * the code cleanup (residual hash() and error handling)
 * fixed ambiguities in the usage and errors texts
Fuzz:
 * simplified validating and selecting functions with the unified validator
Layout:
 * fixed typos
Docs:
 * fixed wrong JSON terminology

Maria Kustova (5):
  docs: Specification for the image fuzzer
  runner: Tool for fuzz tests execution
  fuzz: Fuzzing functions for qcow2 images
  layout: Generator of fuzzed qcow2 images
  package: Public API for image-fuzzer/runner/runner.py

 tests/image-fuzzer/docs/image-fuzzer.txt | 239 ++++++++++++++++++++
 tests/image-fuzzer/qcow2/__init__.py     |   1 +
 tests/image-fuzzer/qcow2/fuzz.py         | 329 ++++++++++++++++++++++++++++
 tests/image-fuzzer/qcow2/layout.py       | 359 ++++++++++++++++++++++++++++++
 tests/image-fuzzer/runner/runner.py      | 360 +++++++++++++++++++++++++++++++
 5 files changed, 1288 insertions(+)
 create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
 create mode 100644 tests/image-fuzzer/qcow2/__init__.py
 create mode 100644 tests/image-fuzzer/qcow2/fuzz.py
 create mode 100644 tests/image-fuzzer/qcow2/layout.py
 create mode 100755 tests/image-fuzzer/runner/runner.py

-- 
1.9.3

[Qemu-devel] [PATCH V4 1/5] docs: Specification for the image fuzzer

[Maria Kustova]

'Overall fuzzer requirements' chapter contains the current product vision and
features done and to be done. This chapter is still in progress.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/docs/image-fuzzer.txt | 239 +++++++++++++++++++++++++++++++
 1 file changed, 239 insertions(+)
 create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt

diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
new file mode 100644
index 0000000..2c4f346
--- /dev/null
+++ b/tests/image-fuzzer/docs/image-fuzzer.txt
@@ -0,0 +1,239 @@
+# Specification for the fuzz testing tool
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Image fuzzer
+============
+
+Description
+-----------
+
+The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img
+by providing to them randomly corrupted images.
+Test images are generated from scratch and have valid inner structure with some
+elements, e.g. L1/L2 tables, having random invalid values.
+
+
+Test runner
+-----------
+
+The test runner generates test images, executes tests utilizing generated
+images, indicates their results and collects all test related artifacts (logs,
+core dumps, test images).
+The test means execution of all available commands under test with the same
+generated test image.
+By default, the test runner generates new tests and executes them until
+keyboard interruption. But if a test seed is specified via the '--seed' runner
+parameter, then only one test with this seed will be executed, after its finish
+the runner will exit.
+
+The runner uses an external image fuzzer to generate test images. An image
+generator should be specified as a mandatory parameter of the test runner.
+Details about interactions between the runner and fuzzers see "Module
+interfaces".
+
+The runner activates generation of core dumps during test executions, but it
+assumes that core dumps will be generated in the current working directory.
+For comprehensive test results, please, set up your test environment
+properly.
+
+Paths to binaries under test (SUTs) qemu-img and qemu-io are retrieved from
+environment variables. If the environment check fails the runner will
+use SUTs installed in system paths.
+qemu-img is required for creation of backing files, so it's mandatory to set
+the related environment variable if it's not installed in the system path.
+For details about environment variables see qemu-iotests/check.
+
+The runner accepts via the '--config' argument a JSON array of fields expected
+to be fuzzed, e.g.
+
+       '[["feature_name_table"], ["header", "l1_table_offset"]]'
+
+Each sublist can be have one or two strings defining image structure elements.
+In the latter case a parent element should be placed on the first position,
+and a field name on the second one.
+
+The runner accepts a list of commands under test as a JSON array via
+the '--command' argument. Each command is a list containing a SUT and all its
+arguments, e.g.
+
+       runner.py -c '[["qemu-io", "$test_img", "-c", "write $off $len"]]'
+     /tmp/test ../qcow2
+
+For variable arguments next aliases can be used:
+    - $test_img for a fuzzed img
+    - $off for an offset in the fuzzed image
+    - $len for a data size
+
+Values for last two aliases will be generated based on the size of the virtal
+disk in the fuzzed image.
+In case when no commands are specified the runner will execute commands from
+the default list:
+    - qemu-img check
+    - qemu-img info
+    - qemu-img convert
+    - qemu-io -c read
+    - qemu-io -c write
+    - qemu-io -c aio_read
+    - qemu-io -c aio_write
+    - qemu-io -c flush
+    - qemu-io -c discard
+    - qemu-io -c truncate
+
+
+Qcow2 image generator
+---------------------
+
+The 'qcow2' generator is a Python package providing 'create_image' method as
+a single public API. See details in 'Test runner/image fuzzer' in 'Module
+interfaces'.
+
+Qcow2 contains two submodules: fuzz.py and layout.py.
+
+'fuzz.py' contains all fuzzing functions, one per image field. It's assumed
+that after code analysis every field will have own constraints for its value.
+For now only universal potentially dangerous values are used, e.g. type limits
+for integers or unsafe symbols as '%s' for strings. For bitmasks random amount
+of bits are set to ones. All fuzzed values are checked on non-equality to the
+current valid value of the field. In case of equality the value will be
+regenerated.
+
+'layout.py' creates a random valid image, fuzzes a random subset of the image
+fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
+If a fuzzer configuration is specified, then it has the next interpretation:
+
+    1. If a list contains a parent image element only, then some random portion
+    of fields of this element will be fuzzed every test.
+    The same behavior is applied for the entire image if no configuration is
+    used. This case is useful for the test specialization.
+
+    2. If a list contains a parent element and a field name, then a field
+    will be always fuzzed for every test. This case is useful for regression
+    testing.
+
+For now only header fields and header extensions are generated.
+
+
+Module interfaces
+-----------------
+
+* Test runner/image fuzzer
+
+The runner calls an image generator specifying the path to a test image file,
+path to a backing file and its format and a fuzzer configuration.
+An image generator is expected to provide a
+
+   'create_image(test_img_path, backing_file_path=None,
+                 backing_file_format=None, fuzz_config=None)'
+
+method that creates a test image, writes it to the specified file and returns
+the size of the virtual disk.
+The file should be created if it doesn't exist or overwritten otherwise.
+fuzz_config has a form of a list of lists. Every sublist can have one
+or two elements: first element is a name of a parent image element, second one
+if exists is a name of a field in this element.
+Example,
+        [['header', 'l1_table_offset'],
+         ['header', 'nb_snapshots'],
+         ['feature_name_table']]
+
+Random seed is set by the runner at every test execution for the regression
+purpose, so an image generator is not recommended to modify it internally.
+
+
+Overall fuzzer requirements
+===========================
+
+Input data:
+----------
+
+ - image template (generator)
+ - work directory
+ - action vector (optional)
+ - seed (optional)
+ - SUT and its arguments (optional)
+
+
+Fuzzer requirements:
+-------------------
+
+1.  Should be able to inject random data
+2.  Should be able to select a random value from the manually pregenerated
+    vector (boundary values, e.g. max/min cluster size)
+3.  Image template should describe a general structure invariant for all
+    test images (image format description)
+4.  Image template should be autonomous and other fuzzer parts should not
+    rely on it
+5.  Image template should contain reference rules (not only block+size
+    description)
+6.  Should generate the test image with the correct structure based on an image
+    template
+7.  Should accept a seed as an argument (for regression purpose)
+8.  Should generate a seed if it is not specified as an input parameter.
+9.  The same seed should generate the same image for the same action vector,
+    specified or generated.
+10. Should accept a vector of actions as an argument (for test reproducing and
+    for test case specification, e.g. group of tests for header structure,
+    group of test for snapshots, etc)
+11. Action vector should be randomly generated from the pool of available
+    actions, if it is not specified as an input parameter
+12. Pool of actions should be defined automatically based on an image template
+13. Should accept a SUT and its call parameters as an argument or select them
+    randomly otherwise. As far as it's expected to be rarely changed, the list
+    of all possible test commands can be available in the test runner
+    internally.
+14. Should support an external cancellation of a test run
+15. Seed and action vector should be logged (for regression purpose)
+16. All files related to test result should be collected: a test image,
+    SUT logs, fuzzer logs and crash dumps
+17. Should be compatible with python version 2.4-2.7
+18. Usage of external libraries should be limited as much as possible.
+
+
+Image formats:
+-------------
+
+Main target image format is qcow2, but support of image templates should
+provide an ability to add any other image format.
+
+
+Effectiveness:
+-------------
+
+Fuzzer can be controlled via template, seed and action vector;
+it makes the fuzzer itself invariant to an image format and test logic.
+It should be able to perform rather complex and precise tests, that can be
+specified via action vector. Otherwise, knowledge about an image structure
+allows the fuzzer to generate the pool of all available areas can be fuzzed
+and randomly select some of them and so compose its own action vector.
+Also complexity of a template defines complexity of the fuzzer, so its
+functionality can be varied from simple model-independent fuzzing to smart
+model-based one.
+
+
+Glossary:
+--------
+
+Action vector is a sequence of structure elements retrieved from an image
+format, each of them will be fuzzed for the test image. It's a subset of
+elements of the action pool. Example: header, refcount table, etc.
+Action pool is all available elements of an image structure that generated
+automatically from an image template.
+Image template is a formal description of an image structure and relations
+between image blocks.
+Test image is an output image of the fuzzer defined by the current seed and
+action vector.
-- 
1.9.3

[Qemu-devel] [PATCH V4 2/5] runner: Tool for fuzz tests execution

[Maria Kustova]

The purpose of the test runner is to prepare the test environment (e.g. create
a work directory, a test image, etc), execute a program under test with
parameters, indicate a test failure if the program was killed during the test
execution and collect core dumps, logs and other test artifacts.

The test runner doesn't depend on an image format or a program will be tested,
so it can be used with any external image generator and program under test.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/runner/runner.py | 360 ++++++++++++++++++++++++++++++++++++
 1 file changed, 360 insertions(+)
 create mode 100755 tests/image-fuzzer/runner/runner.py

diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
new file mode 100755
index 0000000..3e9e65d
--- /dev/null
+++ b/tests/image-fuzzer/runner/runner.py
@@ -0,0 +1,360 @@
+#!/usr/bin/env python
+
+# Tool for running fuzz tests
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys, os, signal
+import subprocess
+import random
+from itertools import count
+from shutil import rmtree
+import getopt
+try:
+    import json
+except ImportError:
+    try:
+        import simplejson as json
+    except ImportError:
+        print "Warning: Module for JSON processing is not found.\n" + \
+            "'--config' and '--command' options are not supported."
+import resource
+resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
+
+
+def multilog(msg, *output):
+    """ Write an object to all of specified file descriptors
+    """
+
+    for fd in output:
+        fd.write(msg)
+        fd.flush()
+
+
+def str_signal(sig):
+    """ Convert a numeric value of a system signal to the string one
+    defined by the current operational system
+    """
+
+    for k, v in signal.__dict__.items():
+        if v == sig:
+            return k
+
+
+class TestException(Exception):
+    """Exception for errors risen by TestEnv objects"""
+    pass
+
+
+class TestEnv(object):
+    """ Trivial test object
+
+    The class sets up test environment, generates backing and test images
+    and executes application under tests with specified arguments and a test
+    image provided.
+    All logs are collected.
+    Summary log will contain short descriptions and statuses of tests in
+    a run.
+    Test log will include application (e.g. 'qemu-img') logs besides info sent
+    to the summary log.
+    """
+
+    def __init__(self, test_id, seed, work_dir, run_log,
+                 cleanup=True, log_all=False):
+        """Set test environment in a specified work directory.
+
+        Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
+        'QEMU_IO' environment variables
+        """
+        if seed is not None:
+            self.seed = seed
+        else:
+            self.seed = str(random.randint(0, sys.maxint))
+        random.seed(self.seed)
+
+        self.init_path = os.getcwd()
+        self.work_dir = work_dir
+        self.current_dir = os.path.join(work_dir, 'test-' + test_id)
+        self.qemu_img = \
+                        os.environ.get('QEMU_IMG', 'qemu-img')\
+                                  .strip().split(' ')
+        self.qemu_io = \
+                       os.environ.get('QEMU_IO', 'qemu-io').strip().split(' ')
+        self.commands = [['qemu-img', 'check', '-f', 'qcow2', '$test_img'],
+                         ['qemu-img', 'info', '-f', 'qcow2', '$test_img'],
+                         ['qemu-io', '$test_img', '-c', 'read $off $len'],
+                         ['qemu-io', '$test_img', '-c', 'write $off $len'],
+                         ['qemu-io', '$test_img', '-c',
+                          'aio_read $off $len'],
+                         ['qemu-io', '$test_img', '-c',
+                          'aio_write $off $len'],
+                         ['qemu-io', '$test_img', '-c', 'flush'],
+                         ['qemu-io', '$test_img', '-c',
+                          'discard $off $len'],
+                         ['qemu-io', '$test_img', '-c',
+                          'truncate $off']]
+        for fmt in ['raw', 'vmdk', 'vdi', 'cow', 'qcow2', 'file',
+                    'qed', 'vpc']:
+            self.commands.append(
+                         ['qemu-img', 'convert', '-f', 'qcow2', '-O', fmt,
+                          '$test_img', 'converted_image.' + fmt])
+
+        try:
+            os.makedirs(self.current_dir)
+        except OSError, e:
+            print >>sys.stderr, \
+                "Error: The working directory '%s' cannot be used. Reason: %s"\
+                % (self.work_dir, e[1])
+            raise TestException
+        self.log = open(os.path.join(self.current_dir, "test.log"), "w")
+        self.parent_log = open(run_log, "a")
+        self.failed = False
+        self.cleanup = cleanup
+        self.log_all = log_all
+
+    def _test_app(self, q_args):
+        """ Start application under test with specified arguments and return
+        an exit code or a kill signal depending on the result of execution.
+        """
+        devnull = open('/dev/null', 'r+')
+        return subprocess.call(q_args, stdin=devnull, stdout=self.log,
+                               stderr=self.log)
+
+    def _create_backing_file(self):
+        """Create a backing file in the current directory and return
+        its name and file format
+
+        Format of a backing file is randomly chosen from all formats supported
+        by 'qemu-img create'
+        """
+        # All formats qemu-img can create images of.
+        backing_file_fmt = random.choice(['raw', 'vmdk', 'vdi', 'cow', 'qcow2',
+                                          'file', 'qed', 'vpc'])
+        backing_file_name = 'backing_img.' + backing_file_fmt
+        # Size of the backing file varies from 1 to 10 MB
+        backing_file_size = random.randint(1, 10)*(1 << 20)
+        cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
+                               backing_file_name, str(backing_file_size)]
+        devnull = open('/dev/null', 'r+')
+        retcode = subprocess.call(cmd, stdin=devnull, stdout=self.log,
+                                  stderr=self.log)
+        if retcode == 0:
+            return [backing_file_name, backing_file_fmt]
+        else:
+            return [None, None]
+
+    def execute(self, input_commands=None, fuzz_config=None):
+        """ Execute a test.
+
+        The method creates backing and test images, runs test app and analyzes
+        its exit status. If the application was killed by a signal, the test
+        is marked as failed.
+        """
+        if input_commands is None:
+            commands = self.commands
+        else:
+            commands = input_commands
+        os.chdir(self.current_dir)
+        backing_file_name, backing_file_fmt = self._create_backing_file()
+        img_size = image_generator.create_image('test_image',
+                                                backing_file_name,
+                                                backing_file_fmt,
+                                                fuzz_config)
+        for item in commands:
+            start = random.randint(0, img_size)
+            end = random.randint(start, img_size)
+            current_cmd = list(self.__dict__[item[0].replace('-', '_')])
+            # Replace all placeholders with their real values
+            for v in item[1:]:
+                c = v.replace('$test_img', 'test_image').\
+                    replace('$off', str(start)).\
+                    replace('$len', str(end - start))
+                current_cmd.append(c)
+            # Log string with the test header
+            test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
+                           "Backing file: %s\n" \
+                           % (self.seed, " ".join(current_cmd),
+                              self.current_dir, backing_file_name)
+            try:
+                retcode = self._test_app(current_cmd)
+            except OSError, e:
+                multilog(test_summary + "Error: Start of '%s' failed. " \
+                         "Reason: %s\n\n" % (os.path.basename(
+                             current_cmd[0]), e[1]),
+                         sys.stderr, self.log, self.parent_log)
+                raise TestException
+
+            if retcode < 0:
+                multilog(test_summary + "FAIL: Test terminated by signal " +
+                         "%s\n\n" % str_signal(-retcode), sys.stderr, self.log,
+                         self.parent_log)
+                self.failed = True
+            else:
+                if self.log_all:
+                    multilog(test_summary + "PASS: Application exited with" + \
+                             " the code '%d'\n\n" % retcode, sys.stdout,
+                             self.log, self.parent_log)
+
+    def finish(self):
+        """ Restore environment after a test execution. Remove folders of
+        passed tests
+        """
+        self.log.close()
+        self.parent_log.close()
+        os.chdir(self.init_path)
+        if self.cleanup and not self.failed:
+            rmtree(self.current_dir)
+
+if __name__ == '__main__':
+
+    def usage():
+        print """
+        Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
+
+        Set up test environment in TEST_DIR and run a test in it. A module for
+        test image generation should be specified via IMG_GENERATOR.
+        Example:
+        runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test ../qcow2
+
+        Optional arguments:
+          -h, --help                    display this help and exit
+          -c, --command=JSON            run tests for all commands specified in
+                                        the JSON array
+          -s, --seed=STRING             seed for a test image generation,
+                                        by default will be generated randomly
+          --config=JSON                 take fuzzer configuration from the JSON
+                                        array
+          -k, --keep_passed             don't remove folders of passed tests
+          -v, --verbose                 log information about passed tests
+
+        JSON:
+
+        '--command' accepts a JSON array of commands. Each command presents
+        an application under test with all its paramaters as a list of strings,
+        e.g.
+          ["qemu-io", "$test_img", "-c", "write $off $len"]
+
+        Supported application aliases: 'qemu-img' and 'qemu-io'.
+        Supported argument aliases: $test_img for the fuzzed image, $off
+        for an offset, $len for length.
+
+        Values for $off and $len will be generated based on the virtual disk
+        size of the fuzzed image
+        Paths to 'qemu-img' and 'qemu-io' are retrevied from 'QEMU_IMG' and
+        'QEMU_IO' environment variables
+
+        '--config' accepts a JSON array of fields to be fuzzed, e.g.
+          '[["header"], ["header", "version"]]'
+        Each of the list elements can consist of a complex image element only
+        as ["header"] or ["feature_name_table"] or an exact field as
+        ["header", "version"]. In the first case random portion of the element
+        fields will be fuzzed, in the second one the specified field will be
+        fuzzed always.
+
+        If '--config' argument is specified, fields not listed in
+        the configuration array will not be fuzzed.
+        """
+
+    def run_test(test_id, seed, work_dir, run_log, cleanup, log_all,
+                 command, fuzz_config):
+        """Setup environment for one test and execute this test"""
+        try:
+            test = TestEnv(test_id, seed, work_dir, run_log, cleanup,
+                           log_all)
+        except TestException:
+            sys.exit(1)
+
+        # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
+        # block
+        try:
+            try:
+                test.execute(command, fuzz_config)
+            # Silent exit on user break
+            except TestException:
+                sys.exit(1)
+        finally:
+            test.finish()
+
+    try:
+        opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
+                                       ['command=', 'help', 'seed=', 'config=',
+                                        'keep_passed', 'verbose'])
+    except getopt.error, e:
+        print "Error: %s\n\nTry 'runner.py --help' for more information" % e
+        sys.exit(1)
+
+    command = None
+    cleanup = True
+    log_all = False
+    seed = None
+    config = None
+    for opt, arg in opts:
+        if opt in ('-h', '--help'):
+            usage()
+            sys.exit()
+        elif opt in ('-c', '--command'):
+            try:
+                command = json.loads(arg)
+            except (TypeError, ValueError, NameError), e:
+                print "Error: JSON array of test commands cannot be loaded"\
+                    "\nReason: %s" % e
+                sys.exit(1)
+        elif opt in ('-k', '--keep_passed'):
+            cleanup = False
+        elif opt in ('-v', '--verbose'):
+            log_all = True
+        elif opt in ('-s', '--seed'):
+            seed = arg
+        elif opt == '--config':
+            try:
+                config = json.loads(arg)
+            except (TypeError, ValueError, NameError), e:
+                print "Error: JSON array with the fuzzer configuration " \
+                    "cannot be loaded\nReason: %s" % e
+                sys.exit(1)
+
+    if not len(args) == 2:
+        print "Expected two parameters\nTry 'runner.py --help' " \
+            "for more information"
+        sys.exit(1)
+
+    work_dir = os.path.realpath(args[0])
+    # run_log is created in 'main', because multiple tests are expected to
+    # log in it
+    run_log = os.path.join(work_dir, 'run.log')
+
+    # Add the path to the image generator module to sys.path
+    sys.path.append(os.path.dirname(os.path.realpath(args[1])))
+    # Remove a script extension from image generator module if any
+    generator_name = os.path.splitext(os.path.basename(args[1]))[0]
+    try:
+        image_generator = __import__(generator_name)
+    except ImportError, e:
+        print "Error: The image generator '%s' cannot be imported.\n" \
+            "Reason: %s" % (generator_name, e)
+        sys.exit(1)
+    # If a seed is specified, only one test will be executed.
+    # Otherwise runner will terminate after a keyboard interruption
+    for test_id in count(1):
+        try:
+            run_test(str(test_id), seed, work_dir, run_log, cleanup,
+                     log_all, command, config)
+        except (KeyboardInterrupt, SystemExit):
+            sys.exit(1)
+
+        if seed is not None:
+            break
-- 
1.9.3

[Qemu-devel] [PATCH V4 3/5] fuzz: Fuzzing functions for qcow2 images

[Maria Kustova]

The fuzz submodule of the qcow2 image generator contains fuzzing functions for
image fields.
Each fuzzing function contains a list of constraints and a call of a helper
function that randomly selects a fuzzed value satisfied to one of constraints.
For now constraints include only known as invalid or potentially dangerous
values. But after investigation of code coverage by fuzz tests they will be
expanded by heuristic values based on inner checks and flows of a program
under test.

Now fuzzing of a header, header extensions and a backing file name is
supported.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/fuzz.py | 329 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 329 insertions(+)
 create mode 100644 tests/image-fuzzer/qcow2/fuzz.py

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
new file mode 100644
index 0000000..ef9198f
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -0,0 +1,329 @@
+# Fuzzing functions for qcow2 fields
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+
+
+UINT32 = 0xffffffff
+UINT64 = 0xffffffffffffffff
+# Most significant bit orders
+UINT32_M = 31
+UINT64_M = 63
+# Fuzz vectors
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
+            UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
+                       UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+                       UINT64]
+STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
+            '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
+            '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
+            '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+            '%s x 129', '%x x 257']
+
+
+def random_from_intervals(intervals):
+    """Select a random integer number from the list of specified intervals
+
+    Each interval is a tuple of lower and upper limits of the interval. The
+    limits are included. Intervals in a list should not overlap.
+    """
+    total = reduce(lambda x, y: x + y[1] - y[0] + 1, intervals, 0)
+    r = random.randint(0, total-1) + intervals[0][0]
+    temp = zip(intervals, intervals[1:])
+    for x in temp:
+        r = r + (r > x[0][1])*(x[1][0] - x[0][1] - 1)
+    return r
+
+
+def random_bits(bit_ranges):
+    """Generate random binary mask with ones in the specified bit ranges
+
+    Each bit_ranges is a list of tuples of lower and upper limits of bit
+    positions will be fuzzed. The limits are included. Random amount of bits
+    in range limits will be set to ones. The mask is returned in decimal
+    integer format.
+    """
+    bit_numbers = []
+    # Select random amount of random positions in bit_ranges
+    for rng in bit_ranges:
+        bit_numbers += random.sample(range(rng[0], rng[1] + 1),
+                                     random.randint(0, rng[1] - rng[0] + 1))
+    val = 0
+    # Set bits on selected positions to ones
+    for bit in bit_numbers:
+        val |= 1 << bit
+    return val
+
+
+def truncate_string(strings, length):
+    """Return strings truncated to specified length"""
+    if type(strings) == list:
+        return [s[:length] for s in strings]
+    else:
+        return strings[:length]
+
+
+def validator(current, pick, choices):
+    """Returns a value not equal to the current selected by the pick
+    function from choices
+    """
+    while True:
+        val = pick(choices)
+        if not val == current:
+            return val
+
+
+def int_validator(current, intervals):
+    """Return a random value from intervals not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+    return validator(current, random_from_intervals, intervals)
+
+
+def bit_validator(current, bit_ranges):
+    """Return a random bit mask not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+    return validator(current, random_bits, bit_ranges)
+
+
+def string_validator(current, strings):
+    """Return a random string value from the list not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+    return validator(current, random.choice, strings)
+
+
+def selector(current, constraints, validate=int_validator):
+    """Select one value from all defined by constraints
+
+    Each constraint produces one random value satisfying to it. The function
+    randomly selects one value satisfying at least one constraint (depending on
+    constraints overlaps).
+    """
+
+    def iter_validate(c):
+        """Apply validate() only to constraints represented as lists
+
+        This auxiliary function replaces short circuit conditions not supported
+        in Python 2.4
+        """
+        if type(c) == list:
+            return validate(current, c)
+        else:
+            return c
+    fuzz_values = [iter_validate(c) for c in constraints]
+    # Remove current for cases it's implicitly specified in constraints
+    # Duplicate validator functionality to prevent decreasing of probability
+    # to get one of allowable values
+    # TODO: remove validators after implementation of intelligent selection
+    # of fields will be fuzzed
+    try:
+        fuzz_values.remove(current)
+    except ValueError:
+        pass
+    return random.choice(fuzz_values)
+
+
+def magic(current):
+    """Fuzz magic header field
+
+    The function just returns the current magic value and provides uniformity
+    of calls for all fuzzing functions
+    """
+    return current
+
+
+def version(current):
+    """Fuzz version header field"""
+    constraints = UINT32_V + [
+        [(2, 3)],  # correct values
+        [(0, 1), (4, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def backing_file_offset(current):
+    """Fuzz backing file offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def backing_file_size(current):
+    """Fuzz backing file size header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def cluster_bits(current):
+    """Fuzz cluster bits header field"""
+    constraints = UINT32_V + [
+        [(9, 20)],  # correct values
+        [(0, 9), (20, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def size(current):
+    """Fuzz image size header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def crypt_method(current):
+    """Fuzz crypt method header field"""
+    constraints = UINT32_V + [
+        1,
+        [(2, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def l1_size(current):
+    """Fuzz L1 table size header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def l1_table_offset(current):
+    """Fuzz L1 table offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def refcount_table_offset(current):
+    """Fuzz refcount table offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def refcount_table_clusters(current):
+    """Fuzz refcount table clusters header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def nb_snapshots(current):
+    """Fuzz number of snapshots header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def snapshots_offset(current):
+    """Fuzz snapshots offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def incompatible_features(current):
+    """Fuzz incompatible features header field"""
+    constraints = [
+        [(0, 1)],  # allowable values
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, bit_validator)
+
+
+def compatible_features(current):
+    """Fuzz compatible features header field"""
+    constraints = [
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, bit_validator)
+
+
+def autoclear_features(current):
+    """Fuzz autoclear features header field"""
+    constraints = [
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, bit_validator)
+
+
+def refcount_order(current):
+    """Fuzz number of refcount order header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def header_length(current):
+    """Fuzz number of refcount order header field"""
+    constraints = UINT32_V + [
+        72,
+        104,
+        [(0, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def bf_name(current):
+    """Fuzz the backing file name"""
+    constraints = [
+        truncate_string(STRING_V, len(current))
+    ]
+    return selector(current, constraints, string_validator)
+
+
+def ext_magic(current):
+    """Fuzz magic field of a header extension"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def ext_length(current):
+    """Fuzz length field of a header extension"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def bf_format(current):
+    """Fuzz backing file format in the corresponding header extension"""
+    constraints = [
+        truncate_string(STRING_V, len(current)),
+        truncate_string(STRING_V, (len(current) + 7) & ~7)  # Fuzz padding
+    ]
+    return selector(current, constraints, string_validator)
+
+
+def feature_type(current):
+    """Fuzz feature type field of a feature name table header extension"""
+    constraints = [
+        [(0, 255)]
+    ]
+    return selector(current, constraints)
+
+
+def feature_bit_number(current):
+    """Fuzz bit number field of a feature name table header extension"""
+    constraints = [
+        [(0, 255)]
+    ]
+    return selector(current, constraints)
+
+
+def feature_name(current):
+    """Fuzz feature name field of a feature name table header extension"""
+    constraints = [
+        truncate_string(STRING_V, len(current)),
+        truncate_string(STRING_V, 46)  # Fuzz padding (field length = 46)
+    ]
+    return selector(current, constraints, string_validator)
-- 
1.9.3

[Qemu-devel] [PATCH V4 4/5] layout: Generator of fuzzed qcow2 images

[Maria Kustova]

The layout submodule of the qcow2 package creates a random valid image,
randomly selects some amount of its fields, fuzzes them and write the fuzzed
image to the file. Fuzzing process can be controlled by an external
configuration.

Now only a header, header extensions and a backing file name are generated.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/layout.py | 359 +++++++++++++++++++++++++++++++++++++
 1 file changed, 359 insertions(+)
 create mode 100644 tests/image-fuzzer/qcow2/layout.py

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
new file mode 100644
index 0000000..2bad223
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -0,0 +1,359 @@
+# Generator of fuzzed qcow2 images
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+import struct
+import fuzz
+
+MAX_IMAGE_SIZE = 10*2**20
+# Standard sizes
+UINT32_S = 4
+UINT64_S = 8
+
+
+class Field(object):
+    """Atomic image element (field)
+
+    The class represents an image field as quadruple of a data format
+    of value necessary for its packing to binary form, an offset from
+    the beginning of the image, a value and a name.
+
+    The field can be iterated as a list [format, offset, value]
+    """
+    __slots__ = ('fmt', 'offset', 'value', 'name')
+
+    def __init__(self, fmt, offset, val, name):
+        self.fmt = fmt
+        self.offset = offset
+        self.value = val
+        self.name = name
+
+    def __iter__(self):
+        return (x for x in [self.fmt, self.offset, self.value])
+
+    def __repr__(self):
+        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
+            (self.fmt, self.offset, str(self.value), self.name)
+
+
+class FieldsList(object):
+    """List of fields
+
+    The class allows access to a field in the list by its name and joins
+    several list in one via in-place addition
+    """
+    def __init__(self, meta_data=None):
+        if meta_data is None:
+            self.data = []
+        else:
+            self.data = [Field(f[0], f[1], f[2], f[3])
+                         for f in meta_data]
+
+    def __getitem__(self, name):
+        return [x for x in self.data if x.name == name]
+
+    def __iter__(self):
+        return (x for x in self.data)
+
+    def __iadd__(self, other):
+        self.data += other.data
+        return self
+
+    def __len__(self):
+        return len(self.data)
+
+
+class Image(object):
+    """ Qcow2 image object
+
+    This class allows to create qcow2 images with random valid structures and
+    values, fuzz them via external qcow2.fuzz module and write the result to
+    a file
+    """
+    @staticmethod
+    def _size_params():
+        """Generate a random image size aligned to a random correct
+        cluster size
+        """
+        cluster_bits = random.randrange(9, 21)
+        cluster_size = 1 << cluster_bits
+        img_size = random.randrange(0, MAX_IMAGE_SIZE + 1,
+                                    cluster_size)
+        return [cluster_bits, img_size]
+
+    @staticmethod
+    def _header(cluster_bits, img_size, backing_file_name=None):
+        """Generate a random valid header"""
+        meta_header = [
+            ['>4s', 0, "QFI\xfb", 'magic'],
+            ['>I', 4, random.randint(2, 3), 'version'],
+            ['>Q', 8, 0, 'backing_file_offset'],
+            ['>I', 16, 0, 'backing_file_size'],
+            ['>I', 20, cluster_bits, 'cluster_bits'],
+            ['>Q', 24, img_size, 'size'],
+            ['>I', 32, 0, 'crypt_method'],
+            ['>I', 36, 0, 'l1_size'],
+            ['>Q', 40, 0, 'l1_table_offset'],
+            ['>Q', 48, 0, 'refcount_table_offset'],
+            ['>I', 56, 0, 'refcount_table_clusters'],
+            ['>I', 60, 0, 'nb_snapshots'],
+            ['>Q', 64, 0, 'snapshots_offset'],
+            ['>Q', 72, 0, 'incompatible_features'],
+            ['>Q', 80, 0, 'compatible_features'],
+            ['>Q', 88, 0, 'autoclear_features'],
+            # Only refcount_order = 4 is supported by current (07.2014)
+            # implementation of QEMU
+            ['>I', 96, 4, 'refcount_order'],
+            ['>I', 100, 0, 'header_length']
+        ]
+        v_header = FieldsList(meta_header)
+
+        if v_header['version'][0].value == 2:
+            v_header['header_length'][0].value = 72
+        else:
+            v_header['incompatible_features'][0].value = random.getrandbits(2)
+            v_header['compatible_features'][0].value = random.getrandbits(1)
+            v_header['header_length'][0].value = 104
+
+        max_header_len = struct.calcsize(v_header['header_length'][0].fmt) + \
+                         v_header['header_length'][0].offset
+        end_of_extension_area_len = 2*UINT32_S
+        free_space = (1 << cluster_bits) - (max_header_len +
+                                            end_of_extension_area_len)
+        # If the backing file name specified and there is enough space for it
+        # in the first cluster, then it's placed in the very end of the first
+        # cluster.
+        if (backing_file_name is not None) and \
+           (free_space >= len(backing_file_name)):
+            v_header['backing_file_size'][0].value = len(backing_file_name)
+            v_header['backing_file_offset'][0].value = (1 << cluster_bits) - \
+                                                       len(backing_file_name)
+
+        return v_header
+
+    @staticmethod
+    def _backing_file_name(header, backing_file_name=None):
+        """Add the name of the backing file at the offset specified
+        in the header
+        """
+        if (backing_file_name is not None) and \
+           (not header['backing_file_offset'][0].value == 0):
+            data_len = len(backing_file_name)
+            data_fmt = '>' + str(data_len) + 's'
+            data_field = FieldsList([
+                [data_fmt, header['backing_file_offset'][0].value,
+                 backing_file_name, 'bf_name']
+            ])
+        else:
+            data_field = FieldsList()
+
+        return data_field
+
+    @staticmethod
+    def _backing_file_format(header, backing_file_fmt=None):
+        """Generate the header extension for the backing file
+        format
+        """
+        # Calculation of the free space available in the first cluster
+        offset = struct.calcsize(header['header_length'][0].fmt) + \
+                 header['header_length'][0].offset
+        end_of_extension_area_len = 2*UINT32_S
+        high_border = (header['backing_file_offset'][0].value or
+                       ((1 << header['cluster_bits'][0].value) - 1)) - \
+            end_of_extension_area_len
+        free_space = high_border - offset
+        ext_size = 2*UINT32_S + ((len(backing_file_fmt) + 7) & ~7)
+
+        if (backing_file_fmt is not None) and (free_space >= ext_size):
+            ext_data_len = len(backing_file_fmt)
+            ext_data_fmt = '>' + str(ext_data_len) + 's'
+            ext_padding_len = 7 - (ext_data_len - 1) % 8
+            ext = FieldsList([
+                ['>I', offset, 0xE2792ACA, 'ext_magic'],
+                ['>I', offset + UINT32_S, ext_data_len, 'ext_length'],
+                [ext_data_fmt, offset + UINT32_S*2, backing_file_fmt,
+                 'bf_format']
+            ])
+            offset = ext['bf_format'][0].offset + \
+                     struct.calcsize(ext['bf_format'][0].fmt) + ext_padding_len
+        else:
+            ext = FieldsList()
+        return (ext, offset)
+
+    @staticmethod
+    def _feature_name_table(header, offset):
+        """Generate a random header extension for names of features used in
+        the image
+        """
+        def gen_feat_ids():
+            """Return random feature type and feature bit"""
+            return (random.randint(0, 2), random.randint(0, 63))
+
+        end_of_extension_area_len = 2*UINT32_S
+        high_border = (header['backing_file_offset'][0].value or
+                       (1 << header['cluster_bits'][0].value) - 1) - \
+            end_of_extension_area_len
+        free_space = high_border - offset
+        # Sum of sizes of 'magic' and 'length' header extension fields
+        ext_header_len = 2*UINT32_S
+        fnt_entry_size = 6*UINT64_S
+        num_fnt_entries = min(10, (free_space - ext_header_len)/fnt_entry_size)
+        if not num_fnt_entries == 0:
+            feature_tables = []
+            feature_ids = []
+            inner_offset = offset + ext_header_len
+            feat_name = 'some cool feature'
+            while len(feature_tables) < num_fnt_entries*3:
+                feat_type, feat_bit = gen_feat_ids()
+                # Remove duplicates
+                while (feat_type, feat_bit) in feature_ids:
+                    feat_type, feat_bit = gen_feat_ids()
+                feature_ids.append((feat_type, feat_bit))
+                feat_fmt = '>' + str(len(feat_name)) + 's'
+                feature_tables += [['B', inner_offset,
+                                    feat_type, 'feature_type'],
+                                   ['B', inner_offset + 1, feat_bit,
+                                    'feature_bit_number'],
+                                   [feat_fmt, inner_offset + 2,
+                                    feat_name, 'feature_name']
+                ]
+                inner_offset += fnt_entry_size
+            # No padding for the extension is necessary, because
+            # the extension length is multiple of 8
+            ext = FieldsList([
+                ['>I', offset, 0x6803f857, 'ext_magic'],
+                ['>I', offset + UINT32_S, len(feature_tables)*48, 'ext_length']
+            ] + feature_tables)
+            offset = inner_offset
+        else:
+            ext = FieldsList()
+
+        return (ext, offset)
+
+    @staticmethod
+    def _end_of_extension_area(offset):
+        """Generate a mandatory header extension marking end of header
+        extensions
+        """
+        ext = FieldsList([
+            ['>I', offset, 0, 'ext_magic'],
+            ['>I', offset + UINT32_S, 0, 'ext_length']
+        ])
+        return ext
+
+    def __init__(self, backing_file_name=None, backing_file_fmt=None):
+        """Create a random valid qcow2 image with the correct inner structure
+        and allowable values
+        """
+        # Image size is saved as an attribute for the runner needs
+        cluster_bits, self.image_size = self._size_params()
+        # Saved as an attribute, because it's necessary for writing
+        self.cluster_size = 1 << cluster_bits
+        self.header = self._header(cluster_bits, self.image_size,
+                                   backing_file_name)
+        self.backing_file_name = self._backing_file_name(self.header,
+                                                         backing_file_name)
+        self.backing_file_format, \
+            offset = self._backing_file_format(self.header,
+                                               backing_file_fmt)
+        self.feature_name_table, \
+            offset = self._feature_name_table(self.header, offset)
+        self.end_of_extension_area = self._end_of_extension_area(offset)
+        # Container for entire image
+        self.data = FieldsList()
+        # Percentage of fields will be fuzzed
+        self.bias = random.uniform(0.2, 0.5)
+
+    def __iter__(self):
+        return (x for x in [self.header,
+                            self.backing_file_format,
+                            self.feature_name_table,
+                            self.end_of_extension_area,
+                            self.backing_file_name])
+
+    def _join(self):
+        """Join all image structure elements as header, tables, etc in one
+        list of fields
+        """
+        if len(self.data) == 0:
+            for v in self:
+                self.data += v
+
+    def fuzz(self, fields_to_fuzz=None):
+        """Fuzz an image by corrupting values of a random subset of its fields
+
+        Without parameters the method fuzzes an entire image.
+        If 'fields_to_fuzz' is specified then only fields in this list will be
+        fuzzed. 'fields_to_fuzz' can contain both individual fields and more
+        general image elements as a header or tables.
+        In the first case the field will be fuzzed always.
+        In the second a random subset of fields will be selected and fuzzed.
+        """
+        def coin():
+            """Return boolean value proportional to a portion of fields to be
+            fuzzed
+            """
+            return random.random() < self.bias
+
+        if fields_to_fuzz is None:
+            self._join()
+            for field in self.data:
+                if coin():
+                    field.value = getattr(fuzz, field.name)(field.value)
+        else:
+            for item in fields_to_fuzz:
+                if len(item) == 1:
+                    for field in self.__dict__[item[0]]:
+                        if coin():
+                            field.value = getattr(fuzz,
+                                                  field.name)(field.value)
+                else:
+                    for field in self.__dict__[item[0]][item[1]]:
+                        try:
+                            field.value = getattr(fuzz, field.name)(
+                                field.value)
+                        except AttributeError:
+                            # Some fields can be skipped depending on
+                            # references, e.g. FNT header extension is not
+                            # generated for a feature mask header field
+                            # equal to zero
+                            pass
+
+    def write(self, filename):
+        """Writes an entire image to the file"""
+        image_file = open(filename, 'w')
+        self._join()
+        for field in self.data:
+            image_file.seek(field.offset)
+            image_file.write(struct.pack(field.fmt, field.value))
+        image_file.seek(0, 2)
+        size = image_file.tell()
+        roundup = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
+        if roundup > size:
+            image_file.seek(roundup - 1)
+            image_file.write("\0")
+        image_file.close()
+
+
+def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
+                 fields_to_fuzz=None):
+    """Create a fuzzed image and write it to the specified file"""
+    image = Image(backing_file_name, backing_file_fmt)
+    image.fuzz(fields_to_fuzz)
+    image.write(test_img_path)
+    return image.image_size
-- 
1.9.3

[Qemu-devel] [PATCH V4 5/5] package: Public API for image-fuzzer/runner/runner.py

[Maria Kustova]

__init__.py provides the public API required by the test runner

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/__init__.py | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 tests/image-fuzzer/qcow2/__init__.py

diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
new file mode 100644
index 0000000..e2ebe19
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -0,0 +1 @@
+from layout import create_image
-- 
1.9.3

Re: [Qemu-devel] [PATCH V4 2/5] runner: Tool for fuzz tests execution

[Fam Zheng]

On Mon, 07/21 14:18, Maria Kustova wrote:
> The purpose of the test runner is to prepare the test environment (e.g. create
> a work directory, a test image, etc), execute a program under test with
> parameters, indicate a test failure if the program was killed during the test
> execution and collect core dumps, logs and other test artifacts.
> 
> The test runner doesn't depend on an image format or a program will be tested,
> so it can be used with any external image generator and program under test.
> 
> Signed-off-by: Maria Kustova <maria.k@catit.be>

Looks good. Only two minor comments below but neither is a stopper.

> ---
>  tests/image-fuzzer/runner/runner.py | 360 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 360 insertions(+)
>  create mode 100755 tests/image-fuzzer/runner/runner.py
> 
> diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
> new file mode 100755
> index 0000000..3e9e65d
> --- /dev/null
> +++ b/tests/image-fuzzer/runner/runner.py
> @@ -0,0 +1,360 @@
> +#!/usr/bin/env python
> +
> +# Tool for running fuzz tests
> +#
> +# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
> +#
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import sys, os, signal
> +import subprocess
> +import random
> +from itertools import count
> +from shutil import rmtree
> +import getopt
> +try:
> +    import json
> +except ImportError:
> +    try:
> +        import simplejson as json
> +    except ImportError:
> +        print "Warning: Module for JSON processing is not found.\n" + \
> +            "'--config' and '--command' options are not supported."
> +import resource
> +resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
> +
> +
> +def multilog(msg, *output):
> +    """ Write an object to all of specified file descriptors
> +    """
> +
> +    for fd in output:
> +        fd.write(msg)
> +        fd.flush()
> +
> +
> +def str_signal(sig):
> +    """ Convert a numeric value of a system signal to the string one
> +    defined by the current operational system
> +    """
> +
> +    for k, v in signal.__dict__.items():
> +        if v == sig:
> +            return k
> +
> +
> +class TestException(Exception):
> +    """Exception for errors risen by TestEnv objects"""
> +    pass
> +
> +
> +class TestEnv(object):
> +    """ Trivial test object
> +
> +    The class sets up test environment, generates backing and test images
> +    and executes application under tests with specified arguments and a test
> +    image provided.
> +    All logs are collected.
> +    Summary log will contain short descriptions and statuses of tests in
> +    a run.
> +    Test log will include application (e.g. 'qemu-img') logs besides info sent
> +    to the summary log.
> +    """
> +
> +    def __init__(self, test_id, seed, work_dir, run_log,
> +                 cleanup=True, log_all=False):
> +        """Set test environment in a specified work directory.
> +
> +        Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
> +        'QEMU_IO' environment variables
> +        """
> +        if seed is not None:
> +            self.seed = seed
> +        else:
> +            self.seed = str(random.randint(0, sys.maxint))
> +        random.seed(self.seed)
> +
> +        self.init_path = os.getcwd()
> +        self.work_dir = work_dir
> +        self.current_dir = os.path.join(work_dir, 'test-' + test_id)
> +        self.qemu_img = \
> +                        os.environ.get('QEMU_IMG', 'qemu-img')\
> +                                  .strip().split(' ')
> +        self.qemu_io = \
> +                       os.environ.get('QEMU_IO', 'qemu-io').strip().split(' ')
> +        self.commands = [['qemu-img', 'check', '-f', 'qcow2', '$test_img'],
> +                         ['qemu-img', 'info', '-f', 'qcow2', '$test_img'],
> +                         ['qemu-io', '$test_img', '-c', 'read $off $len'],
> +                         ['qemu-io', '$test_img', '-c', 'write $off $len'],
> +                         ['qemu-io', '$test_img', '-c',
> +                          'aio_read $off $len'],
> +                         ['qemu-io', '$test_img', '-c',
> +                          'aio_write $off $len'],
> +                         ['qemu-io', '$test_img', '-c', 'flush'],
> +                         ['qemu-io', '$test_img', '-c',
> +                          'discard $off $len'],
> +                         ['qemu-io', '$test_img', '-c',
> +                          'truncate $off']]
> +        for fmt in ['raw', 'vmdk', 'vdi', 'cow', 'qcow2', 'file',
> +                    'qed', 'vpc']:
> +            self.commands.append(
> +                         ['qemu-img', 'convert', '-f', 'qcow2', '-O', fmt,
> +                          '$test_img', 'converted_image.' + fmt])
> +
> +        try:
> +            os.makedirs(self.current_dir)
> +        except OSError, e:
> +            print >>sys.stderr, \
> +                "Error: The working directory '%s' cannot be used. Reason: %s"\
> +                % (self.work_dir, e[1])
> +            raise TestException
> +        self.log = open(os.path.join(self.current_dir, "test.log"), "w")
> +        self.parent_log = open(run_log, "a")
> +        self.failed = False
> +        self.cleanup = cleanup
> +        self.log_all = log_all
> +
> +    def _test_app(self, q_args):
> +        """ Start application under test with specified arguments and return
> +        an exit code or a kill signal depending on the result of execution.
> +        """
> +        devnull = open('/dev/null', 'r+')
> +        return subprocess.call(q_args, stdin=devnull, stdout=self.log,
> +                               stderr=self.log)
> +
> +    def _create_backing_file(self):
> +        """Create a backing file in the current directory and return
> +        its name and file format
> +
> +        Format of a backing file is randomly chosen from all formats supported
> +        by 'qemu-img create'
> +        """
> +        # All formats qemu-img can create images of.
> +        backing_file_fmt = random.choice(['raw', 'vmdk', 'vdi', 'cow', 'qcow2',
> +                                          'file', 'qed', 'vpc'])
> +        backing_file_name = 'backing_img.' + backing_file_fmt
> +        # Size of the backing file varies from 1 to 10 MB
> +        backing_file_size = random.randint(1, 10)*(1 << 20)

Missing whitespaces around '*'.

> +        cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
> +                               backing_file_name, str(backing_file_size)]
> +        devnull = open('/dev/null', 'r+')
> +        retcode = subprocess.call(cmd, stdin=devnull, stdout=self.log,
> +                                  stderr=self.log)
> +        if retcode == 0:

We can assert retcode == 0.

> +            return [backing_file_name, backing_file_fmt]
> +        else:
> +            return [None, None]
> +
> +    def execute(self, input_commands=None, fuzz_config=None):
> +        """ Execute a test.
> +
> +        The method creates backing and test images, runs test app and analyzes
> +        its exit status. If the application was killed by a signal, the test
> +        is marked as failed.
> +        """
> +        if input_commands is None:
> +            commands = self.commands
> +        else:
> +            commands = input_commands
> +        os.chdir(self.current_dir)
> +        backing_file_name, backing_file_fmt = self._create_backing_file()
> +        img_size = image_generator.create_image('test_image',
> +                                                backing_file_name,
> +                                                backing_file_fmt,
> +                                                fuzz_config)
> +        for item in commands:
> +            start = random.randint(0, img_size)
> +            end = random.randint(start, img_size)
> +            current_cmd = list(self.__dict__[item[0].replace('-', '_')])
> +            # Replace all placeholders with their real values
> +            for v in item[1:]:
> +                c = v.replace('$test_img', 'test_image').\
> +                    replace('$off', str(start)).\
> +                    replace('$len', str(end - start))
> +                current_cmd.append(c)
> +            # Log string with the test header
> +            test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
> +                           "Backing file: %s\n" \
> +                           % (self.seed, " ".join(current_cmd),
> +                              self.current_dir, backing_file_name)
> +            try:
> +                retcode = self._test_app(current_cmd)
> +            except OSError, e:
> +                multilog(test_summary + "Error: Start of '%s' failed. " \
> +                         "Reason: %s\n\n" % (os.path.basename(
> +                             current_cmd[0]), e[1]),
> +                         sys.stderr, self.log, self.parent_log)
> +                raise TestException
> +
> +            if retcode < 0:
> +                multilog(test_summary + "FAIL: Test terminated by signal " +
> +                         "%s\n\n" % str_signal(-retcode), sys.stderr, self.log,
> +                         self.parent_log)
> +                self.failed = True
> +            else:
> +                if self.log_all:
> +                    multilog(test_summary + "PASS: Application exited with" + \
> +                             " the code '%d'\n\n" % retcode, sys.stdout,
> +                             self.log, self.parent_log)
> +
> +    def finish(self):
> +        """ Restore environment after a test execution. Remove folders of
> +        passed tests
> +        """
> +        self.log.close()
> +        self.parent_log.close()
> +        os.chdir(self.init_path)
> +        if self.cleanup and not self.failed:
> +            rmtree(self.current_dir)
> +
> +if __name__ == '__main__':
> +
> +    def usage():
> +        print """
> +        Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
> +
> +        Set up test environment in TEST_DIR and run a test in it. A module for
> +        test image generation should be specified via IMG_GENERATOR.
> +        Example:
> +        runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test ../qcow2
> +
> +        Optional arguments:
> +          -h, --help                    display this help and exit
> +          -c, --command=JSON            run tests for all commands specified in
> +                                        the JSON array
> +          -s, --seed=STRING             seed for a test image generation,
> +                                        by default will be generated randomly
> +          --config=JSON                 take fuzzer configuration from the JSON
> +                                        array
> +          -k, --keep_passed             don't remove folders of passed tests
> +          -v, --verbose                 log information about passed tests
> +
> +        JSON:
> +
> +        '--command' accepts a JSON array of commands. Each command presents
> +        an application under test with all its paramaters as a list of strings,
> +        e.g.
> +          ["qemu-io", "$test_img", "-c", "write $off $len"]
> +
> +        Supported application aliases: 'qemu-img' and 'qemu-io'.
> +        Supported argument aliases: $test_img for the fuzzed image, $off
> +        for an offset, $len for length.
> +
> +        Values for $off and $len will be generated based on the virtual disk
> +        size of the fuzzed image
> +        Paths to 'qemu-img' and 'qemu-io' are retrevied from 'QEMU_IMG' and
> +        'QEMU_IO' environment variables
> +
> +        '--config' accepts a JSON array of fields to be fuzzed, e.g.
> +          '[["header"], ["header", "version"]]'
> +        Each of the list elements can consist of a complex image element only
> +        as ["header"] or ["feature_name_table"] or an exact field as
> +        ["header", "version"]. In the first case random portion of the element
> +        fields will be fuzzed, in the second one the specified field will be
> +        fuzzed always.
> +
> +        If '--config' argument is specified, fields not listed in
> +        the configuration array will not be fuzzed.
> +        """
> +
> +    def run_test(test_id, seed, work_dir, run_log, cleanup, log_all,
> +                 command, fuzz_config):
> +        """Setup environment for one test and execute this test"""
> +        try:
> +            test = TestEnv(test_id, seed, work_dir, run_log, cleanup,
> +                           log_all)
> +        except TestException:
> +            sys.exit(1)
> +
> +        # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
> +        # block
> +        try:
> +            try:
> +                test.execute(command, fuzz_config)
> +            # Silent exit on user break
> +            except TestException:
> +                sys.exit(1)
> +        finally:
> +            test.finish()
> +
> +    try:
> +        opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
> +                                       ['command=', 'help', 'seed=', 'config=',
> +                                        'keep_passed', 'verbose'])
> +    except getopt.error, e:
> +        print "Error: %s\n\nTry 'runner.py --help' for more information" % e
> +        sys.exit(1)
> +
> +    command = None
> +    cleanup = True
> +    log_all = False
> +    seed = None
> +    config = None
> +    for opt, arg in opts:
> +        if opt in ('-h', '--help'):
> +            usage()
> +            sys.exit()
> +        elif opt in ('-c', '--command'):
> +            try:
> +                command = json.loads(arg)
> +            except (TypeError, ValueError, NameError), e:
> +                print "Error: JSON array of test commands cannot be loaded"\
> +                    "\nReason: %s" % e
> +                sys.exit(1)
> +        elif opt in ('-k', '--keep_passed'):
> +            cleanup = False
> +        elif opt in ('-v', '--verbose'):
> +            log_all = True
> +        elif opt in ('-s', '--seed'):
> +            seed = arg
> +        elif opt == '--config':
> +            try:
> +                config = json.loads(arg)
> +            except (TypeError, ValueError, NameError), e:
> +                print "Error: JSON array with the fuzzer configuration " \
> +                    "cannot be loaded\nReason: %s" % e
> +                sys.exit(1)
> +
> +    if not len(args) == 2:
> +        print "Expected two parameters\nTry 'runner.py --help' " \
> +            "for more information"
> +        sys.exit(1)
> +
> +    work_dir = os.path.realpath(args[0])
> +    # run_log is created in 'main', because multiple tests are expected to
> +    # log in it
> +    run_log = os.path.join(work_dir, 'run.log')
> +
> +    # Add the path to the image generator module to sys.path
> +    sys.path.append(os.path.dirname(os.path.realpath(args[1])))
> +    # Remove a script extension from image generator module if any
> +    generator_name = os.path.splitext(os.path.basename(args[1]))[0]
> +    try:
> +        image_generator = __import__(generator_name)
> +    except ImportError, e:
> +        print "Error: The image generator '%s' cannot be imported.\n" \
> +            "Reason: %s" % (generator_name, e)
> +        sys.exit(1)
> +    # If a seed is specified, only one test will be executed.
> +    # Otherwise runner will terminate after a keyboard interruption
> +    for test_id in count(1):
> +        try:
> +            run_test(str(test_id), seed, work_dir, run_log, cleanup,
> +                     log_all, command, config)
> +        except (KeyboardInterrupt, SystemExit):
> +            sys.exit(1)
> +
> +        if seed is not None:
> +            break
> -- 
> 1.9.3
> 

Re: [Qemu-devel] [PATCH V4 2/5] runner: Tool for fuzz tests execution

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]

On Mon, Jul 21, 2014 at 02:18:09PM +0400, Maria Kustova wrote:
> +    def execute(self, input_commands=None, fuzz_config=None):
> +        """ Execute a test.
> +
> +        The method creates backing and test images, runs test app and analyzes
> +        its exit status. If the application was killed by a signal, the test
> +        is marked as failed.
> +        """
> +        if input_commands is None:
> +            commands = self.commands
> +        else:
> +            commands = input_commands
> +        os.chdir(self.current_dir)
> +        backing_file_name, backing_file_fmt = self._create_backing_file()
> +        img_size = image_generator.create_image('test_image',
> +                                                backing_file_name,
> +                                                backing_file_fmt,
> +                                                fuzz_config)
> +        for item in commands:
> +            start = random.randint(0, img_size)
> +            end = random.randint(start, img_size)
> +            current_cmd = list(self.__dict__[item[0].replace('-', '_')])

This special case for self.qemu_img and self.qemu_io is not obvious to
me when reading the code.  It would be clearer to use the same
$qemu_img/$qemu_io substitution mechanism as for $test_img/$off/$len
below.

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V4 3/5] fuzz: Fuzzing functions for qcow2 images

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

On Mon, Jul 21, 2014 at 02:18:10PM +0400, Maria Kustova wrote:
> +def random_from_intervals(intervals):
> +    """Select a random integer number from the list of specified intervals
> +
> +    Each interval is a tuple of lower and upper limits of the interval. The
> +    limits are included. Intervals in a list should not overlap.
> +    """
> +    total = reduce(lambda x, y: x + y[1] - y[0] + 1, intervals, 0)
> +    r = random.randint(0, total-1) + intervals[0][0]
> +    temp = zip(intervals, intervals[1:])
> +    for x in temp:
> +        r = r + (r > x[0][1])*(x[1][0] - x[0][1] - 1)

Please use space around arithmetic operators:
r = r + (r > x[0][1]) * (x[1][0] - x[0][1] - 1)

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V4 2/5] runner: Tool for fuzz tests execution

[M.Kustova]

On Fri, Aug 1, 2014 at 9:46 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, Jul 21, 2014 at 02:18:09PM +0400, Maria Kustova wrote:
>> +    def execute(self, input_commands=None, fuzz_config=None):
>> +        """ Execute a test.
>> +
>> +        The method creates backing and test images, runs test app and analyzes
>> +        its exit status. If the application was killed by a signal, the test
>> +        is marked as failed.
>> +        """
>> +        if input_commands is None:
>> +            commands = self.commands
>> +        else:
>> +            commands = input_commands
>> +        os.chdir(self.current_dir)
>> +        backing_file_name, backing_file_fmt = self._create_backing_file()
>> +        img_size = image_generator.create_image('test_image',
>> +                                                backing_file_name,
>> +                                                backing_file_fmt,
>> +                                                fuzz_config)
>> +        for item in commands:
>> +            start = random.randint(0, img_size)
>> +            end = random.randint(start, img_size)
>> +            current_cmd = list(self.__dict__[item[0].replace('-', '_')])
>
> This special case for self.qemu_img and self.qemu_io is not obvious to
> me when reading the code.  It would be clearer to use the same
> $qemu_img/$qemu_io substitution mechanism as for $test_img/$off/$len
> below.
Separate processing of test programs is due to the impossibility to
laconically replace a string with an unpacked list of strings.
But I agree that using of __dict__ is not safe and have to be redesigned.

BR, M.