[Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Michael Roth]

These patches apply to master (2-04-2011), and can also be obtained from:
git://repo.or.cz/qemu/mdroth.git qtest_v1

OVERVIEW:

QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.

The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:

init():
  Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers.

run():
  Test logic that interacts with the now-created machine.

cleanup():
  Currently unused, but potentially allows for chaining multiple tests together. Currently we run one module, then exit.

As mentioned these are very early starting points. We're mostly looking for input from the community on the basic approach and overall requirements for an acceptable framework. A basic RTC test module is provided as an example.

BUILD/EXAMPLE USAGE:

 $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
 $ make
 $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
 Available test modules:
 rtc
 $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
 ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
 ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200

GENERAL PLAN:

 - Provide libraries for common operations like PCI device enumeration, APIC configuration, default-configured machine setup, interrupt handling, etc.
 - Develop tests as machine/target specific, potentially make some tests re-usable as interfaces are better defined
 - Do port i/o via cpu_in/cpu_out commands
 - Do guest memory access via a CPUPhysMemoryClient interface
 - Allow interrupts to be sent by writing to an FD, detection in test modules via select()/read()

TODO:

 - A means to propagate test returns values to main i/o thread
 - Better defined test harness for individual test cases and/or modules, likely via GLib
 - Support for multiple test threads in a single test module for scalability testing
 - Modify vl.c hooks so tests can configure their own timers/clocksources
 - More test modules, improve current rtc module
 - Further implementing/fleshing out of the overall plan

Comments/feedback are welcome!

 Makefile.objs     |    4 +-
 Makefile.target   |    1 +
 configure         |   20 ++++++
 module.h          |    5 +-
 qemu-options.hx   |    9 +++
 qtest/qtest-rtc.c |  170 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 qtest/qtest.c     |   65 ++++++++++++++++++++
 qtest/qtest.h     |   37 ++++++++++++
 roms/seabios      |    2 +-
 vl.c              |   35 +++++++++++
 10 files changed, 345 insertions(+), 3 deletions(-)

[Qemu-devel] [RFC][PATCH v1 1/4] qtest: add qtest to supported qemu modules

[Michael Roth]

Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 module.h |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/module.h b/module.h
index 9263f1c..d2bf34d 100644
--- a/module.h
+++ b/module.h
@@ -24,12 +24,15 @@ typedef enum {
     MODULE_INIT_BLOCK,
     MODULE_INIT_DEVICE,
     MODULE_INIT_MACHINE,
-    MODULE_INIT_MAX
+    MODULE_INIT_QTEST_REGISTRY,
+    MODULE_INIT_MAX,
 } module_init_type;
 
 #define block_init(function) module_init(function, MODULE_INIT_BLOCK)
 #define device_init(function) module_init(function, MODULE_INIT_DEVICE)
 #define machine_init(function) module_init(function, MODULE_INIT_MACHINE)
+#define qtest_init_registry(function) \
+    module_init(function, MODULE_INIT_QTEST_REGISTRY)
 
 void register_module_init(void (*fn)(void), module_init_type type);
 
-- 
1.7.0.4

[Qemu-devel] [RFC][PATCH v1 2/4] qtest: basic test infrastructure and vl.c hooks

[Michael Roth]

Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 qtest/qtest.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qtest/qtest.h |   37 ++++++++++++++++++++++++++++++++
 vl.c          |   24 +++++++++++++++++++++
 3 files changed, 126 insertions(+), 0 deletions(-)
 create mode 100644 qtest/qtest.c
 create mode 100644 qtest/qtest.h

diff --git a/qtest/qtest.c b/qtest/qtest.c
new file mode 100644
index 0000000..fa30bdc
--- /dev/null
+++ b/qtest/qtest.c
@@ -0,0 +1,65 @@
+#include "qtest.h"
+#include "qemu-thread.h"
+
+static QTAILQ_HEAD(, QTestModule) qtest_module_list =
+QTAILQ_HEAD_INITIALIZER(qtest_module_list);
+
+static const QTestModule *qtest_module_by_name(const char *test_name)
+{
+    const QTestModule *module;
+    QTAILQ_FOREACH(module, &qtest_module_list, next) {
+        if (strcmp(module->name, test_name) == 0) {
+            return module;
+        }
+    }
+
+    return NULL;
+}
+
+void qtest_list_modules(void)
+{
+    const QTestModule *module;
+    printf("Available test modules:\n");
+    QTAILQ_FOREACH(module, &qtest_module_list, next) {
+        printf("%s\n", module->name);
+    }
+}
+
+void qtest_register(QTestModule *module)
+{
+    QTAILQ_INSERT_TAIL(&qtest_module_list, module, next);
+}
+
+void qtest_init(const char *test_name)
+{
+    const QTestModule *module = qtest_module_by_name(test_name);
+
+    if (!module) {
+        fprintf(stderr, "unknown qtest module: %s\n", test_name);
+        exit(1);
+    }
+    module->init();
+}
+
+void qtest_cleanup(const char *test_name)
+{
+    const QTestModule *module = qtest_module_by_name(test_name);
+
+    if (!module) {
+        fprintf(stderr, "unknown qtest module: %s\n", test_name);
+        exit(1);
+    }
+    module->cleanup();
+}
+
+void qtest_start(const char *test_name, void *thread_args)
+{
+    const QTestModule *module = qtest_module_by_name(test_name);
+    QemuThread thread;
+
+    if (!module) {
+        fprintf(stderr, "unknown qtest module: %s\n", test_name);
+        exit(1);
+    }
+    qemu_thread_create(&thread, module->start, thread_args);
+}
diff --git a/qtest/qtest.h b/qtest/qtest.h
new file mode 100644
index 0000000..7e3daed
--- /dev/null
+++ b/qtest/qtest.h
@@ -0,0 +1,37 @@
+/*
+ * qtest - general interface definitions
+ *
+ * Copyright IBM Corp. 2011
+ *
+ * Authors:
+ *  Michael Roth      <mdroth@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QTEST_H
+#define QTEST_H
+
+#include "qemu-common.h"
+#include "qemu-queue.h"
+
+typedef void *(QTestStartFunc)(void *thread_args);
+typedef void (QTestCleanupFunc)(void);
+typedef void (QTestInitFunc)(void);
+
+typedef struct QTestModule {
+    const char *name;
+    QTestInitFunc *init;
+    QTestCleanupFunc *cleanup;
+    QTestStartFunc *start;
+    QTAILQ_ENTRY(QTestModule) next;
+} QTestModule;
+
+void qtest_register(QTestModule *module);
+void qtest_init(const char *test_name);
+void qtest_cleanup(const char *test_name);
+void qtest_start(const char *test_name, void *thread_args);
+void qtest_list_modules(void);
+
+#endif
diff --git a/vl.c b/vl.c
index 655617f..0b5b613 100644
--- a/vl.c
+++ b/vl.c
@@ -164,6 +164,9 @@ int main(int argc, char **argv)
 #include "arch_init.h"
 
 #include "ui/qemu-spice.h"
+#ifdef CONFIG_QTEST
+#include "qtest/qtest.h"
+#endif
 
 //#define DEBUG_NET
 //#define DEBUG_SLIRP
@@ -1930,6 +1933,9 @@ int main(int argc, char **argv, char **envp)
     int show_vnc_port = 0;
     int defconfig = 1;
 
+#ifdef CONFIG_QTEST
+    const char *qtest_module = NULL;
+#endif
 #ifdef CONFIG_SIMPLE_TRACE
     const char *trace_file = NULL;
 #endif
@@ -1943,6 +1949,9 @@ int main(int argc, char **argv, char **envp)
     QLIST_INIT (&vm_change_state_head);
     os_setup_early_signal_handling();
 
+#ifdef CONFIG_QTEST
+    module_call_init(MODULE_INIT_QTEST_REGISTRY);
+#endif
     module_call_init(MODULE_INIT_MACHINE);
     machine = find_default_machine();
     cpu_model = NULL;
@@ -2859,7 +2868,11 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+#ifdef CONFIG_QTEST
+    if (!qtest_module && kvm_allowed) {
+#else
     if (kvm_allowed) {
+#endif
         int ret = kvm_init();
         if (ret < 0) {
             if (!kvm_available()) {
@@ -2986,6 +2999,14 @@ int main(int argc, char **argv, char **envp)
         exit(1);
 
     module_call_init(MODULE_INIT_DEVICE);
+#ifdef CONFIG_QTEST
+    if (qtest_module) {
+        module_call_init(MODULE_INIT_QTEST_REGISTRY);
+        qtest_init(qtest_module);
+        qtest_start(qtest_module, NULL);
+        goto out_machine_creation;
+    }
+#endif
 
     if (qemu_opts_foreach(qemu_find_opts("device"), device_help_func, NULL, 0) != 0)
         exit(0);
@@ -3101,6 +3122,9 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+#ifdef CONFIG_QTEST
+out_machine_creation:
+#endif
     qdev_machine_creation_done();
 
     if (rom_load_all() != 0) {
-- 
1.7.0.4

[Qemu-devel] [RFC][PATCH v1 3/4] qtest: add basic rtc test module

[Michael Roth]

This test module initializes a basic system with an ISA bus, a PIC
(currently not used in the test), and an RTC. Currently only drift
testing is performed at various frequencies. This is currently more
of a proof of concept than a rigorous test, and will be improved
over time.

Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 qtest/qtest-rtc.c |  170 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 170 insertions(+), 0 deletions(-)
 create mode 100644 qtest/qtest-rtc.c

diff --git a/qtest/qtest-rtc.c b/qtest/qtest-rtc.c
new file mode 100644
index 0000000..a494fd5
--- /dev/null
+++ b/qtest/qtest-rtc.c
@@ -0,0 +1,170 @@
+#include <stdio.h>
+#include "qtest/qtest.h"
+#include "sysemu.h"
+#include "hw/pc.h"
+#include "hw/isa.h"
+#include "hw/irq.h"
+#include "hw/mc146818rtc.h"
+
+#define RAM_SIZE_B 128 * 1024 * 1024
+#define DEBUG_QT
+#ifdef DEBUG_QT
+#define TRACE(msg, ...) do { \
+    fprintf(stderr, "%s:%s():L%d: " msg "\n", \
+        __FILE__, __FUNCTION__, __LINE__, ## __VA_ARGS__); \
+} while(0)
+#else
+#define TRACE(msg, ...) \
+    do { } while (0)
+#endif
+
+static enum {
+    HANDLER_MODE_NORMAL = 0,
+    HANDLER_MODE_DRIFT_TEST_INIT,
+    HANDLER_MODE_DRIFT_TEST_STARTED,
+    HANDLER_MODE_ONESHOT_TEST_INIT,
+    HANDLER_MODE_ONESHOT_TEST_STARTED,
+} handler_mode = HANDLER_MODE_NORMAL;
+
+static uint32_t irq_count, irq_count_max;
+static struct timeval ts1, ts2;
+
+static void cmos_write(int addr, unsigned int val)
+{
+    cpu_outb(0x70, addr - 0x70);
+    cpu_outb(0x71, val);
+}
+
+static unsigned int cmos_read(int addr)
+{
+    cpu_outb(0x70, addr - 0x70);
+    return cpu_inb(0x71);
+}
+
+static void rtc_irq_handler(void *opaque, int n, int level)
+{
+    int rtc_reg_c = 0;
+    if (level) {
+        switch (handler_mode) {
+        case HANDLER_MODE_DRIFT_TEST_INIT:
+            irq_count = 0;
+            gettimeofday(&ts1, NULL);
+            handler_mode = HANDLER_MODE_DRIFT_TEST_STARTED;
+            break;
+        case HANDLER_MODE_DRIFT_TEST_STARTED:
+            if (++irq_count == irq_count_max) { 
+                gettimeofday(&ts2, NULL);
+                handler_mode = HANDLER_MODE_NORMAL;
+            }
+            break;
+        default:
+            break;
+        }
+        /* reset irq line */
+        rtc_reg_c = cmos_read(0x7C);
+    }
+    //TRACE("irq: %d, level: %d, 0x7c: %d", n, level, rtc_reg_c);
+}
+
+#define DRIFT_RATIO_MAX .05
+
+static void test_drift(void)
+{
+    uint32_t hz, start, stop, duration_ms, exp_duration_ms;
+    float drift_ratio;
+
+    handler_mode = HANDLER_MODE_NORMAL;
+    /* enable timer interrupts */
+    cmos_write(0x7B, cmos_read(0x7B) | 0x40);
+
+    /* set frequency to 2hz (32*1024khz >> (div - 1)), div is bottom 4 bits */
+    cmos_write(0x7A, (cmos_read(0x7A) & 0xF0) | 0x0F);
+    hz = 2;
+    irq_count_max = hz*5;
+    handler_mode = HANDLER_MODE_DRIFT_TEST_INIT;
+    while (handler_mode != HANDLER_MODE_NORMAL) {
+        sleep(1);
+    }
+    start = ts1.tv_sec * 1000 * 1000 + ts1.tv_usec;
+    stop = ts2.tv_sec * 1000 * 1000 + ts2.tv_usec;
+    duration_ms = (stop - start) / 1000;
+    exp_duration_ms = irq_count_max / 2 * 1000;
+    drift_ratio = fabs(1.0 - (float)duration_ms/exp_duration_ms);
+    TRACE("hz: %u, duration_ms: %u, exp_duration: %u, drift ratio: %f",
+           hz, duration_ms, exp_duration_ms, drift_ratio);
+    assert(drift_ratio <= .05);
+
+    /* set frequency to 1024hz */
+    cmos_write(0x7A, (cmos_read(0x7A) & 0xF0) | 0x06);
+    hz = 1024;
+    irq_count_max = hz * 5;
+    handler_mode = HANDLER_MODE_DRIFT_TEST_INIT;
+    while (handler_mode != HANDLER_MODE_NORMAL) {
+        sleep(1);
+    }
+    start = ts1.tv_sec * 1000 * 1000 + ts1.tv_usec;
+    stop = ts2.tv_sec * 1000 * 1000 + ts2.tv_usec;
+    duration_ms = (stop - start) / 1000;
+    exp_duration_ms = irq_count_max / hz * 1000;
+    drift_ratio = fabs(1.0 - (float)duration_ms/exp_duration_ms);
+    TRACE("hz: %u, duration_ms: %u, exp_duration: %u, drift ratio: %f",
+           hz, duration_ms, exp_duration_ms, drift_ratio);
+    assert(drift_ratio <= .05);
+}
+
+static void qtest_rtc_init(void)
+{
+    ram_addr_t below_4g_mem_size, above_4g_mem_size;
+    IsaIrqState *isa_irq_state;
+    ISADevice *rtc_state;
+    qemu_irq *rtc_irq, *cpu_irq, *isa_irq, *i8259;
+
+    /* allocate ram */
+    pc_memory_init(RAM_SIZE_B, NULL, NULL, NULL,
+                   &below_4g_mem_size, &above_4g_mem_size);
+    /*
+    pc_cmos_init(below_4g_mem_size, above_4g_mem_size, NULL,
+                 NULL, NULL, NULL, NULL);
+                 */
+
+    /* set up the isa bus. we'll use an intercept handler for the rtc
+     * interrupts, but set the PIC up in case we want to use it later
+     */
+    cpu_irq = pc_allocate_cpu_irq();
+    i8259 = i8259_init(cpu_irq[0]);
+    isa_irq_state = qemu_mallocz(sizeof(*isa_irq_state));
+    isa_irq_state->i8259 = i8259;
+    isa_irq = qemu_allocate_irqs(isa_irq_handler, isa_irq_state, 24);
+    isa_bus_new(NULL);
+    isa_bus_irqs(isa_irq);
+
+    /* set up rtc */
+    rtc_irq = qemu_allocate_irqs(rtc_irq_handler, NULL, 1);
+    rtc_state = rtc_init(2000, rtc_irq[0]);
+}
+
+static void qtest_rtc_cleanup(void)
+{
+}
+
+static void *qtest_rtc_start(void *thread_args)
+{
+    test_drift();
+    //g_assert(false);
+    qemu_system_shutdown_request();
+    return NULL;
+}
+
+static QTestModule qtest_module = {
+    .name = "rtc",
+    .init = qtest_rtc_init,
+    .cleanup = qtest_rtc_cleanup,
+    .start = qtest_rtc_start,
+};
+
+static void qtest_module_register(void)
+{
+    qtest_register(&qtest_module);
+}
+
+qtest_init_registry(qtest_module_register)
-- 
1.7.0.4

[Qemu-devel] [RFC][PATCH v1 4/4] qtest: build qtest and add -test <testname> cmdline opt

[Michael Roth]

Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 Makefile.objs   |    4 +++-
 Makefile.target |    1 +
 configure       |   20 ++++++++++++++++++++
 qemu-options.hx |    9 +++++++++
 vl.c            |   11 +++++++++++
 5 files changed, 44 insertions(+), 1 deletions(-)

diff --git a/Makefile.objs b/Makefile.objs
index f1c7bfe..ec0cd91 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -139,6 +139,9 @@ ui-obj-y += vnc-jobs-sync.o
 endif
 common-obj-y += $(addprefix ui/, $(ui-obj-y))
 
+qtest-obj-y += qtest.o
+common-obj-$(CONFIG_QTEST) += $(addprefix qtest/, $(qtest-obj-y))
+
 common-obj-y += iov.o acl.o
 common-obj-$(CONFIG_THREAD) += qemu-thread.o
 common-obj-$(CONFIG_IOTHREAD) += compatfd.o
@@ -315,4 +318,3 @@ endif
 vl.o: QEMU_CFLAGS+=$(GPROF_CFLAGS)
 
 vl.o: QEMU_CFLAGS+=$(SDL_CFLAGS)
-
diff --git a/Makefile.target b/Makefile.target
index b0ba95f..5c775bc 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -76,6 +76,7 @@ libobj-$(CONFIG_NOSOFTFLOAT) += fpu/softfloat-native.o
 libobj-y += op_helper.o helper.o
 ifeq ($(TARGET_BASE_ARCH), i386)
 libobj-y += cpuid.o
+libobj-$(CONFIG_QTEST) += ../qtest/qtest_rtc.o
 endif
 libobj-$(CONFIG_NEED_MMU) += mmu.o
 libobj-$(TARGET_ARM) += neon_helper.o iwmmxt_helper.o
diff --git a/configure b/configure
index 598e8e1..355928e 100755
--- a/configure
+++ b/configure
@@ -174,6 +174,7 @@ trace_backend="nop"
 trace_file="trace"
 spice=""
 rbd=""
+qtest=""
 
 # parse CC options first
 for opt do
@@ -719,6 +720,10 @@ for opt do
   ;;
   --enable-rbd) rbd="yes"
   ;;
+  --disable-qtest) qtest="no"
+  ;;
+  --enable-qtest) qtest="yes"
+  ;;
   *) echo "ERROR: unknown option $opt"; show_help="yes"
   ;;
   esac
@@ -914,6 +919,9 @@ echo "                           Default:trace-<pid>"
 echo "  --disable-spice          disable spice"
 echo "  --enable-spice           enable spice"
 echo "  --enable-rbd             enable building the rados block device (rbd)"
+echo "  --disable-qtest          disable qtest unit testing support"
+echo "  --enable-qtest           enable qtest unit testing support"
+echo "                           (requires --enable-io-thread)"
 echo ""
 echo "NOTE: The object files are built at the place where configure is launched"
 exit 1
@@ -1656,6 +1664,8 @@ EOF
 fi
 
 ##########################################
+
+##########################################
 # kvm probe
 if test "$kvm" != "no" ; then
     cat > $TMPC <<EOF
@@ -2472,6 +2482,7 @@ echo "Trace output file $trace_file-<pid>"
 echo "spice support     $spice"
 echo "rbd support       $rbd"
 echo "xfsctl support    $xfs"
+echo "QTest support     $qtest"
 
 if test $sdl_too_old = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -2784,6 +2795,15 @@ if test "$trace_backend" = "dtrace" -a "$trace_backend_stap" = "yes" ; then
 fi
 echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
 
+if test "$qtest" = "yes"; then
+    if test "$io_thread" = "yes"; then
+        echo "CONFIG_QTEST=y" >> $config_host_mak
+    else
+        echo "ERROR: --enable-io-thread is currently required for qtest"
+        exit 1
+    fi
+fi
+
 echo "TOOLS=$tools" >> $config_host_mak
 echo "ROMS=$roms" >> $config_host_mak
 echo "MAKE=$make" >> $config_host_mak
diff --git a/qemu-options.hx b/qemu-options.hx
index 945edf3..6f7e00b 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -2339,6 +2339,15 @@ STEXI
 Specify a trace file to log output traces to.
 ETEXI
 #endif
+#ifdef CONFIG_QTEST
+DEF("test", HAS_ARG, QEMU_OPTION_test,
+    "-test <testname>\n", QEMU_ARCH_ALL)
+STEXI
+@item -test @var{testname}
+@findex -test
+Execute test case specified by @var{testname}. Use "?" to list available tests.
+ETEXI
+#endif
 
 HXCOMM This is the last statement. Insert new options before this line!
 STEXI
diff --git a/vl.c b/vl.c
index 0b5b613..4daf948 100644
--- a/vl.c
+++ b/vl.c
@@ -2722,6 +2722,17 @@ int main(int argc, char **argv, char **envp)
                 }
                 break;
 #endif
+#ifdef CONFIG_QTEST
+            case QEMU_OPTION_test:
+                {
+                    if (*optarg == '?') {
+                        qtest_list_modules();
+                        exit(0);
+                    }
+                    qtest_module = optarg;
+                    break;
+                }
+#endif
             case QEMU_OPTION_readconfig:
                 {
                     int ret = qemu_read_config_file(optarg);
-- 
1.7.0.4

Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Blue Swirl]

On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth <mdroth@linux.vnet.ibm.com> wrote:
> These patches apply to master (2-04-2011), and can also be obtained from:
> git://repo.or.cz/qemu/mdroth.git qtest_v1
>
> OVERVIEW:
>
> QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
>
> The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
>
> init():
>  Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers.
>
> run():
>  Test logic that interacts with the now-created machine.
>
> cleanup():
>  Currently unused, but potentially allows for chaining multiple tests together. Currently we run one module, then exit.
>
> As mentioned these are very early starting points. We're mostly looking for input from the community on the basic approach and overall requirements for an acceptable framework. A basic RTC test module is provided as an example.
>
> BUILD/EXAMPLE USAGE:
>
>  $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
>  $ make
>  $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
>  Available test modules:
>  rtc
>  $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
>  ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
>  ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
>
> GENERAL PLAN:
>
>  - Provide libraries for common operations like PCI device enumeration, APIC configuration, default-configured machine setup, interrupt handling, etc.
>  - Develop tests as machine/target specific, potentially make some tests re-usable as interfaces are better defined
>  - Do port i/o via cpu_in/cpu_out commands
>  - Do guest memory access via a CPUPhysMemoryClient interface
>  - Allow interrupts to be sent by writing to an FD, detection in test modules via select()/read()
>
> TODO:
>
>  - A means to propagate test returns values to main i/o thread
>  - Better defined test harness for individual test cases and/or modules, likely via GLib
>  - Support for multiple test threads in a single test module for scalability testing
>  - Modify vl.c hooks so tests can configure their own timers/clocksources
>  - More test modules, improve current rtc module
>  - Further implementing/fleshing out of the overall plan
>
> Comments/feedback are welcome!

Would it be possible to couple this with the tracing or Kemari somehow
so that you could capture, say, block device traces and feed them to
test setup?

Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Michael Roth]

On 02/09/2011 01:42 PM, Blue Swirl wrote:
> On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth<mdroth@linux.vnet.ibm.com>  wrote:
>> These patches apply to master (2-04-2011), and can also be obtained from:
>> git://repo.or.cz/qemu/mdroth.git qtest_v1
>>
>> OVERVIEW:
>>
>> QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
>>
>> The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
>>
>> init():
>>   Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers.
>>
>> run():
>>   Test logic that interacts with the now-created machine.
>>
>> cleanup():
>>   Currently unused, but potentially allows for chaining multiple tests together. Currently we run one module, then exit.
>>
>> As mentioned these are very early starting points. We're mostly looking for input from the community on the basic approach and overall requirements for an acceptable framework. A basic RTC test module is provided as an example.
>>
>> BUILD/EXAMPLE USAGE:
>>
>>   $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
>>   $ make
>>   $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
>>   Available test modules:
>>   rtc
>>   $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
>>   ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
>>   ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
>>
>> GENERAL PLAN:
>>
>>   - Provide libraries for common operations like PCI device enumeration, APIC configuration, default-configured machine setup, interrupt handling, etc.
>>   - Develop tests as machine/target specific, potentially make some tests re-usable as interfaces are better defined
>>   - Do port i/o via cpu_in/cpu_out commands
>>   - Do guest memory access via a CPUPhysMemoryClient interface
>>   - Allow interrupts to be sent by writing to an FD, detection in test modules via select()/read()
>>
>> TODO:
>>
>>   - A means to propagate test returns values to main i/o thread
>>   - Better defined test harness for individual test cases and/or modules, likely via GLib
>>   - Support for multiple test threads in a single test module for scalability testing
>>   - Modify vl.c hooks so tests can configure their own timers/clocksources
>>   - More test modules, improve current rtc module
>>   - Further implementing/fleshing out of the overall plan
>>
>> Comments/feedback are welcome!
>
> Would it be possible to couple this with the tracing or Kemari somehow
> so that you could capture, say, block device traces and feed them to
> test setup?

I would think so...it's a pretty open ended framework, a unit test 
could, say, read in block device traces in some pre-defined format and 
then execute those against a block device. We're also planning on adding 
command-line parameters for tests, so a unit test could actually be used 
as a general testing utility. for instance:

qemu -test block-trace-virtio -test-opts 
tracefile=<file>,target_img=<img>,target_fmt=qcow2,comparison_img=<img>

Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Stefan Hajnoczi]

On Wed, Feb 9, 2011 at 8:39 PM, Michael Roth <mdroth@linux.vnet.ibm.com> wrote:
> On 02/09/2011 01:42 PM, Blue Swirl wrote:
>>
>> On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth<mdroth@linux.vnet.ibm.com>
>>  wrote:
>>>
>>> These patches apply to master (2-04-2011), and can also be obtained from:
>>> git://repo.or.cz/qemu/mdroth.git qtest_v1
>>>
>>> OVERVIEW:
>>>
>>> QEMU currently lacks a standard means to do targeted unit testing of the
>>> device model. Frameworks like kvm-autotest interact via guest OS, which
>>> provide a highly abstracted interface to the underlying machine, and are
>>> susceptable to bugs in the guest OS itself. This allows for reasonable test
>>> coverage of guest functionality as a whole, but reduces the accuracy and
>>> specificity with which we can exercise paths in the underlying devices.
>>>
>>> The following patches provide the basic beginnings of a test framework
>>> which replaces vcpu threads with test threads that interact with the
>>> underlying machine directly, allowing for directed unit/performance testing
>>> of individual devices. Test modules are built directly into the qemu binary,
>>> and each module provides the following interfaces:
>>>
>>> init():
>>>  Called in place of qemu's normal machine initialization to setup up
>>> devices explicitly. A full machine can be created here by calling into the
>>> normal init path, as well as minimal machines with a select set of
>>> buses/devices/IRQ handlers.
>>>
>>> run():
>>>  Test logic that interacts with the now-created machine.
>>>
>>> cleanup():
>>>  Currently unused, but potentially allows for chaining multiple tests
>>> together. Currently we run one module, then exit.
>>>
>>> As mentioned these are very early starting points. We're mostly looking
>>> for input from the community on the basic approach and overall requirements
>>> for an acceptable framework. A basic RTC test module is provided as an
>>> example.
>>>
>>> BUILD/EXAMPLE USAGE:
>>>
>>>  $ ./configure --target-list=x86_64-softmmu --enable-qtest
>>> --enable-io-thread
>>>  $ make
>>>  $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
>>>  Available test modules:
>>>  rtc
>>>  $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
>>>  ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999,
>>> exp_duration: 5000, drift ratio: 0.000200
>>>  ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999,
>>> exp_duration: 5000, drift ratio: 0.000200
>>>
>>> GENERAL PLAN:
>>>
>>>  - Provide libraries for common operations like PCI device enumeration,
>>> APIC configuration, default-configured machine setup, interrupt handling,
>>> etc.
>>>  - Develop tests as machine/target specific, potentially make some tests
>>> re-usable as interfaces are better defined
>>>  - Do port i/o via cpu_in/cpu_out commands
>>>  - Do guest memory access via a CPUPhysMemoryClient interface
>>>  - Allow interrupts to be sent by writing to an FD, detection in test
>>> modules via select()/read()
>>>
>>> TODO:
>>>
>>>  - A means to propagate test returns values to main i/o thread
>>>  - Better defined test harness for individual test cases and/or modules,
>>> likely via GLib
>>>  - Support for multiple test threads in a single test module for
>>> scalability testing
>>>  - Modify vl.c hooks so tests can configure their own timers/clocksources
>>>  - More test modules, improve current rtc module
>>>  - Further implementing/fleshing out of the overall plan
>>>
>>> Comments/feedback are welcome!
>>
>> Would it be possible to couple this with the tracing or Kemari somehow
>> so that you could capture, say, block device traces and feed them to
>> test setup?
>
> I would think so...it's a pretty open ended framework, a unit test could,
> say, read in block device traces in some pre-defined format and then execute
> those against a block device. We're also planning on adding command-line
> parameters for tests, so a unit test could actually be used as a general
> testing utility. for instance:

That's a good point.  Testing network, block, serial, etc device
emulation requires mock host devices (netdev, drive, chardev).

Net needs a "replay" net client.  A "dump" net client already exists
for capturing packets.

Block has no record/replay but the Linux blktrace format might be
good.  There's also Kevin's blkdebug and CQ's blksim which might be
extendable.

For chardev perhaps the existing options are already powerful enough,
otherwise something like expect would be neat.

> qemu -test block-trace-virtio -test-opts
> tracefile=<file>,target_img=<img>,target_fmt=qcow2,comparison_img=<img>

Or more like the -device, -chardev, etc syntax:

qemu -test block-trace-virtio,tracefile=<file>,target_img=<img>,target_fmt=qcow2,comparison_img=<img>

Stefan