[Qemu-devel] [PATCH 0/4] add "make check"

[Qemu-devel] [PATCH 0/4] add "make check"

[Gerd Hoffmann]

  Hi,

This patch series intends to make unit testing easier.  It adds a new
"make check" target which can be used to run all unit tests which are
currently in the tree.  It also enables the unit tests by default, so
you don't have to re-run configure with a special switch.

cheers,
  Gerd

Gerd Hoffmann (4):
  Probe for libcheck by default.
  move checks to separate variable
  add "make check" target
  add test-coroutine to checks

 Makefile  |   10 ++++++++--
 configure |    7 ++++---
 2 files changed, 12 insertions(+), 5 deletions(-)

[Qemu-devel] [PATCH 1/4] Probe for libcheck by default.

[Gerd Hoffmann]

Probe for libcheck and build checks (if found) by default.
Can be explicitly disabled using --disable-check-utests.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 configure |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/configure b/configure
index 1340c33..920de9e 100755
--- a/configure
+++ b/configure
@@ -170,7 +170,7 @@ mixemu="no"
 aix="no"
 blobs="yes"
 pkgversion=""
-check_utests="no"
+check_utests=""
 user_pie="no"
 zero_malloc=""
 trace_backend="nop"
-- 
1.7.1

[Qemu-devel] [PATCH 2/4] move checks to separate variable

[Gerd Hoffmann]

Create a new CHECKS variable.  Put the checks there instead
of adding them to the TOOLS variable.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 Makefile  |    4 ++--
 configure |    5 +++--
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 8606849..2963c8b 100644
--- a/Makefile
+++ b/Makefile
@@ -71,7 +71,7 @@ defconfig:
 
 -include config-all-devices.mak
 
-build-all: $(DOCS) $(TOOLS) recurse-all
+build-all: $(DOCS) $(TOOLS) $(CHECKS) recurse-all
 
 config-host.h: config-host.h-timestamp
 config-host.h-timestamp: config-host.mak
@@ -205,7 +205,7 @@ clean:
 # avoid old build problems by removing potentially incorrect old files
 	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
 	rm -f qemu-options.def
-	rm -f *.o *.d *.a *.lo $(TOOLS) qemu-ga TAGS cscope.* *.pod *~ */*~
+	rm -f *.o *.d *.a *.lo $(TOOLS) $(CHECKS) qemu-ga TAGS cscope.* *.pod *~ */*~
 	rm -Rf .libs
 	rm -f slirp/*.o slirp/*.d audio/*.o audio/*.d block/*.o block/*.d net/*.o net/*.d fsdev/*.o fsdev/*.d ui/*.o ui/*.d qapi/*.o qapi/*.d qga/*.o qga/*.d
 	rm -f qemu-img-cmds.h
diff --git a/configure b/configure
index 920de9e..44b3295 100755
--- a/configure
+++ b/configure
@@ -2630,8 +2630,8 @@ if test "$softmmu" = yes ; then
       tools="qemu-ga\$(EXESUF) $tools"
     fi
     if [ "$check_utests" = "yes" ]; then
-      tools="check-qint check-qstring check-qdict check-qlist $tools"
-      tools="check-qfloat check-qjson $tools"
+      checks="check-qint check-qstring check-qdict check-qlist"
+      checks="check-qfloat check-qjson $checks"
     fi
   fi
 fi
@@ -3078,6 +3078,7 @@ fi
 echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
 
 echo "TOOLS=$tools" >> $config_host_mak
+echo "CHECKS=$checks" >> $config_host_mak
 echo "ROMS=$roms" >> $config_host_mak
 echo "MAKE=$make" >> $config_host_mak
 echo "INSTALL=$install" >> $config_host_mak
-- 
1.7.1

[Qemu-devel] [PATCH 3/4] add "make check" target

[Gerd Hoffmann]

Add some Makefile glue so we have a simple "make check"
to run the unit tests.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 Makefile |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 2963c8b..80c0bc2 100644
--- a/Makefile
+++ b/Makefile
@@ -289,6 +289,12 @@ endif
 test speed: all
 	$(MAKE) -C tests $@
 
+.PHONY: check
+check: $(patsubst %,run-check-%,$(CHECKS))
+
+run-check-%: %
+	./$<
+
 .PHONY: TAGS
 TAGS:
 	find "$(SRC_PATH)" -name '*.[hc]' -print0 | xargs -0 etags
-- 
1.7.1

[Qemu-devel] [PATCH 4/4] add test-coroutine to checks

[Gerd Hoffmann]

There is a "test-coroutine" which isn't in the list.
Add it so "make check" runs it too.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 configure |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/configure b/configure
index 44b3295..b677801 100755
--- a/configure
+++ b/configure
@@ -2631,7 +2631,7 @@ if test "$softmmu" = yes ; then
     fi
     if [ "$check_utests" = "yes" ]; then
       checks="check-qint check-qstring check-qdict check-qlist"
-      checks="check-qfloat check-qjson $checks"
+      checks="check-qfloat check-qjson test-coroutine $checks"
     fi
   fi
 fi
-- 
1.7.1

Re: [Qemu-devel] [PATCH 1/4] Probe for libcheck by default.

[Anthony Liguori]

On 09/01/2011 10:42 AM, Gerd Hoffmann wrote:
> Probe for libcheck and build checks (if found) by default.
> Can be explicitly disabled using --disable-check-utests.
>
> Signed-off-by: Gerd Hoffmann<kraxel@redhat.com>
> ---
>   configure |    2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)

I think we should convert the check tests to gtest, and then have make 
check use gtester and gtester-report generate a single report of all of 
the test cases.

We could then have build bot run make check and post the output.  I 
don't want to end up with a bunch of non gtest unit tests...

Regards,

Anthony Liguori

>
> diff --git a/configure b/configure
> index 1340c33..920de9e 100755
> --- a/configure
> +++ b/configure
> @@ -170,7 +170,7 @@ mixemu="no"
>   aix="no"
>   blobs="yes"
>   pkgversion=""
> -check_utests="no"
> +check_utests=""
>   user_pie="no"
>   zero_malloc=""
>   trace_backend="nop"

Re: [Qemu-devel] [PATCH 1/4] Probe for libcheck by default.

[Gerd Hoffmann]

On 09/01/11 21:37, Anthony Liguori wrote:
> On 09/01/2011 10:42 AM, Gerd Hoffmann wrote:
>> Probe for libcheck and build checks (if found) by default.
>> Can be explicitly disabled using --disable-check-utests.
>>
>> Signed-off-by: Gerd Hoffmann<kraxel@redhat.com>
>> ---
>> configure | 2 +-
>> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> I think we should convert the check tests to gtest, and then have make
> check use gtester and gtester-report generate a single report of all of
> the test cases.

I wouldn't object, that is *way* beyond the scope of this little patch 
series though.

> We could then have build bot run make check and post the output.

Running "make check" in buildbot is indeed the motivation to do this ;)

> I don't
> want to end up with a bunch of non gtest unit tests...

This patch series doesn't add any.  It just adds some build system glue 
so "make check" runs the existing stuff.

cheers,
   Gerd

Re: [Qemu-devel] [PATCH 1/4] Probe for libcheck by default.

[Markus Armbruster]

Gerd Hoffmann <kraxel@redhat.com> writes:

> On 09/01/11 21:37, Anthony Liguori wrote:
>> On 09/01/2011 10:42 AM, Gerd Hoffmann wrote:
>>> Probe for libcheck and build checks (if found) by default.
>>> Can be explicitly disabled using --disable-check-utests.
>>>
>>> Signed-off-by: Gerd Hoffmann<kraxel@redhat.com>
>>> ---
>>> configure | 2 +-
>>> 1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> I think we should convert the check tests to gtest, and then have make
>> check use gtester and gtester-report generate a single report of all of
>> the test cases.
>
> I wouldn't object, that is *way* beyond the scope of this little patch
> series though.

One step at a time.

>> We could then have build bot run make check and post the output.
>
> Running "make check" in buildbot is indeed the motivation to do this ;)
>
>> I don't
>> want to end up with a bunch of non gtest unit tests...
>
> This patch series doesn't add any.

Yep.  If you want to standardize on gtest, you get to convert or remove
the existing tests.  Until then, we use what we have.

>                                     It just adds some build system
> glue so "make check" runs the existing stuff.

Running self-tests shouldn't be made any harder than that.

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Markus Armbruster]

Gerd Hoffmann <kraxel@redhat.com> writes:

>   Hi,
>
> This patch series intends to make unit testing easier.  It adds a new
> "make check" target which can be used to run all unit tests which are
> currently in the tree.  It also enables the unit tests by default, so
> you don't have to re-run configure with a special switch.

Reviewed-by: Markus Armbruster <armbru@redhat.com>

One test fails, but Luiz has a fix in his tree.

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Eduardo Habkost]

On Mon, Sep 05, 2011 at 09:55:20AM +0200, Markus Armbruster wrote:
> Gerd Hoffmann <kraxel@redhat.com> writes:
> 
> >   Hi,
> >
> > This patch series intends to make unit testing easier.  It adds a new
> > "make check" target which can be used to run all unit tests which are
> > currently in the tree.  It also enables the unit tests by default, so
> > you don't have to re-run configure with a special switch.
> 
> Reviewed-by: Markus Armbruster <armbru@redhat.com>
> 
> One test fails, but Luiz has a fix in his tree.

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Now all tests are passing. Why this was not applied yet?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Anthony Liguori]

On 10/24/2011 01:43 PM, Eduardo Habkost wrote:
> On Mon, Sep 05, 2011 at 09:55:20AM +0200, Markus Armbruster wrote:
>> Gerd Hoffmann<kraxel@redhat.com>  writes:
>>
>>>    Hi,
>>>
>>> This patch series intends to make unit testing easier.  It adds a new
>>> "make check" target which can be used to run all unit tests which are
>>> currently in the tree.  It also enables the unit tests by default, so
>>> you don't have to re-run configure with a special switch.
>>
>> Reviewed-by: Markus Armbruster<armbru@redhat.com>
>>
>> One test fails, but Luiz has a fix in his tree.
>
> Reviewed-by: Eduardo Habkost<ehabkost@redhat.com>
>
> Now all tests are passing. Why this was not applied yet?

I was hoping for more, but maybe we just need to start here and grow 
organically, I'll queue it again.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Kevin Wolf]

Am 24.10.2011 20:57, schrieb Anthony Liguori:
> On 10/24/2011 01:43 PM, Eduardo Habkost wrote:
>> On Mon, Sep 05, 2011 at 09:55:20AM +0200, Markus Armbruster wrote:
>>> Gerd Hoffmann<kraxel@redhat.com>  writes:
>>>
>>>>    Hi,
>>>>
>>>> This patch series intends to make unit testing easier.  It adds a new
>>>> "make check" target which can be used to run all unit tests which are
>>>> currently in the tree.  It also enables the unit tests by default, so
>>>> you don't have to re-run configure with a special switch.
>>>
>>> Reviewed-by: Markus Armbruster<armbru@redhat.com>
>>>
>>> One test fails, but Luiz has a fix in his tree.
>>
>> Reviewed-by: Eduardo Habkost<ehabkost@redhat.com>
>>
>> Now all tests are passing. Why this was not applied yet?
> 
> I was hoping for more, but maybe we just need to start here and grow 
> organically, I'll queue it again.

A while ago I played with some simple IDE tests. It basically was a
small x86 kernel with an empty image that sends IDE commands and prints
some results, and a script that invokes the guest and checks whether the
test has passed or failed.

So at first I started with my own multiboot kernel and copied over some
parts of kvm-unittest's libc. Clearly not the best idea once it's more
than a couple of lines, so at some point I took the code and integrated
with my real kvm-unittests repository.

Now I don't have to duplicate code any more, but at the same time
there's no chance that a 'make check' in an upstream qemu tree could run
this. Tests for other devices will have exactly the same problem.

Any suggestions on how to go forward with this kind of tests? Should
this go into qemu or into kvm-unittests? Or should kvm-unittests be
merged into the qemu tree? Or is the approach completely wrong?

Kevin

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 1598 bytes --]

  Hi,

>> I was hoping for more, but maybe we just need to start here and grow 
>> organically, I'll queue it again.
> 
> A while ago I played with some simple IDE tests. It basically was a
> small x86 kernel with an empty image that sends IDE commands and prints
> some results, and a script that invokes the guest and checks whether the
> test has passed or failed.

That reminds me that I've started toying with running tests inside a
guest too.  Stopped working on it a while back due to other priorities.
 Attached what I have so far.

> So at first I started with my own multiboot kernel and copied over some
> parts of kvm-unittest's libc. Clearly not the best idea once it's more
> than a couple of lines, so at some point I took the code and integrated
> with my real kvm-unittests repository.
> 
> Now I don't have to duplicate code any more, but at the same time
> there's no chance that a 'make check' in an upstream qemu tree could run
> this. Tests for other devices will have exactly the same problem.
> 
> Any suggestions on how to go forward with this kind of tests? Should
> this go into qemu or into kvm-unittests? Or should kvm-unittests be
> merged into the qemu tree? Or is the approach completely wrong?

I think we should have some framework to run tests inside the guest in
the qemu source tree.  I'm not sure kvm-unittests is the right tool for
the job though.  It is quite low-level and mainly targets the kvm bits
inside the linux kernel.  Testing -- for example -- usb device emulation
would pretty much require writing a usb stack for kvm-unitests ...

cheers,
  Gerd

[-- Attachment #2: 0001-initramfs-test-framework.patch --]
[-- Type: text/plain, Size: 16053 bytes --]

From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
From: Gerd Hoffmann <kraxel@redhat.com>
Date: Tue, 13 Sep 2011 17:38:37 +0200
Subject: [PATCH] initramfs test framework

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 initramfs/.gitignore         |    3 +
 initramfs/10-qemu-udev.rules |    5 +
 initramfs/Makefile           |   36 +++++++
 initramfs/README             |   44 ++++++++
 initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
 initramfs/initramfs-boot     |   32 ++++++
 initramfs/initramfs-create   |  111 +++++++++++++++++++++
 initramfs/test-ehci          |    3 +
 initramfs/test-ehci.good     |    8 ++
 initramfs/test-hello.c       |    7 ++
 initramfs/test-hello.good    |    1 +
 initramfs/test-uhci          |    3 +
 initramfs/test-uhci.good     |    3 +
 13 files changed, 481 insertions(+), 0 deletions(-)
 create mode 100644 initramfs/.gitignore
 create mode 100644 initramfs/10-qemu-udev.rules
 create mode 100644 initramfs/Makefile
 create mode 100644 initramfs/README
 create mode 100644 initramfs/init.c
 create mode 100755 initramfs/initramfs-boot
 create mode 100755 initramfs/initramfs-create
 create mode 100755 initramfs/test-ehci
 create mode 100644 initramfs/test-ehci.good
 create mode 100644 initramfs/test-hello.c
 create mode 100644 initramfs/test-hello.good
 create mode 100755 initramfs/test-uhci
 create mode 100644 initramfs/test-uhci.good

diff --git a/initramfs/.gitignore b/initramfs/.gitignore
new file mode 100644
index 0000000..8ece42c
--- /dev/null
+++ b/initramfs/.gitignore
@@ -0,0 +1,3 @@
+initramfs.cpio.gz
+init
+test-hello
diff --git a/initramfs/10-qemu-udev.rules b/initramfs/10-qemu-udev.rules
new file mode 100644
index 0000000..fb5cc0a
--- /dev/null
+++ b/initramfs/10-qemu-udev.rules
@@ -0,0 +1,5 @@
+# load modules
+DRIVER!="?*", ENV{MODALIAS}=="?*", RUN+="/sbin/modprobe -b $env{MODALIAS}"
+
+# virtio console
+KERNEL=="vport*", ATTR{name}=="?*", SYMLINK+="virtio-ports/$attr{name}"
diff --git a/initramfs/Makefile b/initramfs/Makefile
new file mode 100644
index 0000000..2db2b76
--- /dev/null
+++ b/initramfs/Makefile
@@ -0,0 +1,36 @@
+
+CC	:= gcc
+CFLAGS	:= -O2 -g -Wall
+LDFLAGS	:= -lutil
+
+TOOLS	:= init
+TEST_C	:= test-hello
+TEST_SH	:= test-uhci test-ehci
+
+TARGETS := $(TOOLS) $(TEST_C) initramfs.cpio.gz
+TESTS	:= $(TEST_C) $(TEST_SH)
+
+run-test-uhci : QEMU_ARGS := -usb -device usb-tablet
+run-test-ehci : QEMU_ARGS := -readconfig ../docs/ich9-ehci-uhci.cfg
+
+default all: $(TARGETS)
+
+clean:
+	rm -f $(TARGETS) *.o *~
+	rm -f org.qemu.initramfs.log
+
+init: init.o
+test-hello: test-hello.o
+
+initramfs.cpio.gz: $(TOOLS) $(TESTS) initramfs-create
+	./initramfs-create $@ $(TESTS)
+
+boot shell: $(TARGETS)
+	./initramfs-boot
+
+run-test-%: $(TARGETS)
+	./initramfs-boot /tests/test-$* $(QEMU_ARGS)
+	diff -u org.qemu.initramfs.log test-$*.good
+
+run-tests: $(patsubst %,run-%,$(TESTS))
+
diff --git a/initramfs/README b/initramfs/README
new file mode 100644
index 0000000..a9504ad
--- /dev/null
+++ b/initramfs/README
@@ -0,0 +1,44 @@
+
+This is an experimental test framework.
+
+Design goals
+------------
+
+  * Allow running tests within a guest.
+  * Be small enougth that it can easily be included in
+    the qemu source tree.
+  * Don't require setup and/or downloading stuff
+    (i.e. guest images) from the internet.
+  * Be easy to use.
+
+
+How it works
+------------
+
+It creates a linux initramfs from the bits found on the host machine,
+then boots the host kernel with the initramfs just created within
+qemu.  A special init handles the setup (create core devices, mount
+filesystems, start udev), command execution and shutdown.
+
+Obviously requires a linux host.  It also needs udev for device setup
+and module loading and virtio-serial support for logging.
+
+
+Getting started
+---------------
+
+Just type "make boot", a few seconds later you should be greeted by
+the guests bash prompt.  You can look around now.  Exiting the shell
+will shutdown the guest.  You'll find the shell output logged in the
+"org.qemu.initramfs.log" file.
+
+
+Run tests
+---------
+
+Type "make run-test-$name" for a single test or "make run-tests" to
+run all of them.  This will run the test program instead of the shell,
+then compare the actual output with the expected output.
+
+This can probably be refined ...
+
diff --git a/initramfs/init.c b/initramfs/init.c
new file mode 100644
index 0000000..94a8764
--- /dev/null
+++ b/initramfs/init.c
@@ -0,0 +1,225 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <string.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <pty.h>
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/mount.h>
+#include <sys/reboot.h>
+#include <sys/select.h>
+
+#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
+
+static const char *logdev = "/dev/virtio-ports/org.qemu.initramfs.log";
+
+static struct {
+    mode_t     mode;
+    const char *path;
+    int        major;
+    int        minor;
+} cdevs[] = {
+    { 0666, "/dev/null",    1,  3 },
+    { 0600, "/dev/console", 5,  1 },
+    { 0666, "/dev/ptmx",    5,  2 },
+    { 0660, "/dev/kmsg",    1, 11 },
+};
+
+static struct {
+    const char *dest;
+    const char *type;
+} fs[] = {
+    { "/proc", "proc" },
+    { "/sys", "sysfs" },
+    { "/sys/kernel/debug", "debugfs" },
+    { "/dev/pts", "devpts" },
+    { "/dev/shm", "tmpfs" },
+};
+
+int run(const char *cmd, ...)
+{
+    va_list args;
+    char *argv[16], *arg;
+    int i, pid;
+
+    va_start(args, cmd);
+    argv[0] = (char*)cmd;
+    for (i = 1; i < ARRAY_SIZE(argv)-1; i++) {
+        arg = va_arg(args, char*);
+        if (arg == NULL) {
+            break;
+        }
+        argv[i] = arg;
+    }
+    argv[i] = NULL;
+    va_end(args);
+
+    pid = fork();
+    if (pid != 0) {
+        /* parent */
+        if (pid < 0) {
+            perror("fork");
+        }
+        return pid;
+    }
+    /* child */
+    execv(cmd, argv);
+    fprintf(stderr, "exec %s: %s\n", cmd, strerror(errno));
+    exit(1);
+}
+
+int forward(int from_fd, int to_fd, int log_fd)
+{
+    int len, pos, rc;
+    char buf[512];
+
+    len = read(from_fd, buf, sizeof(buf));
+    if (len < 0) {
+        return len;
+    }
+    for (pos = 0; pos < len; pos += rc) {
+        rc = write(to_fd, buf + pos, len - pos);
+        if (rc < 0) {
+            return rc;
+        }
+    }
+    if (log_fd != -1) {
+        for (pos = 0; pos < len; pos += rc) {
+            rc = write(log_fd, buf + pos, len - pos);
+            if (rc < 0) {
+                return rc;
+            }
+        }
+    }
+    return len;
+}
+
+static void
+tty_raw(int fd)
+{
+    struct termios tattr;
+
+    tcgetattr(fd, &tattr);
+    tattr.c_lflag &= ~(ICANON|ECHO);
+    tattr.c_cc[VMIN] = 1;
+    tattr.c_cc[VTIME] = 0;
+    tcsetattr(fd, TCSAFLUSH, &tattr);
+}
+
+int main(int argc, char *argv[])
+{
+    const char *command;
+    pid_t cmdpid, pid;
+    int status, i, rc, pty, log;
+    bool cmd_exit, eof_seen;
+
+    /* say hi */
+    fprintf(stderr, "-*- qemu initramfs starting -*-\n");
+
+    /* mount filesystems */
+    for (i = 0; i < ARRAY_SIZE(fs); i++) {
+        rc = mount(fs[i].type, fs[i].dest, fs[i].type, 0, NULL);
+        if (rc < 0) {
+            fprintf(stderr, "mount %s at %s: %s\n",
+                    fs[i].type, fs[i].dest, strerror(errno));
+        }
+    }
+
+    /* create basic chardevs */
+    for (i = 0; i < ARRAY_SIZE(cdevs); i++) {
+        rc = mknod(cdevs[i].path, cdevs[i].mode | S_IFCHR,
+                   makedev(cdevs[i].major, cdevs[i].minor));
+        if (rc < 0 && errno != EEXIST) {
+            fprintf(stderr, "mknod %s : %s\n",
+                    cdevs[i].path, strerror(errno));
+        }
+    }
+
+    /* start udev, let it create devices */
+    fprintf(stderr, "-*- starting udev -*-\n");
+    run("/sbin/depmod", "-a", NULL);
+    run("/sbin/udevd", "--daemon", "--resolve-names=never", NULL);
+    run("/sbin/udevadm", "trigger", "--action=add", NULL);
+
+    /* wait until virtio-serial is up'n'running */
+    for (log = -1, i = 0; log == -1 && i < 32; i++) {
+        sleep(1);
+        run("/sbin/udevadm", "settle", NULL);
+        log = open(logdev, O_WRONLY);
+    }
+    if (log == -1) {
+        fprintf(stderr, "open %s: %s\n", logdev, strerror(errno));
+    }
+
+    /* setup environment */
+    setenv("PATH", "/sbin:/bin", 1);
+
+    /* run our command in a pseuto tty */
+    command = getenv("QEMU_RUN");
+    if (command == NULL) {
+        command = "/bin/bash";
+    }
+    fprintf(stderr, "-*- running %s -*-\n", command);
+
+    tty_raw(0);
+    cmdpid = forkpty(&pty, NULL, NULL, NULL);
+    if (cmdpid == 0) {
+        /* child */
+        execl(command, command, NULL);
+        fprintf(stderr, "exec %s: %s\n", command, strerror(errno));
+        exit(1);
+    }
+
+    /* main loop */
+    cmd_exit = false;
+    eof_seen = false;
+    do {
+        fd_set rd;
+
+        /* reap children and zombies */
+        for (;;) {
+            pid = waitpid(-1, &status, WNOHANG);
+            if (pid <= 0) {
+                break;
+            }
+            if (pid == cmdpid) {
+                cmd_exit = true;
+            }
+        }
+
+        /* forward data between stdio (aka /dev/console) and pseudo tty */
+        if (!eof_seen) {
+            FD_ZERO(&rd);
+            FD_SET(0, &rd);
+            FD_SET(pty, &rd);
+            rc = select(pty+1, &rd, NULL, NULL, NULL);
+            if (rc > 0) {
+                if (FD_ISSET(0, &rd)) {
+                    /* stdin -> pseudo tty */
+                    if (forward(0, pty, -1) <= 0) {
+                        eof_seen = 1;
+                    }
+                }
+                if (FD_ISSET(pty, &rd)) {
+                    /* pseudo tty -> stdout + log */
+                    if (forward(pty, 1, log) <= 0) {
+                        eof_seen = 1;
+                    }
+                }
+            }
+        }
+    } while (!cmd_exit || !eof_seen);
+
+    /* powerdown vm */
+    fprintf(stderr, "-*- qemu initramfs done -*-\n");
+    reboot(RB_POWER_OFF);
+
+    /* keep gcc happy ;) */
+    return 0;
+}
diff --git a/initramfs/initramfs-boot b/initramfs/initramfs-boot
new file mode 100755
index 0000000..4884a6d
--- /dev/null
+++ b/initramfs/initramfs-boot
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+base="$(dirname $0)"
+command="$1"
+shift;
+
+case "$(uname -m)" in
+	x86_64)	qemu="x86_64-softmmu/qemu-system-x86_64"
+		;;
+	i?86)	qemu="i386-softmmu/qemu-system-i386"
+		;;
+	*)	echo "unknown arch: $(uname -m)"
+		exit 1
+		;;
+esac
+
+kernel="$(echo /boot/vmlinu*$(uname -r)*)"
+initrd="initramfs.cpio.gz"
+append="console=ttyS0"
+if test "$command" != ""; then
+	append="$append QEMU_RUN=$command"
+fi
+
+exec ../$qemu -nographic -no-reboot \
+	-machine accel=kvm:tcg \
+	-kernel "$kernel" \
+	-initrd "$initrd" \
+	-append "$append" \
+	-chardev file,id=cmdlog,path=org.qemu.initramfs.log \
+	-device virtio-serial \
+	-device virtserialport,name=org.qemu.initramfs.log,chardev=cmdlog \
+	"$@"
diff --git a/initramfs/initramfs-create b/initramfs/initramfs-create
new file mode 100755
index 0000000..4525c11
--- /dev/null
+++ b/initramfs/initramfs-create
@@ -0,0 +1,111 @@
+#!/bin/bash
+#
+# Create a simple linux initramfs with the stuff found on the host.
+# Intented to be used for a quick boot test with the host kernel.
+#
+
+base="$(dirname $0)"
+file="${1-initramfs.cpio.gz}"; shift
+tests="$*"
+
+# create work dir
+WORK="${TMPDIR-/tmp}/${0##*/}-$$"
+mkdir "$WORK" || exit 1
+trap 'rm -rf "$WORK"' EXIT
+
+
+##############################################################################
+# helper functions
+
+function add_dirs() {
+	local dest="$WORK/fs"
+	local dir
+	for dir in dev dev/pts dev/shm etc proc sys \
+		   var var/tmp /var/log tmp
+	do
+		mkdir -p "$dest/$dir"
+	done
+}
+
+function add_binary() {
+	local dest="$WORK/fs$1"; shift
+	local item
+
+       	mkdir -p "$dest"
+	for item in $*; do
+		cp -L "$(which $item)" "$dest" || exit 1
+	done
+}
+
+function add_symlink() {
+	local target="$1"
+	local name="$2"
+	ln -s "$target" "$WORK/fs$name"
+}
+
+function add_data() {
+	local dest="$WORK/fs"
+	local item
+
+	for item in $*; do
+	       	mkdir -p $(dirname "$dest/$item")
+		cp -L "$item" "$dest/$item" || exit 1
+	done
+}
+
+function add_data_to_dir() {
+	local dest="$WORK/fs$1"; shift
+	local item
+
+       	mkdir -p "$dest"
+	for item in $*; do
+		cp -L "$item" "$dest" || exit 1
+	done
+}
+
+function add_libs() {
+	local item dest
+	for item in $(ldd $WORK/fs/init $WORK/fs/bin/* $WORK/fs/sbin/* \
+			$WORK/fs/tests/* $WORK/fs/lib/udev/*id); do
+		test -f "$item" || continue
+		test -f "$WORK/fs$item" && continue
+		dest=$(dirname "$WORK/fs$item")
+		mkdir -p "$dest"
+		cp -L "$item" "$dest"
+	done
+}
+
+function add_modules() {
+	local item
+
+	echo -n > "$WORK/modules"
+	for item in $*; do
+		modprobe --show-depends $item 2>/dev/null \
+			| awk '{ print $2 }' >> "$WORK/modules"
+	done
+	add_data $(sort "$WORK/modules" | uniq)
+}
+
+
+##############################################################################
+# main
+
+add_dirs
+add_binary / $base/init
+for t in $tests; do
+	add_binary /tests $base/$t;
+done
+add_binary /bin bash ls cat more dmesg ps uname find sort grep uniq
+add_binary /sbin lspci lsusb mount umount udevd udevadm blkid
+add_binary /sbin depmod insmod lsmod modinfo modprobe rmmod ip
+add_symlink bash /bin/sh
+add_data /usr/share/hwdata/*.ids
+add_data /lib/udev/*id
+add_data /lib/udev/rules.d/60-persistent-storage.rules
+add_data_to_dir /lib/udev/rules.d $base/10-qemu-udev.rules
+add_libs
+add_modules virtio_pci virtio_blk virtio_net virtio_console \
+	virtio_balloon virtio-rng 9pnet_virtio 9p \
+	ata_piix ahci sd_mod sr_mod sg e1000 8139cp
+
+(cd $WORK/fs; find -print | cpio -o -R 0:0 -H newc) | gzip > "$file"
diff --git a/initramfs/test-ehci b/initramfs/test-ehci
new file mode 100755
index 0000000..bd4bb5c
--- /dev/null
+++ b/initramfs/test-ehci
@@ -0,0 +1,3 @@
+#!/bin/sh
+/sbin/lspci -s1d
+/sbin/lsusb | sort
diff --git a/initramfs/test-ehci.good b/initramfs/test-ehci.good
new file mode 100644
index 0000000..dd9c6b8
--- /dev/null
+++ b/initramfs/test-ehci.good
@@ -0,0 +1,8 @@
+00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
+00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
+00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
+00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
+Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
diff --git a/initramfs/test-hello.c b/initramfs/test-hello.c
new file mode 100644
index 0000000..e5ce2d2
--- /dev/null
+++ b/initramfs/test-hello.c
@@ -0,0 +1,7 @@
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+    printf("Hello world!\n");
+    return 0;
+}
diff --git a/initramfs/test-hello.good b/initramfs/test-hello.good
new file mode 100644
index 0000000..dfd6895
--- /dev/null
+++ b/initramfs/test-hello.good
@@ -0,0 +1 @@
+Hello world!
diff --git a/initramfs/test-uhci b/initramfs/test-uhci
new file mode 100755
index 0000000..0af70e0
--- /dev/null
+++ b/initramfs/test-uhci
@@ -0,0 +1,3 @@
+#!/bin/sh
+/sbin/lspci -s1.2
+/sbin/lsusb | sort
\ No newline at end of file
diff --git a/initramfs/test-uhci.good b/initramfs/test-uhci.good
new file mode 100644
index 0000000..c87271e
--- /dev/null
+++ b/initramfs/test-uhci.good
@@ -0,0 +1,3 @@
+00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
+Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
-- 
1.7.1

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Kevin Wolf]

Am 25.10.2011 15:27, schrieb Gerd Hoffmann:
>   Hi,
> 
>>> I was hoping for more, but maybe we just need to start here and grow 
>>> organically, I'll queue it again.
>>
>> A while ago I played with some simple IDE tests. It basically was a
>> small x86 kernel with an empty image that sends IDE commands and prints
>> some results, and a script that invokes the guest and checks whether the
>> test has passed or failed.
> 
> That reminds me that I've started toying with running tests inside a
> guest too.  Stopped working on it a while back due to other priorities.
>  Attached what I have so far.
> 
>> So at first I started with my own multiboot kernel and copied over some
>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>> than a couple of lines, so at some point I took the code and integrated
>> with my real kvm-unittests repository.
>>
>> Now I don't have to duplicate code any more, but at the same time
>> there's no chance that a 'make check' in an upstream qemu tree could run
>> this. Tests for other devices will have exactly the same problem.
>>
>> Any suggestions on how to go forward with this kind of tests? Should
>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>> merged into the qemu tree? Or is the approach completely wrong?
> 
> I think we should have some framework to run tests inside the guest in
> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
> the job though.  It is quite low-level and mainly targets the kvm bits
> inside the linux kernel.  Testing -- for example -- usb device emulation
> would pretty much require writing a usb stack for kvm-unitests ...

Which is more or less the only way to do a reasonably complete test
(particularly including the error handling paths that a current Linux
might never exercise) at least for the host controllers.

I'm not sure if you can send arbitrary USB packets from Linux (is USB
passthrough really direct passthrough or does the kernel mess with it?),
so maybe for everything above host controllers you could use that. If
you can't, you're back to your own USB stack.

Kevin

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Eduardo Habkost]

On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
[...]
> > A while ago I played with some simple IDE tests. It basically was a
> > small x86 kernel with an empty image that sends IDE commands and prints
> > some results, and a script that invokes the guest and checks whether the
> > test has passed or failed.
> 
> That reminds me that I've started toying with running tests inside a
> guest too.  Stopped working on it a while back due to other priorities.
>  Attached what I have so far.
> 
> > So at first I started with my own multiboot kernel and copied over some
> > parts of kvm-unittest's libc. Clearly not the best idea once it's more
> > than a couple of lines, so at some point I took the code and integrated
> > with my real kvm-unittests repository.
> > 
> > Now I don't have to duplicate code any more, but at the same time
> > there's no chance that a 'make check' in an upstream qemu tree could run
> > this. Tests for other devices will have exactly the same problem.
> > 
> > Any suggestions on how to go forward with this kind of tests? Should
> > this go into qemu or into kvm-unittests? Or should kvm-unittests be
> > merged into the qemu tree? Or is the approach completely wrong?
> 
> I think we should have some framework to run tests inside the guest in
> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
> the job though.  It is quite low-level and mainly targets the kvm bits
> inside the linux kernel.  Testing -- for example -- usb device emulation
> would pretty much require writing a usb stack for kvm-unitests ...

We have a framework to run tests inside a fully-installed guest, that's
KVM-Autotest. But maybe it's too much for the kind of tests you need, I
don't know. There are different "levels" of testing, with different
reequirements, and we need to have good tools for all levels.

Just trying to enumerate the kind of tests somebody may need:

A) Simple unit tests for internal qemu C functions
   - 'make check' can run them, using either libcheck or gtest.
B) Functional tests that tests actual virtualization/emulation, but only
   of some specific subsystems, not using a fully-featured qemu process.
   - We don't have anything like that, today, right? I am not sure we
     need it.
C) Functional tests that just need to run a small binary with no OS
   installed in the guest, but running a fully-feature qemu process.
   - The tests in the 'tests' directory do this, right? kvm-unittests
     does this, right?
D) Functional tests that need a minimal OS installed, with, e.g., at
   least a Linux kernel and a shell.
   - This is what Gerd's patch below does, right? Also, KVM-Autotest can
     be used for this.
E) Functional tests that need a full OS installed and configured.
   - Today we use KVM-Autotest for this.


Does the above model look correct/complete, or is there some case I
missed?


> 
> cheers,
>   Gerd

> From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
> From: Gerd Hoffmann <kraxel@redhat.com>
> Date: Tue, 13 Sep 2011 17:38:37 +0200
> Subject: [PATCH] initramfs test framework
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  initramfs/.gitignore         |    3 +
>  initramfs/10-qemu-udev.rules |    5 +
>  initramfs/Makefile           |   36 +++++++
>  initramfs/README             |   44 ++++++++
>  initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
>  initramfs/initramfs-boot     |   32 ++++++
>  initramfs/initramfs-create   |  111 +++++++++++++++++++++
>  initramfs/test-ehci          |    3 +
>  initramfs/test-ehci.good     |    8 ++
>  initramfs/test-hello.c       |    7 ++
>  initramfs/test-hello.good    |    1 +
>  initramfs/test-uhci          |    3 +
>  initramfs/test-uhci.good     |    3 +
>  13 files changed, 481 insertions(+), 0 deletions(-)
>  create mode 100644 initramfs/.gitignore
>  create mode 100644 initramfs/10-qemu-udev.rules
>  create mode 100644 initramfs/Makefile
>  create mode 100644 initramfs/README
>  create mode 100644 initramfs/init.c
>  create mode 100755 initramfs/initramfs-boot
>  create mode 100755 initramfs/initramfs-create
>  create mode 100755 initramfs/test-ehci
>  create mode 100644 initramfs/test-ehci.good
>  create mode 100644 initramfs/test-hello.c
>  create mode 100644 initramfs/test-hello.good
>  create mode 100755 initramfs/test-uhci
>  create mode 100644 initramfs/test-uhci.good
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Anthony Liguori]

On 10/25/2011 07:21 AM, Kevin Wolf wrote:
> Am 24.10.2011 20:57, schrieb Anthony Liguori:
>> On 10/24/2011 01:43 PM, Eduardo Habkost wrote:
>>> On Mon, Sep 05, 2011 at 09:55:20AM +0200, Markus Armbruster wrote:
>>>> Gerd Hoffmann<kraxel@redhat.com>   writes:
>>>>
>>>>>     Hi,
>>>>>
>>>>> This patch series intends to make unit testing easier.  It adds a new
>>>>> "make check" target which can be used to run all unit tests which are
>>>>> currently in the tree.  It also enables the unit tests by default, so
>>>>> you don't have to re-run configure with a special switch.
>>>>
>>>> Reviewed-by: Markus Armbruster<armbru@redhat.com>
>>>>
>>>> One test fails, but Luiz has a fix in his tree.
>>>
>>> Reviewed-by: Eduardo Habkost<ehabkost@redhat.com>
>>>
>>> Now all tests are passing. Why this was not applied yet?
>>
>> I was hoping for more, but maybe we just need to start here and grow
>> organically, I'll queue it again.
>
> A while ago I played with some simple IDE tests. It basically was a
> small x86 kernel with an empty image that sends IDE commands and prints
> some results, and a script that invokes the guest and checks whether the
> test has passed or failed.
>
> So at first I started with my own multiboot kernel and copied over some
> parts of kvm-unittest's libc. Clearly not the best idea once it's more
> than a couple of lines, so at some point I took the code and integrated
> with my real kvm-unittests repository.

I did something very similar.  I just added PCI support to kvm-unittests such 
that I could write a small virtio driver.

I actually do think tests like this could live in a make check framework in QEMU.

Regards,

Anthony Liguori

>
> Now I don't have to duplicate code any more, but at the same time
> there's no chance that a 'make check' in an upstream qemu tree could run
> this. Tests for other devices will have exactly the same problem.
>
> Any suggestions on how to go forward with this kind of tests? Should
> this go into qemu or into kvm-unittests? Or should kvm-unittests be
> merged into the qemu tree? Or is the approach completely wrong?
>
> Kevin
>

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Kevin Wolf]

Am 25.10.2011 17:03, schrieb Eduardo Habkost:
> On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
> [...]
>>> A while ago I played with some simple IDE tests. It basically was a
>>> small x86 kernel with an empty image that sends IDE commands and prints
>>> some results, and a script that invokes the guest and checks whether the
>>> test has passed or failed.
>>
>> That reminds me that I've started toying with running tests inside a
>> guest too.  Stopped working on it a while back due to other priorities.
>>  Attached what I have so far.
>>
>>> So at first I started with my own multiboot kernel and copied over some
>>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>>> than a couple of lines, so at some point I took the code and integrated
>>> with my real kvm-unittests repository.
>>>
>>> Now I don't have to duplicate code any more, but at the same time
>>> there's no chance that a 'make check' in an upstream qemu tree could run
>>> this. Tests for other devices will have exactly the same problem.
>>>
>>> Any suggestions on how to go forward with this kind of tests? Should
>>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>>> merged into the qemu tree? Or is the approach completely wrong?
>>
>> I think we should have some framework to run tests inside the guest in
>> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
>> the job though.  It is quite low-level and mainly targets the kvm bits
>> inside the linux kernel.  Testing -- for example -- usb device emulation
>> would pretty much require writing a usb stack for kvm-unitests ...
> 
> We have a framework to run tests inside a fully-installed guest, that's
> KVM-Autotest. But maybe it's too much for the kind of tests you need, I
> don't know. There are different "levels" of testing, with different
> reequirements, and we need to have good tools for all levels.

Autotest works for an entirely different use case.

Autotest is the right tool when you can let the tests run over night or
over the weekend. Good to run regularly on the current git tree, the
right thing to do to test release candidates.

It's the wrong tool for testing the patch that I'm currently working on.
The focus here is on being able to run very quick tests. I don't want to
wait for a guest installation to complete, and I don't even necessarily
want to wait for a full-blown Linux guest to boot up.

> Just trying to enumerate the kind of tests somebody may need:
> 
> A) Simple unit tests for internal qemu C functions
>    - 'make check' can run them, using either libcheck or gtest.
> B) Functional tests that tests actual virtualization/emulation, but only
>    of some specific subsystems, not using a fully-featured qemu process.
>    - We don't have anything like that, today, right? I am not sure we
>      need it.

I think qemu-iotests could be considered an instance of B)

> C) Functional tests that just need to run a small binary with no OS
>    installed in the guest, but running a fully-feature qemu process.
>    - The tests in the 'tests' directory do this, right? kvm-unittests
>      does this, right?

Not sure what test/ does, but for kvm-unittests yes. And this is also
what I was talking about.

> D) Functional tests that need a minimal OS installed, with, e.g., at
>    least a Linux kernel and a shell.
>    - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>      be used for this.
> E) Functional tests that need a full OS installed and configured.
>    - Today we use KVM-Autotest for this.
> 
> 
> Does the above model look correct/complete, or is there some case I
> missed?

I think it covers what we need.

Kevin

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Gerd Hoffmann]

  Hi,

> Just trying to enumerate the kind of tests somebody may need:
> 
> A) Simple unit tests for internal qemu C functions
>    - 'make check' can run them, using either libcheck or gtest.

Yes, those tests should work and succeed anywhere where qemu compiles.

> B) Functional tests that tests actual virtualization/emulation, but only
>    of some specific subsystems, not using a fully-featured qemu process.
>    - We don't have anything like that, today, right? I am not sure we
>      need it.

qemu-kvm used to have such tests.  There is no point in not using qemu
here though, thats why they where transformed into (C).

> C) Functional tests that just need to run a small binary with no OS
>    installed in the guest, but running a fully-feature qemu process.
>    - The tests in the 'tests' directory do this, right? kvm-unittests
>      does this, right?

kvm-unittests yes.

tests/ seems to be a somewhat bitrotted random collection of tests.

> D) Functional tests that need a minimal OS installed, with, e.g., at
>    least a Linux kernel and a shell.
>    - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>      be used for this.

What the patch tries is to use the bits which are on the host anyway to
run guest tests (obviously linux-on-linux only), so we can do parts of
(E) with less configuration, less internet bandwith (no iso downloads),
less disk space requirements (no iso + disk storage) and faster (no
guest installs needed).  Didn't try yet to mount the host filesystem
using 9p yet, with that we could possibly take the approach quite far as
we suddenly don't have just a "minimal os" any more but alot more.

Using the host bits is at the same time a possible trouble spot.  There
are lots of different distros with more or less subtile differences out
there.  Each machine has a different set of packages installed.  So
having this "just work" everythere probably isn't that easy ...

> E) Functional tests that need a full OS installed and configured.
>    - Today we use KVM-Autotest for this.

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Lucas Meneghel Rodrigues]

On 10/25/2011 01:03 PM, Eduardo Habkost wrote:
> On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
> [...]
>>> A while ago I played with some simple IDE tests. It basically was a
>>> small x86 kernel with an empty image that sends IDE commands and prints
>>> some results, and a script that invokes the guest and checks whether the
>>> test has passed or failed.
>>
>> That reminds me that I've started toying with running tests inside a
>> guest too.  Stopped working on it a while back due to other priorities.
>>   Attached what I have so far.
>>
>>> So at first I started with my own multiboot kernel and copied over some
>>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>>> than a couple of lines, so at some point I took the code and integrated
>>> with my real kvm-unittests repository.
>>>
>>> Now I don't have to duplicate code any more, but at the same time
>>> there's no chance that a 'make check' in an upstream qemu tree could run
>>> this. Tests for other devices will have exactly the same problem.
>>>
>>> Any suggestions on how to go forward with this kind of tests? Should
>>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>>> merged into the qemu tree? Or is the approach completely wrong?
>>
>> I think we should have some framework to run tests inside the guest in
>> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
>> the job though.  It is quite low-level and mainly targets the kvm bits
>> inside the linux kernel.  Testing -- for example -- usb device emulation
>> would pretty much require writing a usb stack for kvm-unitests ...
>
> We have a framework to run tests inside a fully-installed guest, that's
> KVM-Autotest. But maybe it's too much for the kind of tests you need, I
> don't know. There are different "levels" of testing, with different
> reequirements, and we need to have good tools for all levels.
>
> Just trying to enumerate the kind of tests somebody may need:
>
> A) Simple unit tests for internal qemu C functions
>     - 'make check' can run them, using either libcheck or gtest.
> B) Functional tests that tests actual virtualization/emulation, but only
>     of some specific subsystems, not using a fully-featured qemu process.
>     - We don't have anything like that, today, right? I am not sure we
>       need it.
> C) Functional tests that just need to run a small binary with no OS
>     installed in the guest, but running a fully-feature qemu process.
>     - The tests in the 'tests' directory do this, right? kvm-unittests
>       does this, right?
> D) Functional tests that need a minimal OS installed, with, e.g., at
>     least a Linux kernel and a shell.
>     - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>       be used for this.

Yes, it can. There's instrumentation to get ssh sessions, serial 
sessions, execute monitor commands... Only need to specify the path to a 
pre-installed minimal guest, skip install tests and have the bare 
minimal subset of tests needed. If only a small subset of tests are run 
with a minimal image, a kvm autotest job is quick to run and suitable 
for development use. I've looked at Gerd's initial patch to add an 
initramfs for such lightweight testing, and the init ramdisk generation 
could be implemented in autotest.

However...

Problem today is that KVM autotest is considered to be hard to approach 
and integrate with qemu developer's workflow. I won't argue it's not, so 
I'm admitting failure here. We've been juggling to keep it stable for 
the regular, full blown QE case, with expanding test and targets 
coverage (see libvirt test), and yet trying to make it better documented 
and sufficiently easy to approach to do simpler ad hoc testing. I wish 
it was easier to accomplish the later, though.

So, an in-tree solution for D) like the one Gerd started to work on 
seems better, as it won't generate frustration from folks having to 
learn kvm autotest, which will ultimately lead to them ignoring the effort.

> E) Functional tests that need a full OS installed and configured.
>     - Today we use KVM-Autotest for this.
>
>
> Does the above model look correct/complete, or is there some case I
> missed?
>
>
>>
>> cheers,
>>    Gerd
>
>>  From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
>> From: Gerd Hoffmann<kraxel@redhat.com>
>> Date: Tue, 13 Sep 2011 17:38:37 +0200
>> Subject: [PATCH] initramfs test framework
>>
>> Signed-off-by: Gerd Hoffmann<kraxel@redhat.com>
>> ---
>>   initramfs/.gitignore         |    3 +
>>   initramfs/10-qemu-udev.rules |    5 +
>>   initramfs/Makefile           |   36 +++++++
>>   initramfs/README             |   44 ++++++++
>>   initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
>>   initramfs/initramfs-boot     |   32 ++++++
>>   initramfs/initramfs-create   |  111 +++++++++++++++++++++
>>   initramfs/test-ehci          |    3 +
>>   initramfs/test-ehci.good     |    8 ++
>>   initramfs/test-hello.c       |    7 ++
>>   initramfs/test-hello.good    |    1 +
>>   initramfs/test-uhci          |    3 +
>>   initramfs/test-uhci.good     |    3 +
>>   13 files changed, 481 insertions(+), 0 deletions(-)
>>   create mode 100644 initramfs/.gitignore
>>   create mode 100644 initramfs/10-qemu-udev.rules
>>   create mode 100644 initramfs/Makefile
>>   create mode 100644 initramfs/README
>>   create mode 100644 initramfs/init.c
>>   create mode 100755 initramfs/initramfs-boot
>>   create mode 100755 initramfs/initramfs-create
>>   create mode 100755 initramfs/test-ehci
>>   create mode 100644 initramfs/test-ehci.good
>>   create mode 100644 initramfs/test-hello.c
>>   create mode 100644 initramfs/test-hello.good
>>   create mode 100755 initramfs/test-uhci
>>   create mode 100644 initramfs/test-uhci.good
>>
>

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Anthony Liguori]

On 10/25/2011 10:22 AM, Kevin Wolf wrote:
> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
> I think qemu-iotests could be considered an instance of B)
>
>> C) Functional tests that just need to run a small binary with no OS
>>     installed in the guest, but running a fully-feature qemu process.
>>     - The tests in the 'tests' directory do this, right? kvm-unittests
>>       does this, right?
>
> Not sure what test/ does, but for kvm-unittests yes. And this is also
> what I was talking about.

Thinking more about this...

We could add a new '-x-test-server CHR' option.  When this option is added, it 
would do the following:

1) Open CHR character device
2) Use /dev/shm for guest memory
3) Listen for connections on CHR
4) When something connects to CHR
  a) reset device model
  b) send /dev/shm fd over CHR
  c) register CPU physical memory client
     1. upon CPU physical memory changes, send the change info over CHR
  d) instead of doing [kvm_]cpu_exec(), block reading on CHR

So when you launch qemu with -x-test-server, it'll sit there doing nothing 
terribly useful.  But this lets you write a program that connects to CHR, and 
then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via 
mmap()'ing the passed fd using the client mapping table, you can essentially 
write kvm-unittest style tests while still having full access to libc.

And since each test program can reset QEMU after running, you could very nicely 
tie into something like gtest as a unit test framework.  I think it's pretty 
appealing from a debugability perspective too.

It also means that it's possible to have 100% C test cases such that you could 
still build something like ppc64-softmmu and run it against the written test 
cases without having to really understand ppc64 assembly or have a ppc64 build 
environment (to generate native binaries to run under ppc64 TCG).

I think this could work out fairly well as a unit test framework.

Regards,

Anthony Liguori

>
>> D) Functional tests that need a minimal OS installed, with, e.g., at
>>     least a Linux kernel and a shell.
>>     - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>>       be used for this.
>> E) Functional tests that need a full OS installed and configured.
>>     - Today we use KVM-Autotest for this.
>>
>>
>> Does the above model look correct/complete, or is there some case I
>> missed?
>
> I think it covers what we need.
>
> Kevin
>

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Kevin Wolf]

Am 26.10.2011 22:49, schrieb Anthony Liguori:
> On 10/25/2011 10:22 AM, Kevin Wolf wrote:
>> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
>> I think qemu-iotests could be considered an instance of B)
>>
>>> C) Functional tests that just need to run a small binary with no OS
>>>     installed in the guest, but running a fully-feature qemu process.
>>>     - The tests in the 'tests' directory do this, right? kvm-unittests
>>>       does this, right?
>>
>> Not sure what test/ does, but for kvm-unittests yes. And this is also
>> what I was talking about.
> 
> Thinking more about this...
> 
> We could add a new '-x-test-server CHR' option.  When this option is added, it 
> would do the following:
> 
> 1) Open CHR character device
> 2) Use /dev/shm for guest memory
> 3) Listen for connections on CHR
> 4) When something connects to CHR
>   a) reset device model
>   b) send /dev/shm fd over CHR
>   c) register CPU physical memory client
>      1. upon CPU physical memory changes, send the change info over CHR
>   d) instead of doing [kvm_]cpu_exec(), block reading on CHR
> 
> So when you launch qemu with -x-test-server, it'll sit there doing nothing 
> terribly useful.  But this lets you write a program that connects to CHR, and 
> then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via 
> mmap()'ing the passed fd using the client mapping table, you can essentially 
> write kvm-unittest style tests while still having full access to libc.

IRQs need to go through the connection as well.

Oh, and you would finally have a C user for libqmp. The test cases
definitely need to be able to access the monitor. For example I would
really love to have test cases for the I/O error paths that stop the VM
(or actually it's the resume that must be tested).

> And since each test program can reset QEMU after running, you could very nicely 
> tie into something like gtest as a unit test framework.  I think it's pretty 
> appealing from a debugability perspective too.
> 
> It also means that it's possible to have 100% C test cases such that you could 
> still build something like ppc64-softmmu and run it against the written test 
> cases without having to really understand ppc64 assembly or have a ppc64 build 
> environment (to generate native binaries to run under ppc64 TCG).
> 
> I think this could work out fairly well as a unit test framework.

Sounds great, where are the patches? ;-)

Kevin

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Anthony Liguori]

On 10/27/2011 03:20 AM, Kevin Wolf wrote:
> Am 26.10.2011 22:49, schrieb Anthony Liguori:
>> On 10/25/2011 10:22 AM, Kevin Wolf wrote:
>>> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
>>> I think qemu-iotests could be considered an instance of B)
>>>
>>>> C) Functional tests that just need to run a small binary with no OS
>>>>      installed in the guest, but running a fully-feature qemu process.
>>>>      - The tests in the 'tests' directory do this, right? kvm-unittests
>>>>        does this, right?
>>>
>>> Not sure what test/ does, but for kvm-unittests yes. And this is also
>>> what I was talking about.
>>
>> Thinking more about this...
>>
>> We could add a new '-x-test-server CHR' option.  When this option is added, it
>> would do the following:
>>
>> 1) Open CHR character device
>> 2) Use /dev/shm for guest memory
>> 3) Listen for connections on CHR
>> 4) When something connects to CHR
>>    a) reset device model
>>    b) send /dev/shm fd over CHR
>>    c) register CPU physical memory client
>>       1. upon CPU physical memory changes, send the change info over CHR
>>    d) instead of doing [kvm_]cpu_exec(), block reading on CHR
>>
>> So when you launch qemu with -x-test-server, it'll sit there doing nothing
>> terribly useful.  But this lets you write a program that connects to CHR, and
>> then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via
>> mmap()'ing the passed fd using the client mapping table, you can essentially
>> write kvm-unittest style tests while still having full access to libc.
>
> IRQs need to go through the connection as well.

Yes, forgot to mention that.

>
> Oh, and you would finally have a C user for libqmp. The test cases
> definitely need to be able to access the monitor. For example I would
> really love to have test cases for the I/O error paths that stop the VM
> (or actually it's the resume that must be tested).

Yeah, tunnelling a monitor session sounds like a really good idea.

>> And since each test program can reset QEMU after running, you could very nicely
>> tie into something like gtest as a unit test framework.  I think it's pretty
>> appealing from a debugability perspective too.
>>
>> It also means that it's possible to have 100% C test cases such that you could
>> still build something like ppc64-softmmu and run it against the written test
>> cases without having to really understand ppc64 assembly or have a ppc64 build
>> environment (to generate native binaries to run under ppc64 TCG).
>>
>> I think this could work out fairly well as a unit test framework.
>
> Sounds great, where are the patches? ;-)

Heh, need to find a volunteer although I spent a few minutes this afternoon 
trying to figure out how hard it would be.  Turns out, it's much simpler than I 
expected if you do the same trick that Xen does.  Instead of mucking with 
hooking cpu_exec, Xen simply starts the CPUs in the halted state such that TCG 
simply never runs.  The following patch is all we really need.

test_init() just needs to register the appropriate file descriptor callbacks and 
then in the data path dispatch PIO/MMIO.  It would also need to override 
cpu_interrupt_handler to intercept interrupt operations.

diff --git a/hw/pc.c b/hw/pc.c
index eb4c2d8..f3fd32d 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -923,12 +923,18 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
      }
  }

+extern int test_allowed;
+
  static void pc_cpu_reset(void *opaque)
  {
      CPUState *env = opaque;

      cpu_reset(env);
-    env->halted = !cpu_is_bsp(env);
+    if (test_allowed) {
+        env->halted = 1;
+    } else {
+        env->halted = !cpu_is_bsp(env);
+    }
  }

  static CPUState *pc_new_cpu(const char *cpu_model)
diff --git a/vl.c b/vl.c
index 1ddb17b..adc626a 100644
--- a/vl.c
+++ b/vl.c
@@ -1988,6 +1988,19 @@ static int tcg_init(void)
      return 0;
  }

+static int test_init(void)
+{
+    printf("Hello World\n");
+    return 0;
+}
+
+static int test_available(void)
+{
+    return 1;
+}
+
+int test_allowed = 1;
+
  static struct {
      const char *opt_name;
      const char *name;
@@ -1998,6 +2011,7 @@ static struct {
      { "tcg", "tcg", tcg_available, tcg_init, &tcg_allowed },
      { "xen", "Xen", xen_available, xen_init, &xen_allowed },
      { "kvm", "KVM", kvm_available, kvm_init, &kvm_allowed },
+    { "test", "Test", test_available, test_init, &test_allowed },
  };

  static int configure_accelerator(void)


> Kevin
>

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Michael Roth]

On Thu, 27 Oct 2011 12:58:32 -0500, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 10/27/2011 03:20 AM, Kevin Wolf wrote:
> > Am 26.10.2011 22:49, schrieb Anthony Liguori:
> >> On 10/25/2011 10:22 AM, Kevin Wolf wrote:
> >>> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
> >>> I think qemu-iotests could be considered an instance of B)
> >>>
> >>>> C) Functional tests that just need to run a small binary with no OS
> >>>>      installed in the guest, but running a fully-feature qemu process.
> >>>>      - The tests in the 'tests' directory do this, right? kvm-unittests
> >>>>        does this, right?
> >>>
> >>> Not sure what test/ does, but for kvm-unittests yes. And this is also
> >>> what I was talking about.
> >>
> >> Thinking more about this...
> >>
> >> We could add a new '-x-test-server CHR' option.  When this option is added, it
> >> would do the following:
> >>
> >> 1) Open CHR character device
> >> 2) Use /dev/shm for guest memory
> >> 3) Listen for connections on CHR
> >> 4) When something connects to CHR
> >>    a) reset device model
> >>    b) send /dev/shm fd over CHR
> >>    c) register CPU physical memory client
> >>       1. upon CPU physical memory changes, send the change info over CHR
> >>    d) instead of doing [kvm_]cpu_exec(), block reading on CHR
> >>
> >> So when you launch qemu with -x-test-server, it'll sit there doing nothing
> >> terribly useful.  But this lets you write a program that connects to CHR, and
> >> then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via
> >> mmap()'ing the passed fd using the client mapping table, you can essentially
> >> write kvm-unittest style tests while still having full access to libc.
> >
> > IRQs need to go through the connection as well.
> 
> Yes, forgot to mention that.
> 
> >
> > Oh, and you would finally have a C user for libqmp. The test cases
> > definitely need to be able to access the monitor. For example I would
> > really love to have test cases for the I/O error paths that stop the VM
> > (or actually it's the resume that must be tested).
> 
> Yeah, tunnelling a monitor session sounds like a really good idea.
> 
> >> And since each test program can reset QEMU after running, you could very nicely
> >> tie into something like gtest as a unit test framework.  I think it's pretty
> >> appealing from a debugability perspective too.
> >>
> >> It also means that it's possible to have 100% C test cases such that you could
> >> still build something like ppc64-softmmu and run it against the written test
> >> cases without having to really understand ppc64 assembly or have a ppc64 build
> >> environment (to generate native binaries to run under ppc64 TCG).
> >>
> >> I think this could work out fairly well as a unit test framework.
> >
> > Sounds great, where are the patches? ;-)
> 
> Heh, need to find a volunteer although I spent a few minutes this afternoon 
> trying to figure out how hard it would be.  Turns out, it's much simpler than I 
> expected if you do the same trick that Xen does.  Instead of mucking with 
> hooking cpu_exec, Xen simply starts the CPUs in the halted state such that TCG 
> simply never runs.  The following patch is all we really need.
> 
> test_init() just needs to register the appropriate file descriptor callbacks and 
> then in the data path dispatch PIO/MMIO.  It would also need to override 
> cpu_interrupt_handler to intercept interrupt operations.
> 
> diff --git a/hw/pc.c b/hw/pc.c
> index eb4c2d8..f3fd32d 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -923,12 +923,18 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
>       }
>   }
> 
> +extern int test_allowed;
> +
>   static void pc_cpu_reset(void *opaque)
>   {
>       CPUState *env = opaque;
> 
>       cpu_reset(env);
> -    env->halted = !cpu_is_bsp(env);
> +    if (test_allowed) {
> +        env->halted = 1;
> +    } else {
> +        env->halted = !cpu_is_bsp(env);
> +    }
>   }
> 
>   static CPUState *pc_new_cpu(const char *cpu_model)
> diff --git a/vl.c b/vl.c
> index 1ddb17b..adc626a 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -1988,6 +1988,19 @@ static int tcg_init(void)
>       return 0;
>   }
> 
> +static int test_init(void)
> +{
> +    printf("Hello World\n");
> +    return 0;
> +}
> +
> +static int test_available(void)
> +{
> +    return 1;
> +}
> +
> +int test_allowed = 1;
> +
>   static struct {
>       const char *opt_name;
>       const char *name;
> @@ -1998,6 +2011,7 @@ static struct {
>       { "tcg", "tcg", tcg_available, tcg_init, &tcg_allowed },
>       { "xen", "Xen", xen_available, xen_init, &xen_allowed },
>       { "kvm", "KVM", kvm_available, kvm_init, &kvm_allowed },
> +    { "test", "Test", test_available, test_init, &test_allowed },
>   };
> 
>   static int configure_accelerator(void)
> 

vl.c seems to have changed a bit since then, but I think this was the same basic approach we took with QTest:

http://repo.or.cz/w/qemu/mdroth.git/commitdiff/2e55d8a892297512f6c7883aff2c4b082bd6a3d5?hp=07d8a50cb0e096c5cdc5a81b8030e40833664dd3

http://wiki.qemu.org/Features/QTest

There was even a brief period where we were using gtest, but GLib hadn't been merged yet so we pulled it "temporarially" before going off and working on other stuff.

> 
> > Kevin
> >
> 
> 

-- 
Sincerely,
Mike Roth
IBM Linux Technology Center

Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Anthony Liguori]

On 09/01/2011 10:42 AM, Gerd Hoffmann wrote:
>    Hi,
>
> This patch series intends to make unit testing easier.  It adds a new
> "make check" target which can be used to run all unit tests which are
> currently in the tree.  It also enables the unit tests by default, so
> you don't have to re-run configure with a special switch.

Applied all.  Thanks.

Regards,

Anthony Liguori

>
> cheers,
>    Gerd
>
> Gerd Hoffmann (4):
>    Probe for libcheck by default.
>    move checks to separate variable
>    add "make check" target
>    add test-coroutine to checks
>
>   Makefile  |   10 ++++++++--
>   configure |    7 ++++---
>   2 files changed, 12 insertions(+), 5 deletions(-)
>
>
>