[Qemu-devel] [PULL 0/3] Block and testing patches

[Qemu-devel] [PULL 0/3] Block and testing patches

[Fam Zheng]

The following changes since commit 9b2e891ec5ccdb4a7d583b77988848282606fdea:

  Merge remote-tracking branch 'remotes/marcel/tags/rdma-pull-request' into staging (2018-12-22 11:25:31 +0000)

are available in the git repository at:

  https://famz@github.com/famz/qemu tags/block-and-testing-pull-request

for you to fetch changes up to 3baa6942169cce0f58b88484ad010742c382480a:

  tests: vm: auto_install OpenBSD (2019-01-02 20:51:15 +0800)

----------------------------------------------------------------
Pull request

----------------------------------------------------------------

Fam Zheng (1):
  tests: vm: auto_install OpenBSD

Li Feng (1):
  block/nvme: optimize the performance of nvme driver based on vfio-pci

Philippe Mathieu-Daudé (1):
  docker: Use a stable snapshot for Debian Sid

 block/nvme.c                               | 16 +++---
 tests/docker/dockerfiles/debian-sid.docker |  4 ++
 tests/vm/basevm.py                         |  6 +--
 tests/vm/openbsd                           | 85 +++++++++++++++++++++++++-----
 4 files changed, 86 insertions(+), 25 deletions(-)

-- 
2.11.0

[Qemu-devel] [PULL 1/3] block/nvme: optimize the performance of nvme driver based on vfio-pci

[Fam Zheng]

From: Li Feng <lifeng1519@gmail.com>

When the IO size is larger than 2 pages, we move the the pointer one by
one in the pagelist, this is inefficient.

This is a simple benchmark result:

Before:
$ qemu-io -c 'write 0 1G' nvme://0000:00:04.0/1

wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.41 (424.504 MiB/sec and 0.4146 ops/sec)

 $ qemu-io -c 'read 0 1G' nvme://0000:00:04.0/1

read 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.03 (503.055 MiB/sec and 0.4913 ops/sec)

After:
$ qemu-io -c 'write 0 1G' nvme://0000:00:04.0/1

wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.17 (471.517 MiB/sec and 0.4605 ops/sec)

 $ qemu-io -c 'read 0 1G' nvme://0000:00:04.0/1

read 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:01.94 (526.770 MiB/sec and 0.5144 ops/sec)

Signed-off-by: Li Feng <lifeng1519@gmail.com>
Message-Id: <20181101103807.25862-1-lifeng1519@gmail.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/nvme.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/block/nvme.c b/block/nvme.c
index 29294038fc..982097b5b1 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -837,7 +837,7 @@ try_map:
         }
 
         for (j = 0; j < qiov->iov[i].iov_len / s->page_size; j++) {
-            pagelist[entries++] = iova + j * s->page_size;
+            pagelist[entries++] = cpu_to_le64(iova + j * s->page_size);
         }
         trace_nvme_cmd_map_qiov_iov(s, i, qiov->iov[i].iov_base,
                                     qiov->iov[i].iov_len / s->page_size);
@@ -850,20 +850,16 @@ try_map:
     case 0:
         abort();
     case 1:
-        cmd->prp1 = cpu_to_le64(pagelist[0]);
+        cmd->prp1 = pagelist[0];
         cmd->prp2 = 0;
         break;
     case 2:
-        cmd->prp1 = cpu_to_le64(pagelist[0]);
-        cmd->prp2 = cpu_to_le64(pagelist[1]);;
+        cmd->prp1 = pagelist[0];
+        cmd->prp2 = pagelist[1];
         break;
     default:
-        cmd->prp1 = cpu_to_le64(pagelist[0]);
-        cmd->prp2 = cpu_to_le64(req->prp_list_iova);
-        for (i = 0; i < entries - 1; ++i) {
-            pagelist[i] = cpu_to_le64(pagelist[i + 1]);
-        }
-        pagelist[entries - 1] = 0;
+        cmd->prp1 = pagelist[0];
+        cmd->prp2 = cpu_to_le64(req->prp_list_iova + sizeof(uint64_t));
         break;
     }
     trace_nvme_cmd_map_qiov(s, cmd, req, qiov, entries);
-- 
2.11.0

[Qemu-devel] [PULL 2/3] docker: Use a stable snapshot for Debian Sid

[Fam Zheng]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

The Debian Sid repository is not garanteed to be stable, as his
'unstable' name suggest :)

To allow quick testing, Debian maintainers might push packages
various time a day. Sometime package dependencies might break,
which is annoying when using this repository for stable development
(which is not recommended, but Sid provides edge packages we use
for testing).

Debian provides repositories snapshots which are suitable for our
use. Pick a recent date that works. When required, update to newer
releases will be easy.

This fixes current issues with this image:

  $ make docker-image-debian-sid
  [...]
  The following packages have unmet dependencies:
   build-essential : Depends: dpkg-dev (>= 1.17.11) but it is not going to be installed
   git : Depends: perl but it is not going to be installed
         Depends: liberror-perl but it is not going to be installed
   pkg-config : Depends: libdpkg-perl but it is not going to be installed
   texinfo : Depends: perl (>= 5.26.2-6) but it is not going to be installed
             Depends: libtext-unidecode-perl but it is not going to be installed
             Depends: libxml-libxml-perl but it is not going to be installed
  E: Unable to correct problems, you have held broken packages.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20181101183705.5422-1-philmd@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/dockerfiles/debian-sid.docker | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker
index 9a3d168705..4e4cda0ba5 100644
--- a/tests/docker/dockerfiles/debian-sid.docker
+++ b/tests/docker/dockerfiles/debian-sid.docker
@@ -13,6 +13,10 @@
 
 FROM debian:sid-slim
 
+# Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
+ENV DEBIAN_SNAPSHOT_DATE "20181030"
+RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
+
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
 
-- 
2.11.0

[Qemu-devel] [PULL 3/3] tests: vm: auto_install OpenBSD

[Fam Zheng]

From: Fam Zheng <famz@redhat.com>

Upgrade OpenBSD to 6.4 using auto_install. Especially, drop SDL1,
include SDL2.

Also do the build in $HOME since both /var/tmp and /tmp are tmpfs with
limited capacities.

Signed-off-by: Fam Zheng <famz@redhat.com>

Message-Id: <20181031025712.18602-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/vm/basevm.py |  6 ++--
 tests/vm/openbsd   | 85 ++++++++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 76 insertions(+), 15 deletions(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 5caf77d6b8..6fb446d4c5 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -68,8 +68,6 @@ class BaseVM(object):
         self._args = [ \
             "-nodefaults", "-m", "4G",
             "-cpu", "max",
-            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
-            "-device", "virtio-net-pci,netdev=vnet",
             "-vnc", "127.0.0.1:0,to=20",
             "-serial", "file:%s" % os.path.join(self._tmpdir, "serial.out")]
         if vcpus and vcpus > 1:
@@ -146,8 +144,10 @@ class BaseVM(object):
                             "-device",
                             "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)]
 
-    def boot(self, img, extra_args=[]):
+    def boot(self, img, extra_args=[], extra_usernet_args=""):
         args = self._args + [
+            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + extra_usernet_args,
+            "-device", "virtio-net-pci,netdev=vnet",
             "-device", "VGA",
             "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img,
             "-device", "virtio-blk,drive=drive0,bootindex=0"]
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index cfe0572c59..99a7e98d80 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -14,6 +14,9 @@
 import os
 import sys
 import subprocess
+import time
+import atexit
+import tempfile
 import basevm
 
 class OpenBSDVM(basevm.BaseVM):
@@ -21,25 +24,83 @@ class OpenBSDVM(basevm.BaseVM):
     arch = "x86_64"
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf $HOME/qemu-test.*
+        cd $(mktemp -d $HOME/qemu-test.XXXXXX);
         tar -xf /dev/rsd1c;
-        ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
+        ./configure {configure_opts};
         gmake --output-sync -j{jobs} {verbose};
         # XXX: "gmake check" seems to always hang or fail
         #gmake --output-sync -j{jobs} check {verbose};
     """
 
+    def _install_os(self, img):
+        tmpdir = tempfile.mkdtemp()
+        pxeboot = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/pxeboot",
+                sha256sum="d87ab39d941ff926d693943a927585945456ccedb76ea504a251b4b93cd4c266")
+        bsd_rd = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/bsd.rd",
+                sha256sum="89505c683cbcd75582fe475e847ed53d89e2b8180c3e3d61f4eb4b76b5e11f5c")
+        install = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/install64.iso",
+                sha256sum='81833b79e23dc0f961ac5fb34484bca66386deb3181ddb8236870fa4f488cdd2')
+        subprocess.check_call(["qemu-img", "create", img, "32G"])
+        subprocess.check_call(["cp", pxeboot, os.path.join(tmpdir, "auto_install")])
+        subprocess.check_call(["cp", bsd_rd, os.path.join(tmpdir, "bsd")])
+
+        self._gen_install_conf(tmpdir)
+        # BOOTP filename being auto_install makes sure OpenBSD installer
+        # not prompt for "auto install mode"
+        usernet_args = ",tftp=%s,bootfile=/auto_install" % tmpdir
+        usernet_args += ",tftp-server-name=10.0.2.4"
+        usernet_args += ",guestfwd=tcp:10.0.2.4:80-cmd:cat %s" % \
+            os.path.join(tmpdir, "install.conf")
+        self.boot(img,
+                  extra_args=["-boot", "once=n", "-no-reboot",
+                              "-cdrom", install],
+                  extra_usernet_args=usernet_args)
+        self.wait()
+
+    def _gen_install_conf(self, tmpdir):
+        contents = """\
+HTTP/1.0 200 OK
+
+System hostname = qemu-openbsd
+Password for root = qemupass
+Public ssh key for root = {pub_key}
+Allow root ssh login = yes
+Network interfaces = vio0
+IPv4 address for vio0 = dhcp
+Setup a user = qemu
+Password for user = qemupass
+Public ssh key for user = {pub_key}
+What timezone are you in = US/Eastern
+Server = fastly.cdn.openbsd.org
+Use http = yes
+Default IPv4 route = 10.0.2.2
+Location of sets = cd0
+Set name(s) = all
+Continue without verification = yes
+""".format(pub_key=basevm.SSH_PUB_KEY)
+        with open(os.path.join(tmpdir, "install.conf"), "w") as f:
+            f.write(contents)
+
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
-                sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')
-        img_tmp_xz = img + ".tmp.xz"
-        img_tmp = img + ".tmp"
-        subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "-df", img_tmp_xz])
-        if os.path.exists(img):
-            os.remove(img)
-        os.rename(img_tmp, img)
+
+        self._install_os(img + ".tmp")
+
+        self.boot(img + ".tmp")
+        self.wait_ssh()
+
+        self.ssh_root("usermod -G operator qemu")
+        self.ssh_root("echo https://fastly.cdn.openbsd.org/pub/OpenBSD > /etc/installurl")
+        for pkg in ["git", "gmake", "glib2", "bison", "sdl2"]:
+            self.ssh_root("pkg_add " + pkg)
+        self.ssh_root("ln -sf /usr/local/bin/python2.7 /usr/local/bin/python")
+        self.ssh_root("ln -sf /usr/local/bin/python2.7-2to3 /usr/local/bin/2to3")
+        self.ssh_root("ln -sf /usr/local/bin/python2.7-config /usr/local/bin/python-config")
+        self.ssh_root("ln -sf /usr/local/bin/pydoc2.7 /usr/local/bin/pydoc")
+        self.ssh_root("shutdown -p now")
+        self.wait()
+
+        subprocess.check_call(["mv", img + ".tmp", img])
 
 if __name__ == "__main__":
     sys.exit(basevm.main(OpenBSDVM))
-- 
2.11.0

Re: [Qemu-devel] [PULL 3/3] tests: vm: auto_install OpenBSD

[Carlo Arenas]

On Sat, Jan 5, 2019 at 9:39 PM Fam Zheng <fam@euphon.net> wrote:
> +        self.ssh_root("shutdown -p now")

could use `halt -p` as a more direct/obvious/effective command

Re: [Qemu-devel] [PULL 0/3] Block and testing patches

[Peter Maydell]

On Sun, 6 Jan 2019 at 05:35, Fam Zheng <fam@euphon.net> wrote:
>
> The following changes since commit 9b2e891ec5ccdb4a7d583b77988848282606fdea:
>
>   Merge remote-tracking branch 'remotes/marcel/tags/rdma-pull-request' into staging (2018-12-22 11:25:31 +0000)
>
> are available in the git repository at:
>
>   https://famz@github.com/famz/qemu tags/block-and-testing-pull-request

I think something in your workflow has put in the wrong URL
for your repo here: this looks like the one you use to log
in with (ie to push with), not the general one other people
pull from the repo with. I manually used the github URL I
have on record for you (git://github.com/famz/qemu).


> for you to fetch changes up to 3baa6942169cce0f58b88484ad010742c382480a:
>
>   tests: vm: auto_install OpenBSD (2019-01-02 20:51:15 +0800)

This makes my OpenBSD build test fail:

DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -vnc
127.0.0.1:0,to=20 -serial file:/home/peter.maydell
/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out -smp 18 -enable-kvm
-netdev user,id=vnet,hostfwd=:127.
0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:8
0-cmd:cat /tmp/tmpXrGR4K/install.conf -device
virtio-net-pci,netdev=vnet -device VGA -drive file=tests/v
m/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device
virtio-blk,drive=drive0,bootindex=0 -boot on
ce=n -no-reboot -cdrom
/home/peter.maydell/.cache/qemu-vm/download/535472578c58eb79f8b3c168a22fa5bcd2c1e
6d4
DEBUG:QMP:>>> {'execute': 'quit'}
DEBUG:qemu:Error launching VM
DEBUG:qemu:Command: 'qemu-system-x86_64 -chardev
socket,id=mon,path=/var/tmp/tmp2kj6Xg/qemu-35162-monito
r.sock -mon chardev=mon,mode=control -display none -vga none
-nodefaults -m 4G -cpu max -vnc 127.0.0.1:0
,to=20 -serial file:/home/peter.maydell/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out
-smp 18 -enable
-kvm -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-serve
r-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf -device virtio-net-pci,netd
ev=vnet -device VGA -drive
file=tests/vm/openbsd.img.tmp,if=none,id=drive0,cache=writeback
-device virti
o-blk,drive=drive0,bootindex=0 -boot once=n -no-reboot -cdrom
/home/peter.maydell/.cache/qemu-vm/downloa
d/535472578c58eb79f8b3c168a22fa5bcd2c1e6d4'
DEBUG:qemu:Output: "qemu-system-x86_64: -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf: Invalid parameter 'tftp-server-name'\n"
ERROR:root:Failed to launch QEMU, command line:
ERROR:root:qemu-system-x86_64 -nodefaults -m 4G -cpu max -vnc
127.0.0.1:0,to=20 -serial
file:/home/peter.maydell/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out
-smp 18 -enable-kvm -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf -device virtio-net-pci,netdev=vnet -device
VGA -drive file=tests/vm/openbsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -boot once=n -no-reboot
-cdrom /home/peter.maydell/.cache/qemu-vm/download/535472578c58eb79f8b3c168a22fa5bcd2c1e6d4
ERROR:root:Log:
ERROR:root:qemu-system-x86_64: -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf: Invalid parameter 'tftp-server-name'

ERROR:root:QEMU version >= 2.10 is required
Failed to prepare guest environment

The host is an Ubuntu 18.04.1, with QEMU version
2.11.1(Debian 1:2.11+dfsg-1ubuntu7.8), so that should
be good enough for a ">= 2.10" requirement.
I notice that our qapi/net.json says tftp-server-name is
"Since 3.1", though, so maybe the error message is wrong.

I don't really want to have to build and maintain a non-distro
QEMU binary for running these VM tests if we can avoid it.

thanks
-- PMM