* [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.
@ 2019-05-10 10:46 Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 01/13] scripts: use git archive in archive-source Gerd Hoffmann
` (13 more replies)
0 siblings, 14 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
This patch series changes the way virtual machines for test builds are
managed. They are created locally on the developer machine now. The
installer is booted on the serial console and the scripts walks through
the dialogs to install and configure the guest.
That takes the download.patchew.org server out of the loop and makes it
alot easier to tweak the guest images (adding build dependencies for
example).
The install scripts take care to apply host proxy settings (from *_proxy
environment variables) to the guest, so any package downloads will be
routed through the proxy and can be cached that way. This also makes
them work behind strict firewalls.
There are also a bunch of smaller tweaks for tests/vm to fix issues I
was struggling with. See commit messages of individual patches for
details.
Gerd Hoffmann (13):
scripts: use git archive in archive-source
tests/vm: send proxy environment variables over ssh
tests/vm: use ssh with pty unconditionally
tests/vm: run test builds on snapshot
tests/vm: proper guest shutdown
tests/vm: add vm-boot-{ssh,serial}-<guest> targets
tests/vm: add DEBUG=1 to help text
tests/vm: serial console support helpers
tests/vm: openbsd autoinstall, using serial console
tests/vm: freebsd autoinstall, using serial console
tests/vm: netbsd autoinstall, using serial console
tests/vm: fedora autoinstall, using serial console
tests/vm: ubuntu.i386: apt proxy setup
tests/vm/basevm.py | 134 +++++++++++++++++++++++----
scripts/archive-source.sh | 72 +++++++--------
tests/vm/Makefile.include | 25 ++++-
tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++
tests/vm/freebsd | 175 +++++++++++++++++++++++++++++++++--
tests/vm/netbsd | 187 ++++++++++++++++++++++++++++++++++++--
tests/vm/openbsd | 154 ++++++++++++++++++++++++++++---
tests/vm/ubuntu.i386 | 4 +
8 files changed, 845 insertions(+), 93 deletions(-)
create mode 100755 tests/vm/fedora
--
2.18.1
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 01/13] scripts: use git archive in archive-source
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh Gerd Hoffmann
` (12 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Use git archive to create tarballs of qemu and submodules instead of
cloning the repository and the submodules. This is a order of magnitude
faster because it doesn't fetch the submodules from the internet each
time the script runs.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
scripts/archive-source.sh | 72 +++++++++++++++++----------------------
1 file changed, 31 insertions(+), 41 deletions(-)
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 8b8994826000..93a48d2063f9 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -19,16 +19,25 @@ if test $# -lt 1; then
fi
tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_tdir=$(mktemp -d "${tar_file%.tar}.sub.XXXXXXXX")
+sub_file="${sub_tdir}/submodule.tar"
# We want a predictable list of submodules for builds, that is
# independent of what the developer currently has initialized
# in their checkout, because the build environment is completely
# different to the host OS.
submodules="dtc slirp ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
+sub_deinit=""
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+function cleanup() {
+ local status=$?
+ rm -rf "$sub_tdir"
+ if test "$sub_deinit" != ""; then
+ git submodule deinit $sub_deinit
+ fi
+ exit $status
+}
+trap "cleanup" 0 1 2 3 15
if git diff-index --quiet HEAD -- &>/dev/null
then
@@ -36,45 +45,26 @@ then
else
HEAD=$(git stash create)
fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-for sm in $submodules; do
- if test -d "$sm/.git"
- then
- git clone --shared "$sm" "$vroot_dir/$sm"
- test $? -ne 0 && error "failed to clone submodule $sm"
- fi
-done
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
for sm in $submodules; do
- git submodule update --init $sm
- test $? -ne 0 && error "failed to init submodule $sm"
+ status="$(git submodule status "$sm")"
+ smhash="${status#[ +-]}"
+ smhash="${smhash%% *}"
+ case "$status" in
+ -*)
+ sub_deinit="$sub_deinit $sm"
+ git submodule update --init "$sm"
+ test $? -ne 0 && error "failed to update submodule $sm"
+ ;;
+ +*)
+ echo "WARNING: submodule $sm is out of sync"
+ ;;
+ esac
+ (cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file"
+ test $? -ne 0 && error "failed to archive submodule $sm ($smhash)"
+ tar --concatenate --file "$tar_file" "$sub_file"
+ test $? -ne 0 && error "failed append submodule $sm to $tar_file"
done
-
-if test -n "$submodules"; then
- {
- git ls-files || error "git ls-files failed"
- for sm in $submodules; do
- (cd $sm; git ls-files) | sed "s:^:$sm/:"
- if test "${PIPESTATUS[*]}" != "0 0"; then
- error "git ls-files in submodule $sm failed"
- fi
- done
- } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
- git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
- error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
exit 0
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 01/13] scripts: use git archive in archive-source Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-18 22:15 ` Philippe Mathieu-Daudé
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 03/13] tests/vm: use ssh with pty unconditionally Gerd Hoffmann
` (11 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Packages are fetched via proxy that way, if configured on the host.
That might be required to pass firewalls, and it allows to route
package downloads through a caching proxy server.
Needs AcceptEnv setup in sshd_config on the guest side to work.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/basevm.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 0556bdcf9e9f..6b46674f4497 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -38,6 +38,13 @@ class BaseVM(object):
GUEST_PASS = "qemupass"
ROOT_PASS = "qemupass"
+ envvars = [
+ "https_proxy",
+ "http_proxy",
+ "ftp_proxy",
+ "no_proxy",
+ ]
+
# The script to run in the guest that builds QEMU
BUILD_SCRIPT = ""
# The guest name, to be overridden by subclasses
@@ -105,6 +112,8 @@ class BaseVM(object):
"-o", "UserKnownHostsFile=" + os.devnull,
"-o", "ConnectTimeout=1",
"-p", self.ssh_port, "-i", self._ssh_key_file]
+ for var in self.envvars:
+ ssh_cmd += ['-o', "SendEnv=%s" % var ]
if interactive:
ssh_cmd += ['-t']
assert not isinstance(cmd, str)
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 03/13] tests/vm: use ssh with pty unconditionally
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 01/13] scripts: use git archive in archive-source Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 04/13] tests/vm: run test builds on snapshot Gerd Hoffmann
` (10 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Allways ask ssh to run with a pseudo terminal.
Not having a terminal causes problems now and then.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/basevm.py | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 6b46674f4497..89bfa0fe4425 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -106,16 +106,14 @@ class BaseVM(object):
os.rename(fname + ".download", fname)
return fname
- def _ssh_do(self, user, cmd, check, interactive=False):
- ssh_cmd = ["ssh", "-q",
+ def _ssh_do(self, user, cmd, check):
+ ssh_cmd = ["ssh", "-q", "-t",
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=" + os.devnull,
"-o", "ConnectTimeout=1",
"-p", self.ssh_port, "-i", self._ssh_key_file]
for var in self.envvars:
ssh_cmd += ['-o', "SendEnv=%s" % var ]
- if interactive:
- ssh_cmd += ['-t']
assert not isinstance(cmd, str)
ssh_cmd += ["%s@127.0.0.1" % user] + list(cmd)
logging.debug("ssh_cmd: %s", " ".join(ssh_cmd))
@@ -127,9 +125,6 @@ class BaseVM(object):
def ssh(self, *cmd):
return self._ssh_do(self.GUEST_USER, cmd, False)
- def ssh_interactive(self, *cmd):
- return self._ssh_do(self.GUEST_USER, cmd, False, True)
-
def ssh_root(self, *cmd):
return self._ssh_do("root", cmd, False)
@@ -283,9 +278,9 @@ def main(vmcls):
return 2
if args.interactive:
- if vm.ssh_interactive(*cmd) == 0:
+ if vm.ssh(*cmd) == 0:
return 0
- vm.ssh_interactive()
+ vm.ssh()
return 3
else:
if vm.ssh(*cmd) != 0:
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 04/13] tests/vm: run test builds on snapshot
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (2 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 03/13] tests/vm: use ssh with pty unconditionally Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 05/13] tests/vm: proper guest shutdown Gerd Hoffmann
` (9 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
The build script doesn't shutdown the guest VMs properly,
which results in filesystem corruption and guest boot
failures sooner or later.
Use the --snapshot to run builds on a snapshot,
That way killing the VM doesn't corrupt the base image.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/Makefile.include | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 992d823f6bd2..d628e0a986f6 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -53,6 +53,7 @@ vm-build-%: $(IMAGES_DIR)/%.img
$(if $(V),--verbose) \
--image "$<" \
$(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \
+ --snapshot \
--build-qemu $(SRC_PATH) -- \
$(if $(TARGET_LIST),--target-list=$(TARGET_LIST)) \
$(if $(EXTRA_CONFIGURE_OPTS),$(EXTRA_CONFIGURE_OPTS)), \
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 05/13] tests/vm: proper guest shutdown
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (3 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 04/13] tests/vm: run test builds on snapshot Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 06/13] tests/vm: add vm-boot-{ssh, serial}-<guest> targets Gerd Hoffmann
` (8 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
When not running in snapshot mode ask the guest to poweroff and wait for
this to finish instead of simply quitting qemu, so the guest can flush
pending updates to disk.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/basevm.py | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 89bfa0fe4425..cae507323596 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -51,6 +51,8 @@ class BaseVM(object):
name = "#base"
# The guest architecture, to be overridden by subclasses
arch = "#arch"
+ # command to halt the guest, can be overridden by subclasses
+ poweroff = "poweroff"
def __init__(self, debug=False, vcpus=None):
self._guest = None
self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",
@@ -201,6 +203,10 @@ class BaseVM(object):
def wait(self):
self._guest.wait()
+ def graceful_shutdown(self):
+ self.ssh_root(self.poweroff)
+ self._guest.wait()
+
def qmp(self, *args, **kwargs):
return self._guest.qmp(*args, **kwargs)
@@ -277,11 +283,13 @@ def main(vmcls):
traceback.print_exc()
return 2
- if args.interactive:
- if vm.ssh(*cmd) == 0:
- return 0
+ exitcode = 0
+ if vm.ssh(*cmd) != 0:
+ exitcode = 3
+ if exitcode != 0 and args.interactive:
vm.ssh()
- return 3
- else:
- if vm.ssh(*cmd) != 0:
- return 3
+
+ if not args.snapshot:
+ vm.graceful_shutdown()
+
+ return exitcode
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 06/13] tests/vm: add vm-boot-{ssh, serial}-<guest> targets
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (4 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 05/13] tests/vm: proper guest shutdown Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text Gerd Hoffmann
` (7 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
For testing/troubleshooting convinience.
make vm-boot-serial-<guest>
Boot guest, with the serial console on stdio.
make vm-boot-ssh-<guest>
Boot guest, login via ssh.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/Makefile.include | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index d628e0a986f6..47084d5717c6 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -20,6 +20,10 @@ vm-test:
@echo " vm-build-all - Build QEMU in all VMs"
@echo " vm-clean-all - Clean up VM images"
@echo
+ @echo "For trouble-shooting:"
+ @echo " vm-boot-serial-<guest> - Boot guest, serial console on stdio"
+ @echo " vm-boot-ssh-<guest> - Boot guest and login via ssh"
+ @echo
@echo "Special variables:"
@echo " BUILD_TARGET=foo - override the build target"
@echo " TARGET_LIST=a,b,c - Override target list in builds."
@@ -59,3 +63,18 @@ vm-build-%: $(IMAGES_DIR)/%.img
$(if $(EXTRA_CONFIGURE_OPTS),$(EXTRA_CONFIGURE_OPTS)), \
" VM-BUILD $*")
+vm-boot-serial-%: $(IMAGES_DIR)/%.img
+ qemu-system-x86_64 -enable-kvm -m 4G -smp 2 -nographic \
+ -drive if=none,id=vblk,cache=writeback,file="$<" \
+ -netdev user,id=vnet \
+ -device virtio-blk-pci,drive=vblk \
+ -device virtio-net-pci,netdev=vnet \
+ || true
+
+vm-boot-ssh-%: $(IMAGES_DIR)/%.img
+ $(call quiet-command, \
+ $(SRC_PATH)/tests/vm/$* \
+ --image "$<" \
+ --interactive \
+ false, \
+ " VM-BOOT-SSH $*")
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (5 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 06/13] tests/vm: add vm-boot-{ssh, serial}-<guest> targets Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-18 22:23 ` Philippe Mathieu-Daudé
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 08/13] tests/vm: serial console support helpers Gerd Hoffmann
` (6 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/Makefile.include | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 47084d5717c6..8714b5947958 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -25,6 +25,8 @@ vm-test:
@echo " vm-boot-ssh-<guest> - Boot guest and login via ssh"
@echo
@echo "Special variables:"
+ @echo " DEBUG=1 - be verbose, also start interactive"
+ @echo " shell on build failures"
@echo " BUILD_TARGET=foo - override the build target"
@echo " TARGET_LIST=a,b,c - Override target list in builds."
@echo ' EXTRA_CONFIGURE_OPTS="..."'
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 08/13] tests/vm: serial console support helpers
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (6 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 09/13] tests/vm: openbsd autoinstall, using serial console Gerd Hoffmann
` (5 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Add a bunch of helpers to talk to the guest using the
serial console.
Also drop the hard-coded -serial parameter for the vm
so QEMUMachine.set_console() actually works.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/basevm.py | 85 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 82 insertions(+), 3 deletions(-)
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index cae507323596..c4f67ee4c065 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -2,10 +2,11 @@
#
# VM testing base class
#
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
#
# Authors:
# Fam Zheng <famz@redhat.com>
+# Gerd Hoffmann <kraxel@redhat.com>
#
# This code is licensed under the GPL version 2 or later. See
# the COPYING file in the top-level directory.
@@ -13,7 +14,9 @@
from __future__ import print_function
import os
+import re
import sys
+import socket
import logging
import time
import datetime
@@ -79,8 +82,7 @@ class BaseVM(object):
"-cpu", "max",
"-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
"-device", "virtio-net-pci,netdev=vnet",
- "-vnc", "127.0.0.1:0,to=20",
- "-serial", "file:%s" % os.path.join(self._tmpdir, "serial.out")]
+ "-vnc", "127.0.0.1:0,to=20"]
if vcpus and vcpus > 1:
self._args += ["-smp", str(vcpus)]
if kvm_available(self.arch):
@@ -161,6 +163,8 @@ class BaseVM(object):
logging.debug("QEMU args: %s", " ".join(args))
qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch)
guest = QEMUMachine(binary=qemu_bin, args=args)
+ guest.set_machine('pc')
+ guest.set_console()
try:
guest.launch()
except:
@@ -183,6 +187,81 @@ class BaseVM(object):
raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
usernet_info)
+ def console_init(self, timeout = 120):
+ vm = self._guest
+ vm.console_socket.settimeout(timeout)
+
+ def console_log(self, text):
+ for line in re.split("[\r\n]", text):
+ # filter out terminal escape sequences
+ line = re.sub("\x1b\[[0-9;?]*[a-zA-Z]", "", line)
+ line = re.sub("\x1b\([0-9;?]*[a-zA-Z]", "", line)
+ # replace unprintable chars
+ line = re.sub("\x1b", "<esc>", line)
+ line = re.sub("[\x00-\x1f]", ".", line)
+ if line == "":
+ continue
+ # log console line
+ sys.stderr.write("con recv: %s\n" % line)
+
+ def console_wait(self, expect):
+ vm = self._guest
+ output = ""
+ while True:
+ try:
+ chars = vm.console_socket.recv(1024)
+ except socket.timeout:
+ sys.stderr.write("console: *** read timeout ***\n")
+ sys.stderr.write("console: waiting for: '%s'\n" % expect)
+ sys.stderr.write("console: line buffer:\n")
+ sys.stderr.write("\n")
+ self.console_log(output.rstrip())
+ sys.stderr.write("\n")
+ raise
+ output += chars
+ if expect in output:
+ break
+ if "\r" in output or "\n" in output:
+ lines = re.split("[\r\n]", output)
+ output = lines.pop()
+ if self.debug:
+ self.console_log("\n".join(lines))
+ if self.debug:
+ self.console_log(output)
+
+ def console_send(self, command):
+ vm = self._guest
+ if self.debug:
+ logline = re.sub("\n", "<enter>", command)
+ logline = re.sub("[\x00-\x1f]", ".", logline)
+ sys.stderr.write("con send: %s\n" % logline)
+ for char in list(command):
+ vm.console_socket.send(char)
+ time.sleep(0.01)
+
+ def console_wait_send(self, wait, command):
+ self.console_wait(wait)
+ self.console_send(command)
+
+ def console_ssh_init(self, prompt, user, pw):
+ sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.rstrip()
+ self.console_wait_send("login:", "%s\n" % user)
+ self.console_wait_send("Password:", "%s\n" % pw)
+ self.console_wait_send(prompt, "mkdir .ssh\n")
+ self.console_wait_send(prompt, sshkey_cmd)
+ self.console_wait_send(prompt, "chmod 755 .ssh\n")
+ self.console_wait_send(prompt, "chmod 644 .ssh/authorized_keys\n")
+
+ def console_sshd_config(self, prompt):
+ self.console_wait(prompt)
+ self.console_send("echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config\n")
+ for var in self.envvars:
+ self.console_wait(prompt)
+ self.console_send("echo 'AcceptEnv %s' >> /etc/ssh/sshd_config\n" % var)
+
+ def print_step(self, text):
+ sys.stderr.write("### %s ...\n" % text)
+
def wait_ssh(self, seconds=300):
starttime = datetime.datetime.now()
endtime = starttime + datetime.timedelta(seconds=seconds)
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 09/13] tests/vm: openbsd autoinstall, using serial console
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (7 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 08/13] tests/vm: serial console support helpers Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd " Gerd Hoffmann
` (4 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally. Install to disk, using the serial
console. Create qemu user, configure ssh login. Install packages
needed for qemu builds.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/openbsd | 154 +++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 141 insertions(+), 13 deletions(-)
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index 2105c01a267a..2753cfb1139a 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -2,10 +2,11 @@
#
# OpenBSD VM image
#
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
#
# Authors:
# Fam Zheng <famz@redhat.com>
+# Gerd Hoffmann <kraxel@redhat.com>
#
# This code is licensed under the GPL version 2 or later. See
# the COPYING file in the top-level directory.
@@ -13,34 +14,161 @@
import os
import sys
+import socket
import subprocess
import basevm
class OpenBSDVM(basevm.BaseVM):
name = "openbsd"
arch = "x86_64"
+
+ link = "https://cdn.openbsd.org/pub/OpenBSD/6.5/amd64/install65.iso"
+ csum = "38d1f8cadd502f1c27bf05c5abde6cc505dd28f3f34f8a941048ff9a54f9f608"
+ size = "20G"
+ pkgs = [
+ # tools
+ "git",
+ "pkgconf",
+ "bzip2", "xz",
+
+ # gnu tools
+ "bash",
+ "gmake",
+ "gsed",
+ "bison",
+
+ # libs: usb
+ "libusb1",
+
+ # libs: crypto
+ "gnutls",
+
+ # libs: images
+ "jpeg",
+ "png",
+
+ # libs: ui
+ "sdl2",
+ "gtk+3",
+ "libxkbcommon",
+ ]
+
BUILD_SCRIPT = """
set -e;
- rm -rf /var/tmp/qemu-test.*
- cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ rm -rf /home/qemu/qemu-test.*
+ cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+ mkdir src build; cd src;
tar -xf /dev/rsd1c;
- ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
- gmake --output-sync -j{jobs} {verbose};
- # XXX: "gmake check" seems to always hang or fail
- #gmake --output-sync -j{jobs} check {verbose};
+ cd ../build
+ ../src/configure --cc=cc --python=python3 {configure_opts};
+ gmake --output-sync -j{jobs} {target} {verbose};
"""
+ poweroff = "halt -p"
def build_image(self, img):
- cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
- sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')
- img_tmp_xz = img + ".tmp.xz"
+ self.print_step("Downloading install iso")
+ cimg = self._download_with_cache(self.link, sha256sum=self.csum)
img_tmp = img + ".tmp"
- sys.stderr.write("Extracting the image...\n")
- subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
- subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+ iso = img + ".install.iso"
+
+ self.print_step("Preparing iso and disk image")
+ subprocess.check_call(["cp", "-f", cimg, iso])
+ subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+ img_tmp, self.size])
+
+ self.print_step("Booting installer")
+ self.boot(img_tmp, extra_args = [
+ "-machine", "graphics=off",
+ "-cdrom", iso
+ ])
+ self.console_init()
+ self.console_wait_send("boot>", "set tty com0\n")
+ self.console_wait_send("boot>", "\n")
+
+ # pre-install configuration
+ self.console_wait_send("(I)nstall", "i\n")
+ self.console_wait_send("Terminal type", "xterm\n")
+ self.console_wait_send("System hostname", "openbsd\n")
+ self.console_wait_send("Which network interface", "vio0\n")
+ self.console_wait_send("IPv4 address", "dhcp\n")
+ self.console_wait_send("IPv6 address", "none\n")
+ self.console_wait_send("Which network interface", "done\n")
+ self.console_wait_send("DNS domain name", "localnet\n")
+ self.console_wait("Password for root account")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait("Password for root account")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait_send("Start sshd(8)", "yes\n")
+ self.console_wait_send("X Window System", "\n")
+ self.console_wait_send("xenodm", "\n")
+ self.console_wait_send("console to com0", "\n")
+ self.console_wait_send("Which speed", "\n")
+
+ self.console_wait("Setup a user")
+ self.console_send("%s\n" % self.GUEST_USER)
+ self.console_wait("Full name")
+ self.console_send("%s\n" % self.GUEST_USER)
+ self.console_wait("Password")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait("Password")
+ self.console_send("%s\n" % self.GUEST_PASS)
+
+ self.console_wait_send("Allow root ssh login", "yes\n")
+ self.console_wait_send("timezone", "UTC\n")
+ self.console_wait_send("root disk", "\n")
+ self.console_wait_send("(W)hole disk", "\n")
+ self.console_wait_send("(A)uto layout", "\n")
+ self.console_wait_send("Location of sets", "cd0\n")
+ self.console_wait_send("Pathname to the sets", "\n")
+ self.console_wait_send("Set name(s)", "\n")
+ self.console_wait_send("without verification", "yes\n")
+
+ self.print_step("Installation started now, this will take a while")
+ self.console_wait_send("Location of sets", "done\n")
+
+ self.console_wait("successfully completed")
+ self.print_step("Installation finished, rebooting")
+ self.console_wait_send("(R)eboot", "reboot\n")
+
+ # setup qemu user
+ prompt = "$"
+ self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+ self.console_wait_send(prompt, "exit\n")
+
+ # setup root user
+ prompt = "openbsd#"
+ self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+ self.console_sshd_config(prompt)
+
+ # setup virtio-blk #1 (tarfile)
+ self.console_wait(prompt)
+ self.console_send("echo 'chmod 666 /dev/rsd1c' >> /etc/rc.local\n")
+
+ # enable w+x for /home
+ self.console_wait(prompt)
+ self.console_send("sed -i -e '/home/s/rw,/rw,wxallowed,/' /etc/fstab\n")
+
+ # use http (be proxy cache friendly)
+ self.console_wait(prompt)
+ self.console_send("sed -i -e 's/https/http/' /etc/installurl\n")
+
+ self.print_step("Configuration finished, rebooting")
+ self.console_wait_send(prompt, "reboot\n")
+ self.console_wait("login:")
+ self.wait_ssh()
+
+ self.print_step("Installing packages")
+ self.ssh_root_check("pkg_add %s\n" % " ".join(self.pkgs))
+
+ # shutdown
+ self.ssh_root(self.poweroff)
+ self.wait()
+
if os.path.exists(img):
os.remove(img)
os.rename(img_tmp, img)
+ os.remove(iso)
+ self.print_step("All done")
if __name__ == "__main__":
sys.exit(basevm.main(OpenBSDVM))
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (8 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 09/13] tests/vm: openbsd autoinstall, using serial console Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-18 22:55 ` Philippe Mathieu-Daudé
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd " Gerd Hoffmann
` (3 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally. Install to disk, using the serial
console. Create qemu user, configure ssh login. Install packages
needed for qemu builds.
Note that freebsd package downloads are delivered as non-cachable
content, so I had to configure squid with "ignore-no-store
ignore-private ignore-reload" for pkgmir.geo.freebsd.org to make the
caching actually work.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/freebsd | 175 ++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 165 insertions(+), 10 deletions(-)
diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index b0066017a617..57e5c97f3b26 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -2,43 +2,198 @@
#
# FreeBSD VM image
#
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
#
# Authors:
# Fam Zheng <famz@redhat.com>
+# Gerd Hoffmann <kraxel@redhat.com>
#
# This code is licensed under the GPL version 2 or later. See
# the COPYING file in the top-level directory.
#
import os
+import re
import sys
+import time
+import socket
import subprocess
import basevm
class FreeBSDVM(basevm.BaseVM):
name = "freebsd"
arch = "x86_64"
+
+ link = "https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-disc1.iso.xz"
+ csum = "1d40015bea89d05b8bd13e2ed80c40b522a9ec1abd8e7c8b80954fb485fb99db"
+ size = "20G"
+ pkgs = [
+ # build tools
+ "git",
+ "pkgconf",
+ "bzip2",
+
+ # gnu tools
+ "bash",
+ "gmake",
+ "gsed",
+ "flex", "bison",
+
+ # libs: crypto
+ "gnutls",
+
+ # libs: images
+ "jpeg-turbo",
+ "png",
+
+ # libs: ui
+ "sdl2",
+ "gtk3",
+ "libxkbcommon",
+
+ # libs: opengl
+ "libepoxy",
+ "mesa-libs",
+ ]
+
BUILD_SCRIPT = """
set -e;
- rm -rf /var/tmp/qemu-test.*
- cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ rm -rf /home/qemu/qemu-test.*
+ cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+ mkdir src build; cd src;
tar -xf /dev/vtbd1;
- ./configure {configure_opts};
+ cd ../build
+ ../src/configure --python=python3.6 {configure_opts};
gmake --output-sync -j{jobs} {target} {verbose};
"""
+ def console_boot_serial(self):
+ self.console_wait_send("Autoboot", "3")
+ self.console_wait_send("OK", "set console=comconsole\n")
+ self.console_wait_send("OK", "boot\n")
+
def build_image(self, img):
- cimg = self._download_with_cache("http://download.patchew.org/freebsd-11.1-amd64.img.xz",
- sha256sum='adcb771549b37bc63826c501f05121a206ed3d9f55f49145908f7e1432d65891')
- img_tmp_xz = img + ".tmp.xz"
+ self.print_step("Downloading install iso")
+ cimg = self._download_with_cache(self.link, sha256sum=self.csum)
img_tmp = img + ".tmp"
- sys.stderr.write("Extracting the image...\n")
- subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
- subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+ iso = img + ".install.iso"
+ iso_xz = iso + ".xz"
+
+ self.print_step("Preparing iso and disk image")
+ subprocess.check_call(["cp", "-f", cimg, iso_xz])
+ subprocess.check_call(["xz", "-dvf", iso_xz])
+ subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+ img_tmp, self.size])
+
+ self.print_step("Booting installer")
+ self.boot(img_tmp, extra_args = [
+ "-machine", "graphics=off",
+ "-cdrom", iso
+ ])
+ self.console_init()
+ self.console_boot_serial()
+ self.console_wait_send("Console type", "xterm\n")
+
+ # pre-install configuration
+ self.console_wait_send("Welcome", "\n")
+ self.console_wait_send("Keymap Selection", "\n")
+ self.console_wait_send("Set Hostname", "freebsd\n")
+ self.console_wait_send("Distribution Select", "\n")
+ self.console_wait_send("Partitioning", "\n")
+ self.console_wait_send("Partition", "\n")
+ self.console_wait_send("Scheme", "\n")
+ self.console_wait_send("Editor", "f")
+ self.console_wait_send("Confirmation", "c")
+
+ self.print_step("Installation started now, this will take a while")
+
+ # post-install configuration
+ self.console_wait("New Password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait("Retype New Password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+
+ self.console_wait_send("Network Configuration", "\n")
+ self.console_wait_send("IPv4", "y")
+ self.console_wait_send("DHCP", "y")
+ self.console_wait_send("IPv6", "n")
+ self.console_wait_send("Resolver", "\n")
+
+ self.console_wait_send("Time Zone Selector", "a\n")
+ self.console_wait_send("Confirmation", "y")
+ self.console_wait_send("Time & Date", "\n")
+ self.console_wait_send("Time & Date", "\n")
+
+ self.console_wait_send("System Configuration", "\n")
+ self.console_wait_send("System Hardening", "\n")
+
+ # qemu user
+ self.console_wait_send("Add User Accounts", "y")
+ self.console_wait("Username")
+ self.console_send("%s\n" % self.GUEST_USER)
+ self.console_wait("Full name")
+ self.console_send("%s\n" % self.GUEST_USER)
+ self.console_wait_send("Uid", "\n")
+ self.console_wait_send("Login group", "\n")
+ self.console_wait_send("Login group", "\n")
+ self.console_wait_send("Login class", "\n")
+ self.console_wait_send("Shell", "\n")
+ self.console_wait_send("Home directory", "\n")
+ self.console_wait_send("Home directory perm", "\n")
+ self.console_wait_send("Use password", "\n")
+ self.console_wait_send("Use an empty password", "\n")
+ self.console_wait_send("Use a random password", "\n")
+ self.console_wait("Enter password:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait("Enter password again:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait_send("Lock out", "\n")
+ self.console_wait_send("OK", "yes\n")
+ self.console_wait_send("Add another user", "no\n")
+
+ self.console_wait_send("Final Configuration", "\n")
+ self.console_wait_send("Manual Configuration", "\n")
+ self.console_wait_send("Complete", "\n")
+
+ self.print_step("Installation finished, rebooting")
+ self.console_boot_serial()
+
+ # setup qemu user
+ prompt = "$"
+ self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+ self.console_wait_send(prompt, "exit\n")
+
+ # setup root user
+ prompt = "root@freebsd:~ #"
+ self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+ self.console_sshd_config(prompt)
+
+ # setup serial console
+ self.console_wait(prompt)
+ self.console_send("echo 'console=comconsole' >> /boot/loader.conf\n")
+
+ # setup virtio-blk #1 (tarfile)
+ self.console_wait(prompt)
+ self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
+
+ self.print_step("Configuration finished, rebooting")
+ self.console_wait_send(prompt, "reboot\n")
+ self.console_wait("login:")
+ self.wait_ssh()
+
+ self.print_step("Installing packages")
+ self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
+
+ # shutdown
+ self.ssh_root(self.poweroff)
+ self.console_wait("Uptime:")
+ self.wait()
+
if os.path.exists(img):
os.remove(img)
os.rename(img_tmp, img)
+ os.remove(iso)
+ self.print_step("All done")
if __name__ == "__main__":
sys.exit(basevm.main(FreeBSDVM))
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd autoinstall, using serial console
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (9 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd " Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-16 5:16 ` Kamil Rytarowski
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 12/13] tests/vm: fedora " Gerd Hoffmann
` (2 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally. Install to disk, using the serial
console. Create qemu user, configure ssh login. Install packages
needed for qemu builds.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/netbsd | 187 +++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 177 insertions(+), 10 deletions(-)
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 4c6624ea5ed5..6dbfc1b0fbe3 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -2,10 +2,11 @@
#
# NetBSD VM image
#
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
#
# Authors:
# Fam Zheng <famz@redhat.com>
+# Gerd Hoffmann <kraxel@redhat.com>
#
# This code is licensed under the GPL version 2 or later. See
# the COPYING file in the top-level directory.
@@ -13,32 +14,198 @@
import os
import sys
+import time
import subprocess
import basevm
class NetBSDVM(basevm.BaseVM):
name = "netbsd"
arch = "x86_64"
+
+ link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+ size = "20G"
+ pkgs = [
+ # tools
+ "git-base",
+ "pkgconf",
+ "xz",
+ "python37",
+
+ # gnu tools
+ "bash",
+ "gmake",
+ "gsed",
+ "flex", "bison",
+
+ # libs: crypto
+ "gnutls",
+
+ # libs: images
+ "jpeg",
+ "png",
+
+ # libs: ui
+ "SDL2",
+ "gtk3+",
+ "libxkbcommon",
+ ]
+
BUILD_SCRIPT = """
set -e;
- rm -rf /var/tmp/qemu-test.*
- cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ rm -rf /home/qemu/qemu-test.*
+ cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+ mkdir src build; cd src;
tar -xf /dev/rld1a;
- ./configure --python=python2.7 {configure_opts};
+ cd ../build
+ ../src/configure --python=python3.7 --disable-opengl {configure_opts};
gmake --output-sync -j{jobs} {target} {verbose};
"""
+ poweroff = "/sbin/poweroff"
def build_image(self, img):
- cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
- sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
- img_tmp_xz = img + ".tmp.xz"
+ cimg = self._download_with_cache(self.link)
img_tmp = img + ".tmp"
- sys.stderr.write("Extracting the image...\n")
- subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
- subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+ iso = img + ".install.iso"
+
+ self.print_step("Preparing iso and disk image")
+ subprocess.check_call(["cp", "-f", cimg, iso])
+ subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+ img_tmp, self.size])
+
+ self.print_step("Booting installer")
+ self.boot(img_tmp, extra_args = [
+ "-device", "VGA",
+ "-machine", "graphics=off",
+ "-cdrom", iso
+ ])
+ self.console_init()
+ self.console_wait("Primary Bootstrap")
+
+ # serial console boot menu output doesn't work for some
+ # reason, so we have to fly blind ...
+ for char in list("5consdev com0\n"):
+ time.sleep(0.2)
+ self.console_send(char)
+ self.console_wait("")
+ self.console_wait_send("> ", "boot\n")
+
+ self.console_wait_send("Terminal type", "xterm\n")
+ self.console_wait_send("a: Installation messages", "a\n")
+ self.console_wait_send("b: US-English", "b\n")
+ self.console_wait_send("a: Install NetBSD", "a\n")
+ self.console_wait("Shall we continue?")
+ self.console_wait_send("b: Yes", "b\n")
+
+ self.console_wait_send("a: ld0", "a\n")
+ self.console_wait_send("a: This is the correct", "a\n")
+ self.console_wait_send("b: Use the entire disk", "b\n")
+ self.console_wait("NetBSD bootcode")
+ self.console_wait_send("a: Yes", "a\n")
+ self.console_wait_send("b: Use existing part", "b\n")
+ self.console_wait_send("x: Partition sizes ok", "x\n")
+ self.console_wait_send("for your NetBSD disk", "\n")
+ self.console_wait("Shall we continue?")
+ self.console_wait_send("b: Yes", "b\n")
+
+ self.console_wait_send("b: Use serial port com0", "b\n")
+ self.console_wait_send("f: Set serial baud rate", "f\n")
+ self.console_wait_send("a: 9600", "a\n")
+ self.console_wait_send("x: Exit", "x\n")
+
+ self.console_wait_send("a: Full installation", "a\n")
+ self.console_wait_send("a: CD-ROM", "a\n")
+
+ self.print_step("Installation started now, this will take a while")
+ self.console_wait_send("Hit enter to continue", "\n")
+
+ self.console_wait_send("d: Change root password", "d\n")
+ self.console_wait_send("a: Yes", "a\n")
+ self.console_wait("New password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait("New password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait("Retype new password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+
+ self.console_wait_send("o: Add a user", "o\n")
+ self.console_wait("username")
+ self.console_send("%s\n" % self.GUEST_USER)
+ self.console_wait("to group wheel")
+ self.console_wait_send("a: Yes", "a\n")
+ self.console_wait_send("a: /bin/sh", "a\n")
+ self.console_wait("New password:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait("New password:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait("Retype new password:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+
+ self.console_wait_send("a: Configure network", "a\n")
+ self.console_wait_send("a: vioif0", "a\n")
+ self.console_wait_send("Network media type", "\n")
+ self.console_wait("autoconfiguration")
+ self.console_wait_send("a: Yes", "a\n")
+ self.console_wait_send("DNS domain", "localnet\n")
+ self.console_wait("Are they OK?")
+ self.console_wait_send("a: Yes", "a\n")
+ self.console_wait("installed in /etc")
+ self.console_wait_send("a: Yes", "a\n")
+
+ self.console_wait_send("e: Enable install", "e\n")
+ proxy = os.environ.get("http_proxy")
+ if not proxy is None:
+ self.console_wait_send("f: Proxy", "f\n")
+ self.console_wait("Proxy")
+ self.console_send("%s\n" % proxy)
+ self.console_wait_send("x: Install pkgin", "x\n")
+ self.console_init(1200)
+ self.console_wait_send("Hit enter to continue", "\n")
+ self.console_init()
+
+ self.console_wait_send("g: Enable sshd", "g\n")
+ self.console_wait_send("x: Finished conf", "x\n")
+ self.console_wait_send("Hit enter to continue", "\n")
+
+ self.print_step("Installation finished, rebooting")
+ self.console_wait_send("d: Reboot the computer", "d\n")
+
+ # setup qemu user
+ prompt = "localhost$"
+ self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+ self.console_wait_send(prompt, "exit\n")
+
+ # setup root user
+ prompt = "localhost#"
+ self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+ self.console_sshd_config(prompt)
+
+ # setup virtio-blk #1 (tarfile)
+ self.console_wait(prompt)
+ self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
+
+ # turn off mprotect (conflicts with tcg)
+ self.console_wait(prompt)
+ self.console_send("echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf\n")
+
+ self.print_step("Configuration finished, rebooting")
+ self.console_wait_send(prompt, "reboot\n")
+ self.console_wait("login:")
+ self.wait_ssh()
+
+ self.print_step("Installing packages")
+ self.ssh_root_check("pkgin update\n")
+ self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
+
+ # shutdown
+ self.ssh_root(self.poweroff)
+ self.console_wait("entering state S5")
+ self.wait()
+
if os.path.exists(img):
os.remove(img)
os.rename(img_tmp, img)
+ os.remove(iso)
+ self.print_step("All done")
if __name__ == "__main__":
sys.exit(basevm.main(NetBSDVM))
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 12/13] tests/vm: fedora autoinstall, using serial console
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (10 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd " Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup Gerd Hoffmann
2019-05-17 1:04 ` [Qemu-devel] QEMU on OpenBSD is broken? (was: Re: [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.) Thomas Huth
13 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Download the install iso and prepare the image locally. Install to
disk, using the serial console. Create qemu user, configure ssh login.
Install packages needed for qemu builds.
Yes, we have docker images for fedora. But for trouble-shooting it
might be helpful to have a vm too. When vm builds fail you can use
it to figure whenever the vm setup or the guest os is the problem.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/basevm.py | 9 +-
tests/vm/Makefile.include | 3 +-
tests/vm/fedora | 187 ++++++++++++++++++++++++++++++++++++++
3 files changed, 197 insertions(+), 2 deletions(-)
create mode 100755 tests/vm/fedora
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index c4f67ee4c065..9cec3af7cc7e 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -204,7 +204,7 @@ class BaseVM(object):
# log console line
sys.stderr.write("con recv: %s\n" % line)
- def console_wait(self, expect):
+ def console_wait(self, expect, expectalt = None):
vm = self._guest
output = ""
while True:
@@ -213,6 +213,8 @@ class BaseVM(object):
except socket.timeout:
sys.stderr.write("console: *** read timeout ***\n")
sys.stderr.write("console: waiting for: '%s'\n" % expect)
+ if not expectalt is None:
+ sys.stderr.write("console: waiting for: '%s' (alt)\n" % expectalt)
sys.stderr.write("console: line buffer:\n")
sys.stderr.write("\n")
self.console_log(output.rstrip())
@@ -221,6 +223,8 @@ class BaseVM(object):
output += chars
if expect in output:
break
+ if not expectalt is None and expectalt in output:
+ break
if "\r" in output or "\n" in output:
lines = re.split("[\r\n]", output)
output = lines.pop()
@@ -228,6 +232,9 @@ class BaseVM(object):
self.console_log("\n".join(lines))
if self.debug:
self.console_log(output)
+ if not expectalt is None and expectalt in output:
+ return False
+ return True
def console_send(self, command):
vm = self._guest
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 8714b5947958..6e24ee786910 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -2,7 +2,7 @@
.PHONY: vm-build-all vm-clean-all
-IMAGES := ubuntu.i386 freebsd netbsd openbsd centos
+IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora
IMAGES_DIR := $(HOME)/.cache/qemu-vm/images
IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES))
@@ -16,6 +16,7 @@ vm-test:
@echo " vm-build-netbsd - Build QEMU in NetBSD VM"
@echo " vm-build-openbsd - Build QEMU in OpenBSD VM"
@echo " vm-build-centos - Build QEMU in CentOS VM, with Docker"
+ @echo " vm-build-fedora - Build QEMU in Fedora VM"
@echo ""
@echo " vm-build-all - Build QEMU in all VMs"
@echo " vm-clean-all - Clean up VM images"
diff --git a/tests/vm/fedora b/tests/vm/fedora
new file mode 100755
index 000000000000..c5621d083221
--- /dev/null
+++ b/tests/vm/fedora
@@ -0,0 +1,187 @@
+#!/usr/bin/env python
+#
+# Fedora VM image
+#
+# Copyright 2019 Red Hat Inc.
+#
+# Authors:
+# Gerd Hoffmann <kraxel@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import re
+import sys
+import time
+import socket
+import subprocess
+import basevm
+
+class FedoraVM(basevm.BaseVM):
+ name = "fedora"
+ arch = "x86_64"
+
+ base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
+ link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
+ repo = base + "Server/x86_64/os/"
+ full = base + "Everything/x86_64/os/"
+ csum = "5e4eac4566d8c572bfb3bcf54b7d6c82006ec3c6c882a2c9235c6d3494d7b100"
+ size = "20G"
+ pkgs = [
+ # tools
+ 'git-core',
+ 'flex', 'bison',
+ 'gcc', 'binutils', 'make',
+
+ # perl
+ 'perl-Test-Harness',
+
+ # libs: usb
+ '"pkgconfig(libusb-1.0)"',
+ '"pkgconfig(libusbredirparser-0.5)"',
+
+ # libs: crypto
+ '"pkgconfig(gnutls)"',
+
+ # libs: ui
+ '"pkgconfig(sdl2)"',
+ '"pkgconfig(gtk+-3.0)"',
+ '"pkgconfig(ncursesw)"',
+
+ # libs: audio
+ '"pkgconfig(libpulse)"',
+ '"pkgconfig(alsa)"',
+ ]
+
+ BUILD_SCRIPT = """
+ set -e;
+ rm -rf /home/qemu/qemu-test.*
+ cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+ mkdir src build; cd src;
+ tar -xf /dev/vdb;
+ cd ../build
+ ../src/configure --python=python3 {configure_opts};
+ gmake --output-sync -j{jobs} {target} {verbose};
+ """
+
+ def build_image(self, img):
+ self.print_step("Downloading install iso")
+ cimg = self._download_with_cache(self.link, sha256sum=self.csum)
+ img_tmp = img + ".tmp"
+ iso = img + ".install.iso"
+
+ self.print_step("Preparing iso and disk image")
+ subprocess.check_call(["cp", "-f", cimg, iso])
+ subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+ img_tmp, self.size])
+
+ self.print_step("Booting installer")
+ self.boot(img_tmp, extra_args = [
+ "-machine", "graphics=off",
+ "-cdrom", iso
+ ])
+ self.console_init(300)
+ self.console_wait("installation process.")
+ time.sleep(0.3)
+ self.console_send("\t")
+ time.sleep(0.3)
+ self.console_send(" console=ttyS0")
+ proxy = os.environ.get("http_proxy")
+ if not proxy is None:
+ self.console_send(" proxy=%s" % proxy)
+ self.console_send(" inst.repo=%s" % self.repo)
+ self.console_send("\n")
+
+ self.console_wait_send("2) Use text mode", "2\n")
+
+ self.console_wait_send("5) [!] Installation Dest", "5\n")
+ self.console_wait_send("1) [x]", "c\n")
+ self.console_wait_send("2) [ ] Use All Space", "2\n")
+ self.console_wait_send("2) [x] Use All Space", "c\n")
+ self.console_wait_send("1) [ ] Standard Part", "1\n")
+ self.console_wait_send("1) [x] Standard Part", "c\n")
+
+ self.console_wait_send("7) [!] Root password", "7\n")
+ self.console_wait("Password:")
+ self.console_send("%s\n" % self.ROOT_PASS)
+ self.console_wait("Password (confirm):")
+ self.console_send("%s\n" % self.ROOT_PASS)
+
+ self.console_wait_send("8) [ ] User creation", "8\n")
+ self.console_wait_send("1) [ ] Create user", "1\n")
+ self.console_wait_send("3) User name", "3\n")
+ self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER)
+ self.console_wait_send("4) [ ] Use password", "4\n")
+ self.console_wait_send("5) Password", "5\n")
+ self.console_wait("Password:")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait("Password (confirm):")
+ self.console_send("%s\n" % self.GUEST_PASS)
+ self.console_wait_send("7) Groups", "c\n")
+
+ while True:
+ good = self.console_wait("3) [x] Installation",
+ "3) [!] Installation")
+ self.console_send("r\n")
+ if good:
+ break
+ time.sleep(10)
+
+ while True:
+ good = self.console_wait("4) [x] Software",
+ "4) [!] Software")
+ self.console_send("r\n")
+ if good:
+ break
+ time.sleep(10)
+ self.console_send("r\n" % self.GUEST_PASS)
+
+ self.console_wait_send("'b' to begin install", "b\n")
+
+ self.print_step("Installation started now, this will take a while")
+
+ self.console_wait_send("Installation complete", "\n")
+ self.print_step("Installation finished, rebooting")
+
+ # setup qemu user
+ prompt = " ~]$"
+ self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+ self.console_wait_send(prompt, "exit\n")
+
+ # setup root user
+ prompt = " ~]#"
+ self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+ self.console_sshd_config(prompt)
+
+ # setup virtio-blk #1 (tarfile)
+ self.console_wait(prompt)
+ self.console_send("echo 'KERNEL==\"vdb\" MODE=\"666\"' >> %s\n" %
+ "/etc/udev/rules.d/99-qemu.rules")
+
+ self.print_step("Configuration finished, rebooting")
+ self.console_wait_send(prompt, "reboot\n")
+ self.console_wait("login:")
+ self.wait_ssh()
+
+ self.print_step("Installing packages")
+ self.ssh_root_check("rm -vf /etc/yum.repos.d/fedora*.repo\n")
+ self.ssh_root_check("echo '[fedora]' >> /etc/yum.repos.d/qemu.repo\n")
+ self.ssh_root_check("echo 'baseurl=%s' >> /etc/yum.repos.d/qemu.repo\n" % self.full)
+ self.ssh_root_check("echo 'gpgcheck=0' >> /etc/yum.repos.d/qemu.repo\n")
+ self.ssh_root_check("dnf install -y %s\n" % " ".join(self.pkgs))
+
+ # shutdown
+ self.ssh_root(self.poweroff)
+ self.console_wait("sleep state S5")
+ self.wait()
+
+ if os.path.exists(img):
+ os.remove(img)
+ os.rename(img_tmp, img)
+ os.remove(iso)
+ self.print_step("All done")
+
+if __name__ == "__main__":
+ sys.exit(basevm.main(FedoraVM))
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (11 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 12/13] tests/vm: fedora " Gerd Hoffmann
@ 2019-05-10 10:46 ` Gerd Hoffmann
2019-05-18 22:18 ` Philippe Mathieu-Daudé
2019-05-17 1:04 ` [Qemu-devel] QEMU on OpenBSD is broken? (was: Re: [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.) Thomas Huth
13 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-10 10:46 UTC (permalink / raw)
To: qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Gerd Hoffmann, Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
Configure apt proxy so package downloads
can be cached and can pass firewalls.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
tests/vm/ubuntu.i386 | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
index a22d137e76df..b869afd212fa 100755
--- a/tests/vm/ubuntu.i386
+++ b/tests/vm/ubuntu.i386
@@ -51,6 +51,10 @@ class UbuntuX86VM(basevm.BaseVM):
" ssh-authorized-keys:\n",
" - %s\n" % basevm.SSH_PUB_KEY,
"locale: en_US.UTF-8\n"])
+ proxy = os.environ.get("http_proxy")
+ if not proxy is None:
+ udata.writelines(["apt:\n",
+ " proxy: %s" % proxy])
udata.close()
subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
"-volid", "cidata", "-joliet", "-rock",
--
2.18.1
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd autoinstall, using serial console
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd " Gerd Hoffmann
@ 2019-05-16 5:16 ` Kamil Rytarowski
0 siblings, 0 replies; 27+ messages in thread
From: Kamil Rytarowski @ 2019-05-16 5:16 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Ed Maste, Alex Bennée, Kamil Rytarowski,
Philippe Mathieu-Daudé,
Li-Wen Hsu, Brad Smith
[-- Attachment #1: Type: text/plain, Size: 9755 bytes --]
On 10.05.2019 12:46, Gerd Hoffmann wrote:
> Instead of fetching the prebuilt image from patchew download the install
> iso and prepare the image locally. Install to disk, using the serial
> console. Create qemu user, configure ssh login. Install packages
> needed for qemu builds.
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Kamil Rytarowski <n54@gmx.com>
> ---
> tests/vm/netbsd | 187 +++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 177 insertions(+), 10 deletions(-)
>
> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
> index 4c6624ea5ed5..6dbfc1b0fbe3 100755
> --- a/tests/vm/netbsd
> +++ b/tests/vm/netbsd
> @@ -2,10 +2,11 @@
> #
> # NetBSD VM image
> #
> -# Copyright 2017 Red Hat Inc.
> +# Copyright 2017-2019 Red Hat Inc.
> #
> # Authors:
> # Fam Zheng <famz@redhat.com>
> +# Gerd Hoffmann <kraxel@redhat.com>
> #
> # This code is licensed under the GPL version 2 or later. See
> # the COPYING file in the top-level directory.
> @@ -13,32 +14,198 @@
>
> import os
> import sys
> +import time
> import subprocess
> import basevm
>
> class NetBSDVM(basevm.BaseVM):
> name = "netbsd"
> arch = "x86_64"
> +
> + link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
> + size = "20G"
> + pkgs = [
> + # tools
> + "git-base",
> + "pkgconf",
> + "xz",
> + "python37",
> +
> + # gnu tools
> + "bash",
> + "gmake",
> + "gsed",
> + "flex", "bison",
> +
> + # libs: crypto
> + "gnutls",
> +
> + # libs: images
> + "jpeg",
> + "png",
> +
> + # libs: ui
> + "SDL2",
> + "gtk3+",
> + "libxkbcommon",
> + ]
> +
> BUILD_SCRIPT = """
> set -e;
> - rm -rf /var/tmp/qemu-test.*
> - cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
> + rm -rf /home/qemu/qemu-test.*
> + cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
> + mkdir src build; cd src;
> tar -xf /dev/rld1a;
> - ./configure --python=python2.7 {configure_opts};
> + cd ../build
> + ../src/configure --python=python3.7 --disable-opengl {configure_opts};
> gmake --output-sync -j{jobs} {target} {verbose};
> """
> + poweroff = "/sbin/poweroff"
>
> def build_image(self, img):
> - cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
> - sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
> - img_tmp_xz = img + ".tmp.xz"
> + cimg = self._download_with_cache(self.link)
> img_tmp = img + ".tmp"
> - sys.stderr.write("Extracting the image...\n")
> - subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
> - subprocess.check_call(["xz", "-dvf", img_tmp_xz])
> + iso = img + ".install.iso"
> +
> + self.print_step("Preparing iso and disk image")
> + subprocess.check_call(["cp", "-f", cimg, iso])
> + subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> + img_tmp, self.size])
> +
> + self.print_step("Booting installer")
> + self.boot(img_tmp, extra_args = [
> + "-device", "VGA",
> + "-machine", "graphics=off",
> + "-cdrom", iso
> + ])
> + self.console_init()
> + self.console_wait("Primary Bootstrap")
> +
> + # serial console boot menu output doesn't work for some
> + # reason, so we have to fly blind ...
> + for char in list("5consdev com0\n"):
> + time.sleep(0.2)
> + self.console_send(char)
> + self.console_wait("")
> + self.console_wait_send("> ", "boot\n")
> +
> + self.console_wait_send("Terminal type", "xterm\n")
> + self.console_wait_send("a: Installation messages", "a\n")
> + self.console_wait_send("b: US-English", "b\n")
> + self.console_wait_send("a: Install NetBSD", "a\n")
> + self.console_wait("Shall we continue?")
> + self.console_wait_send("b: Yes", "b\n")
> +
> + self.console_wait_send("a: ld0", "a\n")
> + self.console_wait_send("a: This is the correct", "a\n")
> + self.console_wait_send("b: Use the entire disk", "b\n")
> + self.console_wait("NetBSD bootcode")
> + self.console_wait_send("a: Yes", "a\n")
> + self.console_wait_send("b: Use existing part", "b\n")
> + self.console_wait_send("x: Partition sizes ok", "x\n")
> + self.console_wait_send("for your NetBSD disk", "\n")
> + self.console_wait("Shall we continue?")
> + self.console_wait_send("b: Yes", "b\n")
> +
> + self.console_wait_send("b: Use serial port com0", "b\n")
> + self.console_wait_send("f: Set serial baud rate", "f\n")
> + self.console_wait_send("a: 9600", "a\n")
> + self.console_wait_send("x: Exit", "x\n")
> +
> + self.console_wait_send("a: Full installation", "a\n")
> + self.console_wait_send("a: CD-ROM", "a\n")
> +
> + self.print_step("Installation started now, this will take a while")
> + self.console_wait_send("Hit enter to continue", "\n")
> +
> + self.console_wait_send("d: Change root password", "d\n")
> + self.console_wait_send("a: Yes", "a\n")
> + self.console_wait("New password:")
> + self.console_send("%s\n" % self.ROOT_PASS)
> + self.console_wait("New password:")
> + self.console_send("%s\n" % self.ROOT_PASS)
> + self.console_wait("Retype new password:")
> + self.console_send("%s\n" % self.ROOT_PASS)
> +
> + self.console_wait_send("o: Add a user", "o\n")
> + self.console_wait("username")
> + self.console_send("%s\n" % self.GUEST_USER)
> + self.console_wait("to group wheel")
> + self.console_wait_send("a: Yes", "a\n")
> + self.console_wait_send("a: /bin/sh", "a\n")
> + self.console_wait("New password:")
> + self.console_send("%s\n" % self.GUEST_PASS)
> + self.console_wait("New password:")
> + self.console_send("%s\n" % self.GUEST_PASS)
> + self.console_wait("Retype new password:")
> + self.console_send("%s\n" % self.GUEST_PASS)
> +
> + self.console_wait_send("a: Configure network", "a\n")
> + self.console_wait_send("a: vioif0", "a\n")
> + self.console_wait_send("Network media type", "\n")
> + self.console_wait("autoconfiguration")
> + self.console_wait_send("a: Yes", "a\n")
> + self.console_wait_send("DNS domain", "localnet\n")
> + self.console_wait("Are they OK?")
> + self.console_wait_send("a: Yes", "a\n")
> + self.console_wait("installed in /etc")
> + self.console_wait_send("a: Yes", "a\n")
> +
> + self.console_wait_send("e: Enable install", "e\n")
> + proxy = os.environ.get("http_proxy")
> + if not proxy is None:
> + self.console_wait_send("f: Proxy", "f\n")
> + self.console_wait("Proxy")
> + self.console_send("%s\n" % proxy)
> + self.console_wait_send("x: Install pkgin", "x\n")
> + self.console_init(1200)
> + self.console_wait_send("Hit enter to continue", "\n")
> + self.console_init()
> +
> + self.console_wait_send("g: Enable sshd", "g\n")
> + self.console_wait_send("x: Finished conf", "x\n")
> + self.console_wait_send("Hit enter to continue", "\n")
> +
> + self.print_step("Installation finished, rebooting")
> + self.console_wait_send("d: Reboot the computer", "d\n")
> +
> + # setup qemu user
> + prompt = "localhost$"
> + self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
> + self.console_wait_send(prompt, "exit\n")
> +
> + # setup root user
> + prompt = "localhost#"
> + self.console_ssh_init(prompt, "root", self.ROOT_PASS)
> + self.console_sshd_config(prompt)
> +
> + # setup virtio-blk #1 (tarfile)
> + self.console_wait(prompt)
> + self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
> +
> + # turn off mprotect (conflicts with tcg)
> + self.console_wait(prompt)
> + self.console_send("echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf\n")
> +
> + self.print_step("Configuration finished, rebooting")
> + self.console_wait_send(prompt, "reboot\n")
> + self.console_wait("login:")
> + self.wait_ssh()
> +
> + self.print_step("Installing packages")
> + self.ssh_root_check("pkgin update\n")
> + self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
> +
> + # shutdown
> + self.ssh_root(self.poweroff)
> + self.console_wait("entering state S5")
> + self.wait()
> +
> if os.path.exists(img):
> os.remove(img)
> os.rename(img_tmp, img)
> + os.remove(iso)
> + self.print_step("All done")
>
> if __name__ == "__main__":
> sys.exit(basevm.main(NetBSDVM))
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Qemu-devel] QEMU on OpenBSD is broken? (was: Re: [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.)
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
` (12 preceding siblings ...)
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup Gerd Hoffmann
@ 2019-05-17 1:04 ` Thomas Huth
2019-05-18 21:27 ` [Qemu-devel] QEMU on OpenBSD is broken? Jim Payne
13 siblings, 1 reply; 27+ messages in thread
From: Thomas Huth @ 2019-05-17 1:04 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel, Brad Smith
Cc: Fam Zheng, Peter Maydell, Ed Maste, Philippe Mathieu-Daudé,
Kamil Rytarowski, Alex Bennée, Li-Wen Hsu
On 10/05/2019 12.46, Gerd Hoffmann wrote:
> This patch series changes the way virtual machines for test builds are
> managed. They are created locally on the developer machine now. The
> installer is booted on the serial console and the scripts walks through
> the dialogs to install and configure the guest.
>
> That takes the download.patchew.org server out of the loop and makes it
> alot easier to tweak the guest images (adding build dependencies for
> example).
>
> The install scripts take care to apply host proxy settings (from *_proxy
> environment variables) to the guest, so any package downloads will be
> routed through the proxy and can be cached that way. This also makes
> them work behind strict firewalls.
>
> There are also a bunch of smaller tweaks for tests/vm to fix issues I
> was struggling with. See commit messages of individual patches for
> details.
>
> Gerd Hoffmann (13):
> scripts: use git archive in archive-source
> tests/vm: send proxy environment variables over ssh
> tests/vm: use ssh with pty unconditionally
> tests/vm: run test builds on snapshot
> tests/vm: proper guest shutdown
> tests/vm: add vm-boot-{ssh,serial}-<guest> targets
> tests/vm: add DEBUG=1 to help text
> tests/vm: serial console support helpers
> tests/vm: openbsd autoinstall, using serial console
> tests/vm: freebsd autoinstall, using serial console
> tests/vm: netbsd autoinstall, using serial console
> tests/vm: fedora autoinstall, using serial console
> tests/vm: ubuntu.i386: apt proxy setup
freebsd, netbsd and fedora targets work fine for me, so for the patches
1 - 8 and 10 - 12 :
Tested-by: Thomas Huth <thuth@redhat.com>
openbsd still fails for me:
TEST check-qtest-arm: tests/tmp105-test
TEST check-qtest-arm: tests/pca9552-test
TEST check-qtest-arm: tests/ds1338-test
TEST check-qtest-arm: tests/microbit-test
TEST check-qtest-arm: tests/m25p80-test
TEST check-qtest-arm: tests/test-arm-mptimer
TEST check-qtest-arm: tests/boot-serial-test
qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
Broken pipe
/home/qemu/qemu-test.Ka98K9/src/tests/libqtest.c:135: kill_qemu() tried
to terminate QEMU process but encountered exit status 1
ERROR - too few tests run (expected 2, got 0)
Abort trap (core dumped)
gmake: *** [/home/qemu/qemu-test.Ka98K9/src/tests/Makefile.include:884:
check-qtest-arm] Error 1
Brad, does current master work for you on OpenBSD? ... looking at the
history of the openbsd script, it seems like "make check" is broken on
OpenBSD since 2017 ... any chance that this could ever be fixed?
Thomas
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] QEMU on OpenBSD is broken?
2019-05-17 1:04 ` [Qemu-devel] QEMU on OpenBSD is broken? (was: Re: [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.) Thomas Huth
@ 2019-05-18 21:27 ` Jim Payne
2019-05-19 4:27 ` Brad Smith
2019-05-20 7:54 ` Gerd Hoffmann
0 siblings, 2 replies; 27+ messages in thread
From: Jim Payne @ 2019-05-18 21:27 UTC (permalink / raw)
To: Thomas Huth, Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Peter Maydell, Ed Maste, Philippe Mathieu-Daudé,
Kamil Rytarowski, Alex Bennée, Li-Wen Hsu
On 5/16/2019 9:04 PM, Thomas Huth wrote:
> On 10/05/2019 12.46, Gerd Hoffmann wrote:
>> This patch series changes the way virtual machines for test builds are
>> managed. They are created locally on the developer machine now. The
>> installer is booted on the serial console and the scripts walks through
>> the dialogs to install and configure the guest.
>>
>> That takes the download.patchew.org server out of the loop and makes it
>> alot easier to tweak the guest images (adding build dependencies for
>> example).
>>
>> The install scripts take care to apply host proxy settings (from *_proxy
>> environment variables) to the guest, so any package downloads will be
>> routed through the proxy and can be cached that way. This also makes
>> them work behind strict firewalls.
>>
>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>> was struggling with. See commit messages of individual patches for
>> details.
>>
>> Gerd Hoffmann (13):
>> scripts: use git archive in archive-source
>> tests/vm: send proxy environment variables over ssh
>> tests/vm: use ssh with pty unconditionally
>> tests/vm: run test builds on snapshot
>> tests/vm: proper guest shutdown
>> tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>> tests/vm: add DEBUG=1 to help text
>> tests/vm: serial console support helpers
>> tests/vm: openbsd autoinstall, using serial console
>> tests/vm: freebsd autoinstall, using serial console
>> tests/vm: netbsd autoinstall, using serial console
>> tests/vm: fedora autoinstall, using serial console
>> tests/vm: ubuntu.i386: apt proxy setup
> freebsd, netbsd and fedora targets work fine for me, so for the patches
> 1 - 8 and 10 - 12 :
>
> Tested-by: Thomas Huth <thuth@redhat.com>
>
> openbsd still fails for me:
>
> TEST check-qtest-arm: tests/tmp105-test
> TEST check-qtest-arm: tests/pca9552-test
> TEST check-qtest-arm: tests/ds1338-test
> TEST check-qtest-arm: tests/microbit-test
> TEST check-qtest-arm: tests/m25p80-test
> TEST check-qtest-arm: tests/test-arm-mptimer
> TEST check-qtest-arm: tests/boot-serial-test
> qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
> Broken pipe
How much memory is trying to be allocated here?
The default maximum data size is set to 768MB. If there is a requirement
to go beyond
that then the default has to be adjusted in /etc/login.conf.
datasize-max and datasize-cur
default:\
:path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin
/usr/local/bin /usr/local/sbin:\
:umask=022:\
:datasize-max=768M:\
:datasize-cur=768M:\
:maxproc-max=256:\
:maxproc-cur=128:\
:openfiles-max=1024:\
:openfiles-cur=512:\
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh Gerd Hoffmann
@ 2019-05-18 22:15 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 27+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-05-18 22:15 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Ed Maste, Kamil Rytarowski, Alex Bennée,
Li-Wen Hsu, Brad Smith
On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
> Packages are fetched via proxy that way, if configured on the host.
> That might be required to pass firewalls, and it allows to route
> package downloads through a caching proxy server.
>
> Needs AcceptEnv setup in sshd_config on the guest side to work.
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> tests/vm/basevm.py | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 0556bdcf9e9f..6b46674f4497 100755
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -38,6 +38,13 @@ class BaseVM(object):
> GUEST_PASS = "qemupass"
> ROOT_PASS = "qemupass"
>
> + envvars = [
> + "https_proxy",
> + "http_proxy",
> + "ftp_proxy",
> + "no_proxy",
> + ]
> +
> # The script to run in the guest that builds QEMU
> BUILD_SCRIPT = ""
> # The guest name, to be overridden by subclasses
> @@ -105,6 +112,8 @@ class BaseVM(object):
> "-o", "UserKnownHostsFile=" + os.devnull,
> "-o", "ConnectTimeout=1",
> "-p", self.ssh_port, "-i", self._ssh_key_file]
> + for var in self.envvars:
> + ssh_cmd += ['-o', "SendEnv=%s" % var ]
> if interactive:
> ssh_cmd += ['-t']
> assert not isinstance(cmd, str)
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup Gerd Hoffmann
@ 2019-05-18 22:18 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 27+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-05-18 22:18 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Ed Maste, Kamil Rytarowski, Alex Bennée,
Li-Wen Hsu, Brad Smith
On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
> Configure apt proxy so package downloads
> can be cached and can pass firewalls.
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
> tests/vm/ubuntu.i386 | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
> index a22d137e76df..b869afd212fa 100755
> --- a/tests/vm/ubuntu.i386
> +++ b/tests/vm/ubuntu.i386
> @@ -51,6 +51,10 @@ class UbuntuX86VM(basevm.BaseVM):
> " ssh-authorized-keys:\n",
> " - %s\n" % basevm.SSH_PUB_KEY,
> "locale: en_US.UTF-8\n"])
> + proxy = os.environ.get("http_proxy")
> + if not proxy is None:
> + udata.writelines(["apt:\n",
> + " proxy: %s" % proxy])
> udata.close()
> subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
> "-volid", "cidata", "-joliet", "-rock",
>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text Gerd Hoffmann
@ 2019-05-18 22:23 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 27+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-05-18 22:23 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Ed Maste, Kamil Rytarowski, Alex Bennée,
Li-Wen Hsu, Brad Smith
Hi Gerd,
On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
> tests/vm/Makefile.include | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index 47084d5717c6..8714b5947958 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -25,6 +25,8 @@ vm-test:
> @echo " vm-boot-ssh-<guest> - Boot guest and login via ssh"
> @echo
> @echo "Special variables:"
> + @echo " DEBUG=1 - be verbose, also start interactive"
> + @echo " shell on build failures"
Can you replace <tabs> by <spaces>? See:
Special variables:
DEBUG=1 - be verbose, also start interactive
shell on build failures
BUILD_TARGET=foo - override the build target
TARGET_LIST=a,b,c - Override target list in builds.
EXTRA_CONFIGURE_OPTS="..."
Using spaces:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> @echo " BUILD_TARGET=foo - override the build target"
> @echo " TARGET_LIST=a,b,c - Override target list in builds."
> @echo ' EXTRA_CONFIGURE_OPTS="..."'
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd " Gerd Hoffmann
@ 2019-05-18 22:55 ` Philippe Mathieu-Daudé
2019-05-20 5:26 ` Gerd Hoffmann
0 siblings, 1 reply; 27+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-05-18 22:55 UTC (permalink / raw)
To: Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Ed Maste, Kamil Rytarowski, Alex Bennée,
Li-Wen Hsu, Brad Smith
Hi Gerd,
On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
> Instead of fetching the prebuilt image from patchew download the install
> iso and prepare the image locally. Install to disk, using the serial
> console. Create qemu user, configure ssh login. Install packages
> needed for qemu builds.
I'm impressed how charmly this works :)
3 comments so far.
1/ We could record (in tests/vm/freebsd header?) roughly how many local
storage will be used (or display in 'make vm-help'?). FYI this image
takes ~3.1GiB.
2/ "Autoboot in 9 seconds, hit [Enter] to boot or any other key to stop"
3/ I am a bit annoyed it overwrote my previous
~/.cache/qemu-vm/images/freebsd.img VM. Not sure what's the best hash to
use, maybe "git log -n 1 --pretty=format:%H -- tests/vm/freebsd"?
(Similarly for other images).
> Note that freebsd package downloads are delivered as non-cachable
> content, so I had to configure squid with "ignore-no-store
> ignore-private ignore-reload" for pkgmir.geo.freebsd.org to make the
> caching actually work.
>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> tests/vm/freebsd | 175 ++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 165 insertions(+), 10 deletions(-)
>
> diff --git a/tests/vm/freebsd b/tests/vm/freebsd
> index b0066017a617..57e5c97f3b26 100755
> --- a/tests/vm/freebsd
> +++ b/tests/vm/freebsd
> @@ -2,43 +2,198 @@
> #
> # FreeBSD VM image
> #
> -# Copyright 2017 Red Hat Inc.
> +# Copyright 2017-2019 Red Hat Inc.
> #
> # Authors:
> # Fam Zheng <famz@redhat.com>
> +# Gerd Hoffmann <kraxel@redhat.com>
> #
> # This code is licensed under the GPL version 2 or later. See
> # the COPYING file in the top-level directory.
> #
>
> import os
> +import re
> import sys
> +import time
> +import socket
> import subprocess
> import basevm
>
> class FreeBSDVM(basevm.BaseVM):
> name = "freebsd"
> arch = "x86_64"
> +
> + link = "https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-disc1.iso.xz"
> + csum = "1d40015bea89d05b8bd13e2ed80c40b522a9ec1abd8e7c8b80954fb485fb99db"
> + size = "20G"
> + pkgs = [
> + # build tools
> + "git",
> + "pkgconf",
> + "bzip2",
> +
> + # gnu tools
> + "bash",
> + "gmake",
> + "gsed",
> + "flex", "bison",
> +
> + # libs: crypto
> + "gnutls",
> +
> + # libs: images
> + "jpeg-turbo",
> + "png",
> +
> + # libs: ui
> + "sdl2",
> + "gtk3",
> + "libxkbcommon",
> +
> + # libs: opengl
> + "libepoxy",
> + "mesa-libs",
> + ]
> +
> BUILD_SCRIPT = """
> set -e;
> - rm -rf /var/tmp/qemu-test.*
> - cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
> + rm -rf /home/qemu/qemu-test.*
> + cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
> + mkdir src build; cd src;
> tar -xf /dev/vtbd1;
> - ./configure {configure_opts};
> + cd ../build
> + ../src/configure --python=python3.6 {configure_opts};
> gmake --output-sync -j{jobs} {target} {verbose};
> """
>
> + def console_boot_serial(self):
> + self.console_wait_send("Autoboot", "3")
> + self.console_wait_send("OK", "set console=comconsole\n")
> + self.console_wait_send("OK", "boot\n")
> +
> def build_image(self, img):
> - cimg = self._download_with_cache("http://download.patchew.org/freebsd-11.1-amd64.img.xz",
> - sha256sum='adcb771549b37bc63826c501f05121a206ed3d9f55f49145908f7e1432d65891')
> - img_tmp_xz = img + ".tmp.xz"
> + self.print_step("Downloading install iso")
> + cimg = self._download_with_cache(self.link, sha256sum=self.csum)
> img_tmp = img + ".tmp"
> - sys.stderr.write("Extracting the image...\n")
> - subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
> - subprocess.check_call(["xz", "-dvf", img_tmp_xz])
> + iso = img + ".install.iso"
> + iso_xz = iso + ".xz"
> +
> + self.print_step("Preparing iso and disk image")
> + subprocess.check_call(["cp", "-f", cimg, iso_xz])
> + subprocess.check_call(["xz", "-dvf", iso_xz])
> + subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> + img_tmp, self.size])
> +
> + self.print_step("Booting installer")
> + self.boot(img_tmp, extra_args = [
> + "-machine", "graphics=off",
> + "-cdrom", iso
> + ])
> + self.console_init()
> + self.console_boot_serial()
> + self.console_wait_send("Console type", "xterm\n")
> +
> + # pre-install configuration
> + self.console_wait_send("Welcome", "\n")
> + self.console_wait_send("Keymap Selection", "\n")
> + self.console_wait_send("Set Hostname", "freebsd\n")
> + self.console_wait_send("Distribution Select", "\n")
> + self.console_wait_send("Partitioning", "\n")
> + self.console_wait_send("Partition", "\n")
> + self.console_wait_send("Scheme", "\n")
> + self.console_wait_send("Editor", "f")
> + self.console_wait_send("Confirmation", "c")
> +
> + self.print_step("Installation started now, this will take a while")
> +
> + # post-install configuration
> + self.console_wait("New Password:")
> + self.console_send("%s\n" % self.ROOT_PASS)
> + self.console_wait("Retype New Password:")
> + self.console_send("%s\n" % self.ROOT_PASS)
> +
> + self.console_wait_send("Network Configuration", "\n")
> + self.console_wait_send("IPv4", "y")
> + self.console_wait_send("DHCP", "y")
> + self.console_wait_send("IPv6", "n")
> + self.console_wait_send("Resolver", "\n")
> +
> + self.console_wait_send("Time Zone Selector", "a\n")
> + self.console_wait_send("Confirmation", "y")
> + self.console_wait_send("Time & Date", "\n")
> + self.console_wait_send("Time & Date", "\n")
> +
> + self.console_wait_send("System Configuration", "\n")
> + self.console_wait_send("System Hardening", "\n")
> +
> + # qemu user
> + self.console_wait_send("Add User Accounts", "y")
> + self.console_wait("Username")
> + self.console_send("%s\n" % self.GUEST_USER)
> + self.console_wait("Full name")
> + self.console_send("%s\n" % self.GUEST_USER)
> + self.console_wait_send("Uid", "\n")
> + self.console_wait_send("Login group", "\n")
> + self.console_wait_send("Login group", "\n")
> + self.console_wait_send("Login class", "\n")
> + self.console_wait_send("Shell", "\n")
> + self.console_wait_send("Home directory", "\n")
> + self.console_wait_send("Home directory perm", "\n")
> + self.console_wait_send("Use password", "\n")
> + self.console_wait_send("Use an empty password", "\n")
> + self.console_wait_send("Use a random password", "\n")
> + self.console_wait("Enter password:")
> + self.console_send("%s\n" % self.GUEST_PASS)
> + self.console_wait("Enter password again:")
> + self.console_send("%s\n" % self.GUEST_PASS)
> + self.console_wait_send("Lock out", "\n")
> + self.console_wait_send("OK", "yes\n")
> + self.console_wait_send("Add another user", "no\n")
> +
> + self.console_wait_send("Final Configuration", "\n")
> + self.console_wait_send("Manual Configuration", "\n")
> + self.console_wait_send("Complete", "\n")
> +
> + self.print_step("Installation finished, rebooting")
> + self.console_boot_serial()
> +
> + # setup qemu user
> + prompt = "$"
> + self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
> + self.console_wait_send(prompt, "exit\n")
> +
> + # setup root user
> + prompt = "root@freebsd:~ #"
> + self.console_ssh_init(prompt, "root", self.ROOT_PASS)
> + self.console_sshd_config(prompt)
> +
> + # setup serial console
> + self.console_wait(prompt)
> + self.console_send("echo 'console=comconsole' >> /boot/loader.conf\n")
> +
> + # setup virtio-blk #1 (tarfile)
> + self.console_wait(prompt)
> + self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
> +
> + self.print_step("Configuration finished, rebooting")
> + self.console_wait_send(prompt, "reboot\n")
> + self.console_wait("login:")
> + self.wait_ssh()
> +
> + self.print_step("Installing packages")
> + self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
> +
> + # shutdown
> + self.ssh_root(self.poweroff)
> + self.console_wait("Uptime:")
> + self.wait()
> +
> if os.path.exists(img):
> os.remove(img)
> os.rename(img_tmp, img)
> + os.remove(iso)
> + self.print_step("All done")
>
> if __name__ == "__main__":
> sys.exit(basevm.main(FreeBSDVM))
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] QEMU on OpenBSD is broken?
2019-05-18 21:27 ` [Qemu-devel] QEMU on OpenBSD is broken? Jim Payne
@ 2019-05-19 4:27 ` Brad Smith
2019-05-20 7:54 ` Gerd Hoffmann
1 sibling, 0 replies; 27+ messages in thread
From: Brad Smith @ 2019-05-19 4:27 UTC (permalink / raw)
To: Thomas Huth, Gerd Hoffmann, qemu-devel
Cc: Fam Zheng, Peter Maydell, Ed Maste, Philippe Mathieu-Daudé,
Kamil Rytarowski, Alex Bennée, Li-Wen Hsu
I just noticed when I had replied that my e-mail was sent from a different
name, by accident, as I was testing something with my e-mail client.
On 5/18/2019 5:27 PM, Jim Payne wrote:
> On 5/16/2019 9:04 PM, Thomas Huth wrote:
>
>> On 10/05/2019 12.46, Gerd Hoffmann wrote:
>>> This patch series changes the way virtual machines for test builds are
>>> managed. They are created locally on the developer machine now. The
>>> installer is booted on the serial console and the scripts walks through
>>> the dialogs to install and configure the guest.
>>>
>>> That takes the download.patchew.org server out of the loop and makes it
>>> alot easier to tweak the guest images (adding build dependencies for
>>> example).
>>>
>>> The install scripts take care to apply host proxy settings (from
>>> *_proxy
>>> environment variables) to the guest, so any package downloads will be
>>> routed through the proxy and can be cached that way. This also makes
>>> them work behind strict firewalls.
>>>
>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>>> was struggling with. See commit messages of individual patches for
>>> details.
>>>
>>> Gerd Hoffmann (13):
>>> scripts: use git archive in archive-source
>>> tests/vm: send proxy environment variables over ssh
>>> tests/vm: use ssh with pty unconditionally
>>> tests/vm: run test builds on snapshot
>>> tests/vm: proper guest shutdown
>>> tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>>> tests/vm: add DEBUG=1 to help text
>>> tests/vm: serial console support helpers
>>> tests/vm: openbsd autoinstall, using serial console
>>> tests/vm: freebsd autoinstall, using serial console
>>> tests/vm: netbsd autoinstall, using serial console
>>> tests/vm: fedora autoinstall, using serial console
>>> tests/vm: ubuntu.i386: apt proxy setup
>> freebsd, netbsd and fedora targets work fine for me, so for the patches
>> 1 - 8 and 10 - 12 :
>>
>> Tested-by: Thomas Huth <thuth@redhat.com>
>>
>> openbsd still fails for me:
>>
>> TEST check-qtest-arm: tests/tmp105-test
>> TEST check-qtest-arm: tests/pca9552-test
>> TEST check-qtest-arm: tests/ds1338-test
>> TEST check-qtest-arm: tests/microbit-test
>> TEST check-qtest-arm: tests/m25p80-test
>> TEST check-qtest-arm: tests/test-arm-mptimer
>> TEST check-qtest-arm: tests/boot-serial-test
>> qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate
>> memory
>> Broken pipe
>
> How much memory is trying to be allocated here?
>
> The default maximum data size is set to 768MB. If there is a
> requirement to go beyond
> that then the default has to be adjusted in /etc/login.conf.
>
> datasize-max and datasize-cur
>
> default:\
> :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin
> /usr/local/bin /usr/local/sbin:\
> :umask=022:\
> :datasize-max=768M:\
> :datasize-cur=768M:\
> :maxproc-max=256:\
> :maxproc-cur=128:\
> :openfiles-max=1024:\
> :openfiles-cur=512:\
>
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-18 22:55 ` Philippe Mathieu-Daudé
@ 2019-05-20 5:26 ` Gerd Hoffmann
2019-05-20 7:14 ` Thomas Huth
0 siblings, 1 reply; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-20 5:26 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Fam Zheng, Ed Maste, qemu-devel, Kamil Rytarowski,
Alex Bennée, Li-Wen Hsu, Brad Smith
On Sun, May 19, 2019 at 12:55:09AM +0200, Philippe Mathieu-Daudé wrote:
> Hi Gerd,
>
> On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
> > Instead of fetching the prebuilt image from patchew download the install
> > iso and prepare the image locally. Install to disk, using the serial
> > console. Create qemu user, configure ssh login. Install packages
> > needed for qemu builds.
>
> I'm impressed how charmly this works :)
>
> 3 comments so far.
>
> 1/ We could record (in tests/vm/freebsd header?) roughly how many local
> storage will be used (or display in 'make vm-help'?). FYI this image
> takes ~3.1GiB.
vm-help would be more useful I think.
> 2/ "Autoboot in 9 seconds, hit [Enter] to boot or any other key to stop"
Probably tweakable via loader.conf, I'll check.
> 3/ I am a bit annoyed it overwrote my previous
> ~/.cache/qemu-vm/images/freebsd.img VM. Not sure what's the best hash to
> use, maybe "git log -n 1 --pretty=format:%H -- tests/vm/freebsd"?
> (Similarly for other images).
Then use ~/.cache/qemu-vm/images/freebsd-${hash}.img ?
Which will fill ~/.cache/qemu-vm/images with stale images over time,
which isn't great either ...
Or do you have something else in mind?
cheers,
Gerd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-20 5:26 ` Gerd Hoffmann
@ 2019-05-20 7:14 ` Thomas Huth
2019-05-20 11:24 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 27+ messages in thread
From: Thomas Huth @ 2019-05-20 7:14 UTC (permalink / raw)
To: Gerd Hoffmann, Philippe Mathieu-Daudé
Cc: Fam Zheng, Ed Maste, qemu-devel, Kamil Rytarowski,
Alex Bennée, Li-Wen Hsu, Brad Smith
On 20/05/2019 07.26, Gerd Hoffmann wrote:
> On Sun, May 19, 2019 at 12:55:09AM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Gerd,
>>
>> On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
>>> Instead of fetching the prebuilt image from patchew download the install
>>> iso and prepare the image locally. Install to disk, using the serial
>>> console. Create qemu user, configure ssh login. Install packages
>>> needed for qemu builds.
>>
>> I'm impressed how charmly this works :)
>>
>> 3 comments so far.
>>
>> 1/ We could record (in tests/vm/freebsd header?) roughly how many local
>> storage will be used (or display in 'make vm-help'?). FYI this image
>> takes ~3.1GiB.
>
> vm-help would be more useful I think.
>
>> 2/ "Autoboot in 9 seconds, hit [Enter] to boot or any other key to stop"
>
> Probably tweakable via loader.conf, I'll check.
>
>> 3/ I am a bit annoyed it overwrote my previous
>> ~/.cache/qemu-vm/images/freebsd.img VM. Not sure what's the best hash to
>> use, maybe "git log -n 1 --pretty=format:%H -- tests/vm/freebsd"?
>> (Similarly for other images).
>
> Then use ~/.cache/qemu-vm/images/freebsd-${hash}.img ?
>
> Which will fill ~/.cache/qemu-vm/images with stale images over time,
> which isn't great either ...
>
> Or do you have something else in mind?
I'd vote to keep the current approach to overwrite "freebsd.img". It's a
"cache" directory - if you want to have something persisten, Philippe, I
think you should simply copy your image to a different location instead.
Thomas
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] QEMU on OpenBSD is broken?
2019-05-18 21:27 ` [Qemu-devel] QEMU on OpenBSD is broken? Jim Payne
2019-05-19 4:27 ` Brad Smith
@ 2019-05-20 7:54 ` Gerd Hoffmann
1 sibling, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-20 7:54 UTC (permalink / raw)
To: Jim Payne
Cc: Fam Zheng, Peter Maydell, Thomas Huth, Ed Maste,
Philippe Mathieu-Daudé,
qemu-devel, Kamil Rytarowski, Alex Bennée, Li-Wen Hsu
Hi,
> > qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
> > Broken pipe
>
> How much memory is trying to be allocated here?
>
> The default maximum data size is set to 768MB. If there is a requirement to
> go beyond
> that then the default has to be adjusted in /etc/login.conf.
Yes, adjusting the limits fixes that.
thanks,
Gerd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-20 7:14 ` Thomas Huth
@ 2019-05-20 11:24 ` Philippe Mathieu-Daudé
2019-05-20 12:18 ` Gerd Hoffmann
0 siblings, 1 reply; 27+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-05-20 11:24 UTC (permalink / raw)
To: Thomas Huth, Gerd Hoffmann
Cc: Fam Zheng, Ed Maste, qemu-devel, Kamil Rytarowski,
Alex Bennée, Li-Wen Hsu, Brad Smith
On 5/20/19 9:14 AM, Thomas Huth wrote:
> On 20/05/2019 07.26, Gerd Hoffmann wrote:
>> On Sun, May 19, 2019 at 12:55:09AM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi Gerd,
>>>
>>> On 5/10/19 12:46 PM, Gerd Hoffmann wrote:
>>>> Instead of fetching the prebuilt image from patchew download the install
>>>> iso and prepare the image locally. Install to disk, using the serial
>>>> console. Create qemu user, configure ssh login. Install packages
>>>> needed for qemu builds.
>>>
>>> I'm impressed how charmly this works :)
>>>
>>> 3 comments so far.
>>>
>>> 1/ We could record (in tests/vm/freebsd header?) roughly how many local
>>> storage will be used (or display in 'make vm-help'?). FYI this image
>>> takes ~3.1GiB.
>>
>> vm-help would be more useful I think.
>>
>>> 2/ "Autoboot in 9 seconds, hit [Enter] to boot or any other key to stop"
>>
>> Probably tweakable via loader.conf, I'll check.
>>
>>> 3/ I am a bit annoyed it overwrote my previous
>>> ~/.cache/qemu-vm/images/freebsd.img VM. Not sure what's the best hash to
>>> use, maybe "git log -n 1 --pretty=format:%H -- tests/vm/freebsd"?
>>> (Similarly for other images).
>>
>> Then use ~/.cache/qemu-vm/images/freebsd-${hash}.img ?
>>
>> Which will fill ~/.cache/qemu-vm/images with stale images over time,
>> which isn't great either ...
>>
>> Or do you have something else in mind?
Maybe we can simply use a ${VM_IMG_VERSION} and manually increase it
when there are significant changes.
We could add a "make vm-clean-cache-older-than-%" parsing the
"20days/3months/2y" formats?
> I'd vote to keep the current approach to overwrite "freebsd.img". It's a
> "cache" directory - if you want to have something persisten, Philippe, I
> think you should simply copy your image to a different location instead.
My concern came when thinking about bisecting:
$ git log --oneline \
$(git log --pretty=format:%H \
--diff-filter=A -- tests/vm/freebsd)..origin/master | wc -l
13016
But then I only used "make check-vm" while bisecting twice, so I can
stop worrying about testing old VMs.
Regards,
Phil.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd autoinstall, using serial console
2019-05-20 11:24 ` Philippe Mathieu-Daudé
@ 2019-05-20 12:18 ` Gerd Hoffmann
0 siblings, 0 replies; 27+ messages in thread
From: Gerd Hoffmann @ 2019-05-20 12:18 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Fam Zheng, Thomas Huth, Ed Maste, qemu-devel, Kamil Rytarowski,
Alex Bennée, Li-Wen Hsu, Brad Smith
Hi,
> > I'd vote to keep the current approach to overwrite "freebsd.img". It's a
> > "cache" directory - if you want to have something persisten, Philippe, I
> > think you should simply copy your image to a different location instead.
>
> My concern came when thinking about bisecting:
/me tricked make with "touch ~/.cache/qemu-vm/images/${name}.img" to
avoid image rebuilds while bisecting.
A bit hacky, but I think better than building fragile infrastructure
for rare usecases ...
cheers,
Gerd
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2019-05-20 12:19 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-10 10:46 [Qemu-devel] [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 01/13] scripts: use git archive in archive-source Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 02/13] tests/vm: send proxy environment variables over ssh Gerd Hoffmann
2019-05-18 22:15 ` Philippe Mathieu-Daudé
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 03/13] tests/vm: use ssh with pty unconditionally Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 04/13] tests/vm: run test builds on snapshot Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 05/13] tests/vm: proper guest shutdown Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 06/13] tests/vm: add vm-boot-{ssh, serial}-<guest> targets Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 07/13] tests/vm: add DEBUG=1 to help text Gerd Hoffmann
2019-05-18 22:23 ` Philippe Mathieu-Daudé
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 08/13] tests/vm: serial console support helpers Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 09/13] tests/vm: openbsd autoinstall, using serial console Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 10/13] tests/vm: freebsd " Gerd Hoffmann
2019-05-18 22:55 ` Philippe Mathieu-Daudé
2019-05-20 5:26 ` Gerd Hoffmann
2019-05-20 7:14 ` Thomas Huth
2019-05-20 11:24 ` Philippe Mathieu-Daudé
2019-05-20 12:18 ` Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd " Gerd Hoffmann
2019-05-16 5:16 ` Kamil Rytarowski
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 12/13] tests/vm: fedora " Gerd Hoffmann
2019-05-10 10:46 ` [Qemu-devel] [PATCH v2 13/13] tests/vm: ubuntu.i386: apt proxy setup Gerd Hoffmann
2019-05-18 22:18 ` Philippe Mathieu-Daudé
2019-05-17 1:04 ` [Qemu-devel] QEMU on OpenBSD is broken? (was: Re: [PATCH v2 00/13] tests/vm: serial console autoinstall, misc fixes.) Thomas Huth
2019-05-18 21:27 ` [Qemu-devel] QEMU on OpenBSD is broken? Jim Payne
2019-05-19 4:27 ` Brad Smith
2019-05-20 7:54 ` Gerd Hoffmann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.