[Qemu-devel] [PATCH 0/5] tests/acceptance: Add bFLT loader linux-user test

[Qemu-devel] [PATCH 0/5] tests/acceptance: Add bFLT loader linux-user test

[Philippe Mathieu-Daudé]

Hi,

This series refactor the current avocado_qemu Test class to allow
MachineTest and LinuxUserTest classes.
We can then add linux-user tests.
A simple/quick one is added as example, it runs a STM32 busybox
binary (in the bFLT format) on Travis-CI.
The same test can be used to confirm LP bug 1833668 [*] is fixed.

Regards,

Phil.

[*] https://bugs.launchpad.net/qemu/+bug/1833668

Philippe Mathieu-Daudé (5):
  tests/acceptance: Rename avocado_qemu.Test as MachineTest
  tests/acceptance: Make pick_default_qemu_bin() more generic
  tests/acceptance: Introduce LinuxUserTest base class
  tests/acceptance: Add bFLT loader linux-user test
  .travis.yml: Let the avocado job run linux-user tests

 .travis.yml                               |  2 +-
 docs/devel/testing.rst                    |  8 ++--
 tests/acceptance/avocado_qemu/__init__.py | 26 +++++++++---
 tests/acceptance/boot_linux_console.py    |  4 +-
 tests/acceptance/cpu_queries.py           |  4 +-
 tests/acceptance/empty_cpu_model.py       |  4 +-
 tests/acceptance/linux_initrd.py          |  4 +-
 tests/acceptance/linux_ssh_mips_malta.py  |  4 +-
 tests/acceptance/load_bflt.py             | 52 +++++++++++++++++++++++
 tests/acceptance/migration.py             |  4 +-
 tests/acceptance/version.py               |  4 +-
 tests/acceptance/virtio_version.py        |  4 +-
 tests/acceptance/vnc.py                   |  4 +-
 13 files changed, 95 insertions(+), 29 deletions(-)
 create mode 100644 tests/acceptance/load_bflt.py

-- 
2.20.1

[Qemu-devel] [PATCH 1/5] tests/acceptance: Rename avocado_qemu.Test as MachineTest

[Philippe Mathieu-Daudé]

This class is used to test QEMU machines, rename it as MachineTest.
This will allow us to add a UserTest class for qemu-user tests.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 docs/devel/testing.rst                    | 8 ++++----
 tests/acceptance/avocado_qemu/__init__.py | 7 ++++++-
 tests/acceptance/boot_linux_console.py    | 4 ++--
 tests/acceptance/cpu_queries.py           | 4 ++--
 tests/acceptance/empty_cpu_model.py       | 4 ++--
 tests/acceptance/linux_initrd.py          | 4 ++--
 tests/acceptance/linux_ssh_mips_malta.py  | 4 ++--
 tests/acceptance/migration.py             | 4 ++--
 tests/acceptance/version.py               | 4 ++--
 tests/acceptance/virtio_version.py        | 4 ++--
 tests/acceptance/vnc.py                   | 4 ++--
 11 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index da2d0fc964..5b911e153d 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -599,7 +599,7 @@ class.  Here's a simple usage example:
   from avocado_qemu import Test
 
 
-  class Version(Test):
+  class Version(MachineTest):
       """
       :avocado: tags=quick
       """
@@ -623,7 +623,7 @@ in the current directory, tagged as "quick", run:
 
   avocado run -t quick .
 
-The ``avocado_qemu.Test`` base test class
+The ``avocado_qemu.MachineTest`` base test class
 -----------------------------------------
 
 The ``avocado_qemu.Test`` class has a number of characteristics that
@@ -644,10 +644,10 @@ and hypothetical example follows:
 
 .. code::
 
-  from avocado_qemu import Test
+  from avocado_qemu import MachineTest
 
 
-  class MultipleMachines(Test):
+  class MultipleMachines(MachineTest):
       """
       :avocado: enable
       """
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 2b236a1cf0..84fe090458 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -52,7 +52,6 @@ def pick_default_qemu_bin(arch=None):
 
 class Test(avocado.Test):
     def setUp(self):
-        self._vms = {}
         arches = self.tags.get('arch', [])
         if len(arches) == 1:
             arch = arches.pop()
@@ -65,6 +64,12 @@ class Test(avocado.Test):
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
 
+
+class MachineTest(Test):
+    def setUp(self):
+        self._vms = {}
+        super().setUp()
+
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin)
         if args:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 32159503e9..1a7b378413 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -14,12 +14,12 @@ import lzma
 import gzip
 import shutil
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 from avocado.utils import process
 from avocado.utils import archive
 
 
-class BootLinuxConsole(Test):
+class BootLinuxConsole(MachineTest):
     """
     Boots a Linux kernel and checks that the console is operational and the
     kernel command line is properly passed from QEMU to the kernel
diff --git a/tests/acceptance/cpu_queries.py b/tests/acceptance/cpu_queries.py
index e71edec39f..30f545fe48 100644
--- a/tests/acceptance/cpu_queries.py
+++ b/tests/acceptance/cpu_queries.py
@@ -10,9 +10,9 @@
 
 import logging
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
-class QueryCPUModelExpansion(Test):
+class QueryCPUModelExpansion(MachineTest):
     """
     Run query-cpu-model-expansion for each CPU model, and validate results
     """
diff --git a/tests/acceptance/empty_cpu_model.py b/tests/acceptance/empty_cpu_model.py
index 3f4f663582..a4e9cc62f8 100644
--- a/tests/acceptance/empty_cpu_model.py
+++ b/tests/acceptance/empty_cpu_model.py
@@ -8,9 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 import subprocess
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
-class EmptyCPUModel(Test):
+class EmptyCPUModel(MachineTest):
     def test(self):
         cmd = [self.qemu_bin, '-S', '-display', 'none', '-machine', 'none', '-cpu', '']
         r = subprocess.run(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index 23be5a63aa..b9b2d3dab1 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -12,10 +12,10 @@ import logging
 import tempfile
 from avocado.utils.process import run
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
 
-class LinuxInitrd(Test):
+class LinuxInitrd(MachineTest):
     """
     Checks QEMU evaluates correctly the initrd file passed as -initrd option.
 
diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py
index aafb0c39f6..8451d05f7c 100644
--- a/tests/acceptance/linux_ssh_mips_malta.py
+++ b/tests/acceptance/linux_ssh_mips_malta.py
@@ -13,12 +13,12 @@ import paramiko
 import time
 
 from avocado import skipIf
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 from avocado.utils import process
 from avocado.utils import archive
 
 
-class LinuxSSH(Test):
+class LinuxSSH(MachineTest):
 
     timeout = 150 # Not for 'configure --enable-debug --enable-debug-tcg'
 
diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
index 6115cf6c24..7fdb717628 100644
--- a/tests/acceptance/migration.py
+++ b/tests/acceptance/migration.py
@@ -10,13 +10,13 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
 from avocado.utils import network
 from avocado.utils import wait
 
 
-class Migration(Test):
+class Migration(MachineTest):
     """
     :avocado: enable
     """
diff --git a/tests/acceptance/version.py b/tests/acceptance/version.py
index 67c2192c93..e11661f780 100644
--- a/tests/acceptance/version.py
+++ b/tests/acceptance/version.py
@@ -9,10 +9,10 @@
 # later.  See the COPYING file in the top-level directory.
 
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
 
-class Version(Test):
+class Version(MachineTest):
     """
     :avocado: tags=quick
     """
diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
index 8b97453ff8..04c68e41e6 100644
--- a/tests/acceptance/virtio_version.py
+++ b/tests/acceptance/virtio_version.py
@@ -13,7 +13,7 @@ import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu import QEMUMachine
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
 # Virtio Device IDs:
 VIRTIO_NET = 1
@@ -55,7 +55,7 @@ def get_pci_interfaces(vm, devtype):
     interfaces = ('pci-express-device', 'conventional-pci-device')
     return [i for i in interfaces if devtype_implements(vm, devtype, i)]
 
-class VirtioVersionCheck(Test):
+class VirtioVersionCheck(MachineTest):
     """
     Check if virtio-version-specific device types result in the
     same device tree created by `disable-modern` and
diff --git a/tests/acceptance/vnc.py b/tests/acceptance/vnc.py
index 064ceabcc1..2c532dab5e 100644
--- a/tests/acceptance/vnc.py
+++ b/tests/acceptance/vnc.py
@@ -8,10 +8,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from avocado_qemu import Test
+from avocado_qemu import MachineTest
 
 
-class Vnc(Test):
+class Vnc(MachineTest):
     """
     :avocado: tags=vnc,quick
     """
-- 
2.20.1

[Qemu-devel] [PATCH 2/5] tests/acceptance: Make pick_default_qemu_bin() more generic

[Philippe Mathieu-Daudé]

Make pick_default_qemu_bin() generic to find qemu-system or
qemu-user binaries.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 84fe090458..6785c7399e 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -23,7 +23,7 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin(arch=None):
+def pick_default_qemu_bin(path_fmt, bin_fmt, arch=None):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
@@ -39,8 +39,7 @@ def pick_default_qemu_bin(arch=None):
     """
     if arch is None:
         arch = os.uname()[4]
-    qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
-                                          "qemu-system-%s" % arch)
+    qemu_bin_relative_path = os.path.join(path_fmt % arch, bin_fmt % arch)
     if is_readable_executable_file(qemu_bin_relative_path):
         return qemu_bin_relative_path
 
@@ -51,14 +50,15 @@ def pick_default_qemu_bin(arch=None):
 
 
 class Test(avocado.Test):
-    def setUp(self):
+    def base_setUp(self, path_fmt, bin_fmt):
         arches = self.tags.get('arch', [])
         if len(arches) == 1:
             arch = arches.pop()
         else:
             arch = None
         self.arch = self.params.get('arch', default=arch)
-        default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
+        default_qemu_bin = pick_default_qemu_bin(path_fmt, bin_fmt,
+                                                 arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
         if self.qemu_bin is None:
@@ -68,7 +68,7 @@ class Test(avocado.Test):
 class MachineTest(Test):
     def setUp(self):
         self._vms = {}
-        super().setUp()
+        self.base_setUp("%s-softmmu", "qemu-system-%s")
 
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin)
-- 
2.20.1

[Qemu-devel] [PATCH 3/5] tests/acceptance: Introduce LinuxUserTest base class

[Philippe Mathieu-Daudé]

Similarly to the MachineTest base class, this class contains
methods common to linux-user tests.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 6785c7399e..0a9277c991 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -13,6 +13,7 @@ import sys
 import uuid
 
 import avocado
+from avocado.utils import process
 
 SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
@@ -90,3 +91,11 @@ class MachineTest(Test):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
+
+
+class LinuxUserTest(Test):
+    def setUp(self):
+        self.base_setUp("%s-linux-user", "qemu-%s")
+
+    def run(self, cmd):
+        return process.run("%s %s" % (self.qemu_bin, cmd))
-- 
2.20.1

[Qemu-devel] [PATCH 4/5] tests/acceptance: Add bFLT loader linux-user test

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Add a very quick test that runs a busybox binary in bFLT format:

  $ avocado --show=app run tests/acceptance/load_bflt.py
  JOB ID     : db94d5960ce564c50904d666a7e259148c27e88f
  JOB LOG    : ~/avocado/job-results/job-2019-06-25T10.52-db94d59/job.log
   (1/1) tests/acceptance/load_bflt.py:LoadBFLT.test_stm32: PASS (0.15 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 0.54 s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
This test currently fails on Aarch64 host, see LP#1833668
https://bugs.launchpad.net/qemu/+bug/1833668
---
 tests/acceptance/load_bflt.py | 52 +++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 tests/acceptance/load_bflt.py

diff --git a/tests/acceptance/load_bflt.py b/tests/acceptance/load_bflt.py
new file mode 100644
index 0000000000..fdcaf8ad88
--- /dev/null
+++ b/tests/acceptance/load_bflt.py
@@ -0,0 +1,52 @@
+# Test the bFLT format
+#
+# Copyright (C) 2019 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import bz2
+import subprocess
+
+from avocado_qemu import LinuxUserTest
+
+
+class LoadBFLT(LinuxUserTest):
+
+    def extract_cpio(self, cpio_path):
+        """
+        Extracts a cpio archive into the test workdir
+
+        :param cpio_path: path to the cpio archive
+        """
+        cwd = os.getcwd()
+        os.chdir(self.workdir)
+        with bz2.open(cpio_path, 'rb') as archive_cpio:
+            subprocess.run(['cpio', '-i'], input=archive_cpio.read(),
+                           stderr=subprocess.DEVNULL)
+        os.chdir(cwd)
+
+    def test_stm32(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=linux_user
+        :avocado: tags=quick
+        """
+        # See https://elinux.org/STM32#User_Space
+        rootfs_url = ('https://elinux.org/images/5/51/'
+                      'Stm32_mini_rootfs.cpio.bz2')
+        rootfs_hash = '9f065e6ba40cce7411ba757f924f30fcc57951e6'
+        rootfs_path_bz2 = self.fetch_asset(rootfs_url, asset_hash=rootfs_hash)
+        busybox_path = self.workdir + "/bin/busybox"
+
+        self.extract_cpio(rootfs_path_bz2)
+
+        cmd = ''
+        res = self.run("%s %s" % (busybox_path, cmd))
+        ver = 'BusyBox v1.24.0.git (2015-02-03 22:17:13 CET) multi-call binary.'
+        self.assertIn(ver, res.stdout_text)
+
+        cmd = 'uname -a'
+        res = self.run("%s %s" % (busybox_path, cmd))
+        unm = 'armv7l GNU/Linux'
+        self.assertIn(unm, res.stdout_text)
-- 
2.20.1

[Qemu-devel] [PATCH 5/5] .travis.yml: Let the avocado job run linux-user tests

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index aeb9b211cd..2ece761ac7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -231,7 +231,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,arm-linux-user"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
-- 
2.20.1

Re: [Qemu-devel] [PATCH 0/5] tests/acceptance: Add bFLT loader linux-user test

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190625101524.13447-1-philmd@redhat.com/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa
=== TEST SCRIPT END ===

  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o

Warning, treated as error:
/var/tmp/patchew-tester-tmp-63tbziko/src/docs/devel/testing.rst:627:Title underline too short.

The ``avocado_qemu.MachineTest`` base test class


The full log is available at
http://patchew.org/logs/20190625101524.13447-1-philmd@redhat.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH 0/5] tests/acceptance: Add bFLT loader linux-user test

[Philippe Mathieu-Daudé]

On 6/25/19 12:47 PM, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20190625101524.13447-1-philmd@redhat.com/
> 
> This series failed build test on s390x host. Please find the details below.
> 
> === TEST SCRIPT END ===
> 
>   CC      qapi/qmp-registry.o
>   CC      qapi/qmp-dispatch.o
> 
> Warning, treated as error:
> /var/tmp/patchew-tester-tmp-63tbziko/src/docs/devel/testing.rst:627:Title underline too short.
>
> The ``avocado_qemu.MachineTest`` base test class

Ah, this warning was not obvious to figure out...

The fix is trivial:

-- >8 --

 The ``avocado_qemu.MachineTest`` base test class
------------------------------------------
+------------------------------------------------

---