[Qemu-devel] [PULL 00/19] Python queue, 2019-05-02

[Qemu-devel] [PULL 00/19] Python queue, 2019-05-02

[Eduardo Habkost]

The following changes since commit 8482ff2eb3bb95020eb2f370a9b3ea26511e41df:

  Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging (2019-05-02 12:04:51 +0100)

are available in the Git repository at:

  git://github.com/ehabkost/qemu.git tags/python-next-pull-request

for you to fetch changes up to faf441429adfe5767be52c5dcdb8bc03161d064f:

  configure: automatically pick python3 is available (2019-05-02 21:33:27 -0300)

----------------------------------------------------------------
Python queue, 2019-05-02

* configure: automatically pick python3 is available
  (Daniel P. Berrangé)

* tests/acceptance (Cleber Rosa, Philippe Mathieu-Daudé):
  * Multi-architecture test support
  * Multiple arch-specific boot_linux_console test cases
  * Increase verbosity of avocado by default
  * docstring improvements

----------------------------------------------------------------

Cleber Rosa (17):
  tests/acceptance: show avocado test execution by default
  tests/acceptance: improve docstring on pick_default_qemu_bin()
  tests/acceptance: fix doc reference to avocado_qemu directory
  tests/acceptance: introduce arch parameter and attribute
  tests/acceptance: use "arch:" tag to filter target specific tests
  tests/acceptance: look for target architecture in test tags first
  tests/boot_linux_console: rename the x86_64 after the arch and machine
  tests/boot_linux_console: update the x86_64 kernel
  tests/boot_linux_console: add common kernel command line options
  tests/boot_linux_console: increase timeout
  tests/boot_linux_console: refactor the console watcher into utility
    method
  scripts/qemu.py: support adding a console with the default serial
    device
  tests/boot_linux_console: add a test for mips64el + malta
  tests/boot_linux_console: add a test for aarch64 + virt
  tests/boot_linux_console: add a test for arm + virt
  tests/boot_linux_console: add a test for s390x + s390-ccw-virtio
  tests/boot_linux_console: add a test for alpha + clipper

Daniel P. Berrangé (1):
  configure: automatically pick python3 is available

Philippe Mathieu-Daudé (1):
  tests/boot_linux_console: add a test for mips + malta

 docs/devel/testing.rst                    |  35 +++-
 configure                                 |  18 +-
 .travis.yml                               |   4 +-
 python/qemu/__init__.py                   |  50 ++---
 tests/Makefile.include                    |   5 +-
 tests/acceptance/avocado_qemu/__init__.py |  23 ++-
 tests/acceptance/boot_linux_console.py    | 211 ++++++++++++++++++++--
 tests/acceptance/linux_initrd.py          |   2 +-
 tests/acceptance/virtio_version.py        |   2 +-
 9 files changed, 285 insertions(+), 65 deletions(-)

-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 01/19] tests/acceptance: show avocado test execution by default

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

The current version of the "check-acceptance" target will only show
one line for execution of all tests.  That's probably OK if the tests
to be run are quick enough and they're always the same.

But, there's already one test alone that takes on average ~5 seconds
to run, we intend to adapt the list of tests to match the user's build
environment (among other choices).

Because of that, let's present the default Avocado UI by default.
Users can always choose a different output by setting the AVOCADO_SHOW
variable.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-2-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml            | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 2e06aee9d0..e6f951e01a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -224,7 +224,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
-        - TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
+        - TEST_CMD="make check-acceptance"
       addons:
         apt:
           packages:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index e2432d5e77..445aa0bb12 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1133,7 +1133,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
-AVOCADO_SHOW=none
+AVOCADO_SHOW=app
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 02/19] tests/acceptance: improve docstring on pick_default_qemu_bin()

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Making it clear what is returned by this utility function.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-3-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index a66ec72daa..b7c5d598d8 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -27,6 +27,10 @@ def pick_default_qemu_bin():
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
+
+    :returns: the path to the default QEMU binary or None if one could not
+              be found
+    :rtype: str or None
     """
     arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 03/19] tests/acceptance: fix doc reference to avocado_qemu directory

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

The "this directory" reference is misleading and confusing, it's a
leftover from when this text was proposed in a README file inside
the "tests/acceptance/avocado_qemu" directory.

When that text was moved to the top level docs directory, the
reference was not updated.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-4-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/devel/testing.rst | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 60f897d915..7c1b199323 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -590,8 +590,9 @@ Alternatively, follow the instructions on this link:
 Overview
 --------
 
-This directory provides the ``avocado_qemu`` Python module, containing
-the ``avocado_qemu.Test`` class.  Here's a simple usage example:
+The ``tests/acceptance/avocado_qemu`` directory provides the
+``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
+class.  Here's a simple usage example:
 
 .. code::
 
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 04/19] tests/acceptance: introduce arch parameter and attribute

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

It's useful to define the architecture that should be used in
situations such as:
 * the intended target of the QEMU binary to be used on tests
 * the architecture of code to be run within the QEMU binary, such
   as a kernel image or a full blown guest OS image

This commit introduces both a test parameter and a test instance
attribute, that will contain such a value.

Now, when the "arch" test parameter is given, it will influence the
selection of the default QEMU binary, if one is not given explicitly
by means of the "qemu_img" parameter.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-5-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/devel/testing.rst                    | 28 +++++++++++++++++++++++
 tests/acceptance/avocado_qemu/__init__.py | 14 +++++++++---
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 7c1b199323..83bf9f09ac 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -727,6 +727,21 @@ vm
 A QEMUMachine instance, initially configured according to the given
 ``qemu_bin`` parameter.
 
+arch
+~~~~
+
+The architecture can be used on different levels of the stack, e.g. by
+the framework or by the test itself.  At the framework level, it will
+currently influence the selection of a QEMU binary (when one is not
+explicitly given).
+
+Tests are also free to use this attribute value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+The ``arch`` attribute will be set to the test parameter of the same
+name, and if one is not given explicitly, it will be set to ``None``.
+
 qemu_bin
 ~~~~~~~~
 
@@ -749,6 +764,19 @@ like the following:
 
   PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
 
+arch
+~~~~
+
+The architecture that will influence the selection of a QEMU binary
+(when one is not explicitly given).
+
+Tests are also free to use this parameter value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+This parameter has a direct relation with the ``arch`` attribute.  If
+not given, it will default to None.
+
 qemu_bin
 ~~~~~~~~
 
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index b7c5d598d8..688a0746a2 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -23,16 +23,22 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin():
+def pick_default_qemu_bin(arch=None):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
 
+    :param arch: the arch to use when looking for a QEMU binary (the target
+                 will match the arch given).  If None (the default), arch
+                 will be the current host system arch (as given by
+                 :func:`os.uname`).
+    :type arch: str
     :returns: the path to the default QEMU binary or None if one could not
               be found
     :rtype: str or None
     """
-    arch = os.uname()[4]
+    if arch is None:
+        arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
                                           "qemu-system-%s" % arch)
     if is_readable_executable_file(qemu_bin_relative_path):
@@ -47,8 +53,10 @@ def pick_default_qemu_bin():
 class Test(avocado.Test):
     def setUp(self):
         self._vms = {}
+        self.arch = self.params.get('arch')
+        default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
-                                        default=pick_default_qemu_bin())
+                                        default=default_qemu_bin)
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
 
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 05/19] tests/acceptance: use "arch:" tag to filter target specific tests

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Currently, some tests contains target architecture information, in the
form of a "x86_64" tag.  But that tag is not respected in the default
execution, that is, "make check-acceptance" doesn't do anything with
it.

That said, even the target architecture handling currently present in
the "avocado_qemu.Test" class is pretty limited.  For instance, by
default, it chooses a target based on the host architecture.

Because the original implementation of the tags feature in Avocado did
not include any time of namespace or "key:val" mechanism, no tag has
relation to another tag.  The new implementation of the tags feature
from version 67.0 onwards, allows "key:val" tags, and because of that,
a test can be classified with a tag in a given key.  For instance, the
new proposed version of the "boot_linux_console.py" test, which
downloads and attempts to run a x86_64 kernel, is now tagged as:

  :avocado: tags=arch:x86_64

This means that it can be filtered (out) when no x86_64 target is
available.  At the same time, tests that don't have a "arch:" tag,
will not be filtered out.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-6-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/Makefile.include                 | 3 +++
 tests/acceptance/boot_linux_console.py | 2 +-
 tests/acceptance/linux_initrd.py       | 2 +-
 tests/acceptance/virtio_version.py     | 2 +-
 4 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 445aa0bb12..7c8b9c84b2 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1134,6 +1134,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
+AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
@@ -1159,6 +1160,8 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
+            $(AVOCADO_TAGS) \
             --failfast=on $(SRC_PATH)/tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index beeb1e59e8..fa4594f612 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -18,7 +18,7 @@ class BootLinuxConsole(Test):
     Boots a x86_64 Linux kernel and checks that the console is operational
     and the kernel command line is properly passed from QEMU to the kernel
 
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index fbdb48e43f..23be5a63aa 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -19,7 +19,7 @@ class LinuxInitrd(Test):
     """
     Checks QEMU evaluates correctly the initrd file passed as -initrd option.
 
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 300
diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
index 37fc01ea18..8b97453ff8 100644
--- a/tests/acceptance/virtio_version.py
+++ b/tests/acceptance/virtio_version.py
@@ -61,7 +61,7 @@ class VirtioVersionCheck(Test):
     same device tree created by `disable-modern` and
     `disable-legacy`.
 
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 06/19] tests/acceptance: look for target architecture in test tags first

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

A test can, optionally, be tagged for one or many architectures.  If a
test has been tagged for a single architecture, there's a high chance
that the test won't run on other architectures.  This changes the
default order of choosing a default target architecture to use based
on the 'arch' tag value first.

The precedence order is for choosing a QEMU binary to use for a test
is now:

 * qemu_bin parameter
 * arch parameter
 * arch tag value (for example, x86_64 if ":avocado: tags=arch:x86_64
   is used)

This means that if one runs:

 $ avocado run -p qemu_bin=/usr/bin/qemu-system-x86_64 test.py

No arch parameter or tag will influence the selection of the QEMU
target binary.  If one runs:

 $ avocado run -p arch=ppc64 test.py

The target binary selection mechanism will attempt to find a binary
such as "ppc64-softmmu/qemu-system-ppc64".  And finally, if one runs
a test that is tagged (in its docstring) with "arch:aarch64":

 $ avocado run aarch64.py

The target binary selection mechanism will attempt to find a binary
such as "aarch64-softmmu/qemu-system-aarch64".

At this time, no provision is made to cancel the execution of tests if
the arch parameter given (manually) does not match the test "arch"
tag, but it may be a useful default behavior to be added in the
future.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-7-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/devel/testing.rst                    | 4 +++-
 tests/acceptance/avocado_qemu/__init__.py | 7 ++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 83bf9f09ac..da2d0fc964 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -740,7 +740,9 @@ A test may, for instance, use the same value when selecting the
 architecture of a kernel or disk image to boot a VM with.
 
 The ``arch`` attribute will be set to the test parameter of the same
-name, and if one is not given explicitly, it will be set to ``None``.
+name.  If one is not given explicitly, it will either be set to
+``None``, or, if the test is tagged with one (and only one)
+``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
 
 qemu_bin
 ~~~~~~~~
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 688a0746a2..2b236a1cf0 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -53,7 +53,12 @@ def pick_default_qemu_bin(arch=None):
 class Test(avocado.Test):
     def setUp(self):
         self._vms = {}
-        self.arch = self.params.get('arch')
+        arches = self.tags.get('arch', [])
+        if len(arches) == 1:
+            arch = arches.pop()
+        else:
+            arch = None
+        self.arch = self.params.get('arch', default=arch)
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 07/19] tests/boot_linux_console: rename the x86_64 after the arch and machine

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Given that the test is specific to x86_64 and pc, and new tests are
going to be added to the same class, let's rename it accordingly.
Also, let's make the class documentation not architecture specific.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-8-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index fa4594f612..4d57741bd8 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -15,15 +15,17 @@ from avocado_qemu import Test
 
 class BootLinuxConsole(Test):
     """
-    Boots a x86_64 Linux kernel and checks that the console is operational
-    and the kernel command line is properly passed from QEMU to the kernel
-
-    :avocado: tags=arch:x86_64
+    Boots a Linux kernel and checks that the console is operational and the
+    kernel command line is properly passed from QEMU to the kernel
     """
 
     timeout = 60
 
-    def test(self):
+    def test_x86_64_pc(self):
+        """
+        :avocado: tags=arch:x86_64
+        :avocado: tags=machine:pc
+        """
         kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
         kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 08/19] tests/boot_linux_console: update the x86_64 kernel

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Update to the stock Fedora 29 kernel, from the Fedora 28.  New tests
will be added using the 29 kernel, so for consistency, let's also
update it here.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
CC: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190312171824.5134-9-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 4d57741bd8..03a4445176 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -26,9 +26,9 @@ class BootLinuxConsole(Test):
         :avocado: tags=arch:x86_64
         :avocado: tags=machine:pc
         """
-        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
-                      'Everything/x86_64/os/images/pxeboot/vmlinuz')
-        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
+        kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
+                      'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
+        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_machine('pc')
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 09/19] tests/boot_linux_console: add common kernel command line options

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

The 'printk.time=0' option makes it easier to parse the console
output.  Let's set it as a default, and reusable, kernel command line
options for this and future similar tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-10-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 03a4445176..a94f5cd2c8 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -21,6 +21,8 @@ class BootLinuxConsole(Test):
 
     timeout = 60
 
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -33,7 +35,7 @@ class BootLinuxConsole(Test):
 
         self.vm.set_machine('pc')
         self.vm.set_console()
-        kernel_command_line = 'console=ttyS0'
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

When running on very low powered environments, some tests may time out
causing false negatives.  As a conservative change, and for
considering that human time (investigating false negatives) is worth
more than some extra machine cycles (and time), let's increase the
overall timeout.

CC: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-11-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index a94f5cd2c8..bfcae3771b 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -19,7 +19,7 @@ class BootLinuxConsole(Test):
     kernel command line is properly passed from QEMU to the kernel
     """
 
-    timeout = 60
+    timeout = 90
 
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 11/19] tests/boot_linux_console: refactor the console watcher into utility method

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

This introduces a utility method that monitors the console device and
looks for either a message that signals the test success or failure.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-12-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 30 ++++++++++++++++++--------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index bfcae3771b..32f1d4d0bf 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -23,6 +23,25 @@ class BootLinuxConsole(Test):
 
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
+    def wait_for_console_pattern(self, success_message,
+                                 failure_message='Kernel panic - not syncing'):
+        """
+        Waits for messages to appear on the console, while logging the content
+
+        :param success_message: if this message appears, test succeeds
+        :param failure_message: if this message appears, test fails
+        """
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline()
+            console_logger.debug(msg.strip())
+            if success_message in msg:
+                break
+            if failure_message in msg:
+                fail = 'Failure message found in console: %s' % failure_message
+                self.fail(fail)
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -39,12 +58,5 @@ class BootLinuxConsole(Test):
         self.vm.add_args('-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-        console = self.vm.console_socket.makefile()
-        console_logger = logging.getLogger('console')
-        while True:
-            msg = console.readline()
-            console_logger.debug(msg.strip())
-            if 'Kernel command line: %s' % kernel_command_line in msg:
-                break
-            if 'Kernel panic - not syncing' in msg:
-                self.fail("Kernel panic reached")
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 12/19] scripts/qemu.py: support adding a console with the default serial device

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

The set_console() utility function either adds a device based on the
explicitly given device type, or adds a known good type of device
based on the machine type.

But, for a number of machine types, it may be impossible or
inconvenient to add the devices by means of "-device" command line
options, and then it may better to just use the "-serial" option and
let QEMU itself, based on the machine type, set the device
accordingly.

To achieve that, the behavior of set_console() now flags the intention
to add a console device on launch(), and if no explicit device type is
given the "-serial" option is going to be added to the QEMU command
line, instead of raising exceptions.

Based on testing with different machine types, the CONSOLE_DEV_TYPES
is not necessary anymore, so it's being removed, as is the logic to
use it.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-13-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 python/qemu/__init__.py | 50 ++++++++++++++---------------------------
 1 file changed, 17 insertions(+), 33 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index fd144c0006..81d9657ec0 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -41,17 +41,6 @@ def kvm_available(target_arch=None):
     return os.access("/dev/kvm", os.R_OK | os.W_OK)
 
 
-#: Maps machine types to the preferred console device types
-CONSOLE_DEV_TYPES = {
-    r'^clipper$': 'isa-serial',
-    r'^malta': 'isa-serial',
-    r'^(pc.*|q35.*|isapc)$': 'isa-serial',
-    r'^(40p|powernv|prep)$': 'isa-serial',
-    r'^pseries.*': 'spapr-vty',
-    r'^s390-ccw-virtio.*': 'sclpconsole',
-    }
-
-
 class QEMUMachineError(Exception):
     """
     Exception called when an error in QEMUMachine happens.
@@ -130,6 +119,7 @@ class QEMUMachine(object):
         self._temp_dir = None
         self._launched = False
         self._machine = None
+        self._console_set = False
         self._console_device_type = None
         self._console_address = None
         self._console_socket = None
@@ -248,13 +238,17 @@ class QEMUMachine(object):
                 '-display', 'none', '-vga', 'none']
         if self._machine is not None:
             args.extend(['-machine', self._machine])
-        if self._console_device_type is not None:
+        if self._console_set:
             self._console_address = os.path.join(self._temp_dir,
                                                  self._name + "-console.sock")
             chardev = ('socket,id=console,path=%s,server,nowait' %
                        self._console_address)
-            device = '%s,chardev=console' % self._console_device_type
-            args.extend(['-chardev', chardev, '-device', device])
+            args.extend(['-chardev', chardev])
+            if self._console_device_type is None:
+                args.extend(['-serial', 'chardev:console'])
+            else:
+                device = '%s,chardev=console' % self._console_device_type
+                args.extend(['-device', device])
         return args
 
     def _pre_launch(self):
@@ -480,30 +474,20 @@ class QEMUMachine(object):
         line.
 
         This is a convenience method that will either use the provided
-        device type, of if not given, it will used the device type set
-        on CONSOLE_DEV_TYPES.
+        device type, or default to a "-serial chardev:console" command
+        line argument.
 
         The actual setting of command line arguments will be be done at
         machine launch time, as it depends on the temporary directory
         to be created.
 
-        @param device_type: the device type, such as "isa-serial"
-        @raises: QEMUMachineAddDeviceError if the device type is not given
-                 and can not be determined.
-        """
-        if device_type is None:
-            if self._machine is None:
-                raise QEMUMachineAddDeviceError("Can not add a console device:"
-                                                " QEMU instance without a "
-                                                "defined machine type")
-            for regex, device in CONSOLE_DEV_TYPES.items():
-                if re.match(regex, self._machine):
-                    device_type = device
-                    break
-            if device_type is None:
-                raise QEMUMachineAddDeviceError("Can not add a console device:"
-                                                " no matching console device "
-                                                "type definition")
+        @param device_type: the device type, such as "isa-serial".  If
+                            None is given (the default value) a "-serial
+                            chardev:console" command line argument will
+                            be used instead, resorting to the machine's
+                            default device type.
+        """
+        self._console_set = True
         self._console_device_type = device_type
 
     @property
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 13/19] tests/boot_linux_console: add a test for mips + malta

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.  Also, it relies on the serial
device set by the machine itself.

If mips is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:mips tests/acceptance
    $ avocado run -t machine:malta tests/acceptance
    $ avocado run -t endian:big tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20190312171824.5134-14-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 41 ++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index e6f951e01a..f6f5d39aca 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 32f1d4d0bf..028fefd7ee 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -8,9 +8,12 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import os
 import logging
 
 from avocado_qemu import Test
+from avocado.utils import process
+from avocado.utils import archive
 
 
 class BootLinuxConsole(Test):
@@ -42,6 +45,21 @@ class BootLinuxConsole(Test):
                 fail = 'Failure message found in console: %s' % failure_message
                 self.fail(fail)
 
+    def extract_from_deb(self, deb, path):
+        """
+        Extracts a file from a deb package into the test workdir
+
+        :param deb: path to the deb archive
+        :param file: path within the deb archive of the file to be extracted
+        :returns: path of the extracted file
+        """
+        cwd = os.getcwd()
+        os.chdir(self.workdir)
+        process.run("ar x %s data.tar.gz" % deb)
+        archive.extract("data.tar.gz", self.workdir)
+        os.chdir(cwd)
+        return self.workdir + path
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -60,3 +78,26 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_mips_malta(self):
+        """
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20130217T032700Z/pool/main/l/linux-2.6/'
+                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
+        deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-2.6.32-5-4kc-malta')
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 14/19] tests/boot_linux_console: add a test for mips64el + malta

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.

If mips64el is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips64el"
tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:mips64el tests/acceptance
    $ avocado run -t machine:malta tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Message-Id: <20190312171824.5134-15-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 33 ++++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index f6f5d39aca..cb46098972 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 028fefd7ee..38b33b25f7 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -101,3 +101,36 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_mips64el_malta(self):
+        """
+        This test requires the ar tool to extract "data.tar.gz" from
+        the Debian package.
+
+        The kernel can be rebuilt using this Debian kernel source [1] and
+        following the instructions on [2].
+
+        [1] http://snapshot.debian.org/package/linux-2.6/2.6.32-48/
+            #linux-source-2.6.32_2.6.32-48
+        [2] https://kernel-team.pages.debian.net/kernel-handbook/
+            ch-common-tasks.html#s-common-official
+
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:malta
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20130217T032700Z/pool/main/l/linux-2.6/'
+                   'linux-image-2.6.32-5-5kc-malta_2.6.32-48_mipsel.deb')
+        deb_hash = '1aaec92083bf22fda31e0d27fa8d9a388e5fc3d5'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-2.6.32-5-5kc-malta')
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 15/19] tests/boot_linux_console: add a test for aarch64 + virt

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Just like the previous tests, boots a Linux kernel on a aarch64 target
using the virt machine.

One special option added is the CPU type, given that the kernel
selected fails to boot on the virt machine's default CPU (cortex-a15).

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20190312171824.5134-18-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index cb46098972..753e40a115 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 38b33b25f7..6d9362922f 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -134,3 +134,24 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_aarch64_virt(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
+                      'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
+        kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('virt')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-cpu', 'cortex-a53',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 16/19] tests/boot_linux_console: add a test for arm + virt

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Just like the previous tests, boots a Linux kernel on an arm target
using the virt machine.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20190312171824.5134-19-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 753e40a115..268df60345 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 6d9362922f..54b6e2625d 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -155,3 +155,23 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_arm_virt(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
+                      'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
+        kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('virt')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 17/19] tests/boot_linux_console: add a test for s390x + s390-ccw-virtio

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Just like the previous tests, boots a Linux kernel on a s390x target
using the s390-ccw-virtio machine.

Because it's not possible to have multiple VT220 consoles,
'-nodefaults' is used, so that the one set with set_console() works
correctly.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20190312171824.5134-20-crosa@redhat.com>
[ehabkost: Updated kernel URL to point to fedoraproject.org]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 268df60345..9b271b963d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 54b6e2625d..7c1099e5c8 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -175,3 +175,23 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_s390x_s390_ccw_virtio(self):
+        """
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390_ccw_virtio
+        """
+        kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/'
+                      'releases/29/Everything/s390x/os/images/kernel.img')
+        kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('s390-ccw-virtio')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
+        self.vm.add_args('-nodefaults',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 18/19] tests/boot_linux_console: add a test for alpha + clipper

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.  One extra command added to
the QEMU command line is '-vga std', because the kernel used is
known to crash without it.

If alpha is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:alpha" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:alpha tests/acceptance
    $ avocado run -t machine:clipper tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20190312171824.5134-21-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 9b271b963d..66448d99d6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -223,7 +223,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 7c1099e5c8..d5c500ea30 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -195,3 +195,25 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_alpha_clipper(self):
+        """
+        :avocado: tags=arch:alpha
+        :avocado: tags=machine:clipper
+        """
+        kernel_url = ('http://archive.debian.org/debian/dists/lenny/main/'
+                      'installer-alpha/current/images/cdrom/vmlinuz')
+        kernel_hash = '3a943149335529e2ed3e74d0d787b85fb5671ba3'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        uncompressed_kernel = archive.uncompress(kernel_path, self.workdir)
+
+        self.vm.set_machine('clipper')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-vga', 'std',
+                         '-kernel', uncompressed_kernel,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Eduardo Habkost]

From: Daniel P. Berrangé <berrange@redhat.com>

Unless overridden via an env var or configure arg, QEMU will only look
for the 'python' binary in $PATH. This is unhelpful on distros which
are only shipping Python 3.x (eg Fedora) in their default install as,
if they comply with PEP 394, the bare 'python' binary won't exist.

This changes configure so that by default it will search for all three
common python binaries, preferring to find Python 3.x versions.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190327170701.23798-1-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 configure | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index 60719ddcc5..f88011b94d 100755
--- a/configure
+++ b/configure
@@ -899,7 +899,18 @@ fi
 
 : ${make=${MAKE-make}}
 : ${install=${INSTALL-install}}
-: ${python=${PYTHON-python}}
+# We prefer python 3.x. A bare 'python' is traditionally
+# python 2.x, but some distros have it as python 3.x, so
+# we check that before python2
+python=
+for binary in "${PYTHON-python3}" python python2
+do
+    if has "$binary"
+    then
+        python="$binary"
+        break
+    fi
+done
 : ${smbd=${SMBD-/usr/sbin/smbd}}
 
 # Default objcc to clang if available, otherwise use CC
@@ -1818,8 +1829,9 @@ EOF
 exit 0
 fi
 
-if ! has $python; then
-  error_exit "Python not found. Use --python=/path/to/python"
+if test -z "$python"
+then
+    error_exit "Python not found. Use --python=/path/to/python"
 fi
 
 # Note that if the Python conditional here evaluates True we will exit
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PULL 00/19] Python queue, 2019-05-02

[Peter Maydell]

On Fri, 3 May 2019 at 01:41, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> The following changes since commit 8482ff2eb3bb95020eb2f370a9b3ea26511e41df:
>
>   Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging (2019-05-02 12:04:51 +0100)
>
> are available in the Git repository at:
>
>   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
>
> for you to fetch changes up to faf441429adfe5767be52c5dcdb8bc03161d064f:
>
>   configure: automatically pick python3 is available (2019-05-02 21:33:27 -0300)
>
> ----------------------------------------------------------------
> Python queue, 2019-05-02
>
> * configure: automatically pick python3 is available
>   (Daniel P. Berrangé)
>
> * tests/acceptance (Cleber Rosa, Philippe Mathieu-Daudé):
>   * Multi-architecture test support
>   * Multiple arch-specific boot_linux_console test cases
>   * Increase verbosity of avocado by default
>   * docstring improvements
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.1
for any user-visible changes.

-- PMM

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Thomas Huth]

On 03/05/2019 02.41, Eduardo Habkost wrote:
> From: Daniel P. Berrangé <berrange@redhat.com>
> 
> Unless overridden via an env var or configure arg, QEMU will only look
> for the 'python' binary in $PATH. This is unhelpful on distros which
> are only shipping Python 3.x (eg Fedora) in their default install as,
> if they comply with PEP 394, the bare 'python' binary won't exist.
> 
> This changes configure so that by default it will search for all three
> common python binaries, preferring to find Python 3.x versions.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  configure | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)

I haven't bisected it, but I think this patch here broke the gitlab-ci tests:

 https://gitlab.com/huth/qemu/-/jobs/206806257

Seems like the test is now failing when you don't have an UTF-8 locale:

 LANG=C make check-qapi-schema
 [...]
 TEST    tests/qapi-schema/union-base-empty.out
 --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
 +++ -	2019-05-03 15:42:01.561762978 +0000
 @@ -1 +1 @@
 -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
 +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
 /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
 make: *** [check-tests/qapi-schema/unicode-str.json] Error 1

Any ideas how to fix this?

 Thomas

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> On 03/05/2019 02.41, Eduardo Habkost wrote:
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > 
> > Unless overridden via an env var or configure arg, QEMU will only look
> > for the 'python' binary in $PATH. This is unhelpful on distros which
> > are only shipping Python 3.x (eg Fedora) in their default install as,
> > if they comply with PEP 394, the bare 'python' binary won't exist.
> > 
> > This changes configure so that by default it will search for all three
> > common python binaries, preferring to find Python 3.x versions.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  configure | 18 +++++++++++++++---
> >  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> 
>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> Seems like the test is now failing when you don't have an UTF-8 locale:
> 
>  LANG=C make check-qapi-schema
>  [...]
>  TEST    tests/qapi-schema/union-base-empty.out
>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>  +++ -	2019-05-03 15:42:01.561762978 +0000
>  @@ -1 +1 @@
>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> 
> Any ideas how to fix this?

python3 is basically doomed if you use the C locale for LC_CTYPE, as
it is not 8-bit clean.

If a python3 program is liable to see UTF-8 input data, the following
env should generally be set when running python:

   LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8  

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> On 03/05/2019 02.41, Eduardo Habkost wrote:
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > 
> > Unless overridden via an env var or configure arg, QEMU will only look
> > for the 'python' binary in $PATH. This is unhelpful on distros which
> > are only shipping Python 3.x (eg Fedora) in their default install as,
> > if they comply with PEP 394, the bare 'python' binary won't exist.
> > 
> > This changes configure so that by default it will search for all three
> > common python binaries, preferring to find Python 3.x versions.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  configure | 18 +++++++++++++++---
> >  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> 
>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> Seems like the test is now failing when you don't have an UTF-8 locale:
> 
>  LANG=C make check-qapi-schema
>  [...]
>  TEST    tests/qapi-schema/union-base-empty.out
>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>  +++ -	2019-05-03 15:42:01.561762978 +0000
>  @@ -1 +1 @@
>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> 
> Any ideas how to fix this?

python3 is basically doomed if you use the C locale for LC_CTYPE, as
it is not 8-bit clean.

If a python3 program is liable to see UTF-8 input data, the following
env should generally be set when running python:

   LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8  

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 05:54:35PM +0100, Daniel P. Berrangé wrote:
> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> > On 03/05/2019 02.41, Eduardo Habkost wrote:
> > > From: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > Unless overridden via an env var or configure arg, QEMU will only look
> > > for the 'python' binary in $PATH. This is unhelpful on distros which
> > > are only shipping Python 3.x (eg Fedora) in their default install as,
> > > if they comply with PEP 394, the bare 'python' binary won't exist.
> > > 
> > > This changes configure so that by default it will search for all three
> > > common python binaries, preferring to find Python 3.x versions.
> > > 
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > ---
> > >  configure | 18 +++++++++++++++---
> > >  1 file changed, 15 insertions(+), 3 deletions(-)
> > 
> > I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/206806257
> > 
> > Seems like the test is now failing when you don't have an UTF-8 locale:
> > 
> >  LANG=C make check-qapi-schema
> >  [...]
> >  TEST    tests/qapi-schema/union-base-empty.out
> >  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
> >  +++ -	2019-05-03 15:42:01.561762978 +0000
> >  @@ -1 +1 @@
> >  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
> >  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
> >  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
> >  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> > 
> > Any ideas how to fix this?
> 
> python3 is basically doomed if you use the C locale for LC_CTYPE, as
> it is not 8-bit clean.
> 
> If a python3 program is liable to see UTF-8 input data, the following
> env should generally be set when running python:
> 
>    LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8

Oh, actually I forgot we did that and then changed approach in QEMU,
see these:

commit 0d6b93deeeb3cc190692d629f5927befdc8b1fb8
Author: Matthias Maier <tamiko@43-1.org>
Date:   Mon Jun 18 19:59:58 2018 +0200

    Revert commit d4e5ec877ca
    
    This commit removes the PYTHON_UTF8 workaround. The problem with setting
    
      LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
    
    is that the en_US.UTF-8 locale might not be available. In this case
    setting above locales results in build errors even though another UTF-8
    locale was originally set [1]. The only stable way of fixing the
    encoding problem is by specifying the encoding in Python, like the
    previous commit does.
    
    [1] https://bugs.gentoo.org/657766
    
    Signed-off-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
    Signed-off-by: Matthias Maier <tamiko@43-1.org>
    Message-Id: <20180618175958.29073-3-armbru@redhat.com>
    Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>
    [Commit message tweaked]
    Signed-off-by: Markus Armbruster <armbru@redhat.com>

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <armbru@redhat.com>
Date:   Mon Jun 18 19:59:57 2018 +0200

    qapi: Open files with encoding='utf-8'
    
    Python 2 happily reads UTF-8 files in text mode, but Python 3 requires
    either UTF-8 locale or an explicit encoding passed to open().  Commit
    d4e5ec877ca fixed this by setting the en_US.UTF-8 locale.  Falls apart
    when the locale isn't be available.
    
    Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use
    binary mode instead, with manual conversion from bytes to str.  Works,
    but opening with an explicit encoding is simpler, so do that.
    
    Since Python 2's open() doesn't support the encoding parameter, we
    need to suppress it with a version check.
    
    Reported-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
    Reported-by: Matthias Maier <tamiko@43-1.org>
    Signed-off-by: Markus Armbruster <armbru@redhat.com>
    Message-Id: <20180618175958.29073-2-armbru@redhat.com>
    Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Daniel P. Berrangé]

On Fri, May 03, 2019 at 05:54:35PM +0100, Daniel P. Berrangé wrote:
> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> > On 03/05/2019 02.41, Eduardo Habkost wrote:
> > > From: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > Unless overridden via an env var or configure arg, QEMU will only look
> > > for the 'python' binary in $PATH. This is unhelpful on distros which
> > > are only shipping Python 3.x (eg Fedora) in their default install as,
> > > if they comply with PEP 394, the bare 'python' binary won't exist.
> > > 
> > > This changes configure so that by default it will search for all three
> > > common python binaries, preferring to find Python 3.x versions.
> > > 
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > ---
> > >  configure | 18 +++++++++++++++---
> > >  1 file changed, 15 insertions(+), 3 deletions(-)
> > 
> > I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/206806257
> > 
> > Seems like the test is now failing when you don't have an UTF-8 locale:
> > 
> >  LANG=C make check-qapi-schema
> >  [...]
> >  TEST    tests/qapi-schema/union-base-empty.out
> >  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
> >  +++ -	2019-05-03 15:42:01.561762978 +0000
> >  @@ -1 +1 @@
> >  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
> >  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
> >  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
> >  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> > 
> > Any ideas how to fix this?
> 
> python3 is basically doomed if you use the C locale for LC_CTYPE, as
> it is not 8-bit clean.
> 
> If a python3 program is liable to see UTF-8 input data, the following
> env should generally be set when running python:
> 
>    LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8

Oh, actually I forgot we did that and then changed approach in QEMU,
see these:

commit 0d6b93deeeb3cc190692d629f5927befdc8b1fb8
Author: Matthias Maier <tamiko@43-1.org>
Date:   Mon Jun 18 19:59:58 2018 +0200

    Revert commit d4e5ec877ca
    
    This commit removes the PYTHON_UTF8 workaround. The problem with setting
    
      LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
    
    is that the en_US.UTF-8 locale might not be available. In this case
    setting above locales results in build errors even though another UTF-8
    locale was originally set [1]. The only stable way of fixing the
    encoding problem is by specifying the encoding in Python, like the
    previous commit does.
    
    [1] https://bugs.gentoo.org/657766
    
    Signed-off-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
    Signed-off-by: Matthias Maier <tamiko@43-1.org>
    Message-Id: <20180618175958.29073-3-armbru@redhat.com>
    Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>
    [Commit message tweaked]
    Signed-off-by: Markus Armbruster <armbru@redhat.com>

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <armbru@redhat.com>
Date:   Mon Jun 18 19:59:57 2018 +0200

    qapi: Open files with encoding='utf-8'
    
    Python 2 happily reads UTF-8 files in text mode, but Python 3 requires
    either UTF-8 locale or an explicit encoding passed to open().  Commit
    d4e5ec877ca fixed this by setting the en_US.UTF-8 locale.  Falls apart
    when the locale isn't be available.
    
    Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use
    binary mode instead, with manual conversion from bytes to str.  Works,
    but opening with an explicit encoding is simpler, so do that.
    
    Since Python 2's open() doesn't support the encoding parameter, we
    need to suppress it with a version check.
    
    Reported-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
    Reported-by: Matthias Maier <tamiko@43-1.org>
    Signed-off-by: Markus Armbruster <armbru@redhat.com>
    Message-Id: <20180618175958.29073-2-armbru@redhat.com>
    Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Philippe Mathieu-Daudé]

On 5/3/19 6:41 PM, Thomas Huth wrote:
> On 03/05/2019 02.41, Eduardo Habkost wrote:
>> From: Daniel P. Berrangé <berrange@redhat.com>
>>
>> Unless overridden via an env var or configure arg, QEMU will only look
>> for the 'python' binary in $PATH. This is unhelpful on distros which
>> are only shipping Python 3.x (eg Fedora) in their default install as,
>> if they comply with PEP 394, the bare 'python' binary won't exist.
>>
>> This changes configure so that by default it will search for all three
>> common python binaries, preferring to find Python 3.x versions.
>>
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>>  configure | 18 +++++++++++++++---
>>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> 
>  https://gitlab.com/huth/qemu/-/jobs/206806257

What's the easier way to notice that automatically?

The quicker fix I have is Peter setup'ing a GitLab account mirroring his
repo:staging branch, and warn him, but that won't scale much.

> Seems like the test is now failing when you don't have an UTF-8 locale:
> 
>  LANG=C make check-qapi-schema
>  [...]
>  TEST    tests/qapi-schema/union-base-empty.out
>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>  +++ -	2019-05-03 15:42:01.561762978 +0000
>  @@ -1 +1 @@
>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> 
> Any ideas how to fix this?
> 
>  Thomas
> 

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Eduardo Habkost]

On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> On 03/05/2019 02.41, Eduardo Habkost wrote:
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > 
> > Unless overridden via an env var or configure arg, QEMU will only look
> > for the 'python' binary in $PATH. This is unhelpful on distros which
> > are only shipping Python 3.x (eg Fedora) in their default install as,
> > if they comply with PEP 394, the bare 'python' binary won't exist.
> > 
> > This changes configure so that by default it will search for all three
> > common python binaries, preferring to find Python 3.x versions.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  configure | 18 +++++++++++++++---
> >  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> 
>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> Seems like the test is now failing when you don't have an UTF-8 locale:
> 
>  LANG=C make check-qapi-schema

I couldn't reproduce it this way, probably because I'm running Python 3.7 which
implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").

But I can force it to break using:

  PYTHONIOENCODING=ascii make check-qapi-schema

>  [...]
>  TEST    tests/qapi-schema/union-base-empty.out
>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>  +++ -	2019-05-03 15:42:01.561762978 +0000
>  @@ -1 +1 @@
>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> 
> Any ideas how to fix this?

Probably we just need to specify an explicit encoding at the statement that
prints the error message to stderr.  I will give it a try.

-- 
Eduardo

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Eduardo Habkost]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 2161 bytes --]

On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> On 03/05/2019 02.41, Eduardo Habkost wrote:
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > 
> > Unless overridden via an env var or configure arg, QEMU will only look
> > for the 'python' binary in $PATH. This is unhelpful on distros which
> > are only shipping Python 3.x (eg Fedora) in their default install as,
> > if they comply with PEP 394, the bare 'python' binary won't exist.
> > 
> > This changes configure so that by default it will search for all three
> > common python binaries, preferring to find Python 3.x versions.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  configure | 18 +++++++++++++++---
> >  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> 
>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> Seems like the test is now failing when you don't have an UTF-8 locale:
> 
>  LANG=C make check-qapi-schema

I couldn't reproduce it this way, probably because I'm running Python 3.7 which
implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").

But I can force it to break using:

  PYTHONIOENCODING=ascii make check-qapi-schema

>  [...]
>  TEST    tests/qapi-schema/union-base-empty.out
>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>  +++ -	2019-05-03 15:42:01.561762978 +0000
>  @@ -1 +1 @@
>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> 
> Any ideas how to fix this?

Probably we just need to specify an explicit encoding at the statement that
prints the error message to stderr.  I will give it a try.

-- 
Eduardo

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Eduardo Habkost]

On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote:
> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> > On 03/05/2019 02.41, Eduardo Habkost wrote:
> > > From: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > Unless overridden via an env var or configure arg, QEMU will only look
> > > for the 'python' binary in $PATH. This is unhelpful on distros which
> > > are only shipping Python 3.x (eg Fedora) in their default install as,
> > > if they comply with PEP 394, the bare 'python' binary won't exist.
> > > 
> > > This changes configure so that by default it will search for all three
> > > common python binaries, preferring to find Python 3.x versions.
> > > 
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > ---
> > >  configure | 18 +++++++++++++++---
> > >  1 file changed, 15 insertions(+), 3 deletions(-)
> > 
> > I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/206806257
> > 
> > Seems like the test is now failing when you don't have an UTF-8 locale:
> > 
> >  LANG=C make check-qapi-schema
> 
> I couldn't reproduce it this way, probably because I'm running Python 3.7 which
> implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").
> 
> But I can force it to break using:
> 
>   PYTHONIOENCODING=ascii make check-qapi-schema
> 
> >  [...]
> >  TEST    tests/qapi-schema/union-base-empty.out
> >  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
> >  +++ -	2019-05-03 15:42:01.561762978 +0000
> >  @@ -1 +1 @@
> >  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
> >  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
> >  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
> >  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> > 
> > Any ideas how to fix this?
> 
> Probably we just need to specify an explicit encoding at the statement that
> prints the error message to stderr.  I will give it a try.

Forcing a specific encoding inside test-qapi.py would very easy
on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older
versions.  I believe this is the simplest way to fix the problem
on Python 3.5 and 3.6.

Can somebody confirm this really fixes the problem on gitlab-ci?

---
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 7c8b9c84b2..af88ab6f8b 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")

-- 
Eduardo

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Eduardo Habkost]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 3312 bytes --]

On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote:
> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
> > On 03/05/2019 02.41, Eduardo Habkost wrote:
> > > From: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > Unless overridden via an env var or configure arg, QEMU will only look
> > > for the 'python' binary in $PATH. This is unhelpful on distros which
> > > are only shipping Python 3.x (eg Fedora) in their default install as,
> > > if they comply with PEP 394, the bare 'python' binary won't exist.
> > > 
> > > This changes configure so that by default it will search for all three
> > > common python binaries, preferring to find Python 3.x versions.
> > > 
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com>
> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > ---
> > >  configure | 18 +++++++++++++++---
> > >  1 file changed, 15 insertions(+), 3 deletions(-)
> > 
> > I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/206806257
> > 
> > Seems like the test is now failing when you don't have an UTF-8 locale:
> > 
> >  LANG=C make check-qapi-schema
> 
> I couldn't reproduce it this way, probably because I'm running Python 3.7 which
> implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").
> 
> But I can force it to break using:
> 
>   PYTHONIOENCODING=ascii make check-qapi-schema
> 
> >  [...]
> >  TEST    tests/qapi-schema/union-base-empty.out
> >  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
> >  +++ -	2019-05-03 15:42:01.561762978 +0000
> >  @@ -1 +1 @@
> >  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
> >  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
> >  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
> >  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
> > 
> > Any ideas how to fix this?
> 
> Probably we just need to specify an explicit encoding at the statement that
> prints the error message to stderr.  I will give it a try.

Forcing a specific encoding inside test-qapi.py would very easy
on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older
versions.  I believe this is the simplest way to fix the problem
on Python 3.5 and 3.6.

Can somebody confirm this really fixes the problem on gitlab-ci?

---
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 7c8b9c84b2..af88ab6f8b 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")

-- 
Eduardo

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Thomas Huth]

On 03/05/2019 19.04, Philippe Mathieu-Daudé wrote:
> On 5/3/19 6:41 PM, Thomas Huth wrote:
>> On 03/05/2019 02.41, Eduardo Habkost wrote:
>>> From: Daniel P. Berrangé <berrange@redhat.com>
>>>
>>> Unless overridden via an env var or configure arg, QEMU will only look
>>> for the 'python' binary in $PATH. This is unhelpful on distros which
>>> are only shipping Python 3.x (eg Fedora) in their default install as,
>>> if they comply with PEP 394, the bare 'python' binary won't exist.
>>>
>>> This changes configure so that by default it will search for all three
>>> common python binaries, preferring to find Python 3.x versions.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>> ---
>>>  configure | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
>>
>>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> What's the easier way to notice that automatically?

Maybe Peter could tweak one of his acceptance test environments to use a
non-UTF-8 locale?

 Thomas

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Thomas Huth]

On 03/05/2019 19.04, Philippe Mathieu-Daudé wrote:
> On 5/3/19 6:41 PM, Thomas Huth wrote:
>> On 03/05/2019 02.41, Eduardo Habkost wrote:
>>> From: Daniel P. Berrangé <berrange@redhat.com>
>>>
>>> Unless overridden via an env var or configure arg, QEMU will only look
>>> for the 'python' binary in $PATH. This is unhelpful on distros which
>>> are only shipping Python 3.x (eg Fedora) in their default install as,
>>> if they comply with PEP 394, the bare 'python' binary won't exist.
>>>
>>> This changes configure so that by default it will search for all three
>>> common python binaries, preferring to find Python 3.x versions.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>> ---
>>>  configure | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>
>> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
>>
>>  https://gitlab.com/huth/qemu/-/jobs/206806257
> 
> What's the easier way to notice that automatically?

Maybe Peter could tweak one of his acceptance test environments to use a
non-UTF-8 locale?

 Thomas

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Thomas Huth]

On 03/05/2019 23.34, Eduardo Habkost wrote:
> On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote:
>> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
>>> On 03/05/2019 02.41, Eduardo Habkost wrote:
>>>> From: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> Unless overridden via an env var or configure arg, QEMU will only look
>>>> for the 'python' binary in $PATH. This is unhelpful on distros which
>>>> are only shipping Python 3.x (eg Fedora) in their default install as,
>>>> if they comply with PEP 394, the bare 'python' binary won't exist.
>>>>
>>>> This changes configure so that by default it will search for all three
>>>> common python binaries, preferring to find Python 3.x versions.
>>>>
>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
>>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> ---
>>>>  configure | 18 +++++++++++++++---
>>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>
>>> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
>>>
>>>  https://gitlab.com/huth/qemu/-/jobs/206806257
>>>
>>> Seems like the test is now failing when you don't have an UTF-8 locale:
>>>
>>>  LANG=C make check-qapi-schema
>>
>> I couldn't reproduce it this way, probably because I'm running Python 3.7 which
>> implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").
>>
>> But I can force it to break using:
>>
>>   PYTHONIOENCODING=ascii make check-qapi-schema
>>
>>>  [...]
>>>  TEST    tests/qapi-schema/union-base-empty.out
>>>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>>>  +++ -	2019-05-03 15:42:01.561762978 +0000
>>>  @@ -1 +1 @@
>>>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>>>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>>>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>>>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
>>>
>>> Any ideas how to fix this?
>>
>> Probably we just need to specify an explicit encoding at the statement that
>> prints the error message to stderr.  I will give it a try.
> 
> Forcing a specific encoding inside test-qapi.py would very easy
> on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older
> versions.  I believe this is the simplest way to fix the problem
> on Python 3.5 and 3.6.
> 
> Can somebody confirm this really fixes the problem on gitlab-ci?
> 
> ---
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 7c8b9c84b2..af88ab6f8b 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
>  .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
>  $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
>  	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
> -		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
> +		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
>  		$^ >$*.test.out 2>$*.test.err; \
>  		echo $$? >$*.test.exit, \
>  		"TEST","$*.out")

Yes, thanks, this seems to fix this issue:

 https://gitlab.com/huth/qemu/-/jobs/207041381

It also works on my laptop now (which uses Python 3.6).

 Thomas

Re: [Qemu-devel] [PULL 19/19] configure: automatically pick python3 is available

[Thomas Huth]

On 03/05/2019 23.34, Eduardo Habkost wrote:
> On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote:
>> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote:
>>> On 03/05/2019 02.41, Eduardo Habkost wrote:
>>>> From: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> Unless overridden via an env var or configure arg, QEMU will only look
>>>> for the 'python' binary in $PATH. This is unhelpful on distros which
>>>> are only shipping Python 3.x (eg Fedora) in their default install as,
>>>> if they comply with PEP 394, the bare 'python' binary won't exist.
>>>>
>>>> This changes configure so that by default it will search for all three
>>>> common python binaries, preferring to find Python 3.x versions.
>>>>
>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>> Message-Id: <20190327170701.23798-1-berrange@redhat.com>
>>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> ---
>>>>  configure | 18 +++++++++++++++---
>>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>
>>> I haven't bisected it, but I think this patch here broke the gitlab-ci tests:
>>>
>>>  https://gitlab.com/huth/qemu/-/jobs/206806257
>>>
>>> Seems like the test is now failing when you don't have an UTF-8 locale:
>>>
>>>  LANG=C make check-qapi-schema
>>
>> I couldn't reproduce it this way, probably because I'm running Python 3.7 which
>> implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale").
>>
>> But I can force it to break using:
>>
>>   PYTHONIOENCODING=ascii make check-qapi-schema
>>
>>>  [...]
>>>  TEST    tests/qapi-schema/union-base-empty.out
>>>  --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err	2019-05-03 15:21:39.000000000 +0000
>>>  +++ -	2019-05-03 15:42:01.561762978 +0000
>>>  @@ -1 +1 @@
>>>  -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
>>>  +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'
>>>  /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed
>>>  make: *** [check-tests/qapi-schema/unicode-str.json] Error 1
>>>
>>> Any ideas how to fix this?
>>
>> Probably we just need to specify an explicit encoding at the statement that
>> prints the error message to stderr.  I will give it a try.
> 
> Forcing a specific encoding inside test-qapi.py would very easy
> on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older
> versions.  I believe this is the simplest way to fix the problem
> on Python 3.5 and 3.6.
> 
> Can somebody confirm this really fixes the problem on gitlab-ci?
> 
> ---
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 7c8b9c84b2..af88ab6f8b 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
>  .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
>  $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
>  	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
> -		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
> +		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
>  		$^ >$*.test.out 2>$*.test.err; \
>  		echo $$? >$*.test.exit, \
>  		"TEST","$*.out")

Yes, thanks, this seems to fix this issue:

 https://gitlab.com/huth/qemu/-/jobs/207041381

It also works on my laptop now (which uses Python 3.6).

 Thomas

Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Gerd Hoffmann]

On Thu, May 02, 2019 at 09:41:21PM -0300, Eduardo Habkost wrote:
> From: Cleber Rosa <crosa@redhat.com>
> 
> When running on very low powered environments, some tests may time out
> causing false negatives.  As a conservative change, and for
> considering that human time (investigating false negatives) is worth
> more than some extra machine cycles (and time), let's increase the
> overall timeout.

Tests can also timeout due to slow downloads of test kernels.
Any chance to run the downloads without timeout?

cheers,
  Gerd

Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Cleber Rosa]

On Wed, May 08, 2019 at 12:28:24PM +0200, Gerd Hoffmann wrote:
> On Thu, May 02, 2019 at 09:41:21PM -0300, Eduardo Habkost wrote:
> > From: Cleber Rosa <crosa@redhat.com>
> > 
> > When running on very low powered environments, some tests may time out
> > causing false negatives.  As a conservative change, and for
> > considering that human time (investigating false negatives) is worth
> > more than some extra machine cycles (and time), let's increase the
> > overall timeout.
> 
> Tests can also timeout due to slow downloads of test kernels.
> Any chance to run the downloads without timeout?

I acknowledge this is an issue, and have thought about two possible
ways to solve it:

 1) Downloading/caching/checking all the test assets in a job "pre-tests"
    plugin.  This would involve/require:

   - https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring
   - https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher

 2) Report the test phase (say, setUp()) to the test runner, which
    would allow the runner to:

   - Not take that time into account for the test (similar to
     https://github.com/avocado-framework/avocado/pull/3076)
   - Retry the setUp a number of times (if the test opts in or doesn't
     opt out)

I'm very much interested in your opinion so we can evolve the idea into
implementation.

Thanks!
- Cleber

> 
> cheers,
>   Gerd
> 

Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Gerd Hoffmann]

  Hi,

> > Tests can also timeout due to slow downloads of test kernels.
> > Any chance to run the downloads without timeout?
> 
> I acknowledge this is an issue, and have thought about two possible
> ways to solve it:
> 
>  1) Downloading/caching/checking all the test assets in a job "pre-tests"
>     plugin.

>  2) Report the test phase (say, setUp()) to the test runner, which
>     would allow the runner to:

> I'm very much interested in your opinion so we can evolve the idea into
> implementation.

(1) is the better approach I think.  That way you can give better
feedback to the user, along the lines of "download 2 of 5 in progress".
Also it allows for better management of the assets, you can have tools
to list them / fetch them / copy them from one to another test machine
/ find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching
tests to Fedora 29).

(2) is probably still useful, to report phases of longer running
tests and maybe have separate timeouts for each phase.  But for
assets I think approach (1) is better than a "download asset"
phase.

cheers,
  Gerd

Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Cleber Rosa]

On Thu, May 09, 2019 at 06:40:40AM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > Tests can also timeout due to slow downloads of test kernels.
> > > Any chance to run the downloads without timeout?
> > 
> > I acknowledge this is an issue, and have thought about two possible
> > ways to solve it:
> > 
> >  1) Downloading/caching/checking all the test assets in a job "pre-tests"
> >     plugin.
> 
> >  2) Report the test phase (say, setUp()) to the test runner, which
> >     would allow the runner to:
> 
> > I'm very much interested in your opinion so we can evolve the idea into
> > implementation.
> 
> (1) is the better approach I think.  That way you can give better
> feedback to the user, along the lines of "download 2 of 5 in progress".
> Also it allows for better management of the assets, you can have tools
> to list them / fetch them / copy them from one to another test machine
> / find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching
> tests to Fedora 29).
> 
> (2) is probably still useful, to report phases of longer running
> tests and maybe have separate timeouts for each phase.  But for
> assets I think approach (1) is better than a "download asset"
> phase.
>

I also think that approach #1 is simpler and saner, but thinking about
where we're going with the test runner development, I started to have
doubts about it.  The reason is that we're adding parallel and multi
environment (process, machine, container) execution capabilities to the
runner. Dogfood version is here:

  https://github.com/avocado-framework/avocado/pull/3111

By doing all download/caching before any single test starts, we may be
wasting a lot of CPU time that could be used for running tests, making
the overall job execution much longer.

And with regards to test phases, the runner could distinguish between
setup and test phases, possibly re-running the setup phase a number of
times until sucessfull completion, improving even further the
reliability of the *test* results (excluding the setup phase).

Anyway, this is only brainstorm at this point, so I'll let you know
when this moves forward.

> cheers,
>   Gerd
> 
> 

FIY: one of the new runner planned features, is based on a question
use case you mention, about how to run tests built into a minimal
image, such as an initrd.

Regards,
- Cleber.

Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Gerd Hoffmann]

  Hi,

> I also think that approach #1 is simpler and saner, but thinking about
> where we're going with the test runner development, I started to have
> doubts about it.  The reason is that we're adding parallel and multi
> environment (process, machine, container) execution capabilities to the
> runner. Dogfood version is here:
> 
>   https://github.com/avocado-framework/avocado/pull/3111
> 
> By doing all download/caching before any single test starts, we may be
> wasting a lot of CPU time that could be used for running tests, making
> the overall job execution much longer.

Well, if the internet link is the bottleneck running downloads in
parallel wouldn't speed up things ...

Also given that avocado caches the downloads the common case will
probably be to not download anything.

> FIY: one of the new runner planned features, is based on a question
> use case you mention, about how to run tests built into a minimal
> image, such as an initrd.

FYI: Here is what I did: https://git.kraxel.org/cgit/drminfo/tree/tests

cheers,
  Gerd