[Qemu-devel] [PATCH v3 00/20] Acceptance Tests: target architecture support

[Qemu-devel] [PATCH v3 00/20] Acceptance Tests: target architecture support

[Cleber Rosa]

The current acceptance tests don't provide any type of architecture
information that can be used to influence the selection of the QEMU
binary used on them[1].  If one is running tests on a x86_64 host, the
default QEMU binary will be "x86_64-softmmu/qemu-system-x86_64".

Given the nature of QEMU, some tests will be architecture agnostic,
while others will be architecture dependent.  The "check-qtest" and
"check-qtest-TARGET" make targets exemplify that pattern.

For the acceptance tests, the same requirement exists.  Tests should
be allowed to influence the binary used, and when they don't, a
default selection mechanism should kick in[2].  The proposed solution
here requires only that an Avocado tag is set, such as:

   class My(Test):
       def test_nx_cpu_flag(self):
           """
           :avocado: tags=arch:x86_64
           """
           test_code()

The value of the "arch" key, in this case, "x86_64" will be used when
selecting the QEMU binary to use in the test.  At the same time, if
"x86_64-softmmu" is not a built target, the test will be filtered out
by "make check-acceptance"[3].

Besides the convention explained above, where the binary will be
selected from the "arch" tag, it's also possible to set an "arch"
*parameter* that will also influence the QEMU binary selection:

  $ avocado run -p arch=aarch64 works-on-many-arches.py

Finally, it's also posible to set the "qemu_bin" parameter, which will
define (instead of just influencing) the QEMU binary to be used:

 $ avocado run -p qemu_bin=qemu-bin-aarch64 test.py

As examples for the idea proposed here, a number of "boot linux
console" tests have been added, for a number of different target
architectures.  When the build environment includes them (as it has
been added to Travis CI jobs) the architecture specific tests will be
automatically executed.

As mentioned previously, this patch series include ideas present in
other patch series, and from different authors.  I tried by best
to include the information about authorship, but if I missed any,
please accept my apologies and let me know.

---

[1] - The "boot_linux_console.py" contains a "x86_64" test tag, but
      that is informational only, in the sense that it's not consumed
      by the test itself, or used by "make check-acceptance" to filter
      out tests.

[2] - This patch series doesn't attempt to change the default selection
      mechanism.  Possible changes in this area may include looking for
      any one built binary first, no matter the host architecture.

[3] - On a previous proposed version, the test class would look at the
      "arch" parameter given, and would cancel the test if there wasn't
      a match.

---

Changes from v2:
================

 * On "Acceptance tests: introduce arch parameter and attribute":
   - Made the documentation on the "arch" attribute and parameter
     clearer as to what the framework does with them, and what tests
     may use them for (Cornelia).

 * On "Acceptance tests: use "arch:" tag to filter target specific tests":
   - Bumped avocado-framework to the latest release (68.0, instead of
     67.0 on v2).

 * On "Boot Linux Console Test: add a test for mips64el + malta":
   - Broke down kernel source/doc URLs lines on docstring to make
     patchew happy about number of columns in line

 * On "Boot Linux Console Test: add a test for aarch64 + virt"
   - Broke down "kernel_command_line" assignment line to make patchew
     happy about number of columns in line

 * On "Boot Linux Console Test: add a test for arm + virt"
   - Broke down "kernel_command_line" assignment line to make patchew
     happy about number of columns in line

Open issues during v2:
======================

 * The timeout change to 90s may not be necessary, or the best
   idea, given that a possible tcg+ppc64 performance regression
   has been identified.
   - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00338.html

 * A possible race condition has been identified in the aarch64
   target, when running on an environment with more than 1 CPUs
   (initially reported by Wainer).
   - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00192.html

Changes from v1:
================

 * On "Acceptance tests: introduce arch parameter and attribute":
   - Added explicit *host system architecture* to the "arch" parameter
     behavior documentation (Caio / Philippe)
   - Added explicit arch parameter name in call to pick_default_qemu_bin()
     (Caio)
   - Fixed the documentation about the value of the "arch" attribute when
     a parameter is not given (Wainer).

 * On "Acceptance tests: use "arch:" tag to filter target specific tests":
   - Updated "arch" tag on tests "linux_initrd.py" and
     "virtio_version.py" (Cornelia)
   - Fixed the documentation about the value of the "arch" attribute (Wainer)

 * On "Acceptance tests: look for target architecture in test tags first"
   - Fixed the documentation given that starting with this patch, the
     "arch" attribute can also be depend on the "arch:" tag (based on
     previous patch point raised by Wainer).

 * New patch "Boot Linux Console Test: add common kernel command line
   options" (Philippe)

 * New patch "Boot Linux Console Test: increase timeout" (based on
   reports from Alex)

 * On "scripts/qemu.py: support adding a console with the default serial device":
   - Removed all non-essential console device type defintions from
     CONSOLE_DEV_TYPES.  At this point, setting the console will
     attempt to use "-serial" on all but the "s390-ccw-virtio*"
     machines (Philippe).
   - Updated documentation to make it clear that a specific device
     type will only be picked from CONSOLE_DEV_TYPES if the machine
     type is set, and a matching entry exists (Wainer).

 * On "Boot Linux Console Test: add a test for mips + malta":
   - Replaced "not easily reproducible kernel" for a kernel built and
     packaged by the Debian distro (Philippe, and previously Alex)
   - With the change to a kernel within a Debian package, and its use
     on another commit, added an utility function,
     "extract_from_deb()", to extract the kernel from the package
     (Philippe)
   - Switched to using common kernel command line options (Philippe)

 * On "Boot Linux Console Test: add a test for mips64el + malta":
   - Used the "extract_from_deb()" utility function introduced in the
     previous patch (Philippe)
   - Switched to using common kernel command line options (Philippe)
   - Fixed numbered references on the docstring (Wainer)
   - Fixed tag name on commit message (s/mips/mips64el/) (Wainer)

 * On "Boot Linux Console Test: add a test for ppc64 + pseries":
    - Switched to using common kernel command line options (Philippe)

 * On "Boot Linux Console Test: add a test for aarch64 + virt":
    - Switched to using common kernel command line options (Philippe)

 * On "Boot Linux Console Test: add a test for arm + virt"
     - Switched to using common kernel command line options (Philippe)

 * On "Boot Linux Console Test: add a test for s390x + s390-ccw-virtio":
     - Switched to using common kernel command line options (Philippe)

 * On "Boot Linux Console Test: add a test for alpha + clipper":
    - Switched to using common kernel command line options (Philippe)

---

Git Info:
  - URI: https://github.com/clebergnu/qemu/tree/sent/target_arch_v3
  - Remote: https://github.com/clebergnu/qemu
  - Branch: sent/target_arch_v3

Travis CI:
  - Build: https://travis-ci.org/clebergnu/qemu/builds/496229528
  - Relevant Job: https://travis-ci.org/clebergnu/qemu/jobs/496229549
  - Tests added here: https://travis-ci.org/clebergnu/qemu/jobs/496229549#L3085

Cleber Rosa (19):
  scripts/qemu.py: log QEMU launch command line
  Acceptance tests: show avocado test execution by default
  Acceptance tests: improve docstring on pick_default_qemu_bin()
  Acceptance tests: fix doc reference to avocado_qemu directory
  Acceptance tests: introduce arch parameter and attribute
  Acceptance tests: use "arch:" tag to filter target specific tests
  Acceptance tests: look for target architecture in test tags first
  Boot Linux Console Test: rename the x86_64 after the arch and machine
  Boot Linux Console Test: update the x86_64 kernel
  Boot Linux Console Test: add common kernel command line options
  Boot Linux Console Test: increase timeout
  Boot Linux Console Test: refactor the console watcher into utility
    method
  scripts/qemu.py: support adding a console with the default serial
    device
  Boot Linux Console Test: add a test for mips64el + malta
  Boot Linux Console Test: add a test for ppc64 + pseries
  Boot Linux Console Test: add a test for aarch64 + virt
  Boot Linux Console Test: add a test for arm + virt
  Boot Linux Console Test: add a test for s390x + s390-ccw-virtio
  Boot Linux Console Test: add a test for alpha + clipper

Philippe Mathieu-Daudé (1):
  Boot Linux Console Test: add a test for mips + malta

 .travis.yml                               |   4 +-
 docs/devel/testing.rst                    |  35 +++-
 scripts/qemu.py                           |  40 ++--
 tests/Makefile.include                    |   5 +-
 tests/acceptance/avocado_qemu/__init__.py |  23 ++-
 tests/acceptance/boot_linux_console.py    | 230 ++++++++++++++++++++--
 tests/acceptance/linux_initrd.py          |   2 +-
 tests/acceptance/virtio_version.py        |   2 +-
 tests/requirements.txt                    |   2 +-
 9 files changed, 294 insertions(+), 49 deletions(-)

-- 
2.20.1

[Qemu-devel] [PATCH v3 01/20] scripts/qemu.py: log QEMU launch command line

[Cleber Rosa]

Even when the launch of QEMU succeeds, it's useful to have the command
line recorded.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 scripts/qemu.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 32b00af5cc..ee85309923 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -320,6 +320,7 @@ class QEMUMachine(object):
         self._pre_launch()
         self._qemu_full_args = (self._wrapper + [self._binary] +
                                 self._base_args() + self._args)
+        LOG.debug('VM launch command: %r', ' '.join(self._qemu_full_args))
         self._popen = subprocess.Popen(self._qemu_full_args,
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
-- 
2.20.1

[Qemu-devel] [PATCH v3 02/20] Acceptance tests: show avocado test execution by default

[Cleber Rosa]

The current version of the "check-acceptance" target will only show
one line for execution of all tests.  That's probably OK if the tests
to be run are quick enough and they're always the same.

But, there's already one test alone that takes on average ~5 seconds
to run, we intend to adapt the list of tests to match the user's build
environment (among other choices).

Because of that, let's present the default Avocado UI by default.
Users can always choose a different output by setting the AVOCADO_SHOW
variable.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .travis.yml            | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index baa06b976a..42971484ab 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -199,7 +199,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
-        - TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
+        - TEST_CMD="make check-acceptance"
       addons:
         apt:
           packages:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index b39e989f72..93ea42553e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1089,7 +1089,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
-AVOCADO_SHOW=none
+AVOCADO_SHOW=app
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
-- 
2.20.1

[Qemu-devel] [PATCH v3 03/20] Acceptance tests: improve docstring on pick_default_qemu_bin()

[Cleber Rosa]

Making it clear what is returned by this utility function.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 1e54fd5932..d8d5b48dac 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -27,6 +27,10 @@ def pick_default_qemu_bin():
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
+
+    :returns: the path to the default QEMU binary or None if one could not
+              be found
+    :rtype: str or None
     """
     arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
-- 
2.20.1

[Qemu-devel] [PATCH v3 04/20] Acceptance tests: fix doc reference to avocado_qemu directory

[Cleber Rosa]

The "this directory" reference is misleading and confusing, it's a
leftover from when this text was proposed in a README file inside
the "tests/acceptance/avocado_qemu" directory.

When that text was moved to the top level docs directory, the
reference was not updated.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 docs/devel/testing.rst | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 135743a2bf..ceaaafc69f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -590,8 +590,9 @@ Alternatively, follow the instructions on this link:
 Overview
 --------
 
-This directory provides the ``avocado_qemu`` Python module, containing
-the ``avocado_qemu.Test`` class.  Here's a simple usage example:
+The ``tests/acceptance/avocado_qemu`` directory provides the
+``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
+class.  Here's a simple usage example:
 
 .. code::
 
-- 
2.20.1

[Qemu-devel] [PATCH v3 05/20] Acceptance tests: introduce arch parameter and attribute

[Cleber Rosa]

It's useful to define the architecture that should be used in
situations such as:
 * the intended target of the QEMU binary to be used on tests
 * the architecture of code to be run within the QEMU binary, such
   as a kernel image or a full blown guest OS image

This commit introduces both a test parameter and a test instance
attribute, that will contain such a value.

Now, when the "arch" test parameter is given, it will influence the
selection of the default QEMU binary, if one is not given explicitly
by means of the "qemu_img" parameter.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 docs/devel/testing.rst                    | 28 +++++++++++++++++++++++
 tests/acceptance/avocado_qemu/__init__.py | 14 +++++++++---
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index ceaaafc69f..6035db1b44 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -689,6 +689,21 @@ vm
 A QEMUMachine instance, initially configured according to the given
 ``qemu_bin`` parameter.
 
+arch
+~~~~
+
+The architecture can be used on different levels of the stack, e.g. by
+the framework or by the test itself.  At the framework level, it will
+will currently influence the selection of a QEMU binary (when one is
+not explicitly given).
+
+Tests are also free to use this attribute value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+The ``arch`` attribute will be set to the test parameter of the same
+name, and if one is not given explicitly, it will be set to ``None``.
+
 qemu_bin
 ~~~~~~~~
 
@@ -711,6 +726,19 @@ like the following:
 
   PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
 
+arch
+~~~~
+
+The architecture that will influence the selection of a QEMU binary
+(when one is not explicitly given).
+
+Tests are also free to use this parameter value, for their own needs.
+A test may, for instance, use the same value when selecting the
+architecture of a kernel or disk image to boot a VM with.
+
+This parameter has a direct relation with the ``arch`` attribute.  If
+not given, it will default to None.
+
 qemu_bin
 ~~~~~~~~
 
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index d8d5b48dac..f580582602 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -23,16 +23,22 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin():
+def pick_default_qemu_bin(arch=None):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
 
+    :param arch: the arch to use when looking for a QEMU binary (the target
+                 will match the arch given).  If None (the default) arch
+                 will be the current host system arch (as given by
+                 :func:`os.uname`).
+    :type arch: str
     :returns: the path to the default QEMU binary or None if one could not
               be found
     :rtype: str or None
     """
-    arch = os.uname()[4]
+    if arch is None:
+        arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
                                           "qemu-system-%s" % arch)
     if is_readable_executable_file(qemu_bin_relative_path):
@@ -47,8 +53,10 @@ def pick_default_qemu_bin():
 class Test(avocado.Test):
     def setUp(self):
         self.vm = None
+        self.arch = self.params.get('arch')
+        default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
-                                        default=pick_default_qemu_bin())
+                                        default=default_qemu_bin)
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
         self.vm = QEMUMachine(self.qemu_bin)
-- 
2.20.1

[Qemu-devel] [PATCH v3 06/20] Acceptance tests: use "arch:" tag to filter target specific tests

[Cleber Rosa]

Currently, the only test that contains some target architecture
information is "boot_linux_console.py" which test contains a "x86_64"
tag.  But that tag is not respected in the default execution, that is,
"make check-acceptance" doesn't do anything with it.

That said, even the target architecture handling currently present in
the "avocado_qemu.Test" class, class is pretty limited.  For instance,
by default, it chooses a target based on the host architecture.

Because the original implementation of the tags feature in Avocado did
not include any time of namespace or "key:val" mechanism, no tag has
relation to another tag.  The new implementation of the tags feature
from version 67.0 onwards, allows "key:val" tags, and because of that,
a test can be classified with a tag in a given key.  For instance, the
new proposed version of the "boot_linux_console.py" test, which
downloads and attempts to run a x86_64 kernel, is now tagged as:

  :avocado: tags=arch:x86_64

This means that it can be filtered (out) when no x86_64 target is
available.  At the same time, tests that don't have a "arch:" tag,
will not be filtered out.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include                 | 3 +++
 tests/acceptance/boot_linux_console.py | 2 +-
 tests/acceptance/linux_initrd.py       | 2 +-
 tests/acceptance/virtio_version.py     | 2 +-
 tests/requirements.txt                 | 2 +-
 5 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 93ea42553e..633992603d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1090,6 +1090,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
+AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
@@ -1115,6 +1116,8 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
+            $(AVOCADO_TAGS) \
             --failfast=on $(SRC_PATH)/tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 98324f7591..46b20bdfe2 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -19,7 +19,7 @@ class BootLinuxConsole(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index 737355c2ef..c75e29be70 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -19,7 +19,7 @@ class LinuxInitrd(Test):
     Checks QEMU evaluates correctly the initrd file passed as -initrd option.
 
     :avocado: enable
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
index ce990250d8..3b280e7fc3 100644
--- a/tests/acceptance/virtio_version.py
+++ b/tests/acceptance/virtio_version.py
@@ -62,7 +62,7 @@ class VirtioVersionCheck(Test):
     `disable-legacy`.
 
     :avocado: enable
-    :avocado: tags=x86_64
+    :avocado: tags=arch:x86_64
     """
 
     # just in case there are failures, show larger diff:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 64c6e27a94..002ded6a22 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,4 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==65.0
+avocado-framework==68.0
-- 
2.20.1

[Qemu-devel] [PATCH v3 07/20] Acceptance tests: look for target architecture in test tags first

[Cleber Rosa]

A test can, optionally, be tagged for one or many architectures.  If a
test has been tagged for a single architecture, there's a high chance
that the test won't run on other architectures.  This changes the
default order of choosing a default target architecture to use based
on the 'arch' tag value first.

The precedence order is for choosing a QEMU binary to use for a test
is now:

 * qemu_bin parameter
 * arch parameter
 * arch tag value (for example, x86_64 if ":avocado: tags=arch:x86_64
   is used)

This means that if one runs:

 $ avocado run -p qemu_bin=/usr/bin/qemu-system-x86_64 test.py

No arch parameter or tag will influence the selection of the QEMU
target binary.  If one runs:

 $ avocado run -p arch=ppc64 test.py

The target binary selection mechanism will attempt to find a binary
such as "ppc64-softmmu/qemu-system-ppc64".  And finally, if one runs
a test that is tagged (in its docstring) with "arch:aarch64":

 $ avocado run aarch64.py

The target binary selection mechanism will attempt to find a binary
such as "aarch64-softmmu/qemu-system-aarch64".

At this time, no provision is made to cancel the execution of tests if
the arch parameter given (manually) does not match the test "arch"
tag, but it may be a useful default behavior to be added in the
future.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst                    | 4 +++-
 tests/acceptance/avocado_qemu/__init__.py | 7 ++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 6035db1b44..87bcf8ef43 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -702,7 +702,9 @@ A test may, for instance, use the same value when selecting the
 architecture of a kernel or disk image to boot a VM with.
 
 The ``arch`` attribute will be set to the test parameter of the same
-name, and if one is not given explicitly, it will be set to ``None``.
+name.  If one is not given explicitly, it will either be set to
+``None``, or, if the test is tagged with one (and only one)
+``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
 
 qemu_bin
 ~~~~~~~~
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index f580582602..9e98d113cb 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -53,7 +53,12 @@ def pick_default_qemu_bin(arch=None):
 class Test(avocado.Test):
     def setUp(self):
         self.vm = None
-        self.arch = self.params.get('arch')
+        arches = self.tags.get('arch', [])
+        if len(arches) == 1:
+            arch = arches.pop()
+        else:
+            arch = None
+        self.arch = self.params.get('arch', default=arch)
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
-- 
2.20.1

[Qemu-devel] [PATCH v3 08/20] Boot Linux Console Test: rename the x86_64 after the arch and machine

[Cleber Rosa]

Given that the test is specific to x86_64 and pc, and new tests are
going to be added to the same class, let's rename it accordingly.
Also, let's make the class documentation not architecture specific.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 46b20bdfe2..89df7f6e4f 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -15,16 +15,19 @@ from avocado_qemu import Test
 
 class BootLinuxConsole(Test):
     """
-    Boots a x86_64 Linux kernel and checks that the console is operational
-    and the kernel command line is properly passed from QEMU to the kernel
+    Boots a Linux kernel and checks that the console is operational and the
+    kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
 
-    def test(self):
+    def test_x86_64_pc(self):
+        """
+        :avocado: tags=arch:x86_64
+        :avocado: tags=machine:pc
+        """
         kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
         kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
-- 
2.20.1

[Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Cleber Rosa]

To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
added using the 29 kernel, so for consistency, let's also update it
here.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 89df7f6e4f..35b31162d4 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
         :avocado: tags=arch:x86_64
         :avocado: tags=machine:pc
         """
-        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
+        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
-        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
+        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_machine('pc')
-- 
2.20.1

[Qemu-devel] [PATCH v3 10/20] Boot Linux Console Test: add common kernel command line options

[Cleber Rosa]

The 'printk.time=0' option makes it easier to parse the console
output.  Let's set it as a default, and reusable, kernel command line
options for this and future similar tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 35b31162d4..cc5dcd7373 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -23,6 +23,8 @@ class BootLinuxConsole(Test):
 
     timeout = 60
 
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -35,7 +37,7 @@ class BootLinuxConsole(Test):
 
         self.vm.set_machine('pc')
         self.vm.set_console()
-        kernel_command_line = 'console=ttyS0'
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-- 
2.20.1

[Qemu-devel] [PATCH v3 11/20] Boot Linux Console Test: increase timeout

[Cleber Rosa]

When running on very low powered environments, some tests may time out
causing false negatives.  As a conservative change, and for
considering that human time (investigating false negatives) is worth
more than some extra machine cycles (and time), let's increase the
overall timeout.

CC: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index cc5dcd7373..fa721a7355 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -21,7 +21,7 @@ class BootLinuxConsole(Test):
     :avocado: enable
     """
 
-    timeout = 60
+    timeout = 90
 
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
-- 
2.20.1

[Qemu-devel] [PATCH v3 12/20] Boot Linux Console Test: refactor the console watcher into utility method

[Cleber Rosa]

This introduces a utility method that monitors the console device and
looks for either a message that signals the test success or failure.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 30 ++++++++++++++++++--------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index fa721a7355..e2ef43e7ce 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -25,6 +25,25 @@ class BootLinuxConsole(Test):
 
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
+    def wait_for_console_pattern(self, success_message,
+                                 failure_message='Kernel panic - not syncing'):
+        """
+        Waits for messages to appear on the console, while logging the content
+
+        :param success_message: if this message appears, test succeeds
+        :param failure_message: if this message appears, test fails
+        """
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline()
+            console_logger.debug(msg.strip())
+            if success_message in msg:
+                break
+            if failure_message in msg:
+                fail = 'Failure message found in console: %s' % failure_message
+                self.fail(fail)
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -41,12 +60,5 @@ class BootLinuxConsole(Test):
         self.vm.add_args('-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-        console = self.vm.console_socket.makefile()
-        console_logger = logging.getLogger('console')
-        while True:
-            msg = console.readline()
-            console_logger.debug(msg.strip())
-            if 'Kernel command line: %s' % kernel_command_line in msg:
-                break
-            if 'Kernel panic - not syncing' in msg:
-                self.fail("Kernel panic reached")
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 13/20] scripts/qemu.py: support adding a console with the default serial device

[Cleber Rosa]

The set_console() utility function traditionally adds a device either
based on the explicitly given device type, or based on the machine type,
a known good type of device.

But, for a number of machine types, it may be impossible or
inconvenient to add the devices my means of "-device" command line
options, and then it may better to just use the "-serial" option and
let QEMU itself, based on the machine type, set the device
accordingly.

To achieve that, the behavior of set_console() now flags the intention
to add a console device on launch(), and if no explicit device type is
given, and there's no definition on CONSOLE_DEV_TYPES, the "-serial"
is going to be added to the QEMU command line, instead of raising
exceptions.

Based on testing with different machine types, the CONSOLE_DEV_TYPES
is now being set to the bare essential entries (one entry to be
honest), for machine types that can not easily give us a working
console with "-serial".

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qemu.py | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index ee85309923..bd1d2e2b9a 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -42,11 +42,6 @@ def kvm_available(target_arch=None):
 
 #: Maps machine types to the preferred console device types
 CONSOLE_DEV_TYPES = {
-    r'^clipper$': 'isa-serial',
-    r'^malta': 'isa-serial',
-    r'^(pc.*|q35.*|isapc)$': 'isa-serial',
-    r'^(40p|powernv|prep)$': 'isa-serial',
-    r'^pseries.*': 'spapr-vty',
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
@@ -129,6 +124,7 @@ class QEMUMachine(object):
         self._temp_dir = None
         self._launched = False
         self._machine = None
+        self._console_set = False
         self._console_device_type = None
         self._console_address = None
         self._console_socket = None
@@ -248,13 +244,17 @@ class QEMUMachine(object):
                 '-display', 'none', '-vga', 'none']
         if self._machine is not None:
             args.extend(['-machine', self._machine])
-        if self._console_device_type is not None:
+        if self._console_set:
             self._console_address = os.path.join(self._temp_dir,
                                                  self._name + "-console.sock")
             chardev = ('socket,id=console,path=%s,server,nowait' %
                        self._console_address)
-            device = '%s,chardev=console' % self._console_device_type
-            args.extend(['-chardev', chardev, '-device', device])
+            args.extend(['-chardev', chardev])
+            if self._console_device_type is None:
+                args.extend(['-serial', 'chardev:console'])
+            else:
+                device = '%s,chardev=console' % self._console_device_type
+                args.extend(['-device', device])
         return args
 
     def _pre_launch(self):
@@ -480,30 +480,29 @@ class QEMUMachine(object):
         line.
 
         This is a convenience method that will either use the provided
-        device type, of if not given, it will used the device type set
-        on CONSOLE_DEV_TYPES.
+        device type, of if not given, it will use the device type set
+        on CONSOLE_DEV_TYPES if a machine type is set, and a matching
+        entry exists on CONSOLE_DEV_TYPES.
 
         The actual setting of command line arguments will be be done at
         machine launch time, as it depends on the temporary directory
         to be created.
 
-        @param device_type: the device type, such as "isa-serial"
+        @param device_type: the device type, such as "isa-serial".  If
+                            None is given (the default value) a "-serial
+                            chardev:console" command line argument will
+                            be used instead, resorting to the machine's
+                            default device type, if a machine type is set,
+                            and a matching entry exists on CONSOLE_DEV_TYPES.
         @raises: QEMUMachineAddDeviceError if the device type is not given
                  and can not be determined.
         """
-        if device_type is None:
-            if self._machine is None:
-                raise QEMUMachineAddDeviceError("Can not add a console device:"
-                                                " QEMU instance without a "
-                                                "defined machine type")
+        self._console_set = True
+        if device_type is None and self._machine is not None:
             for regex, device in CONSOLE_DEV_TYPES.items():
                 if re.match(regex, self._machine):
                     device_type = device
                     break
-            if device_type is None:
-                raise QEMUMachineAddDeviceError("Can not add a console device:"
-                                                " no matching console device "
-                                                "type definition")
         self._console_device_type = device_type
 
     @property
-- 
2.20.1

[Qemu-devel] [PATCH v3 14/20] Boot Linux Console Test: add a test for mips + malta

[Cleber Rosa]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.  Also, it relies on the serial
device set by the machine itself.

If mips is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:mips tests/acceptance
    $ avocado run -t machine:malta tests/acceptance
    $ avocado run -t endian:big tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 41 ++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 42971484ab..0a5e0613be 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index e2ef43e7ce..05e43360b8 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -8,9 +8,12 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import os
 import logging
 
 from avocado_qemu import Test
+from avocado.utils import process
+from avocado.utils import archive
 
 
 class BootLinuxConsole(Test):
@@ -44,6 +47,21 @@ class BootLinuxConsole(Test):
                 fail = 'Failure message found in console: %s' % failure_message
                 self.fail(fail)
 
+    def extract_from_deb(self, deb, path):
+        """
+        Extracts a file from a deb package into the test workdir
+
+        :param deb: path to the deb archive
+        :param file: path within the deb archive of the file to be extracted
+        :returns: path of the extracted file
+        """
+        cwd = os.getcwd()
+        os.chdir(self.workdir)
+        process.run("ar x %s data.tar.gz" % deb)
+        archive.extract("data.tar.gz", self.workdir)
+        os.chdir(cwd)
+        return self.workdir + path
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -62,3 +80,26 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_mips_malta(self):
+        """
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20130217T032700Z/pool/main/l/linux-2.6/'
+                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
+        deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-2.6.32-5-4kc-malta')
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 15/20] Boot Linux Console Test: add a test for mips64el + malta

[Cleber Rosa]

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.

If mips64el is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips64el"
tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:mips64el tests/acceptance
    $ avocado run -t machine:malta tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 0a5e0613be..0260263bb8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 05e43360b8..899c27a9ec 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -10,6 +10,7 @@
 
 import os
 import logging
+import os
 
 from avocado_qemu import Test
 from avocado.utils import process
@@ -103,3 +104,36 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_mips64el_malta(self):
+        """
+        This test requires the ar tool to extract "data.tar.gz" from
+        the Debian package.
+
+        The kernel can be rebuilt using this Debian kernel source [1] and
+        following the instructions on [2].
+
+        [1] http://snapshot.debian.org/package/linux-2.6/2.6.32-48/
+            #linux-source-2.6.32_2.6.32-48
+        [2] https://kernel-team.pages.debian.net/kernel-handbook/
+            ch-common-tasks.html#s-common-official
+
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:malta
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20130217T032700Z/pool/main/l/linux-2.6/'
+                   'linux-image-2.6.32-5-5kc-malta_2.6.32-48_mipsel.deb')
+        deb_hash = '1aaec92083bf22fda31e0d27fa8d9a388e5fc3d5'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-2.6.32-5-5kc-malta')
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 16/20] Boot Linux Console Test: add a test for ppc64 + pseries

[Cleber Rosa]

Just like the previous tests, boots a Linux kernel on a ppc64 target
using the pseries machine.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 0260263bb8..b5abe130f1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 899c27a9ec..6bc9a6b303 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -137,3 +137,22 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_ppc64_pseries(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        """
+        kernel_url = ('http://mirrors.rit.edu/fedora/fedora-secondary/'
+                      'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
+        kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('pseries')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 17/20] Boot Linux Console Test: add a test for aarch64 + virt

[Cleber Rosa]

Just like the previous tests, boots a Linux kernel on a aarch64 target
using the virt machine.

One special option added is the CPU type, given that the kernel
selected fails to boot on the virt machine's default CPU (cortex-a15).

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index b5abe130f1..60a4bc00b8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 6bc9a6b303..1f2dfa3654 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -156,3 +156,24 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_aarch64_virt(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://sjc.edge.kernel.org/fedora-buffet/fedora/linux/'
+                      'releases/29/Server/aarch64/os/images/pxeboot/vmlinuz')
+        kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('virt')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-cpu', 'cortex-a53',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 18/20] Boot Linux Console Test: add a test for arm + virt

[Cleber Rosa]

Just like the previous tests, boots a Linux kernel on an arm target
using the virt machine.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 60a4bc00b8..f34bd8dc2b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 1f2dfa3654..311f6fbb96 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -177,3 +177,23 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_arm_virt(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://sjc.edge.kernel.org/fedora-buffet/fedora/linux/'
+                      'releases/29/Server/armhfp/os/images/pxeboot/vmlinuz')
+        kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('virt')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 19/20] Boot Linux Console Test: add a test for s390x + s390-ccw-virtio

[Cleber Rosa]

Just like the previous tests, boots a Linux kernel on a s390x target
using the s390-ccw-virtio machine.

Because it's not possible to have multiple VT220 consoles,
'-nodefaults' is used, so that the one set with set_console() works
correctly.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index f34bd8dc2b..82d680c437 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 311f6fbb96..aa581aa7de 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -197,3 +197,23 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_s390x_s390_ccw_virtio(self):
+        """
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390_ccw_virtio
+        """
+        kernel_url = ('http://mirrors.rit.edu/fedora/fedora-secondary/releases'
+                      '/29/Server/s390x/os/images/kernel.img')
+        kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('s390-ccw-virtio')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
+        self.vm.add_args('-nodefaults',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

[Qemu-devel] [PATCH v3 20/20] Boot Linux Console Test: add a test for alpha + clipper

[Cleber Rosa]

Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
board and verify the serial is working.  One extra command added to
the QEMU command line is '-vga std', because the kernel used is
known to crash without it.

If alpha is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:alpha" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:alpha tests/acceptance
    $ avocado run -t machine:clipper tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 82d680c437..ea63cac9e9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -198,7 +198,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
         - TEST_CMD="make check-acceptance"
       addons:
         apt:
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index aa581aa7de..d866886067 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -217,3 +217,25 @@ class BootLinuxConsole(Test):
         self.vm.launch()
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
+
+    def test_alpha_clipper(self):
+        """
+        :avocado: tags=arch:alpha
+        :avocado: tags=machine:clipper
+        """
+        kernel_url = ('http://archive.debian.org/debian/dists/lenny/main/'
+                      'installer-alpha/current/images/cdrom/vmlinuz')
+        kernel_hash = '3a943149335529e2ed3e74d0d787b85fb5671ba3'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        uncompressed_kernel = archive.uncompress(kernel_path, self.workdir)
+
+        self.vm.set_machine('clipper')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        self.vm.add_args('-vga', 'std',
+                         '-kernel', uncompressed_kernel,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
-- 
2.20.1

Re: [Qemu-devel] [PATCH v3 15/20] Boot Linux Console Test: add a test for mips64el + malta

[Aleksandar Markovic]

> From: Cleber Rosa <crosa@redhat.com>
> Subject: [PATCH v3 15/20] Boot Linux Console Test: add a test for mips64el + malta
> 
> Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
> board and verify the serial is working.
> 

Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>

Re: [Qemu-devel] [PATCH v3 14/20] Boot Linux Console Test: add a test for mips + malta

[Aleksandar Markovic]

> From: Cleber Rosa <crosa@redhat.com>
> Subject: [PATCH v3 14/20] Boot Linux Console Test: add a test for mips + malta
> 
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
> board and verify the serial is working.  Also, it relies on the serial
> device set by the machine itself.

Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>

Re: [Qemu-devel] [PATCH v3 01/20] scripts/qemu.py: log QEMU launch command line

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:34 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> Even when the launch of QEMU succeeds, it's useful to have the command
> line recorded.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  scripts/qemu.py | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 02/20] Acceptance tests: show avocado test execution by default

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:35 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> The current version of the "check-acceptance" target will only show
> one line for execution of all tests.  That's probably OK if the tests
> to be run are quick enough and they're always the same.
> 
> But, there's already one test alone that takes on average ~5 seconds
> to run, we intend to adapt the list of tests to match the user's build
> environment (among other choices).
> 
> Because of that, let's present the default Avocado UI by default.
> Users can always choose a different output by setting the AVOCADO_SHOW
> variable.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .travis.yml            | 2 +-
>  tests/Makefile.include | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 03/20] Acceptance tests: improve docstring on pick_default_qemu_bin()

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:36 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> Making it clear what is returned by this utility function.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 4 ++++
>  1 file changed, 4 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 04/20] Acceptance tests: fix doc reference to avocado_qemu directory

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:37 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> The "this directory" reference is misleading and confusing, it's a
> leftover from when this text was proposed in a README file inside
> the "tests/acceptance/avocado_qemu" directory.
> 
> When that text was moved to the top level docs directory, the
> reference was not updated.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  docs/devel/testing.rst | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 05/20] Acceptance tests: introduce arch parameter and attribute

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:38 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> It's useful to define the architecture that should be used in
> situations such as:
>  * the intended target of the QEMU binary to be used on tests
>  * the architecture of code to be run within the QEMU binary, such
>    as a kernel image or a full blown guest OS image
> 
> This commit introduces both a test parameter and a test instance
> attribute, that will contain such a value.
> 
> Now, when the "arch" test parameter is given, it will influence the
> selection of the default QEMU binary, if one is not given explicitly
> by means of the "qemu_img" parameter.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  docs/devel/testing.rst                    | 28 +++++++++++++++++++++++
>  tests/acceptance/avocado_qemu/__init__.py | 14 +++++++++---
>  2 files changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index ceaaafc69f..6035db1b44 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -689,6 +689,21 @@ vm
>  A QEMUMachine instance, initially configured according to the given
>  ``qemu_bin`` parameter.
>  
> +arch
> +~~~~
> +
> +The architecture can be used on different levels of the stack, e.g. by
> +the framework or by the test itself.  At the framework level, it will
> +will currently influence the selection of a QEMU binary (when one is

s/will will/will/

> +not explicitly given).
> +
> +Tests are also free to use this attribute value, for their own needs.
> +A test may, for instance, use the same value when selecting the
> +architecture of a kernel or disk image to boot a VM with.
> +
> +The ``arch`` attribute will be set to the test parameter of the same
> +name, and if one is not given explicitly, it will be set to ``None``.
> +
>  qemu_bin
>  ~~~~~~~~
>  
> @@ -711,6 +726,19 @@ like the following:
>  
>    PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
>  
> +arch
> +~~~~
> +
> +The architecture that will influence the selection of a QEMU binary
> +(when one is not explicitly given).
> +
> +Tests are also free to use this parameter value, for their own needs.
> +A test may, for instance, use the same value when selecting the
> +architecture of a kernel or disk image to boot a VM with.
> +
> +This parameter has a direct relation with the ``arch`` attribute.  If
> +not given, it will default to None.
> +
>  qemu_bin
>  ~~~~~~~~
>  
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index d8d5b48dac..f580582602 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -23,16 +23,22 @@ def is_readable_executable_file(path):
>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>  
>  
> -def pick_default_qemu_bin():
> +def pick_default_qemu_bin(arch=None):
>      """
>      Picks the path of a QEMU binary, starting either in the current working
>      directory or in the source tree root directory.
>  
> +    :param arch: the arch to use when looking for a QEMU binary (the target
> +                 will match the arch given).  If None (the default) arch

s/None (the default)/None (the default),/ makes that sentence less
confusing :)

> +                 will be the current host system arch (as given by
> +                 :func:`os.uname`).
> +    :type arch: str
>      :returns: the path to the default QEMU binary or None if one could not
>                be found
>      :rtype: str or None
>      """
> -    arch = os.uname()[4]
> +    if arch is None:
> +        arch = os.uname()[4]
>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>                                            "qemu-system-%s" % arch)
>      if is_readable_executable_file(qemu_bin_relative_path):
> @@ -47,8 +53,10 @@ def pick_default_qemu_bin():
>  class Test(avocado.Test):
>      def setUp(self):
>          self.vm = None
> +        self.arch = self.params.get('arch')
> +        default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
>          self.qemu_bin = self.params.get('qemu_bin',
> -                                        default=pick_default_qemu_bin())
> +                                        default=default_qemu_bin)
>          if self.qemu_bin is None:
>              self.cancel("No QEMU binary defined or found in the source tree")
>          self.vm = QEMUMachine(self.qemu_bin)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 06/20] Acceptance tests: use "arch:" tag to filter target specific tests

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:39 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> Currently, the only test that contains some target architecture
> information is "boot_linux_console.py" which test contains a "x86_64"

But there are two others changed by you here, aren't there?

> tag.  But that tag is not respected in the default execution, that is,
> "make check-acceptance" doesn't do anything with it.
> 
> That said, even the target architecture handling currently present in
> the "avocado_qemu.Test" class, class is pretty limited.  For instance,

s/class, class/class/

> by default, it chooses a target based on the host architecture.
> 
> Because the original implementation of the tags feature in Avocado did
> not include any time of namespace or "key:val" mechanism, no tag has
> relation to another tag.  The new implementation of the tags feature
> from version 67.0 onwards, allows "key:val" tags, and because of that,
> a test can be classified with a tag in a given key.  For instance, the
> new proposed version of the "boot_linux_console.py" test, which
> downloads and attempts to run a x86_64 kernel, is now tagged as:
> 
>   :avocado: tags=arch:x86_64
> 
> This means that it can be filtered (out) when no x86_64 target is
> available.  At the same time, tests that don't have a "arch:" tag,
> will not be filtered out.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include                 | 3 +++
>  tests/acceptance/boot_linux_console.py | 2 +-
>  tests/acceptance/linux_initrd.py       | 2 +-
>  tests/acceptance/virtio_version.py     | 2 +-
>  tests/requirements.txt                 | 2 +-
>  5 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 93ea42553e..633992603d 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -1090,6 +1090,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # Any number of command separated loggers are accepted.  For more
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=app
> +AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
>  
>  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> @@ -1115,6 +1116,8 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>  	$(call quiet-command, \
>              $(TESTS_VENV_DIR)/bin/python -m avocado \
>              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> +            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> +            $(AVOCADO_TAGS) \
>              --failfast=on $(SRC_PATH)/tests/acceptance, \
>              "AVOCADO", "tests/acceptance")
>  
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 98324f7591..46b20bdfe2 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -19,7 +19,7 @@ class BootLinuxConsole(Test):
>      and the kernel command line is properly passed from QEMU to the kernel
>  
>      :avocado: enable
> -    :avocado: tags=x86_64
> +    :avocado: tags=arch:x86_64
>      """
>  
>      timeout = 60
> diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> index 737355c2ef..c75e29be70 100644
> --- a/tests/acceptance/linux_initrd.py
> +++ b/tests/acceptance/linux_initrd.py
> @@ -19,7 +19,7 @@ class LinuxInitrd(Test):
>      Checks QEMU evaluates correctly the initrd file passed as -initrd option.
>  
>      :avocado: enable
> -    :avocado: tags=x86_64
> +    :avocado: tags=arch:x86_64
>      """
>  
>      timeout = 60
> diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
> index ce990250d8..3b280e7fc3 100644
> --- a/tests/acceptance/virtio_version.py
> +++ b/tests/acceptance/virtio_version.py
> @@ -62,7 +62,7 @@ class VirtioVersionCheck(Test):
>      `disable-legacy`.
>  
>      :avocado: enable
> -    :avocado: tags=x86_64
> +    :avocado: tags=arch:x86_64
>      """
>  
>      # just in case there are failures, show larger diff:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 64c6e27a94..002ded6a22 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,4 +1,4 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -avocado-framework==65.0
> +avocado-framework==68.0

I think you should explain why you bump the required version to 68.0
instead of the 67.0 you mention above.

Re: [Qemu-devel] [PATCH v3 07/20] Acceptance tests: look for target architecture in test tags first

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:40 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> A test can, optionally, be tagged for one or many architectures.  If a
> test has been tagged for a single architecture, there's a high chance
> that the test won't run on other architectures.  This changes the
> default order of choosing a default target architecture to use based
> on the 'arch' tag value first.
> 
> The precedence order is for choosing a QEMU binary to use for a test
> is now:
> 
>  * qemu_bin parameter
>  * arch parameter
>  * arch tag value (for example, x86_64 if ":avocado: tags=arch:x86_64
>    is used)
> 
> This means that if one runs:
> 
>  $ avocado run -p qemu_bin=/usr/bin/qemu-system-x86_64 test.py
> 
> No arch parameter or tag will influence the selection of the QEMU
> target binary.  If one runs:
> 
>  $ avocado run -p arch=ppc64 test.py
> 
> The target binary selection mechanism will attempt to find a binary
> such as "ppc64-softmmu/qemu-system-ppc64".  And finally, if one runs
> a test that is tagged (in its docstring) with "arch:aarch64":
> 
>  $ avocado run aarch64.py
> 
> The target binary selection mechanism will attempt to find a binary
> such as "aarch64-softmmu/qemu-system-aarch64".
> 
> At this time, no provision is made to cancel the execution of tests if
> the arch parameter given (manually) does not match the test "arch"
> tag, but it may be a useful default behavior to be added in the
> future.

Would it be useful as well to cancel a test if the passed-in qemu_bin
parameter clashed with the tagged architecture?

> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/testing.rst                    | 4 +++-
>  tests/acceptance/avocado_qemu/__init__.py | 7 ++++++-
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 6035db1b44..87bcf8ef43 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -702,7 +702,9 @@ A test may, for instance, use the same value when selecting the
>  architecture of a kernel or disk image to boot a VM with.
>  
>  The ``arch`` attribute will be set to the test parameter of the same
> -name, and if one is not given explicitly, it will be set to ``None``.
> +name.  If one is not given explicitly, it will either be set to
> +``None``, or, if the test is tagged with one (and only one)
> +``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
>  
>  qemu_bin
>  ~~~~~~~~
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index f580582602..9e98d113cb 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -53,7 +53,12 @@ def pick_default_qemu_bin(arch=None):
>  class Test(avocado.Test):
>      def setUp(self):
>          self.vm = None
> -        self.arch = self.params.get('arch')
> +        arches = self.tags.get('arch', [])
> +        if len(arches) == 1:
> +            arch = arches.pop()
> +        else:
> +            arch = None
> +        self.arch = self.params.get('arch', default=arch)
>          default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
>          self.qemu_bin = self.params.get('qemu_bin',
>                                          default=default_qemu_bin)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 08/20] Boot Linux Console Test: rename the x86_64 after the arch and machine

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:41 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> Given that the test is specific to x86_64 and pc, and new tests are
> going to be added to the same class, let's rename it accordingly.
> Also, let's make the class documentation not architecture specific.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:42 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> added using the 29 kernel, so for consistency, let's also update it
> here.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 89df7f6e4f..35b31162d4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
>          :avocado: tags=arch:x86_64
>          :avocado: tags=machine:pc
>          """
> -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
>                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>  
>          self.vm.set_machine('pc')

This looks reasonable; but an obvious follow-up question is how we
handle updates to new distro levels in general.

I don't think we'll need to run acceptance tests on old qemus, though.

Re: [Qemu-devel] [PATCH v3 10/20] Boot Linux Console Test: add common kernel command line options

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:43 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> The 'printk.time=0' option makes it easier to parse the console
> output.  Let's set it as a default, and reusable, kernel command line
> options for this and future similar tests.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 11/20] Boot Linux Console Test: increase timeout

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:44 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> When running on very low powered environments, some tests may time out
> causing false negatives.  As a conservative change, and for
> considering that human time (investigating false negatives) is worth
> more than some extra machine cycles (and time), let's increase the
> overall timeout.
> 
> CC: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index cc5dcd7373..fa721a7355 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -21,7 +21,7 @@ class BootLinuxConsole(Test):
>      :avocado: enable
>      """
>  
> -    timeout = 60
> +    timeout = 90
>  
>      KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
>  

Not sure what a good value is, but it is easy to tweak later if needed.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 12/20] Boot Linux Console Test: refactor the console watcher into utility method

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:45 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> This introduces a utility method that monitors the console device and
> looks for either a message that signals the test success or failure.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 30 ++++++++++++++++++--------
>  1 file changed, 21 insertions(+), 9 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 13/20] scripts/qemu.py: support adding a console with the default serial device

[Cornelia Huck]

On Wed, 20 Feb 2019 19:57:46 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> The set_console() utility function traditionally adds a device either
> based on the explicitly given device type, or based on the machine type,
> a known good type of device.

Hm, I find this sentence hard to parse... maybe it should be "either
adds a device..." and "adds a known good type of device"?

> 
> But, for a number of machine types, it may be impossible or
> inconvenient to add the devices my means of "-device" command line

s/my/by/

> options, and then it may better to just use the "-serial" option and
> let QEMU itself, based on the machine type, set the device
> accordingly.
> 
> To achieve that, the behavior of set_console() now flags the intention
> to add a console device on launch(), and if no explicit device type is
> given, and there's no definition on CONSOLE_DEV_TYPES, the "-serial"
> is going to be added to the QEMU command line, instead of raising
> exceptions.
> 
> Based on testing with different machine types, the CONSOLE_DEV_TYPES
> is now being set to the bare essential entries (one entry to be
> honest), for machine types that can not easily give us a working
> console with "-serial".
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qemu.py | 39 +++++++++++++++++++--------------------
>  1 file changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index ee85309923..bd1d2e2b9a 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -42,11 +42,6 @@ def kvm_available(target_arch=None):
>  
>  #: Maps machine types to the preferred console device types
>  CONSOLE_DEV_TYPES = {
> -    r'^clipper$': 'isa-serial',
> -    r'^malta': 'isa-serial',
> -    r'^(pc.*|q35.*|isapc)$': 'isa-serial',
> -    r'^(40p|powernv|prep)$': 'isa-serial',
> -    r'^pseries.*': 'spapr-vty',
>      r'^s390-ccw-virtio.*': 'sclpconsole',

FWIW, s390x has supported the '-serial' parameter since 3.0 (commit
052888f043bacb18231046045b1f9cd946703170). Maybe you can drop this now?
If not, what error are you getting?

Re: [Qemu-devel] [PATCH v3 18/20] Boot Linux Console Test: add a test for arm + virt

[Philippe Mathieu-Daudé]

Hi Cleber,

On 2/21/19 1:57 AM, Cleber Rosa wrote:
> Just like the previous tests, boots a Linux kernel on an arm target
> using the virt machine.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> ---
>  .travis.yml                            |  2 +-
>  tests/acceptance/boot_linux_console.py | 20 ++++++++++++++++++++
>  2 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 60a4bc00b8..f34bd8dc2b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -198,7 +198,7 @@ matrix:
>  
>      # Acceptance (Functional) tests
>      - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu"
> +        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu,aarch64-softmmu,arm-softmmu"
>          - TEST_CMD="make check-acceptance"
>        addons:
>          apt:
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 1f2dfa3654..311f6fbb96 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -177,3 +177,23 @@ class BootLinuxConsole(Test):
>          self.vm.launch()
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
> +
> +    def test_arm_virt(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:virt
> +        """
> +        kernel_url = ('https://sjc.edge.kernel.org/fedora-buffet/fedora/linux/'
> +                      'releases/29/Server/armhfp/os/images/pxeboot/vmlinuz')
> +        kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> +        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> +
> +        self.vm.set_machine('virt')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> +                               'console=ttyAMA0')
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> 

This one sometime hangs:

$ make check-acceptance AVOCADO_SHOW=app,console
  AVOCADO tests/acceptance
JOB ID     : 9e363f10363c90882716d6be254f07eb5f355172
JOB LOG    :
/source/qemu/build/gcc/tests/results/job-2019-03-02T21.00-9e363f1/job.log
...
 (06/16)
/source/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt:
/console: Booting Linux on physical CPU 0x0
console: Linux version 4.18.16-300.fc29.armv7hl
(mockbuild@buildvm-armv7-06.arm.fedoraproject.org) (gcc version 8.2.1
20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Sun Oct 21 00:56:28 UTC 2018
console: CPU: ARMv7 Processor [412fc0f1] revision 1 (ARMv7), cr=10c5387d
console: CPU: div instructions available: patching division code
console: CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
console: OF: fdt: Machine model: linux,dummy-virt
console: Memory policy: Data cache writealloc
console: efi: Getting EFI parameters from FDT:
console: efi: UEFI not found.
console: cma: Failed to reserve 64 MiB
console: psci: probing for conduit method from DT.
console: psci: PSCIv0.2 detected in firmware.
console: psci: Using standard PSCI v0.2 function IDs
console: psci: Trusted OS migration not required
console: percpu: Embedded 16 pages/cpu @(ptrval) s34956 r8192 d22388 u65536
console: Built 1 zonelists, mobility grouping on.  Total pages: 32480
console: Kernel command line: printk.time=0 console=ttyAMA0
INTERRUPTED: Runner error occurred: Timeout reached\nOriginal status:
PASS\n{'name':
'06-/source/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt',
'logdir':
'/source/qemu/build/gcc/tests/results/job-2019-03-02T21.00-df1a431...
(90.60 s)
...
RESULTS    : PASS 9 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 1 |
CANCEL 6
JOB TIME   : 105.35 s
make: *** [/source/qemu/tests/Makefile.include:1116: check-acceptance]
Error 8

$ uname -rom
4.20.11-200.fc29.x86_64 x86_64 GNU/Linux

The string is here, so I wonder if something is wrong in
wait_for_console_pattern().

The debug log isn't helpful:

2019-03-02 21:10:27,062 test             L0602 INFO | START
06-/source/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt
2019-03-02 21:10:27,062 test             L0298 DEBUG| DATA
(filename=output.expected) => NOT FOUND (data sources: variant, test, file)
2019-03-02 21:10:27,063 parameters       L0146 DEBUG| PARAMS (key=arch,
path=*, default=arm) => 'arm'
2019-03-02 21:10:27,063 parameters       L0146 DEBUG| PARAMS
(key=qemu_bin, path=*, default=arm-softmmu/qemu-system-arm) =>
'arm-softmmu/qemu-system-arm'
2019-03-02 21:10:27,070 qemu             L0323 DEBUG| VM launch command:
'arm-softmmu/qemu-system-arm -chardev
socket,id=mon,path=/var/tmp/tmpv_rh95rv/qemu-25227-monitor.sock -mon
chardev=mon,mode=control -display none -vga none -machine virt -chardev
socket,id=console,path=/var/tmp/tmpv_rh95rv/qemu-25227-console.sock,server,nowait
-serial chardev:console -kernel
/avocado/data/cache/by_location/096923daa6943c0c1287e76b52cdd56a882fef8a/vmlinuz
-append printk.time=0 console=ttyAMA0'
2019-03-02 21:10:27,157 qmp              L0167 DEBUG| >>> {'execute':
'qmp_capabilities'}
2019-03-02 21:10:27,176 qmp              L0175 DEBUG| <<< {'return': {}}
2019-03-02 21:10:28,446 qmp              L0167 DEBUG| >>> {'execute':
'quit'}
2019-03-02 21:11:57,654 qemu             L0363 WARNI| qemu received
signal 9: arm-softmmu/qemu-system-arm -chardev
socket,id=mon,path=/var/tmp/tmpv_rh95rv/qemu-25227-monitor.sock -mon
chardev=mon,mode=control -display none -vga none -machine virt -chardev
socket,id=console,path=/var/tmp/tmpv_rh95rv/qemu-25227-console.sock,server,nowait
-serial chardev:console -kernel
/avocado/data/cache/by_location/096923daa6943c0c1287e76b52cdd56a882fef8a/vmlinuz
-append printk.time=0 console=ttyAMA0
2019-03-02 21:11:57,655 test             L0298 DEBUG| DATA
(filename=output.expected) => NOT FOUND (data sources: variant, test, file)
2019-03-02 21:11:57,655 test             L0298 DEBUG| DATA
(filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
2019-03-02 21:11:57,655 test             L0298 DEBUG| DATA
(filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
2019-03-02 21:11:57,682 sysinfo          L0355 DEBUG| Not logging
/var/log/messages (lack of permissions)
2019-03-02 21:11:57,682 test             L1013 INFO | PASS
06-/source/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt
2019-03-02 21:11:57,682 test             L0996 INFO |
2019-03-02 21:11:57,690 runner           L0061 ERROR| Runner error
occurred: Timeout reached

Regards,

Phil.

Re: [Qemu-devel] [PATCH v3 15/20] Boot Linux Console Test: add a test for mips64el + malta

[Philippe Mathieu-Daudé]

On 2/21/19 1:57 AM, Cleber Rosa wrote:
> Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
> board and verify the serial is working.
> 
> If mips64el is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:mips64el"
> tags.
> 
> Alternatively, this test can be run using:
> 
>     $ avocado run -t arch:mips64el tests/acceptance
>     $ avocado run -t machine:malta tests/acceptance
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .travis.yml                            |  2 +-
>  tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++++
>  2 files changed, 35 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 0a5e0613be..0260263bb8 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -198,7 +198,7 @@ matrix:
>  
>      # Acceptance (Functional) tests
>      - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
> +        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"
>          - TEST_CMD="make check-acceptance"
>        addons:
>          apt:
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 05e43360b8..899c27a9ec 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -10,6 +10,7 @@
>  
>  import os
>  import logging
> +import os

One import is enough :P

>  
>  from avocado_qemu import Test
>  from avocado.utils import process
> @@ -103,3 +104,36 @@ class BootLinuxConsole(Test):
>          self.vm.launch()
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
> +
> +    def test_mips64el_malta(self):
> +        """
> +        This test requires the ar tool to extract "data.tar.gz" from
> +        the Debian package.
> +
> +        The kernel can be rebuilt using this Debian kernel source [1] and
> +        following the instructions on [2].
> +
> +        [1] http://snapshot.debian.org/package/linux-2.6/2.6.32-48/
> +            #linux-source-2.6.32_2.6.32-48
> +        [2] https://kernel-team.pages.debian.net/kernel-handbook/
> +            ch-common-tasks.html#s-common-official
> +
> +        :avocado: tags=arch:mips64el
> +        :avocado: tags=machine:malta
> +        """
> +        deb_url = ('http://snapshot.debian.org/archive/debian/'
> +                   '20130217T032700Z/pool/main/l/linux-2.6/'
> +                   'linux-image-2.6.32-5-5kc-malta_2.6.32-48_mipsel.deb')
> +        deb_hash = '1aaec92083bf22fda31e0d27fa8d9a388e5fc3d5'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        kernel_path = self.extract_from_deb(deb_path,
> +                                            '/boot/vmlinux-2.6.32-5-5kc-malta')
> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> 

Re: [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Philippe Mathieu-Daudé]

On 2/21/19 1:57 AM, Cleber Rosa wrote:
> To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be

[Update] to ... ?

> added using the 29 kernel, so for consistency, let's also update it
> here.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  tests/acceptance/boot_linux_console.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 89df7f6e4f..35b31162d4 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
>          :avocado: tags=arch:x86_64
>          :avocado: tags=machine:pc
>          """
> -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
>                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>  
>          self.vm.set_machine('pc')
> 

Re: [Qemu-devel] [PATCH v3 11/20] Boot Linux Console Test: increase timeout

[Philippe Mathieu-Daudé]

On 2/21/19 1:57 AM, Cleber Rosa wrote:
> When running on very low powered environments, some tests may time out
> causing false negatives.  As a conservative change, and for
> considering that human time (investigating false negatives) is worth
> more than some extra machine cycles (and time), let's increase the
> overall timeout.

However this is annoying when a test is stuck and you work on battery...

> 
> CC: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index cc5dcd7373..fa721a7355 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -21,7 +21,7 @@ class BootLinuxConsole(Test):
>      :avocado: enable
>      """
>  
> -    timeout = 60
> +    timeout = 90
>  
>      KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
>  
> 

Re: [Qemu-devel] [PATCH v3 14/20] Boot Linux Console Test: add a test for mips + malta

[Philippe Mathieu-Daudé]

On 2/21/19 1:57 AM, Cleber Rosa wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> Similar to the x86_64 + pc test, it boots a Linux kernel on a Malta
> board and verify the serial is working.  Also, it relies on the serial
> device set by the machine itself.
> 
> If mips is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:mips" tags.
> 
> Alternatively, this test can be run using:
> 
>     $ avocado run -t arch:mips tests/acceptance
>     $ avocado run -t machine:malta tests/acceptance
>     $ avocado run -t endian:big tests/acceptance
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .travis.yml                            |  2 +-
>  tests/acceptance/boot_linux_console.py | 41 ++++++++++++++++++++++++++
>  2 files changed, 42 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 42971484ab..0a5e0613be 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -198,7 +198,7 @@ matrix:
>  
>      # Acceptance (Functional) tests
>      - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu"
>          - TEST_CMD="make check-acceptance"
>        addons:
>          apt:
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e2ef43e7ce..05e43360b8 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -8,9 +8,12 @@
>  # This work is licensed under the terms of the GNU GPL, version 2 or
>  # later.  See the COPYING file in the top-level directory.
>  
> +import os
>  import logging
>  
>  from avocado_qemu import Test
> +from avocado.utils import process
> +from avocado.utils import archive
>  
>  
>  class BootLinuxConsole(Test):
> @@ -44,6 +47,21 @@ class BootLinuxConsole(Test):
>                  fail = 'Failure message found in console: %s' % failure_message
>                  self.fail(fail)
>  
> +    def extract_from_deb(self, deb, path):
> +        """
> +        Extracts a file from a deb package into the test workdir
> +
> +        :param deb: path to the deb archive
> +        :param file: path within the deb archive of the file to be extracted
> +        :returns: path of the extracted file
> +        """
> +        cwd = os.getcwd()
> +        os.chdir(self.workdir)
> +        process.run("ar x %s data.tar.gz" % deb)
> +        archive.extract("data.tar.gz", self.workdir)
> +        os.chdir(cwd)
> +        return self.workdir + path

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> +
>      def test_x86_64_pc(self):
>          """
>          :avocado: tags=arch:x86_64
> @@ -62,3 +80,26 @@ class BootLinuxConsole(Test):
>          self.vm.launch()
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
> +
> +    def test_mips_malta(self):
> +        """
> +        :avocado: tags=arch:mips
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:big
> +        """
> +        deb_url = ('http://snapshot.debian.org/archive/debian/'
> +                   '20130217T032700Z/pool/main/l/linux-2.6/'
> +                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
> +        deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        kernel_path = self.extract_from_deb(deb_path,
> +                                            '/boot/vmlinux-2.6.32-5-4kc-malta')
> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> 

Re: [Qemu-devel] [PATCH v3 05/20] Acceptance tests: introduce arch parameter and attribute

[Cleber Rosa]

On Fri, Mar 01, 2019 at 11:27:10AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:38 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > It's useful to define the architecture that should be used in
> > situations such as:
> >  * the intended target of the QEMU binary to be used on tests
> >  * the architecture of code to be run within the QEMU binary, such
> >    as a kernel image or a full blown guest OS image
> > 
> > This commit introduces both a test parameter and a test instance
> > attribute, that will contain such a value.
> > 
> > Now, when the "arch" test parameter is given, it will influence the
> > selection of the default QEMU binary, if one is not given explicitly
> > by means of the "qemu_img" parameter.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >  docs/devel/testing.rst                    | 28 +++++++++++++++++++++++
> >  tests/acceptance/avocado_qemu/__init__.py | 14 +++++++++---
> >  2 files changed, 39 insertions(+), 3 deletions(-)
> > 
> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > index ceaaafc69f..6035db1b44 100644
> > --- a/docs/devel/testing.rst
> > +++ b/docs/devel/testing.rst
> > @@ -689,6 +689,21 @@ vm
> >  A QEMUMachine instance, initially configured according to the given
> >  ``qemu_bin`` parameter.
> >  
> > +arch
> > +~~~~
> > +
> > +The architecture can be used on different levels of the stack, e.g. by
> > +the framework or by the test itself.  At the framework level, it will
> > +will currently influence the selection of a QEMU binary (when one is
> 
> s/will will/will/

Thanks for spotting this!

> 
> > +not explicitly given).
> > +
> > +Tests are also free to use this attribute value, for their own needs.
> > +A test may, for instance, use the same value when selecting the
> > +architecture of a kernel or disk image to boot a VM with.
> > +
> > +The ``arch`` attribute will be set to the test parameter of the same
> > +name, and if one is not given explicitly, it will be set to ``None``.
> > +
> >  qemu_bin
> >  ~~~~~~~~
> >  
> > @@ -711,6 +726,19 @@ like the following:
> >  
> >    PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
> >  
> > +arch
> > +~~~~
> > +
> > +The architecture that will influence the selection of a QEMU binary
> > +(when one is not explicitly given).
> > +
> > +Tests are also free to use this parameter value, for their own needs.
> > +A test may, for instance, use the same value when selecting the
> > +architecture of a kernel or disk image to boot a VM with.
> > +
> > +This parameter has a direct relation with the ``arch`` attribute.  If
> > +not given, it will default to None.
> > +
> >  qemu_bin
> >  ~~~~~~~~
> >  
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> > index d8d5b48dac..f580582602 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -23,16 +23,22 @@ def is_readable_executable_file(path):
> >      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
> >  
> >  
> > -def pick_default_qemu_bin():
> > +def pick_default_qemu_bin(arch=None):
> >      """
> >      Picks the path of a QEMU binary, starting either in the current working
> >      directory or in the source tree root directory.
> >  
> > +    :param arch: the arch to use when looking for a QEMU binary (the target
> > +                 will match the arch given).  If None (the default) arch
> 
> s/None (the default)/None (the default),/ makes that sentence less
> confusing :)
> 

Good point, thanks!

- Cleber.

Re: [Qemu-devel] [PATCH v3 06/20] Acceptance tests: use "arch:" tag to filter target specific tests

[Cleber Rosa]

On Fri, Mar 01, 2019 at 11:32:27AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:39 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > Currently, the only test that contains some target architecture
> > information is "boot_linux_console.py" which test contains a "x86_64"
> 
> But there are two others changed by you here, aren't there?
>

Yes, the "only test" is now outdated.  Fixing it.

> > tag.  But that tag is not respected in the default execution, that is,
> > "make check-acceptance" doesn't do anything with it.
> > 
> > That said, even the target architecture handling currently present in
> > the "avocado_qemu.Test" class, class is pretty limited.  For instance,
> 
> s/class, class/class/

You are so good at proof reading! Kudos!

> 
> > by default, it chooses a target based on the host architecture.
> > 
> > Because the original implementation of the tags feature in Avocado did
> > not include any time of namespace or "key:val" mechanism, no tag has
> > relation to another tag.  The new implementation of the tags feature
> > from version 67.0 onwards, allows "key:val" tags, and because of that,
> > a test can be classified with a tag in a given key.  For instance, the
> > new proposed version of the "boot_linux_console.py" test, which
> > downloads and attempts to run a x86_64 kernel, is now tagged as:
> > 
> >   :avocado: tags=arch:x86_64
> > 
> > This means that it can be filtered (out) when no x86_64 target is
> > available.  At the same time, tests that don't have a "arch:" tag,
> > will not be filtered out.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/Makefile.include                 | 3 +++
> >  tests/acceptance/boot_linux_console.py | 2 +-
> >  tests/acceptance/linux_initrd.py       | 2 +-
> >  tests/acceptance/virtio_version.py     | 2 +-
> >  tests/requirements.txt                 | 2 +-
> >  5 files changed, 7 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 93ea42553e..633992603d 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -1090,6 +1090,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >  # Any number of command separated loggers are accepted.  For more
> >  # information please refer to "avocado --help".
> >  AVOCADO_SHOW=app
> > +AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
> >  
> >  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
> >  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> > @@ -1115,6 +1116,8 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> >  	$(call quiet-command, \
> >              $(TESTS_VENV_DIR)/bin/python -m avocado \
> >              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > +            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> > +            $(AVOCADO_TAGS) \
> >              --failfast=on $(SRC_PATH)/tests/acceptance, \
> >              "AVOCADO", "tests/acceptance")
> >  
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 98324f7591..46b20bdfe2 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -19,7 +19,7 @@ class BootLinuxConsole(Test):
> >      and the kernel command line is properly passed from QEMU to the kernel
> >  
> >      :avocado: enable
> > -    :avocado: tags=x86_64
> > +    :avocado: tags=arch:x86_64
> >      """
> >  
> >      timeout = 60
> > diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
> > index 737355c2ef..c75e29be70 100644
> > --- a/tests/acceptance/linux_initrd.py
> > +++ b/tests/acceptance/linux_initrd.py
> > @@ -19,7 +19,7 @@ class LinuxInitrd(Test):
> >      Checks QEMU evaluates correctly the initrd file passed as -initrd option.
> >  
> >      :avocado: enable
> > -    :avocado: tags=x86_64
> > +    :avocado: tags=arch:x86_64
> >      """
> >  
> >      timeout = 60
> > diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
> > index ce990250d8..3b280e7fc3 100644
> > --- a/tests/acceptance/virtio_version.py
> > +++ b/tests/acceptance/virtio_version.py
> > @@ -62,7 +62,7 @@ class VirtioVersionCheck(Test):
> >      `disable-legacy`.
> >  
> >      :avocado: enable
> > -    :avocado: tags=x86_64
> > +    :avocado: tags=arch:x86_64
> >      """
> >  
> >      # just in case there are failures, show larger diff:
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index 64c6e27a94..002ded6a22 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -1,4 +1,4 @@
> >  # Add Python module requirements, one per line, to be installed
> >  # in the tests/venv Python virtual environment. For more info,
> >  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> > -avocado-framework==65.0
> > +avocado-framework==68.0
> 
> I think you should explain why you bump the required version to 68.0
> instead of the 67.0 you mention above.

Actually, this bit will be removed in the rebase, because other commit
(9531d26c106) already merged did that for other reason.  So, it looks
like the explanation of the needed feature being present in 67.0 (and
later) is adequate.

Thanks,
- Cleber.

Re: [Qemu-devel] [PATCH v3 07/20] Acceptance tests: look for target architecture in test tags first

[Cleber Rosa]

On Fri, Mar 01, 2019 at 11:37:19AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:40 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > A test can, optionally, be tagged for one or many architectures.  If a
> > test has been tagged for a single architecture, there's a high chance
> > that the test won't run on other architectures.  This changes the
> > default order of choosing a default target architecture to use based
> > on the 'arch' tag value first.
> > 
> > The precedence order is for choosing a QEMU binary to use for a test
> > is now:
> > 
> >  * qemu_bin parameter
> >  * arch parameter
> >  * arch tag value (for example, x86_64 if ":avocado: tags=arch:x86_64
> >    is used)
> > 
> > This means that if one runs:
> > 
> >  $ avocado run -p qemu_bin=/usr/bin/qemu-system-x86_64 test.py
> > 
> > No arch parameter or tag will influence the selection of the QEMU
> > target binary.  If one runs:
> > 
> >  $ avocado run -p arch=ppc64 test.py
> > 
> > The target binary selection mechanism will attempt to find a binary
> > such as "ppc64-softmmu/qemu-system-ppc64".  And finally, if one runs
> > a test that is tagged (in its docstring) with "arch:aarch64":
> > 
> >  $ avocado run aarch64.py
> > 
> > The target binary selection mechanism will attempt to find a binary
> > such as "aarch64-softmmu/qemu-system-aarch64".
> > 
> > At this time, no provision is made to cancel the execution of tests if
> > the arch parameter given (manually) does not match the test "arch"
> > tag, but it may be a useful default behavior to be added in the
> > future.
> 
> Would it be useful as well to cancel a test if the passed-in qemu_bin
> parameter clashed with the tagged architecture?
> 

It would... in fact, a much older version of this proposal did that.
I'm waiting for the most basic things to get in, before adding yet
another "special" behavior.

Thanks for the review and suggestion.
- Cleber.

> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/testing.rst                    | 4 +++-
> >  tests/acceptance/avocado_qemu/__init__.py | 7 ++++++-
> >  2 files changed, 9 insertions(+), 2 deletions(-)
> > 
> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > index 6035db1b44..87bcf8ef43 100644
> > --- a/docs/devel/testing.rst
> > +++ b/docs/devel/testing.rst
> > @@ -702,7 +702,9 @@ A test may, for instance, use the same value when selecting the
> >  architecture of a kernel or disk image to boot a VM with.
> >  
> >  The ``arch`` attribute will be set to the test parameter of the same
> > -name, and if one is not given explicitly, it will be set to ``None``.
> > +name.  If one is not given explicitly, it will either be set to
> > +``None``, or, if the test is tagged with one (and only one)
> > +``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
> >  
> >  qemu_bin
> >  ~~~~~~~~
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> > index f580582602..9e98d113cb 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -53,7 +53,12 @@ def pick_default_qemu_bin(arch=None):
> >  class Test(avocado.Test):
> >      def setUp(self):
> >          self.vm = None
> > -        self.arch = self.params.get('arch')
> > +        arches = self.tags.get('arch', [])
> > +        if len(arches) == 1:
> > +            arch = arches.pop()
> > +        else:
> > +            arch = None
> > +        self.arch = self.params.get('arch', default=arch)
> >          default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
> >          self.qemu_bin = self.params.get('qemu_bin',
> >                                          default=default_qemu_bin)
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Cleber Rosa]

On Fri, Mar 01, 2019 at 11:41:44AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:42 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> > added using the 29 kernel, so for consistency, let's also update it
> > here.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> > ---
> >  tests/acceptance/boot_linux_console.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 89df7f6e4f..35b31162d4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:x86_64
> >          :avocado: tags=machine:pc
> >          """
> > -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
> >                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> > +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> >          self.vm.set_machine('pc')
> 
> This looks reasonable; but an obvious follow-up question is how we
> handle updates to new distro levels in general.
> 
> I don't think we'll need to run acceptance tests on old qemus, though.

No, I don't think we have to bother with that.  Backport(er)s would be
expected to handle this (if they ever come to existence).

You do have a point about consistenly updating those images, though.
I won't pretend to have an answer for that just yet... but it's
possible that these "kernel_urls" can be turned into parameters
that will eventually be expand based on the user's selected distro
and version (similar to the Avocado vmimage library[1]).

- Cleber.

[1] https://avocado-framework.readthedocs.io/en/69.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get

Re: [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Cleber Rosa]

On Sat, Mar 02, 2019 at 10:42:13PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/21/19 1:57 AM, Cleber Rosa wrote:
> > To the stock Fedora 29 kernel, from the Fedora 28.  New tests will be
> 
> [Update] to ... ?
>

Right... I used this line as a continuation of the commit title,
but probably shouldn't have done that.

> > added using the 29 kernel, so for consistency, let's also update it
> > here.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>

Thanks!
- Cleber.

> > ---
> >  tests/acceptance/boot_linux_console.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index 89df7f6e4f..35b31162d4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:x86_64
> >          :avocado: tags=machine:pc
> >          """
> > -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > +        kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/'
> >                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > -        kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> > +        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> >          self.vm.set_machine('pc')
> > 

Re: [Qemu-devel] [PATCH v3 11/20] Boot Linux Console Test: increase timeout

[Cleber Rosa]

On Sat, Mar 02, 2019 at 10:43:52PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/21/19 1:57 AM, Cleber Rosa wrote:
> > When running on very low powered environments, some tests may time out
> > causing false negatives.  As a conservative change, and for
> > considering that human time (investigating false negatives) is worth
> > more than some extra machine cycles (and time), let's increase the
> > overall timeout.
> 
> However this is annoying when a test is stuck and you work on battery...
>

I agree, but at this point, the more we try to fix all use cases, the
furthest we get to something reasonable IMO.

Any ideas on how to do better here while still keeping things simple?

- Cleber.

> > 
> > CC: Alex Bennée <alex.bennee@linaro.org>
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/acceptance/boot_linux_console.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index cc5dcd7373..fa721a7355 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -21,7 +21,7 @@ class BootLinuxConsole(Test):
> >      :avocado: enable
> >      """
> >  
> > -    timeout = 60
> > +    timeout = 90
> >  
> >      KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> >  
> > 

Re: [Qemu-devel] [PATCH v3 13/20] scripts/qemu.py: support adding a console with the default serial device

[Cleber Rosa]

On Fri, Mar 01, 2019 at 11:55:13AM +0100, Cornelia Huck wrote:
> On Wed, 20 Feb 2019 19:57:46 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
> > The set_console() utility function traditionally adds a device either
> > based on the explicitly given device type, or based on the machine type,
> > a known good type of device.
> 
> Hm, I find this sentence hard to parse... maybe it should be "either
> adds a device..." and "adds a known good type of device"?
>

OK, sounds good, but I may change it a little bit given that you
following feedback made me change (positively) a lot here.

> > 
> > But, for a number of machine types, it may be impossible or
> > inconvenient to add the devices my means of "-device" command line
> 
> s/my/by/
>

Same here.

> > options, and then it may better to just use the "-serial" option and
> > let QEMU itself, based on the machine type, set the device
> > accordingly.
> > 
> > To achieve that, the behavior of set_console() now flags the intention
> > to add a console device on launch(), and if no explicit device type is
> > given, and there's no definition on CONSOLE_DEV_TYPES, the "-serial"
> > is going to be added to the QEMU command line, instead of raising
> > exceptions.
> > 
> > Based on testing with different machine types, the CONSOLE_DEV_TYPES
> > is now being set to the bare essential entries (one entry to be
> > honest), for machine types that can not easily give us a working
> > console with "-serial".
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  scripts/qemu.py | 39 +++++++++++++++++++--------------------
> >  1 file changed, 19 insertions(+), 20 deletions(-)
> > 
> > diff --git a/scripts/qemu.py b/scripts/qemu.py
> > index ee85309923..bd1d2e2b9a 100644
> > --- a/scripts/qemu.py
> > +++ b/scripts/qemu.py
> > @@ -42,11 +42,6 @@ def kvm_available(target_arch=None):
> >  
> >  #: Maps machine types to the preferred console device types
> >  CONSOLE_DEV_TYPES = {
> > -    r'^clipper$': 'isa-serial',
> > -    r'^malta': 'isa-serial',
> > -    r'^(pc.*|q35.*|isapc)$': 'isa-serial',
> > -    r'^(40p|powernv|prep)$': 'isa-serial',
> > -    r'^pseries.*': 'spapr-vty',
> >      r'^s390-ccw-virtio.*': 'sclpconsole',
> 
> FWIW, s390x has supported the '-serial' parameter since 3.0 (commit
> 052888f043bacb18231046045b1f9cd946703170). Maybe you can drop this now?

Yes, thanks for pointing that out!  I'm simplifying the set_console()
behavior and dropping the CONSOLE_DEV_TYPES dict completely.

> If not, what error are you getting?

No errors, it seems to work great now.

- Cleber.

Re: [Qemu-devel] [PATCH v3 15/20] Boot Linux Console Test: add a test for mips64el + malta

[Cleber Rosa]

On Sat, Mar 02, 2019 at 10:39:47PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/21/19 1:57 AM, Cleber Rosa wrote:
> > +import os
> 
> One import is enough :P
>

Oh my... thanks for spotting this.

- Cleber.