All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
@ 2018-06-22  0:44 Philippe Mathieu-Daudé
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property Philippe Mathieu-Daudé
                   ` (7 more replies)
  0 siblings, 8 replies; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé,
	qemu-devel, Lukáš Doktor, Fam Zheng, Aurelien Jarno

Hi,

Another neanderthal approach to add multi-arch acceptance tests using
Avocado.

Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
at using Avocado in QEMU.
Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
a bit further:
- we can now run other archs
- Avocado is integrated into Travis CI

I'm not sure Travis is the best CI available for those tests, but it runs
quite fast:

      ...
      LINK    x86_64-softmmu/qemu-system-x86_64
    JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
    JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
     (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
     (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
     (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
     (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
     (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
     (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
     (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
    RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 6.21 s

    Done. Your build exited with 0.

    Job ran for 5 min 34 sec

I couldn't add the Alpha tests because all the vmlinux kernels I found
online are gzipped, so I'm waiting some upstream Avocado merges before.

Cleber: I hope you can help me pythonizing this series :p

Regards,

Phil.

[0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
[1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html

Philippe Mathieu-Daudé (6):
  avocado: Add a Test.arch property
  tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
  tests/acceptance: Improve the Avocado tags
  tests/acceptance: Add a BootLinuxConsoleMips test
  tests/acceptance: Add a kludge to not use the default console
  travis: Add Avocado tests

 .travis.yml                               |  8 +++++
 scripts/qemu.py                           |  6 ++--
 tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
 tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
 4 files changed, 67 insertions(+), 8 deletions(-)

-- 
2.18.0.rc2

^ permalink raw reply	[flat|nested] 28+ messages in thread

* [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-28 16:10   ` Alex Bennée
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test Philippe Mathieu-Daudé
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng

Tests can change this property to run tests in other
architectures than the host one.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 1e54fd5932..6e9601f5e9 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -13,6 +13,7 @@ import sys
 
 import avocado
 
+HOST_ARCH = os.uname()[4]
 SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
 SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
@@ -23,12 +24,11 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin():
+def pick_default_qemu_bin(arch):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
     """
-    arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
                                           "qemu-system-%s" % arch)
     if is_readable_executable_file(qemu_bin_relative_path):
@@ -41,10 +41,19 @@ def pick_default_qemu_bin():
 
 
 class Test(avocado.Test):
+    _arch = HOST_ARCH
+
+    @property
+    def arch(self):
+        """
+        Returns the architecture required to run the current test
+        """
+        return self._arch
+
     def setUp(self):
         self.vm = None
-        self.qemu_bin = self.params.get('qemu_bin',
-                                        default=pick_default_qemu_bin())
+        qemu_bin = pick_default_qemu_bin(self.arch)
+        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
         self.vm = QEMUMachine(self.qemu_bin)
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-28 17:17   ` Alex Bennée
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 3/6] tests/acceptance: Improve the Avocado tags Philippe Mathieu-Daudé
                   ` (5 subsequent siblings)
  7 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 98324f7591..f6a34d75a5 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -13,7 +13,7 @@ import logging
 from avocado_qemu import Test
 
 
-class BootLinuxConsole(Test):
+class BootLinuxConsoleX86_64(Test):
     """
     Boots a x86_64 Linux kernel and checks that the console is operational
     and the kernel command line is properly passed from QEMU to the kernel
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [Qemu-devel] [PATCH v2 3/6] tests/acceptance: Improve the Avocado tags
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property Philippe Mathieu-Daudé
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test Philippe Mathieu-Daudé
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng

Introduce 'arch' and 'endian' namespaces.

For example, all little-endian tests can be run using:

    $ avocado run -t endian:little tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index f6a34d75a5..17dc8d58c1 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -19,7 +19,8 @@ class BootLinuxConsoleX86_64(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=x86_64
+    :avocado: tags=endian:little
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
                   ` (2 preceding siblings ...)
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 3/6] tests/acceptance: Improve the Avocado tags Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-28 16:23   ` Alex Bennée
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console Philippe Mathieu-Daudé
                   ` (3 subsequent siblings)
  7 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng, Aurelien Jarno

Similar to the BootLinuxConsoleX86_64 test:
boot a Linux kernel on a Malta board and verify the serial is working.

This test can be run using:

    $ avocado run -t endian:big tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 17dc8d58c1..72cf5e943c 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
                 break
             if 'Kernel panic - not syncing' in msg:
                 self.fail("Kernel panic reached")
+
+
+class BootLinuxConsoleMips(Test):
+    """
+    Boots a mips Linux kernel and checks that the console is operational
+    and the kernel command line is properly passed from QEMU to the kernel
+
+    :avocado: enable
+    :avocado: tags=endian:big
+    :avocado: tags=arch:mips
+    :avocado: tags=board:malta
+    """
+
+    arch = "mips"
+    timeout = 60
+
+    def test(self):
+        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
+                      'vmlinux-3.2.0-4-4kc-malta')
+        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = 'console=ttyS0 printk.time=0'
+        self.vm.add_args('-m', "64",
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline()
+            console_logger.debug(msg.strip())
+            if 'Kernel command line: %s' % kernel_command_line in msg:
+                break
+            if 'Kernel panic - not syncing' in msg:
+                self.fail("Kernel panic reached")
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
                   ` (3 preceding siblings ...)
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-28 16:33   ` Alex Bennée
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 6/6] travis: Add Avocado tests Philippe Mathieu-Daudé
                   ` (2 subsequent siblings)
  7 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng

The board already instantiate the proper devices, we don't want to
add extra devices but connect the chardev to one of the serial already
available.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qemu.py                        | 6 ++++--
 tests/acceptance/boot_linux_console.py | 3 ++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index f099ce7278..7a975f4538 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -211,8 +211,10 @@ class QEMUMachine(object):
                                                  self._name + "-console.sock")
             chardev = ('socket,id=console,path=%s,server,nowait' %
                        self._console_address)
-            device = '%s,chardev=console' % self._console_device_type
-            args.extend(['-chardev', chardev, '-device', device])
+            args.extend(['-chardev', chardev])
+            if len(self._console_device_type):
+                device = '%s,chardev=console' % self._console_device_type
+                args.extend(['-device', device])
         return args
 
     def _pre_launch(self):
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 72cf5e943c..510742c4c7 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -69,9 +69,10 @@ class BootLinuxConsoleMips(Test):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_machine('malta')
-        self.vm.set_console()
+        self.vm.set_console("") # FIXME this disable isa-serial to use -serial
         kernel_command_line = 'console=ttyS0 printk.time=0'
         self.vm.add_args('-m', "64",
+                         '-serial', "chardev:console", # FIXME ... here.
                          '-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [Qemu-devel] [PATCH v2 6/6] travis: Add Avocado tests
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
                   ` (4 preceding siblings ...)
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console Philippe Mathieu-Daudé
@ 2018-06-22  0:44 ` Philippe Mathieu-Daudé
  2018-06-28 15:46 ` [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Alex Bennée
  2018-06-28 16:01 ` Alex Bennée
  7 siblings, 0 replies; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-22  0:44 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost
  Cc: Philippe Mathieu-Daudé, qemu-devel, Fam Zheng

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
https://travis-ci.org/philmd/qemu/jobs/395266626

  ...
  LINK    x86_64-softmmu/qemu-system-x86_64
JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
 (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
 (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
 (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
 (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
 (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
 (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
 (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 6.21 s

Done. Your build exited with 0.

Job ran for 5 min 34 sec
---
 .travis.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index fabfe9ec34..7b224add2d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -153,3 +153,11 @@ matrix:
         - TEST_CMD=""
       before_script:
         - ./configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread -fuse-ld=gold" || { cat config.log && exit 1; }
+    # Avocado tests
+    - env: CONFIG="--target-list=x86_64-softmmu,mips-softmmu --enable-trace-backends=nop --disable-tools --disable-docs"
+           TEST_CMD="avocado run tests/acceptance"
+      python:
+        - "3.6"
+      before_script:
+        - pip install --user avocado-framework
+        - ./configure ${CONFIG} || { cat config.log && exit 1; }
-- 
2.18.0.rc2

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
                   ` (5 preceding siblings ...)
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 6/6] travis: Add Avocado tests Philippe Mathieu-Daudé
@ 2018-06-28 15:46 ` Alex Bennée
  2018-06-28 16:26   ` Cleber Rosa
  2018-06-28 16:01 ` Alex Bennée
  7 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 15:46 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Lukáš Doktor,
	Fam Zheng, Aurelien Jarno


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
> Another neanderthal approach to add multi-arch acceptance tests using
> Avocado.
>
> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
> at using Avocado in QEMU.
> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
> a bit further:
> - we can now run other archs
> - Avocado is integrated into Travis CI
>
> I'm not sure Travis is the best CI available for those tests, but it runs
> quite fast:
>
>       ...
>       LINK    x86_64-softmmu/qemu-system-x86_64
>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 6.21 s

Hmm however if I do:

   pip install --user avocado-framework
   avocado run tests/acceptance

I get:

   Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
   Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
   Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
   Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
   Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
   You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log

Which I seem to remember from last time I looked into this. I think
avocado makes a bunch of assumptions about the filesystem and OS layout
which is hard to gloss over for something you should be able to run
straight out of the source tree.


>
>     Done. Your build exited with 0.
>
>     Job ran for 5 min 34 sec
>
> I couldn't add the Alpha tests because all the vmlinux kernels I found
> online are gzipped, so I'm waiting some upstream Avocado merges before.
>
> Cleber: I hope you can help me pythonizing this series :p
>
> Regards,
>
> Phil.
>
> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>
> Philippe Mathieu-Daudé (6):
>   avocado: Add a Test.arch property
>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>   tests/acceptance: Improve the Avocado tags
>   tests/acceptance: Add a BootLinuxConsoleMips test
>   tests/acceptance: Add a kludge to not use the default console
>   travis: Add Avocado tests
>
>  .travis.yml                               |  8 +++++
>  scripts/qemu.py                           |  6 ++--
>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>  4 files changed, 67 insertions(+), 8 deletions(-)


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
                   ` (6 preceding siblings ...)
  2018-06-28 15:46 ` [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Alex Bennée
@ 2018-06-28 16:01 ` Alex Bennée
  2018-07-27  5:33   ` Philippe Mathieu-Daudé
  7 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 16:01 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Lukáš Doktor,
	Fam Zheng, Aurelien Jarno


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
> Another neanderthal approach to add multi-arch acceptance tests using
> Avocado.
>
> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
> at using Avocado in QEMU.
> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
> a bit further:
> - we can now run other archs
> - Avocado is integrated into Travis CI
>
> I'm not sure Travis is the best CI available for those tests, but it runs
> quite fast:
>
>       ...
>       LINK    x86_64-softmmu/qemu-system-x86_64
>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>       INTERRUPT 0 | CANCEL 0

That said on my AArch64 server it fairs better:

15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
 (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
 (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
 (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
 (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
 (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
 (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
 (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
JOB TIME   : 2.47 s

Although given the errors in the logs I'm surprised at ERROR 0

>     JOB TIME   : 6.21 s
>
>     Done. Your build exited with 0.
>
>     Job ran for 5 min 34 sec
>
> I couldn't add the Alpha tests because all the vmlinux kernels I found
> online are gzipped, so I'm waiting some upstream Avocado merges before.
>
> Cleber: I hope you can help me pythonizing this series :p
>
> Regards,
>
> Phil.
>
> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>
> Philippe Mathieu-Daudé (6):
>   avocado: Add a Test.arch property
>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>   tests/acceptance: Improve the Avocado tags
>   tests/acceptance: Add a BootLinuxConsoleMips test
>   tests/acceptance: Add a kludge to not use the default console
>   travis: Add Avocado tests
>
>  .travis.yml                               |  8 +++++
>  scripts/qemu.py                           |  6 ++--
>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>  4 files changed, 67 insertions(+), 8 deletions(-)


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property Philippe Mathieu-Daudé
@ 2018-06-28 16:10   ` Alex Bennée
  2018-06-28 21:54     ` Alex Bennée
  0 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 16:10 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Fam Zheng


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Tests can change this property to run tests in other
> architectures than the host one.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index 1e54fd5932..6e9601f5e9 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -13,6 +13,7 @@ import sys
>
>  import avocado
>
> +HOST_ARCH = os.uname()[4]

The python docs seem to point to platform as a more stable way of
querying this stuff:

  platform.machine() => 'x86_64' or 'aarch64'

>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>
>
> -def pick_default_qemu_bin():
> +def pick_default_qemu_bin(arch):
>      """
>      Picks the path of a QEMU binary, starting either in the current working
>      directory or in the source tree root directory.
>      """
> -    arch = os.uname()[4]
>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>                                            "qemu-system-%s" % arch)
>      if is_readable_executable_file(qemu_bin_relative_path):
> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>
>
>  class Test(avocado.Test):
> +    _arch = HOST_ARCH
> +
> +    @property
> +    def arch(self):
> +        """
> +        Returns the architecture required to run the current test
> +        """
> +        return self._arch
> +
>      def setUp(self):
>          self.vm = None
> -        self.qemu_bin = self.params.get('qemu_bin',
> -                                        default=pick_default_qemu_bin())
> +        qemu_bin = pick_default_qemu_bin(self.arch)
> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>          if self.qemu_bin is None:
>              self.cancel("No QEMU binary defined or found in the source tree")
>          self.vm = QEMUMachine(self.qemu_bin)


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test Philippe Mathieu-Daudé
@ 2018-06-28 16:23   ` Alex Bennée
  2018-06-28 17:40     ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 16:23 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Fam Zheng, Aurelien Jarno


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Similar to the BootLinuxConsoleX86_64 test:
> boot a Linux kernel on a Malta board and verify the serial is working.
>
> This test can be run using:
>
>     $ avocado run -t endian:big tests/acceptance
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 17dc8d58c1..72cf5e943c 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>                  break
>              if 'Kernel panic - not syncing' in msg:
>                  self.fail("Kernel panic reached")
> +
> +
> +class BootLinuxConsoleMips(Test):
> +    """
> +    Boots a mips Linux kernel and checks that the console is operational
> +    and the kernel command line is properly passed from QEMU to the kernel
> +
> +    :avocado: enable
> +    :avocado: tags=endian:big
> +    :avocado: tags=arch:mips
> +    :avocado: tags=board:malta
> +    """
> +
> +    arch = "mips"
> +    timeout = 60
> +
> +    def test(self):
> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> +                      'vmlinux-3.2.0-4-4kc-malta')
> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)

I'm uncomfortable using "random" binaries of websites as the source of
our test kernels. I can see the justification for distro kernels as they
at least have the infrastructure to rebuild from source if you really
want to, but even then the distros don't cover a lot of the
architectures.

I had experimented with using docker based builds for building test
fixtures (see tests/docker/dockerbuilds):

   https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2

As these tests are fairly simple boot tests that just need kernels maybe
we could look at tooling up the generation of these images in a
repeatable way - similar to the way vmtest builds VMs.

> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = 'console=ttyS0 printk.time=0'
> +        self.vm.add_args('-m', "64",
> +                         '-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console = self.vm.console_socket.makefile()
> +        console_logger = logging.getLogger('console')
> +        while True:
> +            msg = console.readline()
> +            console_logger.debug(msg.strip())
> +            if 'Kernel command line: %s' % kernel_command_line in msg:
> +                break
> +            if 'Kernel panic - not syncing' in msg:
> +                self.fail("Kernel panic reached")

Of course for bonus points a simple rootfs with hackbench or some such
in it would be nice. But I appreciate this makes the building job a lot
more complex than just a kernel.

--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-28 15:46 ` [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Alex Bennée
@ 2018-06-28 16:26   ` Cleber Rosa
  2018-06-28 17:13     ` Alex Bennée
  0 siblings, 1 reply; 28+ messages in thread
From: Cleber Rosa @ 2018-06-28 16:26 UTC (permalink / raw)
  To: Alex Bennée, Philippe Mathieu-Daudé
  Cc: Eduardo Habkost, qemu-devel, Lukáš Doktor, Fam Zheng,
	Aurelien Jarno



On 06/28/2018 11:46 AM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi,
>>
>> Another neanderthal approach to add multi-arch acceptance tests using
>> Avocado.
>>
>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>> at using Avocado in QEMU.
>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>> a bit further:
>> - we can now run other archs
>> - Avocado is integrated into Travis CI
>>
>> I'm not sure Travis is the best CI available for those tests, but it runs
>> quite fast:
>>
>>       ...
>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 6.21 s
> 
> Hmm however if I do:
> 
>    pip install --user avocado-framework
>    avocado run tests/acceptance
> 
> I get:
> 
>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')

Alex,

These are errors related to your installation of Avocado-*VT*[1].

>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
> 

And these from Avocado-*Virt*[2].  Neither of those are used in the
tests under "tests/acceptance".

> Which I seem to remember from last time I looked into this. I think
> avocado makes a bunch of assumptions about the filesystem and OS layout
> which is hard to gloss over for something you should be able to run
> straight out of the source tree.
> 
> 

These Avocado-*VT* issues deserve a better look indeed.  Would you
please describe (if not here, under a GH issue) your environment?

Thanks,
- Cleber.

[1] - https://github.com/avocado-framework/avocado-vt
[2] - https://github.com/avocado-framework/avocado-virt

>>
>>     Done. Your build exited with 0.
>>
>>     Job ran for 5 min 34 sec
>>
>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>
>> Cleber: I hope you can help me pythonizing this series :p
>>
>> Regards,
>>
>> Phil.
>>
>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>
>> Philippe Mathieu-Daudé (6):
>>   avocado: Add a Test.arch property
>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>   tests/acceptance: Improve the Avocado tags
>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>   tests/acceptance: Add a kludge to not use the default console
>>   travis: Add Avocado tests
>>
>>  .travis.yml                               |  8 +++++
>>  scripts/qemu.py                           |  6 ++--
>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>  4 files changed, 67 insertions(+), 8 deletions(-)
> 
> 
> --
> Alex Bennée
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console
  2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console Philippe Mathieu-Daudé
@ 2018-06-28 16:33   ` Alex Bennée
  0 siblings, 0 replies; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 16:33 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Fam Zheng


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> The board already instantiate the proper devices, we don't want to
> add extra devices but connect the chardev to one of the serial already
> available.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  scripts/qemu.py                        | 6 ++++--
>  tests/acceptance/boot_linux_console.py | 3 ++-
>  2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index f099ce7278..7a975f4538 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -211,8 +211,10 @@ class QEMUMachine(object):
>                                                   self._name + "-console.sock")
>              chardev = ('socket,id=console,path=%s,server,nowait' %
>                         self._console_address)
> -            device = '%s,chardev=console' % self._console_device_type
> -            args.extend(['-chardev', chardev, '-device', device])
> +            args.extend(['-chardev', chardev])
> +            if len(self._console_device_type):
> +                device = '%s,chardev=console' % self._console_device_type
> +                args.extend(['-device', device])
>          return args
>
>      def _pre_launch(self):
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 72cf5e943c..510742c4c7 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -69,9 +69,10 @@ class BootLinuxConsoleMips(Test):
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
>          self.vm.set_machine('malta')
> -        self.vm.set_console()
> +        self.vm.set_console("") # FIXME this disable isa-serial to
>          use -serial

This is reminiscent of assumptions libvirt made about serial consoles.
Does -serial not work on x86 for old style setups? I'm sure I have
x86-64 boots with -serial mon:stdio lines.

>          kernel_command_line = 'console=ttyS0 printk.time=0'
>          self.vm.add_args('-m', "64",
> +                         '-serial', "chardev:console", # FIXME ... here.
>                           '-kernel', kernel_path,
>                           '-append', kernel_command_line)
>          self.vm.launch()


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-28 16:26   ` Cleber Rosa
@ 2018-06-28 17:13     ` Alex Bennée
  2018-06-28 17:24       ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 17:13 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Philippe Mathieu-Daudé,
	Eduardo Habkost, qemu-devel, Lukáš Doktor, Fam Zheng,
	Aurelien Jarno


Cleber Rosa <crosa@redhat.com> writes:

> On 06/28/2018 11:46 AM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Hi,
>>>
>>> Another neanderthal approach to add multi-arch acceptance tests using
>>> Avocado.
>>>
>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>> at using Avocado in QEMU.
>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>> a bit further:
>>> - we can now run other archs
>>> - Avocado is integrated into Travis CI
>>>
>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>> quite fast:
>>>
>>>       ...
>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>>     JOB TIME   : 6.21 s
>>
>> Hmm however if I do:
>>
>>    pip install --user avocado-framework
>>    avocado run tests/acceptance
>>
>> I get:
>>
>>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
>
> Alex,
>
> These are errors related to your installation of Avocado-*VT*[1].
>
>>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
>>
>
> And these from Avocado-*Virt*[2].  Neither of those are used in the
> tests under "tests/acceptance".
>
>> Which I seem to remember from last time I looked into this. I think
>> avocado makes a bunch of assumptions about the filesystem and OS layout
>> which is hard to gloss over for something you should be able to run
>> straight out of the source tree.
>>
>>
>
> These Avocado-*VT* issues deserve a better look indeed.  Would you
> please describe (if not here, under a GH issue) your environment?

Ahh probably from last time I tried to get things up and running. I
rm -rf ~/.local and things seem to be working well now.

However I was under the impression that VT or virt was needed for this
testing. Or is this an old assumption?

>
> Thanks,
> - Cleber.
>
> [1] - https://github.com/avocado-framework/avocado-vt
> [2] - https://github.com/avocado-framework/avocado-virt
>
>>>
>>>     Done. Your build exited with 0.
>>>
>>>     Job ran for 5 min 34 sec
>>>
>>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>>
>>> Cleber: I hope you can help me pythonizing this series :p
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>>
>>> Philippe Mathieu-Daudé (6):
>>>   avocado: Add a Test.arch property
>>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>>   tests/acceptance: Improve the Avocado tags
>>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>>   tests/acceptance: Add a kludge to not use the default console
>>>   travis: Add Avocado tests
>>>
>>>  .travis.yml                               |  8 +++++
>>>  scripts/qemu.py                           |  6 ++--
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>>  4 files changed, 67 insertions(+), 8 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
  2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test Philippe Mathieu-Daudé
@ 2018-06-28 17:17   ` Alex Bennée
  0 siblings, 0 replies; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 17:17 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Fam Zheng


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 98324f7591..f6a34d75a5 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -13,7 +13,7 @@ import logging
>  from avocado_qemu import Test
>
>
> -class BootLinuxConsole(Test):
> +class BootLinuxConsoleX86_64(Test):
>      """
>      Boots a x86_64 Linux kernel and checks that the console is operational
>      and the kernel command line is properly passed from QEMU to the kernel

Not your bug but it also needs to be better at choosing a QEMU to run:

2018-06-28 16:10:24,736 download         L0066 INFO | Fetching https://mirrors.kernel.org/fedora/releases/28/Everything/x86_64/os/images/pxeboot/vmlinuz -> /home/alex.bennee/avocado/data/cache/vmlinuz.gqWhs5
2018-06-28 16:10:40,841 qemu             L0268 DEBUG| Error launching VM
2018-06-28 16:10:40,841 qemu             L0270 DEBUG| Command: 'aarch64-softmmu/qemu-system-aarch64 -chardev socket,id=mon,path=/var/tmp/tmpdXbtIA/qemu-16646-monitor.sock -mon chardev=mon,mode=control -display none -vga none -machine pc -chardev socket,id=console,path=/var/tmp/tmpdXbtIA/qemu-16646-console.sock,server,nowait -device isa-serial,chardev=console -kernel /home/alex.bennee/avocado/data/cache/vmlinuz -append console=ttyS0'


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-28 17:13     ` Alex Bennée
@ 2018-06-28 17:24       ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-28 17:24 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa
  Cc: Eduardo Habkost, qemu-devel, Lukáš Doktor, Fam Zheng,
	Aurelien Jarno

On 06/28/2018 02:13 PM, Alex Bennée wrote:
> 
> Cleber Rosa <crosa@redhat.com> writes:
> 
>> On 06/28/2018 11:46 AM, Alex Bennée wrote:
>>>
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Hi,
>>>>
>>>> Another neanderthal approach to add multi-arch acceptance tests using
>>>> Avocado.
>>>>
>>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>>> at using Avocado in QEMU.
>>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>>> a bit further:
>>>> - we can now run other archs
>>>> - Avocado is integrated into Travis CI
>>>>
>>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>>> quite fast:
>>>>
>>>>       ...
>>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>>>     JOB TIME   : 6.21 s
>>>
>>> Hmm however if I do:
>>>
>>>    pip install --user avocado-framework
>>>    avocado run tests/acceptance
>>>
>>> I get:
>>>
>>>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>>>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>>>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
>>
>> Alex,
>>
>> These are errors related to your installation of Avocado-*VT*[1].
>>
>>>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>>>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>>>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
>>>
>>
>> And these from Avocado-*Virt*[2].  Neither of those are used in the
>> tests under "tests/acceptance".
>>
>>> Which I seem to remember from last time I looked into this. I think
>>> avocado makes a bunch of assumptions about the filesystem and OS layout
>>> which is hard to gloss over for something you should be able to run
>>> straight out of the source tree.
>>>
>>>
>>
>> These Avocado-*VT* issues deserve a better look indeed.  Would you
>> please describe (if not here, under a GH issue) your environment?
> 
> Ahh probably from last time I tried to get things up and running. I
> rm -rf ~/.local and things seem to be working well now.

Should we check for a prerequisite version?

> 
> However I was under the impression that VT or virt was needed for this
> testing. Or is this an old assumption?

I don't use neither of these for my qemu-system testing, but I'm more
interested in testing system targets that are different from my host.

I think the VT/virt make sens when you test on a same (target, host)
tuple and worry about perfs.

<snip>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-28 16:23   ` Alex Bennée
@ 2018-06-28 17:40     ` Philippe Mathieu-Daudé
  2018-06-28 18:36       ` Alex Bennée
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-28 17:40 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Aurelien Jarno, Peter Maydell
  Cc: Eduardo Habkost, qemu-devel, Fam Zheng

On 06/28/2018 01:23 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Similar to the BootLinuxConsoleX86_64 test:
>> boot a Linux kernel on a Malta board and verify the serial is working.
>>
>> This test can be run using:
>>
>>     $ avocado run -t endian:big tests/acceptance
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>  1 file changed, 38 insertions(+)
>>
>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>> index 17dc8d58c1..72cf5e943c 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>                  break
>>              if 'Kernel panic - not syncing' in msg:
>>                  self.fail("Kernel panic reached")
>> +
>> +
>> +class BootLinuxConsoleMips(Test):
>> +    """
>> +    Boots a mips Linux kernel and checks that the console is operational
>> +    and the kernel command line is properly passed from QEMU to the kernel
>> +
>> +    :avocado: enable
>> +    :avocado: tags=endian:big
>> +    :avocado: tags=arch:mips
>> +    :avocado: tags=board:malta
>> +    """
>> +
>> +    arch = "mips"
>> +    timeout = 60
>> +
>> +    def test(self):
>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>> +                      'vmlinux-3.2.0-4-4kc-malta')
>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>> +        kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
> 
> I'm uncomfortable using "random" binaries of websites as the source of
> our test kernels. I can see the justification for distro kernels as they
> at least have the infrastructure to rebuild from source if you really
> want to, but even then the distros don't cover a lot of the
> architectures.

And now I notice I made an mistake here :) I guess remember the Avocado
team started using the SHA-1 hash as default and I suggested them to be
able to use other hashes for this particular case, since Aurelien
provided the MD5 hashes signed by his GPG key, which is signed/trusted
by Peter and used to merge mips32 pulls.

That would verify the QEMU community circle of trust right?

I don't think Avocado should parse the FTP/HTTP signed indexes, but a
manual verification when merging this series should suffice.

> 
> I had experimented with using docker based builds for building test
> fixtures (see tests/docker/dockerbuilds):
> 
>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
> 
> As these tests are fairly simple boot tests that just need kernels maybe
> we could look at tooling up the generation of these images in a
> repeatable way - similar to the way vmtest builds VMs.

Yes, I have another acceptance branch where I cross-build an old mipssim
kernel to test the board, using the following:
http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html

But preparing a Docker cross image, fetching the Linux kernel source,
building it, takes a lot of time/storage I'd rather avoid; at least with
Aurelien kernels, since they are known to work since years.

>> +
>> +        self.vm.set_machine('malta')
>> +        self.vm.set_console()
>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>> +        self.vm.add_args('-m', "64",
>> +                         '-kernel', kernel_path,
>> +                         '-append', kernel_command_line)
>> +        self.vm.launch()
>> +        console = self.vm.console_socket.makefile()
>> +        console_logger = logging.getLogger('console')
>> +        while True:
>> +            msg = console.readline()
>> +            console_logger.debug(msg.strip())
>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>> +                break
>> +            if 'Kernel panic - not syncing' in msg:
>> +                self.fail("Kernel panic reached")
> 
> Of course for bonus points a simple rootfs with hackbench or some such
> in it would be nice. But I appreciate this makes the building job a lot
> more complex than just a kernel.

My idea is to use the rootfs for larger tests, and tag the "Kernel
panic" tests as "quick", so we can have a "make acceptance-speed" or
similar.

We can already test than many devices were initialized correctly quickly
looking at this console output.

Regards,

Phil.

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-28 17:40     ` Philippe Mathieu-Daudé
@ 2018-06-28 18:36       ` Alex Bennée
  2018-06-28 22:45         ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 18:36 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Aurelien Jarno, Peter Maydell, Eduardo Habkost,
	qemu-devel, Fam Zheng


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Similar to the BootLinuxConsoleX86_64 test:
>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>
>>> This test can be run using:
>>>
>>>     $ avocado run -t endian:big tests/acceptance
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>  1 file changed, 38 insertions(+)
>>>
>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>> index 17dc8d58c1..72cf5e943c 100644
>>> --- a/tests/acceptance/boot_linux_console.py
>>> +++ b/tests/acceptance/boot_linux_console.py
>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>                  break
>>>              if 'Kernel panic - not syncing' in msg:
>>>                  self.fail("Kernel panic reached")
>>> +
>>> +
>>> +class BootLinuxConsoleMips(Test):
>>> +    """
>>> +    Boots a mips Linux kernel and checks that the console is operational
>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>> +
>>> +    :avocado: enable
>>> +    :avocado: tags=endian:big
>>> +    :avocado: tags=arch:mips
>>> +    :avocado: tags=board:malta
>>> +    """
>>> +
>>> +    arch = "mips"
>>> +    timeout = 60
>>> +
>>> +    def test(self):
>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>> +        kernel_path = self.fetch_asset(kernel_url,
>>> asset_hash=kernel_hash)
>>
>> I'm uncomfortable using "random" binaries of websites as the source of
>> our test kernels. I can see the justification for distro kernels as they
>> at least have the infrastructure to rebuild from source if you really
>> want to, but even then the distros don't cover a lot of the
>> architectures.
>
> And now I notice I made an mistake here :) I guess remember the Avocado
> team started using the SHA-1 hash as default and I suggested them to be
> able to use other hashes for this particular case, since Aurelien
> provided the MD5 hashes signed by his GPG key, which is signed/trusted
> by Peter and used to merge mips32 pulls.
>
> That would verify the QEMU community circle of trust right?

It's not so much the chain of trust and more repeatability of building
the test cases. I trust Aurel32's binaries are good test cases for MIPS
but at the moment I have no idea how to rebuild them which is a bit of
an issue given it is covered by the GPL.

>
> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
> manual verification when merging this series should suffice.
>
>>
>> I had experimented with using docker based builds for building test
>> fixtures (see tests/docker/dockerbuilds):
>>
>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>
>> As these tests are fairly simple boot tests that just need kernels maybe
>> we could look at tooling up the generation of these images in a
>> repeatable way - similar to the way vmtest builds VMs.
>
> Yes, I have another acceptance branch where I cross-build an old mipssim
> kernel to test the board, using the following:
> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>
> But preparing a Docker cross image, fetching the Linux kernel source,
> building it, takes a lot of time/storage I'd rather avoid; at least with
> Aurelien kernels, since they are known to work since years.

Well you can throw away the docker image as long as we have a mechanism
to dump out the final binary. I have no interest in forcing people to
keep checked out code using up space, just that they can re-build if
they need to.

>
>>> +
>>> +        self.vm.set_machine('malta')
>>> +        self.vm.set_console()
>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>> +        self.vm.add_args('-m', "64",
>>> +                         '-kernel', kernel_path,
>>> +                         '-append', kernel_command_line)
>>> +        self.vm.launch()
>>> +        console = self.vm.console_socket.makefile()
>>> +        console_logger = logging.getLogger('console')
>>> +        while True:
>>> +            msg = console.readline()
>>> +            console_logger.debug(msg.strip())
>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>> +                break
>>> +            if 'Kernel panic - not syncing' in msg:
>>> +                self.fail("Kernel panic reached")
>>
>> Of course for bonus points a simple rootfs with hackbench or some such
>> in it would be nice. But I appreciate this makes the building job a lot
>> more complex than just a kernel.
>
> My idea is to use the rootfs for larger tests, and tag the "Kernel
> panic" tests as "quick", so we can have a "make acceptance-speed" or
> similar.
>
> We can already test than many devices were initialized correctly quickly
> looking at this console output.

Yeah the kernel boot up is a pretty good smoke test (it's all most of
kernelci manages as well). However it certainly doesn't fully exercise
the translator. I've lost track of the number of bugs we found after
successfully booting an kernel because we were doing exhaustive
instruction testing.

>
> Regards,
>
> Phil.


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property
  2018-06-28 16:10   ` Alex Bennée
@ 2018-06-28 21:54     ` Alex Bennée
  2018-06-28 22:03       ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Alex Bennée @ 2018-06-28 21:54 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Fam Zheng


Alex Bennée <alex.bennee@linaro.org> writes:

> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Tests can change this property to run tests in other
>> architectures than the host one.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>> index 1e54fd5932..6e9601f5e9 100644
>> --- a/tests/acceptance/avocado_qemu/__init__.py
>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>> @@ -13,6 +13,7 @@ import sys
>>
>>  import avocado
>>
>> +HOST_ARCH = os.uname()[4]
>
> The python docs seem to point to platform as a more stable way of
> querying this stuff:
>
>   platform.machine() => 'x86_64' or 'aarch64'
>
>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>
>>
>> -def pick_default_qemu_bin():
>> +def pick_default_qemu_bin(arch):
>>      """
>>      Picks the path of a QEMU binary, starting either in the current working
>>      directory or in the source tree root directory.
>>      """
>> -    arch = os.uname()[4]
>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>                                            "qemu-system-%s" % arch)
>>      if is_readable_executable_file(qemu_bin_relative_path):
>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>
>>
>>  class Test(avocado.Test):
>> +    _arch = HOST_ARCH

But actually this is wrong - because the host arch may not boot machines
defined in the tests. I suspect each superclass needs to explicitly set
it's arch and we should assert it has done so here.

>> +
>> +    @property
>> +    def arch(self):
>> +        """
>> +        Returns the architecture required to run the current test
>> +        """
>> +        return self._arch
>> +
>>      def setUp(self):
>>          self.vm = None
>> -        self.qemu_bin = self.params.get('qemu_bin',
>> -                                        default=pick_default_qemu_bin())
>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>          if self.qemu_bin is None:
>>              self.cancel("No QEMU binary defined or found in the source tree")
>>          self.vm = QEMUMachine(self.qemu_bin)


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property
  2018-06-28 21:54     ` Alex Bennée
@ 2018-06-28 22:03       ` Philippe Mathieu-Daudé
  2018-07-27  5:35         ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-28 22:03 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa; +Cc: Eduardo Habkost, qemu-devel, Fam Zheng

On 06/28/2018 06:54 PM, Alex Bennée wrote:
> Alex Bennée <alex.bennee@linaro.org> writes:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Tests can change this property to run tests in other
>>> architectures than the host one.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>>> index 1e54fd5932..6e9601f5e9 100644
>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>> @@ -13,6 +13,7 @@ import sys
>>>
>>>  import avocado
>>>
>>> +HOST_ARCH = os.uname()[4]
>>
>> The python docs seem to point to platform as a more stable way of
>> querying this stuff:
>>
>>   platform.machine() => 'x86_64' or 'aarch64'
>>
>>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>>
>>>
>>> -def pick_default_qemu_bin():
>>> +def pick_default_qemu_bin(arch):
>>>      """
>>>      Picks the path of a QEMU binary, starting either in the current working
>>>      directory or in the source tree root directory.
>>>      """
>>> -    arch = os.uname()[4]
>>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>>                                            "qemu-system-%s" % arch)
>>>      if is_readable_executable_file(qemu_bin_relative_path):
>>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>>
>>>
>>>  class Test(avocado.Test):
>>> +    _arch = HOST_ARCH
> 
> But actually this is wrong - because the host arch may not boot machines
> defined in the tests. I suspect each superclass needs to explicitly set
> it's arch and we should assert it has done so here.

Hmm but arch-specific tests are protected by the 'arch' tag:

class BootLinuxConsoleMips(Test):
    """
    :avocado: enable
    :avocado: tags=endian:big
    :avocado: tags=arch:mips
    :avocado: tags=board:malta
    """

Oh no they aren't, it is just a way to filter which selection of tests
to run :|

Cleber can you help us here?

>>> +
>>> +    @property
>>> +    def arch(self):
>>> +        """
>>> +        Returns the architecture required to run the current test
>>> +        """
>>> +        return self._arch
>>> +> 
> 
> --
> Alex Bennée
> 
>>>      def setUp(self):
>>>          self.vm = None
>>> -        self.qemu_bin = self.params.get('qemu_bin',
>>> -                                        default=pick_default_qemu_bin())
>>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>>          if self.qemu_bin is None:
>>>              self.cancel("No QEMU binary defined or found in the source tree")
>>>          self.vm = QEMUMachine(self.qemu_bin)

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-28 18:36       ` Alex Bennée
@ 2018-06-28 22:45         ` Philippe Mathieu-Daudé
  2018-07-04 19:56           ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-06-28 22:45 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa
  Cc: Aurelien Jarno, Peter Maydell, Eduardo Habkost, qemu-devel,
	Fam Zheng, Lukáš Doktor

On 06/28/2018 03:36 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Similar to the BootLinuxConsoleX86_64 test:
>>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>>
>>>> This test can be run using:
>>>>
>>>>     $ avocado run -t endian:big tests/acceptance
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>>  1 file changed, 38 insertions(+)
>>>>
>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>> index 17dc8d58c1..72cf5e943c 100644
>>>> --- a/tests/acceptance/boot_linux_console.py
>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>>                  break
>>>>              if 'Kernel panic - not syncing' in msg:
>>>>                  self.fail("Kernel panic reached")
>>>> +
>>>> +
>>>> +class BootLinuxConsoleMips(Test):
>>>> +    """
>>>> +    Boots a mips Linux kernel and checks that the console is operational
>>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>>> +
>>>> +    :avocado: enable
>>>> +    :avocado: tags=endian:big
>>>> +    :avocado: tags=arch:mips
>>>> +    :avocado: tags=board:malta
>>>> +    """
>>>> +
>>>> +    arch = "mips"
>>>> +    timeout = 60
>>>> +
>>>> +    def test(self):
>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>> asset_hash=kernel_hash)
>>>
>>> I'm uncomfortable using "random" binaries of websites as the source of
>>> our test kernels. I can see the justification for distro kernels as they
>>> at least have the infrastructure to rebuild from source if you really
>>> want to, but even then the distros don't cover a lot of the
>>> architectures.
>>
>> And now I notice I made an mistake here :) I guess remember the Avocado
>> team started using the SHA-1 hash as default and I suggested them to be
>> able to use other hashes for this particular case, since Aurelien
>> provided the MD5 hashes signed by his GPG key, which is signed/trusted
>> by Peter and used to merge mips32 pulls.
>>
>> That would verify the QEMU community circle of trust right?
> 
> It's not so much the chain of trust and more repeatability of building
> the test cases. I trust Aurel32's binaries are good test cases for MIPS
> but at the moment I have no idea how to rebuild them which is a bit of
> an issue given it is covered by the GPL.

OK! I didn't even think about that since this is a "Linux" kernel and a
"Debian" rootfs provided by a Debian developer, hosted on a Debian
server, so I'm pretty sure this is GPL compliant, but it makes sens.

I'll find a way to have reproducible test binaries for acceptance testing.

>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
>> manual verification when merging this series should suffice.
>>
>>>
>>> I had experimented with using docker based builds for building test
>>> fixtures (see tests/docker/dockerbuilds):
>>>
>>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>>
>>> As these tests are fairly simple boot tests that just need kernels maybe
>>> we could look at tooling up the generation of these images in a
>>> repeatable way - similar to the way vmtest builds VMs.
>>
>> Yes, I have another acceptance branch where I cross-build an old mipssim
>> kernel to test the board, using the following:
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>>
>> But preparing a Docker cross image, fetching the Linux kernel source,
>> building it, takes a lot of time/storage I'd rather avoid; at least with
>> Aurelien kernels, since they are known to work since years.
> 
> Well you can throw away the docker image as long as we have a mechanism
> to dump out the final binary. I have no interest in forcing people to
> keep checked out code using up space, just that they can re-build if
> they need to.

The Avocado guys said they have plenty of storage for assets.
A cool feature would be to have some config built by some CI builder
that then sign and push reproducible config + generated binary to the
assets storage automatically.

>>>> +
>>>> +        self.vm.set_machine('malta')
>>>> +        self.vm.set_console()
>>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>>> +        self.vm.add_args('-m', "64",
>>>> +                         '-kernel', kernel_path,
>>>> +                         '-append', kernel_command_line)
>>>> +        self.vm.launch()
>>>> +        console = self.vm.console_socket.makefile()
>>>> +        console_logger = logging.getLogger('console')
>>>> +        while True:
>>>> +            msg = console.readline()
>>>> +            console_logger.debug(msg.strip())
>>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>>> +                break
>>>> +            if 'Kernel panic - not syncing' in msg:
>>>> +                self.fail("Kernel panic reached")
>>>
>>> Of course for bonus points a simple rootfs with hackbench or some such
>>> in it would be nice. But I appreciate this makes the building job a lot
>>> more complex than just a kernel.
>>
>> My idea is to use the rootfs for larger tests, and tag the "Kernel
>> panic" tests as "quick", so we can have a "make acceptance-speed" or
>> similar.
>>
>> We can already test than many devices were initialized correctly quickly
>> looking at this console output.
> 
> Yeah the kernel boot up is a pretty good smoke test (it's all most of
> kernelci manages as well). However it certainly doesn't fully exercise
> the translator. I've lost track of the number of bugs we found after
> successfully booting an kernel because we were doing exhaustive
> instruction testing.

This is not indeed a translator test, but simple enough to test than
devices are instantiated, mapped at the correct address, detected by
Linux, probe the device enough to initialize it.

I actually wrote this test after breaking a I/O device why a previous
Super I/O refactor, this test would have failed.

You could say a simple "info mtree" parser would notice it too... but
acceptance testing is closer to real-world imho, and would be also
useful to catch changes in the guest code (as in "QEMU expects things
that way, but the Linux kernel updated to something new").

Regards,

Phil.

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-06-28 22:45         ` Philippe Mathieu-Daudé
@ 2018-07-04 19:56           ` Philippe Mathieu-Daudé
  2018-07-04 20:47             ` Eduardo Habkost
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-07-04 19:56 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa
  Cc: Aurelien Jarno, Peter Maydell, Eduardo Habkost, qemu-devel,
	Fam Zheng, Lukáš Doktor, David Gibson, BALATON Zoltan,
	Laurent Vivier

[-- Attachment #1: Type: text/plain, Size: 8315 bytes --]

Hi Alex, Cleber.

On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 03:36 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>>
>>>>> Similar to the BootLinuxConsoleX86_64 test:
>>>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>>>
>>>>> This test can be run using:
>>>>>
>>>>>     $ avocado run -t endian:big tests/acceptance
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>> ---
>>>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>>>  1 file changed, 38 insertions(+)
>>>>>
>>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>>> index 17dc8d58c1..72cf5e943c 100644
>>>>> --- a/tests/acceptance/boot_linux_console.py
>>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>>>                  break
>>>>>              if 'Kernel panic - not syncing' in msg:
>>>>>                  self.fail("Kernel panic reached")
>>>>> +
>>>>> +
>>>>> +class BootLinuxConsoleMips(Test):
>>>>> +    """
>>>>> +    Boots a mips Linux kernel and checks that the console is operational
>>>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>>>> +
>>>>> +    :avocado: enable
>>>>> +    :avocado: tags=endian:big
>>>>> +    :avocado: tags=arch:mips
>>>>> +    :avocado: tags=board:malta
>>>>> +    """
>>>>> +
>>>>> +    arch = "mips"
>>>>> +    timeout = 60
>>>>> +
>>>>> +    def test(self):
>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>>> asset_hash=kernel_hash)
>>>>
>>>> I'm uncomfortable using "random" binaries of websites as the source of
>>>> our test kernels. I can see the justification for distro kernels as they
>>>> at least have the infrastructure to rebuild from source if you really
>>>> want to, but even then the distros don't cover a lot of the
>>>> architectures.

Alex: I could find all the Linux kernel I'm interested to console-test
with Avocado on the http://snapshot.debian.org/ archive website.

For example Aurelien's one (more up-to-date) is available here:
http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48

I also added a SH-4 test for the SM501 series of Zoltan BALATON using
the kernel extracted from this distrib built kernel:
http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30

The Debian distribution also provide the source package and the kernels
can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
recent kernels).

Would it be enough to satisfy the GPL requirements to provided that info
in the header and use these handy pre-compiled kernels?

Cleber: As you added tarballs and zip support, I'd also need a "dpkg"
support in your fetch_asset() archive management...

Cleber: And another future request is to be able to extract files from
ISO images with the guestfish tool :P
To be able to add a test for the Apple Machintosh Quadra 800 machine:
http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg08139.html

>>> And now I notice I made an mistake here :) I guess remember the Avocado
>>> team started using the SHA-1 hash as default and I suggested them to be
>>> able to use other hashes for this particular case, since Aurelien
>>> provided the MD5 hashes signed by his GPG key, which is signed/trusted
>>> by Peter and used to merge mips32 pulls.
>>>
>>> That would verify the QEMU community circle of trust right?
>>
>> It's not so much the chain of trust and more repeatability of building
>> the test cases. I trust Aurel32's binaries are good test cases for MIPS
>> but at the moment I have no idea how to rebuild them which is a bit of
>> an issue given it is covered by the GPL.
> 
> OK! I didn't even think about that since this is a "Linux" kernel and a
> "Debian" rootfs provided by a Debian developer, hosted on a Debian
> server, so I'm pretty sure this is GPL compliant, but it makes sens.
> 
> I'll find a way to have reproducible test binaries for acceptance testing.
> 
>>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
>>> manual verification when merging this series should suffice.
>>>
>>>>
>>>> I had experimented with using docker based builds for building test
>>>> fixtures (see tests/docker/dockerbuilds):
>>>>
>>>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>>>
>>>> As these tests are fairly simple boot tests that just need kernels maybe
>>>> we could look at tooling up the generation of these images in a
>>>> repeatable way - similar to the way vmtest builds VMs.
>>>
>>> Yes, I have another acceptance branch where I cross-build an old mipssim
>>> kernel to test the board, using the following:
>>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>>>
>>> But preparing a Docker cross image, fetching the Linux kernel source,
>>> building it, takes a lot of time/storage I'd rather avoid; at least with
>>> Aurelien kernels, since they are known to work since years.
>>
>> Well you can throw away the docker image as long as we have a mechanism
>> to dump out the final binary. I have no interest in forcing people to
>> keep checked out code using up space, just that they can re-build if
>> they need to.
> 
> The Avocado guys said they have plenty of storage for assets.
> A cool feature would be to have some config built by some CI builder
> that then sign and push reproducible config + generated binary to the
> assets storage automatically.
> 
>>>>> +
>>>>> +        self.vm.set_machine('malta')
>>>>> +        self.vm.set_console()
>>>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>>>> +        self.vm.add_args('-m', "64",
>>>>> +                         '-kernel', kernel_path,
>>>>> +                         '-append', kernel_command_line)
>>>>> +        self.vm.launch()
>>>>> +        console = self.vm.console_socket.makefile()
>>>>> +        console_logger = logging.getLogger('console')
>>>>> +        while True:
>>>>> +            msg = console.readline()
>>>>> +            console_logger.debug(msg.strip())
>>>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>>>> +                break
>>>>> +            if 'Kernel panic - not syncing' in msg:
>>>>> +                self.fail("Kernel panic reached")
>>>>
>>>> Of course for bonus points a simple rootfs with hackbench or some such
>>>> in it would be nice. But I appreciate this makes the building job a lot
>>>> more complex than just a kernel.
>>>
>>> My idea is to use the rootfs for larger tests, and tag the "Kernel
>>> panic" tests as "quick", so we can have a "make acceptance-speed" or
>>> similar.
>>>
>>> We can already test than many devices were initialized correctly quickly
>>> looking at this console output.
>>
>> Yeah the kernel boot up is a pretty good smoke test (it's all most of
>> kernelci manages as well). However it certainly doesn't fully exercise
>> the translator. I've lost track of the number of bugs we found after
>> successfully booting an kernel because we were doing exhaustive
>> instruction testing.
> 
> This is not indeed a translator test, but simple enough to test than
> devices are instantiated, mapped at the correct address, detected by
> Linux, probe the device enough to initialize it.
> 
> I actually wrote this test after breaking a I/O device why a previous
> Super I/O refactor, this test would have failed.
> 
> You could say a simple "info mtree" parser would notice it too... but
> acceptance testing is closer to real-world imho, and would be also
> useful to catch changes in the guest code (as in "QEMU expects things
> that way, but the Linux kernel updated to something new").
> 
> Regards,
> 
> Phil.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-07-04 19:56           ` Philippe Mathieu-Daudé
@ 2018-07-04 20:47             ` Eduardo Habkost
  2018-07-04 21:44               ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 28+ messages in thread
From: Eduardo Habkost @ 2018-07-04 20:47 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Alex Bennée, Cleber Rosa, Aurelien Jarno, Peter Maydell,
	qemu-devel, Fam Zheng, Lukáš Doktor, David Gibson,
	BALATON Zoltan, Laurent Vivier

On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> > On 06/28/2018 03:36 PM, Alex Bennée wrote:
> >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
[...]
> >>>>> +    def test(self):
> >>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> >>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
> >>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> >>>>> +        kernel_path = self.fetch_asset(kernel_url,
> >>>>> asset_hash=kernel_hash)
> >>>>
> >>>> I'm uncomfortable using "random" binaries of websites as the source of
> >>>> our test kernels. I can see the justification for distro kernels as they
> >>>> at least have the infrastructure to rebuild from source if you really
> >>>> want to, but even then the distros don't cover a lot of the
> >>>> architectures.
> 
> Alex: I could find all the Linux kernel I'm interested to console-test
> with Avocado on the http://snapshot.debian.org/ archive website.
> 
> For example Aurelien's one (more up-to-date) is available here:
> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
> 
> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
> the kernel extracted from this distrib built kernel:
> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
> 
> The Debian distribution also provide the source package and the kernels
> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
> recent kernels).
> 
> Would it be enough to satisfy the GPL requirements to provided that info
> in the header and use these handy pre-compiled kernels?

(IANAL, etc.)

Personally, I would try to avoid dealing with the "written offer"
option of the GPL, and just publish the sources in the same
medium (HTTP download repository, git repository, etc) as the
binary.

There was a related discussion about distribution of firmware
binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>.
It looks like we never applied the docs/firmware.txt patch,
though.

-- 
Eduardo

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-07-04 20:47             ` Eduardo Habkost
@ 2018-07-04 21:44               ` Philippe Mathieu-Daudé
  2018-07-04 22:32                 ` Eduardo Habkost
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-07-04 21:44 UTC (permalink / raw)
  To: Eduardo Habkost
  Cc: Alex Bennée, Cleber Rosa, Aurelien Jarno, Peter Maydell,
	qemu-devel, Fam Zheng, Lukáš Doktor, David Gibson,
	BALATON Zoltan, Laurent Vivier, John Snow

On 07/04/2018 05:47 PM, Eduardo Habkost wrote:
> On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
>> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
>>> On 06/28/2018 03:36 PM, Alex Bennée wrote:
>>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
> [...]
>>>>>>> +    def test(self):
>>>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>>>>> asset_hash=kernel_hash)
>>>>>>
>>>>>> I'm uncomfortable using "random" binaries of websites as the source of
>>>>>> our test kernels. I can see the justification for distro kernels as they
>>>>>> at least have the infrastructure to rebuild from source if you really
>>>>>> want to, but even then the distros don't cover a lot of the
>>>>>> architectures.
>>
>> Alex: I could find all the Linux kernel I'm interested to console-test
>> with Avocado on the http://snapshot.debian.org/ archive website.
>>
>> For example Aurelien's one (more up-to-date) is available here:
>> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
>>
>> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
>> the kernel extracted from this distrib built kernel:
>> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
>>
>> The Debian distribution also provide the source package and the kernels
>> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
>> recent kernels).
>>
>> Would it be enough to satisfy the GPL requirements to provided that info
>> in the header and use these handy pre-compiled kernels?
> 
> (IANAL, etc.)
> 
> Personally, I would try to avoid dealing with the "written offer"
> option of the GPL, and just publish the sources in the same
> medium (HTTP download repository, git repository, etc) as the
> binary.

About Debian packaged kernel: the sources are in the same directory:

http://snapshot.debian.org/package/linux-2.6/2.6.32-30/

linux-2.6_2.6.32.orig.tar.gz
linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb
linux-support-2.6.32-5_2.6.32-30_all.deb

The other way, trying to compile a 2013 kernel is painful because the
oldest GCC I succeed installing complain about many errors.

The next absurd option I have is install a 2013 userland distrib and use
qemu-user to cross^H^H^H^H(?? I dunno how to name it)-compile this
kernel.... "backward-compile"? This might make sens with the coming
quantum computers.

> There was a related discussion about distribution of firmware
> binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>.
> It looks like we never applied the docs/firmware.txt patch,
> though.

This case is not about firmware we need to run QEMU, but Linux kernel to
run acceptance tests using QEMU. But same legal stuffs, you know...

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test
  2018-07-04 21:44               ` Philippe Mathieu-Daudé
@ 2018-07-04 22:32                 ` Eduardo Habkost
  0 siblings, 0 replies; 28+ messages in thread
From: Eduardo Habkost @ 2018-07-04 22:32 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Alex Bennée, Cleber Rosa, Aurelien Jarno, Peter Maydell,
	qemu-devel, Fam Zheng, Lukáš Doktor, David Gibson,
	BALATON Zoltan, Laurent Vivier, John Snow

On Wed, Jul 04, 2018 at 06:44:02PM -0300, Philippe Mathieu-Daudé wrote:
> On 07/04/2018 05:47 PM, Eduardo Habkost wrote:
> > On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
> >> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> >>> On 06/28/2018 03:36 PM, Alex Bennée wrote:
> >>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
> > [...]
> >>>>>>> +    def test(self):
> >>>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> >>>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
> >>>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> >>>>>>> +        kernel_path = self.fetch_asset(kernel_url,
> >>>>>>> asset_hash=kernel_hash)
> >>>>>>
> >>>>>> I'm uncomfortable using "random" binaries of websites as the source of
> >>>>>> our test kernels. I can see the justification for distro kernels as they
> >>>>>> at least have the infrastructure to rebuild from source if you really
> >>>>>> want to, but even then the distros don't cover a lot of the
> >>>>>> architectures.
> >>
> >> Alex: I could find all the Linux kernel I'm interested to console-test
> >> with Avocado on the http://snapshot.debian.org/ archive website.
> >>
> >> For example Aurelien's one (more up-to-date) is available here:
> >> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
> >>
> >> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
> >> the kernel extracted from this distrib built kernel:
> >> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
> >>
> >> The Debian distribution also provide the source package and the kernels
> >> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
> >> recent kernels).
> >>
> >> Would it be enough to satisfy the GPL requirements to provided that info
> >> in the header and use these handy pre-compiled kernels?
> > 
> > (IANAL, etc.)
> > 
> > Personally, I would try to avoid dealing with the "written offer"
> > option of the GPL, and just publish the sources in the same
> > medium (HTTP download repository, git repository, etc) as the
> > binary.
> 
> About Debian packaged kernel: the sources are in the same directory:
> 
> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/
> 
> linux-2.6_2.6.32.orig.tar.gz
> linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb
> linux-support-2.6.32-5_2.6.32-30_all.deb

Oh, I thought you were talking about publishing the binaries
elsewhere.  If we're not the ones publishing the binaries (we're
just downloading it), then this is somebody else's problem.  :)

Would it be OK to make test code download directly from
snapshot.debian.org, though?  Their home page have instructions
to use apt repositories there, so I guess it's reasonable use of
the service.

-- 
Eduardo

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-06-28 16:01 ` Alex Bennée
@ 2018-07-27  5:33   ` Philippe Mathieu-Daudé
  2018-07-27 16:30     ` Alex Bennée
  0 siblings, 1 reply; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-07-27  5:33 UTC (permalink / raw)
  To: Alex Bennée
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Lukáš Doktor,
	Fam Zheng, Aurelien Jarno

Hi Alex,

On 06/28/2018 01:01 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>> Another neanderthal approach to add multi-arch acceptance tests using
>> Avocado.
>>
>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>> at using Avocado in QEMU.
>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>> a bit further:
>> - we can now run other archs
>> - Avocado is integrated into Travis CI
>>
>> I'm not sure Travis is the best CI available for those tests, but it runs
>> quite fast:
>>
>>       ...
>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>       INTERRUPT 0 | CANCEL 0
> 
> That said on my AArch64 server it fairs better:
> 
> 15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
> JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
> JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
>  (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
>  (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
>  (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
>  (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
>  (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
>  (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
>  (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
> JOB TIME   : 2.47 s
> 
> Although given the errors in the logs I'm surprised at ERROR 0

Why are you surprised? You were expecting the BootLinuxConsoleMips to fail?

> 
>>     JOB TIME   : 6.21 s
>>
>>     Done. Your build exited with 0.
>>
>>     Job ran for 5 min 34 sec
>>
>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>
>> Cleber: I hope you can help me pythonizing this series :p
>>
>> Regards,
>>
>> Phil.
>>
>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>
>> Philippe Mathieu-Daudé (6):
>>   avocado: Add a Test.arch property
>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>   tests/acceptance: Improve the Avocado tags
>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>   tests/acceptance: Add a kludge to not use the default console
>>   travis: Add Avocado tests
>>
>>  .travis.yml                               |  8 +++++
>>  scripts/qemu.py                           |  6 ++--
>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>  4 files changed, 67 insertions(+), 8 deletions(-)
> 
> 
> --
> Alex Bennée
> 

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property
  2018-06-28 22:03       ` Philippe Mathieu-Daudé
@ 2018-07-27  5:35         ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 28+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-07-27  5:35 UTC (permalink / raw)
  To: Alex Bennée, Cleber Rosa, Eduardo Habkost; +Cc: qemu-devel, Fam Zheng

Hi Cleber,

On 06/28/2018 07:03 PM, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 06:54 PM, Alex Bennée wrote:
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Tests can change this property to run tests in other
>>>> architectures than the host one.
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>>>> index 1e54fd5932..6e9601f5e9 100644
>>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>>> @@ -13,6 +13,7 @@ import sys
>>>>
>>>>  import avocado
>>>>
>>>> +HOST_ARCH = os.uname()[4]
>>>
>>> The python docs seem to point to platform as a more stable way of
>>> querying this stuff:
>>>
>>>   platform.machine() => 'x86_64' or 'aarch64'
>>>
>>>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>>>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>>>
>>>>
>>>> -def pick_default_qemu_bin():
>>>> +def pick_default_qemu_bin(arch):
>>>>      """
>>>>      Picks the path of a QEMU binary, starting either in the current working
>>>>      directory or in the source tree root directory.
>>>>      """
>>>> -    arch = os.uname()[4]
>>>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>>>                                            "qemu-system-%s" % arch)
>>>>      if is_readable_executable_file(qemu_bin_relative_path):
>>>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>>>
>>>>
>>>>  class Test(avocado.Test):
>>>> +    _arch = HOST_ARCH
>>
>> But actually this is wrong - because the host arch may not boot machines
>> defined in the tests. I suspect each superclass needs to explicitly set
>> it's arch and we should assert it has done so here.

Do you have any suggestion on how we could clear this?

> 
> Hmm but arch-specific tests are protected by the 'arch' tag:
> 
> class BootLinuxConsoleMips(Test):
>     """
>     :avocado: enable
>     :avocado: tags=endian:big
>     :avocado: tags=arch:mips
>     :avocado: tags=board:malta
>     """
> 
> Oh no they aren't, it is just a way to filter which selection of tests
> to run :|

>>>> +
>>>> +    @property
>>>> +    def arch(self):
>>>> +        """
>>>> +        Returns the architecture required to run the current test
>>>> +        """
>>>> +        return self._arch
>>>> +> 
>>
>> --
>> Alex Bennée
>>
>>>>      def setUp(self):
>>>>          self.vm = None
>>>> -        self.qemu_bin = self.params.get('qemu_bin',
>>>> -                                        default=pick_default_qemu_bin())
>>>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>>>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>>>          if self.qemu_bin is None:
>>>>              self.cancel("No QEMU binary defined or found in the source tree")
>>>>          self.vm = QEMUMachine(self.qemu_bin)

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job
  2018-07-27  5:33   ` Philippe Mathieu-Daudé
@ 2018-07-27 16:30     ` Alex Bennée
  0 siblings, 0 replies; 28+ messages in thread
From: Alex Bennée @ 2018-07-27 16:30 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Cleber Rosa, Eduardo Habkost, qemu-devel, Lukáš Doktor,
	Fam Zheng, Aurelien Jarno


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 06/28/2018 01:01 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>> Another neanderthal approach to add multi-arch acceptance tests using
>>> Avocado.
>>>
>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>> at using Avocado in QEMU.
>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>> a bit further:
>>> - we can now run other archs
>>> - Avocado is integrated into Travis CI
>>>
>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>> quite fast:
>>>
>>>       ...
>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>>       INTERRUPT 0 | CANCEL 0
>>
>> That said on my AArch64 server it fairs better:
>>
>> 15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
>> JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
>> JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
>>  (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
>>  (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
>>  (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
>>  (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
>>  (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
>>  (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
>>  (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
>> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
>> JOB TIME   : 2.47 s
>>
>> Although given the errors in the logs I'm surprised at ERROR 0
>
> Why are you surprised? You were expecting the BootLinuxConsoleMips to
> fail?

No but a bunch have CANCELed... maybe they are skipping but if the test
is run and it fails to complete I would expect that to be an error.

>
>>
>>>     JOB TIME   : 6.21 s
>>>
>>>     Done. Your build exited with 0.
>>>
>>>     Job ran for 5 min 34 sec
>>>
>>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>>
>>> Cleber: I hope you can help me pythonizing this series :p
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>>
>>> Philippe Mathieu-Daudé (6):
>>>   avocado: Add a Test.arch property
>>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>>   tests/acceptance: Improve the Avocado tags
>>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>>   tests/acceptance: Add a kludge to not use the default console
>>>   travis: Add Avocado tests
>>>
>>>  .travis.yml                               |  8 +++++
>>>  scripts/qemu.py                           |  6 ++--
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>>  4 files changed, 67 insertions(+), 8 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

^ permalink raw reply	[flat|nested] 28+ messages in thread

end of thread, other threads:[~2018-07-27 16:30 UTC | newest]

Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-22  0:44 [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Philippe Mathieu-Daudé
2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property Philippe Mathieu-Daudé
2018-06-28 16:10   ` Alex Bennée
2018-06-28 21:54     ` Alex Bennée
2018-06-28 22:03       ` Philippe Mathieu-Daudé
2018-07-27  5:35         ` Philippe Mathieu-Daudé
2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test Philippe Mathieu-Daudé
2018-06-28 17:17   ` Alex Bennée
2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 3/6] tests/acceptance: Improve the Avocado tags Philippe Mathieu-Daudé
2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test Philippe Mathieu-Daudé
2018-06-28 16:23   ` Alex Bennée
2018-06-28 17:40     ` Philippe Mathieu-Daudé
2018-06-28 18:36       ` Alex Bennée
2018-06-28 22:45         ` Philippe Mathieu-Daudé
2018-07-04 19:56           ` Philippe Mathieu-Daudé
2018-07-04 20:47             ` Eduardo Habkost
2018-07-04 21:44               ` Philippe Mathieu-Daudé
2018-07-04 22:32                 ` Eduardo Habkost
2018-06-22  0:44 ` [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console Philippe Mathieu-Daudé
2018-06-28 16:33   ` Alex Bennée
2018-06-22  0:44 ` [Qemu-devel] [PATCH v2 6/6] travis: Add Avocado tests Philippe Mathieu-Daudé
2018-06-28 15:46 ` [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job Alex Bennée
2018-06-28 16:26   ` Cleber Rosa
2018-06-28 17:13     ` Alex Bennée
2018-06-28 17:24       ` Philippe Mathieu-Daudé
2018-06-28 16:01 ` Alex Bennée
2018-07-27  5:33   ` Philippe Mathieu-Daudé
2018-07-27 16:30     ` Alex Bennée

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.