[Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Philippe Mathieu-Daudé]

Hi,

Another neanderthal approach to add multi-arch acceptance tests using
Avocado.

Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
at using Avocado in QEMU.
Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
a bit further:
- we can now run other archs
- Avocado is integrated into Travis CI

I'm not sure Travis is the best CI available for those tests, but it runs
quite fast:

      ...
      LINK    x86_64-softmmu/qemu-system-x86_64
    JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
    JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
     (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
     (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
     (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
     (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
     (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
     (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
     (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
    RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 6.21 s

    Done. Your build exited with 0.

    Job ran for 5 min 34 sec

I couldn't add the Alpha tests because all the vmlinux kernels I found
online are gzipped, so I'm waiting some upstream Avocado merges before.

Cleber: I hope you can help me pythonizing this series :p

Regards,

Phil.

[0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
[1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html

Philippe Mathieu-Daudé (6):
  avocado: Add a Test.arch property
  tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
  tests/acceptance: Improve the Avocado tags
  tests/acceptance: Add a BootLinuxConsoleMips test
  tests/acceptance: Add a kludge to not use the default console
  travis: Add Avocado tests

 .travis.yml                               |  8 +++++
 scripts/qemu.py                           |  6 ++--
 tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
 tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
 4 files changed, 67 insertions(+), 8 deletions(-)

-- 
2.18.0.rc2

[Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property

[Philippe Mathieu-Daudé]

Tests can change this property to run tests in other
architectures than the host one.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 1e54fd5932..6e9601f5e9 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -13,6 +13,7 @@ import sys
 
 import avocado
 
+HOST_ARCH = os.uname()[4]
 SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
 SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
@@ -23,12 +24,11 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin():
+def pick_default_qemu_bin(arch):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
     """
-    arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
                                           "qemu-system-%s" % arch)
     if is_readable_executable_file(qemu_bin_relative_path):
@@ -41,10 +41,19 @@ def pick_default_qemu_bin():
 
 
 class Test(avocado.Test):
+    _arch = HOST_ARCH
+
+    @property
+    def arch(self):
+        """
+        Returns the architecture required to run the current test
+        """
+        return self._arch
+
     def setUp(self):
         self.vm = None
-        self.qemu_bin = self.params.get('qemu_bin',
-                                        default=pick_default_qemu_bin())
+        qemu_bin = pick_default_qemu_bin(self.arch)
+        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
         self.vm = QEMUMachine(self.qemu_bin)
-- 
2.18.0.rc2

Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Tests can change this property to run tests in other
> architectures than the host one.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index 1e54fd5932..6e9601f5e9 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -13,6 +13,7 @@ import sys
>
>  import avocado
>
> +HOST_ARCH = os.uname()[4]

The python docs seem to point to platform as a more stable way of
querying this stuff:

  platform.machine() => 'x86_64' or 'aarch64'

>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>
>
> -def pick_default_qemu_bin():
> +def pick_default_qemu_bin(arch):
>      """
>      Picks the path of a QEMU binary, starting either in the current working
>      directory or in the source tree root directory.
>      """
> -    arch = os.uname()[4]
>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>                                            "qemu-system-%s" % arch)
>      if is_readable_executable_file(qemu_bin_relative_path):
> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>
>
>  class Test(avocado.Test):
> +    _arch = HOST_ARCH
> +
> +    @property
> +    def arch(self):
> +        """
> +        Returns the architecture required to run the current test
> +        """
> +        return self._arch
> +
>      def setUp(self):
>          self.vm = None
> -        self.qemu_bin = self.params.get('qemu_bin',
> -                                        default=pick_default_qemu_bin())
> +        qemu_bin = pick_default_qemu_bin(self.arch)
> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>          if self.qemu_bin is None:
>              self.cancel("No QEMU binary defined or found in the source tree")
>          self.vm = QEMUMachine(self.qemu_bin)


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Tests can change this property to run tests in other
>> architectures than the host one.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>> index 1e54fd5932..6e9601f5e9 100644
>> --- a/tests/acceptance/avocado_qemu/__init__.py
>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>> @@ -13,6 +13,7 @@ import sys
>>
>>  import avocado
>>
>> +HOST_ARCH = os.uname()[4]
>
> The python docs seem to point to platform as a more stable way of
> querying this stuff:
>
>   platform.machine() => 'x86_64' or 'aarch64'
>
>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>
>>
>> -def pick_default_qemu_bin():
>> +def pick_default_qemu_bin(arch):
>>      """
>>      Picks the path of a QEMU binary, starting either in the current working
>>      directory or in the source tree root directory.
>>      """
>> -    arch = os.uname()[4]
>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>                                            "qemu-system-%s" % arch)
>>      if is_readable_executable_file(qemu_bin_relative_path):
>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>
>>
>>  class Test(avocado.Test):
>> +    _arch = HOST_ARCH

But actually this is wrong - because the host arch may not boot machines
defined in the tests. I suspect each superclass needs to explicitly set
it's arch and we should assert it has done so here.

>> +
>> +    @property
>> +    def arch(self):
>> +        """
>> +        Returns the architecture required to run the current test
>> +        """
>> +        return self._arch
>> +
>>      def setUp(self):
>>          self.vm = None
>> -        self.qemu_bin = self.params.get('qemu_bin',
>> -                                        default=pick_default_qemu_bin())
>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>          if self.qemu_bin is None:
>>              self.cancel("No QEMU binary defined or found in the source tree")
>>          self.vm = QEMUMachine(self.qemu_bin)


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property

[Philippe Mathieu-Daudé]

On 06/28/2018 06:54 PM, Alex Bennée wrote:
> Alex Bennée <alex.bennee@linaro.org> writes:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Tests can change this property to run tests in other
>>> architectures than the host one.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>>> index 1e54fd5932..6e9601f5e9 100644
>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>> @@ -13,6 +13,7 @@ import sys
>>>
>>>  import avocado
>>>
>>> +HOST_ARCH = os.uname()[4]
>>
>> The python docs seem to point to platform as a more stable way of
>> querying this stuff:
>>
>>   platform.machine() => 'x86_64' or 'aarch64'
>>
>>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>>
>>>
>>> -def pick_default_qemu_bin():
>>> +def pick_default_qemu_bin(arch):
>>>      """
>>>      Picks the path of a QEMU binary, starting either in the current working
>>>      directory or in the source tree root directory.
>>>      """
>>> -    arch = os.uname()[4]
>>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>>                                            "qemu-system-%s" % arch)
>>>      if is_readable_executable_file(qemu_bin_relative_path):
>>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>>
>>>
>>>  class Test(avocado.Test):
>>> +    _arch = HOST_ARCH
> 
> But actually this is wrong - because the host arch may not boot machines
> defined in the tests. I suspect each superclass needs to explicitly set
> it's arch and we should assert it has done so here.

Hmm but arch-specific tests are protected by the 'arch' tag:

class BootLinuxConsoleMips(Test):
    """
    :avocado: enable
    :avocado: tags=endian:big
    :avocado: tags=arch:mips
    :avocado: tags=board:malta
    """

Oh no they aren't, it is just a way to filter which selection of tests
to run :|

Cleber can you help us here?

>>> +
>>> +    @property
>>> +    def arch(self):
>>> +        """
>>> +        Returns the architecture required to run the current test
>>> +        """
>>> +        return self._arch
>>> +> 
> 
> --
> Alex Bennée
> 
>>>      def setUp(self):
>>>          self.vm = None
>>> -        self.qemu_bin = self.params.get('qemu_bin',
>>> -                                        default=pick_default_qemu_bin())
>>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>>          if self.qemu_bin is None:
>>>              self.cancel("No QEMU binary defined or found in the source tree")
>>>          self.vm = QEMUMachine(self.qemu_bin)

Re: [Qemu-devel] [RFC PATCH v2 1/6] avocado: Add a Test.arch property

[Philippe Mathieu-Daudé]

Hi Cleber,

On 06/28/2018 07:03 PM, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 06:54 PM, Alex Bennée wrote:
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Tests can change this property to run tests in other
>>>> architectures than the host one.
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>  tests/acceptance/avocado_qemu/__init__.py | 17 +++++++++++++----
>>>>  1 file changed, 13 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
>>>> index 1e54fd5932..6e9601f5e9 100644
>>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>>> @@ -13,6 +13,7 @@ import sys
>>>>
>>>>  import avocado
>>>>
>>>> +HOST_ARCH = os.uname()[4]
>>>
>>> The python docs seem to point to platform as a more stable way of
>>> querying this stuff:
>>>
>>>   platform.machine() => 'x86_64' or 'aarch64'
>>>
>>>>  SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
>>>>  SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
>>>>  sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
>>>> @@ -23,12 +24,11 @@ def is_readable_executable_file(path):
>>>>      return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>>>>
>>>>
>>>> -def pick_default_qemu_bin():
>>>> +def pick_default_qemu_bin(arch):
>>>>      """
>>>>      Picks the path of a QEMU binary, starting either in the current working
>>>>      directory or in the source tree root directory.
>>>>      """
>>>> -    arch = os.uname()[4]
>>>>      qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
>>>>                                            "qemu-system-%s" % arch)
>>>>      if is_readable_executable_file(qemu_bin_relative_path):
>>>> @@ -41,10 +41,19 @@ def pick_default_qemu_bin():
>>>>
>>>>
>>>>  class Test(avocado.Test):
>>>> +    _arch = HOST_ARCH
>>
>> But actually this is wrong - because the host arch may not boot machines
>> defined in the tests. I suspect each superclass needs to explicitly set
>> it's arch and we should assert it has done so here.

Do you have any suggestion on how we could clear this?

> 
> Hmm but arch-specific tests are protected by the 'arch' tag:
> 
> class BootLinuxConsoleMips(Test):
>     """
>     :avocado: enable
>     :avocado: tags=endian:big
>     :avocado: tags=arch:mips
>     :avocado: tags=board:malta
>     """
> 
> Oh no they aren't, it is just a way to filter which selection of tests
> to run :|

>>>> +
>>>> +    @property
>>>> +    def arch(self):
>>>> +        """
>>>> +        Returns the architecture required to run the current test
>>>> +        """
>>>> +        return self._arch
>>>> +> 
>>
>> --
>> Alex Bennée
>>
>>>>      def setUp(self):
>>>>          self.vm = None
>>>> -        self.qemu_bin = self.params.get('qemu_bin',
>>>> -                                        default=pick_default_qemu_bin())
>>>> +        qemu_bin = pick_default_qemu_bin(self.arch)
>>>> +        self.qemu_bin = self.params.get('qemu_bin', default=qemu_bin)
>>>>          if self.qemu_bin is None:
>>>>              self.cancel("No QEMU binary defined or found in the source tree")
>>>>          self.vm = QEMUMachine(self.qemu_bin)

[Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 98324f7591..f6a34d75a5 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -13,7 +13,7 @@ import logging
 from avocado_qemu import Test
 
 
-class BootLinuxConsole(Test):
+class BootLinuxConsoleX86_64(Test):
     """
     Boots a x86_64 Linux kernel and checks that the console is operational
     and the kernel command line is properly passed from QEMU to the kernel
-- 
2.18.0.rc2

Re: [Qemu-devel] [PATCH v2 2/6] tests/acceptance: Rename the x86-64 specific BootLinuxConsole test

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 98324f7591..f6a34d75a5 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -13,7 +13,7 @@ import logging
>  from avocado_qemu import Test
>
>
> -class BootLinuxConsole(Test):
> +class BootLinuxConsoleX86_64(Test):
>      """
>      Boots a x86_64 Linux kernel and checks that the console is operational
>      and the kernel command line is properly passed from QEMU to the kernel

Not your bug but it also needs to be better at choosing a QEMU to run:

2018-06-28 16:10:24,736 download         L0066 INFO | Fetching https://mirrors.kernel.org/fedora/releases/28/Everything/x86_64/os/images/pxeboot/vmlinuz -> /home/alex.bennee/avocado/data/cache/vmlinuz.gqWhs5
2018-06-28 16:10:40,841 qemu             L0268 DEBUG| Error launching VM
2018-06-28 16:10:40,841 qemu             L0270 DEBUG| Command: 'aarch64-softmmu/qemu-system-aarch64 -chardev socket,id=mon,path=/var/tmp/tmpdXbtIA/qemu-16646-monitor.sock -mon chardev=mon,mode=control -display none -vga none -machine pc -chardev socket,id=console,path=/var/tmp/tmpdXbtIA/qemu-16646-console.sock,server,nowait -device isa-serial,chardev=console -kernel /home/alex.bennee/avocado/data/cache/vmlinuz -append console=ttyS0'


--
Alex Bennée

[Qemu-devel] [PATCH v2 3/6] tests/acceptance: Improve the Avocado tags

[Philippe Mathieu-Daudé]

Introduce 'arch' and 'endian' namespaces.

For example, all little-endian tests can be run using:

    $ avocado run -t endian:little tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index f6a34d75a5..17dc8d58c1 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -19,7 +19,8 @@ class BootLinuxConsoleX86_64(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=x86_64
+    :avocado: tags=endian:little
+    :avocado: tags=arch:x86_64
     """
 
     timeout = 60
-- 
2.18.0.rc2

[Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

Similar to the BootLinuxConsoleX86_64 test:
boot a Linux kernel on a Malta board and verify the serial is working.

This test can be run using:

    $ avocado run -t endian:big tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 17dc8d58c1..72cf5e943c 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
                 break
             if 'Kernel panic - not syncing' in msg:
                 self.fail("Kernel panic reached")
+
+
+class BootLinuxConsoleMips(Test):
+    """
+    Boots a mips Linux kernel and checks that the console is operational
+    and the kernel command line is properly passed from QEMU to the kernel
+
+    :avocado: enable
+    :avocado: tags=endian:big
+    :avocado: tags=arch:mips
+    :avocado: tags=board:malta
+    """
+
+    arch = "mips"
+    timeout = 60
+
+    def test(self):
+        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
+                      'vmlinux-3.2.0-4-4kc-malta')
+        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = 'console=ttyS0 printk.time=0'
+        self.vm.add_args('-m', "64",
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline()
+            console_logger.debug(msg.strip())
+            if 'Kernel command line: %s' % kernel_command_line in msg:
+                break
+            if 'Kernel panic - not syncing' in msg:
+                self.fail("Kernel panic reached")
-- 
2.18.0.rc2

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Similar to the BootLinuxConsoleX86_64 test:
> boot a Linux kernel on a Malta board and verify the serial is working.
>
> This test can be run using:
>
>     $ avocado run -t endian:big tests/acceptance
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 17dc8d58c1..72cf5e943c 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>                  break
>              if 'Kernel panic - not syncing' in msg:
>                  self.fail("Kernel panic reached")
> +
> +
> +class BootLinuxConsoleMips(Test):
> +    """
> +    Boots a mips Linux kernel and checks that the console is operational
> +    and the kernel command line is properly passed from QEMU to the kernel
> +
> +    :avocado: enable
> +    :avocado: tags=endian:big
> +    :avocado: tags=arch:mips
> +    :avocado: tags=board:malta
> +    """
> +
> +    arch = "mips"
> +    timeout = 60
> +
> +    def test(self):
> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> +                      'vmlinux-3.2.0-4-4kc-malta')
> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)

I'm uncomfortable using "random" binaries of websites as the source of
our test kernels. I can see the justification for distro kernels as they
at least have the infrastructure to rebuild from source if you really
want to, but even then the distros don't cover a lot of the
architectures.

I had experimented with using docker based builds for building test
fixtures (see tests/docker/dockerbuilds):

   https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2

As these tests are fairly simple boot tests that just need kernels maybe
we could look at tooling up the generation of these images in a
repeatable way - similar to the way vmtest builds VMs.

> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = 'console=ttyS0 printk.time=0'
> +        self.vm.add_args('-m', "64",
> +                         '-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console = self.vm.console_socket.makefile()
> +        console_logger = logging.getLogger('console')
> +        while True:
> +            msg = console.readline()
> +            console_logger.debug(msg.strip())
> +            if 'Kernel command line: %s' % kernel_command_line in msg:
> +                break
> +            if 'Kernel panic - not syncing' in msg:
> +                self.fail("Kernel panic reached")

Of course for bonus points a simple rootfs with hackbench or some such
in it would be nice. But I appreciate this makes the building job a lot
more complex than just a kernel.

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

On 06/28/2018 01:23 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Similar to the BootLinuxConsoleX86_64 test:
>> boot a Linux kernel on a Malta board and verify the serial is working.
>>
>> This test can be run using:
>>
>>     $ avocado run -t endian:big tests/acceptance
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>  1 file changed, 38 insertions(+)
>>
>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>> index 17dc8d58c1..72cf5e943c 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>                  break
>>              if 'Kernel panic - not syncing' in msg:
>>                  self.fail("Kernel panic reached")
>> +
>> +
>> +class BootLinuxConsoleMips(Test):
>> +    """
>> +    Boots a mips Linux kernel and checks that the console is operational
>> +    and the kernel command line is properly passed from QEMU to the kernel
>> +
>> +    :avocado: enable
>> +    :avocado: tags=endian:big
>> +    :avocado: tags=arch:mips
>> +    :avocado: tags=board:malta
>> +    """
>> +
>> +    arch = "mips"
>> +    timeout = 60
>> +
>> +    def test(self):
>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>> +                      'vmlinux-3.2.0-4-4kc-malta')
>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>> +        kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
> 
> I'm uncomfortable using "random" binaries of websites as the source of
> our test kernels. I can see the justification for distro kernels as they
> at least have the infrastructure to rebuild from source if you really
> want to, but even then the distros don't cover a lot of the
> architectures.

And now I notice I made an mistake here :) I guess remember the Avocado
team started using the SHA-1 hash as default and I suggested them to be
able to use other hashes for this particular case, since Aurelien
provided the MD5 hashes signed by his GPG key, which is signed/trusted
by Peter and used to merge mips32 pulls.

That would verify the QEMU community circle of trust right?

I don't think Avocado should parse the FTP/HTTP signed indexes, but a
manual verification when merging this series should suffice.

> 
> I had experimented with using docker based builds for building test
> fixtures (see tests/docker/dockerbuilds):
> 
>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
> 
> As these tests are fairly simple boot tests that just need kernels maybe
> we could look at tooling up the generation of these images in a
> repeatable way - similar to the way vmtest builds VMs.

Yes, I have another acceptance branch where I cross-build an old mipssim
kernel to test the board, using the following:
http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html

But preparing a Docker cross image, fetching the Linux kernel source,
building it, takes a lot of time/storage I'd rather avoid; at least with
Aurelien kernels, since they are known to work since years.

>> +
>> +        self.vm.set_machine('malta')
>> +        self.vm.set_console()
>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>> +        self.vm.add_args('-m', "64",
>> +                         '-kernel', kernel_path,
>> +                         '-append', kernel_command_line)
>> +        self.vm.launch()
>> +        console = self.vm.console_socket.makefile()
>> +        console_logger = logging.getLogger('console')
>> +        while True:
>> +            msg = console.readline()
>> +            console_logger.debug(msg.strip())
>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>> +                break
>> +            if 'Kernel panic - not syncing' in msg:
>> +                self.fail("Kernel panic reached")
> 
> Of course for bonus points a simple rootfs with hackbench or some such
> in it would be nice. But I appreciate this makes the building job a lot
> more complex than just a kernel.

My idea is to use the rootfs for larger tests, and tag the "Kernel
panic" tests as "quick", so we can have a "make acceptance-speed" or
similar.

We can already test than many devices were initialized correctly quickly
looking at this console output.

Regards,

Phil.

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Similar to the BootLinuxConsoleX86_64 test:
>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>
>>> This test can be run using:
>>>
>>>     $ avocado run -t endian:big tests/acceptance
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>  1 file changed, 38 insertions(+)
>>>
>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>> index 17dc8d58c1..72cf5e943c 100644
>>> --- a/tests/acceptance/boot_linux_console.py
>>> +++ b/tests/acceptance/boot_linux_console.py
>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>                  break
>>>              if 'Kernel panic - not syncing' in msg:
>>>                  self.fail("Kernel panic reached")
>>> +
>>> +
>>> +class BootLinuxConsoleMips(Test):
>>> +    """
>>> +    Boots a mips Linux kernel and checks that the console is operational
>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>> +
>>> +    :avocado: enable
>>> +    :avocado: tags=endian:big
>>> +    :avocado: tags=arch:mips
>>> +    :avocado: tags=board:malta
>>> +    """
>>> +
>>> +    arch = "mips"
>>> +    timeout = 60
>>> +
>>> +    def test(self):
>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>> +        kernel_path = self.fetch_asset(kernel_url,
>>> asset_hash=kernel_hash)
>>
>> I'm uncomfortable using "random" binaries of websites as the source of
>> our test kernels. I can see the justification for distro kernels as they
>> at least have the infrastructure to rebuild from source if you really
>> want to, but even then the distros don't cover a lot of the
>> architectures.
>
> And now I notice I made an mistake here :) I guess remember the Avocado
> team started using the SHA-1 hash as default and I suggested them to be
> able to use other hashes for this particular case, since Aurelien
> provided the MD5 hashes signed by his GPG key, which is signed/trusted
> by Peter and used to merge mips32 pulls.
>
> That would verify the QEMU community circle of trust right?

It's not so much the chain of trust and more repeatability of building
the test cases. I trust Aurel32's binaries are good test cases for MIPS
but at the moment I have no idea how to rebuild them which is a bit of
an issue given it is covered by the GPL.

>
> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
> manual verification when merging this series should suffice.
>
>>
>> I had experimented with using docker based builds for building test
>> fixtures (see tests/docker/dockerbuilds):
>>
>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>
>> As these tests are fairly simple boot tests that just need kernels maybe
>> we could look at tooling up the generation of these images in a
>> repeatable way - similar to the way vmtest builds VMs.
>
> Yes, I have another acceptance branch where I cross-build an old mipssim
> kernel to test the board, using the following:
> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>
> But preparing a Docker cross image, fetching the Linux kernel source,
> building it, takes a lot of time/storage I'd rather avoid; at least with
> Aurelien kernels, since they are known to work since years.

Well you can throw away the docker image as long as we have a mechanism
to dump out the final binary. I have no interest in forcing people to
keep checked out code using up space, just that they can re-build if
they need to.

>
>>> +
>>> +        self.vm.set_machine('malta')
>>> +        self.vm.set_console()
>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>> +        self.vm.add_args('-m', "64",
>>> +                         '-kernel', kernel_path,
>>> +                         '-append', kernel_command_line)
>>> +        self.vm.launch()
>>> +        console = self.vm.console_socket.makefile()
>>> +        console_logger = logging.getLogger('console')
>>> +        while True:
>>> +            msg = console.readline()
>>> +            console_logger.debug(msg.strip())
>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>> +                break
>>> +            if 'Kernel panic - not syncing' in msg:
>>> +                self.fail("Kernel panic reached")
>>
>> Of course for bonus points a simple rootfs with hackbench or some such
>> in it would be nice. But I appreciate this makes the building job a lot
>> more complex than just a kernel.
>
> My idea is to use the rootfs for larger tests, and tag the "Kernel
> panic" tests as "quick", so we can have a "make acceptance-speed" or
> similar.
>
> We can already test than many devices were initialized correctly quickly
> looking at this console output.

Yeah the kernel boot up is a pretty good smoke test (it's all most of
kernelci manages as well). However it certainly doesn't fully exercise
the translator. I've lost track of the number of bugs we found after
successfully booting an kernel because we were doing exhaustive
instruction testing.

>
> Regards,
>
> Phil.


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

On 06/28/2018 03:36 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Similar to the BootLinuxConsoleX86_64 test:
>>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>>
>>>> This test can be run using:
>>>>
>>>>     $ avocado run -t endian:big tests/acceptance
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>>  1 file changed, 38 insertions(+)
>>>>
>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>> index 17dc8d58c1..72cf5e943c 100644
>>>> --- a/tests/acceptance/boot_linux_console.py
>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>>                  break
>>>>              if 'Kernel panic - not syncing' in msg:
>>>>                  self.fail("Kernel panic reached")
>>>> +
>>>> +
>>>> +class BootLinuxConsoleMips(Test):
>>>> +    """
>>>> +    Boots a mips Linux kernel and checks that the console is operational
>>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>>> +
>>>> +    :avocado: enable
>>>> +    :avocado: tags=endian:big
>>>> +    :avocado: tags=arch:mips
>>>> +    :avocado: tags=board:malta
>>>> +    """
>>>> +
>>>> +    arch = "mips"
>>>> +    timeout = 60
>>>> +
>>>> +    def test(self):
>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>> asset_hash=kernel_hash)
>>>
>>> I'm uncomfortable using "random" binaries of websites as the source of
>>> our test kernels. I can see the justification for distro kernels as they
>>> at least have the infrastructure to rebuild from source if you really
>>> want to, but even then the distros don't cover a lot of the
>>> architectures.
>>
>> And now I notice I made an mistake here :) I guess remember the Avocado
>> team started using the SHA-1 hash as default and I suggested them to be
>> able to use other hashes for this particular case, since Aurelien
>> provided the MD5 hashes signed by his GPG key, which is signed/trusted
>> by Peter and used to merge mips32 pulls.
>>
>> That would verify the QEMU community circle of trust right?
> 
> It's not so much the chain of trust and more repeatability of building
> the test cases. I trust Aurel32's binaries are good test cases for MIPS
> but at the moment I have no idea how to rebuild them which is a bit of
> an issue given it is covered by the GPL.

OK! I didn't even think about that since this is a "Linux" kernel and a
"Debian" rootfs provided by a Debian developer, hosted on a Debian
server, so I'm pretty sure this is GPL compliant, but it makes sens.

I'll find a way to have reproducible test binaries for acceptance testing.

>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
>> manual verification when merging this series should suffice.
>>
>>>
>>> I had experimented with using docker based builds for building test
>>> fixtures (see tests/docker/dockerbuilds):
>>>
>>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>>
>>> As these tests are fairly simple boot tests that just need kernels maybe
>>> we could look at tooling up the generation of these images in a
>>> repeatable way - similar to the way vmtest builds VMs.
>>
>> Yes, I have another acceptance branch where I cross-build an old mipssim
>> kernel to test the board, using the following:
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>>
>> But preparing a Docker cross image, fetching the Linux kernel source,
>> building it, takes a lot of time/storage I'd rather avoid; at least with
>> Aurelien kernels, since they are known to work since years.
> 
> Well you can throw away the docker image as long as we have a mechanism
> to dump out the final binary. I have no interest in forcing people to
> keep checked out code using up space, just that they can re-build if
> they need to.

The Avocado guys said they have plenty of storage for assets.
A cool feature would be to have some config built by some CI builder
that then sign and push reproducible config + generated binary to the
assets storage automatically.

>>>> +
>>>> +        self.vm.set_machine('malta')
>>>> +        self.vm.set_console()
>>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>>> +        self.vm.add_args('-m', "64",
>>>> +                         '-kernel', kernel_path,
>>>> +                         '-append', kernel_command_line)
>>>> +        self.vm.launch()
>>>> +        console = self.vm.console_socket.makefile()
>>>> +        console_logger = logging.getLogger('console')
>>>> +        while True:
>>>> +            msg = console.readline()
>>>> +            console_logger.debug(msg.strip())
>>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>>> +                break
>>>> +            if 'Kernel panic - not syncing' in msg:
>>>> +                self.fail("Kernel panic reached")
>>>
>>> Of course for bonus points a simple rootfs with hackbench or some such
>>> in it would be nice. But I appreciate this makes the building job a lot
>>> more complex than just a kernel.
>>
>> My idea is to use the rootfs for larger tests, and tag the "Kernel
>> panic" tests as "quick", so we can have a "make acceptance-speed" or
>> similar.
>>
>> We can already test than many devices were initialized correctly quickly
>> looking at this console output.
> 
> Yeah the kernel boot up is a pretty good smoke test (it's all most of
> kernelci manages as well). However it certainly doesn't fully exercise
> the translator. I've lost track of the number of bugs we found after
> successfully booting an kernel because we were doing exhaustive
> instruction testing.

This is not indeed a translator test, but simple enough to test than
devices are instantiated, mapped at the correct address, detected by
Linux, probe the device enough to initialize it.

I actually wrote this test after breaking a I/O device why a previous
Super I/O refactor, this test would have failed.

You could say a simple "info mtree" parser would notice it too... but
acceptance testing is closer to real-world imho, and would be also
useful to catch changes in the guest code (as in "QEMU expects things
that way, but the Linux kernel updated to something new").

Regards,

Phil.

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 8315 bytes --]

Hi Alex, Cleber.

On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 03:36 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>>
>>>>> Similar to the BootLinuxConsoleX86_64 test:
>>>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>>>
>>>>> This test can be run using:
>>>>>
>>>>>     $ avocado run -t endian:big tests/acceptance
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>> ---
>>>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>>>  1 file changed, 38 insertions(+)
>>>>>
>>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>>> index 17dc8d58c1..72cf5e943c 100644
>>>>> --- a/tests/acceptance/boot_linux_console.py
>>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>>>                  break
>>>>>              if 'Kernel panic - not syncing' in msg:
>>>>>                  self.fail("Kernel panic reached")
>>>>> +
>>>>> +
>>>>> +class BootLinuxConsoleMips(Test):
>>>>> +    """
>>>>> +    Boots a mips Linux kernel and checks that the console is operational
>>>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>>>> +
>>>>> +    :avocado: enable
>>>>> +    :avocado: tags=endian:big
>>>>> +    :avocado: tags=arch:mips
>>>>> +    :avocado: tags=board:malta
>>>>> +    """
>>>>> +
>>>>> +    arch = "mips"
>>>>> +    timeout = 60
>>>>> +
>>>>> +    def test(self):
>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>>> asset_hash=kernel_hash)
>>>>
>>>> I'm uncomfortable using "random" binaries of websites as the source of
>>>> our test kernels. I can see the justification for distro kernels as they
>>>> at least have the infrastructure to rebuild from source if you really
>>>> want to, but even then the distros don't cover a lot of the
>>>> architectures.

Alex: I could find all the Linux kernel I'm interested to console-test
with Avocado on the http://snapshot.debian.org/ archive website.

For example Aurelien's one (more up-to-date) is available here:
http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48

I also added a SH-4 test for the SM501 series of Zoltan BALATON using
the kernel extracted from this distrib built kernel:
http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30

The Debian distribution also provide the source package and the kernels
can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
recent kernels).

Would it be enough to satisfy the GPL requirements to provided that info
in the header and use these handy pre-compiled kernels?

Cleber: As you added tarballs and zip support, I'd also need a "dpkg"
support in your fetch_asset() archive management...

Cleber: And another future request is to be able to extract files from
ISO images with the guestfish tool :P
To be able to add a test for the Apple Machintosh Quadra 800 machine:
http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg08139.html

>>> And now I notice I made an mistake here :) I guess remember the Avocado
>>> team started using the SHA-1 hash as default and I suggested them to be
>>> able to use other hashes for this particular case, since Aurelien
>>> provided the MD5 hashes signed by his GPG key, which is signed/trusted
>>> by Peter and used to merge mips32 pulls.
>>>
>>> That would verify the QEMU community circle of trust right?
>>
>> It's not so much the chain of trust and more repeatability of building
>> the test cases. I trust Aurel32's binaries are good test cases for MIPS
>> but at the moment I have no idea how to rebuild them which is a bit of
>> an issue given it is covered by the GPL.
> 
> OK! I didn't even think about that since this is a "Linux" kernel and a
> "Debian" rootfs provided by a Debian developer, hosted on a Debian
> server, so I'm pretty sure this is GPL compliant, but it makes sens.
> 
> I'll find a way to have reproducible test binaries for acceptance testing.
> 
>>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
>>> manual verification when merging this series should suffice.
>>>
>>>>
>>>> I had experimented with using docker based builds for building test
>>>> fixtures (see tests/docker/dockerbuilds):
>>>>
>>>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>>>
>>>> As these tests are fairly simple boot tests that just need kernels maybe
>>>> we could look at tooling up the generation of these images in a
>>>> repeatable way - similar to the way vmtest builds VMs.
>>>
>>> Yes, I have another acceptance branch where I cross-build an old mipssim
>>> kernel to test the board, using the following:
>>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>>>
>>> But preparing a Docker cross image, fetching the Linux kernel source,
>>> building it, takes a lot of time/storage I'd rather avoid; at least with
>>> Aurelien kernels, since they are known to work since years.
>>
>> Well you can throw away the docker image as long as we have a mechanism
>> to dump out the final binary. I have no interest in forcing people to
>> keep checked out code using up space, just that they can re-build if
>> they need to.
> 
> The Avocado guys said they have plenty of storage for assets.
> A cool feature would be to have some config built by some CI builder
> that then sign and push reproducible config + generated binary to the
> assets storage automatically.
> 
>>>>> +
>>>>> +        self.vm.set_machine('malta')
>>>>> +        self.vm.set_console()
>>>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>>>> +        self.vm.add_args('-m', "64",
>>>>> +                         '-kernel', kernel_path,
>>>>> +                         '-append', kernel_command_line)
>>>>> +        self.vm.launch()
>>>>> +        console = self.vm.console_socket.makefile()
>>>>> +        console_logger = logging.getLogger('console')
>>>>> +        while True:
>>>>> +            msg = console.readline()
>>>>> +            console_logger.debug(msg.strip())
>>>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>>>> +                break
>>>>> +            if 'Kernel panic - not syncing' in msg:
>>>>> +                self.fail("Kernel panic reached")
>>>>
>>>> Of course for bonus points a simple rootfs with hackbench or some such
>>>> in it would be nice. But I appreciate this makes the building job a lot
>>>> more complex than just a kernel.
>>>
>>> My idea is to use the rootfs for larger tests, and tag the "Kernel
>>> panic" tests as "quick", so we can have a "make acceptance-speed" or
>>> similar.
>>>
>>> We can already test than many devices were initialized correctly quickly
>>> looking at this console output.
>>
>> Yeah the kernel boot up is a pretty good smoke test (it's all most of
>> kernelci manages as well). However it certainly doesn't fully exercise
>> the translator. I've lost track of the number of bugs we found after
>> successfully booting an kernel because we were doing exhaustive
>> instruction testing.
> 
> This is not indeed a translator test, but simple enough to test than
> devices are instantiated, mapped at the correct address, detected by
> Linux, probe the device enough to initialize it.
> 
> I actually wrote this test after breaking a I/O device why a previous
> Super I/O refactor, this test would have failed.
> 
> You could say a simple "info mtree" parser would notice it too... but
> acceptance testing is closer to real-world imho, and would be also
> useful to catch changes in the guest code (as in "QEMU expects things
> that way, but the Linux kernel updated to something new").
> 
> Regards,
> 
> Phil.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Eduardo Habkost]

On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> > On 06/28/2018 03:36 PM, Alex Bennée wrote:
> >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
[...]
> >>>>> +    def test(self):
> >>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> >>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
> >>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> >>>>> +        kernel_path = self.fetch_asset(kernel_url,
> >>>>> asset_hash=kernel_hash)
> >>>>
> >>>> I'm uncomfortable using "random" binaries of websites as the source of
> >>>> our test kernels. I can see the justification for distro kernels as they
> >>>> at least have the infrastructure to rebuild from source if you really
> >>>> want to, but even then the distros don't cover a lot of the
> >>>> architectures.
> 
> Alex: I could find all the Linux kernel I'm interested to console-test
> with Avocado on the http://snapshot.debian.org/ archive website.
> 
> For example Aurelien's one (more up-to-date) is available here:
> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
> 
> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
> the kernel extracted from this distrib built kernel:
> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
> 
> The Debian distribution also provide the source package and the kernels
> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
> recent kernels).
> 
> Would it be enough to satisfy the GPL requirements to provided that info
> in the header and use these handy pre-compiled kernels?

(IANAL, etc.)

Personally, I would try to avoid dealing with the "written offer"
option of the GPL, and just publish the sources in the same
medium (HTTP download repository, git repository, etc) as the
binary.

There was a related discussion about distribution of firmware
binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>.
It looks like we never applied the docs/firmware.txt patch,
though.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

On 07/04/2018 05:47 PM, Eduardo Habkost wrote:
> On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
>> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
>>> On 06/28/2018 03:36 PM, Alex Bennée wrote:
>>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
> [...]
>>>>>>> +    def test(self):
>>>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>>>>> asset_hash=kernel_hash)
>>>>>>
>>>>>> I'm uncomfortable using "random" binaries of websites as the source of
>>>>>> our test kernels. I can see the justification for distro kernels as they
>>>>>> at least have the infrastructure to rebuild from source if you really
>>>>>> want to, but even then the distros don't cover a lot of the
>>>>>> architectures.
>>
>> Alex: I could find all the Linux kernel I'm interested to console-test
>> with Avocado on the http://snapshot.debian.org/ archive website.
>>
>> For example Aurelien's one (more up-to-date) is available here:
>> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
>>
>> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
>> the kernel extracted from this distrib built kernel:
>> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
>>
>> The Debian distribution also provide the source package and the kernels
>> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
>> recent kernels).
>>
>> Would it be enough to satisfy the GPL requirements to provided that info
>> in the header and use these handy pre-compiled kernels?
> 
> (IANAL, etc.)
> 
> Personally, I would try to avoid dealing with the "written offer"
> option of the GPL, and just publish the sources in the same
> medium (HTTP download repository, git repository, etc) as the
> binary.

About Debian packaged kernel: the sources are in the same directory:

http://snapshot.debian.org/package/linux-2.6/2.6.32-30/

linux-2.6_2.6.32.orig.tar.gz
linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb
linux-support-2.6.32-5_2.6.32-30_all.deb

The other way, trying to compile a 2013 kernel is painful because the
oldest GCC I succeed installing complain about many errors.

The next absurd option I have is install a 2013 userland distrib and use
qemu-user to cross^H^H^H^H(?? I dunno how to name it)-compile this
kernel.... "backward-compile"? This might make sens with the coming
quantum computers.

> There was a related discussion about distribution of firmware
> binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>.
> It looks like we never applied the docs/firmware.txt patch,
> though.

This case is not about firmware we need to run QEMU, but Linux kernel to
run acceptance tests using QEMU. But same legal stuffs, you know...

Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Eduardo Habkost]

On Wed, Jul 04, 2018 at 06:44:02PM -0300, Philippe Mathieu-Daudé wrote:
> On 07/04/2018 05:47 PM, Eduardo Habkost wrote:
> > On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote:
> >> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> >>> On 06/28/2018 03:36 PM, Alex Bennée wrote:
> >>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
> > [...]
> >>>>>>> +    def test(self):
> >>>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
> >>>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
> >>>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> >>>>>>> +        kernel_path = self.fetch_asset(kernel_url,
> >>>>>>> asset_hash=kernel_hash)
> >>>>>>
> >>>>>> I'm uncomfortable using "random" binaries of websites as the source of
> >>>>>> our test kernels. I can see the justification for distro kernels as they
> >>>>>> at least have the infrastructure to rebuild from source if you really
> >>>>>> want to, but even then the distros don't cover a lot of the
> >>>>>> architectures.
> >>
> >> Alex: I could find all the Linux kernel I'm interested to console-test
> >> with Avocado on the http://snapshot.debian.org/ archive website.
> >>
> >> For example Aurelien's one (more up-to-date) is available here:
> >> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48
> >>
> >> I also added a SH-4 test for the SM501 series of Zoltan BALATON using
> >> the kernel extracted from this distrib built kernel:
> >> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30
> >>
> >> The Debian distribution also provide the source package and the kernels
> >> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
> >> recent kernels).
> >>
> >> Would it be enough to satisfy the GPL requirements to provided that info
> >> in the header and use these handy pre-compiled kernels?
> > 
> > (IANAL, etc.)
> > 
> > Personally, I would try to avoid dealing with the "written offer"
> > option of the GPL, and just publish the sources in the same
> > medium (HTTP download repository, git repository, etc) as the
> > binary.
> 
> About Debian packaged kernel: the sources are in the same directory:
> 
> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/
> 
> linux-2.6_2.6.32.orig.tar.gz
> linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb
> linux-support-2.6.32-5_2.6.32-30_all.deb

Oh, I thought you were talking about publishing the binaries
elsewhere.  If we're not the ones publishing the binaries (we're
just downloading it), then this is somebody else's problem.  :)

Would it be OK to make test code download directly from
snapshot.debian.org, though?  Their home page have instructions
to use apt repositories there, so I guess it's reasonable use of
the service.

-- 
Eduardo

[Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console

[Philippe Mathieu-Daudé]

The board already instantiate the proper devices, we don't want to
add extra devices but connect the chardev to one of the serial already
available.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qemu.py                        | 6 ++++--
 tests/acceptance/boot_linux_console.py | 3 ++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index f099ce7278..7a975f4538 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -211,8 +211,10 @@ class QEMUMachine(object):
                                                  self._name + "-console.sock")
             chardev = ('socket,id=console,path=%s,server,nowait' %
                        self._console_address)
-            device = '%s,chardev=console' % self._console_device_type
-            args.extend(['-chardev', chardev, '-device', device])
+            args.extend(['-chardev', chardev])
+            if len(self._console_device_type):
+                device = '%s,chardev=console' % self._console_device_type
+                args.extend(['-device', device])
         return args
 
     def _pre_launch(self):
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 72cf5e943c..510742c4c7 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -69,9 +69,10 @@ class BootLinuxConsoleMips(Test):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_machine('malta')
-        self.vm.set_console()
+        self.vm.set_console("") # FIXME this disable isa-serial to use -serial
         kernel_command_line = 'console=ttyS0 printk.time=0'
         self.vm.add_args('-m', "64",
+                         '-serial', "chardev:console", # FIXME ... here.
                          '-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
-- 
2.18.0.rc2

Re: [Qemu-devel] [RFC PATCH v2 5/6] tests/acceptance: Add a kludge to not use the default console

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> The board already instantiate the proper devices, we don't want to
> add extra devices but connect the chardev to one of the serial already
> available.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  scripts/qemu.py                        | 6 ++++--
>  tests/acceptance/boot_linux_console.py | 3 ++-
>  2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index f099ce7278..7a975f4538 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -211,8 +211,10 @@ class QEMUMachine(object):
>                                                   self._name + "-console.sock")
>              chardev = ('socket,id=console,path=%s,server,nowait' %
>                         self._console_address)
> -            device = '%s,chardev=console' % self._console_device_type
> -            args.extend(['-chardev', chardev, '-device', device])
> +            args.extend(['-chardev', chardev])
> +            if len(self._console_device_type):
> +                device = '%s,chardev=console' % self._console_device_type
> +                args.extend(['-device', device])
>          return args
>
>      def _pre_launch(self):
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 72cf5e943c..510742c4c7 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -69,9 +69,10 @@ class BootLinuxConsoleMips(Test):
>          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>
>          self.vm.set_machine('malta')
> -        self.vm.set_console()
> +        self.vm.set_console("") # FIXME this disable isa-serial to
>          use -serial

This is reminiscent of assumptions libvirt made about serial consoles.
Does -serial not work on x86 for old style setups? I'm sure I have
x86-64 boots with -serial mon:stdio lines.

>          kernel_command_line = 'console=ttyS0 printk.time=0'
>          self.vm.add_args('-m', "64",
> +                         '-serial', "chardev:console", # FIXME ... here.
>                           '-kernel', kernel_path,
>                           '-append', kernel_command_line)
>          self.vm.launch()


--
Alex Bennée

[Qemu-devel] [PATCH v2 6/6] travis: Add Avocado tests

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
https://travis-ci.org/philmd/qemu/jobs/395266626

  ...
  LINK    x86_64-softmmu/qemu-system-x86_64
JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
 (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
 (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
 (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
 (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
 (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
 (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
 (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 6.21 s

Done. Your build exited with 0.

Job ran for 5 min 34 sec
---
 .travis.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index fabfe9ec34..7b224add2d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -153,3 +153,11 @@ matrix:
         - TEST_CMD=""
       before_script:
         - ./configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread -fuse-ld=gold" || { cat config.log && exit 1; }
+    # Avocado tests
+    - env: CONFIG="--target-list=x86_64-softmmu,mips-softmmu --enable-trace-backends=nop --disable-tools --disable-docs"
+           TEST_CMD="avocado run tests/acceptance"
+      python:
+        - "3.6"
+      before_script:
+        - pip install --user avocado-framework
+        - ./configure ${CONFIG} || { cat config.log && exit 1; }
-- 
2.18.0.rc2

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
> Another neanderthal approach to add multi-arch acceptance tests using
> Avocado.
>
> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
> at using Avocado in QEMU.
> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
> a bit further:
> - we can now run other archs
> - Avocado is integrated into Travis CI
>
> I'm not sure Travis is the best CI available for those tests, but it runs
> quite fast:
>
>       ...
>       LINK    x86_64-softmmu/qemu-system-x86_64
>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 6.21 s

Hmm however if I do:

   pip install --user avocado-framework
   avocado run tests/acceptance

I get:

   Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
   Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
   Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
   Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
   Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
   You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log

Which I seem to remember from last time I looked into this. I think
avocado makes a bunch of assumptions about the filesystem and OS layout
which is hard to gloss over for something you should be able to run
straight out of the source tree.


>
>     Done. Your build exited with 0.
>
>     Job ran for 5 min 34 sec
>
> I couldn't add the Alpha tests because all the vmlinux kernels I found
> online are gzipped, so I'm waiting some upstream Avocado merges before.
>
> Cleber: I hope you can help me pythonizing this series :p
>
> Regards,
>
> Phil.
>
> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>
> Philippe Mathieu-Daudé (6):
>   avocado: Add a Test.arch property
>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>   tests/acceptance: Improve the Avocado tags
>   tests/acceptance: Add a BootLinuxConsoleMips test
>   tests/acceptance: Add a kludge to not use the default console
>   travis: Add Avocado tests
>
>  .travis.yml                               |  8 +++++
>  scripts/qemu.py                           |  6 ++--
>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>  4 files changed, 67 insertions(+), 8 deletions(-)


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Cleber Rosa]

On 06/28/2018 11:46 AM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi,
>>
>> Another neanderthal approach to add multi-arch acceptance tests using
>> Avocado.
>>
>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>> at using Avocado in QEMU.
>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>> a bit further:
>> - we can now run other archs
>> - Avocado is integrated into Travis CI
>>
>> I'm not sure Travis is the best CI available for those tests, but it runs
>> quite fast:
>>
>>       ...
>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 6.21 s
> 
> Hmm however if I do:
> 
>    pip install --user avocado-framework
>    avocado run tests/acceptance
> 
> I get:
> 
>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')

Alex,

These are errors related to your installation of Avocado-*VT*[1].

>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
> 

And these from Avocado-*Virt*[2].  Neither of those are used in the
tests under "tests/acceptance".

> Which I seem to remember from last time I looked into this. I think
> avocado makes a bunch of assumptions about the filesystem and OS layout
> which is hard to gloss over for something you should be able to run
> straight out of the source tree.
> 
> 

These Avocado-*VT* issues deserve a better look indeed.  Would you
please describe (if not here, under a GH issue) your environment?

Thanks,
- Cleber.

[1] - https://github.com/avocado-framework/avocado-vt
[2] - https://github.com/avocado-framework/avocado-virt

>>
>>     Done. Your build exited with 0.
>>
>>     Job ran for 5 min 34 sec
>>
>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>
>> Cleber: I hope you can help me pythonizing this series :p
>>
>> Regards,
>>
>> Phil.
>>
>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>
>> Philippe Mathieu-Daudé (6):
>>   avocado: Add a Test.arch property
>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>   tests/acceptance: Improve the Avocado tags
>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>   tests/acceptance: Add a kludge to not use the default console
>>   travis: Add Avocado tests
>>
>>  .travis.yml                               |  8 +++++
>>  scripts/qemu.py                           |  6 ++--
>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>  4 files changed, 67 insertions(+), 8 deletions(-)
> 
> 
> --
> Alex Bennée
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On 06/28/2018 11:46 AM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Hi,
>>>
>>> Another neanderthal approach to add multi-arch acceptance tests using
>>> Avocado.
>>>
>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>> at using Avocado in QEMU.
>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>> a bit further:
>>> - we can now run other archs
>>> - Avocado is integrated into Travis CI
>>>
>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>> quite fast:
>>>
>>>       ...
>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>>     JOB TIME   : 6.21 s
>>
>> Hmm however if I do:
>>
>>    pip install --user avocado-framework
>>    avocado run tests/acceptance
>>
>> I get:
>>
>>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
>
> Alex,
>
> These are errors related to your installation of Avocado-*VT*[1].
>
>>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
>>
>
> And these from Avocado-*Virt*[2].  Neither of those are used in the
> tests under "tests/acceptance".
>
>> Which I seem to remember from last time I looked into this. I think
>> avocado makes a bunch of assumptions about the filesystem and OS layout
>> which is hard to gloss over for something you should be able to run
>> straight out of the source tree.
>>
>>
>
> These Avocado-*VT* issues deserve a better look indeed.  Would you
> please describe (if not here, under a GH issue) your environment?

Ahh probably from last time I tried to get things up and running. I
rm -rf ~/.local and things seem to be working well now.

However I was under the impression that VT or virt was needed for this
testing. Or is this an old assumption?

>
> Thanks,
> - Cleber.
>
> [1] - https://github.com/avocado-framework/avocado-vt
> [2] - https://github.com/avocado-framework/avocado-virt
>
>>>
>>>     Done. Your build exited with 0.
>>>
>>>     Job ran for 5 min 34 sec
>>>
>>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>>
>>> Cleber: I hope you can help me pythonizing this series :p
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>>
>>> Philippe Mathieu-Daudé (6):
>>>   avocado: Add a Test.arch property
>>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>>   tests/acceptance: Improve the Avocado tags
>>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>>   tests/acceptance: Add a kludge to not use the default console
>>>   travis: Add Avocado tests
>>>
>>>  .travis.yml                               |  8 +++++
>>>  scripts/qemu.py                           |  6 ++--
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>>  4 files changed, 67 insertions(+), 8 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Philippe Mathieu-Daudé]

On 06/28/2018 02:13 PM, Alex Bennée wrote:
> 
> Cleber Rosa <crosa@redhat.com> writes:
> 
>> On 06/28/2018 11:46 AM, Alex Bennée wrote:
>>>
>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>>> Hi,
>>>>
>>>> Another neanderthal approach to add multi-arch acceptance tests using
>>>> Avocado.
>>>>
>>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>>> at using Avocado in QEMU.
>>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>>> a bit further:
>>>> - we can now run other archs
>>>> - Avocado is integrated into Travis CI
>>>>
>>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>>> quite fast:
>>>>
>>>>       ...
>>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>>>     JOB TIME   : 6.21 s
>>>
>>> Hmm however if I do:
>>>
>>>    pip install --user avocado-framework
>>>    avocado run tests/acceptance
>>>
>>> I get:
>>>
>>>    Failed to load plugin from module "avocado_vt.plugins.vt_list": ImportError('No module named netaddr',)
>>>    Failed to load plugin from module "avocado_vt.plugins.vt": ImportError('No module named netaddr',)
>>>    Failed to load plugin from module "avocado_vt.plugins.vt_bootstrap": OSError(13, 'Permission denied')
>>
>> Alex,
>>
>> These are errors related to your installation of Avocado-*VT*[1].
>>
>>>    Error running method "run" of plugin "virt": 'Namespace' object has no attribute 'default_avocado_params'
>>>    Avocado crashed unexpectedly: [Errno 13] Permission denied: '/var/lib/libvirt/images/avocado_sg6NRi'
>>>    You can find details in /var/tmp/avocado-traceback-2018-06-28_16:44:25-DGb1CG.log
>>>
>>
>> And these from Avocado-*Virt*[2].  Neither of those are used in the
>> tests under "tests/acceptance".
>>
>>> Which I seem to remember from last time I looked into this. I think
>>> avocado makes a bunch of assumptions about the filesystem and OS layout
>>> which is hard to gloss over for something you should be able to run
>>> straight out of the source tree.
>>>
>>>
>>
>> These Avocado-*VT* issues deserve a better look indeed.  Would you
>> please describe (if not here, under a GH issue) your environment?
> 
> Ahh probably from last time I tried to get things up and running. I
> rm -rf ~/.local and things seem to be working well now.

Should we check for a prerequisite version?

> 
> However I was under the impression that VT or virt was needed for this
> testing. Or is this an old assumption?

I don't use neither of these for my qemu-system testing, but I'm more
interested in testing system targets that are different from my host.

I think the VT/virt make sens when you test on a same (target, host)
tuple and worry about perfs.

<snip>

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
> Another neanderthal approach to add multi-arch acceptance tests using
> Avocado.
>
> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
> at using Avocado in QEMU.
> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
> a bit further:
> - we can now run other archs
> - Avocado is integrated into Travis CI
>
> I'm not sure Travis is the best CI available for those tests, but it runs
> quite fast:
>
>       ...
>       LINK    x86_64-softmmu/qemu-system-x86_64
>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>       INTERRUPT 0 | CANCEL 0

That said on my AArch64 server it fairs better:

15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
 (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
 (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
 (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
 (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
 (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
 (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
 (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
JOB TIME   : 2.47 s

Although given the errors in the logs I'm surprised at ERROR 0

>     JOB TIME   : 6.21 s
>
>     Done. Your build exited with 0.
>
>     Job ran for 5 min 34 sec
>
> I couldn't add the Alpha tests because all the vmlinux kernels I found
> online are gzipped, so I'm waiting some upstream Avocado merges before.
>
> Cleber: I hope you can help me pythonizing this series :p
>
> Regards,
>
> Phil.
>
> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>
> Philippe Mathieu-Daudé (6):
>   avocado: Add a Test.arch property
>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>   tests/acceptance: Improve the Avocado tags
>   tests/acceptance: Add a BootLinuxConsoleMips test
>   tests/acceptance: Add a kludge to not use the default console
>   travis: Add Avocado tests
>
>  .travis.yml                               |  8 +++++
>  scripts/qemu.py                           |  6 ++--
>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>  4 files changed, 67 insertions(+), 8 deletions(-)


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Philippe Mathieu-Daudé]

Hi Alex,

On 06/28/2018 01:01 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>> Another neanderthal approach to add multi-arch acceptance tests using
>> Avocado.
>>
>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>> at using Avocado in QEMU.
>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>> a bit further:
>> - we can now run other archs
>> - Avocado is integrated into Travis CI
>>
>> I'm not sure Travis is the best CI available for those tests, but it runs
>> quite fast:
>>
>>       ...
>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>       INTERRUPT 0 | CANCEL 0
> 
> That said on my AArch64 server it fairs better:
> 
> 15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
> JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
> JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
>  (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
>  (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
>  (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
>  (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
>  (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
>  (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
>  (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
> JOB TIME   : 2.47 s
> 
> Although given the errors in the logs I'm surprised at ERROR 0

Why are you surprised? You were expecting the BootLinuxConsoleMips to fail?

> 
>>     JOB TIME   : 6.21 s
>>
>>     Done. Your build exited with 0.
>>
>>     Job ran for 5 min 34 sec
>>
>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>
>> Cleber: I hope you can help me pythonizing this series :p
>>
>> Regards,
>>
>> Phil.
>>
>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>
>> Philippe Mathieu-Daudé (6):
>>   avocado: Add a Test.arch property
>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>   tests/acceptance: Improve the Avocado tags
>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>   tests/acceptance: Add a kludge to not use the default console
>>   travis: Add Avocado tests
>>
>>  .travis.yml                               |  8 +++++
>>  scripts/qemu.py                           |  6 ++--
>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>  4 files changed, 67 insertions(+), 8 deletions(-)
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 06/28/2018 01:01 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>
>>> Another neanderthal approach to add multi-arch acceptance tests using
>>> Avocado.
>>>
>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>> at using Avocado in QEMU.
>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>> a bit further:
>>> - we can now run other archs
>>> - Avocado is integrated into Travis CI
>>>
>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>> quite fast:
>>>
>>>       ...
>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>     JOB LOG    : /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>      (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS (3.89 s)
>>>      (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS (1.83 s)
>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.05 s)
>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>      (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.05 s)
>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.05 s)
>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>>       INTERRUPT 0 | CANCEL 0
>>
>> That said on my AArch64 server it fairs better:
>>
>> 15:59:19 [alex.bennee@libgomp-d05-dev:~/l/qemu.git] review/avocado-v2 + avocado run tests/acceptance
>> JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
>> JOB LOG    : /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
>>  (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: CANCEL (0.00 s)
>>  (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: PASS (2.03 s)
>>  (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: CANCEL (0.00 s)
>>  (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
>>  (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 s)
>>  (6/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: CANCEL (0.00 s)
>>  (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
>> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 6
>> JOB TIME   : 2.47 s
>>
>> Although given the errors in the logs I'm surprised at ERROR 0
>
> Why are you surprised? You were expecting the BootLinuxConsoleMips to
> fail?

No but a bunch have CANCELed... maybe they are skipping but if the test
is run and it fails to complete I would expect that to be an error.

>
>>
>>>     JOB TIME   : 6.21 s
>>>
>>>     Done. Your build exited with 0.
>>>
>>>     Job ran for 5 min 34 sec
>>>
>>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>>
>>> Cleber: I hope you can help me pythonizing this series :p
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>>
>>> Philippe Mathieu-Daudé (6):
>>>   avocado: Add a Test.arch property
>>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>>   tests/acceptance: Improve the Avocado tags
>>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>>   tests/acceptance: Add a kludge to not use the default console
>>>   travis: Add Avocado tests
>>>
>>>  .travis.yml                               |  8 +++++
>>>  scripts/qemu.py                           |  6 ++--
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>>  4 files changed, 67 insertions(+), 8 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée