[Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Philippe Mathieu-Daudé]

Hi,

It was a rainy week-end here, so I invested it to automatize some
of my MIPS tests.

The BootLinuxSshTest is not Global warming friendly, it is not
meant to run on a CI system but rather on a workstation previous
to post a pull request.
It can surely be improved, but it is a good starting point.

Regards,

Phil.

Philippe Mathieu-Daudé (4):
  BootLinuxConsoleTest: Let extract_from_deb handle various compressions
  BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU
  BootLinuxConsoleTest: Run kerneltests BusyBox on Malta
  BootLinuxSshTest: Test some userspace commands on Malta

 MAINTAINERS                              |   1 +
 tests/acceptance/boot_linux_console.py   | 112 ++++++++++-
 tests/acceptance/linux_ssh_mips_malta.py | 229 +++++++++++++++++++++++
 tests/requirements.txt                   |   1 +
 4 files changed, 341 insertions(+), 2 deletions(-)
 create mode 100644 tests/acceptance/linux_ssh_mips_malta.py

-- 
2.19.1

[Qemu-devel] [PATCH 1/4] BootLinuxConsoleTest: Let extract_from_deb handle various compressions

[Philippe Mathieu-Daudé]

Debian binary package format supports various compressions.

Per man deb(5):

  NAME
    deb - Debian binary package format

  FORMAT
    ...
    The third, last required member is named data.tar.  It contains the
    filesystem as a tar archive, either not compressed (supported since
    dpkg 1.10.24), or compressed with gzip (with .gz extension),
    xz (with .xz extension, supported since dpkg 1.15.6),
    bzip2 (with .bz2 extension, supported since dpkg 1.10.24) or
    lzma (with .lzma extension, supported since dpkg 1.13.25).

List the archive files to have the 3rd name with the correct extension.

The function avocado.utils.archive.extract() will handle the different
compression format for us.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
This patch is already in Eduardo's queue, but is required in
this series.
---
 tests/acceptance/boot_linux_console.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index d5c500ea30..6aa084e049 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -55,8 +55,9 @@ class BootLinuxConsole(Test):
         """
         cwd = os.getcwd()
         os.chdir(self.workdir)
-        process.run("ar x %s data.tar.gz" % deb)
-        archive.extract("data.tar.gz", self.workdir)
+        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
+        process.run("ar x %s %s" % (deb, file_path))
+        archive.extract(file_path, self.workdir)
         os.chdir(cwd)
         return self.workdir + path
 
-- 
2.19.1

[Qemu-devel] [PATCH 2/4] BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU

[Philippe Mathieu-Daudé]

Similar to the x86_64/pc test, it boots a Linux kernel on a Malta
machine and verify the serial is working.

Use the documentation added in commit f7d257cb4a17 to test
nanoMIPS kernels and the I7200 CPU.

This test can be run using:

  $ avocado --show=console run -t arch:mipsel tests/acceptance/boot_linux_console.py
  console: [    0.000000] Linux version 4.15.18-00432-gb2eb9a8b (emubuild@mipscs563) (gcc version 6.3.0 (Codescape GNU Tools 2018.04-02 for nanoMIPS Linux)) #1 SMP Wed Jun 27 11:10:08 PDT 2018
  console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
  console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
  console: [    0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
  console: [    0.000000] MIPS: machine is mti,malta
  console: [    0.000000] Determined physical RAM map:
  console: [    0.000000]  memory: 08000000 @ 00000000 (usable)
  console: [    0.000000] earlycon: ns16550a0 at I/O port 0x3f8 (options '38400n8')
  console: [    0.000000] bootconsole [ns16550a0] enabled
  console: [    0.000000] User-defined physical RAM map:
  console: [    0.000000]  memory: 10000000 @ 00000000 (usable)
  console: [    0.000000] Initrd not found or empty - disabling initrd
  console: [    0.000000] MIPS CPS SMP unable to proceed without a CM
  console: [    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
  console: [    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
  console: [    0.000000] This processor doesn't support highmem. -262144k highmem ignored
  console: [    0.000000] Zone ranges:
  console: [    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000]   HighMem  empty
  console: [    0.000000] Movable zone start for each node
  console: [    0.000000] Early memory node ranges
  console: [    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000] random: get_random_bytes called from start_kernel+0x60/0x2f0 with crng_init=0
  console: [    0.000000] percpu: Embedded 16 pages/cpu @(ptrval) s36620 r8192 d20724 u65536
  console: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64960
  console: [    0.000000] Kernel command line: printk.time=0 mem=256m@@0x0 console=ttyS0 earlycon

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 58 ++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 6aa084e049..1c330871c0 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -10,6 +10,8 @@
 
 import os
 import logging
+import lzma
+import shutil
 
 from avocado_qemu import Test
 from avocado.utils import process
@@ -136,6 +138,62 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        kernel_path = self.workdir + "kernel"
+        with lzma.open(kernel_path_xz, 'rb') as f_in:
+            with open(kernel_path, 'wb') as f_out:
+                shutil.copyfileobj(f_in, f_out)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'mem=256m@@0x0 '
+                               + 'console=ttyS0')
+        self.vm.add_args('-no-reboot',
+                         '-cpu', 'I7200',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
+
+    def test_mips_malta32el_nanomips_4k(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page4k.xz')
+        kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
+    def test_mips_malta32el_nanomips_16k_up(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page16k_up.xz')
+        kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
+    def test_mips_malta32el_nanomips_64k_dbg(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page64k_dbg.xz')
+        kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
     def test_aarch64_virt(self):
         """
         :avocado: tags=arch:aarch64
-- 
2.19.1

Re: [Qemu-devel] [PATCH 2/4] BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU

[Aleksandar Markovic]

On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
>
> Similar to the x86_64/pc test, it boots a Linux kernel on a Malta
> machine and verify the serial is working.
>
> Use the documentation added in commit f7d257cb4a17 to test
> nanoMIPS kernels and the I7200 CPU.
>
> This test can be run using:
>
>   $ avocado --show=console run -t arch:mipsel
tests/acceptance/boot_linux_console.py
>   console: [    0.000000] Linux version 4.15.18-00432-gb2eb9a8b
(emubuild@mipscs563) (gcc version 6.3.0 (Codescape GNU Tools 2018.04-02 for
nanoMIPS Linux)) #1 SMP Wed Jun 27 11:10:08 PDT 2018
>   console: [    0.000000] GCRs appear to have been moved (expected them
at 0x1fbf8000)!
>   console: [    0.000000] GCRs appear to have been moved (expected them
at 0x1fbf8000)!
>   console: [    0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
>   console: [    0.000000] MIPS: machine is mti,malta
>   console: [    0.000000] Determined physical RAM map:
>   console: [    0.000000]  memory: 08000000 @ 00000000 (usable)
>   console: [    0.000000] earlycon: ns16550a0 at I/O port 0x3f8 (options
'38400n8')
>   console: [    0.000000] bootconsole [ns16550a0] enabled
>   console: [    0.000000] User-defined physical RAM map:
>   console: [    0.000000]  memory: 10000000 @ 00000000 (usable)
>   console: [    0.000000] Initrd not found or empty - disabling initrd
>   console: [    0.000000] MIPS CPS SMP unable to proceed without a CM
>   console: [    0.000000] Primary instruction cache 32kB, VIPT, 4-way,
linesize 32 bytes.
>   console: [    0.000000] Primary data cache 32kB, 4-way, VIPT, cache
aliases, linesize 32 bytes
>   console: [    0.000000] This processor doesn't support highmem.
-262144k highmem ignored
>   console: [    0.000000] Zone ranges:
>   console: [    0.000000]   Normal   [mem
0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000]   HighMem  empty
>   console: [    0.000000] Movable zone start for each node
>   console: [    0.000000] Early memory node ranges
>   console: [    0.000000]   node   0: [mem
0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000] random: get_random_bytes called from
start_kernel+0x60/0x2f0 with crng_init=0
>   console: [    0.000000] percpu: Embedded 16 pages/cpu @(ptrval) s36620
r8192 d20724 u65536
>   console: [    0.000000] Built 1 zonelists, mobility grouping on.  Total
pages: 64960
>   console: [    0.000000] Kernel command line: printk.time=0 mem=256m@@0x0
console=ttyS0 earlycon
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---

Fantastiquement.

Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>

>  tests/acceptance/boot_linux_console.py | 58 ++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux_console.py
b/tests/acceptance/boot_linux_console.py
> index 6aa084e049..1c330871c0 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -10,6 +10,8 @@
>
>  import os
>  import logging
> +import lzma
> +import shutil
>
>  from avocado_qemu import Test
>  from avocado.utils import process
> @@ -136,6 +138,62 @@ class BootLinuxConsole(Test):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>
> +    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
> +        kernel_path_xz = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> +        kernel_path = self.workdir + "kernel"
> +        with lzma.open(kernel_path_xz, 'rb') as f_in:
> +            with open(kernel_path, 'wb') as f_out:
> +                shutil.copyfileobj(f_in, f_out)
> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> +                               + 'mem=256m@@0x0 '
> +                               + 'console=ttyS0')
> +        self.vm.add_args('-no-reboot',
> +                         '-cpu', 'I7200',
> +                         '-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> +
> +    def test_mips_malta32el_nanomips_4k(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/
'
> +
'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page4k.xz')
> +        kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
> +    def test_mips_malta32el_nanomips_16k_up(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/
'
> +
'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page16k_up.xz')
> +        kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
> +    def test_mips_malta32el_nanomips_64k_dbg(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/
'
> +
'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page64k_dbg.xz')
> +        kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
>      def test_aarch64_virt(self):
>          """
>          :avocado: tags=arch:aarch64
> --
> 2.19.1
>

Re: [Qemu-devel] [PATCH 2/4] BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU

[Cleber Rosa]

On Tue, May 21, 2019 at 01:19:08AM +0200, Philippe Mathieu-Daudé wrote:
> Similar to the x86_64/pc test, it boots a Linux kernel on a Malta
> machine and verify the serial is working.
> 
> Use the documentation added in commit f7d257cb4a17 to test
> nanoMIPS kernels and the I7200 CPU.
> 
> This test can be run using:
> 
>   $ avocado --show=console run -t arch:mipsel tests/acceptance/boot_linux_console.py
>   console: [    0.000000] Linux version 4.15.18-00432-gb2eb9a8b (emubuild@mipscs563) (gcc version 6.3.0 (Codescape GNU Tools 2018.04-02 for nanoMIPS Linux)) #1 SMP Wed Jun 27 11:10:08 PDT 2018
>   console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
>   console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
>   console: [    0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
>   console: [    0.000000] MIPS: machine is mti,malta
>   console: [    0.000000] Determined physical RAM map:
>   console: [    0.000000]  memory: 08000000 @ 00000000 (usable)
>   console: [    0.000000] earlycon: ns16550a0 at I/O port 0x3f8 (options '38400n8')
>   console: [    0.000000] bootconsole [ns16550a0] enabled
>   console: [    0.000000] User-defined physical RAM map:
>   console: [    0.000000]  memory: 10000000 @ 00000000 (usable)
>   console: [    0.000000] Initrd not found or empty - disabling initrd
>   console: [    0.000000] MIPS CPS SMP unable to proceed without a CM
>   console: [    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
>   console: [    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
>   console: [    0.000000] This processor doesn't support highmem. -262144k highmem ignored
>   console: [    0.000000] Zone ranges:
>   console: [    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000]   HighMem  empty
>   console: [    0.000000] Movable zone start for each node
>   console: [    0.000000] Early memory node ranges
>   console: [    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
>   console: [    0.000000] random: get_random_bytes called from start_kernel+0x60/0x2f0 with crng_init=0
>   console: [    0.000000] percpu: Embedded 16 pages/cpu @(ptrval) s36620 r8192 d20724 u65536
>   console: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64960
>   console: [    0.000000] Kernel command line: printk.time=0 mem=256m@@0x0 console=ttyS0 earlycon
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 58 ++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 6aa084e049..1c330871c0 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -10,6 +10,8 @@
>  
>  import os
>  import logging
> +import lzma
> +import shutil
>  
>  from avocado_qemu import Test
>  from avocado.utils import process
> @@ -136,6 +138,62 @@ class BootLinuxConsole(Test):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>  
> +    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):

Nitpick: It's clear that having helper methods that tests build on is
a common pattern (even when we improve the parameter situation, so
that individual tests can be fed their own set of parameters).  What
bothers me a bit is the "_test_" component in their names, because it
makes it a bit harder on my eyes to distinguish between tests and
other methods.

Given that we are just starting to build our conventions, so I guess
I'd make sense to at try follow some prior art.  Looking at
virtio_version.py, I see that Eduardo used "check_" as a prefix for
methods of similar purpose.  Let me know if you think this seems worth
doing.

> +        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> +        kernel_path = self.workdir + "kernel"
> +        with lzma.open(kernel_path_xz, 'rb') as f_in:
> +            with open(kernel_path, 'wb') as f_out:
> +                shutil.copyfileobj(f_in, f_out)
> +

avocado.utils.archive *almost* suits this lzma case (but not quite), so
ack on this.

> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> +                               + 'mem=256m@@0x0 '
> +                               + 'console=ttyS0')

Another nitpick: these last two lines can be a single one (and avoid
one extra string concatenation).

> +        self.vm.add_args('-no-reboot',
> +                         '-cpu', 'I7200',
> +                         '-kernel', kernel_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> +
> +    def test_mips_malta32el_nanomips_4k(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
> +                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page4k.xz')
> +        kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
> +    def test_mips_malta32el_nanomips_16k_up(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
> +                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page16k_up.xz')
> +        kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
> +    def test_mips_malta32el_nanomips_64k_dbg(self):
> +        """
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        """
> +        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
> +                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
> +                      'generic_nano32r6el_page64k_dbg.xz')
> +        kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
> +        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
> +
>      def test_aarch64_virt(self):
>          """
>          :avocado: tags=arch:aarch64
> -- 
> 2.19.1
> 

I'd suggest you to add the mipsel target to the configure list on
.travis.yaml, ir accordance with the previous patches that added tests
for new arches.

Other than that, it looks good to me.

Tested-by: Cleber Rosa <crosa@redhat.com>

[Qemu-devel] [PATCH 3/4] BootLinuxConsoleTest: Run kerneltests BusyBox on Malta

[Philippe Mathieu-Daudé]

This tests boots a Linux kernel on a Malta machine up to a
busybox shell on the serial console. Few commands are executed
before halting the machine (via reboot).

We use the initrd cpio image from the kerneltests project:
https://kerneltests.org/

If MIPS is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips" tags.

Alternatively, this test can be run using:

  $ avocado --show=console run -t arch:mips tests/acceptance/boot_linux_console.py
  [...]
  console: Boot successful.
  [...]
  console: / # uname -a
  console: Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
  console: / # reboot
  console: / # reboot: Restarting system

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 49 ++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 1c330871c0..60ea240ab6 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -11,6 +11,7 @@
 import os
 import logging
 import lzma
+import gzip
 import shutil
 
 from avocado_qemu import Test
@@ -47,6 +48,11 @@ class BootLinuxConsole(Test):
                 fail = 'Failure message found in console: %s' % failure_message
                 self.fail(fail)
 
+    def exec_command_and_wait_for_pattern(self, command, success_message):
+        command += '\n'
+        self.vm.console_socket.sendall(command.encode())
+        self.wait_for_console_pattern(success_message)
+
     def extract_from_deb(self, deb, path):
         """
         Extracts a file from a deb package into the test workdir
@@ -138,6 +144,49 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_mips_malta_cpio(self):
+        """
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20160601T041800Z/pool/main/l/linux/'
+                   'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
+        deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-4.5.0-2-4kc-malta')
+        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
+                      '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
+                      'mips/rootfs.cpio.gz')
+        initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
+        initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
+        initrd_path = self.workdir + "rootfs.cpio"
+
+        with gzip.open(initrd_path_gz, 'rb') as f_in:
+            with open(initrd_path, 'wb') as f_out:
+                shutil.copyfileobj(f_in, f_out)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 console=tty '
+                               + 'rdinit=/sbin/init noreboot')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-initrd', initrd_path,
+                         '-append', kernel_command_line,
+                         '-no-reboot')
+        self.vm.launch()
+        self.wait_for_console_pattern('Boot successful.')
+
+        self.exec_command_and_wait_for_pattern('cat /proc/cpuinfo',
+                                               'BogoMIPS')
+        self.exec_command_and_wait_for_pattern('uname -a',
+                                               'Debian')
+        self.exec_command_and_wait_for_pattern('reboot',
+                                               'reboot: Restarting system')
+
     def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
         kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
         kernel_path = self.workdir + "kernel"
-- 
2.19.1

Re: [Qemu-devel] [PATCH 3/4] BootLinuxConsoleTest: Run kerneltests BusyBox on Malta

[Aleksandar Markovic]

On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
>
> This tests boots a Linux kernel on a Malta machine up to a
> busybox shell on the serial console. Few commands are executed
> before halting the machine (via reboot).
>
> We use the initrd cpio image from the kerneltests project:
> https://kerneltests.org/
>
> If MIPS is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:mips" tags.
>
> Alternatively, this test can be run using:
>
>   $ avocado --show=console run -t arch:mips
tests/acceptance/boot_linux_console.py
>   [...]
>   console: Boot successful.
>   [...]
>   console: / # uname -a
>   console: Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1
(2016-05-29) mips GNU/Linux
>   console: / # reboot
>   console: / # reboot: Restarting system
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---

Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>

>  tests/acceptance/boot_linux_console.py | 49 ++++++++++++++++++++++++++
>  1 file changed, 49 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux_console.py
b/tests/acceptance/boot_linux_console.py
> index 1c330871c0..60ea240ab6 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -11,6 +11,7 @@
>  import os
>  import logging
>  import lzma
> +import gzip
>  import shutil
>
>  from avocado_qemu import Test
> @@ -47,6 +48,11 @@ class BootLinuxConsole(Test):
>                  fail = 'Failure message found in console: %s' %
failure_message
>                  self.fail(fail)
>
> +    def exec_command_and_wait_for_pattern(self, command,
success_message):
> +        command += '\n'
> +        self.vm.console_socket.sendall(command.encode())
> +        self.wait_for_console_pattern(success_message)
> +
>      def extract_from_deb(self, deb, path):
>          """
>          Extracts a file from a deb package into the test workdir
> @@ -138,6 +144,49 @@ class BootLinuxConsole(Test):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>
> +    def test_mips_malta_cpio(self):
> +        """
> +        :avocado: tags=arch:mips
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:big
> +        """
> +        deb_url = ('http://snapshot.debian.org/archive/debian/'
> +                   '20160601T041800Z/pool/main/l/linux/'
> +                   'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
> +        deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        kernel_path = self.extract_from_deb(deb_path,
> +
'/boot/vmlinux-4.5.0-2-4kc-malta')
> +        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> +                      '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
> +                      'mips/rootfs.cpio.gz')
> +        initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
> +        initrd_path_gz = self.fetch_asset(initrd_url,
asset_hash=initrd_hash)
> +        initrd_path = self.workdir + "rootfs.cpio"
> +
> +        with gzip.open(initrd_path_gz, 'rb') as f_in:
> +            with open(initrd_path, 'wb') as f_out:
> +                shutil.copyfileobj(f_in, f_out)
> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> +                               + 'console=ttyS0 console=tty '
> +                               + 'rdinit=/sbin/init noreboot')
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-initrd', initrd_path,
> +                         '-append', kernel_command_line,
> +                         '-no-reboot')
> +        self.vm.launch()
> +        self.wait_for_console_pattern('Boot successful.')
> +
> +        self.exec_command_and_wait_for_pattern('cat /proc/cpuinfo',
> +                                               'BogoMIPS')
> +        self.exec_command_and_wait_for_pattern('uname -a',
> +                                               'Debian')
> +        self.exec_command_and_wait_for_pattern('reboot',
> +                                               'reboot: Restarting
system')
> +
>      def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
>          kernel_path_xz = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
>          kernel_path = self.workdir + "kernel"
> --
> 2.19.1
>

Re: [Qemu-devel] [PATCH 3/4] BootLinuxConsoleTest: Run kerneltests BusyBox on Malta

[Cleber Rosa]

On Tue, May 21, 2019 at 01:19:09AM +0200, Philippe Mathieu-Daudé wrote:
> This tests boots a Linux kernel on a Malta machine up to a
> busybox shell on the serial console. Few commands are executed
> before halting the machine (via reboot).
> 
> We use the initrd cpio image from the kerneltests project:
> https://kerneltests.org/
> 
> If MIPS is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:mips" tags.
> 
> Alternatively, this test can be run using:
> 
>   $ avocado --show=console run -t arch:mips tests/acceptance/boot_linux_console.py
>   [...]
>   console: Boot successful.
>   [...]
>   console: / # uname -a
>   console: Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
>   console: / # reboot

This seems like an ordinary input on the console...

>   console: / # reboot: Restarting system
>

And this seems like a perfectly normal output from the called
command. But, I also noticed this:

    console: cat /proc/cpuinfo
    console: / # cat /proc/cpuinfo
    console: system type            : MIPS Malta

Which seems to indicate an erroneous echo behavior of the console.
Any hints?

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 49 ++++++++++++++++++++++++++
>  1 file changed, 49 insertions(+)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 1c330871c0..60ea240ab6 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -11,6 +11,7 @@
>  import os
>  import logging
>  import lzma
> +import gzip
>  import shutil
>  
>  from avocado_qemu import Test
> @@ -47,6 +48,11 @@ class BootLinuxConsole(Test):
>                  fail = 'Failure message found in console: %s' % failure_message
>                  self.fail(fail)
>  
> +    def exec_command_and_wait_for_pattern(self, command, success_message):
> +        command += '\n'
> +        self.vm.console_socket.sendall(command.encode())
> +        self.wait_for_console_pattern(success_message)
> +
>      def extract_from_deb(self, deb, path):
>          """
>          Extracts a file from a deb package into the test workdir
> @@ -138,6 +144,49 @@ class BootLinuxConsole(Test):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>  
> +    def test_mips_malta_cpio(self):
> +        """
> +        :avocado: tags=arch:mips
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:big
> +        """
> +        deb_url = ('http://snapshot.debian.org/archive/debian/'
> +                   '20160601T041800Z/pool/main/l/linux/'
> +                   'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
> +        deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        kernel_path = self.extract_from_deb(deb_path,
> +                                            '/boot/vmlinux-4.5.0-2-4kc-malta')
> +        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> +                      '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
> +                      'mips/rootfs.cpio.gz')
> +        initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
> +        initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
> +        initrd_path = self.workdir + "rootfs.cpio"
> +
> +        with gzip.open(initrd_path_gz, 'rb') as f_in:
> +            with open(initrd_path, 'wb') as f_out:
> +                shutil.copyfileobj(f_in, f_out)

The last four lines can be replaced by:

        initrd_path = archive.extract(initrd_path_gz,
                                      os.path.join(self.workdir, "rootfs.cpio"))

And with that, the gzip import can also be removed.

> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> +                               + 'console=ttyS0 console=tty '
> +                               + 'rdinit=/sbin/init noreboot')
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-initrd', initrd_path,
> +                         '-append', kernel_command_line,
> +                         '-no-reboot')
> +        self.vm.launch()
> +        self.wait_for_console_pattern('Boot successful.')
> +
> +        self.exec_command_and_wait_for_pattern('cat /proc/cpuinfo',
> +                                               'BogoMIPS')
> +        self.exec_command_and_wait_for_pattern('uname -a',
> +                                               'Debian')
> +        self.exec_command_and_wait_for_pattern('reboot',
> +                                               'reboot: Restarting system')
> +
>      def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
>          kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>          kernel_path = self.workdir + "kernel"
> -- 
> 2.19.1
> 

Tested-by: Cleber Rosa <crosa@redhat.com>

[Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Philippe Mathieu-Daudé]

This tests boot a full VM and check the serial console until
the SSH daemon is running, then start a SSH session and run
some commands.

This test can be run using:

  $ avocado --show=ssh run -t arch:mips tests/acceptance/linux_ssh_mips_malta.py
  ssh: Entering interactive session.
  ssh: # uname -a
  ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips GNU/Linux
  ssh: # lspci -d 11ab:4620
  ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd. GT-64120/64120A/64121A System Controller (rev 10)
  ssh: # cat /sys/bus/i2c/devices/i2c-0/name
  ssh: SMBus PIIX4 adapter at 1100
  ssh: # cat /proc/mtd
  ssh: dev:    size   erasesize  name
  ssh: mtd0: 00100000 00010000 "YAMON"
  ssh: mtd1: 002e0000 00010000 "User FS"
  ssh: mtd2: 00020000 00010000 "Board Config"
  ssh: # md5sum /dev/mtd2ro
  ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
  ssh: # poweroff

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
TODO: do not run this tests by default, use the 'slow' tag
---
 MAINTAINERS                              |   1 +
 tests/acceptance/linux_ssh_mips_malta.py | 229 +++++++++++++++++++++++
 tests/requirements.txt                   |   1 +
 3 files changed, 231 insertions(+)
 create mode 100644 tests/acceptance/linux_ssh_mips_malta.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 9424a490d6..69fa4b3abc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -934,6 +934,7 @@ M: Aurelien Jarno <aurelien@aurel32.net>
 R: Aleksandar Rikalo <arikalo@wavecomp.com>
 S: Maintained
 F: hw/mips/mips_malta.c
+F: tests/acceptance/linux_ssh_mips_malta.py
 
 Mipssim
 M: Aleksandar Markovic <amarkovic@wavecomp.com>
diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py
new file mode 100644
index 0000000000..ceb530ff88
--- /dev/null
+++ b/tests/acceptance/linux_ssh_mips_malta.py
@@ -0,0 +1,229 @@
+# Functional test that boots a VM and run commands via a SSH session
+#
+# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import re
+import base64
+import logging
+import paramiko
+import time
+
+from avocado_qemu import Test
+from avocado.utils import process
+from avocado.utils import archive
+
+
+class LinuxSSH(Test):
+
+    timeout = 150 # Not for 'configure --enable-debug --enable-debug-tcg'
+
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    VM_IP = '127.0.0.1'
+
+    IMAGE_INFO = {
+        'be': {
+            'image_url': 'https://people.debian.org/~aurel32/qemu/mips/'
+                         'debian_wheezy_mips_standard.qcow2',
+            'image_hash': '8987a63270df67345b2135a6b7a4885a35e392d5',
+            'rsa_hostkey': b'AAAAB3NzaC1yc2EAAAADAQABAAABAQCca1VitiyLAdQOld'
+                           b'zT43IOEVJZ0wHD78GJi8wDAjMiYWUzNSSn0rXGQsINHuH5'
+                           b'IlF+kBZsHinb/FtKCAyS9a8uCHhQI4SuB4QhAb0+39MlUw'
+                           b'Mm0CLkctgM2eUUZ6MQMQvDlqnue6CCkxN62EZYbaxmby7j'
+                           b'CQa1125o1HRKBvdGm2zrJWxXAfA+f1v6jHLyE8Jnu83eQ+'
+                           b'BFY25G+Vzx1PVc3zQBwJ8r0NGTRqy2//oWQP0h+bMsgeFe'
+                           b'KH/J3RJM22vg6+I4JAdBFcxnK+l781h1FuRxOn4O/Xslbg'
+                           b'go6WtB4V4TOsw2E/KfxI5IZ/icxF+swVcnvF46Hf3uQc/0'
+                           b'BBqb',
+        },
+        'le': {
+            'image_url': 'https://people.debian.org/~aurel32/qemu/mipsel/'
+                         'debian_wheezy_mipsel_standard.qcow2',
+            'image_hash': '7866764d9de3ef536ffca24c9fb9f04ffdb45802',
+            'rsa_hostkey': b'AAAAB3NzaC1yc2EAAAADAQABAAABAQClXJlBT71HL5yKvv'
+                           b'gfC7jmxSWx5zSBCzET6CLZczwAafSIs7YKfNOy/dQTxhuk'
+                           b'yIGFUugZFoF3E9PzdhunuyvyTd56MPoNIqFbb5rGokwU5I'
+                           b'TOx3dBHZR0mClypL6MVrwe0bsiIb8GhF1zioNwcsaAZnAi'
+                           b'KfXStVDtXvn/kLLq+xLABYt48CC5KYWoFaCoICskLAY+qo'
+                           b'L+LWyAnQisj4jAH8VSaSKIImFpfkHWEXPhHcC4ZBlDKtnH'
+                           b'po9vhfCHgnfW3Pzrqmk8BI4HysqPFVmJWkJGlGUL+sGeg3'
+                           b'ZZolAYuDXGuBrw8ooPJq2v2dOH+z6dyD2q/ypmAbyPqj5C'
+                           b'rc8H',
+        },
+    }
+
+    def wait_for_console_pattern(self, success_message,
+                                 failure_message='Oops'):
+        console = self.vm.console_socket.makefile()
+        console_logger = logging.getLogger('console')
+        while True:
+            msg = console.readline()
+            console_logger.debug(msg.strip())
+            if success_message in msg:
+                break
+            if failure_message in msg:
+                fail = 'Failure message found in console: %s' % failure_message
+                self.fail(fail)
+
+    def get_portfwd(self):
+        res = self.vm.command('human-monitor-command',
+                              command_line='info usernet')
+        line = res.split('\r\n')[2]
+        port = re.split(r'.*TCP.HOST_FORWARD.*127\.0\.0\.1 (\d+)\s+10\..*',
+                        line)[1]
+        self.log.debug("sshd listening on port:" + port)
+        return port
+
+    def ssh_connect(self, username, password, rsa_hostkey_b64=None):
+        self.ssh_logger = logging.getLogger('ssh')
+        self.ssh_username = username
+        self.ssh_ps1 = '# ' if username is 'root' else '$ '
+        self.ssh_client = paramiko.SSHClient()
+        port = self.get_portfwd()
+        if rsa_hostkey_b64:
+            rsa_hostkey_bin = base64.b64decode(rsa_hostkey_b64)
+            rsa_hostkey = paramiko.RSAKey(data = rsa_hostkey_bin)
+            ipport = '[%s]:%s' % (self.VM_IP, port)
+            self.ssh_logger.debug('ipport ' + ipport)
+            self.ssh_client.get_host_keys().add(ipport, 'ssh-rsa', rsa_hostkey)
+        for i in range(10):
+            try:
+                self.ssh_client.connect(self.VM_IP, int(port),
+                                        username, password, banner_timeout=90)
+                self.ssh_logger.info("Entering interactive session.")
+                return
+            except:
+                time.sleep(4)
+                pass
+        self.fail("sshd timeout")
+
+    def ssh_disconnect_vm(self):
+        self.ssh_client.close()
+
+    def ssh_command(self, command, is_root=True):
+        self.ssh_logger.info(self.ssh_ps1 + command)
+        stdin, stdout, stderr = self.ssh_client.exec_command(command)
+        stdout_lines = [line.strip('\n') for line in stdout]
+        for line in stdout_lines:
+            self.ssh_logger.info(line)
+        stderr_lines = [line.strip('\n') for line in stderr]
+        for line in stderr_lines:
+            self.ssh_logger.warning(line)
+        return stdout_lines, stderr_lines
+
+    def boot_debian_wheezy_image_and_ssh_login(self, endianess, kernel_path):
+        image_url = self.IMAGE_INFO[endianess]['image_url']
+        image_hash = self.IMAGE_INFO[endianess]['image_hash']
+        image_path = self.fetch_asset(image_url, asset_hash=image_hash)
+        rsa_hostkey_b64 = self.IMAGE_INFO[endianess]['rsa_hostkey']
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 root=/dev/sda1')
+        self.vm.add_args('-no-reboot',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line,
+                         '-hda', image_path,
+                         '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22',
+                         '-device', 'pcnet,netdev=vnet')
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for sshd')
+        console_pattern = 'Starting OpenBSD Secure Shell server: sshd'
+        self.wait_for_console_pattern(console_pattern)
+        self.log.info('sshd ready')
+
+        self.ssh_connect('root', 'root', rsa_hostkey_b64=rsa_hostkey_b64)
+
+    def shutdown_via_ssh(self):
+        self.ssh_command('poweroff')
+        self.ssh_disconnect_vm()
+        self.wait_for_console_pattern('Power down')
+
+    def run_common_commands(self):
+        stdout, stderr = self.ssh_command('lspci -d 11ab:4620')
+        self.assertIn(True, ["GT-64120" in line for line in stdout])
+
+        stdout, stderr = self.ssh_command('cat /sys/bus/i2c/devices/i2c-0/name')
+        self.assertIn(True, ["SMBus PIIX4 adapter" in line
+                             for line in stdout])
+
+        stdout, stderr = self.ssh_command('cat /proc/mtd')
+        self.assertIn(True, ["YAMON" in line
+                             for line in stdout])
+
+        # Empty 'Board Config'
+        stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro')
+        self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line
+                             for line in stdout])
+
+    def do_test_mips_malta(self, endianess, kernel_path, uname_m):
+        self.boot_debian_wheezy_image_and_ssh_login(endianess, kernel_path)
+
+        stdout, stderr = self.ssh_command('uname -a')
+        self.assertIn(True, [uname_m + " GNU/Linux" in line for line in stdout])
+
+        self.run_common_commands()
+        self.shutdown_via_ssh()
+
+    def test_mips_malta32eb_kernel3_2_0(self):
+        """
+        :avocado: tags=slow
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        :avocado: tags=device:pcnet32
+        """
+        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
+                      'vmlinux-3.2.0-4-4kc-malta')
+        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.do_test_mips_malta('be', kernel_path, 'mips')
+
+    def test_mips_malta32el_kernel3_2_0(self):
+        """
+        :avocado: tags=slow
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        :avocado: tags=device:pcnet32
+        """
+        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
+                      'vmlinux-3.2.0-4-4kc-malta')
+        kernel_hash = 'a66bea5a8adaa2cb3d36a1d4e0ccdb01be8f6c2a'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.do_test_mips_malta('le', kernel_path, 'mips')
+
+    def test_mips_malta64eb_kernel3_2_0(self):
+        """
+        :avocado: tags=slow
+        :avocado: tags=arch:mips64
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        :avocado: tags=device:pcnet32
+        """
+        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
+                      'vmlinux-3.2.0-4-5kc-malta')
+        kernel_hash = 'db6eea7de35d36c77d8c165b6bcb222e16eb91db'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta('be', kernel_path, 'mips64')
+
+    def test_mips_malta64el_kernel3_2_0(self):
+        """
+        :avocado: tags=slow
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        :avocado: tags=device:pcnet32
+        """
+        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
+                      'vmlinux-3.2.0-4-5kc-malta')
+        kernel_hash = '6a7f77245acf231415a0e8b725d91ed2f3487794'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta('le', kernel_path, 'mips64')
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 002ded6a22..3ae0e29ad7 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -2,3 +2,4 @@
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 avocado-framework==68.0
+paramiko
-- 
2.19.1

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Aleksandar Markovic]

On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
>
> This tests boot a full VM and check the serial console until
> the SSH daemon is running, then start a SSH session and run
> some commands.
>
> This test can be run using:
>
>   $ avocado --show=ssh run -t arch:mips
tests/acceptance/linux_ssh_mips_malta.py
>   ssh: Entering interactive session.
>   ssh: # uname -a
>   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
GNU/Linux
>   ssh: # lspci -d 11ab:4620
>   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
GT-64120/64120A/64121A System Controller (rev 10)
>   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
>   ssh: SMBus PIIX4 adapter at 1100
>   ssh: # cat /proc/mtd
>   ssh: dev:    size   erasesize  name
>   ssh: mtd0: 00100000 00010000 "YAMON"
>   ssh: mtd1: 002e0000 00010000 "User FS"
>   ssh: mtd2: 00020000 00010000 "Board Config"
>   ssh: # md5sum /dev/mtd2ro
>   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
>   ssh: # poweroff
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---

Excelent! Frankly, this was something that we in MIPS needed and missed
very much for a long time.

I liked the idea that this test does not run as a default, giving us
opportunity to extend and adjust it in future as we deem appropriate for
our MIPS needs, but without affecting people running default test
execution. In other words, this patch achieves “best of both worlds”.

Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>

> TODO: do not run this tests by default, use the 'slow' tag
> ---
>  MAINTAINERS                              |   1 +
>  tests/acceptance/linux_ssh_mips_malta.py | 229 +++++++++++++++++++++++
>  tests/requirements.txt                   |   1 +
>  3 files changed, 231 insertions(+)
>  create mode 100644 tests/acceptance/linux_ssh_mips_malta.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 9424a490d6..69fa4b3abc 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -934,6 +934,7 @@ M: Aurelien Jarno <aurelien@aurel32.net>
>  R: Aleksandar Rikalo <arikalo@wavecomp.com>
>  S: Maintained
>  F: hw/mips/mips_malta.c
> +F: tests/acceptance/linux_ssh_mips_malta.py
>
>  Mipssim
>  M: Aleksandar Markovic <amarkovic@wavecomp.com>
> diff --git a/tests/acceptance/linux_ssh_mips_malta.py
b/tests/acceptance/linux_ssh_mips_malta.py
> new file mode 100644
> index 0000000000..ceb530ff88
> --- /dev/null
> +++ b/tests/acceptance/linux_ssh_mips_malta.py
> @@ -0,0 +1,229 @@
> +# Functional test that boots a VM and run commands via a SSH session
> +#
> +# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +import re
> +import base64
> +import logging
> +import paramiko
> +import time
> +
> +from avocado_qemu import Test
> +from avocado.utils import process
> +from avocado.utils import archive
> +
> +
> +class LinuxSSH(Test):
> +
> +    timeout = 150 # Not for 'configure --enable-debug --enable-debug-tcg'
> +
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> +    VM_IP = '127.0.0.1'
> +
> +    IMAGE_INFO = {
> +        'be': {
> +            'image_url': 'https://people.debian.org/~aurel32/qemu/mips/'
> +                         'debian_wheezy_mips_standard.qcow2',
> +            'image_hash': '8987a63270df67345b2135a6b7a4885a35e392d5',
> +            'rsa_hostkey':
b'AAAAB3NzaC1yc2EAAAADAQABAAABAQCca1VitiyLAdQOld'
> +
 b'zT43IOEVJZ0wHD78GJi8wDAjMiYWUzNSSn0rXGQsINHuH5'
> +
 b'IlF+kBZsHinb/FtKCAyS9a8uCHhQI4SuB4QhAb0+39MlUw'
> +
 b'Mm0CLkctgM2eUUZ6MQMQvDlqnue6CCkxN62EZYbaxmby7j'
> +
 b'CQa1125o1HRKBvdGm2zrJWxXAfA+f1v6jHLyE8Jnu83eQ+'
> +
 b'BFY25G+Vzx1PVc3zQBwJ8r0NGTRqy2//oWQP0h+bMsgeFe'
> +
 b'KH/J3RJM22vg6+I4JAdBFcxnK+l781h1FuRxOn4O/Xslbg'
> +
 b'go6WtB4V4TOsw2E/KfxI5IZ/icxF+swVcnvF46Hf3uQc/0'
> +                           b'BBqb',
> +        },
> +        'le': {
> +            'image_url': 'https://people.debian.org/~aurel32/qemu/mipsel/
'
> +                         'debian_wheezy_mipsel_standard.qcow2',
> +            'image_hash': '7866764d9de3ef536ffca24c9fb9f04ffdb45802',
> +            'rsa_hostkey':
b'AAAAB3NzaC1yc2EAAAADAQABAAABAQClXJlBT71HL5yKvv'
> +
 b'gfC7jmxSWx5zSBCzET6CLZczwAafSIs7YKfNOy/dQTxhuk'
> +
 b'yIGFUugZFoF3E9PzdhunuyvyTd56MPoNIqFbb5rGokwU5I'
> +
 b'TOx3dBHZR0mClypL6MVrwe0bsiIb8GhF1zioNwcsaAZnAi'
> +
 b'KfXStVDtXvn/kLLq+xLABYt48CC5KYWoFaCoICskLAY+qo'
> +
 b'L+LWyAnQisj4jAH8VSaSKIImFpfkHWEXPhHcC4ZBlDKtnH'
> +
 b'po9vhfCHgnfW3Pzrqmk8BI4HysqPFVmJWkJGlGUL+sGeg3'
> +
 b'ZZolAYuDXGuBrw8ooPJq2v2dOH+z6dyD2q/ypmAbyPqj5C'
> +                           b'rc8H',
> +        },
> +    }
> +
> +    def wait_for_console_pattern(self, success_message,
> +                                 failure_message='Oops'):
> +        console = self.vm.console_socket.makefile()
> +        console_logger = logging.getLogger('console')
> +        while True:
> +            msg = console.readline()
> +            console_logger.debug(msg.strip())
> +            if success_message in msg:
> +                break
> +            if failure_message in msg:
> +                fail = 'Failure message found in console: %s' %
failure_message
> +                self.fail(fail)
> +
> +    def get_portfwd(self):
> +        res = self.vm.command('human-monitor-command',
> +                              command_line='info usernet')
> +        line = res.split('\r\n')[2]
> +        port = re.split(r'.*TCP.HOST_FORWARD.*127\.0\.0\.1
(\d+)\s+10\..*',
> +                        line)[1]
> +        self.log.debug("sshd listening on port:" + port)
> +        return port
> +
> +    def ssh_connect(self, username, password, rsa_hostkey_b64=None):
> +        self.ssh_logger = logging.getLogger('ssh')
> +        self.ssh_username = username
> +        self.ssh_ps1 = '# ' if username is 'root' else '$ '
> +        self.ssh_client = paramiko.SSHClient()
> +        port = self.get_portfwd()
> +        if rsa_hostkey_b64:
> +            rsa_hostkey_bin = base64.b64decode(rsa_hostkey_b64)
> +            rsa_hostkey = paramiko.RSAKey(data = rsa_hostkey_bin)
> +            ipport = '[%s]:%s' % (self.VM_IP, port)
> +            self.ssh_logger.debug('ipport ' + ipport)
> +            self.ssh_client.get_host_keys().add(ipport, 'ssh-rsa',
rsa_hostkey)
> +        for i in range(10):
> +            try:
> +                self.ssh_client.connect(self.VM_IP, int(port),
> +                                        username, password,
banner_timeout=90)
> +                self.ssh_logger.info("Entering interactive session.")
> +                return
> +            except:
> +                time.sleep(4)
> +                pass
> +        self.fail("sshd timeout")
> +
> +    def ssh_disconnect_vm(self):
> +        self.ssh_client.close()
> +
> +    def ssh_command(self, command, is_root=True):
> +        self.ssh_logger.info(self.ssh_ps1 + command)
> +        stdin, stdout, stderr = self.ssh_client.exec_command(command)
> +        stdout_lines = [line.strip('\n') for line in stdout]
> +        for line in stdout_lines:
> +            self.ssh_logger.info(line)
> +        stderr_lines = [line.strip('\n') for line in stderr]
> +        for line in stderr_lines:
> +            self.ssh_logger.warning(line)
> +        return stdout_lines, stderr_lines
> +
> +    def boot_debian_wheezy_image_and_ssh_login(self, endianess,
kernel_path):
> +        image_url = self.IMAGE_INFO[endianess]['image_url']
> +        image_hash = self.IMAGE_INFO[endianess]['image_hash']
> +        image_path = self.fetch_asset(image_url, asset_hash=image_hash)
> +        rsa_hostkey_b64 = self.IMAGE_INFO[endianess]['rsa_hostkey']
> +
> +        self.vm.set_machine('malta')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> +                               + 'console=ttyS0 root=/dev/sda1')
> +        self.vm.add_args('-no-reboot',
> +                         '-kernel', kernel_path,
> +                         '-append', kernel_command_line,
> +                         '-hda', image_path,
> +                         '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0
-:22',
> +                         '-device', 'pcnet,netdev=vnet')
> +        self.vm.launch()
> +
> +        self.log.info('VM launched, waiting for sshd')
> +        console_pattern = 'Starting OpenBSD Secure Shell server: sshd'
> +        self.wait_for_console_pattern(console_pattern)
> +        self.log.info('sshd ready')
> +
> +        self.ssh_connect('root', 'root', rsa_hostkey_b64=rsa_hostkey_b64)
> +
> +    def shutdown_via_ssh(self):
> +        self.ssh_command('poweroff')
> +        self.ssh_disconnect_vm()
> +        self.wait_for_console_pattern('Power down')
> +
> +    def run_common_commands(self):
> +        stdout, stderr = self.ssh_command('lspci -d 11ab:4620')
> +        self.assertIn(True, ["GT-64120" in line for line in stdout])
> +
> +        stdout, stderr = self.ssh_command('cat
/sys/bus/i2c/devices/i2c-0/name')
> +        self.assertIn(True, ["SMBus PIIX4 adapter" in line
> +                             for line in stdout])
> +
> +        stdout, stderr = self.ssh_command('cat /proc/mtd')
> +        self.assertIn(True, ["YAMON" in line
> +                             for line in stdout])
> +
> +        # Empty 'Board Config'
> +        stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro')
> +        self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line
> +                             for line in stdout])
> +
> +    def do_test_mips_malta(self, endianess, kernel_path, uname_m):
> +        self.boot_debian_wheezy_image_and_ssh_login(endianess,
kernel_path)
> +
> +        stdout, stderr = self.ssh_command('uname -a')
> +        self.assertIn(True, [uname_m + " GNU/Linux" in line for line in
stdout])
> +
> +        self.run_common_commands()
> +        self.shutdown_via_ssh()
> +
> +    def test_mips_malta32eb_kernel3_2_0(self):
> +        """
> +        :avocado: tags=slow
> +        :avocado: tags=arch:mips
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:big
> +        :avocado: tags=device:pcnet32
> +        """
> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
> +                      'vmlinux-3.2.0-4-4kc-malta')
> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> +
> +        self.do_test_mips_malta('be', kernel_path, 'mips')
> +
> +    def test_mips_malta32el_kernel3_2_0(self):
> +        """
> +        :avocado: tags=slow
> +        :avocado: tags=arch:mipsel
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        :avocado: tags=device:pcnet32
> +        """
> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
> +                      'vmlinux-3.2.0-4-4kc-malta')
> +        kernel_hash = 'a66bea5a8adaa2cb3d36a1d4e0ccdb01be8f6c2a'
> +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> +
> +        self.do_test_mips_malta('le', kernel_path, 'mips')
> +
> +    def test_mips_malta64eb_kernel3_2_0(self):
> +        """
> +        :avocado: tags=slow
> +        :avocado: tags=arch:mips64
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:big
> +        :avocado: tags=device:pcnet32
> +        """
> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
> +                      'vmlinux-3.2.0-4-5kc-malta')
> +        kernel_hash = 'db6eea7de35d36c77d8c165b6bcb222e16eb91db'
> +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> +        self.do_test_mips_malta('be', kernel_path, 'mips64')
> +
> +    def test_mips_malta64el_kernel3_2_0(self):
> +        """
> +        :avocado: tags=slow
> +        :avocado: tags=arch:mips64el
> +        :avocado: tags=machine:malta
> +        :avocado: tags=endian:little
> +        :avocado: tags=device:pcnet32
> +        """
> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
> +                      'vmlinux-3.2.0-4-5kc-malta')
> +        kernel_hash = '6a7f77245acf231415a0e8b725d91ed2f3487794'
> +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> +        self.do_test_mips_malta('le', kernel_path, 'mips64')
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 002ded6a22..3ae0e29ad7 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,3 +2,4 @@
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>  avocado-framework==68.0
> +paramiko
> --
> 2.19.1
>

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Aleksandar Markovic]

On May 21, 2019 10:18 AM, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
wrote:
>
>
> On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
> >
> > This tests boot a full VM and check the serial console until
> > the SSH daemon is running, then start a SSH session and run
> > some commands.
> >

If there is no objection from Cleber or others, I would like to apply this
patch to my next MIPS pull request. We in Wave just started regular
regression tests of various nature for QEMU 4.1 for MIPS, and would really
like to have this test integrated as soon as possible, that is why I am
rushing a bit. I am talking here about this patch only, since it is a sort
of an independant unit - I will not do anything with other remaining
patches from this series.

Please let me know if you have any concerns.

Sincerely, Aleksandar

> > This test can be run using:
> >
> >   $ avocado --show=ssh run -t arch:mips
tests/acceptance/linux_ssh_mips_malta.py
> >   ssh: Entering interactive session.
> >   ssh: # uname -a
> >   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
GNU/Linux
> >   ssh: # lspci -d 11ab:4620
> >   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
GT-64120/64120A/64121A System Controller (rev 10)
> >   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
> >   ssh: SMBus PIIX4 adapter at 1100
> >   ssh: # cat /proc/mtd
> >   ssh: dev:    size   erasesize  name
> >   ssh: mtd0: 00100000 00010000 "YAMON"
> >   ssh: mtd1: 002e0000 00010000 "User FS"
> >   ssh: mtd2: 00020000 00010000 "Board Config"
> >   ssh: # md5sum /dev/mtd2ro
> >   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
> >   ssh: # poweroff
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
>
> Excelent! Frankly, this was something that we in MIPS needed and missed
very much for a long time.
>
> I liked the idea that this test does not run as a default, giving us
opportunity to extend and adjust it in future as we deem appropriate for
our MIPS needs, but without affecting people running default test
execution. In other words, this patch achieves “best of both worlds”.
>
> Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
>
> > TODO: do not run this tests by default, use the 'slow' tag
> > ---
> >  MAINTAINERS                              |   1 +
> >  tests/acceptance/linux_ssh_mips_malta.py | 229 +++++++++++++++++++++++
> >  tests/requirements.txt                   |   1 +
> >  3 files changed, 231 insertions(+)
> >  create mode 100644 tests/acceptance/linux_ssh_mips_malta.py
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 9424a490d6..69fa4b3abc 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -934,6 +934,7 @@ M: Aurelien Jarno <aurelien@aurel32.net>
> >  R: Aleksandar Rikalo <arikalo@wavecomp.com>
> >  S: Maintained
> >  F: hw/mips/mips_malta.c
> > +F: tests/acceptance/linux_ssh_mips_malta.py
> >
> >  Mipssim
> >  M: Aleksandar Markovic <amarkovic@wavecomp.com>
> > diff --git a/tests/acceptance/linux_ssh_mips_malta.py
b/tests/acceptance/linux_ssh_mips_malta.py
> > new file mode 100644
> > index 0000000000..ceb530ff88
> > --- /dev/null
> > +++ b/tests/acceptance/linux_ssh_mips_malta.py
> > @@ -0,0 +1,229 @@
> > +# Functional test that boots a VM and run commands via a SSH session
> > +#
> > +# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later.  See the COPYING file in the top-level directory.
> > +
> > +import os
> > +import re
> > +import base64
> > +import logging
> > +import paramiko
> > +import time
> > +
> > +from avocado_qemu import Test
> > +from avocado.utils import process
> > +from avocado.utils import archive
> > +
> > +
> > +class LinuxSSH(Test):
> > +
> > +    timeout = 150 # Not for 'configure --enable-debug
--enable-debug-tcg'
> > +
> > +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> > +    VM_IP = '127.0.0.1'
> > +
> > +    IMAGE_INFO = {
> > +        'be': {
> > +            'image_url': 'https://people.debian.org/~aurel32/qemu/mips/
'
> > +                         'debian_wheezy_mips_standard.qcow2',
> > +            'image_hash': '8987a63270df67345b2135a6b7a4885a35e392d5',
> > +            'rsa_hostkey':
b'AAAAB3NzaC1yc2EAAAADAQABAAABAQCca1VitiyLAdQOld'
> > +
 b'zT43IOEVJZ0wHD78GJi8wDAjMiYWUzNSSn0rXGQsINHuH5'
> > +
 b'IlF+kBZsHinb/FtKCAyS9a8uCHhQI4SuB4QhAb0+39MlUw'
> > +
 b'Mm0CLkctgM2eUUZ6MQMQvDlqnue6CCkxN62EZYbaxmby7j'
> > +
 b'CQa1125o1HRKBvdGm2zrJWxXAfA+f1v6jHLyE8Jnu83eQ+'
> > +
 b'BFY25G+Vzx1PVc3zQBwJ8r0NGTRqy2//oWQP0h+bMsgeFe'
> > +
 b'KH/J3RJM22vg6+I4JAdBFcxnK+l781h1FuRxOn4O/Xslbg'
> > +
 b'go6WtB4V4TOsw2E/KfxI5IZ/icxF+swVcnvF46Hf3uQc/0'
> > +                           b'BBqb',
> > +        },
> > +        'le': {
> > +            'image_url': '
https://people.debian.org/~aurel32/qemu/mipsel/'
> > +                         'debian_wheezy_mipsel_standard.qcow2',
> > +            'image_hash': '7866764d9de3ef536ffca24c9fb9f04ffdb45802',
> > +            'rsa_hostkey':
b'AAAAB3NzaC1yc2EAAAADAQABAAABAQClXJlBT71HL5yKvv'
> > +
 b'gfC7jmxSWx5zSBCzET6CLZczwAafSIs7YKfNOy/dQTxhuk'
> > +
 b'yIGFUugZFoF3E9PzdhunuyvyTd56MPoNIqFbb5rGokwU5I'
> > +
 b'TOx3dBHZR0mClypL6MVrwe0bsiIb8GhF1zioNwcsaAZnAi'
> > +
 b'KfXStVDtXvn/kLLq+xLABYt48CC5KYWoFaCoICskLAY+qo'
> > +
 b'L+LWyAnQisj4jAH8VSaSKIImFpfkHWEXPhHcC4ZBlDKtnH'
> > +
 b'po9vhfCHgnfW3Pzrqmk8BI4HysqPFVmJWkJGlGUL+sGeg3'
> > +
 b'ZZolAYuDXGuBrw8ooPJq2v2dOH+z6dyD2q/ypmAbyPqj5C'
> > +                           b'rc8H',
> > +        },
> > +    }
> > +
> > +    def wait_for_console_pattern(self, success_message,
> > +                                 failure_message='Oops'):
> > +        console = self.vm.console_socket.makefile()
> > +        console_logger = logging.getLogger('console')
> > +        while True:
> > +            msg = console.readline()
> > +            console_logger.debug(msg.strip())
> > +            if success_message in msg:
> > +                break
> > +            if failure_message in msg:
> > +                fail = 'Failure message found in console: %s' %
failure_message
> > +                self.fail(fail)
> > +
> > +    def get_portfwd(self):
> > +        res = self.vm.command('human-monitor-command',
> > +                              command_line='info usernet')
> > +        line = res.split('\r\n')[2]
> > +        port = re.split(r'.*TCP.HOST_FORWARD.*127\.0\.0\.1
(\d+)\s+10\..*',
> > +                        line)[1]
> > +        self.log.debug("sshd listening on port:" + port)
> > +        return port
> > +
> > +    def ssh_connect(self, username, password, rsa_hostkey_b64=None):
> > +        self.ssh_logger = logging.getLogger('ssh')
> > +        self.ssh_username = username
> > +        self.ssh_ps1 = '# ' if username is 'root' else '$ '
> > +        self.ssh_client = paramiko.SSHClient()
> > +        port = self.get_portfwd()
> > +        if rsa_hostkey_b64:
> > +            rsa_hostkey_bin = base64.b64decode(rsa_hostkey_b64)
> > +            rsa_hostkey = paramiko.RSAKey(data = rsa_hostkey_bin)
> > +            ipport = '[%s]:%s' % (self.VM_IP, port)
> > +            self.ssh_logger.debug('ipport ' + ipport)
> > +            self.ssh_client.get_host_keys().add(ipport, 'ssh-rsa',
rsa_hostkey)
> > +        for i in range(10):
> > +            try:
> > +                self.ssh_client.connect(self.VM_IP, int(port),
> > +                                        username, password,
banner_timeout=90)
> > +                self.ssh_logger.info("Entering interactive session.")
> > +                return
> > +            except:
> > +                time.sleep(4)
> > +                pass
> > +        self.fail("sshd timeout")
> > +
> > +    def ssh_disconnect_vm(self):
> > +        self.ssh_client.close()
> > +
> > +    def ssh_command(self, command, is_root=True):
> > +        self.ssh_logger.info(self.ssh_ps1 + command)
> > +        stdin, stdout, stderr = self.ssh_client.exec_command(command)
> > +        stdout_lines = [line.strip('\n') for line in stdout]
> > +        for line in stdout_lines:
> > +            self.ssh_logger.info(line)
> > +        stderr_lines = [line.strip('\n') for line in stderr]
> > +        for line in stderr_lines:
> > +            self.ssh_logger.warning(line)
> > +        return stdout_lines, stderr_lines
> > +
> > +    def boot_debian_wheezy_image_and_ssh_login(self, endianess,
kernel_path):
> > +        image_url = self.IMAGE_INFO[endianess]['image_url']
> > +        image_hash = self.IMAGE_INFO[endianess]['image_hash']
> > +        image_path = self.fetch_asset(image_url, asset_hash=image_hash)
> > +        rsa_hostkey_b64 = self.IMAGE_INFO[endianess]['rsa_hostkey']
> > +
> > +        self.vm.set_machine('malta')
> > +        self.vm.set_console()
> > +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> > +                               + 'console=ttyS0 root=/dev/sda1')
> > +        self.vm.add_args('-no-reboot',
> > +                         '-kernel', kernel_path,
> > +                         '-append', kernel_command_line,
> > +                         '-hda', image_path,
> > +                         '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0
-:22',
> > +                         '-device', 'pcnet,netdev=vnet')
> > +        self.vm.launch()
> > +
> > +        self.log.info('VM launched, waiting for sshd')
> > +        console_pattern = 'Starting OpenBSD Secure Shell server: sshd'
> > +        self.wait_for_console_pattern(console_pattern)
> > +        self.log.info('sshd ready')
> > +
> > +        self.ssh_connect('root', 'root',
rsa_hostkey_b64=rsa_hostkey_b64)
> > +
> > +    def shutdown_via_ssh(self):
> > +        self.ssh_command('poweroff')
> > +        self.ssh_disconnect_vm()
> > +        self.wait_for_console_pattern('Power down')
> > +
> > +    def run_common_commands(self):
> > +        stdout, stderr = self.ssh_command('lspci -d 11ab:4620')
> > +        self.assertIn(True, ["GT-64120" in line for line in stdout])
> > +
> > +        stdout, stderr = self.ssh_command('cat
/sys/bus/i2c/devices/i2c-0/name')
> > +        self.assertIn(True, ["SMBus PIIX4 adapter" in line
> > +                             for line in stdout])
> > +
> > +        stdout, stderr = self.ssh_command('cat /proc/mtd')
> > +        self.assertIn(True, ["YAMON" in line
> > +                             for line in stdout])
> > +
> > +        # Empty 'Board Config'
> > +        stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro')
> > +        self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line
> > +                             for line in stdout])
> > +
> > +    def do_test_mips_malta(self, endianess, kernel_path, uname_m):
> > +        self.boot_debian_wheezy_image_and_ssh_login(endianess,
kernel_path)
> > +
> > +        stdout, stderr = self.ssh_command('uname -a')
> > +        self.assertIn(True, [uname_m + " GNU/Linux" in line for line
in stdout])
> > +
> > +        self.run_common_commands()
> > +        self.shutdown_via_ssh()
> > +
> > +    def test_mips_malta32eb_kernel3_2_0(self):
> > +        """
> > +        :avocado: tags=slow
> > +        :avocado: tags=arch:mips
> > +        :avocado: tags=machine:malta
> > +        :avocado: tags=endian:big
> > +        :avocado: tags=device:pcnet32
> > +        """
> > +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
> > +                      'vmlinux-3.2.0-4-4kc-malta')
> > +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
> > +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> > +
> > +        self.do_test_mips_malta('be', kernel_path, 'mips')
> > +
> > +    def test_mips_malta32el_kernel3_2_0(self):
> > +        """
> > +        :avocado: tags=slow
> > +        :avocado: tags=arch:mipsel
> > +        :avocado: tags=machine:malta
> > +        :avocado: tags=endian:little
> > +        :avocado: tags=device:pcnet32
> > +        """
> > +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
> > +                      'vmlinux-3.2.0-4-4kc-malta')
> > +        kernel_hash = 'a66bea5a8adaa2cb3d36a1d4e0ccdb01be8f6c2a'
> > +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> > +
> > +        self.do_test_mips_malta('le', kernel_path, 'mips')
> > +
> > +    def test_mips_malta64eb_kernel3_2_0(self):
> > +        """
> > +        :avocado: tags=slow
> > +        :avocado: tags=arch:mips64
> > +        :avocado: tags=machine:malta
> > +        :avocado: tags=endian:big
> > +        :avocado: tags=device:pcnet32
> > +        """
> > +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
> > +                      'vmlinux-3.2.0-4-5kc-malta')
> > +        kernel_hash = 'db6eea7de35d36c77d8c165b6bcb222e16eb91db'
> > +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> > +        self.do_test_mips_malta('be', kernel_path, 'mips64')
> > +
> > +    def test_mips_malta64el_kernel3_2_0(self):
> > +        """
> > +        :avocado: tags=slow
> > +        :avocado: tags=arch:mips64el
> > +        :avocado: tags=machine:malta
> > +        :avocado: tags=endian:little
> > +        :avocado: tags=device:pcnet32
> > +        """
> > +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
> > +                      'vmlinux-3.2.0-4-5kc-malta')
> > +        kernel_hash = '6a7f77245acf231415a0e8b725d91ed2f3487794'
> > +        kernel_path = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
> > +        self.do_test_mips_malta('le', kernel_path, 'mips64')
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index 002ded6a22..3ae0e29ad7 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -2,3 +2,4 @@
> >  # in the tests/venv Python virtual environment. For more info,
> >  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >  avocado-framework==68.0
> > +paramiko
> > --
> > 2.19.1
> >

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Philippe Mathieu-Daudé]

Hi Aleksandar,

On 5/21/19 11:26 AM, Aleksandar Markovic wrote:
> On May 21, 2019 10:18 AM, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> wrote:
>>
>>
>> On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
>>>
>>> This tests boot a full VM and check the serial console until
>>> the SSH daemon is running, then start a SSH session and run
>>> some commands.
>>>
> 
> If there is no objection from Cleber or others, I would like to apply this
> patch to my next MIPS pull request. We in Wave just started regular
> regression tests of various nature for QEMU 4.1 for MIPS, and would really
> like to have this test integrated as soon as possible, that is why I am
> rushing a bit. I am talking here about this patch only, since it is a sort
> of an independant unit - I will not do anything with other remaining
> patches from this series.

Thanks for the quick review :)

> 
> Please let me know if you have any concerns.

I threw this as it because it is working enough for me, but I'm secretly
hoping Cleber would take it and improve it, because I don't feel
confident with Python, and he better know Avocado ;)
This is why I also split this out of the other BootLinuxConsole test.

Anyway if it doesn't break anything and is already useful, it can get
improved later.

> Sincerely, Aleksandar
> 
>>> This test can be run using:
>>>
>>>   $ avocado --show=ssh run -t arch:mips
> tests/acceptance/linux_ssh_mips_malta.py
>>>   ssh: Entering interactive session.
>>>   ssh: # uname -a
>>>   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
> GNU/Linux
>>>   ssh: # lspci -d 11ab:4620
>>>   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
> GT-64120/64120A/64121A System Controller (rev 10)
>>>   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
>>>   ssh: SMBus PIIX4 adapter at 1100
>>>   ssh: # cat /proc/mtd
>>>   ssh: dev:    size   erasesize  name
>>>   ssh: mtd0: 00100000 00010000 "YAMON"
>>>   ssh: mtd1: 002e0000 00010000 "User FS"
>>>   ssh: mtd2: 00020000 00010000 "Board Config"
>>>   ssh: # md5sum /dev/mtd2ro
>>>   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
>>>   ssh: # poweroff

Aleksandar, do you think of other commands we should run? What commands
do you usually try?

I plan to later install more packages in the VM (in particular test
packages, FPU extensive, device tests and so on).

My rule about device testing is "if the device can be tested in a KVM
guest, don't test it on TCG guest" for obvious reason of speed.

>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>
>> Excelent! Frankly, this was something that we in MIPS needed and missed
> very much for a long time.
>>
>> I liked the idea that this test does not run as a default, giving us
> opportunity to extend and adjust it in future as we deem appropriate for
> our MIPS needs, but without affecting people running default test

Note, the tests have the "slow" avocado tag, but it is not used because
I'm not sure how to modify the 'check-acceptance' rule of
tests/Makefile.include but Cleber will probably fix this ;)

> execution. In other words, this patch achieves “best of both worlds”.
>>
>> Reviewed-by: Aleksandar Markovic <amarkovic@wavecomp.com>
>>
>>> TODO: do not run this tests by default, use the 'slow' tag
>>> ---
>>>  MAINTAINERS                              |   1 +
>>>  tests/acceptance/linux_ssh_mips_malta.py | 229 +++++++++++++++++++++++
>>>  tests/requirements.txt                   |   1 +
>>>  3 files changed, 231 insertions(+)
>>>  create mode 100644 tests/acceptance/linux_ssh_mips_malta.py
>>>
>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>> index 9424a490d6..69fa4b3abc 100644
>>> --- a/MAINTAINERS
>>> +++ b/MAINTAINERS
>>> @@ -934,6 +934,7 @@ M: Aurelien Jarno <aurelien@aurel32.net>
>>>  R: Aleksandar Rikalo <arikalo@wavecomp.com>
>>>  S: Maintained
>>>  F: hw/mips/mips_malta.c
>>> +F: tests/acceptance/linux_ssh_mips_malta.py
>>>
>>>  Mipssim
>>>  M: Aleksandar Markovic <amarkovic@wavecomp.com>
>>> diff --git a/tests/acceptance/linux_ssh_mips_malta.py
> b/tests/acceptance/linux_ssh_mips_malta.py
>>> new file mode 100644
>>> index 0000000000..ceb530ff88
>>> --- /dev/null
>>> +++ b/tests/acceptance/linux_ssh_mips_malta.py
>>> @@ -0,0 +1,229 @@
>>> +# Functional test that boots a VM and run commands via a SSH session
>>> +#
>>> +# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> +#
>>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>>> +# later.  See the COPYING file in the top-level directory.
>>> +
>>> +import os
>>> +import re
>>> +import base64
>>> +import logging
>>> +import paramiko
>>> +import time
>>> +
>>> +from avocado_qemu import Test
>>> +from avocado.utils import process
>>> +from avocado.utils import archive
>>> +
>>> +
>>> +class LinuxSSH(Test):
>>> +
>>> +    timeout = 150 # Not for 'configure --enable-debug
> --enable-debug-tcg'
>>> +
>>> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
>>> +    VM_IP = '127.0.0.1'
>>> +
>>> +    IMAGE_INFO = {
>>> +        'be': {
>>> +            'image_url': 'https://people.debian.org/~aurel32/qemu/mips/
> '
>>> +                         'debian_wheezy_mips_standard.qcow2',
>>> +            'image_hash': '8987a63270df67345b2135a6b7a4885a35e392d5',
>>> +            'rsa_hostkey':
> b'AAAAB3NzaC1yc2EAAAADAQABAAABAQCca1VitiyLAdQOld'
>>> +
>  b'zT43IOEVJZ0wHD78GJi8wDAjMiYWUzNSSn0rXGQsINHuH5'
>>> +
>  b'IlF+kBZsHinb/FtKCAyS9a8uCHhQI4SuB4QhAb0+39MlUw'
>>> +
>  b'Mm0CLkctgM2eUUZ6MQMQvDlqnue6CCkxN62EZYbaxmby7j'
>>> +
>  b'CQa1125o1HRKBvdGm2zrJWxXAfA+f1v6jHLyE8Jnu83eQ+'
>>> +
>  b'BFY25G+Vzx1PVc3zQBwJ8r0NGTRqy2//oWQP0h+bMsgeFe'
>>> +
>  b'KH/J3RJM22vg6+I4JAdBFcxnK+l781h1FuRxOn4O/Xslbg'
>>> +
>  b'go6WtB4V4TOsw2E/KfxI5IZ/icxF+swVcnvF46Hf3uQc/0'
>>> +                           b'BBqb',
>>> +        },
>>> +        'le': {
>>> +            'image_url': '
> https://people.debian.org/~aurel32/qemu/mipsel/'
>>> +                         'debian_wheezy_mipsel_standard.qcow2',
>>> +            'image_hash': '7866764d9de3ef536ffca24c9fb9f04ffdb45802',
>>> +            'rsa_hostkey':
> b'AAAAB3NzaC1yc2EAAAADAQABAAABAQClXJlBT71HL5yKvv'
>>> +
>  b'gfC7jmxSWx5zSBCzET6CLZczwAafSIs7YKfNOy/dQTxhuk'
>>> +
>  b'yIGFUugZFoF3E9PzdhunuyvyTd56MPoNIqFbb5rGokwU5I'
>>> +
>  b'TOx3dBHZR0mClypL6MVrwe0bsiIb8GhF1zioNwcsaAZnAi'
>>> +
>  b'KfXStVDtXvn/kLLq+xLABYt48CC5KYWoFaCoICskLAY+qo'
>>> +
>  b'L+LWyAnQisj4jAH8VSaSKIImFpfkHWEXPhHcC4ZBlDKtnH'
>>> +
>  b'po9vhfCHgnfW3Pzrqmk8BI4HysqPFVmJWkJGlGUL+sGeg3'
>>> +
>  b'ZZolAYuDXGuBrw8ooPJq2v2dOH+z6dyD2q/ypmAbyPqj5C'
>>> +                           b'rc8H',
>>> +        },
>>> +    }
>>> +
>>> +    def wait_for_console_pattern(self, success_message,
>>> +                                 failure_message='Oops'):

Reviewing myself:

This method might go in a generic class to handle buffered IO with QEMU
chardev.

>>> +        console = self.vm.console_socket.makefile()
>>> +        console_logger = logging.getLogger('console')
>>> +        while True:
>>> +            msg = console.readline()
>>> +            console_logger.debug(msg.strip())
>>> +            if success_message in msg:
>>> +                break
>>> +            if failure_message in msg:
>>> +                fail = 'Failure message found in console: %s' %
> failure_message
>>> +                self.fail(fail)
>>> +
>>> +    def get_portfwd(self):
>>> +        res = self.vm.command('human-monitor-command',
>>> +                              command_line='info usernet')
>>> +        line = res.split('\r\n')[2]
>>> +        port = re.split(r'.*TCP.HOST_FORWARD.*127\.0\.0\.1
> (\d+)\s+10\..*',
>>> +                        line)[1]
>>> +        self.log.debug("sshd listening on port:" + port)
>>> +        return port
>>> +
>>> +    def ssh_connect(self, username, password, rsa_hostkey_b64=None):
>>> +        self.ssh_logger = logging.getLogger('ssh')
>>> +        self.ssh_username = username
>>> +        self.ssh_ps1 = '# ' if username is 'root' else '$ '
>>> +        self.ssh_client = paramiko.SSHClient()
>>> +        port = self.get_portfwd()
>>> +        if rsa_hostkey_b64:
>>> +            rsa_hostkey_bin = base64.b64decode(rsa_hostkey_b64)
>>> +            rsa_hostkey = paramiko.RSAKey(data = rsa_hostkey_bin)
>>> +            ipport = '[%s]:%s' % (self.VM_IP, port)
>>> +            self.ssh_logger.debug('ipport ' + ipport)
>>> +            self.ssh_client.get_host_keys().add(ipport, 'ssh-rsa',
> rsa_hostkey)
>>> +        for i in range(10):
>>> +            try:
>>> +                self.ssh_client.connect(self.VM_IP, int(port),
>>> +                                        username, password,
> banner_timeout=90)
>>> +                self.ssh_logger.info("Entering interactive session.")
>>> +                return
>>> +            except:
>>> +                time.sleep(4)
>>> +                pass
>>> +        self.fail("sshd timeout")
>>> +
>>> +    def ssh_disconnect_vm(self):
>>> +        self.ssh_client.close()
>>> +
>>> +    def ssh_command(self, command, is_root=True):
>>> +        self.ssh_logger.info(self.ssh_ps1 + command)
>>> +        stdin, stdout, stderr = self.ssh_client.exec_command(command)
>>> +        stdout_lines = [line.strip('\n') for line in stdout]
>>> +        for line in stdout_lines:
>>> +            self.ssh_logger.info(line)
>>> +        stderr_lines = [line.strip('\n') for line in stderr]
>>> +        for line in stderr_lines:
>>> +            self.ssh_logger.warning(line)
>>> +        return stdout_lines, stderr_lines
>>> +
>>> +    def boot_debian_wheezy_image_and_ssh_login(self, endianess,
> kernel_path):
>>> +        image_url = self.IMAGE_INFO[endianess]['image_url']
>>> +        image_hash = self.IMAGE_INFO[endianess]['image_hash']
>>> +        image_path = self.fetch_asset(image_url, asset_hash=image_hash)
>>> +        rsa_hostkey_b64 = self.IMAGE_INFO[endianess]['rsa_hostkey']
>>> +
>>> +        self.vm.set_machine('malta')
>>> +        self.vm.set_console()
>>> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
>>> +                               + 'console=ttyS0 root=/dev/sda1')
>>> +        self.vm.add_args('-no-reboot',
>>> +                         '-kernel', kernel_path,
>>> +                         '-append', kernel_command_line,
>>> +                         '-hda', image_path,

We should probably use 'snapshot=on' here.

>>> +                         '-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0
> -:22',
>>> +                         '-device', 'pcnet,netdev=vnet')
>>> +        self.vm.launch()
>>> +
>>> +        self.log.info('VM launched, waiting for sshd')
>>> +        console_pattern = 'Starting OpenBSD Secure Shell server: sshd'
>>> +        self.wait_for_console_pattern(console_pattern)
>>> +        self.log.info('sshd ready')
>>> +
>>> +        self.ssh_connect('root', 'root',
> rsa_hostkey_b64=rsa_hostkey_b64)
>>> +
>>> +    def shutdown_via_ssh(self):
>>> +        self.ssh_command('poweroff')
>>> +        self.ssh_disconnect_vm()
>>> +        self.wait_for_console_pattern('Power down')
>>> +
>>> +    def run_common_commands(self):
>>> +        stdout, stderr = self.ssh_command('lspci -d 11ab:4620')
>>> +        self.assertIn(True, ["GT-64120" in line for line in stdout])
>>> +
>>> +        stdout, stderr = self.ssh_command('cat
> /sys/bus/i2c/devices/i2c-0/name')
>>> +        self.assertIn(True, ["SMBus PIIX4 adapter" in line
>>> +                             for line in stdout])
>>> +
>>> +        stdout, stderr = self.ssh_command('cat /proc/mtd')
>>> +        self.assertIn(True, ["YAMON" in line
>>> +                             for line in stdout])
>>> +
>>> +        # Empty 'Board Config'
>>> +        stdout, stderr = self.ssh_command('md5sum /dev/mtd2ro')
>>> +        self.assertIn(True, ["0dfbe8aa4c20b52e1b8bf3cb6cbdf193" in line
>>> +                             for line in stdout])
>>> +
>>> +    def do_test_mips_malta(self, endianess, kernel_path, uname_m):
>>> +        self.boot_debian_wheezy_image_and_ssh_login(endianess,
> kernel_path)
>>> +
>>> +        stdout, stderr = self.ssh_command('uname -a')
>>> +        self.assertIn(True, [uname_m + " GNU/Linux" in line for line
> in stdout])
>>> +
>>> +        self.run_common_commands()
>>> +        self.shutdown_via_ssh()
>>> +
>>> +    def test_mips_malta32eb_kernel3_2_0(self):
>>> +        """
>>> +        :avocado: tags=slow
>>> +        :avocado: tags=arch:mips
>>> +        :avocado: tags=machine:malta
>>> +        :avocado: tags=endian:big
>>> +        :avocado: tags=device:pcnet32
>>> +        """
>>> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)
>>> +
>>> +        self.do_test_mips_malta('be', kernel_path, 'mips')
>>> +
>>> +    def test_mips_malta32el_kernel3_2_0(self):
>>> +        """
>>> +        :avocado: tags=slow
>>> +        :avocado: tags=arch:mipsel
>>> +        :avocado: tags=machine:malta
>>> +        :avocado: tags=endian:little
>>> +        :avocado: tags=device:pcnet32
>>> +        """
>>> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>> +        kernel_hash = 'a66bea5a8adaa2cb3d36a1d4e0ccdb01be8f6c2a'
>>> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)
>>> +
>>> +        self.do_test_mips_malta('le', kernel_path, 'mips')
>>> +
>>> +    def test_mips_malta64eb_kernel3_2_0(self):
>>> +        """
>>> +        :avocado: tags=slow
>>> +        :avocado: tags=arch:mips64
>>> +        :avocado: tags=machine:malta
>>> +        :avocado: tags=endian:big
>>> +        :avocado: tags=device:pcnet32
>>> +        """
>>> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
>>> +                      'vmlinux-3.2.0-4-5kc-malta')
>>> +        kernel_hash = 'db6eea7de35d36c77d8c165b6bcb222e16eb91db'
>>> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)
>>> +        self.do_test_mips_malta('be', kernel_path, 'mips64')
>>> +
>>> +    def test_mips_malta64el_kernel3_2_0(self):
>>> +        """
>>> +        :avocado: tags=slow
>>> +        :avocado: tags=arch:mips64el
>>> +        :avocado: tags=machine:malta
>>> +        :avocado: tags=endian:little
>>> +        :avocado: tags=device:pcnet32
>>> +        """
>>> +        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
>>> +                      'vmlinux-3.2.0-4-5kc-malta')
>>> +        kernel_hash = '6a7f77245acf231415a0e8b725d91ed2f3487794'
>>> +        kernel_path = self.fetch_asset(kernel_url,
> asset_hash=kernel_hash)
>>> +        self.do_test_mips_malta('le', kernel_path, 'mips64')
>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>> index 002ded6a22..3ae0e29ad7 100644
>>> --- a/tests/requirements.txt
>>> +++ b/tests/requirements.txt
>>> @@ -2,3 +2,4 @@
>>>  # in the tests/venv Python virtual environment. For more info,
>>>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>  avocado-framework==68.0
>>> +paramiko
>>> --
>>> 2.19.1
>>>

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Aleksandar Markovic]

> >>> This test can be run using:
> >>>
> >>>   $ avocado --show=ssh run -t arch:mips
> > tests/acceptance/linux_ssh_mips_malta.py
> >>>   ssh: Entering interactive session.
> >>>   ssh: # uname -a
> >>>   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
> > GNU/Linux
> >>>   ssh: # lspci -d 11ab:4620
> >>>   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
> > GT-64120/64120A/64121A System Controller (rev 10)
> >>>   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
> >>>   ssh: SMBus PIIX4 adapter at 1100
> >>>   ssh: # cat /proc/mtd
> >>>   ssh: dev:    size   erasesize  name
> >>>   ssh: mtd0: 00100000 00010000 "YAMON"
> >>>   ssh: mtd1: 002e0000 00010000 "User FS"
> >>>   ssh: mtd2: 00020000 00010000 "Board Config"
> >>>   ssh: # md5sum /dev/mtd2ro
> >>>   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
> >>>   ssh: # poweroff
>
> Aleksandar, do you think of other commands we should run? What commands
> do you usually try?
>

Philippe, I will leave to you the choice of commands. You know that I
have a very high opinion on this test that you designed and also
implemented, I am more than happy even without any additions. But,
by no means, add something whenever you think it is interesting
and useful.

However, I want to bring to your attention something that could also
be of tremendous importance: Can we try communicating with the
newly booted machine via gdb? And do some simple commands,
like load a binary, set a breakpoint, display register content, etc?

Merci mille fois.

Aleksandar

> I plan to later install more packages in the VM (in particular test
> packages, FPU extensive, device tests and so on).
>
> My rule about device testing is "if the device can be tested in a KVM
> guest, don't test it on TCG guest" for obvious reason of speed.

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Philippe Mathieu-Daudé]

On 6/3/19 3:41 PM, Aleksandar Markovic wrote:
>>>>> This test can be run using:
>>>>>
>>>>>   $ avocado --show=ssh run -t arch:mips
>>> tests/acceptance/linux_ssh_mips_malta.py
>>>>>   ssh: Entering interactive session.
>>>>>   ssh: # uname -a
>>>>>   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
>>> GNU/Linux
>>>>>   ssh: # lspci -d 11ab:4620
>>>>>   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
>>> GT-64120/64120A/64121A System Controller (rev 10)
>>>>>   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
>>>>>   ssh: SMBus PIIX4 adapter at 1100
>>>>>   ssh: # cat /proc/mtd
>>>>>   ssh: dev:    size   erasesize  name
>>>>>   ssh: mtd0: 00100000 00010000 "YAMON"
>>>>>   ssh: mtd1: 002e0000 00010000 "User FS"
>>>>>   ssh: mtd2: 00020000 00010000 "Board Config"
>>>>>   ssh: # md5sum /dev/mtd2ro
>>>>>   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
>>>>>   ssh: # poweroff
>>
>> Aleksandar, do you think of other commands we should run? What commands
>> do you usually try?
>>
> 
> Philippe, I will leave to you the choice of commands. You know that I
> have a very high opinion on this test that you designed and also
> implemented, I am more than happy even without any additions. But,
> by no means, add something whenever you think it is interesting
> and useful.
> 
> However, I want to bring to your attention something that could also
> be of tremendous importance: Can we try communicating with the
> newly booted machine via gdb? And do some simple commands,
> like load a binary, set a breakpoint, display register content, etc?

Yes! This is my next step, I'm working on some draft series. It might
takes a while. My idea is to have some prototype working, then send as
RFC and ask for help. I'll Cc you then.

Regards,

Phil.

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Eduardo Habkost]

On Tue, May 21, 2019 at 11:26:35AM +0200, Aleksandar Markovic wrote:
> On May 21, 2019 10:18 AM, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> wrote:
> >
> >
> > On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
> > >
> > > This tests boot a full VM and check the serial console until
> > > the SSH daemon is running, then start a SSH session and run
> > > some commands.
> > >
> 
> If there is no objection from Cleber or others, I would like to apply this
> patch to my next MIPS pull request. We in Wave just started regular
> regression tests of various nature for QEMU 4.1 for MIPS, and would really
> like to have this test integrated as soon as possible, that is why I am
> rushing a bit. I am talking here about this patch only, since it is a sort
> of an independant unit - I will not do anything with other remaining
> patches from this series.
> 
> Please let me know if you have any concerns.

No problem to me.  I don't expect any merge conflicts caused by
the same patch being merged through both trees.

Acked-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH 4/4] BootLinuxSshTest: Test some userspace commands on Malta

[Eduardo Habkost]

On Tue, May 21, 2019 at 10:18:10AM +0200, Aleksandar Markovic wrote:
> On May 21, 2019 1:19 AM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
> >
> > This tests boot a full VM and check the serial console until
> > the SSH daemon is running, then start a SSH session and run
> > some commands.
> >
> > This test can be run using:
> >
> >   $ avocado --show=ssh run -t arch:mips
> tests/acceptance/linux_ssh_mips_malta.py
> >   ssh: Entering interactive session.
> >   ssh: # uname -a
> >   ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips
> GNU/Linux
> >   ssh: # lspci -d 11ab:4620
> >   ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd.
> GT-64120/64120A/64121A System Controller (rev 10)
> >   ssh: # cat /sys/bus/i2c/devices/i2c-0/name
> >   ssh: SMBus PIIX4 adapter at 1100
> >   ssh: # cat /proc/mtd
> >   ssh: dev:    size   erasesize  name
> >   ssh: mtd0: 00100000 00010000 "YAMON"
> >   ssh: mtd1: 002e0000 00010000 "User FS"
> >   ssh: mtd2: 00020000 00010000 "Board Config"
> >   ssh: # md5sum /dev/mtd2ro
> >   ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193  /dev/mtd2ro
> >   ssh: # poweroff
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> 
> Excelent! Frankly, this was something that we in MIPS needed and missed
> very much for a long time.
> 
> I liked the idea that this test does not run as a default, giving us
> opportunity to extend and adjust it in future as we deem appropriate for
> our MIPS needs, but without affecting people running default test
> execution. In other words, this patch achieves “best of both worlds”.
> 

Actually, the test is being run by default, and is timing out on
travis-ci:

https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3297

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> It was a rainy week-end here, so I invested it to automatize some
> of my MIPS tests.
> 
> The BootLinuxSshTest is not Global warming friendly, it is not
> meant to run on a CI system but rather on a workstation previous
> to post a pull request.
> It can surely be improved, but it is a good starting point.

Until we actually have a mechanism to exclude the test case on
travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
please don't merge patch 4/4 yet or it will break travis-ci.

Cleber, Wainer, is it already possible to make "avocado run" skip
tests tagged with "slow"?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa]

----- Original Message -----
> From: "Eduardo Habkost" <ehabkost@redhat.com>
> To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Aleksandar Markovic"
> <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>
> Sent: Wednesday, May 22, 2019 5:12:30 PM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > Hi,
> > 
> > It was a rainy week-end here, so I invested it to automatize some
> > of my MIPS tests.
> > 
> > The BootLinuxSshTest is not Global warming friendly, it is not
> > meant to run on a CI system but rather on a workstation previous
> > to post a pull request.
> > It can surely be improved, but it is a good starting point.
> 
> Until we actually have a mechanism to exclude the test case on
> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> please don't merge patch 4/4 yet or it will break travis-ci.
> 
> Cleber, Wainer, is it already possible to make "avocado run" skip
> tests tagged with "slow"?
> 

The mechanism exists, but we haven't tagged any test so far as slow.

Should we define/document a criteria for a test to be slow?  Given
that this is highly subjective, we have to think of:

 * Will we consider the average or maximum run time (the timeout
   definition)?
 
 * For a single test, what is "slow"? Some rough numbers from Travis
   CI[1] to help us with guidelines:
   - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
   - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
   - linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:  PASS (18.14 s)
   - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)

 * Do we want to set a maximum job timeout?  This way we can skip
   tests after a given amount of time has passed.  Currently we interrupt
   the test running when the job timeout is reached, but it's possible
   to add a option so that no new tests will be started, but currently
   running ones will be waited on.

Regards,
- Cleber. 

[1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518

> --
> Eduardo
> 

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>
>
>
> ----- Original Message -----
> > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
"Aleksandar Markovic"
> > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <
wainersm@redhat.com>
> > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > It was a rainy week-end here, so I invested it to automatize some
> > > of my MIPS tests.
> > >
> > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > meant to run on a CI system but rather on a workstation previous
> > > to post a pull request.
> > > It can surely be improved, but it is a good starting point.
> >
> > Until we actually have a mechanism to exclude the test case on
> > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > please don't merge patch 4/4 yet or it will break travis-ci.
> >
> > Cleber, Wainer, is it already possible to make "avocado run" skip
> > tests tagged with "slow"?
> >
>
> The mechanism exists, but we haven't tagged any test so far as slow.
>
> Should we define/document a criteria for a test to be slow?  Given
> that this is highly subjective, we have to think of:
>
>  * Will we consider the average or maximum run time (the timeout
>    definition)?
>
>  * For a single test, what is "slow"? Some rough numbers from Travis
>    CI[1] to help us with guidelines:
>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
>    -
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
PASS (18.14 s)
>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
>
>  * Do we want to set a maximum job timeout?  This way we can skip
>    tests after a given amount of time has passed.  Currently we interrupt
>    the test running when the job timeout is reached, but it's possible
>    to add a option so that no new tests will be started, but currently
>    running ones will be waited on.
>

As far as integrating the patch into my queue, I did it just an hour or so
prior the objections of others, but will inform Peter to put the pull
request on hold, so it will not go to the main tree.

We in Wave Computing (MIPS) are very happy with this test, even in its
current state, and understand it as an initial version that will be subject
to improvement and expansion. We consider this test seriously and think it
is vital for QEMU for MIPS.

We would like to put it in the “slow” group for the simple reason that, we
gather, this would give us more freedom in future versions.

Yours,
Aleksandar

> Regards,
> - Cleber.
>
> [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
>
> > --
> > Eduardo
> >

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>
>
>
> ----- Original Message -----
> > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
"Aleksandar Markovic"
> > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <
wainersm@redhat.com>
> > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > It was a rainy week-end here, so I invested it to automatize some
> > > of my MIPS tests.
> > >
> > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > meant to run on a CI system but rather on a workstation previous
> > > to post a pull request.
> > > It can surely be improved, but it is a good starting point.
> >
> > Until we actually have a mechanism to exclude the test case on
> > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > please don't merge patch 4/4 yet or it will break travis-ci.
> >
> > Cleber, Wainer, is it already possible to make "avocado run" skip
> > tests tagged with "slow"?
> >
>
> The mechanism exists, but we haven't tagged any test so far as slow.
>

Cleber,

For the test from patch 4/4, there is no dilemma - it should be in the
“slow” group, as Philippe envisioned and said, so that it is not humpered
with stricter requirements for “fast” (default) group. Could you explain us
how to do it, so that we can hopefully finally proceed?

Gratefully,
Aleksandar

> Should we define/document a criteria for a test to be slow?  Given
> that this is highly subjective, we have to think of:
>
>  * Will we consider the average or maximum run time (the timeout
>    definition)?
>
>  * For a single test, what is "slow"? Some rough numbers from Travis
>    CI[1] to help us with guidelines:
>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
>    -
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
PASS (18.14 s)
>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
>
>  * Do we want to set a maximum job timeout?  This way we can skip
>    tests after a given amount of time has passed.  Currently we interrupt
>    the test running when the job timeout is reached, but it's possible
>    to add a option so that no new tests will be started, but currently
>    running ones will be waited on.
>
> Regards,
> - Cleber.
>
> [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
>
> > --
> > Eduardo
> >

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa]

----- Original Message -----
> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>,
> qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Eduardo Habkost" <ehabkost@redhat.com>,
> "Aurelien Jarno" <aurelien@aurel32.net>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> Sent: Wednesday, May 22, 2019 6:43:54 PM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >
> >
> >
> > ----- Original Message -----
> > > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> "Aleksandar Markovic"
> > > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
> amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <
> wainersm@redhat.com>
> > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > >
> > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > > Hi,
> > > >
> > > > It was a rainy week-end here, so I invested it to automatize some
> > > > of my MIPS tests.
> > > >
> > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > meant to run on a CI system but rather on a workstation previous
> > > > to post a pull request.
> > > > It can surely be improved, but it is a good starting point.
> > >
> > > Until we actually have a mechanism to exclude the test case on
> > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > please don't merge patch 4/4 yet or it will break travis-ci.
> > >
> > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > tests tagged with "slow"?
> > >
> >
> > The mechanism exists, but we haven't tagged any test so far as slow.
> >
> 
> Cleber,
> 
> For the test from patch 4/4, there is no dilemma - it should be in the
> “slow” group, as Philippe envisioned and said, so that it is not humpered
> with stricter requirements for “fast” (default) group. Could you explain us
> how to do it, so that we can hopefully finally proceed?
> 

Hi Aleksandar,

The point is that there's no "group" definition at this point.  This is the
core of the discussion.

I think we're close to converging to something simple and effective.  Please
let us know what you think of the proposals given.

Thanks!
- Cleber.

> Gratefully,
> Aleksandar
> 
> > Should we define/document a criteria for a test to be slow?  Given
> > that this is highly subjective, we have to think of:
> >
> >  * Will we consider the average or maximum run time (the timeout
> >    definition)?
> >
> >  * For a single test, what is "slow"? Some rough numbers from Travis
> >    CI[1] to help us with guidelines:
> >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> >    -
> linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> PASS (18.14 s)
> >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> >
> >  * Do we want to set a maximum job timeout?  This way we can skip
> >    tests after a given amount of time has passed.  Currently we interrupt
> >    the test running when the job timeout is reached, but it's possible
> >    to add a option so that no new tests will be started, but currently
> >    running ones will be waited on.
> >
> > Regards,
> > - Cleber.
> >
> > [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
> >
> > > --
> > > Eduardo
> > >
> 

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>
>
>
> ----- Original Message -----
> > From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> > To: "Cleber Rosa" <crosa@redhat.com>
> > Cc: "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Aleksandar
Markovic" <amarkovic@wavecomp.com>,
> > qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
"Eduardo Habkost" <ehabkost@redhat.com>,
> > "Aurelien Jarno" <aurelien@aurel32.net>, "Philippe Mathieu-Daudé" <
f4bug@amsat.org>
> > Sent: Wednesday, May 22, 2019 6:43:54 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com
>,
> > "Aleksandar Markovic"
> > > > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
> > amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos
Moschetta" <
> > wainersm@redhat.com>
> > > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > > >
> > > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé
wrote:
> > > > > Hi,
> > > > >
> > > > > It was a rainy week-end here, so I invested it to automatize some
> > > > > of my MIPS tests.
> > > > >
> > > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > > meant to run on a CI system but rather on a workstation previous
> > > > > to post a pull request.
> > > > > It can surely be improved, but it is a good starting point.
> > > >
> > > > Until we actually have a mechanism to exclude the test case on
> > > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > > please don't merge patch 4/4 yet or it will break travis-ci.
> > > >
> > > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > > tests tagged with "slow"?
> > > >
> > >
> > > The mechanism exists, but we haven't tagged any test so far as slow.
> > >
> >
> > Cleber,
> >
> > For the test from patch 4/4, there is no dilemma - it should be in the
> > “slow” group, as Philippe envisioned and said, so that it is not
humpered
> > with stricter requirements for “fast” (default) group. Could you
explain us
> > how to do it, so that we can hopefully finally proceed?
> >
>
> Hi Aleksandar,
>
> The point is that there's no "group" definition at this point.  This is
the
> core of the discussion.
>
> I think we're close to converging to something simple and effective.
Please
> let us know what you think of the proposals given.
>
> Thanks!
> - Cleber.
>

Cleber, hi.

Thanks for responding.

My views are very similar to Philippe's, but I will provide you with more
details of our (mips) perspective.

As far as black/whitelist issues that is a moot point for us - we only want
to be able to have a way to tag a test within the test itself (so, without
updating some common files, external lists,etc.)

In general, we would like to have a test environment where we would be able
to test what WE deem suitable for us, without feeling that we bother you or
anybody else, or that we are bothered by others.

Let me give you a little extreme example: Let's say we want a complex test
that downloads components from let's say fifty internet location, executes
zillion test cases, and last two days. I wouldn't like anybody to ask me
“Why would you that?” or tell me “You can't do this.” or say “No, we did
not anticipate such tests, patch rejected.” I (we, people from mips) should
be able to define what I (we) need.

Having such test would be a big deal for me, not only that I could run it
manually or automatically every weekend, but I could ask submitters of
critical changes: “Did you run this test that we have in Avocado dir?”,
without specifying test details, procedures, etc. All this is a BIG deal
for me.

On the other hand, I agree that certain group of tests (envisioned for
daily or so Travis CI) should have some stricter limitations and structure.
But right now I feel humpered by it, and this is counterproductive.

So, we want freedom, responsibility and ownersheep of our tests. Please
give us the opportunity to get down on business and start writing tests and
start testing.

Yours,
Aleksandar

> > Gratefully,
> > Aleksandar
> >
> > > Should we define/document a criteria for a test to be slow?  Given
> > > that this is highly subjective, we have to think of:
> > >
> > >  * Will we consider the average or maximum run time (the timeout
> > >    definition)?
> > >
> > >  * For a single test, what is "slow"? Some rough numbers from Travis
> > >    CI[1] to help us with guidelines:
> > >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS
(6.04 s)
> > >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS
(2.91 s)
> > >    -
> >
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > PASS (18.14 s)
> > >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > >
> > >  * Do we want to set a maximum job timeout?  This way we can skip
> > >    tests after a given amount of time has passed.  Currently we
interrupt
> > >    the test running when the job timeout is reached, but it's possible
> > >    to add a option so that no new tests will be started, but currently
> > >    running ones will be waited on.
> > >
> > > Regards,
> > > - Cleber.
> > >
> > > [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
> > >
> > > > --
> > > > Eduardo
> > > >
> >

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Philippe Mathieu-Daudé]

On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>>> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
>>> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>>>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
>>>>>
>>>>> Until we actually have a mechanism to exclude the test case on
>>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
>>>>> please don't merge patch 4/4 yet or it will break travis-ci.
>>>>>
>>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
>>>>> tests tagged with "slow"?
>>>>>
>>>>
>>>> The mechanism exists, but we haven't tagged any test so far as slow.
>>>>
>>>
>>> Cleber,
>>>
>>> For the test from patch 4/4, there is no dilemma - it should be in the
>>> “slow” group, as Philippe envisioned and said, so that it is not
> humpered
>>> with stricter requirements for “fast” (default) group. Could you
> explain us
>>> how to do it, so that we can hopefully finally proceed?
>>>
>>
>> Hi Aleksandar,
>>
>> The point is that there's no "group" definition at this point.  This is
> the
>> core of the discussion.
>>
>> I think we're close to converging to something simple and effective.
> Please
>> let us know what you think of the proposals given.
>>
>> Thanks!
>> - Cleber.
>>
> 
> Cleber, hi.
> 
> Thanks for responding.
> 
> My views are very similar to Philippe's, but I will provide you with more
> details of our (mips) perspective.
> 
> As far as black/whitelist issues that is a moot point for us - we only want
> to be able to have a way to tag a test within the test itself (so, without
> updating some common files, external lists,etc.)
> 
> In general, we would like to have a test environment where we would be able
> to test what WE deem suitable for us, without feeling that we bother you or
> anybody else, or that we are bothered by others.
> 
> Let me give you a little extreme example: Let's say we want a complex test
> that downloads components from let's say fifty internet location, executes
> zillion test cases, and last two days. I wouldn't like anybody to ask me
> “Why would you that?” or tell me “You can't do this.” or say “No, we did
> not anticipate such tests, patch rejected.” I (we, people from mips) should
> be able to define what I (we) need.

Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
maintains tests/tcg/mips/). We should try to keep contribution upstream,
so good idea/pattern can be reused by others.

What I'd like to have with those tests is, at least:

1/ we don't need to run all the tests (but there is a set of 'quick'
tests we can run on daily basis)

2/ maintainers can run their default tests easily (using a combination
of Avocado tags)

3/ if a developer working on the PCI subsystem has to modify the MIPS
subsystem (for example), he should be able to run the MIPS tests before
sending his series.

> Having such test would be a big deal for me, not only that I could run it
> manually or automatically every weekend, but I could ask submitters of
> critical changes: “Did you run this test that we have in Avocado dir?”,
> without specifying test details, procedures, etc. All this is a BIG deal
> for me.
> 
> On the other hand, I agree that certain group of tests (envisioned for
> daily or so Travis CI) should have some stricter limitations and structure.
> But right now I feel humpered by it, and this is counterproductive.
> 
> So, we want freedom, responsibility and ownersheep of our tests. Please
> give us the opportunity to get down on business and start writing tests and
> start testing.
> 
> Yours,
> Aleksandar

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 23, 2019 7:27 PM, "Philippe Mathieu-Daudé" <f4bug@amsat.org> wrote:
>
> On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> > On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >>> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> >>> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >>>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> >>>>>
> >>>>> Until we actually have a mechanism to exclude the test case on
> >>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> >>>>> please don't merge patch 4/4 yet or it will break travis-ci.
> >>>>>
> >>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
> >>>>> tests tagged with "slow"?
> >>>>>
> >>>>
> >>>> The mechanism exists, but we haven't tagged any test so far as slow.
> >>>>
> >>>
> >>> Cleber,
> >>>
> >>> For the test from patch 4/4, there is no dilemma - it should be in the
> >>> “slow” group, as Philippe envisioned and said, so that it is not
> > humpered
> >>> with stricter requirements for “fast” (default) group. Could you
> > explain us
> >>> how to do it, so that we can hopefully finally proceed?
> >>>
> >>
> >> Hi Aleksandar,
> >>
> >> The point is that there's no "group" definition at this point.  This is
> > the
> >> core of the discussion.
> >>
> >> I think we're close to converging to something simple and effective.
> > Please
> >> let us know what you think of the proposals given.
> >>
> >> Thanks!
> >> - Cleber.
> >>
> >
> > Cleber, hi.
> >
> > Thanks for responding.
> >
> > My views are very similar to Philippe's, but I will provide you with
more
> > details of our (mips) perspective.
> >
> > As far as black/whitelist issues that is a moot point for us - we only
want
> > to be able to have a way to tag a test within the test itself (so,
without
> > updating some common files, external lists,etc.)
> >
> > In general, we would like to have a test environment where we would be
able
> > to test what WE deem suitable for us, without feeling that we bother
you or
> > anybody else, or that we are bothered by others.
> >
> > Let me give you a little extreme example: Let's say we want a complex
test
> > that downloads components from let's say fifty internet location,
executes
> > zillion test cases, and last two days. I wouldn't like anybody to ask me
> > “Why would you that?” or tell me “You can't do this.” or say “No, we did
> > not anticipate such tests, patch rejected.” I (we, people from mips)
should
> > be able to define what I (we) need.
>
> Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
> maintains tests/tcg/mips/). We should try to keep contribution upstream,
> so good idea/pattern can be reused by others.
>
> What I'd like to have with those tests is, at least:
>
> 1/ we don't need to run all the tests (but there is a set of 'quick'
> tests we can run on daily basis)
>
> 2/ maintainers can run their default tests easily (using a combination
> of Avocado tags)
>
> 3/ if a developer working on the PCI subsystem has to modify the MIPS
> subsystem (for example), he should be able to run the MIPS tests before
> sending his series.
>

Exactly! Excellent ideas and examples!

> > Having such test would be a big deal for me, not only that I could run
it
> > manually or automatically every weekend, but I could ask submitters of
> > critical changes: “Did you run this test that we have in Avocado dir?”,
> > without specifying test details, procedures, etc. All this is a BIG deal
> > for me.
> >
> > On the other hand, I agree that certain group of tests (envisioned for
> > daily or so Travis CI) should have some stricter limitations and
structure.
> > But right now I feel humpered by it, and this is counterproductive.
> >
> > So, we want freedom, responsibility and ownersheep of our tests. Please
> > give us the opportunity to get down on business and start writing tests
and
> > start testing.
> >
> > Yours,
> > Aleksandar

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Thu, May 23, 2019 at 07:27:35PM +0200, Philippe Mathieu-Daudé wrote:
> On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> > On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >>> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> >>> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >>>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> >>>>>
> >>>>> Until we actually have a mechanism to exclude the test case on
> >>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> >>>>> please don't merge patch 4/4 yet or it will break travis-ci.
> >>>>>
> >>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
> >>>>> tests tagged with "slow"?
> >>>>>
> >>>>
> >>>> The mechanism exists, but we haven't tagged any test so far as slow.
> >>>>
> >>>
> >>> Cleber,
> >>>
> >>> For the test from patch 4/4, there is no dilemma - it should be in the
> >>> “slow” group, as Philippe envisioned and said, so that it is not
> > humpered
> >>> with stricter requirements for “fast” (default) group. Could you
> > explain us
> >>> how to do it, so that we can hopefully finally proceed?
> >>>
> >>
> >> Hi Aleksandar,
> >>
> >> The point is that there's no "group" definition at this point.  This is
> > the
> >> core of the discussion.
> >>
> >> I think we're close to converging to something simple and effective.
> > Please
> >> let us know what you think of the proposals given.
> >>
> >> Thanks!
> >> - Cleber.
> >>
> > 
> > Cleber, hi.
> > 
> > Thanks for responding.
> > 
> > My views are very similar to Philippe's, but I will provide you with more
> > details of our (mips) perspective.
> > 
> > As far as black/whitelist issues that is a moot point for us - we only want
> > to be able to have a way to tag a test within the test itself (so, without
> > updating some common files, external lists,etc.)
> > 
> > In general, we would like to have a test environment where we would be able
> > to test what WE deem suitable for us, without feeling that we bother you or
> > anybody else, or that we are bothered by others.
> > 
> > Let me give you a little extreme example: Let's say we want a complex test
> > that downloads components from let's say fifty internet location, executes
> > zillion test cases, and last two days. I wouldn't like anybody to ask me
> > “Why would you that?” or tell me “You can't do this.” or say “No, we did
> > not anticipate such tests, patch rejected.” I (we, people from mips) should
> > be able to define what I (we) need.
> 
> Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
> maintains tests/tcg/mips/). We should try to keep contribution upstream,
> so good idea/pattern can be reused by others.
> 
> What I'd like to have with those tests is, at least:
> 
> 1/ we don't need to run all the tests (but there is a set of 'quick'
> tests we can run on daily basis)
> 
> 2/ maintainers can run their default tests easily (using a combination
> of Avocado tags)
> 
> 3/ if a developer working on the PCI subsystem has to modify the MIPS
> subsystem (for example), he should be able to run the MIPS tests before
> sending his series.

Keeping the test cases organized in subdirectories are a good
idea, but don't think this is going to help us when we need to
quickly enable/disable specific test cases on some CI systems.

Disabling a test case (or an entire category of test cases) known
to be failing on some CI systems should require a one line patch,
not moving files to a separate directory.

> 
> > Having such test would be a big deal for me, not only that I could run it
> > manually or automatically every weekend, but I could ask submitters of
> > critical changes: “Did you run this test that we have in Avocado dir?”,
> > without specifying test details, procedures, etc. All this is a BIG deal
> > for me.
> > 
> > On the other hand, I agree that certain group of tests (envisioned for
> > daily or so Travis CI) should have some stricter limitations and structure.
> > But right now I feel humpered by it, and this is counterproductive.
> > 
> > So, we want freedom, responsibility and ownersheep of our tests. Please
> > give us the opportunity to get down on business and start writing tests and
> > start testing.
> > 
> > Yours,
> > Aleksandar

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 24, 2019 9:40 PM, "Eduardo Habkost" <ehabkost@redhat.com> wrote:
>
> On Thu, May 23, 2019 at 07:27:35PM +0200, Philippe Mathieu-Daudé wrote:
> > On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> > > On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> > >>> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> > >>> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> > >>>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > >>>>>
> > >>>>> Until we actually have a mechanism to exclude the test case on
> > >>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > >>>>> please don't merge patch 4/4 yet or it will break travis-ci.
> > >>>>>
> > >>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
> > >>>>> tests tagged with "slow"?
> > >>>>>
> > >>>>
> > >>>> The mechanism exists, but we haven't tagged any test so far as
slow.
> > >>>>
> > >>>
> > >>> Cleber,
> > >>>
> > >>> For the test from patch 4/4, there is no dilemma - it should be in
the
> > >>> “slow” group, as Philippe envisioned and said, so that it is not
> > > humpered
> > >>> with stricter requirements for “fast” (default) group. Could you
> > > explain us
> > >>> how to do it, so that we can hopefully finally proceed?
> > >>>
> > >>
> > >> Hi Aleksandar,
> > >>
> > >> The point is that there's no "group" definition at this point.  This
is
> > > the
> > >> core of the discussion.
> > >>
> > >> I think we're close to converging to something simple and effective.
> > > Please
> > >> let us know what you think of the proposals given.
> > >>
> > >> Thanks!
> > >> - Cleber.
> > >>
> > >
> > > Cleber, hi.
> > >
> > > Thanks for responding.
> > >
> > > My views are very similar to Philippe's, but I will provide you with
more
> > > details of our (mips) perspective.
> > >
> > > As far as black/whitelist issues that is a moot point for us - we
only want
> > > to be able to have a way to tag a test within the test itself (so,
without
> > > updating some common files, external lists,etc.)
> > >
> > > In general, we would like to have a test environment where we would
be able
> > > to test what WE deem suitable for us, without feeling that we bother
you or
> > > anybody else, or that we are bothered by others.
> > >
> > > Let me give you a little extreme example: Let's say we want a complex
test
> > > that downloads components from let's say fifty internet location,
executes
> > > zillion test cases, and last two days. I wouldn't like anybody to ask
me
> > > “Why would you that?” or tell me “You can't do this.” or say “No, we
did
> > > not anticipate such tests, patch rejected.” I (we, people from mips)
should
> > > be able to define what I (we) need.
> >
> > Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
> > maintains tests/tcg/mips/). We should try to keep contribution upstream,
> > so good idea/pattern can be reused by others.
> >
> > What I'd like to have with those tests is, at least:
> >
> > 1/ we don't need to run all the tests (but there is a set of 'quick'
> > tests we can run on daily basis)
> >
> > 2/ maintainers can run their default tests easily (using a combination
> > of Avocado tags)
> >
> > 3/ if a developer working on the PCI subsystem has to modify the MIPS
> > subsystem (for example), he should be able to run the MIPS tests before
> > sending his series.
>
> Keeping the test cases organized in subdirectories are a good
> idea, but don't think this is going to help us when we need to
> quickly enable/disable specific test cases on some CI systems.
>

Well, Eduardo, nobody said that directory locations should be used for
enabling/disabling or tagging/untagging tests in the first place. I think
it was clear for everybody from the outset that these features should have
their own mechanisms, which Cleber says already exist, but can't be used
because the test group still can't figure out (in some hamletesque way)
whether to blacklist or to whitelist, or how to name the tag for travis,
and tag for not travis, and if such tags should even exist, etc. - that is
my layman impression from recent discussions. And now when Philippe
suggested (in my opinion logical and reasonable) subdirectory, an endless
discussion begins: “To subdirectory or not to subdirectory? That is the
question.” Meanwhile, 4.1 is inexorably getting closer and closer, and with
each day, the value of any potential tests is decreasing.

Directory structure should be used in its usual and basic way: for
clustering files of similar nature, purpose, or origin, and I do certainly
support any reasonable subdirectory organization for your directory - and
you should think about it, and probably while doing that consult a little
bit other people from all walks of QEMU. We are ready to comply with your
final decision.

The good thing is that nothing is set in stone, everything can be changed
and improved, moving files is easy in git.

All that said, many thanks for reviewing patch 4/4.

Aleksandar



> Disabling a test case (or an entire category of test cases) known
> to be failing on some CI systems should require a one line patch,
> not moving files to a separate directory.
>
> >
> > > Having such test would be a big deal for me, not only that I could
run it
> > > manually or automatically every weekend, but I could ask submitters of
> > > critical changes: “Did you run this test that we have in Avocado
dir?”,
> > > without specifying test details, procedures, etc. All this is a BIG
deal
> > > for me.
> > >
> > > On the other hand, I agree that certain group of tests (envisioned for
> > > daily or so Travis CI) should have some stricter limitations and
structure.
> > > But right now I feel humpered by it, and this is counterproductive.
> > >
> > > So, we want freedom, responsibility and ownersheep of our tests.
Please
> > > give us the opportunity to get down on business and start writing
tests and
> > > start testing.
> > >
> > > Yours,
> > > Aleksandar
>
> --
> Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Fri, May 24, 2019 at 10:32:36PM +0200, Aleksandar Markovic wrote:
> On May 24, 2019 9:40 PM, "Eduardo Habkost" <ehabkost@redhat.com> wrote:
> >
> > On Thu, May 23, 2019 at 07:27:35PM +0200, Philippe Mathieu-Daudé wrote:
> > > On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> > > > On May 23, 2019 3:45 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> > > >>> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> > > >>> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> > > >>>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > >>>>>
> > > >>>>> Until we actually have a mechanism to exclude the test case on
> > > >>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > >>>>> please don't merge patch 4/4 yet or it will break travis-ci.
> > > >>>>>
> > > >>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
> > > >>>>> tests tagged with "slow"?
> > > >>>>>
> > > >>>>
> > > >>>> The mechanism exists, but we haven't tagged any test so far as
> slow.
> > > >>>>
> > > >>>
> > > >>> Cleber,
> > > >>>
> > > >>> For the test from patch 4/4, there is no dilemma - it should be in
> the
> > > >>> “slow” group, as Philippe envisioned and said, so that it is not
> > > > humpered
> > > >>> with stricter requirements for “fast” (default) group. Could you
> > > > explain us
> > > >>> how to do it, so that we can hopefully finally proceed?
> > > >>>
> > > >>
> > > >> Hi Aleksandar,
> > > >>
> > > >> The point is that there's no "group" definition at this point.  This
> is
> > > > the
> > > >> core of the discussion.
> > > >>
> > > >> I think we're close to converging to something simple and effective.
> > > > Please
> > > >> let us know what you think of the proposals given.
> > > >>
> > > >> Thanks!
> > > >> - Cleber.
> > > >>
> > > >
> > > > Cleber, hi.
> > > >
> > > > Thanks for responding.
> > > >
> > > > My views are very similar to Philippe's, but I will provide you with
> more
> > > > details of our (mips) perspective.
> > > >
> > > > As far as black/whitelist issues that is a moot point for us - we
> only want
> > > > to be able to have a way to tag a test within the test itself (so,
> without
> > > > updating some common files, external lists,etc.)
> > > >
> > > > In general, we would like to have a test environment where we would
> be able
> > > > to test what WE deem suitable for us, without feeling that we bother
> you or
> > > > anybody else, or that we are bothered by others.
> > > >
> > > > Let me give you a little extreme example: Let's say we want a complex
> test
> > > > that downloads components from let's say fifty internet location,
> executes
> > > > zillion test cases, and last two days. I wouldn't like anybody to ask
> me
> > > > “Why would you that?” or tell me “You can't do this.” or say “No, we
> did
> > > > not anticipate such tests, patch rejected.” I (we, people from mips)
> should
> > > > be able to define what I (we) need.
> > >
> > > Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
> > > maintains tests/tcg/mips/). We should try to keep contribution upstream,
> > > so good idea/pattern can be reused by others.
> > >
> > > What I'd like to have with those tests is, at least:
> > >
> > > 1/ we don't need to run all the tests (but there is a set of 'quick'
> > > tests we can run on daily basis)
> > >
> > > 2/ maintainers can run their default tests easily (using a combination
> > > of Avocado tags)
> > >
> > > 3/ if a developer working on the PCI subsystem has to modify the MIPS
> > > subsystem (for example), he should be able to run the MIPS tests before
> > > sending his series.
> >
> > Keeping the test cases organized in subdirectories are a good
> > idea, but don't think this is going to help us when we need to
> > quickly enable/disable specific test cases on some CI systems.
> >
> 
> Well, Eduardo, nobody said that directory locations should be used for
> enabling/disabling or tagging/untagging tests in the first place. I think
> it was clear for everybody from the outset that these features should have
> their own mechanisms, which Cleber says already exist, but can't be used
> because the test group still can't figure out (in some hamletesque way)
> whether to blacklist or to whitelist, or how to name the tag for travis,
> and tag for not travis, and if such tags should even exist, etc. - that is
> my layman impression from recent discussions. And now when Philippe
> suggested (in my opinion logical and reasonable) subdirectory, an endless
> discussion begins: “To subdirectory or not to subdirectory? That is the
> question.” Meanwhile, 4.1 is inexorably getting closer and closer, and with
> each day, the value of any potential tests is decreasing.

I understand that seeing the discussions going on and the patches
taking too long to be included might be frustrating.

These discussions shouldn't get into the way of addressing other
problems.  We don't need to wait until all discussions have
finished before proposing new patches or before merging patches
that are considered good.


> 
> Directory structure should be used in its usual and basic way: for
> clustering files of similar nature, purpose, or origin, and I do certainly
> support any reasonable subdirectory organization for your directory - and
> you should think about it, and probably while doing that consult a little
> bit other people from all walks of QEMU. We are ready to comply with your
> final decision.

About subdirectories, specifically, note that I explicitly said
it was a good idea.  If somebody wants to send patches, they are
welcome.

If I'm doing something else that could be blocking people from
getting work done, I'd like to fix that.  I'm aware that
sometimes I take too long to review patches, but I hope other
developers can help us on the review work.

> 
> The good thing is that nothing is set in stone, everything can be changed
> and improved, moving files is easy in git.
> 
> All that said, many thanks for reviewing patch 4/4.
> 
> Aleksandar
> 
> 
> 
> > Disabling a test case (or an entire category of test cases) known
> > to be failing on some CI systems should require a one line patch,
> > not moving files to a separate directory.
> >
> > >
> > > > Having such test would be a big deal for me, not only that I could
> run it
> > > > manually or automatically every weekend, but I could ask submitters of
> > > > critical changes: “Did you run this test that we have in Avocado
> dir?”,
> > > > without specifying test details, procedures, etc. All this is a BIG
> deal
> > > > for me.
> > > >
> > > > On the other hand, I agree that certain group of tests (envisioned for
> > > > daily or so Travis CI) should have some stricter limitations and
> structure.
> > > > But right now I feel humpered by it, and this is counterproductive.
> > > >
> > > > So, we want freedom, responsibility and ownersheep of our tests.
> Please
> > > > give us the opportunity to get down on business and start writing
> tests and
> > > > start testing.
> > > >
> > > > Yours,
> > > > Aleksandar
> >
> > --
> > Eduardo

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Aleksandar Markovic"
> > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>
> > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > 
> > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > > 
> > > It was a rainy week-end here, so I invested it to automatize some
> > > of my MIPS tests.
> > > 
> > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > meant to run on a CI system but rather on a workstation previous
> > > to post a pull request.
> > > It can surely be improved, but it is a good starting point.
> > 
> > Until we actually have a mechanism to exclude the test case on
> > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > please don't merge patch 4/4 yet or it will break travis-ci.
> > 
> > Cleber, Wainer, is it already possible to make "avocado run" skip
> > tests tagged with "slow"?
> > 
> 
> The mechanism exists, but we haven't tagged any test so far as slow.
> 
> Should we define/document a criteria for a test to be slow?  Given
> that this is highly subjective, we have to think of:
> 
>  * Will we consider the average or maximum run time (the timeout
>    definition)?
>  
>  * For a single test, what is "slow"? Some rough numbers from Travis
>    CI[1] to help us with guidelines:
>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
>    - linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:  PASS (18.14 s)
>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)

I don't think we need to overthink this.  Whatever objective
criteria we choose, I'm sure we'll have to adapt them later due
to real world problems.

e.g.: is 396 seconds too slow?  I don't know, it depends: does it
break Travis and other CI systems often because of timeouts?  If
yes, then we should probably tag it as slow.

If having subjective criteria is really a problem (I don't think
it is), then we can call the tag "skip_travis", and stop worrying
about defining what exactly is "slow".


> 
>  * Do we want to set a maximum job timeout?  This way we can skip
>    tests after a given amount of time has passed.  Currently we interrupt
>    the test running when the job timeout is reached, but it's possible
>    to add a option so that no new tests will be started, but currently
>    running ones will be waited on.

I'm not sure I understand the suggestion to skip tests.  If we
skip tests after a timeout, how would we differentiate a test
being expectedly slow from a QEMU hang?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa]

----- Original Message -----
> From: "Eduardo Habkost" <ehabkost@redhat.com>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Philippe Mathieu-Daudé" <f4bug@amsat.org>, qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien
> Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>
> Sent: Wednesday, May 22, 2019 7:07:05 PM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> > > "Aleksandar Markovic"
> > > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic"
> > > <amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta"
> > > <wainersm@redhat.com>
> > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > > 
> > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > > Hi,
> > > > 
> > > > It was a rainy week-end here, so I invested it to automatize some
> > > > of my MIPS tests.
> > > > 
> > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > meant to run on a CI system but rather on a workstation previous
> > > > to post a pull request.
> > > > It can surely be improved, but it is a good starting point.
> > > 
> > > Until we actually have a mechanism to exclude the test case on
> > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > please don't merge patch 4/4 yet or it will break travis-ci.
> > > 
> > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > tests tagged with "slow"?
> > > 
> > 
> > The mechanism exists, but we haven't tagged any test so far as slow.
> > 
> > Should we define/document a criteria for a test to be slow?  Given
> > that this is highly subjective, we have to think of:
> > 
> >  * Will we consider the average or maximum run time (the timeout
> >    definition)?
> >  
> >  * For a single test, what is "slow"? Some rough numbers from Travis
> >    CI[1] to help us with guidelines:
> >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> >    -
> >    linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> >    PASS (18.14 s)
> >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> 
> I don't think we need to overthink this.  Whatever objective
> criteria we choose, I'm sure we'll have to adapt them later due
> to real world problems.
> 
> e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> break Travis and other CI systems often because of timeouts?  If
> yes, then we should probably tag it as slow.
> 

It's not only that.  We're close to a point where we'll need to
determine whether "make check-acceptance" will work as a generic
enough default for most user on their environments and most CI
systems.

As an example, this job ran 5 fairly slow tests (which I'm preparing
to send):

  https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518

Those are justifiably slow, given the fact that they boot a full
Fedora 30 system using TCG.  The job has a cumulative execution time
of ~39 minutes.  That leaves only 11 minutes to spare on the Travis
CI environment.  If they all exercised close to their 600s allowances
(timeout), the Travis job would have failed. 

Having said that, if a CI failure is supposed to be a major breakage,
which I believe it's the right mind set and a worthy goal, we should
limit the amount of tests we run so that their *maximum* execution
time does not exceed the maximum job time limit.

> If having subjective criteria is really a problem (I don't think
> it is), then we can call the tag "skip_travis", and stop worrying
> about defining what exactly is "slow".
> 
> 
> > 
> >  * Do we want to set a maximum job timeout?  This way we can skip
> >    tests after a given amount of time has passed.  Currently we interrupt
> >    the test running when the job timeout is reached, but it's possible
> >    to add a option so that no new tests will be started, but currently
> >    running ones will be waited on.
> 
> I'm not sure I understand the suggestion to skip tests.  If we
> skip tests after a timeout, how would we differentiate a test
> being expectedly slow from a QEMU hang?
> 
> --
> Eduardo
> 

Basically, what I meant is that we could attempt something like:

 * Job "Brave"
  - 50 tests, each with 60 seconds timeout = 50 min max
  - 60 tests, each with 1 second timeout  = 1 min max

If Job "Brave" is run on a system such as Travis, it *can* fail,
because it can go over the maximum Travis CI job limit of 50 min.
We could set an Avocado job timeout of say, 48 minutes, and tell
Avocado to mark the tests it wasn't able to spawn as "SKIPPED",
and do not report an overall error condition.

But, if we want to be more conservative (which I now realize is
the best mindset for this situation), we should stick to something
like:

 * Job "Coward"
  - 47 tests, each with 60 seconds timeout = 47 min max
  - 60 tests, each with 1 second timeout  = 1 min max

So my proposal is that we should:

 * Give ample timeouts to test (at least 2x their average
   run time on Travis CI)

 * Define the standard job (make check-acceptance) as a set
   of tests that can run under the Travis CI job (discounted
   the average QEMU build time)

This means that:

 * We'd tag some tests as "not-default", filtering them out
   of "make check-acceptance"

 * Supposing a developer is using a machine as least as powerful
   as the Travis CI environment, and assuming a build time of
   10 minutes, his "make check-acceptance" maximum execution
   time would be in the order of ~39 minutes.

I can work on adding the missing Avocado features, such as
the ability to list/count the maximum job time for the given test
selection. This should help us to maintain sound CI jobs, and good
user experience.

And finally, I'm sorry that I did overthink this... but I know
that the time for hard choices are coming fast.

Thanks,
- Cleber.

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Wed, May 22, 2019 at 09:04:17PM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > To: "Cleber Rosa" <crosa@redhat.com>
> > Cc: "Philippe Mathieu-Daudé" <f4bug@amsat.org>, qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> > "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien
> > Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>
> > Sent: Wednesday, May 22, 2019 7:07:05 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > 
> > On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> > > > "Aleksandar Markovic"
> > > > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic"
> > > > <amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta"
> > > > <wainersm@redhat.com>
> > > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > > > 
> > > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > > > Hi,
> > > > > 
> > > > > It was a rainy week-end here, so I invested it to automatize some
> > > > > of my MIPS tests.
> > > > > 
> > > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > > meant to run on a CI system but rather on a workstation previous
> > > > > to post a pull request.
> > > > > It can surely be improved, but it is a good starting point.
> > > > 
> > > > Until we actually have a mechanism to exclude the test case on
> > > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > > please don't merge patch 4/4 yet or it will break travis-ci.
> > > > 
> > > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > > tests tagged with "slow"?
> > > > 
> > > 
> > > The mechanism exists, but we haven't tagged any test so far as slow.
> > > 
> > > Should we define/document a criteria for a test to be slow?  Given
> > > that this is highly subjective, we have to think of:
> > > 
> > >  * Will we consider the average or maximum run time (the timeout
> > >    definition)?
> > >  
> > >  * For a single test, what is "slow"? Some rough numbers from Travis
> > >    CI[1] to help us with guidelines:
> > >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> > >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> > >    -
> > >    linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > >    PASS (18.14 s)
> > >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > 
> > I don't think we need to overthink this.  Whatever objective
> > criteria we choose, I'm sure we'll have to adapt them later due
> > to real world problems.
> > 
> > e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> > break Travis and other CI systems often because of timeouts?  If
> > yes, then we should probably tag it as slow.
> > 
> 
> It's not only that.  We're close to a point where we'll need to
> determine whether "make check-acceptance" will work as a generic
> enough default for most user on their environments and most CI
> systems.
> 
> As an example, this job ran 5 fairly slow tests (which I'm preparing
> to send):
> 
>   https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
> 
> Those are justifiably slow, given the fact that they boot a full
> Fedora 30 system using TCG.  The job has a cumulative execution time
> of ~39 minutes.  That leaves only 11 minutes to spare on the Travis
> CI environment.  If they all exercised close to their 600s allowances
> (timeout), the Travis job would have failed. 
> 
> Having said that, if a CI failure is supposed to be a major breakage,
> which I believe it's the right mind set and a worthy goal, we should
> limit the amount of tests we run so that their *maximum* execution
> time does not exceed the maximum job time limit.
> 
> > If having subjective criteria is really a problem (I don't think
> > it is), then we can call the tag "skip_travis", and stop worrying
> > about defining what exactly is "slow".
> > 
> > 
> > > 
> > >  * Do we want to set a maximum job timeout?  This way we can skip
> > >    tests after a given amount of time has passed.  Currently we interrupt
> > >    the test running when the job timeout is reached, but it's possible
> > >    to add a option so that no new tests will be started, but currently
> > >    running ones will be waited on.
> > 
> > I'm not sure I understand the suggestion to skip tests.  If we
> > skip tests after a timeout, how would we differentiate a test
> > being expectedly slow from a QEMU hang?
> > 
> > --
> > Eduardo
> > 
> 
> Basically, what I meant is that we could attempt something like:
> 
>  * Job "Brave"
>   - 50 tests, each with 60 seconds timeout = 50 min max
>   - 60 tests, each with 1 second timeout  = 1 min max
> 
> If Job "Brave" is run on a system such as Travis, it *can* fail,
> because it can go over the maximum Travis CI job limit of 50 min.
> We could set an Avocado job timeout of say, 48 minutes, and tell
> Avocado to mark the tests it wasn't able to spawn as "SKIPPED",
> and do not report an overall error condition.

Oh, that would be a nice feature.  But while we don't have it,
the following proposal would work too.

> 
> But, if we want to be more conservative (which I now realize is
> the best mindset for this situation), we should stick to something
> like:
> 
>  * Job "Coward"
>   - 47 tests, each with 60 seconds timeout = 47 min max
>   - 60 tests, each with 1 second timeout  = 1 min max
> 
> So my proposal is that we should:
> 
>  * Give ample timeouts to test (at least 2x their average
>    run time on Travis CI)
> 
>  * Define the standard job (make check-acceptance) as a set
>    of tests that can run under the Travis CI job (discounted
>    the average QEMU build time)

Agreed.

> 
> This means that:
> 
>  * We'd tag some tests as "not-default", filtering them out
>    of "make check-acceptance"
> 
>  * Supposing a developer is using a machine as least as powerful
>    as the Travis CI environment, and assuming a build time of
>    10 minutes, his "make check-acceptance" maximum execution
>    time would be in the order of ~39 minutes.
> 
> I can work on adding the missing Avocado features, such as
> the ability to list/count the maximum job time for the given test
> selection. This should help us to maintain sound CI jobs, and good
> user experience.

Sounds good to me.

> 
> And finally, I'm sorry that I did overthink this... but I know
> that the time for hard choices are coming fast.

The above proposals are cool, I don't think they are
overthinking.

I only meant that we shouldn't be looking to a formal definition
of "slow", because "what exactly is a slow job?" isn't the
important question we should be asking.  "How to avoid timeouts
on CI jobs" is the important question, and your proposals above
help us address that.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Philippe Mathieu-Daudé]

On 5/23/19 1:07 AM, Eduardo Habkost wrote:
> On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
>> ----- Original Message -----
>>> From: "Eduardo Habkost" <ehabkost@redhat.com>
>>> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
>>>> Hi,
>>>>
>>>> It was a rainy week-end here, so I invested it to automatize some
>>>> of my MIPS tests.
>>>>
>>>> The BootLinuxSshTest is not Global warming friendly, it is not
>>>> meant to run on a CI system but rather on a workstation previous
>>>> to post a pull request.
>>>> It can surely be improved, but it is a good starting point.
>>>
>>> Until we actually have a mechanism to exclude the test case on
>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
>>> please don't merge patch 4/4 yet or it will break travis-ci.
>>>
>>> Cleber, Wainer, is it already possible to make "avocado run" skip
>>> tests tagged with "slow"?
>>>
>>
>> The mechanism exists, but we haven't tagged any test so far as slow.
>>
>> Should we define/document a criteria for a test to be slow?  Given
>> that this is highly subjective, we have to think of:
>>
>>  * Will we consider the average or maximum run time (the timeout
>>    definition)?
>>  
>>  * For a single test, what is "slow"? Some rough numbers from Travis
>>    CI[1] to help us with guidelines:
>>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
>>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
>>    - linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:  PASS (18.14 s)
>>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> 
> I don't think we need to overthink this.  Whatever objective
> criteria we choose, I'm sure we'll have to adapt them later due
> to real world problems.
> 
> e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> break Travis and other CI systems often because of timeouts?  If
> yes, then we should probably tag it as slow.
> 
> If having subjective criteria is really a problem (I don't think
> it is), then we can call the tag "skip_travis", and stop worrying
> about defining what exactly is "slow".

I'd go with a simpler "tags:travis-ci" whitelisting any job expecting to
run smoothly there.

Then we can add "slow" tests without having to worry about blacklisting
for Travis CI.
Also, Other CI can set different timeouts.

I'd like maintainers to add as many tests as they want to upstream, so
these tests can eventually run by anyone, then each maintainer is free
to select which particular set he wants to run as default.

>>  * Do we want to set a maximum job timeout?  This way we can skip
>>    tests after a given amount of time has passed.  Currently we interrupt
>>    the test running when the job timeout is reached, but it's possible
>>    to add a option so that no new tests will be started, but currently
>>    running ones will be waited on.
> 
> I'm not sure I understand the suggestion to skip tests.  If we
> skip tests after a timeout, how would we differentiate a test
> being expectedly slow from a QEMU hang?
> 

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa]

----- Original Message -----
> From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> To: "Eduardo Habkost" <ehabkost@redhat.com>, "Cleber Rosa" <crosa@redhat.com>
> Cc: "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>, "Wainer dos Santos
> Moschetta" <wainersm@redhat.com>, qemu-devel@nongnu.org, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>,
> "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien Jarno" <aurelien@aurel32.net>
> Sent: Thursday, May 23, 2019 5:38:34 AM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On 5/23/19 1:07 AM, Eduardo Habkost wrote:
> > On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> >> ----- Original Message -----
> >>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> >>> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> >>>> Hi,
> >>>>
> >>>> It was a rainy week-end here, so I invested it to automatize some
> >>>> of my MIPS tests.
> >>>>
> >>>> The BootLinuxSshTest is not Global warming friendly, it is not
> >>>> meant to run on a CI system but rather on a workstation previous
> >>>> to post a pull request.
> >>>> It can surely be improved, but it is a good starting point.
> >>>
> >>> Until we actually have a mechanism to exclude the test case on
> >>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> >>> please don't merge patch 4/4 yet or it will break travis-ci.
> >>>
> >>> Cleber, Wainer, is it already possible to make "avocado run" skip
> >>> tests tagged with "slow"?
> >>>
> >>
> >> The mechanism exists, but we haven't tagged any test so far as slow.
> >>
> >> Should we define/document a criteria for a test to be slow?  Given
> >> that this is highly subjective, we have to think of:
> >>
> >>  * Will we consider the average or maximum run time (the timeout
> >>    definition)?
> >>  
> >>  * For a single test, what is "slow"? Some rough numbers from Travis
> >>    CI[1] to help us with guidelines:
> >>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> >>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> >>    -
> >>    linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> >>    PASS (18.14 s)
> >>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > 
> > I don't think we need to overthink this.  Whatever objective
> > criteria we choose, I'm sure we'll have to adapt them later due
> > to real world problems.
> > 
> > e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> > break Travis and other CI systems often because of timeouts?  If
> > yes, then we should probably tag it as slow.
> > 
> > If having subjective criteria is really a problem (I don't think
> > it is), then we can call the tag "skip_travis", and stop worrying
> > about defining what exactly is "slow".
> 
> I'd go with a simpler "tags:travis-ci" whitelisting any job expecting to
> run smoothly there.
> 

My concern is what becomes of "make check-acceptance".  Should we introduce
another target, say, "make check-acceptance-ci" or just change its meaning
and reuse it?

> Then we can add "slow" tests without having to worry about blacklisting
> for Travis CI.
> Also, Other CI can set different timeouts.
> 
> I'd like maintainers to add as many tests as they want to upstream, so
> these tests can eventually run by anyone, then each maintainer is free
> to select which particular set he wants to run as default.
> 

OK, so this matches the idea of carefully curating a set of tests for
CI.  WRT white or blacklisting, I favor the approach that requires the
least effort from the developer to have its test enabled, so I'd go
with blacklisting.  I fear that simple tests will just sit on the repo
without being properly exercised if we need to whitelist them.

But, I'll certainly and gladly accept the majority's opinion here.  

Regards,
- Cleber.

> >>  * Do we want to set a maximum job timeout?  This way we can skip
> >>    tests after a given amount of time has passed.  Currently we interrupt
> >>    the test running when the job timeout is reached, but it's possible
> >>    to add a option so that no new tests will be started, but currently
> >>    running ones will be waited on.
> > 
> > I'm not sure I understand the suggestion to skip tests.  If we
> > skip tests after a timeout, how would we differentiate a test
> > being expectedly slow from a QEMU hang?
> > 
> 

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Thu, May 23, 2019 at 09:28:00AM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> > To: "Eduardo Habkost" <ehabkost@redhat.com>, "Cleber Rosa" <crosa@redhat.com>
> > Cc: "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>, "Wainer dos Santos
> > Moschetta" <wainersm@redhat.com>, qemu-devel@nongnu.org, "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>,
> > "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien Jarno" <aurelien@aurel32.net>
> > Sent: Thursday, May 23, 2019 5:38:34 AM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > 
> > On 5/23/19 1:07 AM, Eduardo Habkost wrote:
> > > On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> > >> ----- Original Message -----
> > >>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > >>> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > >>>> Hi,
> > >>>>
> > >>>> It was a rainy week-end here, so I invested it to automatize some
> > >>>> of my MIPS tests.
> > >>>>
> > >>>> The BootLinuxSshTest is not Global warming friendly, it is not
> > >>>> meant to run on a CI system but rather on a workstation previous
> > >>>> to post a pull request.
> > >>>> It can surely be improved, but it is a good starting point.
> > >>>
> > >>> Until we actually have a mechanism to exclude the test case on
> > >>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > >>> please don't merge patch 4/4 yet or it will break travis-ci.
> > >>>
> > >>> Cleber, Wainer, is it already possible to make "avocado run" skip
> > >>> tests tagged with "slow"?
> > >>>
> > >>
> > >> The mechanism exists, but we haven't tagged any test so far as slow.
> > >>
> > >> Should we define/document a criteria for a test to be slow?  Given
> > >> that this is highly subjective, we have to think of:
> > >>
> > >>  * Will we consider the average or maximum run time (the timeout
> > >>    definition)?
> > >>  
> > >>  * For a single test, what is "slow"? Some rough numbers from Travis
> > >>    CI[1] to help us with guidelines:
> > >>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> > >>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> > >>    -
> > >>    linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > >>    PASS (18.14 s)
> > >>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > > 
> > > I don't think we need to overthink this.  Whatever objective
> > > criteria we choose, I'm sure we'll have to adapt them later due
> > > to real world problems.
> > > 
> > > e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> > > break Travis and other CI systems often because of timeouts?  If
> > > yes, then we should probably tag it as slow.
> > > 
> > > If having subjective criteria is really a problem (I don't think
> > > it is), then we can call the tag "skip_travis", and stop worrying
> > > about defining what exactly is "slow".
> > 
> > I'd go with a simpler "tags:travis-ci" whitelisting any job expecting to
> > run smoothly there.
> > 
> 
> My concern is what becomes of "make check-acceptance".  Should we introduce
> another target, say, "make check-acceptance-ci" or just change its meaning
> and reuse it?

What about "make check-acceptance TAG=travis-ci"?

> 
> > Then we can add "slow" tests without having to worry about blacklisting
> > for Travis CI.
> > Also, Other CI can set different timeouts.
> > 
> > I'd like maintainers to add as many tests as they want to upstream, so
> > these tests can eventually run by anyone, then each maintainer is free
> > to select which particular set he wants to run as default.
> > 
> 
> OK, so this matches the idea of carefully curating a set of tests for
> CI.  WRT white or blacklisting, I favor the approach that requires the
> least effort from the developer to have its test enabled, so I'd go
> with blacklisting.  I fear that simple tests will just sit on the repo
> without being properly exercised if we need to whitelist them.
> 

I agree.  I'd prefer the default case to be simple and not
require extra tags.  (i.e. tests without any tags would be run in
Travis by default).

-- 
Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 23, 2019 11:31 PM, "Eduardo Habkost" <ehabkost@redhat.com> wrote:
>
> On Thu, May 23, 2019 at 09:28:00AM -0400, Cleber Rosa wrote:
> >
> >
> > ----- Original Message -----
> > > From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> > > To: "Eduardo Habkost" <ehabkost@redhat.com>, "Cleber Rosa" <
crosa@redhat.com>
> > > Cc: "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Philippe
Mathieu-Daudé" <f4bug@amsat.org>, "Wainer dos Santos
> > > Moschetta" <wainersm@redhat.com>, qemu-devel@nongnu.org, "Aleksandar
Markovic" <aleksandar.m.mail@gmail.com>,
> > > "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien Jarno" <
aurelien@aurel32.net>
> > > Sent: Thursday, May 23, 2019 5:38:34 AM
> > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > >
> > > On 5/23/19 1:07 AM, Eduardo Habkost wrote:
> > > > On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> > > >> ----- Original Message -----
> > > >>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > >>> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé
wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> It was a rainy week-end here, so I invested it to automatize some
> > > >>>> of my MIPS tests.
> > > >>>>
> > > >>>> The BootLinuxSshTest is not Global warming friendly, it is not
> > > >>>> meant to run on a CI system but rather on a workstation previous
> > > >>>> to post a pull request.
> > > >>>> It can surely be improved, but it is a good starting point.
> > > >>>
> > > >>> Until we actually have a mechanism to exclude the test case on
> > > >>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > >>> please don't merge patch 4/4 yet or it will break travis-ci.
> > > >>>
> > > >>> Cleber, Wainer, is it already possible to make "avocado run" skip
> > > >>> tests tagged with "slow"?
> > > >>>
> > > >>
> > > >> The mechanism exists, but we haven't tagged any test so far as
slow.
> > > >>
> > > >> Should we define/document a criteria for a test to be slow?  Given
> > > >> that this is highly subjective, we have to think of:
> > > >>
> > > >>  * Will we consider the average or maximum run time (the timeout
> > > >>    definition)?
> > > >>
> > > >>  * For a single test, what is "slow"? Some rough numbers from
Travis
> > > >>    CI[1] to help us with guidelines:
> > > >>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS
(6.04 s)
> > > >>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS
(2.91 s)
> > > >>    -
> > > >>
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > > >>    PASS (18.14 s)
> > > >>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > > >
> > > > I don't think we need to overthink this.  Whatever objective
> > > > criteria we choose, I'm sure we'll have to adapt them later due
> > > > to real world problems.
> > > >
> > > > e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> > > > break Travis and other CI systems often because of timeouts?  If
> > > > yes, then we should probably tag it as slow.
> > > >
> > > > If having subjective criteria is really a problem (I don't think
> > > > it is), then we can call the tag "skip_travis", and stop worrying
> > > > about defining what exactly is "slow".
> > >
> > > I'd go with a simpler "tags:travis-ci" whitelisting any job expecting
to
> > > run smoothly there.
> > >
> >
> > My concern is what becomes of "make check-acceptance".  Should we
introduce
> > another target, say, "make check-acceptance-ci" or just change its
meaning
> > and reuse it?
>
> What about "make check-acceptance TAG=travis-ci"?
>
> >
> > > Then we can add "slow" tests without having to worry about
blacklisting
> > > for Travis CI.
> > > Also, Other CI can set different timeouts.
> > >
> > > I'd like maintainers to add as many tests as they want to upstream, so
> > > these tests can eventually run by anyone, then each maintainer is free
> > > to select which particular set he wants to run as default.
> > >
> >
> > OK, so this matches the idea of carefully curating a set of tests for
> > CI.  WRT white or blacklisting, I favor the approach that requires the
> > least effort from the developer to have its test enabled, so I'd go
> > with blacklisting.  I fear that simple tests will just sit on the repo
> > without being properly exercised if we need to whitelist them.
> >
>
> I agree.  I'd prefer the default case to be simple and not
> require extra tags.  (i.e. tests without any tags would be run in
> Travis by default).
>

Eduardo,

You are confusing me here.

You first suggest:

> What about "make check-acceptance TAG=travis-ci"?

... and then say:

> ...tests without any tags would be run in Travis by default.

For casual observers like me it is contradictory, I must be missing
something here, no?

Regards,
Aleksandar

> --
> Eduardo

Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Eduardo Habkost]

On Fri, May 24, 2019 at 03:45:56PM +0200, Aleksandar Markovic wrote:
> On May 23, 2019 11:31 PM, "Eduardo Habkost" <ehabkost@redhat.com> wrote:
> >
> > On Thu, May 23, 2019 at 09:28:00AM -0400, Cleber Rosa wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> > > > To: "Eduardo Habkost" <ehabkost@redhat.com>, "Cleber Rosa" <
> crosa@redhat.com>
> > > > Cc: "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Philippe
> Mathieu-Daudé" <f4bug@amsat.org>, "Wainer dos Santos
> > > > Moschetta" <wainersm@redhat.com>, qemu-devel@nongnu.org, "Aleksandar
> Markovic" <aleksandar.m.mail@gmail.com>,
> > > > "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Aurelien Jarno" <
> aurelien@aurel32.net>
> > > > Sent: Thursday, May 23, 2019 5:38:34 AM
> > > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > > >
> > > > On 5/23/19 1:07 AM, Eduardo Habkost wrote:
> > > > > On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
> > > > >> ----- Original Message -----
> > > > >>> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > > >>> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé
> wrote:
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> It was a rainy week-end here, so I invested it to automatize some
> > > > >>>> of my MIPS tests.
> > > > >>>>
> > > > >>>> The BootLinuxSshTest is not Global warming friendly, it is not
> > > > >>>> meant to run on a CI system but rather on a workstation previous
> > > > >>>> to post a pull request.
> > > > >>>> It can surely be improved, but it is a good starting point.
> > > > >>>
> > > > >>> Until we actually have a mechanism to exclude the test case on
> > > > >>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > > >>> please don't merge patch 4/4 yet or it will break travis-ci.
> > > > >>>
> > > > >>> Cleber, Wainer, is it already possible to make "avocado run" skip
> > > > >>> tests tagged with "slow"?
> > > > >>>
> > > > >>
> > > > >> The mechanism exists, but we haven't tagged any test so far as
> slow.
> > > > >>
> > > > >> Should we define/document a criteria for a test to be slow?  Given
> > > > >> that this is highly subjective, we have to think of:
> > > > >>
> > > > >>  * Will we consider the average or maximum run time (the timeout
> > > > >>    definition)?
> > > > >>
> > > > >>  * For a single test, what is "slow"? Some rough numbers from
> Travis
> > > > >>    CI[1] to help us with guidelines:
> > > > >>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS
> (6.04 s)
> > > > >>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS
> (2.91 s)
> > > > >>    -
> > > > >>
> linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > > > >>    PASS (18.14 s)
> > > > >>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > > > >
> > > > > I don't think we need to overthink this.  Whatever objective
> > > > > criteria we choose, I'm sure we'll have to adapt them later due
> > > > > to real world problems.
> > > > >
> > > > > e.g.: is 396 seconds too slow?  I don't know, it depends: does it
> > > > > break Travis and other CI systems often because of timeouts?  If
> > > > > yes, then we should probably tag it as slow.
> > > > >
> > > > > If having subjective criteria is really a problem (I don't think
> > > > > it is), then we can call the tag "skip_travis", and stop worrying
> > > > > about defining what exactly is "slow".
> > > >
> > > > I'd go with a simpler "tags:travis-ci" whitelisting any job expecting
> to
> > > > run smoothly there.
> > > >
> > >
> > > My concern is what becomes of "make check-acceptance".  Should we
> introduce
> > > another target, say, "make check-acceptance-ci" or just change its
> meaning
> > > and reuse it?
> >
> > What about "make check-acceptance TAG=travis-ci"?
> >
> > >
> > > > Then we can add "slow" tests without having to worry about
> blacklisting
> > > > for Travis CI.
> > > > Also, Other CI can set different timeouts.
> > > >
> > > > I'd like maintainers to add as many tests as they want to upstream, so
> > > > these tests can eventually run by anyone, then each maintainer is free
> > > > to select which particular set he wants to run as default.
> > > >
> > >
> > > OK, so this matches the idea of carefully curating a set of tests for
> > > CI.  WRT white or blacklisting, I favor the approach that requires the
> > > least effort from the developer to have its test enabled, so I'd go
> > > with blacklisting.  I fear that simple tests will just sit on the repo
> > > without being properly exercised if we need to whitelist them.
> > >
> >
> > I agree.  I'd prefer the default case to be simple and not
> > require extra tags.  (i.e. tests without any tags would be run in
> > Travis by default).
> >
> 
> Eduardo,
> 
> You are confusing me here.
> 
> You first suggest:
> 
> > What about "make check-acceptance TAG=travis-ci"?
> 

I was just trying to suggest using make variables as input to
check-acceptance, instead of creating separate makefile rules for
each set of test cases.  But you are right:

> ... and then say:
> 
> > ...tests without any tags would be run in Travis by default.
> 
> For casual observers like me it is contradictory, I must be missing
> something here, no?

Yes, if we use tags to exclude tests, the command line would look
different.  Maybe something like:

  make check-acceptance EXCLUDE_TAGS=skip-travis

The exact format of the arguments don't matter to me, as long as
we don't require people to write new makefile rules just because
they want to run a different set of test cases.

-- 
Eduardo