[PATCH v3 00/11] Record/replay acceptance tests

[PATCH v3 00/11] Record/replay acceptance tests

[Pavel Dovgalyuk]

The following series adds record/replay tests to the acceptance group.
Test pass successfully with the latest submitted record/replay fixes:
 - replay: notify the main loop when there are no instructions
 - replay: synchronize on every virtual timer callback

The provided tests perform kernel boot and disk image boot scenarios.
For all of them recording and replaying phases are executed.
Tests were borrowed from existing boot_linux*.py tests. But some
of the platforms and images were excluded, because icount for them
still has some issues.

Tested-by: Philippe Mathieu-Daude <philmd@redhat.com>

v3 changes:
 - Added record/replay logging (suggested by Philippe Mathieu-Daudé)
 - Changed the console pattern to get longer kernel execution (suggested by Alex Bennée)
 - Coding style fixes
 - Other minor changes
v2 changes:
 - Some test structure refactoring (suggested by Willian Rampazzo)

---

Pavel Dovgaluk (11):
      tests/acceptance: allow console interaction with specific VMs
      tests/acceptance: refactor boot_linux_console test to allow code reuse
      tests/acceptance: add base class record/replay kernel tests
      tests/acceptance: add kernel record/replay test for x86_64
      tests/acceptance: add record/replay test for aarch64
      tests/acceptance: add record/replay test for arm
      tests/acceptance: add record/replay test for ppc64
      tests/acceptance: add record/replay test for m68k
      tests/acceptance: record/replay tests with advcal images
      tests/acceptance: refactor boot_linux to allow code reuse
      tests/acceptance: Linux boot test for record/replay


 0 files changed

--
Pavel Dovgalyuk

[PATCH v3 01/11] tests/acceptance: allow console interaction with specific VMs

[Pavel Dovgalyuk]

Console interaction in avocado scripts was possible only with single
default VM.
This patch modifies the function parameters to allow passing a specific
VM as a parameter to interact with it.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 0 files changed

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 59e7b4f763..77d1c1d9ff 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -69,13 +69,15 @@ def pick_default_qemu_bin(arch=None):
 
 
 def _console_interaction(test, success_message, failure_message,
-                         send_string, keep_sending=False):
+                         send_string, keep_sending=False, vm=None):
     assert not keep_sending or send_string
-    console = test.vm.console_socket.makefile()
+    if vm is None:
+        vm = test.vm
+    console = vm.console_socket.makefile()
     console_logger = logging.getLogger('console')
     while True:
         if send_string:
-            test.vm.console_socket.sendall(send_string.encode())
+            vm.console_socket.sendall(send_string.encode())
             if not keep_sending:
                 send_string = None # send only once
         msg = console.readline().strip()
@@ -115,7 +117,8 @@ def interrupt_interactive_console_until_pattern(test, success_message,
     _console_interaction(test, success_message, failure_message,
                          interrupt_string, True)
 
-def wait_for_console_pattern(test, success_message, failure_message=None):
+def wait_for_console_pattern(test, success_message, failure_message=None,
+                             vm=None):
     """
     Waits for messages to appear on the console, while logging the content
 
@@ -125,7 +128,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None):
     :param success_message: if this message appears, test succeeds
     :param failure_message: if this message appears, test fails
     """
-    _console_interaction(test, success_message, failure_message, None)
+    _console_interaction(test, success_message, failure_message, None, vm=vm)
 
 def exec_command_and_wait_for_pattern(test, command,
                                       success_message, failure_message=None):

[PATCH v3 02/11] tests/acceptance: refactor boot_linux_console test to allow code reuse

[Pavel Dovgalyuk]

This patch splits code in BootLinuxConsole class into two different
classes to allow reusing it by record/replay tests.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 0 files changed

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index c6b06a1a13..12725d4529 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -28,19 +28,13 @@ try:
 except CmdNotFoundError:
     P7ZIP_AVAILABLE = False
 
-class BootLinuxConsole(Test):
-    """
-    Boots a Linux kernel and checks that the console is operational and the
-    kernel command line is properly passed from QEMU to the kernel
-    """
-
-    timeout = 90
-
+class LinuxKernelTest(Test):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
-    def wait_for_console_pattern(self, success_message):
+    def wait_for_console_pattern(self, success_message, vm=None):
         wait_for_console_pattern(self, success_message,
-                                 failure_message='Kernel panic - not syncing')
+                                 failure_message='Kernel panic - not syncing',
+                                 vm=vm)
 
     def extract_from_deb(self, deb, path):
         """
@@ -79,6 +73,13 @@ class BootLinuxConsole(Test):
         os.chdir(cwd)
         return os.path.normpath(os.path.join(self.workdir, path))
 
+class BootLinuxConsole(LinuxKernelTest):
+    """
+    Boots a Linux kernel and checks that the console is operational and the
+    kernel command line is properly passed from QEMU to the kernel
+    """
+    timeout = 90
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64

[PATCH v3 03/11] tests/acceptance: add base class record/replay kernel tests

[Pavel Dovgalyuk]

This patch adds a base for testing kernel boot recording and replaying.
Each test has the phase of recording and phase of replaying.
Virtual machines just boot the kernel and do not interact with
the network.
Structure and image links for the tests are borrowed from boot_linux_console.py
Testing controls the message pattern at the end of the kernel
boot for both record and replay modes. In replay mode QEMU is also
intended to finish the execution automatically.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - changed default value of args to None (suggested by Willian Rampazzo)
 - inherited common functions from boot_linux_console (suggested by Willian Rampazzo)

v3:
 - added logging (suggested by Philippe Mathieu-Daudé)
---
 0 files changed

diff --git a/MAINTAINERS b/MAINTAINERS
index 47ef3139e6..e9a9ce4f66 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2497,6 +2497,7 @@ F: net/filter-replay.c
 F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
+F: tests/acceptance/replay_kernel.py
 
 IOVA Tree
 M: Peter Xu <peterx@redhat.com>
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
new file mode 100644
index 0000000000..77678760be
--- /dev/null
+++ b/tests/acceptance/replay_kernel.py
@@ -0,0 +1,73 @@
+# Record/replay test that boots a Linux kernel
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import logging
+import time
+
+from avocado_qemu import wait_for_console_pattern
+from avocado.utils import process
+from avocado.utils import archive
+from boot_linux_console import LinuxKernelTest
+
+class ReplayKernel(LinuxKernelTest):
+    """
+    Boots a Linux kernel in record mode and checks that the console
+    is operational and the kernel command line is properly passed
+    from QEMU to the kernel.
+    Then replays the same scenario and verifies, that QEMU correctly
+    terminates.
+    """
+
+    timeout = 90
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=1 panic=-1 '
+
+    def run_vm(self, kernel_path, kernel_command_line, console_pattern,
+               record, shift, args, replay_path):
+        logger = logging.getLogger('replay')
+        start_time = time.time()
+        vm = self.get_vm()
+        vm.set_console()
+        if record:
+            logger.info('recording the execution...')
+            mode = 'record'
+        else:
+            logger.info('replaying the execution...')
+            mode = 'replay'
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, replay_path),
+                    '-kernel', kernel_path,
+                    '-append', kernel_command_line,
+                    '-net', 'none',
+                    '-no-reboot')
+        if args:
+            vm.add_args(*args)
+        vm.launch()
+        self.wait_for_console_pattern(console_pattern, vm)
+        if record:
+            vm.shutdown()
+            logger.info('finished the recording with log size %s bytes'
+                % os.path.getsize(replay_path))
+        else:
+            vm.wait()
+            logger.info('successfully fihished the replay')
+        elapsed = time.time() - start_time
+        logger.info('elapsed time %.2f sec' % elapsed)
+        return elapsed
+
+    def run_rr(self, kernel_path, kernel_command_line, console_pattern,
+        shift=7, args=None):
+        replay_path = os.path.join(self.workdir, 'replay.bin')
+        t1 = self.run_vm(kernel_path, kernel_command_line, console_pattern,
+                    True, shift, args, replay_path)
+        t2 = self.run_vm(kernel_path, kernel_command_line, console_pattern,
+                    False, shift, args, replay_path)
+        logger = logging.getLogger('replay')
+        logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))

[PATCH v3 04/11] tests/acceptance: add kernel record/replay test for x86_64

[Pavel Dovgalyuk]

This patch adds a test for record/replay an execution of x86_64 machine.
Execution scenario includes simple kernel boot, which allows testing
basic hardware interaction in RR mode.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 77678760be..5b32126e27 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -71,3 +71,19 @@ class ReplayKernel(LinuxKernelTest):
                     False, shift, args, replay_path)
         logger = logging.getLogger('replay')
         logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
+
+    def test_x86_64_pc(self):
+        """
+        :avocado: tags=arch:x86_64
+        :avocado: tags=machine:pc
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        console_pattern = 'VFS: Cannot open root device'
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)

[PATCH v3 05/11] tests/acceptance: add record/replay test for aarch64

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for aarch64 platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v3:
 - added cpu tag for the test (suggested by Philippe Mathieu-Daudé)
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 5b32126e27..616d2dfc33 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -87,3 +87,22 @@ class ReplayKernel(LinuxKernelTest):
         console_pattern = 'VFS: Cannot open root device'
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)
+
+    def test_aarch64_virt(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        :avocado: tags=cpu:cortex-a53
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        console_pattern = 'VFS: Cannot open root device'
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+            args=('-cpu', 'cortex-a53'))

[PATCH v3 06/11] tests/acceptance: add record/replay test for arm

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for two different arm platforms.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 616d2dfc33..382f1248e0 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -106,3 +106,49 @@ class ReplayKernel(LinuxKernelTest):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern,
             args=('-cpu', 'cortex-a53'))
+
+    def test_arm_virt(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        console_pattern = 'VFS: Cannot open root device'
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
+
+    def test_arm_cubieboard_initrd(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:cubieboard
+        """
+        deb_url = ('https://apt.armbian.com/pool/main/l/'
+                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinuz-4.20.7-sunxi')
+        dtb_path = '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
+        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
+                      '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
+                      'arm/rootfs-armv5.cpio.gz')
+        initrd_hash = '2b50f1873e113523967806f4da2afe385462ff9b'
+        initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
+        initrd_path = os.path.join(self.workdir, 'rootfs.cpio')
+        archive.gzip_uncompress(initrd_path_gz, initrd_path)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyS0,115200 '
+                               'usbcore.nousb '
+                               'panic=-1 noreboot')
+        console_pattern = 'Boot successful.'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1,
+            args=('-dtb', dtb_path, '-initrd', initrd_path, '-no-reboot'))

[PATCH v3 07/11] tests/acceptance: add record/replay test for ppc64

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for ppc64 platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 382f1248e0..738367849f 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -152,3 +152,19 @@ class ReplayKernel(LinuxKernelTest):
         console_pattern = 'Boot successful.'
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1,
             args=('-dtb', dtb_path, '-initrd', initrd_path, '-no-reboot'))
+
+    def test_ppc64_pseries(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+                      '/fedora-secondary/releases/29/Everything/ppc64le/os'
+                      '/ppc/ppc64/vmlinuz')
+        kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
+        # icount is not good enough for PPC64 for complete boot yet
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)

[PATCH v3 08/11] tests/acceptance: add record/replay test for m68k

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for m68k platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 738367849f..c1ec002db6 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -168,3 +168,21 @@ class ReplayKernel(LinuxKernelTest):
         # icount is not good enough for PPC64 for complete boot yet
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def test_m68k_q800(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:q800
+        """
+        deb_url = ('https://snapshot.debian.org/archive/debian-ports'
+                   '/20191021T083923Z/pool-m68k/main'
+                   '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
+        deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-5.3.0-1-m68k')
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyS0 vga=off')
+        console_pattern = 'No filesystem could mount root'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)

[PATCH v3 09/11] tests/acceptance: record/replay tests with advcal images

[Pavel Dovgalyuk]

This patch adds more record/replay tests with kernel images.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - make download path fixed to allow pre-test downloading (suggested by Willian Rampazzo)
---
 0 files changed

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index c1ec002db6..bc21ddf082 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -186,3 +186,108 @@ class ReplayKernel(LinuxKernelTest):
                                'console=ttyS0 vga=off')
         console_pattern = 'No filesystem could mount root'
         self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def do_test_advcal_2018(self, file_path, kernel_name, args=None):
+        archive.extract(file_path, self.workdir)
+
+        for entry in os.scandir(self.workdir):
+            if entry.name.startswith('day') and entry.is_dir():
+                kernel_path = entry.path + '/' + kernel_name
+                break
+
+        kernel_command_line = ''
+        console_pattern = 'QEMU advent calendar'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+            args=args)
+
+    def test_arm_vexpressa9(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:vexpress-a9
+        """
+        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day16.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'winter.zImage',
+            ('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb'))
+
+    def test_m68k_mcf5208evb(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:mcf5208evb
+        """
+        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day07.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
+
+    def test_microblaze_s3adsp1800(self):
+        """
+        :avocado: tags=arch:microblaze
+        :avocado: tags=machine:petalogix-s3adsp1800
+        """
+        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day17.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'ballerina.bin')
+
+    def test_ppc64_e500(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:ppce500
+        """
+        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day19.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
+
+    def test_ppc_g3beige(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:g3beige
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day15.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'invaders.elf',
+            ('-M', 'graphics=off'))
+
+    def test_ppc_mac99(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:mac99
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day15.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'invaders.elf',
+            ('-M', 'graphics=off'))
+
+    def test_sparc_ss20(self):
+        """
+        :avocado: tags=arch:sparc
+        :avocado: tags=machine:SS-20
+        """
+        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day11.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'zImage.elf')
+
+    def test_xtensa_lx60(self):
+        """
+        :avocado: tags=arch:xtensa
+        :avocado: tags=machine:lx60
+        """
+        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day02.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'santas-sleigh-ride.elf',
+            ('-cpu', 'dc233c'))

[PATCH v3 10/11] tests/acceptance: refactor boot_linux to allow code reuse

[Pavel Dovgalyuk]

This patch moves image downloading functions to the separate class to allow
reusing them from record/replay tests.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
index 075a386300..3aa57e88b0 100644
--- a/tests/acceptance/boot_linux.py
+++ b/tests/acceptance/boot_linux.py
@@ -26,22 +26,8 @@ KVM_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "KVM"
 TCG_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "TCG"
 
 
-class BootLinux(Test):
-    """
-    Boots a Linux system, checking for a successful initialization
-    """
-
-    timeout = 900
-    chksum = None
-
-    def setUp(self):
-        super(BootLinux, self).setUp()
-        self.vm.add_args('-smp', '2')
-        self.vm.add_args('-m', '1024')
-        self.prepare_boot()
-        self.prepare_cloudinit()
-
-    def prepare_boot(self):
+class BootLinuxBase(Test):
+    def download_boot(self):
         self.log.debug('Looking for and selecting a qemu-img binary to be '
                        'used to create the bootable snapshot image')
         # If qemu-img has been built, use it, otherwise the system wide one
@@ -60,17 +46,17 @@ class BootLinux(Test):
         if image_arch == 'ppc64':
             image_arch = 'ppc64le'
         try:
-            self.boot = vmimage.get(
+            boot = vmimage.get(
                 'fedora', arch=image_arch, version='31',
                 checksum=self.chksum,
                 algorithm='sha256',
                 cache_dir=self.cache_dirs[0],
                 snapshot_dir=self.workdir)
-            self.vm.add_args('-drive', 'file=%s' % self.boot.path)
         except:
             self.cancel('Failed to download/prepare boot image')
+        return boot.path
 
-    def prepare_cloudinit(self):
+    def download_cloudinit(self):
         self.log.info('Preparing cloudinit image')
         try:
             cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
@@ -81,9 +67,32 @@ class BootLinux(Test):
                           # QEMU's hard coded usermode router address
                           phone_home_host='10.0.2.2',
                           phone_home_port=self.phone_home_port)
-            self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
         except Exception:
             self.cancel('Failed to prepared cloudinit image')
+        return cloudinit_iso
+
+class BootLinux(BootLinuxBase):
+    """
+    Boots a Linux system, checking for a successful initialization
+    """
+
+    timeout = 900
+    chksum = None
+
+    def setUp(self):
+        super(BootLinux, self).setUp()
+        self.vm.add_args('-smp', '2')
+        self.vm.add_args('-m', '1024')
+        self.prepare_boot()
+        self.prepare_cloudinit()
+
+    def prepare_boot(self):
+        path = self.download_boot()
+        self.vm.add_args('-drive', 'file=%s' % path)
+
+    def prepare_cloudinit(self):
+        cloudinit_iso = self.download_cloudinit()
+        self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
 
     def launch_and_wait(self):
         self.vm.set_console()

[PATCH v3 11/11] tests/acceptance: Linux boot test for record/replay

[Pavel Dovgalyuk]

This patch adds a test for record/replay, which boots Linux
image from the disk and interacts with the network.
The idea and code of this test is borrowed from boot_linux.py
However, currently record/replay works only for x86_64,
therefore other tests were excluded.

Each test consists of the following phases:
 - downloading the disk image
 - recording the execution
 - replaying the execution

Replay does not validates the output, but waits until QEMU
finishes the execution. This is reasonable, because
QEMU usually hangs when replay goes wrong.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 0 files changed

diff --git a/MAINTAINERS b/MAINTAINERS
index e9a9ce4f66..97f066a9b2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
 F: tests/acceptance/replay_kernel.py
+F: tests/acceptance/replay_linux.py
 
 IOVA Tree
 M: Peter Xu <peterx@redhat.com>
diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
new file mode 100644
index 0000000000..328b03bb33
--- /dev/null
+++ b/tests/acceptance/replay_linux.py
@@ -0,0 +1,114 @@
+# Record/replay test that boots a complete Linux system via a cloud image
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import logging
+import time
+
+from avocado.utils import cloudinit
+from avocado.utils import network
+from avocado.utils import vmimage
+from avocado.utils import datadrainer
+from avocado.utils.path import find_command
+from boot_linux import BootLinuxBase
+
+class ReplayLinux(BootLinuxBase):
+    """
+    Boots a Linux system, checking for a successful initialization
+    """
+
+    timeout = 1800
+    chksum = None
+    hdd = 'ide-hd'
+    cd = 'ide-cd'
+    bus = 'ide'
+
+    def setUp(self):
+        super(ReplayLinux, self).setUp()
+        self.boot_path = self.download_boot()
+        self.cloudinit_path = self.download_cloudinit()
+
+    def vm_add_disk(self, vm, path, id, device):
+        bus_string = ''
+        if self.bus:
+            bus_string = ',bus=%s.%d' % (self.bus, id,)
+        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
+        vm.add_args('-drive',
+            'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
+        vm.add_args('-device',
+            '%s,drive=disk%s-rr%s' % (device, id, bus_string))
+
+    def launch_and_wait(self, record, args, shift):
+        vm = self.get_vm()
+        vm.add_args('-smp', '1')
+        vm.add_args('-m', '1024')
+        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
+        if args:
+            vm.add_args(*args)
+        self.vm_add_disk(vm, self.boot_path, 0, self.hdd)
+        self.vm_add_disk(vm, self.cloudinit_path, 1, self.cd)
+        logger = logging.getLogger('replay')
+        if record:
+            logger.info('recording the execution...')
+            mode = 'record'
+        else:
+            logger.info('replaying the execution...')
+            mode = 'replay'
+        replay_path = os.path.join(self.workdir, 'replay.bin')
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, replay_path))
+
+        start_time = time.time()
+
+        vm.set_console()
+        vm.launch()
+        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
+                                    logger=self.log.getChild('console'),
+                                    stop_check=(lambda : not vm.is_running()))
+        console_drainer.start()
+        if record:
+            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port),
+                                          self.name)
+            vm.shutdown()
+            logger.info('finished the recording with log size %s bytes'
+                % os.path.getsize(replay_path))
+        else:
+            vm.wait()
+            logger.info('successfully fihished the replay')
+        elapsed = time.time() - start_time
+        logger.info('elapsed time %.2f sec' % elapsed)
+        return elapsed
+
+    def run_rr(self, args=None, shift=7):
+        t1 = self.launch_and_wait(True, args, shift)
+        t2 = self.launch_and_wait(False, args, shift)
+        logger = logging.getLogger('replay')
+        logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
+
+class ReplayLinuxX8664(ReplayLinux):
+    """
+    :avocado: tags=arch:x86_64
+    """
+
+    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
+
+    def test_pc_i440fx(self):
+        """
+        :avocado: tags=machine:pc
+        :avocado: tags=accel:tcg
+        """
+        self.run_rr(shift=1)
+
+    def test_pc_q35(self):
+        """
+        :avocado: tags=machine:q35
+        :avocado: tags=accel:tcg
+        """
+        self.run_rr(shift=3)

Re: [PATCH v3 10/11] tests/acceptance: refactor boot_linux to allow code reuse

[Philippe Mathieu-Daudé]

On 5/29/20 9:05 AM, Pavel Dovgalyuk wrote:
> This patch moves image downloading functions to the separate class to allow
> reusing them from record/replay tests.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  0 files changed
> 
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> index 075a386300..3aa57e88b0 100644
> --- a/tests/acceptance/boot_linux.py
> +++ b/tests/acceptance/boot_linux.py
> @@ -26,22 +26,8 @@ KVM_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "KVM"
>  TCG_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "TCG"
>  
>  
> -class BootLinux(Test):
> -    """
> -    Boots a Linux system, checking for a successful initialization
> -    """
> -
> -    timeout = 900
> -    chksum = None
> -
> -    def setUp(self):
> -        super(BootLinux, self).setUp()
> -        self.vm.add_args('-smp', '2')
> -        self.vm.add_args('-m', '1024')
> -        self.prepare_boot()
> -        self.prepare_cloudinit()
> -
> -    def prepare_boot(self):
> +class BootLinuxBase(Test):
> +    def download_boot(self):
>          self.log.debug('Looking for and selecting a qemu-img binary to be '
>                         'used to create the bootable snapshot image')
>          # If qemu-img has been built, use it, otherwise the system wide one
> @@ -60,17 +46,17 @@ class BootLinux(Test):
>          if image_arch == 'ppc64':
>              image_arch = 'ppc64le'
>          try:
> -            self.boot = vmimage.get(
> +            boot = vmimage.get(
>                  'fedora', arch=image_arch, version='31',
>                  checksum=self.chksum,
>                  algorithm='sha256',
>                  cache_dir=self.cache_dirs[0],
>                  snapshot_dir=self.workdir)
> -            self.vm.add_args('-drive', 'file=%s' % self.boot.path)
>          except:
>              self.cancel('Failed to download/prepare boot image')
> +        return boot.path
>  
> -    def prepare_cloudinit(self):
> +    def download_cloudinit(self):
>          self.log.info('Preparing cloudinit image')
>          try:
>              cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
> @@ -81,9 +67,32 @@ class BootLinux(Test):
>                            # QEMU's hard coded usermode router address
>                            phone_home_host='10.0.2.2',
>                            phone_home_port=self.phone_home_port)
> -            self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
>          except Exception:
>              self.cancel('Failed to prepared cloudinit image')
> +        return cloudinit_iso
> +
> +class BootLinux(BootLinuxBase):
> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 900
> +    chksum = None
> +
> +    def setUp(self):
> +        super(BootLinux, self).setUp()
> +        self.vm.add_args('-smp', '2')
> +        self.vm.add_args('-m', '1024')
> +        self.prepare_boot()
> +        self.prepare_cloudinit()
> +
> +    def prepare_boot(self):
> +        path = self.download_boot()
> +        self.vm.add_args('-drive', 'file=%s' % path)
> +
> +    def prepare_cloudinit(self):
> +        cloudinit_iso = self.download_cloudinit()
> +        self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
>  
>      def launch_and_wait(self):
>          self.vm.set_console()
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v3 00/11] Record/replay acceptance tests

[Philippe Mathieu-Daudé]

On 5/29/20 9:04 AM, Pavel Dovgalyuk wrote:
> The following series adds record/replay tests to the acceptance group.
> Test pass successfully with the latest submitted record/replay fixes:
>  - replay: notify the main loop when there are no instructions
>  - replay: synchronize on every virtual timer callback
> 
> The provided tests perform kernel boot and disk image boot scenarios.
> For all of them recording and replaying phases are executed.
> Tests were borrowed from existing boot_linux*.py tests. But some
> of the platforms and images were excluded, because icount for them
> still has some issues.
> 
> Tested-by: Philippe Mathieu-Daude <philmd@redhat.com>
> 
> v3 changes:
>  - Added record/replay logging (suggested by Philippe Mathieu-Daudé)
>  - Changed the console pattern to get longer kernel execution (suggested by Alex Bennée)
>  - Coding style fixes
>  - Other minor changes
> v2 changes:
>  - Some test structure refactoring (suggested by Willian Rampazzo)
> 
> ---
> 
> Pavel Dovgaluk (11):
>       tests/acceptance: allow console interaction with specific VMs
>       tests/acceptance: refactor boot_linux_console test to allow code reuse
>       tests/acceptance: add base class record/replay kernel tests
>       tests/acceptance: add kernel record/replay test for x86_64
>       tests/acceptance: add record/replay test for aarch64
>       tests/acceptance: add record/replay test for arm
>       tests/acceptance: add record/replay test for ppc64
>       tests/acceptance: add record/replay test for m68k
>       tests/acceptance: record/replay tests with advcal images
>       tests/acceptance: refactor boot_linux to allow code reuse
>       tests/acceptance: Linux boot test for record/replay

Thanks, patches 1/2/10 applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next

- tests/acceptance: allow console interaction with specific VMs
- tests/acceptance: refactor boot_linux_console test to allow code reuse
- tests/acceptance: refactor boot_linux to allow code reuse

Re: [PATCH v3 08/11] tests/acceptance: add record/replay test for m68k

[Laurent Vivier]

Le 29/05/2020 à 09:05, Pavel Dovgalyuk a écrit :
> This patch adds a test for record/replay of the kernel
> image boot for m68k platform.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  0 files changed
> 
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 738367849f..c1ec002db6 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -168,3 +168,21 @@ class ReplayKernel(LinuxKernelTest):
>          # icount is not good enough for PPC64 for complete boot yet
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.run_rr(kernel_path, kernel_command_line, console_pattern)
> +
> +    def test_m68k_q800(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:q800
> +        """
> +        deb_url = ('https://snapshot.debian.org/archive/debian-ports'
> +                   '/20191021T083923Z/pool-m68k/main'
> +                   '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
> +        deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        kernel_path = self.extract_from_deb(deb_path,
> +                                            '/boot/vmlinux-5.3.0-1-m68k')
> +
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> +                               'console=ttyS0 vga=off')
> +        console_pattern = 'No filesystem could mount root'
> +        self.run_rr(kernel_path, kernel_command_line, console_pattern)
> 
> 

Reviewed-by: Laurent Vivier <laurent@vivier.eu>

Re: [PATCH v3 00/11] Record/replay acceptance tests

[Philippe Mathieu-Daudé]

Hi Pavel,

On 5/31/20 5:09 PM, Philippe Mathieu-Daudé wrote:
> On 5/29/20 9:04 AM, Pavel Dovgalyuk wrote:
>> The following series adds record/replay tests to the acceptance group.
>> Test pass successfully with the latest submitted record/replay fixes:
>>  - replay: notify the main loop when there are no instructions
>>  - replay: synchronize on every virtual timer callback
>>
>> The provided tests perform kernel boot and disk image boot scenarios.
>> For all of them recording and replaying phases are executed.
>> Tests were borrowed from existing boot_linux*.py tests. But some
>> of the platforms and images were excluded, because icount for them
>> still has some issues.
>>
>> Tested-by: Philippe Mathieu-Daude <philmd@redhat.com>
>>
>> v3 changes:
>>  - Added record/replay logging (suggested by Philippe Mathieu-Daudé)
>>  - Changed the console pattern to get longer kernel execution (suggested by Alex Bennée)
>>  - Coding style fixes
>>  - Other minor changes
>> v2 changes:
>>  - Some test structure refactoring (suggested by Willian Rampazzo)
>>
>> ---
>>
>> Pavel Dovgaluk (11):
>>       tests/acceptance: allow console interaction with specific VMs
>>       tests/acceptance: refactor boot_linux_console test to allow code reuse
>>       tests/acceptance: add base class record/replay kernel tests
>>       tests/acceptance: add kernel record/replay test for x86_64
>>       tests/acceptance: add record/replay test for aarch64
>>       tests/acceptance: add record/replay test for arm
>>       tests/acceptance: add record/replay test for ppc64
>>       tests/acceptance: add record/replay test for m68k

You forgot to Cc the maintainers for each target/machine tested.

Anyway this series has been on the list for 3 weeks, so I'm
queuing it for next acceptance-testing pull request.

Since the x86 tests generate some timeout errors on Travis-CI,
I squashed the following changes:

-- >8 --
diff --git a/tests/acceptance/replay_kernel.py
b/tests/acceptance/replay_kernel.py
index a25eb3d3ee..8fa6b517c5 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -12,6 +12,7 @@
 import logging
 import time

+from avocado import skipIf
 from avocado_qemu import wait_for_console_pattern
 from avocado.utils import archive
 from avocado.utils import process
@@ -72,6 +73,7 @@ def run_rr(self, kernel_path, kernel_command_line,
console_pattern,
         logger = logging.getLogger('replay')
         logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))

+    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
diff --git a/tests/acceptance/replay_linux.py
b/tests/acceptance/replay_linux.py
index 472ab730cd..8e43e282a7 100644
--- a/tests/acceptance/replay_linux.py
+++ b/tests/acceptance/replay_linux.py
@@ -12,6 +12,7 @@
 import logging
 import time

+from avocado import skipIf
 from avocado.utils import cloudinit
 from avocado.utils import datadrainer
 from avocado.utils import network
@@ -92,6 +93,7 @@ def run_rr(self, args=None, shift=7):
         logger = logging.getLogger('replay')
         logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))

+@skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
 class ReplayLinuxX8664(ReplayLinux):
     """
     :avocado: tags=arch:x86_64
---

Regards,

Phil.

>>       tests/acceptance: record/replay tests with advcal images
>>       tests/acceptance: refactor boot_linux to allow code reuse
>>       tests/acceptance: Linux boot test for record/replay
> 
> Thanks, patches 1/2/10 applied to my python-next tree:
> https://gitlab.com/philmd/qemu/commits/python-next
> 
> - tests/acceptance: allow console interaction with specific VMs
> - tests/acceptance: refactor boot_linux_console test to allow code reuse
> - tests/acceptance: refactor boot_linux to allow code reuse
> 

Re: [PATCH v3 09/11] tests/acceptance: record/replay tests with advcal images

[Philippe Mathieu-Daudé]

Hi Pavel,

On 5/29/20 9:05 AM, Pavel Dovgalyuk wrote:
> This patch adds more record/replay tests with kernel images.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> 
> --
> 
> v2:
>  - make download path fixed to allow pre-test downloading (suggested by Willian Rampazzo)
> ---
>  0 files changed
> 
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index c1ec002db6..bc21ddf082 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -186,3 +186,108 @@ class ReplayKernel(LinuxKernelTest):
>                                 'console=ttyS0 vga=off')
>          console_pattern = 'No filesystem could mount root'
>          self.run_rr(kernel_path, kernel_command_line, console_pattern)
> +
> +    def do_test_advcal_2018(self, file_path, kernel_name, args=None):
> +        archive.extract(file_path, self.workdir)
> +
> +        for entry in os.scandir(self.workdir):
> +            if entry.name.startswith('day') and entry.is_dir():
> +                kernel_path = entry.path + '/' + kernel_name
> +                break
> +
> +        kernel_command_line = ''
> +        console_pattern = 'QEMU advent calendar'
> +        self.run_rr(kernel_path, kernel_command_line, console_pattern,
> +            args=args)
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day16.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'winter.zImage',
> +            ('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb'))
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day07.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day17.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'ballerina.bin')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day19.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day15.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'invaders.elf',
> +            ('-M', 'graphics=off'))
> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day15.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'invaders.elf',
> +            ('-M', 'graphics=off'))

Using QEMU built with -O3, I get:

 (4/4) tests/acceptance/replay_kernel.py:ReplayKernel.test_ppc_mac99:
replay: recording the execution...
replay: finished the recording with log size 21781169 bytes
replay: elapsed time 17.03 sec
replay: replaying the execution...
replay: successfully finished the replay
replay: elapsed time 57.04 sec
replay: replay overhead 234.93%
PASS (74.48 s)

Any idea why there is so much overhead here?

> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day11.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day02.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'santas-sleigh-ride.elf',
> +            ('-cpu', 'dc233c'))
> 

Re: [PATCH v3 00/11] Record/replay acceptance tests

[Pavel Dovgalyuk]

On 21.06.2020 02:36, Philippe Mathieu-Daudé wrote:
> Hi Pavel,
> 
> On 5/31/20 5:09 PM, Philippe Mathieu-Daudé wrote:
>> On 5/29/20 9:04 AM, Pavel Dovgalyuk wrote:
>>> The following series adds record/replay tests to the acceptance group.
>>> Test pass successfully with the latest submitted record/replay fixes:
>>>   - replay: notify the main loop when there are no instructions
>>>   - replay: synchronize on every virtual timer callback
>>>
>>> The provided tests perform kernel boot and disk image boot scenarios.
>>> For all of them recording and replaying phases are executed.
>>> Tests were borrowed from existing boot_linux*.py tests. But some
>>> of the platforms and images were excluded, because icount for them
>>> still has some issues.
>>>
>>> Tested-by: Philippe Mathieu-Daude <philmd@redhat.com>
>>>
>>> v3 changes:
>>>   - Added record/replay logging (suggested by Philippe Mathieu-Daudé)
>>>   - Changed the console pattern to get longer kernel execution (suggested by Alex Bennée)
>>>   - Coding style fixes
>>>   - Other minor changes
>>> v2 changes:
>>>   - Some test structure refactoring (suggested by Willian Rampazzo)
>>>
>>> ---
>>>
>>> Pavel Dovgaluk (11):
>>>        tests/acceptance: allow console interaction with specific VMs
>>>        tests/acceptance: refactor boot_linux_console test to allow code reuse
>>>        tests/acceptance: add base class record/replay kernel tests
>>>        tests/acceptance: add kernel record/replay test for x86_64
>>>        tests/acceptance: add record/replay test for aarch64
>>>        tests/acceptance: add record/replay test for arm
>>>        tests/acceptance: add record/replay test for ppc64
>>>        tests/acceptance: add record/replay test for m68k
> 
> You forgot to Cc the maintainers for each target/machine tested.
> 
> Anyway this series has been on the list for 3 weeks, so I'm
> queuing it for next acceptance-testing pull request.
> 
> Since the x86 tests generate some timeout errors on Travis-CI,

Errors occur with the commits that were not queued yet?

> I squashed the following changes:
> 
> -- >8 --
> diff --git a/tests/acceptance/replay_kernel.py
> b/tests/acceptance/replay_kernel.py
> index a25eb3d3ee..8fa6b517c5 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -12,6 +12,7 @@
>   import logging
>   import time
> 
> +from avocado import skipIf
>   from avocado_qemu import wait_for_console_pattern
>   from avocado.utils import archive
>   from avocado.utils import process
> @@ -72,6 +73,7 @@ def run_rr(self, kernel_path, kernel_command_line,
> console_pattern,
>           logger = logging.getLogger('replay')
>           logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
> 
> +    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
>       def test_x86_64_pc(self):
>           """
>           :avocado: tags=arch:x86_64
> diff --git a/tests/acceptance/replay_linux.py
> b/tests/acceptance/replay_linux.py
> index 472ab730cd..8e43e282a7 100644
> --- a/tests/acceptance/replay_linux.py
> +++ b/tests/acceptance/replay_linux.py
> @@ -12,6 +12,7 @@
>   import logging
>   import time
> 
> +from avocado import skipIf
>   from avocado.utils import cloudinit
>   from avocado.utils import datadrainer
>   from avocado.utils import network
> @@ -92,6 +93,7 @@ def run_rr(self, args=None, shift=7):
>           logger = logging.getLogger('replay')
>           logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
> 
> +@skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
>   class ReplayLinuxX8664(ReplayLinux):
>       """
>       :avocado: tags=arch:x86_64
> ---
> 
> Regards,
> 
> Phil.
> 
>>>        tests/acceptance: record/replay tests with advcal images
>>>        tests/acceptance: refactor boot_linux to allow code reuse
>>>        tests/acceptance: Linux boot test for record/replay
>>
>> Thanks, patches 1/2/10 applied to my python-next tree:
>> https://gitlab.com/philmd/qemu/commits/python-next
>>
>> - tests/acceptance: allow console interaction with specific VMs
>> - tests/acceptance: refactor boot_linux_console test to allow code reuse
>> - tests/acceptance: refactor boot_linux to allow code reuse
>>
> 

Re: [PATCH v3 11/11] tests/acceptance: Linux boot test for record/replay

[Philippe Mathieu-Daudé]

Hi Pavel,

On 5/29/20 9:05 AM, Pavel Dovgalyuk wrote:
> This patch adds a test for record/replay, which boots Linux
> image from the disk and interacts with the network.
> The idea and code of this test is borrowed from boot_linux.py
> However, currently record/replay works only for x86_64,
> therefore other tests were excluded.
> 
> Each test consists of the following phases:
>  - downloading the disk image
>  - recording the execution
>  - replaying the execution
> 
> Replay does not validates the output, but waits until QEMU
> finishes the execution. This is reasonable, because
> QEMU usually hangs when replay goes wrong.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  0 files changed
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e9a9ce4f66..97f066a9b2 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
>  F: docs/replay.txt
>  F: stubs/replay.c
>  F: tests/acceptance/replay_kernel.py
> +F: tests/acceptance/replay_linux.py
>  
>  IOVA Tree
>  M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
> new file mode 100644
> index 0000000000..328b03bb33
> --- /dev/null
> +++ b/tests/acceptance/replay_linux.py
> @@ -0,0 +1,114 @@
> +# Record/replay test that boots a complete Linux system via a cloud image
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +import logging
> +import time
> +
> +from avocado.utils import cloudinit
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import datadrainer
> +from avocado.utils.path import find_command
> +from boot_linux import BootLinuxBase
> +
> +class ReplayLinux(BootLinuxBase):
> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 1800

This test works, but is taking way too long for the default
test suite.

I said in the cover I'd use ...:

@skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')

... to skip on Travis, but I'll change by declaring this test
'slow'. We could use 'tags=slowness:high' but then we need to
modify the 'check-acceptance' default rule to skip tests
matching the tag.

Willian, Cleber, any clever idea?

> +    chksum = None
> +    hdd = 'ide-hd'
> +    cd = 'ide-cd'
> +    bus = 'ide'
> +
> +    def setUp(self):
> +        super(ReplayLinux, self).setUp()
> +        self.boot_path = self.download_boot()
> +        self.cloudinit_path = self.download_cloudinit()
> +
> +    def vm_add_disk(self, vm, path, id, device):
> +        bus_string = ''
> +        if self.bus:
> +            bus_string = ',bus=%s.%d' % (self.bus, id,)
> +        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
> +        vm.add_args('-drive',
> +            'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
> +        vm.add_args('-device',
> +            '%s,drive=disk%s-rr%s' % (device, id, bus_string))
> +
> +    def launch_and_wait(self, record, args, shift):
> +        vm = self.get_vm()
> +        vm.add_args('-smp', '1')
> +        vm.add_args('-m', '1024')
> +        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
> +        if args:
> +            vm.add_args(*args)
> +        self.vm_add_disk(vm, self.boot_path, 0, self.hdd)
> +        self.vm_add_disk(vm, self.cloudinit_path, 1, self.cd)
> +        logger = logging.getLogger('replay')
> +        if record:
> +            logger.info('recording the execution...')
> +            mode = 'record'
> +        else:
> +            logger.info('replaying the execution...')
> +            mode = 'replay'
> +        replay_path = os.path.join(self.workdir, 'replay.bin')
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, replay_path))
> +
> +        start_time = time.time()
> +
> +        vm.set_console()
> +        vm.launch()
> +        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
> +                                    logger=self.log.getChild('console'),
> +                                    stop_check=(lambda : not vm.is_running()))
> +        console_drainer.start()
> +        if record:
> +            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port),
> +                                          self.name)
> +            vm.shutdown()
> +            logger.info('finished the recording with log size %s bytes'
> +                % os.path.getsize(replay_path))
> +        else:
> +            vm.wait()
> +            logger.info('successfully fihished the replay')
> +        elapsed = time.time() - start_time
> +        logger.info('elapsed time %.2f sec' % elapsed)
> +        return elapsed
> +
> +    def run_rr(self, args=None, shift=7):
> +        t1 = self.launch_and_wait(True, args, shift)
> +        t2 = self.launch_and_wait(False, args, shift)
> +        logger = logging.getLogger('replay')
> +        logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
> +
> +class ReplayLinuxX8664(ReplayLinux):
> +    """
> +    :avocado: tags=arch:x86_64
> +    """
> +
> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
> +
> +    def test_pc_i440fx(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=1)
> +
> +    def test_pc_q35(self):
> +        """
> +        :avocado: tags=machine:q35
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=3)
> 

Re: [PATCH v3 11/11] tests/acceptance: Linux boot test for record/replay

[Philippe Mathieu-Daudé]

On 6/22/20 10:03 AM, Philippe Mathieu-Daudé wrote:
> Hi Pavel,
> 
> On 5/29/20 9:05 AM, Pavel Dovgalyuk wrote:
>> This patch adds a test for record/replay, which boots Linux
>> image from the disk and interacts with the network.
>> The idea and code of this test is borrowed from boot_linux.py
>> However, currently record/replay works only for x86_64,
>> therefore other tests were excluded.
>>
>> Each test consists of the following phases:
>>  - downloading the disk image
>>  - recording the execution
>>  - replaying the execution
>>
>> Replay does not validates the output, but waits until QEMU
>> finishes the execution. This is reasonable, because
>> QEMU usually hangs when replay goes wrong.
>>
>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>> ---
>>  0 files changed
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index e9a9ce4f66..97f066a9b2 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
>>  F: docs/replay.txt
>>  F: stubs/replay.c
>>  F: tests/acceptance/replay_kernel.py
>> +F: tests/acceptance/replay_linux.py
>>  
>>  IOVA Tree
>>  M: Peter Xu <peterx@redhat.com>
>> diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
>> new file mode 100644
>> index 0000000000..328b03bb33
>> --- /dev/null
>> +++ b/tests/acceptance/replay_linux.py
>> @@ -0,0 +1,114 @@
>> +# Record/replay test that boots a complete Linux system via a cloud image
>> +#
>> +# Copyright (c) 2020 ISP RAS
>> +#
>> +# Author:
>> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>> +# later.  See the COPYING file in the top-level directory.
>> +
>> +import os
>> +import logging
>> +import time
>> +
>> +from avocado.utils import cloudinit
>> +from avocado.utils import network
>> +from avocado.utils import vmimage
>> +from avocado.utils import datadrainer
>> +from avocado.utils.path import find_command
>> +from boot_linux import BootLinuxBase
>> +
>> +class ReplayLinux(BootLinuxBase):
>> +    """
>> +    Boots a Linux system, checking for a successful initialization
>> +    """
>> +
>> +    timeout = 1800
> 
> This test works, but is taking way too long for the default
> test suite.
> 
> I said in the cover I'd use ...:
> 
> @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> 
> ... to skip on Travis, but I'll change by declaring this test
> 'slow'. We could use 'tags=slowness:high' but then we need to
> modify the 'check-acceptance' default rule to skip tests
> matching the tag.
> 
> Willian, Cleber, any clever idea?

Meanwhile I'm taking this patch out of the queue, so it
won't delay the integration of the other tests.

> 
>> +    chksum = None
>> +    hdd = 'ide-hd'
>> +    cd = 'ide-cd'
>> +    bus = 'ide'
>> +
>> +    def setUp(self):
>> +        super(ReplayLinux, self).setUp()
>> +        self.boot_path = self.download_boot()
>> +        self.cloudinit_path = self.download_cloudinit()
>> +
>> +    def vm_add_disk(self, vm, path, id, device):
>> +        bus_string = ''
>> +        if self.bus:
>> +            bus_string = ',bus=%s.%d' % (self.bus, id,)
>> +        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
>> +        vm.add_args('-drive',
>> +            'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
>> +        vm.add_args('-device',
>> +            '%s,drive=disk%s-rr%s' % (device, id, bus_string))
>> +
>> +    def launch_and_wait(self, record, args, shift):
>> +        vm = self.get_vm()
>> +        vm.add_args('-smp', '1')
>> +        vm.add_args('-m', '1024')
>> +        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
>> +        if args:
>> +            vm.add_args(*args)
>> +        self.vm_add_disk(vm, self.boot_path, 0, self.hdd)
>> +        self.vm_add_disk(vm, self.cloudinit_path, 1, self.cd)
>> +        logger = logging.getLogger('replay')
>> +        if record:
>> +            logger.info('recording the execution...')
>> +            mode = 'record'
>> +        else:
>> +            logger.info('replaying the execution...')
>> +            mode = 'replay'
>> +        replay_path = os.path.join(self.workdir, 'replay.bin')
>> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
>> +                    (shift, mode, replay_path))
>> +
>> +        start_time = time.time()
>> +
>> +        vm.set_console()
>> +        vm.launch()
>> +        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
>> +                                    logger=self.log.getChild('console'),
>> +                                    stop_check=(lambda : not vm.is_running()))
>> +        console_drainer.start()
>> +        if record:
>> +            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port),
>> +                                          self.name)
>> +            vm.shutdown()
>> +            logger.info('finished the recording with log size %s bytes'
>> +                % os.path.getsize(replay_path))
>> +        else:
>> +            vm.wait()
>> +            logger.info('successfully fihished the replay')
>> +        elapsed = time.time() - start_time
>> +        logger.info('elapsed time %.2f sec' % elapsed)
>> +        return elapsed
>> +
>> +    def run_rr(self, args=None, shift=7):
>> +        t1 = self.launch_and_wait(True, args, shift)
>> +        t2 = self.launch_and_wait(False, args, shift)
>> +        logger = logging.getLogger('replay')
>> +        logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
>> +
>> +class ReplayLinuxX8664(ReplayLinux):
>> +    """
>> +    :avocado: tags=arch:x86_64
>> +    """
>> +
>> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
>> +
>> +    def test_pc_i440fx(self):
>> +        """
>> +        :avocado: tags=machine:pc
>> +        :avocado: tags=accel:tcg
>> +        """
>> +        self.run_rr(shift=1)
>> +
>> +    def test_pc_q35(self):
>> +        """
>> +        :avocado: tags=machine:q35
>> +        :avocado: tags=accel:tcg
>> +        """
>> +        self.run_rr(shift=3)
>>
>