[PATCH 0/9] Record/replay acceptance tests

[PATCH 0/9] Record/replay acceptance tests

[Pavel Dovgalyuk]

The following series adds record/replay tests to the acceptance group.
Test pass successfully with the latest submitted record/replay fixes:
 - replay: notify the main loop when there are no instructions
 - replay: synchronize on every virtual timer callback

The provided tests perform kernel boot and disk image boot scenarios.
For all of them recording and replaying phases are executed.
Tests were borrowed from existing boot_linux*.py tests. But some
of the platforms and images were excluded, because icount for them
still has some issues.

---

Pavel Dovgaluk (9):
      tests/acceptance: allow console interaction with specific VMs
      tests/acceptance: add base class record/replay kernel tests
      tests/acceptance: add kernel record/replay test for x86_64
      tests/acceptance: add record/replay test for aarch64
      tests/acceptance: add record/replay test for arm
      tests/acceptance: add record/replay test for ppc64
      tests/acceptance: add record/replay test for m68k
      tests/acceptance: record/replay tests with advcal images
      tests/acceptance: Linux boot test for record/replay


 MAINTAINERS                               |    2 
 tests/acceptance/avocado_qemu/__init__.py |   12 +
 tests/acceptance/replay_kernel.py         |  273 +++++++++++++++++++++++++++++
 tests/acceptance/replay_linux.py          |  140 +++++++++++++++
 4 files changed, 422 insertions(+), 5 deletions(-)
 create mode 100644 tests/acceptance/replay_kernel.py
 create mode 100644 tests/acceptance/replay_linux.py

--
Pavel Dovgalyuk

[PATCH 1/9] tests/acceptance: allow console interaction with specific VMs

[Pavel Dovgalyuk]

Console interaction in avocado scripts was possible only with single
default VM.
This patch modifies the function parameters to allow passing a specific
VM as a parameter to interact with it.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/avocado_qemu/__init__.py |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 59e7b4f763..0bbaa8d2a6 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -69,13 +69,15 @@ def pick_default_qemu_bin(arch=None):
 
 
 def _console_interaction(test, success_message, failure_message,
-                         send_string, keep_sending=False):
+                         send_string, keep_sending=False, vm=None):
     assert not keep_sending or send_string
-    console = test.vm.console_socket.makefile()
+    if vm is None:
+        vm = test.vm
+    console = vm.console_socket.makefile()
     console_logger = logging.getLogger('console')
     while True:
         if send_string:
-            test.vm.console_socket.sendall(send_string.encode())
+            vm.console_socket.sendall(send_string.encode())
             if not keep_sending:
                 send_string = None # send only once
         msg = console.readline().strip()
@@ -115,7 +117,7 @@ def interrupt_interactive_console_until_pattern(test, success_message,
     _console_interaction(test, success_message, failure_message,
                          interrupt_string, True)
 
-def wait_for_console_pattern(test, success_message, failure_message=None):
+def wait_for_console_pattern(test, success_message, failure_message=None, vm=None):
     """
     Waits for messages to appear on the console, while logging the content
 
@@ -125,7 +127,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None):
     :param success_message: if this message appears, test succeeds
     :param failure_message: if this message appears, test fails
     """
-    _console_interaction(test, success_message, failure_message, None)
+    _console_interaction(test, success_message, failure_message, None, vm=vm)
 
 def exec_command_and_wait_for_pattern(test, command,
                                       success_message, failure_message=None):

[PATCH 2/9] tests/acceptance: add base class record/replay kernel tests

[Pavel Dovgalyuk]

This patch adds a base for testing kernel boot recording and replaying.
Each test has the phase of recording and phase of replaying.
Virtual machines just boot the kernel and do not interact with
the network.
Structure and image links for the tests are borrowed from boot_linux_console.py
Testing controls the message pattern at the end of the kernel
boot for both record and replay modes. In replay mode QEMU is also
intended to finish the execution automatically.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 MAINTAINERS                       |    1 
 tests/acceptance/replay_kernel.py |   80 +++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+)
 create mode 100644 tests/acceptance/replay_kernel.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 47ef3139e6..e9a9ce4f66 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2497,6 +2497,7 @@ F: net/filter-replay.c
 F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
+F: tests/acceptance/replay_kernel.py
 
 IOVA Tree
 M: Peter Xu <peterx@redhat.com>
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
new file mode 100644
index 0000000000..3208179789
--- /dev/null
+++ b/tests/acceptance/replay_kernel.py
@@ -0,0 +1,80 @@
+# Record/replay test that boots a Linux kernel
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import gzip
+
+from avocado_qemu import Test
+from avocado_qemu import wait_for_console_pattern
+from avocado.utils import process
+from avocado.utils import archive
+
+class ReplayKernel(Test):
+    """
+    Boots a Linux kernel in record mode and checks that the console
+    is operational and the kernel command line is properly passed
+    from QEMU to the kernel.
+    Then replays the same scenario and verifies, that QEMU correctly
+    terminates.
+    """
+
+    timeout = 90
+
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
+    def wait_for_console_pattern(self, success_message, vm):
+        wait_for_console_pattern(self, success_message,
+                                 failure_message='Kernel panic - not syncing',
+                                 vm=vm)
+
+    def extract_from_deb(self, deb, path):
+        """
+        Extracts a file from a deb package into the test workdir
+
+        :param deb: path to the deb archive
+        :param path: path within the deb archive of the file to be extracted
+        :returns: path of the extracted file
+        """
+        cwd = os.getcwd()
+        os.chdir(self.workdir)
+        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
+        process.run("ar x %s %s" % (deb, file_path))
+        archive.extract(file_path, self.workdir)
+        os.chdir(cwd)
+        # Return complete path to extracted file.  Because callers to
+        # extract_from_deb() specify 'path' with a leading slash, it is
+        # necessary to use os.path.relpath() as otherwise os.path.join()
+        # interprets it as an absolute path and drops the self.workdir part.
+        return os.path.normpath(os.path.join(self.workdir,
+                                             os.path.relpath(path, '/')))
+
+    def run_vm(self, kernel_path, kernel_command_line, console_pattern, record, shift, args):
+        vm = self.get_vm()
+        vm.set_console()
+        if record:
+            mode = 'record'
+        else:
+            mode = 'replay'
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, os.path.join(self.workdir, 'replay.bin')),
+                    '-kernel', kernel_path,
+                    '-append', kernel_command_line,
+                    '-net', 'none',
+                    *args)
+        vm.launch()
+        self.wait_for_console_pattern(console_pattern, vm)
+        if record:
+            vm.shutdown()
+        else:
+            vm.wait()
+
+    def run_rr(self, kernel_path, kernel_command_line, console_pattern, shift=7, args=()):
+        self.run_vm(kernel_path, kernel_command_line, console_pattern, True, shift, args)
+        self.run_vm(kernel_path, kernel_command_line, console_pattern, False, shift, args)

[PATCH 3/9] tests/acceptance: add kernel record/replay test for x86_64

[Pavel Dovgalyuk]

This patch adds a test for record/replay an execution of x86_64 machine.
Execution scenario includes simple kernel boot, which allows testing
basic hardware interaction in RR mode.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 3208179789..00abe1e872 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -78,3 +78,19 @@ class ReplayKernel(Test):
     def run_rr(self, kernel_path, kernel_command_line, console_pattern, shift=7, args=()):
         self.run_vm(kernel_path, kernel_command_line, console_pattern, True, shift, args)
         self.run_vm(kernel_path, kernel_command_line, console_pattern, False, shift, args)
+
+    def test_x86_64_pc(self):
+        """
+        :avocado: tags=arch:x86_64
+        :avocado: tags=machine:pc
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)

[PATCH 4/9] tests/acceptance: add record/replay test for aarch64

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for aarch64 platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 00abe1e872..1d2cfebc85 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -94,3 +94,21 @@ class ReplayKernel(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def test_aarch64_virt(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+            args=('-cpu', 'cortex-a53'))

[PATCH 5/9] tests/acceptance: add record/replay test for arm

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for two different arm platforms.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 1d2cfebc85..41c8ca3c99 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -112,3 +112,49 @@ class ReplayKernel(Test):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern,
             args=('-cpu', 'cortex-a53'))
+
+    def test_arm_virt(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:virt
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
+                      '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
+                      '/vmlinuz')
+        kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def test_arm_cubieboard_initrd(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:cubieboard
+        """
+        deb_url = ('https://apt.armbian.com/pool/main/l/'
+                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
+        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinuz-4.20.7-sunxi')
+        dtb_path = '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
+        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
+                      '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
+                      'arm/rootfs-armv5.cpio.gz')
+        initrd_hash = '2b50f1873e113523967806f4da2afe385462ff9b'
+        initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
+        initrd_path = os.path.join(self.workdir, 'rootfs.cpio')
+        archive.gzip_uncompress(initrd_path_gz, initrd_path)
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyS0,115200 '
+                               'usbcore.nousb '
+                               'panic=-1 noreboot')
+        console_pattern = 'Boot successful.'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+            args=('-dtb', dtb_path, '-initrd', initrd_path, '-no-reboot'))

[PATCH 6/9] tests/acceptance: add record/replay test for ppc64

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for ppc64 platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 41c8ca3c99..9931f87bd5 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -158,3 +158,18 @@ class ReplayKernel(Test):
         console_pattern = 'Boot successful.'
         self.run_rr(kernel_path, kernel_command_line, console_pattern,
             args=('-dtb', dtb_path, '-initrd', initrd_path, '-no-reboot'))
+
+    def test_ppc64_pseries(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+                      '/fedora-secondary/releases/29/Everything/ppc64le/os'
+                      '/ppc/ppc64/vmlinuz')
+        kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)

[PATCH 7/9] tests/acceptance: add record/replay test for m68k

[Pavel Dovgalyuk]

This patch adds a test for record/replay of the kernel
image boot for m68k platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 9931f87bd5..4c786b1565 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -173,3 +173,21 @@ class ReplayKernel(Test):
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def test_m68k_q800(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:q800
+        """
+        deb_url = ('https://snapshot.debian.org/archive/debian-ports'
+                   '/20191021T083923Z/pool-m68k/main'
+                   '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
+        deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-5.3.0-1-m68k')
+
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyS0 vga=off')
+        console_pattern = 'No filesystem could mount root'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern)

[PATCH 8/9] tests/acceptance: record/replay tests with advcal images

[Pavel Dovgalyuk]

This patch adds more record/replay tests with kernel images.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   80 +++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 4c786b1565..3849db7f3a 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -191,3 +191,83 @@ class ReplayKernel(Test):
                                'console=ttyS0 vga=off')
         console_pattern = 'No filesystem could mount root'
         self.run_rr(kernel_path, kernel_command_line, console_pattern)
+
+    def do_test_advcal_2018(self, day, tar_hash, kernel_name, args=()):
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day' + day + '.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        archive.extract(file_path, self.workdir)
+
+        kernel_path = self.workdir + '/day' + day + '/' + kernel_name
+        kernel_command_line = ''
+        console_pattern = 'QEMU advent calendar'
+        self.run_rr(kernel_path, kernel_command_line, console_pattern,
+            args=args)
+
+    def test_arm_vexpressa9(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:vexpress-a9
+        """
+        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
+        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage',
+            ('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb'))
+
+    def test_m68k_mcf5208evb(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:mcf5208evb
+        """
+        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
+        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
+
+    def test_microblaze_s3adsp1800(self):
+        """
+        :avocado: tags=arch:microblaze
+        :avocado: tags=machine:petalogix-s3adsp1800
+        """
+        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
+        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
+
+    def test_ppc64_e500(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:ppce500
+        """
+        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
+        self.do_test_advcal_2018('19', tar_hash, 'uImage', ('-cpu', 'e5500'))
+
+    def test_ppc_g3beige(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:g3beige
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf',
+            ('-M', 'graphics=off'))
+
+    def test_ppc_mac99(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:mac99
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf',
+            ('-M', 'graphics=off'))
+
+    def test_sparc_ss20(self):
+        """
+        :avocado: tags=arch:sparc
+        :avocado: tags=machine:SS-20
+        """
+        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
+        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
+
+    def test_xtensa_lx60(self):
+        """
+        :avocado: tags=arch:xtensa
+        :avocado: tags=machine:lx60
+        """
+        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
+        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf',
+            ('-cpu', 'dc233c'))

[PATCH 9/9] tests/acceptance: Linux boot test for record/replay

[Pavel Dovgalyuk]

This patch adds a test for record/replay, which boots Linux
image from the disk and interacts with the network.
The idea and code of this test is borrowed from boot_linux.py
However, currently record/replay works only for x86_64,
therefore other tests were excluded.

Each test consists of the following phases:
 - downloading the disk image
 - recording the execution
 - replaying the execution

Replay does not validates the output, but waits until QEMU
finishes the execution. This is reasonable, because
QEMU usually hangs when replay goes wrong.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 MAINTAINERS                      |    1 
 tests/acceptance/replay_linux.py |  140 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
 create mode 100644 tests/acceptance/replay_linux.py

diff --git a/MAINTAINERS b/MAINTAINERS
index e9a9ce4f66..97f066a9b2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
 F: tests/acceptance/replay_kernel.py
+F: tests/acceptance/replay_linux.py
 
 IOVA Tree
 M: Peter Xu <peterx@redhat.com>
diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
new file mode 100644
index 0000000000..08eedb23ef
--- /dev/null
+++ b/tests/acceptance/replay_linux.py
@@ -0,0 +1,140 @@
+# Record/replay test that boots a complete Linux system via a cloud image
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado_qemu import Test, BUILD_DIR
+
+from avocado.utils import cloudinit
+from avocado.utils import network
+from avocado.utils import vmimage
+from avocado.utils import datadrainer
+from avocado.utils.path import find_command
+
+class ReplayLinux(Test):
+    """
+    Boots a Linux system, checking for a successful initialization
+    """
+
+    timeout = 1800
+    chksum = None
+    hdd = 'ide-hd'
+    cd = 'ide-cd'
+    bus = ''
+
+    def setUp(self):
+        super(ReplayLinux, self).setUp()
+        self.prepare_boot()
+        self.prepare_cloudinit()
+
+    def vm_add_disk(self, vm, path, id, device):
+        bus_string = ''
+        if self.bus != '':
+            bus_string = ',bus=%s.%d' % (self.bus, id,)
+        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
+        vm.add_args('-drive', 'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
+        vm.add_args('-device', '%s,drive=disk%s-rr%s' % (device, id, bus_string))
+
+    def prepare_boot(self):
+        self.log.debug('Looking for and selecting a qemu-img binary to be '
+                       'used to create the bootable snapshot image')
+        # If qemu-img has been built, use it, otherwise the system wide one
+        # will be used.  If none is available, the test will cancel.
+        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
+        if not os.path.exists(qemu_img):
+            qemu_img = find_command('qemu-img', False)
+        if qemu_img is False:
+            self.cancel('Could not find "qemu-img", which is required to '
+                        'create the bootable image')
+        vmimage.QEMU_IMG = qemu_img
+
+        self.log.info('Downloading/preparing boot image')
+        # Fedora 31 only provides ppc64le images
+        image_arch = self.arch
+        if image_arch == 'ppc64':
+            image_arch = 'ppc64le'
+        try:
+            self.boot = vmimage.get(
+                'fedora', arch=image_arch, version='31',
+                checksum=self.chksum,
+                algorithm='sha256',
+                cache_dir=self.cache_dirs[0],
+                snapshot_dir=self.workdir)
+        except:
+            self.cancel('Failed to download/prepare boot image')
+
+    def prepare_cloudinit(self):
+        self.log.info('Preparing cloudinit image')
+        try:
+            self.cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
+            self.phone_home_port = network.find_free_port()
+            cloudinit.iso(self.cloudinit_iso, self.name,
+                          username='root',
+                          password='password',
+                          # QEMU's hard coded usermode router address
+                          phone_home_host='10.0.2.2',
+                          phone_home_port=self.phone_home_port)
+        except Exception:
+            self.cancel('Failed to prepared cloudinit image')
+
+    def launch_and_wait(self, record, args, shift):
+        vm = self.get_vm()
+        vm.add_args('-smp', '1')
+        vm.add_args('-m', '1024')
+        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
+        vm.add_args(*args)
+        self.vm_add_disk(vm, self.boot.path, 0, self.hdd)
+        self.vm_add_disk(vm, self.cloudinit_iso, 1, self.cd)
+        if record:
+            mode = 'record'
+        else:
+            mode = 'replay'
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, os.path.join(self.workdir, 'replay.bin')))
+
+        vm.set_console()
+        vm.launch()
+        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
+                                                 logger=self.log.getChild('console'),
+                                                 stop_check=(lambda : not vm.is_running()))
+        console_drainer.start()
+        if record:
+            self.log.info('VM launched, waiting for boot confirmation from guest')
+            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
+            vm.shutdown()
+        else:
+            self.log.info('VM launched, waiting the recorded execution to be replayed')
+            vm.wait()
+
+    def run_rr(self, args=(), shift=7):
+        self.launch_and_wait(True, args, shift)
+        self.launch_and_wait(False, args, shift)
+
+class ReplayLinuxX8664(ReplayLinux):
+    """
+    :avocado: tags=arch:x86_64
+    """
+    bus = 'ide'
+
+    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
+
+    def test_pc_i440fx(self):
+        """
+        :avocado: tags=machine:pc
+        :avocado: tags=accel:tcg
+        """
+        self.run_rr(shift=1)
+
+    def test_pc_q35(self):
+        """
+        :avocado: tags=machine:q35
+        :avocado: tags=accel:tcg
+        """
+        self.run_rr(shift=3)

Re: [PATCH 0/9] Record/replay acceptance tests

[no-reply]

Patchew URL: https://patchew.org/QEMU/159040554265.2615.8993443700754452381.stgit@pasha-ThinkPad-X280/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 159040554265.2615.8993443700754452381.stgit@pasha-ThinkPad-X280
Subject: [PATCH 0/9] Record/replay acceptance tests
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20200525134101.24740-1-kraxel@redhat.com -> patchew/20200525134101.24740-1-kraxel@redhat.com
Switched to a new branch 'test'
a859b96 tests/acceptance: Linux boot test for record/replay
14e5ce4 tests/acceptance: record/replay tests with advcal images
5cdf4d3 tests/acceptance: add record/replay test for m68k
0dc5c81 tests/acceptance: add record/replay test for ppc64
ec0d080 tests/acceptance: add record/replay test for arm
7a0dd72 tests/acceptance: add record/replay test for aarch64
3025aed tests/acceptance: add kernel record/replay test for x86_64
9971ff6 tests/acceptance: add base class record/replay kernel tests
2042d8d tests/acceptance: allow console interaction with specific VMs

=== OUTPUT BEGIN ===
1/9 Checking commit 2042d8dddac8 (tests/acceptance: allow console interaction with specific VMs)
WARNING: line over 80 characters
#43: FILE: tests/acceptance/avocado_qemu/__init__.py:120:
+def wait_for_console_pattern(test, success_message, failure_message=None, vm=None):

total: 0 errors, 1 warnings, 34 lines checked

Patch 1/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
2/9 Checking commit 9971ff6fbc70 (tests/acceptance: add base class record/replay kernel tests)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#32: 
new file mode 100644

ERROR: line over 90 characters
#94: FILE: tests/acceptance/replay_kernel.py:58:
+    def run_vm(self, kernel_path, kernel_command_line, console_pattern, record, shift, args):

WARNING: line over 80 characters
#114: FILE: tests/acceptance/replay_kernel.py:78:
+    def run_rr(self, kernel_path, kernel_command_line, console_pattern, shift=7, args=()):

WARNING: line over 80 characters
#115: FILE: tests/acceptance/replay_kernel.py:79:
+        self.run_vm(kernel_path, kernel_command_line, console_pattern, True, shift, args)

WARNING: line over 80 characters
#116: FILE: tests/acceptance/replay_kernel.py:80:
+        self.run_vm(kernel_path, kernel_command_line, console_pattern, False, shift, args)

total: 1 errors, 4 warnings, 87 lines checked

Patch 2/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

3/9 Checking commit 3025aedc97d7 (tests/acceptance: add kernel record/replay test for x86_64)
4/9 Checking commit 7a0dd72514f7 (tests/acceptance: add record/replay test for aarch64)
5/9 Checking commit ec0d0808a993 (tests/acceptance: add record/replay test for arm)
6/9 Checking commit 0dc5c81830b7 (tests/acceptance: add record/replay test for ppc64)
7/9 Checking commit 5cdf4d3ab6fa (tests/acceptance: add record/replay test for m68k)
8/9 Checking commit 14e5ce4abb6e (tests/acceptance: record/replay tests with advcal images)
9/9 Checking commit a859b96e665d (tests/acceptance: Linux boot test for record/replay)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#38: 
new file mode 100644

ERROR: line over 90 characters
#84: FILE: tests/acceptance/replay_linux.py:42:
+        vm.add_args('-drive', 'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))

WARNING: line over 80 characters
#85: FILE: tests/acceptance/replay_linux.py:43:
+        vm.add_args('-device', '%s,drive=disk%s-rr%s' % (device, id, bus_string))

WARNING: line over 80 characters
#147: FILE: tests/acceptance/replay_linux.py:105:
+                                                 logger=self.log.getChild('console'),

ERROR: line over 90 characters
#148: FILE: tests/acceptance/replay_linux.py:106:
+                                                 stop_check=(lambda : not vm.is_running()))

WARNING: line over 80 characters
#151: FILE: tests/acceptance/replay_linux.py:109:
+            self.log.info('VM launched, waiting for boot confirmation from guest')

WARNING: line over 80 characters
#152: FILE: tests/acceptance/replay_linux.py:110:
+            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)

WARNING: line over 80 characters
#155: FILE: tests/acceptance/replay_linux.py:113:
+            self.log.info('VM launched, waiting the recorded execution to be replayed')

total: 2 errors, 6 warnings, 147 lines checked

Patch 9/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/159040554265.2615.8993443700754452381.stgit@pasha-ThinkPad-X280/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH 9/9] tests/acceptance: Linux boot test for record/replay

[Willian Rampazzo]

On Mon, May 25, 2020 at 8:30 AM Pavel Dovgalyuk
<Pavel.Dovgaluk@gmail.com> wrote:
>
> This patch adds a test for record/replay, which boots Linux
> image from the disk and interacts with the network.
> The idea and code of this test is borrowed from boot_linux.py
> However, currently record/replay works only for x86_64,
> therefore other tests were excluded.
>
> Each test consists of the following phases:
>  - downloading the disk image
>  - recording the execution
>  - replaying the execution
>
> Replay does not validates the output, but waits until QEMU
> finishes the execution. This is reasonable, because
> QEMU usually hangs when replay goes wrong.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  MAINTAINERS                      |    1
>  tests/acceptance/replay_linux.py |  140 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 141 insertions(+)
>  create mode 100644 tests/acceptance/replay_linux.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e9a9ce4f66..97f066a9b2 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
>  F: docs/replay.txt
>  F: stubs/replay.c
>  F: tests/acceptance/replay_kernel.py
> +F: tests/acceptance/replay_linux.py
>
>  IOVA Tree
>  M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
> new file mode 100644
> index 0000000000..08eedb23ef
> --- /dev/null
> +++ b/tests/acceptance/replay_linux.py
> @@ -0,0 +1,140 @@
> +# Record/replay test that boots a complete Linux system via a cloud image
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado_qemu import Test, BUILD_DIR
> +
> +from avocado.utils import cloudinit
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import datadrainer
> +from avocado.utils.path import find_command
> +
> +class ReplayLinux(Test):

There is no need to copy/paste the whole BootLinux class. You can
inherit from it and re-implement the lauch_and_wait method. Inheriting
avoids duplication of code.

> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 1800
> +    chksum = None
> +    hdd = 'ide-hd'
> +    cd = 'ide-cd'
> +    bus = ''
> +
> +    def setUp(self):
> +        super(ReplayLinux, self).setUp()
> +        self.prepare_boot()
> +        self.prepare_cloudinit()
> +
> +    def vm_add_disk(self, vm, path, id, device):
> +        bus_string = ''
> +        if self.bus != '':
> +            bus_string = ',bus=%s.%d' % (self.bus, id,)
> +        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
> +        vm.add_args('-drive', 'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
> +        vm.add_args('-device', '%s,drive=disk%s-rr%s' % (device, id, bus_string))
> +
> +    def prepare_boot(self):
> +        self.log.debug('Looking for and selecting a qemu-img binary to be '
> +                       'used to create the bootable snapshot image')
> +        # If qemu-img has been built, use it, otherwise the system wide one
> +        # will be used.  If none is available, the test will cancel.
> +        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
> +        if not os.path.exists(qemu_img):
> +            qemu_img = find_command('qemu-img', False)
> +        if qemu_img is False:
> +            self.cancel('Could not find "qemu-img", which is required to '
> +                        'create the bootable image')
> +        vmimage.QEMU_IMG = qemu_img
> +
> +        self.log.info('Downloading/preparing boot image')
> +        # Fedora 31 only provides ppc64le images
> +        image_arch = self.arch
> +        if image_arch == 'ppc64':
> +            image_arch = 'ppc64le'
> +        try:
> +            self.boot = vmimage.get(
> +                'fedora', arch=image_arch, version='31',
> +                checksum=self.chksum,
> +                algorithm='sha256',
> +                cache_dir=self.cache_dirs[0],
> +                snapshot_dir=self.workdir)
> +        except:
> +            self.cancel('Failed to download/prepare boot image')
> +
> +    def prepare_cloudinit(self):
> +        self.log.info('Preparing cloudinit image')
> +        try:
> +            self.cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
> +            self.phone_home_port = network.find_free_port()
> +            cloudinit.iso(self.cloudinit_iso, self.name,
> +                          username='root',
> +                          password='password',
> +                          # QEMU's hard coded usermode router address
> +                          phone_home_host='10.0.2.2',
> +                          phone_home_port=self.phone_home_port)
> +        except Exception:
> +            self.cancel('Failed to prepared cloudinit image')
> +
> +    def launch_and_wait(self, record, args, shift):
> +        vm = self.get_vm()
> +        vm.add_args('-smp', '1')
> +        vm.add_args('-m', '1024')
> +        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
> +        vm.add_args(*args)
> +        self.vm_add_disk(vm, self.boot.path, 0, self.hdd)
> +        self.vm_add_disk(vm, self.cloudinit_iso, 1, self.cd)
> +        if record:
> +            mode = 'record'
> +        else:
> +            mode = 'replay'
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, os.path.join(self.workdir, 'replay.bin')))
> +
> +        vm.set_console()
> +        vm.launch()
> +        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
> +                                                 logger=self.log.getChild('console'),
> +                                                 stop_check=(lambda : not vm.is_running()))
> +        console_drainer.start()
> +        if record:
> +            self.log.info('VM launched, waiting for boot confirmation from guest')
> +            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
> +            vm.shutdown()
> +        else:
> +            self.log.info('VM launched, waiting the recorded execution to be replayed')
> +            vm.wait()
> +
> +    def run_rr(self, args=(), shift=7):
> +        self.launch_and_wait(True, args, shift)
> +        self.launch_and_wait(False, args, shift)
> +
> +class ReplayLinuxX8664(ReplayLinux):
> +    """
> +    :avocado: tags=arch:x86_64
> +    """
> +    bus = 'ide'
> +
> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
> +
> +    def test_pc_i440fx(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=1)
> +
> +    def test_pc_q35(self):
> +        """
> +        :avocado: tags=machine:q35
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=3)
>
>

Re: [PATCH 8/9] tests/acceptance: record/replay tests with advcal images

[Willian Rampazzo]

On Mon, May 25, 2020 at 8:28 AM Pavel Dovgalyuk
<Pavel.Dovgaluk@gmail.com> wrote:
>
> This patch adds more record/replay tests with kernel images.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  tests/acceptance/replay_kernel.py |   80 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 80 insertions(+)
>
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 4c786b1565..3849db7f3a 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -191,3 +191,83 @@ class ReplayKernel(Test):
>                                 'console=ttyS0 vga=off')
>          console_pattern = 'No filesystem could mount root'
>          self.run_rr(kernel_path, kernel_command_line, console_pattern)
> +
> +    def do_test_advcal_2018(self, day, tar_hash, kernel_name, args=()):
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day' + day + '.tar.xz')

Making the file name flexible helps in the code organization. Still,
in this specific case, due to limitations in the Avocado Asset parser,
this construction is ignored in an `avocado assets fetch <file>`
command. It results in the file being downloaded during the test run
and the time spent to download the files being accounted for in the
test time, and if the files are not saved in the Travis cache after
this test runs, it also means the files will be downloaded again every
time it runs.

The straight forward solution to that is having the complete URL
described and fetched for each test.

> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        archive.extract(file_path, self.workdir)
> +
> +        kernel_path = self.workdir + '/day' + day + '/' + kernel_name
> +        kernel_command_line = ''
> +        console_pattern = 'QEMU advent calendar'
> +        self.run_rr(kernel_path, kernel_command_line, console_pattern,
> +            args=args)
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage',
> +            ('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb'))
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        self.do_test_advcal_2018('19', tar_hash, 'uImage', ('-cpu', 'e5500'))
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf',
> +            ('-M', 'graphics=off'))
> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf',
> +            ('-M', 'graphics=off'))
> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf',
> +            ('-cpu', 'dc233c'))
>
>

Re: [PATCH 9/9] tests/acceptance: Linux boot test for record/replay

[Willian Rampazzo]

On Mon, May 25, 2020 at 8:30 AM Pavel Dovgalyuk
<Pavel.Dovgaluk@gmail.com> wrote:
>
> This patch adds a test for record/replay, which boots Linux
> image from the disk and interacts with the network.
> The idea and code of this test is borrowed from boot_linux.py
> However, currently record/replay works only for x86_64,
> therefore other tests were excluded.
>
> Each test consists of the following phases:
>  - downloading the disk image
>  - recording the execution
>  - replaying the execution
>
> Replay does not validates the output, but waits until QEMU
> finishes the execution. This is reasonable, because
> QEMU usually hangs when replay goes wrong.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  MAINTAINERS                      |    1
>  tests/acceptance/replay_linux.py |  140 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 141 insertions(+)
>  create mode 100644 tests/acceptance/replay_linux.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e9a9ce4f66..97f066a9b2 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2498,6 +2498,7 @@ F: include/sysemu/replay.h
>  F: docs/replay.txt
>  F: stubs/replay.c
>  F: tests/acceptance/replay_kernel.py
> +F: tests/acceptance/replay_linux.py
>
>  IOVA Tree
>  M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
> new file mode 100644
> index 0000000000..08eedb23ef
> --- /dev/null
> +++ b/tests/acceptance/replay_linux.py
> @@ -0,0 +1,140 @@
> +# Record/replay test that boots a complete Linux system via a cloud image
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado_qemu import Test, BUILD_DIR
> +
> +from avocado.utils import cloudinit
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import datadrainer
> +from avocado.utils.path import find_command
> +
> +class ReplayLinux(Test):
> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 1800
> +    chksum = None
> +    hdd = 'ide-hd'
> +    cd = 'ide-cd'
> +    bus = ''
> +
> +    def setUp(self):
> +        super(ReplayLinux, self).setUp()
> +        self.prepare_boot()
> +        self.prepare_cloudinit()
> +
> +    def vm_add_disk(self, vm, path, id, device):
> +        bus_string = ''
> +        if self.bus != '':
> +            bus_string = ',bus=%s.%d' % (self.bus, id,)
> +        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
> +        vm.add_args('-drive', 'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
> +        vm.add_args('-device', '%s,drive=disk%s-rr%s' % (device, id, bus_string))
> +
> +    def prepare_boot(self):
> +        self.log.debug('Looking for and selecting a qemu-img binary to be '
> +                       'used to create the bootable snapshot image')
> +        # If qemu-img has been built, use it, otherwise the system wide one
> +        # will be used.  If none is available, the test will cancel.
> +        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
> +        if not os.path.exists(qemu_img):
> +            qemu_img = find_command('qemu-img', False)
> +        if qemu_img is False:
> +            self.cancel('Could not find "qemu-img", which is required to '
> +                        'create the bootable image')
> +        vmimage.QEMU_IMG = qemu_img
> +
> +        self.log.info('Downloading/preparing boot image')
> +        # Fedora 31 only provides ppc64le images
> +        image_arch = self.arch
> +        if image_arch == 'ppc64':
> +            image_arch = 'ppc64le'
> +        try:
> +            self.boot = vmimage.get(
> +                'fedora', arch=image_arch, version='31',
> +                checksum=self.chksum,
> +                algorithm='sha256',
> +                cache_dir=self.cache_dirs[0],
> +                snapshot_dir=self.workdir)
> +        except:
> +            self.cancel('Failed to download/prepare boot image')
> +
> +    def prepare_cloudinit(self):
> +        self.log.info('Preparing cloudinit image')
> +        try:
> +            self.cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
> +            self.phone_home_port = network.find_free_port()
> +            cloudinit.iso(self.cloudinit_iso, self.name,
> +                          username='root',
> +                          password='password',
> +                          # QEMU's hard coded usermode router address
> +                          phone_home_host='10.0.2.2',
> +                          phone_home_port=self.phone_home_port)
> +        except Exception:
> +            self.cancel('Failed to prepared cloudinit image')
> +
> +    def launch_and_wait(self, record, args, shift):
> +        vm = self.get_vm()
> +        vm.add_args('-smp', '1')
> +        vm.add_args('-m', '1024')
> +        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
> +        vm.add_args(*args)
> +        self.vm_add_disk(vm, self.boot.path, 0, self.hdd)
> +        self.vm_add_disk(vm, self.cloudinit_iso, 1, self.cd)
> +        if record:
> +            mode = 'record'
> +        else:
> +            mode = 'replay'
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, os.path.join(self.workdir, 'replay.bin')))
> +
> +        vm.set_console()
> +        vm.launch()
> +        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
> +                                                 logger=self.log.getChild('console'),
> +                                                 stop_check=(lambda : not vm.is_running()))
> +        console_drainer.start()
> +        if record:
> +            self.log.info('VM launched, waiting for boot confirmation from guest')
> +            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
> +            vm.shutdown()
> +        else:
> +            self.log.info('VM launched, waiting the recorded execution to be replayed')
> +            vm.wait()
> +
> +    def run_rr(self, args=(), shift=7):

I did not spot this in my previous e-mail, sorry about that. Here you
can use the default value of args as None and handle it in the
launch_and_wait method. It is usually recommended to use a None value
for default arguments in Python instead of an empty structure.

> +        self.launch_and_wait(True, args, shift)
> +        self.launch_and_wait(False, args, shift)
> +
> +class ReplayLinuxX8664(ReplayLinux):
> +    """
> +    :avocado: tags=arch:x86_64
> +    """
> +    bus = 'ide'
> +
> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
> +
> +    def test_pc_i440fx(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=1)
> +
> +    def test_pc_q35(self):
> +        """
> +        :avocado: tags=machine:q35
> +        :avocado: tags=accel:tcg
> +        """
> +        self.run_rr(shift=3)
>
>

Re: [PATCH 2/9] tests/acceptance: add base class record/replay kernel tests

[Willian Rampazzo]

On Mon, May 25, 2020 at 8:22 AM Pavel Dovgalyuk
<Pavel.Dovgaluk@gmail.com> wrote:
>
> This patch adds a base for testing kernel boot recording and replaying.
> Each test has the phase of recording and phase of replaying.
> Virtual machines just boot the kernel and do not interact with
> the network.
> Structure and image links for the tests are borrowed from boot_linux_console.py
> Testing controls the message pattern at the end of the kernel
> boot for both record and replay modes. In replay mode QEMU is also
> intended to finish the execution automatically.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  MAINTAINERS                       |    1
>  tests/acceptance/replay_kernel.py |   80 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 81 insertions(+)
>  create mode 100644 tests/acceptance/replay_kernel.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 47ef3139e6..e9a9ce4f66 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2497,6 +2497,7 @@ F: net/filter-replay.c
>  F: include/sysemu/replay.h
>  F: docs/replay.txt
>  F: stubs/replay.c
> +F: tests/acceptance/replay_kernel.py
>
>  IOVA Tree
>  M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> new file mode 100644
> index 0000000000..3208179789
> --- /dev/null
> +++ b/tests/acceptance/replay_kernel.py
> @@ -0,0 +1,80 @@
> +# Record/replay test that boots a Linux kernel
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +import gzip
> +
> +from avocado_qemu import Test
> +from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import process
> +from avocado.utils import archive
> +
> +class ReplayKernel(Test):
> +    """
> +    Boots a Linux kernel in record mode and checks that the console
> +    is operational and the kernel command line is properly passed
> +    from QEMU to the kernel.
> +    Then replays the same scenario and verifies, that QEMU correctly
> +    terminates.
> +    """

The best to do here, IMHO, is to split the BootLinuxConsole class on
boot_linux_console.py into two classes, one with the necessary
utilities inheriting from Test and the second with the tests itself,
inheriting from the first. After that you can also inherit from the
first class in the boot_linux_console.py here and avoid code
duplication.

> +
> +    timeout = 90
> +
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> +
> +    def wait_for_console_pattern(self, success_message, vm):
> +        wait_for_console_pattern(self, success_message,
> +                                 failure_message='Kernel panic - not syncing',
> +                                 vm=vm)
> +
> +    def extract_from_deb(self, deb, path):
> +        """
> +        Extracts a file from a deb package into the test workdir
> +
> +        :param deb: path to the deb archive
> +        :param path: path within the deb archive of the file to be extracted
> +        :returns: path of the extracted file
> +        """
> +        cwd = os.getcwd()
> +        os.chdir(self.workdir)
> +        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
> +        process.run("ar x %s %s" % (deb, file_path))
> +        archive.extract(file_path, self.workdir)
> +        os.chdir(cwd)
> +        # Return complete path to extracted file.  Because callers to
> +        # extract_from_deb() specify 'path' with a leading slash, it is
> +        # necessary to use os.path.relpath() as otherwise os.path.join()
> +        # interprets it as an absolute path and drops the self.workdir part.
> +        return os.path.normpath(os.path.join(self.workdir,
> +                                             os.path.relpath(path, '/')))
> +
> +    def run_vm(self, kernel_path, kernel_command_line, console_pattern, record, shift, args):
> +        vm = self.get_vm()
> +        vm.set_console()
> +        if record:
> +            mode = 'record'
> +        else:
> +            mode = 'replay'
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, os.path.join(self.workdir, 'replay.bin')),
> +                    '-kernel', kernel_path,
> +                    '-append', kernel_command_line,
> +                    '-net', 'none',
> +                    *args)
> +        vm.launch()
> +        self.wait_for_console_pattern(console_pattern, vm)
> +        if record:
> +            vm.shutdown()
> +        else:
> +            vm.wait()
> +
> +    def run_rr(self, kernel_path, kernel_command_line, console_pattern, shift=7, args=()):

Same comment from patch file 9, here you can use the default value of
args as None and handle it in the run_vm method. It is usually
recommended to use a None value for default arguments in Python
instead of an empty structure.

> +        self.run_vm(kernel_path, kernel_command_line, console_pattern, True, shift, args)
> +        self.run_vm(kernel_path, kernel_command_line, console_pattern, False, shift, args)
>
>

Re: [PATCH 1/9] tests/acceptance: allow console interaction with specific VMs

[Willian Rampazzo]

On Mon, May 25, 2020 at 8:20 AM Pavel Dovgalyuk
<Pavel.Dovgaluk@gmail.com> wrote:
>
> Console interaction in avocado scripts was possible only with single
> default VM.
> This patch modifies the function parameters to allow passing a specific
> VM as a parameter to interact with it.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  tests/acceptance/avocado_qemu/__init__.py |   12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index 59e7b4f763..0bbaa8d2a6 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -69,13 +69,15 @@ def pick_default_qemu_bin(arch=None):
>
>
>  def _console_interaction(test, success_message, failure_message,
> -                         send_string, keep_sending=False):
> +                         send_string, keep_sending=False, vm=None):
>      assert not keep_sending or send_string
> -    console = test.vm.console_socket.makefile()
> +    if vm is None:
> +        vm = test.vm
> +    console = vm.console_socket.makefile()
>      console_logger = logging.getLogger('console')
>      while True:
>          if send_string:
> -            test.vm.console_socket.sendall(send_string.encode())
> +            vm.console_socket.sendall(send_string.encode())
>              if not keep_sending:
>                  send_string = None # send only once
>          msg = console.readline().strip()
> @@ -115,7 +117,7 @@ def interrupt_interactive_console_until_pattern(test, success_message,
>      _console_interaction(test, success_message, failure_message,
>                           interrupt_string, True)
>
> -def wait_for_console_pattern(test, success_message, failure_message=None):
> +def wait_for_console_pattern(test, success_message, failure_message=None, vm=None):
>      """
>      Waits for messages to appear on the console, while logging the content
>
> @@ -125,7 +127,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None):
>      :param success_message: if this message appears, test succeeds
>      :param failure_message: if this message appears, test fails
>      """
> -    _console_interaction(test, success_message, failure_message, None)
> +    _console_interaction(test, success_message, failure_message, None, vm=vm)
>
>  def exec_command_and_wait_for_pattern(test, command,
>                                        success_message, failure_message=None):
>
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>