[PATCH 00/16] Acceptance Tests: use Avocado 91.0 features and other improvements

[PATCH 00/16] Acceptance Tests: use Avocado 91.0 features and other improvements

[Cleber Rosa]

This is a collection of patches for the Acceptance Tests to leverage
some of the features of Avocado 91.0.  With the Avocado version bump
by itself, there would be a change in the default "test runner"
implementation that Avocado uses, from the one simply known as
"runner" to the new one called "nrunner".

Among the changes from one implementation to the other, is the fact
that "nrunner" will run tests in parallel by default.  This is *not
yet* enabled by default on "make check-acceptance", but users can
choose to use simply by setting the "AVOCADO_RUNNER" variable, that
is:

  make AVOCADO_RUNNER=nrunner check-acceptance

If you are curious about the architectural differences of the nrunner,
please refer to:

  https://avocado-framework.readthedocs.io/en/91.0/guides/contributor/chapters/runners.html

One other noteworthy proposal is a convention to tag tests that either
have known issues, or that touch on QEMU features that have known
issues.  By tagging those tests accordingly, they will be
automatically excluded from the regular execution with "make
check-acceptance".

Finally, some updates to assets locations and some tests refactors and
cleanups.

Cleber Rosa (16):
  Acceptance Tests: bump Avocado requirement to 91.0
  Acceptance Tests: improve check-acceptance description
  Acceptance Tests: add mechanism for listing tests
  Acceptance Tests: keep track and disable tests with known issues
  Acceptance Tests: add standard clean up at test tearDown()
  Acceptance Tests: use extract from package from avocado.utils
  Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert
  acceptance/tests/vnc.py: use explicit syntax for enabling passwords
  tests/acceptance/boot_xen.py: merge base classes
  tests/acceptance/boot_xen.py: unify tags
  tests/acceptance/boot_xen.py: fetch kernel during test setUp()
  tests/acceptance/boot_xen.py: removed unused import
  tests/acceptance/boot_xen.py: use class attribute
  tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update
  tests/acceptance/ppc_prep_40p.py: clean up unused import
  tests/acceptance/ppc_prep_40p.py: unify tags

 docs/devel/testing.rst                    | 40 ++++++++++
 tests/Makefile.include                    | 15 +++-
 tests/acceptance/avocado_qemu/__init__.py |  1 +
 tests/acceptance/boot_linux_console.py    | 93 +++++++++--------------
 tests/acceptance/boot_xen.py              | 54 ++++---------
 tests/acceptance/machine_rx_gdbsim.py     |  3 +
 tests/acceptance/ppc_prep_40p.py          | 17 ++---
 tests/acceptance/replay_kernel.py         | 18 ++---
 tests/acceptance/tcg_plugins.py           |  2 +-
 tests/acceptance/vnc.py                   |  2 +-
 tests/requirements.txt                    |  2 +-
 11 files changed, 128 insertions(+), 119 deletions(-)

-- 
2.31.1

[PATCH 01/16] Acceptance Tests: bump Avocado requirement to 91.0

[Cleber Rosa]

Avocado 91.0, brings, among other changes, a switch to a new runner
implementation, known as "nrunner".  While my personal testing shows
that it's suitable for running the QEMU tests, there are some
considerations that should be addressed before the change.

For instance, the "nrunner" implementation will run tests in parallel
by default.  With more tests running, and resources shared, some tests
may fail with timeouts on some environments.

So, for now, let's bump the Avocado version to allow for the other
features to be used.  And the "nrunner" implementation to be used by
those that choose to do so.

More information can be found at the release notes at:
   https://avocado-framework.readthedocs.io/en/latest/releases/91_0.html

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst | 12 ++++++++++++
 tests/Makefile.include |  2 ++
 tests/requirements.txt |  2 +-
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..2818c58ff8 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -740,6 +740,18 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Running tests in parallel
+-------------------------
+
+The current ``make check-acceptance`` target continues to use the
+Avocado runner implementation simply known as "runner".  But, it's now
+possible to opt-in and choose the "nrunner" implementation, which,
+among other things, allows for parallel execution of tests:
+
+.. code::
+
+  make AVOCADO_RUNNER=nrunner check-acceptance
+
 Manual Installation
 -------------------
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..ac289a2e41 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -92,6 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
+AVOCADO_RUNNER=runner
 AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
@@ -128,6 +129,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --test-runner=$(AVOCADO_RUNNER) \
             --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
             $(AVOCADO_TAGS) \
             $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
diff --git a/tests/requirements.txt b/tests/requirements.txt
index a21b59b443..40af24c664 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==88.1
+avocado-framework==91.0
 pycdlib==1.11.0
-- 
2.31.1

[PATCH 02/16] Acceptance Tests: improve check-acceptance description

[Cleber Rosa]

The "check-acceptance" make rule won't necessarily run *all* available
tests, because it employs a filter based on the currently configured
targets.  This change in the description of the rule makes that
behavior extra clear.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index ac289a2e41..2c03256ae8 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-softfloat      Run FPU emulation tests"
 endif
-	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
+	@echo " $(MAKE) check-acceptance     Run acceptance (functional) tests for currently configured targets"
 	@echo
 	@echo " $(MAKE) check-report.tap     Generates an aggregated TAP test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
-- 
2.31.1

[PATCH 03/16] Acceptance Tests: add mechanism for listing tests

[Cleber Rosa]

It is helpful to know the tests that would be executed with a "make
check-acceptance" without executing them.  Let's introduce a "make
list-acceptance" rule for that purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 2c03256ae8..d1f90572a7 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -17,6 +17,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
 	@echo " $(MAKE) check-softfloat      Run FPU emulation tests"
 endif
 	@echo " $(MAKE) check-acceptance     Run acceptance (functional) tests for currently configured targets"
+	@echo " $(MAKE) check-acceptance     List acceptance (functional) tests for currently configured targets"
 	@echo
 	@echo " $(MAKE) check-report.tap     Generates an aggregated TAP test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
@@ -135,6 +136,13 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
+list-acceptance: check-venv
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado list \
+            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
+            $(AVOCADO_TAGS) tests/acceptance, \
+            "AVOCADO", "list tests/acceptance")
+
 # Consolidated targets
 
 .PHONY: check-block check check-clean get-vm-images
-- 
2.31.1

[PATCH 04/16] Acceptance Tests: keep track and disable tests with known issues

[Cleber Rosa]

This introduces a convention, under which tests with known open
issues, will be automatically excluded from execution via "make
check-acceptance".  It should by itself improve the reliability
of test jobs, even those with rare failures.

Those tests can still be run with:

  ./tests/venv/bin/avocado run $TEST_ID

With the test identification are obtainable via:

  ./tests/venv/bin/avocado list tests/acceptance

And for tests with known issues:

  ./tests/venv/bin/avocado list -t issue tests/acceptance

A list of tests that are enabled (that is, match the currently
configured targets, and do not have known issues) can be obtained
with:

  make list-acceptance

As a future improvement, a tool can be added to sync the "issue" tags
with issues that end up being resolved closed.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst                 | 28 ++++++++++++++++++++++++++
 tests/Makefile.include                 |  3 ++-
 tests/acceptance/boot_linux_console.py |  3 +++
 tests/acceptance/machine_rx_gdbsim.py  |  3 +++
 4 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 2818c58ff8..f93a9bed3f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1147,6 +1147,34 @@ variable as shown on the code snippet below to skip the test:
   def test(self):
       do_something()
 
+Disabling tests
+---------------
+
+If you want to have a test temporarily disabled (from ``make
+check-acceptance``), you can add an Avocado ``issue`` tag to the test.  Example::
+
+  class Feature(Test):
+
+      def test(self):
+        """
+        :avocado: tags=issue
+        """
+        do_something()
+
+Rather than simply adding an ``issue`` tag, it may be helpful to point
+to an actual issue on the project's issue tracker::
+
+  class Feature(Test):
+
+      def test(self):
+        """
+        Reason for disabling this test is documented and tracked at
+        https://gitlab.com/qemu-project/qemu/-/issues/9999
+
+        :avocado: tags=issue:9999
+        """
+        do_something()
+
 Uninstalling Avocado
 --------------------
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d1f90572a7..eb3e5d2aa6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,7 +94,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
 AVOCADO_RUNNER=runner
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+comma := ,
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=-issue$(comma)arch:%, $(filter %-softmmu,$(TARGETS)))
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 0a49c0e276..4c824bc161 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -485,6 +485,9 @@ def test_arm_raspi2_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:raspi2b
+
+        # https://gitlab.com/qemu-project/qemu/-/issues/636
+        :avocado: tags=issue:636
         """
         deb_url = ('http://archive.raspberrypi.org/debian/'
                    'pool/main/r/raspberrypi-firmware/'
diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
index 32b737b6d8..62893a9989 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -54,6 +54,9 @@ def test_linux_sash(self):
         :avocado: tags=arch:rx
         :avocado: tags=machine:gdbsim-r5f562n7
         :avocado: tags=endian:little
+
+        # https://gitlab.com/qemu-project/qemu/-/issues/507
+        :avocado: tags=issue:507
         """
         dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
         dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
-- 
2.31.1

[PATCH 05/16] Acceptance Tests: add standard clean up at test tearDown()

[Cleber Rosa]

The avocado.Test class, used as the basis of the avocado_qemu.Test
class, performs a clean of temporary directories up as part of its own
tearDown() implementation.

But the avocado_qemu.Test class is currently missing the same clean
up, as it implemented its own tearDown() method without resorting to
the upper class behavior.

This brings avocado_qemu.Test behavior in sync with the standard
avocado.Test behavior and prevents temporary directories from
cluttering the test results directory (unless instructed to do so with
Avocado's "--keep-tmp" option).

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 2c4fef3e14..1e807e2e55 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -276,6 +276,7 @@ def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
         self._sd = None
+        super(Test, self).tearDown()
 
     def fetch_asset(self, name,
                     asset_hash=None, algorithm=None,
-- 
2.31.1

[PATCH 06/16] Acceptance Tests: use extract from package from avocado.utils

[Cleber Rosa]

There's code in avocado.utils to handle package extractions in various
(currently rpm, and deb) formats.  Let's use it.

Reference: https://avocado-framework.readthedocs.io/en/91.0/api/utils/avocado.utils.software_manager.html#avocado.utils.software_manager.SoftwareManager.extract_from_package
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 84 ++++++++++----------------
 tests/acceptance/boot_xen.py           |  6 +-
 tests/acceptance/replay_kernel.py      | 12 ++--
 tests/acceptance/tcg_plugins.py        |  2 +-
 4 files changed, 41 insertions(+), 63 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 4c824bc161..80e3a2f7a6 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -23,6 +23,7 @@
 from avocado.utils import process
 from avocado.utils import archive
 from avocado.utils.path import find_command, CmdNotFoundError
+from avocado.utils.software_manager import SoftwareManager
 
 P7ZIP_AVAILABLE = True
 try:
@@ -54,42 +55,19 @@ def wait_for_console_pattern(self, success_message, vm=None):
                                  failure_message='Kernel panic - not syncing',
                                  vm=vm)
 
-    def extract_from_deb(self, deb, path):
+    def extract_from_pkg(self, pkg, path):
         """
-        Extracts a file from a deb package into the test workdir
+        Extracts a file from a DEB or RPM package into the test workdir
 
-        :param deb: path to the deb archive
-        :param path: path within the deb archive of the file to be extracted
+        :param pkg: path to the DEB or RPM package
+        :param path: path within the DEB or RPM archive of the file to
+                          be extracted
         :returns: path of the extracted file
         """
-        cwd = os.getcwd()
-        os.chdir(self.workdir)
-        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
-        process.run("ar x %s %s" % (deb, file_path))
-        archive.extract(file_path, self.workdir)
-        os.chdir(cwd)
-        # Return complete path to extracted file.  Because callers to
-        # extract_from_deb() specify 'path' with a leading slash, it is
-        # necessary to use os.path.relpath() as otherwise os.path.join()
-        # interprets it as an absolute path and drops the self.workdir part.
+        SoftwareManager.extract_from_package(pkg, self.workdir)
         return os.path.normpath(os.path.join(self.workdir,
                                              os.path.relpath(path, '/')))
 
-    def extract_from_rpm(self, rpm, path):
-        """
-        Extracts a file from an RPM package into the test workdir.
-
-        :param rpm: path to the rpm archive
-        :param path: path within the rpm archive of the file to be extracted
-                     needs to be a relative path (starting with './') because
-                     cpio(1), which is used to extract the file, expects that.
-        :returns: path of the extracted file
-        """
-        cwd = os.getcwd()
-        os.chdir(self.workdir)
-        process.run("rpm2cpio %s | cpio -id %s" % (rpm, path), shell=True)
-        os.chdir(cwd)
-        return os.path.normpath(os.path.join(self.workdir, path))
 
 class BootLinuxConsole(LinuxKernelTest):
     """
@@ -128,7 +106,7 @@ def test_mips_malta(self):
                    'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
         deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-2.6.32-5-4kc-malta')
 
         self.vm.set_console()
@@ -160,7 +138,7 @@ def test_mips64el_malta(self):
                    'linux-image-2.6.32-5-5kc-malta_2.6.32-48_mipsel.deb')
         deb_hash = '1aaec92083bf22fda31e0d27fa8d9a388e5fc3d5'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-2.6.32-5-5kc-malta')
 
         self.vm.set_console()
@@ -181,7 +159,7 @@ def test_mips64el_fuloong2e(self):
                    'linux-image-3.16.0-6-loongson-2e_3.16.56-1+deb8u1_mipsel.deb')
         deb_hash = 'd04d446045deecf7b755ef576551de0c4184dd44'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-3.16.0-6-loongson-2e')
 
         self.vm.set_console()
@@ -203,7 +181,7 @@ def test_mips_malta_cpio(self):
                    'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
         deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-4.5.0-2-4kc-malta')
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
@@ -454,8 +432,8 @@ def do_test_arm_raspi2(self, uart_id):
                    'raspberrypi-kernel_1.20190215-1_armhf.deb')
         deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
-        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
+        kernel_path = self.extract_from_pkg(deb_path, '/boot/kernel7.img')
+        dtb_path = self.extract_from_pkg(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -494,8 +472,8 @@ def test_arm_raspi2_initrd(self):
                    'raspberrypi-kernel_1.20190215-1_armhf.deb')
         deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
-        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
+        kernel_path = self.extract_from_pkg(deb_path, '/boot/kernel7.img')
+        dtb_path = self.extract_from_pkg(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
 
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
@@ -537,10 +515,10 @@ def test_arm_exynos4210_initrd(self):
                    'linux-image-4.19.0-6-armmp_4.19.67-2+deb10u1_armhf.deb')
         deb_hash = 'fa9df4a0d38936cb50084838f2cb933f570d7d82'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-4.19.0-6-armmp')
         dtb_path = '/usr/lib/linux-image-4.19.0-6-armmp/exynos4210-smdkv310.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
 
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
@@ -577,10 +555,10 @@ def test_arm_cubieboard_initrd(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun4i-a10-cubieboard.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
                       'arm/rootfs-armv5.cpio.gz')
@@ -618,10 +596,10 @@ def test_arm_cubieboard_sata(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun4i-a10-cubieboard.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
         rootfs_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
                       'arm/rootfs-armv5.ext2.gz')
@@ -748,10 +726,10 @@ def test_arm_orangepi(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -774,10 +752,10 @@ def test_arm_orangepi_initrd(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
                       'arm/rootfs-armv7a.cpio.gz')
@@ -818,10 +796,10 @@ def test_arm_orangepi_sd(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
         rootfs_url = ('http://storage.kernelci.org/images/rootfs/buildroot/'
                       'kci-2019.02/armel/base/rootfs.ext2.xz')
         rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
@@ -924,7 +902,7 @@ def test_arm_orangepi_uboot_netbsd9(self):
         # OrangePi "PC" device tree blob with 'setenv fdtfile' in U-Boot prompt,
         # before to boot NetBSD.
         uboot_path = '/usr/lib/u-boot/orangepi_plus/u-boot-sunxi-with-spl.bin'
-        uboot_path = self.extract_from_deb(deb_path, uboot_path)
+        uboot_path = self.extract_from_pkg(deb_path, uboot_path)
         image_url = ('https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.0/'
                      'evbarm-earmv7hf/binary/gzimg/armv7.img.gz')
         image_hash = '2babb29d36d8360adcb39c09e31060945259917a'
@@ -1044,7 +1022,7 @@ def test_m68k_q800(self):
                    '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
         deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-5.3.0-1-m68k')
 
         self.vm.set_console()
@@ -1133,8 +1111,8 @@ def test_arm_ast2600_debian(self):
         deb_hash = 'db40d32fe39255d05482bea48d72467b67d6225bb2a2a4d6f618cb8976f1e09e'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash,
                                     algorithm='sha256')
-        kernel_path = self.extract_from_deb(deb_path, '/boot/vmlinuz-5.10.0-3-armmp')
-        dtb_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path, '/boot/vmlinuz-5.10.0-3-armmp')
+        dtb_path = self.extract_from_pkg(deb_path,
                 '/usr/lib/linux-image-5.10.0-3-armmp/aspeed-bmc-opp-tacoma.dtb')
 
         self.vm.set_console()
diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index 3479b5233b..b5860b7d88 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -77,7 +77,7 @@ def test_arm64_xen_411_and_dom0(self):
                    'xen-hypervisor-4.11-arm64_4.11.4%2B37-g3263f257ca-1_arm64.deb')
         xen_sha1 = '034e634d4416adbad1212d59b62bccdcda63e62a'
         xen_deb = self.fetch_asset(xen_url, asset_hash=xen_sha1)
-        xen_path = self.extract_from_deb(xen_deb, "/boot/xen-4.11-arm64")
+        xen_path = self.extract_from_pkg(xen_deb, "/boot/xen-4.11-arm64")
 
         self.launch_xen(xen_path)
 
@@ -95,7 +95,7 @@ def test_arm64_xen_414_and_dom0(self):
                    'xen-hypervisor-4.14-arm64_4.14.0%2B80-gd101b417b7-1_arm64.deb')
         xen_sha1 = 'b9d209dd689ed2b393e625303a225badefec1160'
         xen_deb = self.fetch_asset(xen_url, asset_hash=xen_sha1)
-        xen_path = self.extract_from_deb(xen_deb, "/boot/xen-4.14-arm64")
+        xen_path = self.extract_from_pkg(xen_deb, "/boot/xen-4.14-arm64")
 
         self.launch_xen(xen_path)
 
@@ -112,6 +112,6 @@ def test_arm64_xen_415_and_dom0(self):
                    '?path=%2F&files=xen-upstream-4.15-unstable.deb')
         xen_sha1 = 'fc191172b85cf355abb95d275a24cc0f6d6579d8'
         xen_deb = self.fetch_asset(xen_url, asset_hash=xen_sha1)
-        xen_path = self.extract_from_deb(xen_deb, "/boot/xen-4.15-unstable")
+        xen_path = self.extract_from_pkg(xen_deb, "/boot/xen-4.15-unstable")
 
         self.launch_xen(xen_path)
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index bb32b31240..cfca71e65f 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -106,7 +106,7 @@ def test_mips_malta(self):
                    'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
         deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-2.6.32-5-4kc-malta')
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         console_pattern = 'Kernel command line: %s' % kernel_command_line
@@ -134,7 +134,7 @@ def test_mips64el_malta(self):
                    'linux-image-2.6.32-5-5kc-malta_2.6.32-48_mipsel.deb')
         deb_hash = '1aaec92083bf22fda31e0d27fa8d9a388e5fc3d5'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-2.6.32-5-5kc-malta')
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         console_pattern = 'Kernel command line: %s' % kernel_command_line
@@ -185,10 +185,10 @@ def test_arm_cubieboard_initrd(self):
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
         deb_hash = '9fa84beda245cabf0b4fa84cf6eaa7738ead1da0'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinuz-5.10.16-sunxi')
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun4i-a10-cubieboard.dtb'
-        dtb_path = self.extract_from_deb(deb_path, dtb_path)
+        dtb_path = self.extract_from_pkg(deb_path, dtb_path)
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
                       'arm/rootfs-armv5.cpio.gz')
@@ -233,7 +233,7 @@ def test_m68k_q800(self):
                    '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
         deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-5.3.0-1-m68k')
 
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -368,7 +368,7 @@ def test_mips_malta_cpio(self):
                    'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
         deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
         deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
+        kernel_path = self.extract_from_pkg(deb_path,
                                             '/boot/vmlinux-4.5.0-2-4kc-malta')
         initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
                       '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
diff --git a/tests/acceptance/tcg_plugins.py b/tests/acceptance/tcg_plugins.py
index 9ca1515c3b..bfa7f36104 100644
--- a/tests/acceptance/tcg_plugins.py
+++ b/tests/acceptance/tcg_plugins.py
@@ -59,7 +59,7 @@ def _grab_aarch64_kernel(self):
                       'linux-image-4.19.0-12-arm64_4.19.152-1_arm64.deb')
         kernel_sha1 = '2036c2792f80ac9c4ccaae742b2e0a28385b6010'
         kernel_deb = self.fetch_asset(kernel_url, asset_hash=kernel_sha1)
-        kernel_path = self.extract_from_deb(kernel_deb,
+        kernel_path = self.extract_from_pkg(kernel_deb,
                                             "/boot/vmlinuz-4.19.0-12-arm64")
         return kernel_path
 
-- 
2.31.1

[PATCH 07/16] Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert

[Cleber Rosa]

The certficiate at https://mipsdistros.mips.com/ is currently
expired (since Jan 23, 2021).

Instead of failing to obtaining the files, let's downgrade to HTTP
instead, given that the integrity of the files are already performed
locally, after the download, using the recorded hash.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 6 +++---
 tests/acceptance/replay_kernel.py      | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 80e3a2f7a6..bc0caa1099 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -278,7 +278,7 @@ def test_mips_malta32el_nanomips_4k(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
@@ -291,7 +291,7 @@ def test_mips_malta32el_nanomips_16k_up(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
@@ -304,7 +304,7 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index cfca71e65f..cac795ab4f 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -434,7 +434,7 @@ def test_mips_malta32el_nanomips_4k(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
@@ -448,7 +448,7 @@ def test_mips_malta32el_nanomips_16k_up(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
@@ -462,7 +462,7 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
-- 
2.31.1

[PATCH 08/16] acceptance/tests/vnc.py: use explicit syntax for enabling passwords

[Cleber Rosa]

This matches the command line on 82a17d1d67, where the "on" or "off"
should be explicitly given.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/vnc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/vnc.py b/tests/acceptance/vnc.py
index 22656bbcc2..f301fbb4f5 100644
--- a/tests/acceptance/vnc.py
+++ b/tests/acceptance/vnc.py
@@ -45,7 +45,7 @@ def test_change_password_requires_a_password(self):
                          'Could not set password')
 
     def test_change_password(self):
-        self.vm.add_args('-nodefaults', '-S', '-vnc', ':0,password')
+        self.vm.add_args('-nodefaults', '-S', '-vnc', ':0,password=on')
         self.vm.launch()
         self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled'])
         set_password_response = self.vm.qmp('change-vnc-password',
-- 
2.31.1

[PATCH 09/16] tests/acceptance/boot_xen.py: merge base classes

[Cleber Rosa]

While it's a good practice to have reusable base classes, in this
specific case there's no other user of the BootXenBase class.

By unifying the class used in this test, we can improve readability
and have the opportunity to add some future improvements in a clearer
fashion.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_xen.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index b5860b7d88..4c14f9e2e7 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -18,7 +18,7 @@
 from boot_linux_console import LinuxKernelTest
 
 
-class BootXenBase(LinuxKernelTest):
+class BootXen(LinuxKernelTest):
     """
     Boots a Xen hypervisor with a Linux DomU kernel.
     """
@@ -60,9 +60,6 @@ def launch_xen(self, xen_path):
         console_pattern = 'VFS: Cannot open root device'
         wait_for_console_pattern(self, console_pattern, "Panic on CPU 0:")
 
-
-class BootXen(BootXenBase):
-
     def test_arm64_xen_411_and_dom0(self):
         """
         :avocado: tags=arch:aarch64
-- 
2.31.1

[PATCH 10/16] tests/acceptance/boot_xen.py: unify tags

[Cleber Rosa]

Because all tests share the same tags, it's possible to have all of
them at the class level.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_xen.py | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)

diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index 4c14f9e2e7..e2c59f6592 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -21,6 +21,11 @@
 class BootXen(LinuxKernelTest):
     """
     Boots a Xen hypervisor with a Linux DomU kernel.
+
+    :avocado: tags=arch:aarch64
+    :avocado: tags=accel:tcg
+    :avocado: tags=cpu:cortex-a57
+    :avocado: tags=machine:virt
     """
 
     timeout = 90
@@ -61,13 +66,6 @@ def launch_xen(self, xen_path):
         wait_for_console_pattern(self, console_pattern, "Panic on CPU 0:")
 
     def test_arm64_xen_411_and_dom0(self):
-        """
-        :avocado: tags=arch:aarch64
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:cortex-a57
-        :avocado: tags=machine:virt
-        """
-
         # archive of file from https://deb.debian.org/debian/pool/main/x/xen/
         xen_url = ('https://fileserver.linaro.org/s/JSsewXGZ6mqxPr5/'
                    'download?path=%2F&files='
@@ -79,13 +77,6 @@ def test_arm64_xen_411_and_dom0(self):
         self.launch_xen(xen_path)
 
     def test_arm64_xen_414_and_dom0(self):
-        """
-        :avocado: tags=arch:aarch64
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:cortex-a57
-        :avocado: tags=machine:virt
-        """
-
         # archive of file from https://deb.debian.org/debian/pool/main/x/xen/
         xen_url = ('https://fileserver.linaro.org/s/JSsewXGZ6mqxPr5/'
                    'download?path=%2F&files='
@@ -97,13 +88,6 @@ def test_arm64_xen_414_and_dom0(self):
         self.launch_xen(xen_path)
 
     def test_arm64_xen_415_and_dom0(self):
-        """
-        :avocado: tags=arch:aarch64
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:cortex-a57
-        :avocado: tags=machine:virt
-        """
-
         xen_url = ('https://fileserver.linaro.org/'
                    's/JSsewXGZ6mqxPr5/download'
                    '?path=%2F&files=xen-upstream-4.15-unstable.deb')
-- 
2.31.1

[PATCH 11/16] tests/acceptance/boot_xen.py: fetch kernel during test setUp()

[Cleber Rosa]

The kernel is a common blob used in all tests.  By moving it to the
setUp() method, the "fetch asset" plugin will recognize the kernel and
attempt to fetch it and cache it before the tests are started.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index e2c59f6592..cd916ddba4 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -31,23 +31,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -58,7 +57,7 @@ def launch_xen(self, xen_path):
                          '-append', xen_command_line,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
-- 
2.31.1

[PATCH 12/16] tests/acceptance/boot_xen.py: removed unused import

[Cleber Rosa]

Just a clean up for an unused import.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_xen.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index cd916ddba4..66621fd14d 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -13,7 +13,6 @@
 
 import os
 
-from avocado import skipIf
 from avocado_qemu import wait_for_console_pattern
 from boot_linux_console import LinuxKernelTest
 
-- 
2.31.1

[PATCH 13/16] tests/acceptance/boot_xen.py: use class attribute

[Cleber Rosa]

Rather than defining a single use variable, let's just use the class
attribute directly.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_xen.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/tests/acceptance/boot_xen.py b/tests/acceptance/boot_xen.py
index 66621fd14d..8eec465414 100644
--- a/tests/acceptance/boot_xen.py
+++ b/tests/acceptance/boot_xen.py
@@ -49,11 +49,10 @@ def launch_xen(self, xen_path):
 
         self.vm.set_console()
 
-        xen_command_line = self.XEN_COMMON_COMMAND_LINE
         self.vm.add_args('-machine', 'virtualization=on',
                          '-m', '768',
                          '-kernel', xen_path,
-                         '-append', xen_command_line,
+                         '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
                          % (self.kernel_path))
-- 
2.31.1

[PATCH 14/16] tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update

[Cleber Rosa]

The NetBSD-7.1.2-prep.iso is no longer available on the CDN, but it's
still available in the archive.

Let's update its location so that users without the file on cache can
still fetch it and run the test.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/ppc_prep_40p.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index 2993ee3b07..6b28a69ea5 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -67,8 +67,8 @@ def test_openbios_and_netbsd(self):
         :avocado: tags=machine:40p
         :avocado: tags=os:netbsd
         """
-        drive_url = ('https://cdn.netbsd.org/pub/NetBSD/iso/7.1.2/'
-                     'NetBSD-7.1.2-prep.iso')
+        drive_url = ('https://archive.netbsd.org/pub/NetBSD-archive/'
+                     'NetBSD-7.1.2/iso/NetBSD-7.1.2-prep.iso')
         drive_hash = 'ac6fa2707d888b36d6fa64de6e7fe48e'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash,
                                       algorithm='md5')
-- 
2.31.1

[PATCH 15/16] tests/acceptance/ppc_prep_40p.py: clean up unused import

[Cleber Rosa]

Just a removal of an unused imported symbol.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/ppc_prep_40p.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index 6b28a69ea5..5e61e686bd 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -7,7 +7,6 @@
 
 import os
 
-from avocado import skipIf
 from avocado import skipUnless
 from avocado_qemu import Test
 from avocado_qemu import wait_for_console_pattern
-- 
2.31.1

[PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Cleber Rosa]

The arch and machine tags apply to all tests, so let's define them
only once.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/ppc_prep_40p.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index 5e61e686bd..d1e5674673 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -13,6 +13,10 @@
 
 
 class IbmPrep40pMachine(Test):
+    """
+    :avocado: tags=arch:ppc
+    :avocado: tags=machine:40p
+    """
 
     timeout = 60
 
@@ -24,8 +28,6 @@ class IbmPrep40pMachine(Test):
     @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_factory_firmware_and_netbsd(self):
         """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
         :avocado: tags=os:netbsd
         :avocado: tags=slowness:high
         """
@@ -48,10 +50,6 @@ def test_factory_firmware_and_netbsd(self):
         wait_for_console_pattern(self, 'Model: IBM PPS Model 6015')
 
     def test_openbios_192m(self):
-        """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
-        """
         self.vm.set_console()
         self.vm.add_args('-m', '192') # test fw_cfg
 
@@ -62,8 +60,6 @@ def test_openbios_192m(self):
 
     def test_openbios_and_netbsd(self):
         """
-        :avocado: tags=arch:ppc
-        :avocado: tags=machine:40p
         :avocado: tags=os:netbsd
         """
         drive_url = ('https://archive.netbsd.org/pub/NetBSD-archive/'
-- 
2.31.1

Re: [PATCH 05/16] Acceptance Tests: add standard clean up at test tearDown()

[Willian Rampazzo]

Hi, Cleber,

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> The avocado.Test class, used as the basis of the avocado_qemu.Test
> class, performs a clean of temporary directories up as part of its own
> tearDown() implementation.
>
> But the avocado_qemu.Test class is currently missing the same clean
> up, as it implemented its own tearDown() method without resorting to
> the upper class behavior.
>
> This brings avocado_qemu.Test behavior in sync with the standard
> avocado.Test behavior and prevents temporary directories from
> cluttering the test results directory (unless instructed to do so with
> Avocado's "--keep-tmp" option).
>
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 1 +
>  1 file changed, 1 insertion(+)
>

There was a comment from Wainer in the original patch and as you were
on vacation I submitted a respin last Monday:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05117.html

I also did some refactoring/fixes to the avocado_qemu here:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05212.html

Re: [PATCH 00/16] Acceptance Tests: use Avocado 91.0 features and other improvements

[Pavel Dovgalyuk]

Hi, Cleber!

What about record/replay tests from 25.06?

On 24.09.2021 21:54, Cleber Rosa wrote:
> This is a collection of patches for the Acceptance Tests to leverage
> some of the features of Avocado 91.0.  With the Avocado version bump
> by itself, there would be a change in the default "test runner"
> implementation that Avocado uses, from the one simply known as
> "runner" to the new one called "nrunner".
> 
> Among the changes from one implementation to the other, is the fact
> that "nrunner" will run tests in parallel by default.  This is *not
> yet* enabled by default on "make check-acceptance", but users can
> choose to use simply by setting the "AVOCADO_RUNNER" variable, that
> is:
> 
>    make AVOCADO_RUNNER=nrunner check-acceptance
> 
> If you are curious about the architectural differences of the nrunner,
> please refer to:
> 
>    https://avocado-framework.readthedocs.io/en/91.0/guides/contributor/chapters/runners.html
> 
> One other noteworthy proposal is a convention to tag tests that either
> have known issues, or that touch on QEMU features that have known
> issues.  By tagging those tests accordingly, they will be
> automatically excluded from the regular execution with "make
> check-acceptance".
> 
> Finally, some updates to assets locations and some tests refactors and
> cleanups.
> 
> Cleber Rosa (16):
>    Acceptance Tests: bump Avocado requirement to 91.0
>    Acceptance Tests: improve check-acceptance description
>    Acceptance Tests: add mechanism for listing tests
>    Acceptance Tests: keep track and disable tests with known issues
>    Acceptance Tests: add standard clean up at test tearDown()
>    Acceptance Tests: use extract from package from avocado.utils
>    Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert
>    acceptance/tests/vnc.py: use explicit syntax for enabling passwords
>    tests/acceptance/boot_xen.py: merge base classes
>    tests/acceptance/boot_xen.py: unify tags
>    tests/acceptance/boot_xen.py: fetch kernel during test setUp()
>    tests/acceptance/boot_xen.py: removed unused import
>    tests/acceptance/boot_xen.py: use class attribute
>    tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update
>    tests/acceptance/ppc_prep_40p.py: clean up unused import
>    tests/acceptance/ppc_prep_40p.py: unify tags
> 
>   docs/devel/testing.rst                    | 40 ++++++++++
>   tests/Makefile.include                    | 15 +++-
>   tests/acceptance/avocado_qemu/__init__.py |  1 +
>   tests/acceptance/boot_linux_console.py    | 93 +++++++++--------------
>   tests/acceptance/boot_xen.py              | 54 ++++---------
>   tests/acceptance/machine_rx_gdbsim.py     |  3 +
>   tests/acceptance/ppc_prep_40p.py          | 17 ++---
>   tests/acceptance/replay_kernel.py         | 18 ++---
>   tests/acceptance/tcg_plugins.py           |  2 +-
>   tests/acceptance/vnc.py                   |  2 +-
>   tests/requirements.txt                    |  2 +-
>   11 files changed, 128 insertions(+), 119 deletions(-)
> 

Re: [PATCH 02/16] Acceptance Tests: improve check-acceptance description

[Philippe Mathieu-Daudé]

On 9/24/21 20:54, Cleber Rosa wrote:
> The "check-acceptance" make rule won't necessarily run *all* available
> tests, because it employs a filter based on the currently configured
> targets.  This change in the description of the rule makes that
> behavior extra clear.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH 08/16] acceptance/tests/vnc.py: use explicit syntax for enabling passwords

[Philippe Mathieu-Daudé]

On 9/24/21 20:54, Cleber Rosa wrote:
> This matches the command line on 82a17d1d67, where the "on" or "off"
> should be explicitly given.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/vnc.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH 15/16] tests/acceptance/ppc_prep_40p.py: clean up unused import

[Philippe Mathieu-Daudé]

On 9/24/21 20:55, Cleber Rosa wrote:
> Just a removal of an unused imported symbol.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/ppc_prep_40p.py | 1 -
>  1 file changed, 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH 12/16] tests/acceptance/boot_xen.py: removed unused import

[Philippe Mathieu-Daudé]

On 9/24/21 20:55, Cleber Rosa wrote:
> Just a clean up for an unused import.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_xen.py | 1 -
>  1 file changed, 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Philippe Mathieu-Daudé]

On 9/24/21 20:55, Cleber Rosa wrote:
> The arch and machine tags apply to all tests, so let's define them
> only once.

IIRC this was not the case at the beginning with Avocado 65.0
at least, right? If so, maybe mention since when it is possible?

> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/ppc_prep_40p.py | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
> index 5e61e686bd..d1e5674673 100644
> --- a/tests/acceptance/ppc_prep_40p.py
> +++ b/tests/acceptance/ppc_prep_40p.py
> @@ -13,6 +13,10 @@
>  
>  
>  class IbmPrep40pMachine(Test):
> +    """
> +    :avocado: tags=arch:ppc
> +    :avocado: tags=machine:40p
> +    """
>  
>      timeout = 60
>  
> @@ -24,8 +28,6 @@ class IbmPrep40pMachine(Test):
>      @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
>      def test_factory_firmware_and_netbsd(self):
>          """
> -        :avocado: tags=arch:ppc
> -        :avocado: tags=machine:40p
>          :avocado: tags=os:netbsd
>          :avocado: tags=slowness:high
>          """
> @@ -48,10 +50,6 @@ def test_factory_firmware_and_netbsd(self):
>          wait_for_console_pattern(self, 'Model: IBM PPS Model 6015')
>  
>      def test_openbios_192m(self):
> -        """
> -        :avocado: tags=arch:ppc
> -        :avocado: tags=machine:40p
> -        """
>          self.vm.set_console()
>          self.vm.add_args('-m', '192') # test fw_cfg
>  
> @@ -62,8 +60,6 @@ def test_openbios_192m(self):
>  
>      def test_openbios_and_netbsd(self):
>          """
> -        :avocado: tags=arch:ppc
> -        :avocado: tags=machine:40p
>          :avocado: tags=os:netbsd
>          """
>          drive_url = ('https://archive.netbsd.org/pub/NetBSD-archive/'
> 

Re: [PATCH 14/16] tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update

[Philippe Mathieu-Daudé]

On 9/24/21 20:55, Cleber Rosa wrote:
> The NetBSD-7.1.2-prep.iso is no longer available on the CDN, but it's
> still available in the archive.
> 
> Let's update its location so that users without the file on cache can
> still fetch it and run the test.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/ppc_prep_40p.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH 00/16] Acceptance Tests: use Avocado 91.0 features and other improvements

[Philippe Mathieu-Daudé]

On 9/24/21 20:54, Cleber Rosa wrote:
> This is a collection of patches for the Acceptance Tests to leverage
> some of the features of Avocado 91.0.  With the Avocado version bump
> by itself, there would be a change in the default "test runner"
> implementation that Avocado uses, from the one simply known as
> "runner" to the new one called "nrunner".
> 
> Among the changes from one implementation to the other, is the fact
> that "nrunner" will run tests in parallel by default.  This is *not
> yet* enabled by default on "make check-acceptance", but users can
> choose to use simply by setting the "AVOCADO_RUNNER" variable, that
> is:
> 
>   make AVOCADO_RUNNER=nrunner check-acceptance
> 
> If you are curious about the architectural differences of the nrunner,
> please refer to:
> 
>   https://avocado-framework.readthedocs.io/en/91.0/guides/contributor/chapters/runners.html
> 
> One other noteworthy proposal is a convention to tag tests that either
> have known issues, or that touch on QEMU features that have known
> issues.  By tagging those tests accordingly, they will be
> automatically excluded from the regular execution with "make
> check-acceptance".
> 
> Finally, some updates to assets locations and some tests refactors and
> cleanups.
> 
> Cleber Rosa (16):
>   Acceptance Tests: bump Avocado requirement to 91.0
>   Acceptance Tests: improve check-acceptance description
>   Acceptance Tests: add mechanism for listing tests
>   Acceptance Tests: keep track and disable tests with known issues
>   Acceptance Tests: add standard clean up at test tearDown()
>   Acceptance Tests: use extract from package from avocado.utils
>   Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert
>   acceptance/tests/vnc.py: use explicit syntax for enabling passwords
>   tests/acceptance/boot_xen.py: merge base classes
>   tests/acceptance/boot_xen.py: unify tags
>   tests/acceptance/boot_xen.py: fetch kernel during test setUp()
>   tests/acceptance/boot_xen.py: removed unused import
>   tests/acceptance/boot_xen.py: use class attribute
>   tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update
>   tests/acceptance/ppc_prep_40p.py: clean up unused import
>   tests/acceptance/ppc_prep_40p.py: unify tags

Thanks. Trivial patches 2, 8, 12, 14, 15 applied to my
integration-testing tree.

Re: [PATCH 14/16] tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update

[Reinoud Zandijk]

On Mon, Sep 27, 2021 at 05:26:43PM +0200, Philippe Mathieu-Daudé wrote:
> On 9/24/21 20:55, Cleber Rosa wrote:
> > The NetBSD-7.1.2-prep.iso is no longer available on the CDN, but it's
> > still available in the archive.
> > 
> > Let's update its location so that users without the file on cache can
> > still fetch it and run the test.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/acceptance/ppc_prep_40p.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

No objections but is there a reason why it is still using NetBSD 7.1.2?  I am
not a ppc guy but as a NetBSD developer just curious.
Reinoud

Re: [PATCH 14/16] tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update

[Philippe Mathieu-Daudé]

On 9/30/21 21:19, Reinoud Zandijk wrote:
> On Mon, Sep 27, 2021 at 05:26:43PM +0200, Philippe Mathieu-Daudé wrote:
>> On 9/24/21 20:55, Cleber Rosa wrote:
>>> The NetBSD-7.1.2-prep.iso is no longer available on the CDN, but it's
>>> still available in the archive.
>>>
>>> Let's update its location so that users without the file on cache can
>>> still fetch it and run the test.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/acceptance/ppc_prep_40p.py | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
> No objections but is there a reason why it is still using NetBSD 7.1.2?  I am
> not a ppc guy but as a NetBSD developer just curious.

We know that QEMU 40p machine successfully boots NetBSD 7.1.2 and
we don't want regression with it.

It could boot more recent versions but I haven't tested. Tests
are welcome :) There should be quite easy to add.

Regards,

Phil.

Re: [PATCH 01/16] Acceptance Tests: bump Avocado requirement to 91.0

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> Avocado 91.0, brings, among other changes, a switch to a new runner
> implementation, known as "nrunner".  While my personal testing shows
> that it's suitable for running the QEMU tests, there are some
> considerations that should be addressed before the change.
>
> For instance, the "nrunner" implementation will run tests in parallel
> by default.  With more tests running, and resources shared, some tests
> may fail with timeouts on some environments.
>
> So, for now, let's bump the Avocado version to allow for the other
> features to be used.  And the "nrunner" implementation to be used by
> those that choose to do so.
>
> More information can be found at the release notes at:
>    https://avocado-framework.readthedocs.io/en/latest/releases/91_0.html
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/testing.rst | 12 ++++++++++++
>  tests/Makefile.include |  2 ++
>  tests/requirements.txt |  2 +-
>  3 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 4a0abbf23d..2818c58ff8 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -740,6 +740,18 @@ may be invoked by running:
>
>    tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
>
> +Running tests in parallel
> +-------------------------
> +
> +The current ``make check-acceptance`` target continues to use the
> +Avocado runner implementation simply known as "runner".  But, it's now
> +possible to opt-in and choose the "nrunner" implementation, which,
> +among other things, allows for parallel execution of tests:
> +
> +.. code::
> +
> +  make AVOCADO_RUNNER=nrunner check-acceptance

The examples in the documentation are all setting the environment
variable at the end of the `make` command. For consistency, I would
move the definition of the AVOCADO_RUNNER to the end of the command.

> +
>  Manual Installation
>  -------------------
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 6e16c05f10..ac289a2e41 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -92,6 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # Any number of command separated loggers are accepted.  For more
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=app
> +AVOCADO_RUNNER=runner
>  AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
>
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> @@ -128,6 +129,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>         $(call quiet-command, \
>              $(TESTS_VENV_DIR)/bin/python -m avocado \
>              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> +            --test-runner=$(AVOCADO_RUNNER) \
>              --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
>              $(AVOCADO_TAGS) \
>              $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index a21b59b443..40af24c664 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,5 +1,5 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -avocado-framework==88.1
> +avocado-framework==91.0
>  pycdlib==1.11.0
> --
> 2.31.1
>

Apart from a tiny documentation change, the code is welcome as it
allows using the new Avocado runner.

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 03/16] Acceptance Tests: add mechanism for listing tests

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> It is helpful to know the tests that would be executed with a "make
> check-acceptance" without executing them.  Let's introduce a "make
> list-acceptance" rule for that purpose.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 2c03256ae8..d1f90572a7 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -17,6 +17,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
>         @echo " $(MAKE) check-softfloat      Run FPU emulation tests"
>  endif
>         @echo " $(MAKE) check-acceptance     Run acceptance (functional) tests for currently configured targets"
> +       @echo " $(MAKE) check-acceptance     List acceptance (functional) tests for currently configured targets"

s/check-acceptance/list-acceptance/

>         @echo
>         @echo " $(MAKE) check-report.tap     Generates an aggregated TAP test report"
>         @echo " $(MAKE) check-venv           Creates a Python venv for tests"
> @@ -135,6 +136,13 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>              $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>              "AVOCADO", "tests/acceptance")
>
> +list-acceptance: check-venv
> +       $(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado list \
> +            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> +            $(AVOCADO_TAGS) tests/acceptance, \

It would be interesting to adjust the command to use the
$(AVOCADO_CMDLINE_TAGS) and allow users to restrict the tests listed
by its tags.

> +            "AVOCADO", "list tests/acceptance")
> +
>  # Consolidated targets
>
>  .PHONY: check-block check check-clean get-vm-images
> --
> 2.31.1
>

Re: [PATCH 11/16] tests/acceptance/boot_xen.py: fetch kernel during test setUp()

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:59 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> The kernel is a common blob used in all tests.  By moving it to the
> setUp() method, the "fetch asset" plugin will recognize the kernel and
> attempt to fetch it and cache it before the tests are started.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_xen.py | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 10/16] tests/acceptance/boot_xen.py: unify tags

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:58 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> Because all tests share the same tags, it's possible to have all of
> them at the class level.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_xen.py | 26 +++++---------------------
>  1 file changed, 5 insertions(+), 21 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 09/16] tests/acceptance/boot_xen.py: merge base classes

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:58 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> While it's a good practice to have reusable base classes, in this
> specific case there's no other user of the BootXenBase class.
>
> By unifying the class used in this test, we can improve readability
> and have the opportunity to add some future improvements in a clearer
> fashion.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_xen.py | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 13/16] tests/acceptance/boot_xen.py: use class attribute

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:59 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> Rather than defining a single use variable, let's just use the class
> attribute directly.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/boot_xen.py | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 4:01 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> The arch and machine tags apply to all tests, so let's define them
> only once.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/ppc_prep_40p.py | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Willian Rampazzo]

On Wed, Oct 6, 2021 at 4:17 PM Willian Rampazzo <wrampazz@redhat.com> wrote:
>
> On Fri, Sep 24, 2021 at 4:01 PM Cleber Rosa <crosa@redhat.com> wrote:
> >
> > The arch and machine tags apply to all tests, so let's define them
> > only once.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/acceptance/ppc_prep_40p.py | 12 ++++--------
> >  1 file changed, 4 insertions(+), 8 deletions(-)
> >
>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Also:
Tested-by: Willian Rampazzo <willianr@redhat.com>