[PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

This series adds ARM SMMU and Intel IOMMU functional
tests using Fedora cloud-init images.

ARM SMMU tests feature guests with and without RIL
(range invalidation support) using respectively fedora 33
and 31.  For each, we test the protection of virtio-net-pci
and virtio-block-pci devices. Also strict=no and passthrough
modes are tested. So there is a total of 6 tests.

The series applies on top of Cleber's series:
- [PATCH 0/3] Acceptance Tests: support choosing specific

Note:
- SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
test_smmu_noril_nostrict) pass but the log reports:
"WARN: Test passed but there were warnings during execution."
This seems due to the lack of hash when fetching the kernel and
initrd through fetch_asset():
WARNI| No hash provided. Cannot check the asset file integrity.

History:
v3 -> v4:
- I added Wainer's refactoring of KNOWN_DISTROS
into a class (last patch) and took into account his comments.

v2 -> v3:
- Added Intel IOMMU tests were added. Different
operating modes are tested such as strict, caching mode, pt.

Best Regards

Eric

The series and its dependencies can be found at:
https://github.com/eauger/qemu/tree/avocado-qemu-v4

Eric Auger (3):
  Acceptance Tests: Add default kernel params and pxeboot url to the
    KNOWN_DISTROS collection
  avocado_qemu: Add SMMUv3 tests
  avocado_qemu: Add Intel iommu tests

Wainer dos Santos Moschetta (1):
  avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class

 tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
 tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
 tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
 3 files changed, 332 insertions(+), 33 deletions(-)
 create mode 100644 tests/acceptance/intel_iommu.py
 create mode 100644 tests/acceptance/smmu.py

-- 
2.26.3

[PATCH v4 1/4] Acceptance Tests: Add default kernel params and pxeboot url to the KNOWN_DISTROS collection

[Eric Auger]

When running LinuxTests we may need to run the guest with
custom params. It is practical to store the pxeboot URL
and the default kernel params so that the
tests just need to fetch those and augment the kernel params.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v2 -> v3:
- add fed32 and fed33 checksums

v3 -> v4:
- fix indent issues (Wainer)
- use dl.fedoraproject.org instead of mirror
---
 tests/acceptance/avocado_qemu/__init__.py | 52 ++++++++++++++++++++++-
 1 file changed, 50 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 81ac90bebb..64e9c1952d 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -305,17 +305,59 @@ def ssh_command(self, command):
     'fedora': {
         '31': {
             'x86_64':
-            {'checksum': 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'},
+            {'checksum': 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
+             'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
+                            "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
+             'kernel_params': "root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
+                              "no_timer_check net.ifnames=0 "
+                              "console=tty1 console=ttyS0,115200n8"},
             'aarch64':
-            {'checksum': '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'},
+            {'checksum': '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49',
+             'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
+                            "linux/releases/31/Everything/aarch64/os/images/pxeboot/",
+             'kernel_params': "root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro "
+                              "earlyprintk=pl011,0x9000000 ignore_loglevel "
+                              "no_timer_check printk.time=1 rd_NO_PLYMOUTH "
+                              "console=ttyAMA0 "},
             'ppc64':
             {'checksum': '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'},
             's390x':
             {'checksum': '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'},
             }
+        ,
+        '32': {
+            'aarch64':
+            {'kernel_params': "root=UUID=3df75b65-be8d-4db4-8655-14d95c0e90c5 ro "
+                              "no_timer_check net.ifnames=0 console=tty1 "
+                              "console=ttyS0,115200n8 ",
+             'checksum': 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967',
+             'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
+                            "releases/32/Server/aarch64/os/images/pxeboot/"},
+            }
+        ,
+        '33': {
+            'aarch64':
+            {'kernel_params': "root=UUID=d20b3ffa-6397-4a63-a734-1126a0208f8a ro "
+                              "no_timer_check net.ifnames=0 console=tty1 "
+                              "console=ttyS0,115200n8 console=tty0 ",
+             'checksum': 'e7f75cdfd523fe5ac2ca9eeece68edc1a81f386a17f969c1d1c7c87031008a6b',
+             'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
+                            "releases/33/Server/aarch64/os/images/pxeboot/"},
+            }
         }
     }
 
+def get_known_distro_kernel_params(distro, distro_version, arch):
+    try:
+        return KNOWN_DISTROS.get(distro).get(distro_version).get(arch).get('kernel_params')
+    except AttributeError:
+        return None
+
+def get_known_distro_pxeboot_url(distro, distro_version, arch):
+    try:
+        return KNOWN_DISTROS.get(distro).get(distro_version).get(arch).get('pxeboot_url')
+    except AttributeError:
+        return None
 
 def get_known_distro_checksum(distro, distro_version, arch):
     try:
@@ -449,6 +491,12 @@ def set_up_cloudinit(self, ssh_pubkey=None):
         cloudinit_iso = self.prepare_cloudinit(ssh_pubkey)
         self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
 
+    def get_default_kernel_params(self):
+        return get_known_distro_kernel_params(self.distro, self.distro_version, self.arch)
+
+    def get_pxeboot_url(self):
+        return get_known_distro_pxeboot_url(self.distro, self.distro_version, self.arch)
+
     def launch_and_wait(self, set_up_ssh_connection=True):
         self.vm.set_console()
         self.vm.launch()
-- 
2.26.3

[PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Eric Auger]

Add new tests checking the good behavior of the SMMUv3 protecting
2 virtio pci devices (block and net). We check the guest boots and
we are able to install a package. Different guest configs are tested:
standard, passthrough an strict=0. This is tested with both fedora 31 and
33. The former uses a 5.3 kernel without range invalidation whereas the
latter uses a 5.8 kernel that features range invalidation.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v3 -> v4:
- add tags for machine, distro in the class
- removed smp and memory overrides
- set default param value of common_vm_setup to False

v1 -> v2:
- removed ssh import
- combined add_command_args() and common_vm_setup()
- moved tags in class' docstring and added tags=arch:aarch64
- use self.get_default_kernel_params()
- added RIL tests with fed33 + introduce new tags
---
 tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 132 insertions(+)
 create mode 100644 tests/acceptance/smmu.py

diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
new file mode 100644
index 0000000000..c1d4b88e5f
--- /dev/null
+++ b/tests/acceptance/smmu.py
@@ -0,0 +1,132 @@
+# SMMUv3 Functional tests
+#
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+#  Eric Auger <eric.auger@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado_qemu import LinuxTest, BUILD_DIR
+
+class SMMU(LinuxTest):
+    """
+    :avocado: tags=accel:kvm
+    :avocado: tags=cpu:host
+    :avocado: tags=arch:aarch64
+    :avocado: tags=machine:virt
+    :avocado: tags=distro:fedora
+    :avocado: tags=smmu
+    """
+
+    IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
+    kernel_path = None
+    initrd_path = None
+    kernel_params = None
+
+    def set_up_boot(self):
+        path = self.download_boot()
+        self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
+                         'drive=drv0,id=virtio-disk0,bootindex=1,'
+                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
+        self.vm.add_args('-drive',
+                         'file=%s,if=none,cache=writethrough,id=drv0' % path)
+
+    def setUp(self):
+        super(SMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON)
+
+    def common_vm_setup(self, custom_kernel=False):
+        self.require_accelerator("kvm")
+        self.vm.add_args("-accel", "kvm")
+        self.vm.add_args("-cpu", "host")
+        self.vm.add_args("-machine", "iommu=smmuv3")
+        self.vm.add_args("-d", "guest_errors")
+        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
+                         'edk2-aarch64-code.fd'))
+        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
+        self.vm.add_args('-object',
+                         'rng-random,id=rng0,filename=/dev/urandom')
+
+        if custom_kernel is False:
+            return
+
+        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
+        initrd_url = self.get_pxeboot_url() + 'initrd.img'
+        self.kernel_path = self.fetch_asset(kernel_url)
+        self.initrd_path = self.fetch_asset(initrd_url)
+
+    def run_and_check(self):
+        if self.kernel_path:
+            self.vm.add_args('-kernel', self.kernel_path,
+                             '-append', self.kernel_params,
+                             '-initrd', self.initrd_path)
+        self.launch_and_wait()
+        self.ssh_command('cat /proc/cmdline')
+        self.ssh_command('dnf -y install numactl-devel')
+
+
+    # 5.3 kernel without RIL #
+
+    def test_smmu_noril(self):
+        """
+        :avocado: tags=smmu_noril
+        :avocado: tags=smmu_noril_tests
+        :avocado: tags=distro_version:31
+        """
+        self.common_vm_setup()
+        self.run_and_check()
+
+    def test_smmu_noril_passthrough(self):
+        """
+        :avocado: tags=smmu_noril_passthrough
+        :avocado: tags=smmu_noril_tests
+        :avocado: tags=distro_version:31
+        """
+        self.common_vm_setup(True)
+        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
+        self.run_and_check()
+
+    def test_smmu_noril_nostrict(self):
+        """
+        :avocado: tags=smmu_noril_nostrict
+        :avocado: tags=smmu_noril_tests
+        :avocado: tags=distro_version:31
+        """
+        self.common_vm_setup(True)
+        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
+        self.run_and_check()
+
+    # 5.8 kernel featuring range invalidation
+    # >= v5.7 kernel
+
+    def test_smmu_ril(self):
+        """
+        :avocado: tags=smmu_ril
+        :avocado: tags=smmu_ril_tests
+        :avocado: tags=distro_version:33
+        """
+        self.common_vm_setup()
+        self.run_and_check()
+
+    def test_smmu_ril_passthrough(self):
+        """
+        :avocado: tags=smmu_ril_passthrough
+        :avocado: tags=smmu_ril_tests
+        :avocado: tags=distro_version:33
+        """
+        self.common_vm_setup(True)
+        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
+        self.run_and_check()
+
+    def test_smmu_ril_nostrict(self):
+        """
+        :avocado: tags=smmu_ril_nostrict
+        :avocado: tags=smmu_ril_tests
+        :avocado: tags=distro_version:33
+        """
+        self.common_vm_setup(True)
+        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
+        self.run_and_check()
-- 
2.26.3

[PATCH v4 3/4] avocado_qemu: Add Intel iommu tests

[Eric Auger]

Add Intel IOMMU functional tests based on fedora 31.
Different configs are checked:
- strict
- caching mode, strict
- passthrough.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
---
 tests/acceptance/intel_iommu.py | 115 ++++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)
 create mode 100644 tests/acceptance/intel_iommu.py

diff --git a/tests/acceptance/intel_iommu.py b/tests/acceptance/intel_iommu.py
new file mode 100644
index 0000000000..0b68d3c572
--- /dev/null
+++ b/tests/acceptance/intel_iommu.py
@@ -0,0 +1,115 @@
+# INTEL_IOMMU Functional tests
+#
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+#  Eric Auger <eric.auger@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado_qemu import LinuxTest, BUILD_DIR
+from avocado.utils import ssh
+
+class INTEL_IOMMU(LinuxTest):
+    """
+    :avocado: tags=arch:x86_64
+    :avocado: tags=distro:fedora
+    :avocado: tags=distro_version:31
+    :avocado: tags=machine:q35
+    :avocado: tags=accel:kvm
+    :avocado: tags=intel_iommu
+    """
+
+    IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
+    kernel_path = None
+    initrd_path = None
+    kernel_params = None
+
+    def set_up_boot(self):
+        path = self.download_boot()
+        self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
+                         'drive=drv0,id=virtio-disk0,bootindex=1,'
+                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
+        self.vm.add_args('-device', 'virtio-gpu-pci' + self.IOMMU_ADDON)
+        self.vm.add_args('-drive',
+                         'file=%s,if=none,cache=writethrough,id=drv0' % path)
+
+    def setUp(self):
+        super(INTEL_IOMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON)
+
+    def add_common_args(self):
+        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
+        self.vm.add_args('-object',
+                         'rng-random,id=rng0,filename=/dev/urandom')
+
+    def common_vm_setup(self, custom_kernel=None):
+        self.require_accelerator("kvm")
+        self.add_common_args()
+        self.vm.add_args("-accel", "kvm")
+
+        if custom_kernel is None:
+            return
+
+        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
+        initrd_url = self.get_pxeboot_url() + 'initrd.img'
+        self.kernel_path = self.fetch_asset(kernel_url)
+        self.initrd_path = self.fetch_asset(initrd_url)
+
+    def run_and_check(self):
+        if self.kernel_path:
+            self.vm.add_args('-kernel', self.kernel_path,
+                             '-append', self.kernel_params,
+                             '-initrd', self.initrd_path)
+        self.launch_and_wait()
+        self.ssh_command('cat /proc/cmdline')
+        self.ssh_command('dmesg | grep -e DMAR -e IOMMU')
+        self.ssh_command('find /sys/kernel/iommu_groups/ -type l')
+        self.ssh_command('dnf -y install numactl-devel')
+
+    def test_intel_iommu(self):
+        """
+        :avocado: tags=intel_iommu_intremap
+        """
+
+        self.common_vm_setup(True)
+        self.vm.add_args('-device', 'intel-iommu,intremap=on')
+        self.vm.add_args('-machine', 'kernel_irqchip=split')
+
+        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on'
+        self.run_and_check()
+
+    def test_intel_iommu_strict(self):
+        """
+        :avocado: tags=intel_iommu_strict
+        """
+
+        self.common_vm_setup(True)
+        self.vm.add_args('-device', 'intel-iommu,intremap=on')
+        self.vm.add_args('-machine', 'kernel_irqchip=split')
+        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
+        self.run_and_check()
+
+    def test_intel_iommu_strict_cm(self):
+        """
+        :avocado: tags=intel_iommu_strict_cm
+        """
+
+        self.common_vm_setup(True)
+        self.vm.add_args('-device', 'intel-iommu,intremap=on,caching-mode=on')
+        self.vm.add_args('-machine', 'kernel_irqchip=split')
+        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
+        self.run_and_check()
+
+    def test_intel_iommu_pt(self):
+        """
+        :avocado: tags=intel_iommu_pt
+        """
+
+        self.common_vm_setup(True)
+        self.vm.add_args('-device', 'intel-iommu,intremap=on')
+        self.vm.add_args('-machine', 'kernel_irqchip=split')
+        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on iommu=pt'
+        self.run_and_check()
-- 
2.26.3

[PATCH v4 4/4] avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class

[Eric Auger]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

As the KNOWN_DISTROS grows, more loosely methods will be created in
the avocado_qemu/__init__.py file.

Let's refactor the code so that KNOWN_DISTROS and related methods are
packaged in a class

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

[Eric] rebase and add commit message
---
 tests/acceptance/avocado_qemu/__init__.py | 160 +++++++++++-----------
 tests/acceptance/intel_iommu.py           |  12 +-
 tests/acceptance/smmu.py                  |  12 +-
 3 files changed, 94 insertions(+), 90 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 64e9c1952d..00eb0bfcc8 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -299,72 +299,86 @@ def ssh_command(self, command):
                          f'Guest command failed: {command}')
         return stdout_lines, stderr_lines
 
+class LinuxDistro:
+    """Represents a Linux distribution
 
-#: A collection of known distros and their respective image checksum
-KNOWN_DISTROS = {
-    'fedora': {
-        '31': {
-            'x86_64':
-            {'checksum': 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
-             'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
-                            "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
-             'kernel_params': "root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
+    Holds information of known distros.
+    """
+    #: A collection of known distros and their respective image checksum
+    KNOWN_DISTROS = {
+        'fedora': {
+            '31': {
+                'x86_64':
+                {'checksum': 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
+                'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
+                               "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
+                'kernel_params': "root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
                               "no_timer_check net.ifnames=0 "
                               "console=tty1 console=ttyS0,115200n8"},
-            'aarch64':
-            {'checksum': '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49',
-             'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
-                            "linux/releases/31/Everything/aarch64/os/images/pxeboot/",
-             'kernel_params': "root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro "
-                              "earlyprintk=pl011,0x9000000 ignore_loglevel "
-                              "no_timer_check printk.time=1 rd_NO_PLYMOUTH "
-                              "console=ttyAMA0 "},
-            'ppc64':
-            {'checksum': '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'},
-            's390x':
-            {'checksum': '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'},
-            }
-        ,
-        '32': {
-            'aarch64':
-            {'kernel_params': "root=UUID=3df75b65-be8d-4db4-8655-14d95c0e90c5 ro "
-                              "no_timer_check net.ifnames=0 console=tty1 "
-                              "console=ttyS0,115200n8 ",
-             'checksum': 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967',
-             'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
-                            "releases/32/Server/aarch64/os/images/pxeboot/"},
-            }
-        ,
-        '33': {
-            'aarch64':
-            {'kernel_params': "root=UUID=d20b3ffa-6397-4a63-a734-1126a0208f8a ro "
-                              "no_timer_check net.ifnames=0 console=tty1 "
-                              "console=ttyS0,115200n8 console=tty0 ",
-             'checksum': 'e7f75cdfd523fe5ac2ca9eeece68edc1a81f386a17f969c1d1c7c87031008a6b',
-             'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
-                            "releases/33/Server/aarch64/os/images/pxeboot/"},
+                'aarch64':
+                {'checksum': '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49',
+                 'pxeboot_url': "https://archives.fedoraproject.org/pub/archive/fedora/"
+                                "linux/releases/31/Everything/aarch64/os/images/pxeboot/",
+                 'kernel_params': "root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro "
+                                  "earlyprintk=pl011,0x9000000 ignore_loglevel "
+                                  "no_timer_check printk.time=1 rd_NO_PLYMOUTH "
+                                  "console=ttyAMA0 "},
+                'ppc64':
+                {'checksum': '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'},
+                's390x':
+                {'checksum': '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'},
+                }
+            ,
+            '32': {
+                'aarch64':
+                {'kernel_params': "root=UUID=3df75b65-be8d-4db4-8655-14d95c0e90c5 ro "
+                                  "no_timer_check net.ifnames=0 console=tty1 "
+                                  "console=ttyS0,115200n8 ",
+                 'checksum': 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967',
+                 'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
+                                "releases/32/Server/aarch64/os/images/pxeboot/"},
+                }
+            ,
+            '33': {
+                'aarch64':
+                {'kernel_params': "root=UUID=d20b3ffa-6397-4a63-a734-1126a0208f8a ro "
+                                  "no_timer_check net.ifnames=0 console=tty1 "
+                                  "console=ttyS0,115200n8 console=tty0 ",
+                 'checksum': 'e7f75cdfd523fe5ac2ca9eeece68edc1a81f386a17f969c1d1c7c87031008a6b',
+                 'pxeboot_url': "http://dl.fedoraproject.org/pub/fedora/linux/"
+                                "releases/33/Server/aarch64/os/images/pxeboot/"},
+                }
             }
         }
-    }
 
-def get_known_distro_kernel_params(distro, distro_version, arch):
-    try:
-        return KNOWN_DISTROS.get(distro).get(distro_version).get(arch).get('kernel_params')
-    except AttributeError:
-        return None
+    def __init__(self, name, version, arch):
+        self.name = name
+        self.version = version
+        self.arch = arch
+        try:
+            self._info = self.KNOWN_DISTROS.get(name).get(version).get(arch)
+        except AttributeError:
+            # Unknown distro
+            self._info = {}
 
-def get_known_distro_pxeboot_url(distro, distro_version, arch):
-    try:
-        return KNOWN_DISTROS.get(distro).get(distro_version).get(arch).get('pxeboot_url')
-    except AttributeError:
-        return None
+    @property
+    def default_kernel_params(self):
+        """Gets the default kernel parameters"""
+        return self._info.get('kernel_params', None)
 
-def get_known_distro_checksum(distro, distro_version, arch):
-    try:
-        return KNOWN_DISTROS.get(distro).get(distro_version).get(arch).get('checksum')
-    except AttributeError:
-        return None
+    @property
+    def pxeboot_url(self):
+        """Gets the repository url where pxeboot files can be found"""
+        return self._info.get('pxeboot_url', None)
 
+    @property
+    def checksum(self):
+        """Gets the cloud-image file checksum"""
+        return self._info.get('checksum', None)
+
+    @checksum.setter
+    def checksum(self, value):
+        self._info['checksum'] = value
 
 class LinuxTest(Test, LinuxSSHMixIn):
     """Facilitates having a cloud-image Linux based available.
@@ -374,24 +388,24 @@ class LinuxTest(Test, LinuxSSHMixIn):
     """
 
     timeout = 900
-    distro_checksum = None
+    distro = None
     username = 'root'
     password = 'password'
 
     def _set_distro(self):
-        distro = self.params.get(
+        distro_name = self.params.get(
             'distro',
             default=self._get_unique_tag_val('distro'))
-        if not distro:
-            distro = 'fedora'
-        self.distro = distro
+        if not distro_name:
+            distro_name = 'fedora'
 
         distro_version = self.params.get(
             'distro_version',
             default=self._get_unique_tag_val('distro_version'))
         if not distro_version:
             distro_version = '31'
-        self.distro_version = distro_version
+
+        self.distro = LinuxDistro(distro_name, distro_version, self.arch)
 
         # The distro checksum behaves differently than distro name and
         # version. First, it does not respect a tag with the same
@@ -400,13 +414,9 @@ def _set_distro(self):
         # order of precedence is: parameter, attribute and then value
         # from KNOWN_DISTROS.
         distro_checksum = self.params.get('distro_checksum',
-                                          default=self.distro_checksum)
-        if not distro_checksum:
-            distro_checksum = get_known_distro_checksum(self.distro,
-                                                        self.distro_version,
-                                                        self.arch)
+                                          default=None)
         if distro_checksum:
-            self.distro_checksum = distro_checksum
+            self.distro.checksum = distro_checksum
 
     def setUp(self, ssh_pubkey=None, network_device_type='virtio-net'):
         super(LinuxTest, self).setUp()
@@ -448,14 +458,14 @@ def download_boot(self):
         self.log.info('Downloading/preparing boot image')
         # Fedora 31 only provides ppc64le images
         image_arch = self.arch
-        if self.distro == 'fedora':
+        if self.distro.name == 'fedora':
             if image_arch == 'ppc64':
                 image_arch = 'ppc64le'
 
         try:
             boot = vmimage.get(
-                self.distro, arch=image_arch, version=self.distro_version,
-                checksum=self.distro_checksum,
+                self.distro.name, arch=image_arch, version=self.distro.version,
+                checksum=self.distro.checksum,
                 algorithm='sha256',
                 cache_dir=self.cache_dirs[0],
                 snapshot_dir=self.workdir)
@@ -491,12 +501,6 @@ def set_up_cloudinit(self, ssh_pubkey=None):
         cloudinit_iso = self.prepare_cloudinit(ssh_pubkey)
         self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
 
-    def get_default_kernel_params(self):
-        return get_known_distro_kernel_params(self.distro, self.distro_version, self.arch)
-
-    def get_pxeboot_url(self):
-        return get_known_distro_pxeboot_url(self.distro, self.distro_version, self.arch)
-
     def launch_and_wait(self, set_up_ssh_connection=True):
         self.vm.set_console()
         self.vm.launch()
diff --git a/tests/acceptance/intel_iommu.py b/tests/acceptance/intel_iommu.py
index 0b68d3c572..bf8dea6e4f 100644
--- a/tests/acceptance/intel_iommu.py
+++ b/tests/acceptance/intel_iommu.py
@@ -53,8 +53,8 @@ def common_vm_setup(self, custom_kernel=None):
         if custom_kernel is None:
             return
 
-        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
-        initrd_url = self.get_pxeboot_url() + 'initrd.img'
+        kernel_url = self.distro.pxeboot_url + 'vmlinuz'
+        initrd_url = self.distro.pxeboot_url + 'initrd.img'
         self.kernel_path = self.fetch_asset(kernel_url)
         self.initrd_path = self.fetch_asset(initrd_url)
 
@@ -78,7 +78,7 @@ def test_intel_iommu(self):
         self.vm.add_args('-device', 'intel-iommu,intremap=on')
         self.vm.add_args('-machine', 'kernel_irqchip=split')
 
-        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on'
+        self.kernel_params = self.distro.default_kernel_params + ' quiet intel_iommu=on'
         self.run_and_check()
 
     def test_intel_iommu_strict(self):
@@ -89,7 +89,7 @@ def test_intel_iommu_strict(self):
         self.common_vm_setup(True)
         self.vm.add_args('-device', 'intel-iommu,intremap=on')
         self.vm.add_args('-machine', 'kernel_irqchip=split')
-        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
+        self.kernel_params = self.distro.default_kernel_params + ' quiet intel_iommu=on,strict'
         self.run_and_check()
 
     def test_intel_iommu_strict_cm(self):
@@ -100,7 +100,7 @@ def test_intel_iommu_strict_cm(self):
         self.common_vm_setup(True)
         self.vm.add_args('-device', 'intel-iommu,intremap=on,caching-mode=on')
         self.vm.add_args('-machine', 'kernel_irqchip=split')
-        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
+        self.kernel_params = self.distro.default_kernel_params + ' quiet intel_iommu=on,strict'
         self.run_and_check()
 
     def test_intel_iommu_pt(self):
@@ -111,5 +111,5 @@ def test_intel_iommu_pt(self):
         self.common_vm_setup(True)
         self.vm.add_args('-device', 'intel-iommu,intremap=on')
         self.vm.add_args('-machine', 'kernel_irqchip=split')
-        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on iommu=pt'
+        self.kernel_params = self.distro.default_kernel_params + ' quiet intel_iommu=on iommu=pt'
         self.run_and_check()
diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
index c1d4b88e5f..72e26f002c 100644
--- a/tests/acceptance/smmu.py
+++ b/tests/acceptance/smmu.py
@@ -53,8 +53,8 @@ def common_vm_setup(self, custom_kernel=False):
         if custom_kernel is False:
             return
 
-        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
-        initrd_url = self.get_pxeboot_url() + 'initrd.img'
+        kernel_url = self.distro.pxeboot_url + 'vmlinuz'
+        initrd_url = self.distro.pxeboot_url + 'initrd.img'
         self.kernel_path = self.fetch_asset(kernel_url)
         self.initrd_path = self.fetch_asset(initrd_url)
 
@@ -86,7 +86,7 @@ def test_smmu_noril_passthrough(self):
         :avocado: tags=distro_version:31
         """
         self.common_vm_setup(True)
-        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
+        self.kernel_params = self.distro.default_kernel_params + ' iommu.passthrough=on'
         self.run_and_check()
 
     def test_smmu_noril_nostrict(self):
@@ -96,7 +96,7 @@ def test_smmu_noril_nostrict(self):
         :avocado: tags=distro_version:31
         """
         self.common_vm_setup(True)
-        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
+        self.kernel_params = self.distro.default_kernel_params + ' iommu.strict=0'
         self.run_and_check()
 
     # 5.8 kernel featuring range invalidation
@@ -118,7 +118,7 @@ def test_smmu_ril_passthrough(self):
         :avocado: tags=distro_version:33
         """
         self.common_vm_setup(True)
-        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
+        self.kernel_params = self.distro.default_kernel_params + ' iommu.passthrough=on'
         self.run_and_check()
 
     def test_smmu_ril_nostrict(self):
@@ -128,5 +128,5 @@ def test_smmu_ril_nostrict(self):
         :avocado: tags=distro_version:33
         """
         self.common_vm_setup(True)
-        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
+        self.kernel_params = self.distro.default_kernel_params + ' iommu.strict=0'
         self.run_and_check()
-- 
2.26.3

Re: [PATCH v4 1/4] Acceptance Tests: Add default kernel params and pxeboot url to the KNOWN_DISTROS collection

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 11:36 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> When running LinuxTests we may need to run the guest with
> custom params. It is practical to store the pxeboot URL
> and the default kernel params so that the
> tests just need to fetch those and augment the kernel params.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> v2 -> v3:
> - add fed32 and fed33 checksums
>
> v3 -> v4:
> - fix indent issues (Wainer)
> - use dl.fedoraproject.org instead of mirror
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 52 ++++++++++++++++++++++-
>  1 file changed, 50 insertions(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

Hi Cleber, all,

On 6/29/21 4:36 PM, Eric Auger wrote:
> This series adds ARM SMMU and Intel IOMMU functional
> tests using Fedora cloud-init images.
>
> ARM SMMU tests feature guests with and without RIL
> (range invalidation support) using respectively fedora 33
> and 31.  For each, we test the protection of virtio-net-pci
> and virtio-block-pci devices. Also strict=no and passthrough
> modes are tested. So there is a total of 6 tests.
>
> The series applies on top of Cleber's series:
> - [PATCH 0/3] Acceptance Tests: support choosing specific
>
> Note:
> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
> test_smmu_noril_nostrict) pass but the log reports:
> "WARN: Test passed but there were warnings during execution."
> This seems due to the lack of hash when fetching the kernel and
> initrd through fetch_asset():
> WARNI| No hash provided. Cannot check the asset file integrity.
I wanted to emphasize that point and wondered how we could fix that
issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
Any advice?

Best Regards

Eric
>
> History:
> v3 -> v4:
> - I added Wainer's refactoring of KNOWN_DISTROS
> into a class (last patch) and took into account his comments.
>
> v2 -> v3:
> - Added Intel IOMMU tests were added. Different
> operating modes are tested such as strict, caching mode, pt.
>
> Best Regards
>
> Eric
>
> The series and its dependencies can be found at:
> https://github.com/eauger/qemu/tree/avocado-qemu-v4
>
> Eric Auger (3):
>   Acceptance Tests: Add default kernel params and pxeboot url to the
>     KNOWN_DISTROS collection
>   avocado_qemu: Add SMMUv3 tests
>   avocado_qemu: Add Intel iommu tests
>
> Wainer dos Santos Moschetta (1):
>   avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
>
>  tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
>  tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
>  tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
>  3 files changed, 332 insertions(+), 33 deletions(-)
>  create mode 100644 tests/acceptance/intel_iommu.py
>  create mode 100644 tests/acceptance/smmu.py
>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 5:17 PM Eric Auger <eric.auger@redhat.com> wrote:
>
> Hi Cleber, all,
>
> On 6/29/21 4:36 PM, Eric Auger wrote:
> > This series adds ARM SMMU and Intel IOMMU functional
> > tests using Fedora cloud-init images.
> >
> > ARM SMMU tests feature guests with and without RIL
> > (range invalidation support) using respectively fedora 33
> > and 31.  For each, we test the protection of virtio-net-pci
> > and virtio-block-pci devices. Also strict=no and passthrough
> > modes are tested. So there is a total of 6 tests.
> >
> > The series applies on top of Cleber's series:
> > - [PATCH 0/3] Acceptance Tests: support choosing specific
> >
> > Note:
> > - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
> > test_smmu_noril_nostrict) pass but the log reports:
> > "WARN: Test passed but there were warnings during execution."
> > This seems due to the lack of hash when fetching the kernel and
> > initrd through fetch_asset():
> > WARNI| No hash provided. Cannot check the asset file integrity.
> I wanted to emphasize that point and wondered how we could fix that
> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
> Any advice?

Hi Eric,

We had that discussion some weeks ago regarding the WARN status of a
test when the file hash is not provided for the fetch call. We agreed
that a WARN is not a harmful status, and it would be okay.

When we got the request to add the message regarding a missing hash of
a downloaded file, we concluded that it would not make sense to set it
like a normal message in the logs because no one would open the logs
and see the message if the test succeed.

If you think a WARN may be considered a harmful status, let us know,
and we can try to adjust it so that users see the message when a hash
is not provided without setting the test status as WARN.

On the other hand, you can always add the hash if you have access to it.

I hope it helps somehow,

Willian

>
> Best Regards
>
> Eric
> >
> > History:
> > v3 -> v4:
> > - I added Wainer's refactoring of KNOWN_DISTROS
> > into a class (last patch) and took into account his comments.
> >
> > v2 -> v3:
> > - Added Intel IOMMU tests were added. Different
> > operating modes are tested such as strict, caching mode, pt.
> >
> > Best Regards
> >
> > Eric
> >
> > The series and its dependencies can be found at:
> > https://github.com/eauger/qemu/tree/avocado-qemu-v4
> >
> > Eric Auger (3):
> >   Acceptance Tests: Add default kernel params and pxeboot url to the
> >     KNOWN_DISTROS collection
> >   avocado_qemu: Add SMMUv3 tests
> >   avocado_qemu: Add Intel iommu tests
> >
> > Wainer dos Santos Moschetta (1):
> >   avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
> >
> >  tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
> >  tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
> >  tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
> >  3 files changed, 332 insertions(+), 33 deletions(-)
> >  create mode 100644 tests/acceptance/intel_iommu.py
> >  create mode 100644 tests/acceptance/smmu.py
> >
>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

Hi William,

On 6/29/21 10:38 PM, Willian Rampazzo wrote:
> On Tue, Jun 29, 2021 at 5:17 PM Eric Auger <eric.auger@redhat.com> wrote:
>> Hi Cleber, all,
>>
>> On 6/29/21 4:36 PM, Eric Auger wrote:
>>> This series adds ARM SMMU and Intel IOMMU functional
>>> tests using Fedora cloud-init images.
>>>
>>> ARM SMMU tests feature guests with and without RIL
>>> (range invalidation support) using respectively fedora 33
>>> and 31.  For each, we test the protection of virtio-net-pci
>>> and virtio-block-pci devices. Also strict=no and passthrough
>>> modes are tested. So there is a total of 6 tests.
>>>
>>> The series applies on top of Cleber's series:
>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>>
>>> Note:
>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>>> test_smmu_noril_nostrict) pass but the log reports:
>>> "WARN: Test passed but there were warnings during execution."
>>> This seems due to the lack of hash when fetching the kernel and
>>> initrd through fetch_asset():
>>> WARNI| No hash provided. Cannot check the asset file integrity.
>> I wanted to emphasize that point and wondered how we could fix that
>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
>> Any advice?
> Hi Eric,
>
> We had that discussion some weeks ago regarding the WARN status of a
> test when the file hash is not provided for the fetch call. We agreed
> that a WARN is not a harmful status, and it would be okay.
>
> When we got the request to add the message regarding a missing hash of
> a downloaded file, we concluded that it would not make sense to set it
> like a normal message in the logs because no one would open the logs
> and see the message if the test succeed.
>
> If you think a WARN may be considered a harmful status, let us know,
> and we can try to adjust it so that users see the message when a hash
> is not provided without setting the test status as WARN.
>
> On the other hand, you can always add the hash if you have access to it.
>
> I hope it helps somehow,
It does, thank you for your reply.

Maybe we should tag the tests with warnings as PASS in the last summary
RESULT line and keep the individual test result lines as they are.

Currently it is not obvious the tests has passed if you look at the last
line. See my logs below.



 (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (106.83 s)
 (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough: WARN:
Test passed but there were warnings during execution. Check the log for
details. (112.23 s)
 (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
Test passed but there were warnings during execution. Check the log for
details. (110.77 s)
 (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (135.18 s)
 (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
Test passed but there were warnings during execution. Check the log for
details. (115.98 s)
 (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN: Test
passed but there were warnings during execution. Check the log for
details. (137.16 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0 |
CANCEL 0

Thanks

Eric

>
> Willian
>
>> Best Regards
>>
>> Eric
>>> History:
>>> v3 -> v4:
>>> - I added Wainer's refactoring of KNOWN_DISTROS
>>> into a class (last patch) and took into account his comments.
>>>
>>> v2 -> v3:
>>> - Added Intel IOMMU tests were added. Different
>>> operating modes are tested such as strict, caching mode, pt.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> The series and its dependencies can be found at:
>>> https://github.com/eauger/qemu/tree/avocado-qemu-v4
>>>
>>> Eric Auger (3):
>>>   Acceptance Tests: Add default kernel params and pxeboot url to the
>>>     KNOWN_DISTROS collection
>>>   avocado_qemu: Add SMMUv3 tests
>>>   avocado_qemu: Add Intel iommu tests
>>>
>>> Wainer dos Santos Moschetta (1):
>>>   avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
>>>
>>>  tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
>>>  tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
>>>  tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
>>>  3 files changed, 332 insertions(+), 33 deletions(-)
>>>  create mode 100644 tests/acceptance/intel_iommu.py
>>>  create mode 100644 tests/acceptance/smmu.py
>>>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Wainer dos Santos Moschetta]

Hi,

On 6/29/21 5:17 PM, Eric Auger wrote:
> Hi Cleber, all,
>
> On 6/29/21 4:36 PM, Eric Auger wrote:
>> This series adds ARM SMMU and Intel IOMMU functional
>> tests using Fedora cloud-init images.
>>
>> ARM SMMU tests feature guests with and without RIL
>> (range invalidation support) using respectively fedora 33
>> and 31.  For each, we test the protection of virtio-net-pci
>> and virtio-block-pci devices. Also strict=no and passthrough
>> modes are tested. So there is a total of 6 tests.
>>
>> The series applies on top of Cleber's series:
>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>
>> Note:
>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>> test_smmu_noril_nostrict) pass but the log reports:
>> "WARN: Test passed but there were warnings during execution."
>> This seems due to the lack of hash when fetching the kernel and
>> initrd through fetch_asset():
>> WARNI| No hash provided. Cannot check the asset file integrity.
> I wanted to emphasize that point and wondered how we could fix that
> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
> Any advice?

As Willian mentioned somewhere, to supress the WARN you can pass the 
kernel and initrd checksums (sha1) to the fetch_asset() method.

Below is an draft implementation. It would need to fill out the 
remaining checksums and adjust the `smmu.py` tests.

- Wainer

----

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index 00eb0bfcc8..83637e2654 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -312,6 +312,8 @@ class LinuxDistro:
                  {'checksum': 
'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
                  'pxeboot_url': 
"https://archives.fedoraproject.org/pub/archive/fedora/"
"linux/releases/31/Everything/x86_64/os/images/pxeboot/",
+                'pxeboot_initrd_chksum': 
'dd0340a1b39bd28f88532babd4581c67649ec5b1',
+                'pxeboot_vmlinuz_chksum': 
'5b6f6876e1b5bda314f93893271da0d5777b1f3c',
                  'kernel_params': 
"root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
                                "no_timer_check net.ifnames=0 "
                                "console=tty1 console=ttyS0,115200n8"},
@@ -371,6 +373,16 @@ def pxeboot_url(self):
          """Gets the repository url where pxeboot files can be found"""
          return self._info.get('pxeboot_url', None)

+    @property
+    def pxeboot_initrd_chksum(self):
+        """Gets the pxeboot initrd file checksum"""
+        return self._info.get('pxeboot_initrd_chksum', None)
+
+    @property
+    def pxeboot_vmlinuz_chksum(self):
+        """Gets the pxeboot vmlinuz file checksum"""
+        return self._info.get('pxeboot_vmlinuz_chksum', None)
+
      @property
      def checksum(self):
          """Gets the cloud-image file checksum"""
diff --git a/tests/acceptance/intel_iommu.py 
b/tests/acceptance/intel_iommu.py
index bf8dea6e4f..a2f38ee2e9 100644
--- a/tests/acceptance/intel_iommu.py
+++ b/tests/acceptance/intel_iommu.py
@@ -55,8 +55,10 @@ def common_vm_setup(self, custom_kernel=None):

          kernel_url = self.distro.pxeboot_url + 'vmlinuz'
          initrd_url = self.distro.pxeboot_url + 'initrd.img'
-        self.kernel_path = self.fetch_asset(kernel_url)
-        self.initrd_path = self.fetch_asset(initrd_url)
+        self.kernel_path = self.fetch_asset(kernel_url,
+ asset_hash=self.distro.pxeboot_vmlinuz_chksum)
+        self.initrd_path = self.fetch_asset(initrd_url,
+ asset_hash=self.distro.pxeboot_initrd_chksum)

      def run_and_check(self):
          if self.kernel_path:

>
> Best Regards
>
> Eric
>> History:
>> v3 -> v4:
>> - I added Wainer's refactoring of KNOWN_DISTROS
>> into a class (last patch) and took into account his comments.
>>
>> v2 -> v3:
>> - Added Intel IOMMU tests were added. Different
>> operating modes are tested such as strict, caching mode, pt.
>>
>> Best Regards
>>
>> Eric
>>
>> The series and its dependencies can be found at:
>> https://github.com/eauger/qemu/tree/avocado-qemu-v4
>>
>> Eric Auger (3):
>>    Acceptance Tests: Add default kernel params and pxeboot url to the
>>      KNOWN_DISTROS collection
>>    avocado_qemu: Add SMMUv3 tests
>>    avocado_qemu: Add Intel iommu tests
>>
>> Wainer dos Santos Moschetta (1):
>>    avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
>>
>>   tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
>>   tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
>>   tests/acceptance/smmu.py                  | 132 ++++++++++++++++++++++
>>   3 files changed, 332 insertions(+), 33 deletions(-)
>>   create mode 100644 tests/acceptance/intel_iommu.py
>>   create mode 100644 tests/acceptance/smmu.py
>>

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Wainer dos Santos Moschetta]

Hi,

On 6/29/21 11:36 AM, Eric Auger wrote:
> Add new tests checking the good behavior of the SMMUv3 protecting
> 2 virtio pci devices (block and net). We check the guest boots and
> we are able to install a package. Different guest configs are tested:
> standard, passthrough an strict=0. This is tested with both fedora 31 and
> 33. The former uses a 5.3 kernel without range invalidation whereas the
> latter uses a 5.8 kernel that features range invalidation.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> v3 -> v4:
> - add tags for machine, distro in the class
> - removed smp and memory overrides
> - set default param value of common_vm_setup to False
>
> v1 -> v2:
> - removed ssh import
> - combined add_command_args() and common_vm_setup()
> - moved tags in class' docstring and added tags=arch:aarch64
> - use self.get_default_kernel_params()
> - added RIL tests with fed33 + introduce new tags
> ---
>   tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 132 insertions(+)
>   create mode 100644 tests/acceptance/smmu.py

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

I tested it in a Fedora 32 aarch64 host. The execution output:

# ./tests/venv/bin/avocado run tests/acceptance/smmu.py
JOB ID     : 1625038f5a2ae17c8ba6c503d3df8661ff528942
JOB LOG    : /root/avocado/job-results/job-2021-07-01T13.38-1625038/job.log
  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (175.54 s)
  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (168.39 s)
  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (161.58 s)
  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (150.85 s)
  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (177.56 s)
  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN: Test 
passed but there were warnings during execution. Check the log for 
details. (190.86 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0 | 
CANCEL 0
JOB TIME   : 1026.50 s

One thing that caught my attention was the amount of time spent on each 
test. It spend more than 2 minutes on the package installation 
(`self.ssh_command('dnf -y install numactl-devel')`) in the guest.

Without that operation, it runs way faster:

# ./tests/venv/bin/avocado run tests/acceptance/smmu.py
JOB ID     : 24f22f99169ece37df64d72d2eb373921f378aac
JOB LOG    : /root/avocado/job-results/job-2021-07-01T13.28-24f22f9/job.log
  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (39.61 s)
  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (48.32 s)
  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (48.10 s)
  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (39.22 s)
  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN: 
Test passed but there were warnings during execution. Check the log for 
details. (52.92 s)
  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN: Test 
passed but there were warnings during execution. Check the log for 
details. (50.96 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0 | 
CANCEL 0
JOB TIME   : 280.62 s

Install a package seems a good exerciser for disk I/O and networking, 
but maybe you can use another method for the sake of speed up the tests?

- Wainer

>
> diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
> new file mode 100644
> index 0000000000..c1d4b88e5f
> --- /dev/null
> +++ b/tests/acceptance/smmu.py
> @@ -0,0 +1,132 @@
> +# SMMUv3 Functional tests
> +#
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Eric Auger <eric.auger@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado_qemu import LinuxTest, BUILD_DIR
> +
> +class SMMU(LinuxTest):
> +    """
> +    :avocado: tags=accel:kvm
> +    :avocado: tags=cpu:host
> +    :avocado: tags=arch:aarch64
> +    :avocado: tags=machine:virt
> +    :avocado: tags=distro:fedora
> +    :avocado: tags=smmu
> +    """
> +
> +    IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
> +    kernel_path = None
> +    initrd_path = None
> +    kernel_params = None
> +
> +    def set_up_boot(self):
> +        path = self.download_boot()
> +        self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
> +        self.vm.add_args('-drive',
> +                         'file=%s,if=none,cache=writethrough,id=drv0' % path)
> +
> +    def setUp(self):
> +        super(SMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON)
> +
> +    def common_vm_setup(self, custom_kernel=False):
> +        self.require_accelerator("kvm")
> +        self.vm.add_args("-accel", "kvm")
> +        self.vm.add_args("-cpu", "host")
> +        self.vm.add_args("-machine", "iommu=smmuv3")
> +        self.vm.add_args("-d", "guest_errors")
> +        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
> +                         'edk2-aarch64-code.fd'))
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object',
> +                         'rng-random,id=rng0,filename=/dev/urandom')
> +
> +        if custom_kernel is False:
> +            return
> +
> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
> +        self.kernel_path = self.fetch_asset(kernel_url)
> +        self.initrd_path = self.fetch_asset(initrd_url)
> +
> +    def run_and_check(self):
> +        if self.kernel_path:
> +            self.vm.add_args('-kernel', self.kernel_path,
> +                             '-append', self.kernel_params,
> +                             '-initrd', self.initrd_path)
> +        self.launch_and_wait()
> +        self.ssh_command('cat /proc/cmdline')
> +        self.ssh_command('dnf -y install numactl-devel')
> +
> +
> +    # 5.3 kernel without RIL #
> +
> +    def test_smmu_noril(self):
> +        """
> +        :avocado: tags=smmu_noril
> +        :avocado: tags=smmu_noril_tests
> +        :avocado: tags=distro_version:31
> +        """
> +        self.common_vm_setup()
> +        self.run_and_check()
> +
> +    def test_smmu_noril_passthrough(self):
> +        """
> +        :avocado: tags=smmu_noril_passthrough
> +        :avocado: tags=smmu_noril_tests
> +        :avocado: tags=distro_version:31
> +        """
> +        self.common_vm_setup(True)
> +        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
> +        self.run_and_check()
> +
> +    def test_smmu_noril_nostrict(self):
> +        """
> +        :avocado: tags=smmu_noril_nostrict
> +        :avocado: tags=smmu_noril_tests
> +        :avocado: tags=distro_version:31
> +        """
> +        self.common_vm_setup(True)
> +        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
> +        self.run_and_check()
> +
> +    # 5.8 kernel featuring range invalidation
> +    # >= v5.7 kernel
> +
> +    def test_smmu_ril(self):
> +        """
> +        :avocado: tags=smmu_ril
> +        :avocado: tags=smmu_ril_tests
> +        :avocado: tags=distro_version:33
> +        """
> +        self.common_vm_setup()
> +        self.run_and_check()
> +
> +    def test_smmu_ril_passthrough(self):
> +        """
> +        :avocado: tags=smmu_ril_passthrough
> +        :avocado: tags=smmu_ril_tests
> +        :avocado: tags=distro_version:33
> +        """
> +        self.common_vm_setup(True)
> +        self.kernel_params = self.get_default_kernel_params() + ' iommu.passthrough=on'
> +        self.run_and_check()
> +
> +    def test_smmu_ril_nostrict(self):
> +        """
> +        :avocado: tags=smmu_ril_nostrict
> +        :avocado: tags=smmu_ril_tests
> +        :avocado: tags=distro_version:33
> +        """
> +        self.common_vm_setup(True)
> +        self.kernel_params = self.get_default_kernel_params() + ' iommu.strict=0'
> +        self.run_and_check()

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

Hi Wainer,

On 7/1/21 1:22 AM, Wainer dos Santos Moschetta wrote:
> Hi,
>
> On 6/29/21 5:17 PM, Eric Auger wrote:
>> Hi Cleber, all,
>>
>> On 6/29/21 4:36 PM, Eric Auger wrote:
>>> This series adds ARM SMMU and Intel IOMMU functional
>>> tests using Fedora cloud-init images.
>>>
>>> ARM SMMU tests feature guests with and without RIL
>>> (range invalidation support) using respectively fedora 33
>>> and 31.  For each, we test the protection of virtio-net-pci
>>> and virtio-block-pci devices. Also strict=no and passthrough
>>> modes are tested. So there is a total of 6 tests.
>>>
>>> The series applies on top of Cleber's series:
>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>>
>>> Note:
>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>>> test_smmu_noril_nostrict) pass but the log reports:
>>> "WARN: Test passed but there were warnings during execution."
>>> This seems due to the lack of hash when fetching the kernel and
>>> initrd through fetch_asset():
>>> WARNI| No hash provided. Cannot check the asset file integrity.
>> I wanted to emphasize that point and wondered how we could fix that
>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
>> Any advice?
>
> As Willian mentioned somewhere, to supress the WARN you can pass the
> kernel and initrd checksums (sha1) to the fetch_asset() method.
>
> Below is an draft implementation. It would need to fill out the
> remaining checksums and adjust the `smmu.py` tests.
>
> - Wainer
>
> ----
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py
> b/tests/acceptance/avocado_qemu/__init__.py
> index 00eb0bfcc8..83637e2654 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -312,6 +312,8 @@ class LinuxDistro:
>                  {'checksum':
> 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
>                  'pxeboot_url':
> "https://archives.fedoraproject.org/pub/archive/fedora/"
> "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
> +                'pxeboot_initrd_chksum':
> 'dd0340a1b39bd28f88532babd4581c67649ec5b1',
> +                'pxeboot_vmlinuz_chksum':
> '5b6f6876e1b5bda314f93893271da0d5777b1f3c',
where did you get the checksum? I don't see any at the URL? Did you
generate it yourself?

Thanks

Eric
>                  'kernel_params':
> "root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
>                                "no_timer_check net.ifnames=0 "
>                                "console=tty1 console=ttyS0,115200n8"},
> @@ -371,6 +373,16 @@ def pxeboot_url(self):
>          """Gets the repository url where pxeboot files can be found"""
>          return self._info.get('pxeboot_url', None)
>
> +    @property
> +    def pxeboot_initrd_chksum(self):
> +        """Gets the pxeboot initrd file checksum"""
> +        return self._info.get('pxeboot_initrd_chksum', None)
> +
> +    @property
> +    def pxeboot_vmlinuz_chksum(self):
> +        """Gets the pxeboot vmlinuz file checksum"""
> +        return self._info.get('pxeboot_vmlinuz_chksum', None)
> +
>      @property
>      def checksum(self):
>          """Gets the cloud-image file checksum"""
> diff --git a/tests/acceptance/intel_iommu.py
> b/tests/acceptance/intel_iommu.py
> index bf8dea6e4f..a2f38ee2e9 100644
> --- a/tests/acceptance/intel_iommu.py
> +++ b/tests/acceptance/intel_iommu.py
> @@ -55,8 +55,10 @@ def common_vm_setup(self, custom_kernel=None):
>
>          kernel_url = self.distro.pxeboot_url + 'vmlinuz'
>          initrd_url = self.distro.pxeboot_url + 'initrd.img'
> -        self.kernel_path = self.fetch_asset(kernel_url)
> -        self.initrd_path = self.fetch_asset(initrd_url)
> +        self.kernel_path = self.fetch_asset(kernel_url,
> + asset_hash=self.distro.pxeboot_vmlinuz_chksum)
> +        self.initrd_path = self.fetch_asset(initrd_url,
> + asset_hash=self.distro.pxeboot_initrd_chksum)
>
>      def run_and_check(self):
>          if self.kernel_path:
>
>>
>> Best Regards
>>
>> Eric
>>> History:
>>> v3 -> v4:
>>> - I added Wainer's refactoring of KNOWN_DISTROS
>>> into a class (last patch) and took into account his comments.
>>>
>>> v2 -> v3:
>>> - Added Intel IOMMU tests were added. Different
>>> operating modes are tested such as strict, caching mode, pt.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> The series and its dependencies can be found at:
>>> https://github.com/eauger/qemu/tree/avocado-qemu-v4
>>>
>>> Eric Auger (3):
>>>    Acceptance Tests: Add default kernel params and pxeboot url to the
>>>      KNOWN_DISTROS collection
>>>    avocado_qemu: Add SMMUv3 tests
>>>    avocado_qemu: Add Intel iommu tests
>>>
>>> Wainer dos Santos Moschetta (1):
>>>    avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
>>>
>>>   tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
>>>   tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
>>>   tests/acceptance/smmu.py                  | 132
>>> ++++++++++++++++++++++
>>>   3 files changed, 332 insertions(+), 33 deletions(-)
>>>   create mode 100644 tests/acceptance/intel_iommu.py
>>>   create mode 100644 tests/acceptance/smmu.py
>>>
>

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Eric Auger]

Hi Wainer,

On 7/1/21 8:13 PM, Wainer dos Santos Moschetta wrote:
> Hi,
>
> On 6/29/21 11:36 AM, Eric Auger wrote:
>> Add new tests checking the good behavior of the SMMUv3 protecting
>> 2 virtio pci devices (block and net). We check the guest boots and
>> we are able to install a package. Different guest configs are tested:
>> standard, passthrough an strict=0. This is tested with both fedora 31
>> and
>> 33. The former uses a 5.3 kernel without range invalidation whereas the
>> latter uses a 5.8 kernel that features range invalidation.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>>
>> v3 -> v4:
>> - add tags for machine, distro in the class
>> - removed smp and memory overrides
>> - set default param value of common_vm_setup to False
>>
>> v1 -> v2:
>> - removed ssh import
>> - combined add_command_args() and common_vm_setup()
>> - moved tags in class' docstring and added tags=arch:aarch64
>> - use self.get_default_kernel_params()
>> - added RIL tests with fed33 + introduce new tags
>> ---
>>   tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 132 insertions(+)
>>   create mode 100644 tests/acceptance/smmu.py
>
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> I tested it in a Fedora 32 aarch64 host. The execution output:
>
> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
> JOB ID     : 1625038f5a2ae17c8ba6c503d3df8661ff528942
> JOB LOG    :
> /root/avocado/job-results/job-2021-07-01T13.38-1625038/job.log
>  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (175.54 s)
>  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> WARN: Test passed but there were warnings during execution. Check the
> log for details. (168.39 s)
>  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (161.58 s)
>  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (150.85 s)
>  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (177.56 s)
>  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (190.86 s)
> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
> | CANCEL 0
> JOB TIME   : 1026.50 s
>
> One thing that caught my attention was the amount of time spent on
> each test. It spend more than 2 minutes on the package installation
> (`self.ssh_command('dnf -y install numactl-devel')`) in the guest.
>
> Without that operation, it runs way faster:
>
> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
> JOB ID     : 24f22f99169ece37df64d72d2eb373921f378aac
> JOB LOG    :
> /root/avocado/job-results/job-2021-07-01T13.28-24f22f9/job.log
>  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (39.61 s)
>  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> WARN: Test passed but there were warnings during execution. Check the
> log for details. (48.32 s)
>  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (48.10 s)
>  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (39.22 s)
>  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (52.92 s)
>  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
> Test passed but there were warnings during execution. Check the log
> for details. (50.96 s)
> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
> | CANCEL 0
> JOB TIME   : 280.62 s
>
> Install a package seems a good exerciser for disk I/O and networking,
> but maybe you can use another method for the sake of speed up the tests?

As discussed earlier with Cleber, I am aware the test duration is long
but it was useful finding bugs for SMMU with range invalidation. such a
bug could not be hit with a single boot + ping for instance.

Maybe we should have a mechanism that allows to put some tests out of
the automatic CI?

Thanks

Eric
>
> - Wainer
>
>>
>> diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
>> new file mode 100644
>> index 0000000000..c1d4b88e5f
>> --- /dev/null
>> +++ b/tests/acceptance/smmu.py
>> @@ -0,0 +1,132 @@
>> +# SMMUv3 Functional tests
>> +#
>> +# Copyright (c) 2021 Red Hat, Inc.
>> +#
>> +# Author:
>> +#  Eric Auger <eric.auger@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>> +# later.  See the COPYING file in the top-level directory.
>> +
>> +import os
>> +
>> +from avocado_qemu import LinuxTest, BUILD_DIR
>> +
>> +class SMMU(LinuxTest):
>> +    """
>> +    :avocado: tags=accel:kvm
>> +    :avocado: tags=cpu:host
>> +    :avocado: tags=arch:aarch64
>> +    :avocado: tags=machine:virt
>> +    :avocado: tags=distro:fedora
>> +    :avocado: tags=smmu
>> +    """
>> +
>> +    IOMMU_ADDON =
>> ',iommu_platform=on,disable-modern=off,disable-legacy=on'
>> +    kernel_path = None
>> +    initrd_path = None
>> +    kernel_params = None
>> +
>> +    def set_up_boot(self):
>> +        path = self.download_boot()
>> +        self.vm.add_args('-device',
>> 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
>> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
>> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
>> +        self.vm.add_args('-drive',
>> +                        
>> 'file=%s,if=none,cache=writethrough,id=drv0' % path)
>> +
>> +    def setUp(self):
>> +        super(SMMU, self).setUp(None, 'virtio-net-pci' +
>> self.IOMMU_ADDON)
>> +
>> +    def common_vm_setup(self, custom_kernel=False):
>> +        self.require_accelerator("kvm")
>> +        self.vm.add_args("-accel", "kvm")
>> +        self.vm.add_args("-cpu", "host")
>> +        self.vm.add_args("-machine", "iommu=smmuv3")
>> +        self.vm.add_args("-d", "guest_errors")
>> +        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
>> +                         'edk2-aarch64-code.fd'))
>> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
>> +        self.vm.add_args('-object',
>> +                         'rng-random,id=rng0,filename=/dev/urandom')
>> +
>> +        if custom_kernel is False:
>> +            return
>> +
>> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
>> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
>> +        self.kernel_path = self.fetch_asset(kernel_url)
>> +        self.initrd_path = self.fetch_asset(initrd_url)
>> +
>> +    def run_and_check(self):
>> +        if self.kernel_path:
>> +            self.vm.add_args('-kernel', self.kernel_path,
>> +                             '-append', self.kernel_params,
>> +                             '-initrd', self.initrd_path)
>> +        self.launch_and_wait()
>> +        self.ssh_command('cat /proc/cmdline')
>> +        self.ssh_command('dnf -y install numactl-devel')
>> +
>> +
>> +    # 5.3 kernel without RIL #
>> +
>> +    def test_smmu_noril(self):
>> +        """
>> +        :avocado: tags=smmu_noril
>> +        :avocado: tags=smmu_noril_tests
>> +        :avocado: tags=distro_version:31
>> +        """
>> +        self.common_vm_setup()
>> +        self.run_and_check()
>> +
>> +    def test_smmu_noril_passthrough(self):
>> +        """
>> +        :avocado: tags=smmu_noril_passthrough
>> +        :avocado: tags=smmu_noril_tests
>> +        :avocado: tags=distro_version:31
>> +        """
>> +        self.common_vm_setup(True)
>> +        self.kernel_params = self.get_default_kernel_params() + '
>> iommu.passthrough=on'
>> +        self.run_and_check()
>> +
>> +    def test_smmu_noril_nostrict(self):
>> +        """
>> +        :avocado: tags=smmu_noril_nostrict
>> +        :avocado: tags=smmu_noril_tests
>> +        :avocado: tags=distro_version:31
>> +        """
>> +        self.common_vm_setup(True)
>> +        self.kernel_params = self.get_default_kernel_params() + '
>> iommu.strict=0'
>> +        self.run_and_check()
>> +
>> +    # 5.8 kernel featuring range invalidation
>> +    # >= v5.7 kernel
>> +
>> +    def test_smmu_ril(self):
>> +        """
>> +        :avocado: tags=smmu_ril
>> +        :avocado: tags=smmu_ril_tests
>> +        :avocado: tags=distro_version:33
>> +        """
>> +        self.common_vm_setup()
>> +        self.run_and_check()
>> +
>> +    def test_smmu_ril_passthrough(self):
>> +        """
>> +        :avocado: tags=smmu_ril_passthrough
>> +        :avocado: tags=smmu_ril_tests
>> +        :avocado: tags=distro_version:33
>> +        """
>> +        self.common_vm_setup(True)
>> +        self.kernel_params = self.get_default_kernel_params() + '
>> iommu.passthrough=on'
>> +        self.run_and_check()
>> +
>> +    def test_smmu_ril_nostrict(self):
>> +        """
>> +        :avocado: tags=smmu_ril_nostrict
>> +        :avocado: tags=smmu_ril_tests
>> +        :avocado: tags=distro_version:33
>> +        """
>> +        self.common_vm_setup(True)
>> +        self.kernel_params = self.get_default_kernel_params() + '
>> iommu.strict=0'
>> +        self.run_and_check()
>

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Willian Rampazzo]

Hi Eric,

On Mon, Jul 5, 2021 at 5:00 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> Hi Wainer,
>
> On 7/1/21 8:13 PM, Wainer dos Santos Moschetta wrote:
> > Hi,
> >
> > On 6/29/21 11:36 AM, Eric Auger wrote:
> >> Add new tests checking the good behavior of the SMMUv3 protecting
> >> 2 virtio pci devices (block and net). We check the guest boots and
> >> we are able to install a package. Different guest configs are tested:
> >> standard, passthrough an strict=0. This is tested with both fedora 31
> >> and
> >> 33. The former uses a 5.3 kernel without range invalidation whereas the
> >> latter uses a 5.8 kernel that features range invalidation.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >>
> >> v3 -> v4:
> >> - add tags for machine, distro in the class
> >> - removed smp and memory overrides
> >> - set default param value of common_vm_setup to False
> >>
> >> v1 -> v2:
> >> - removed ssh import
> >> - combined add_command_args() and common_vm_setup()
> >> - moved tags in class' docstring and added tags=arch:aarch64
> >> - use self.get_default_kernel_params()
> >> - added RIL tests with fed33 + introduce new tags
> >> ---
> >>   tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 132 insertions(+)
> >>   create mode 100644 tests/acceptance/smmu.py
> >
> > Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> >
> > Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> >
> > I tested it in a Fedora 32 aarch64 host. The execution output:
> >
> > # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
> > JOB ID     : 1625038f5a2ae17c8ba6c503d3df8661ff528942
> > JOB LOG    :
> > /root/avocado/job-results/job-2021-07-01T13.38-1625038/job.log
> >  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (175.54 s)
> >  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> > WARN: Test passed but there were warnings during execution. Check the
> > log for details. (168.39 s)
> >  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (161.58 s)
> >  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (150.85 s)
> >  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (177.56 s)
> >  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (190.86 s)
> > RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
> > | CANCEL 0
> > JOB TIME   : 1026.50 s
> >
> > One thing that caught my attention was the amount of time spent on
> > each test. It spend more than 2 minutes on the package installation
> > (`self.ssh_command('dnf -y install numactl-devel')`) in the guest.
> >
> > Without that operation, it runs way faster:
> >
> > # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
> > JOB ID     : 24f22f99169ece37df64d72d2eb373921f378aac
> > JOB LOG    :
> > /root/avocado/job-results/job-2021-07-01T13.28-24f22f9/job.log
> >  (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (39.61 s)
> >  (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> > WARN: Test passed but there were warnings during execution. Check the
> > log for details. (48.32 s)
> >  (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (48.10 s)
> >  (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (39.22 s)
> >  (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (52.92 s)
> >  (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
> > Test passed but there were warnings during execution. Check the log
> > for details. (50.96 s)
> > RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
> > | CANCEL 0
> > JOB TIME   : 280.62 s
> >
> > Install a package seems a good exerciser for disk I/O and networking,
> > but maybe you can use another method for the sake of speed up the tests?
>
> As discussed earlier with Cleber, I am aware the test duration is long
> but it was useful finding bugs for SMMU with range invalidation. such a
> bug could not be hit with a single boot + ping for instance.
>
> Maybe we should have a mechanism that allows to put some tests out of
> the automatic CI?
>

You can use the skipIf decorator in the class. See here:
https://gitlab.com/willianrampazzo/qemu/-/commit/6f249845827ed041b55d275a8cb803666ac3c7af

Regards,

> Thanks
>
> Eric
> >
> > - Wainer
> >
> >>
> >> diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
> >> new file mode 100644
> >> index 0000000000..c1d4b88e5f
> >> --- /dev/null
> >> +++ b/tests/acceptance/smmu.py
> >> @@ -0,0 +1,132 @@
> >> +# SMMUv3 Functional tests
> >> +#
> >> +# Copyright (c) 2021 Red Hat, Inc.
> >> +#
> >> +# Author:
> >> +#  Eric Auger <eric.auger@redhat.com>
> >> +#
> >> +# This work is licensed under the terms of the GNU GPL, version 2 or
> >> +# later.  See the COPYING file in the top-level directory.
> >> +
> >> +import os
> >> +
> >> +from avocado_qemu import LinuxTest, BUILD_DIR
> >> +
> >> +class SMMU(LinuxTest):
> >> +    """
> >> +    :avocado: tags=accel:kvm
> >> +    :avocado: tags=cpu:host
> >> +    :avocado: tags=arch:aarch64
> >> +    :avocado: tags=machine:virt
> >> +    :avocado: tags=distro:fedora
> >> +    :avocado: tags=smmu
> >> +    """
> >> +
> >> +    IOMMU_ADDON =
> >> ',iommu_platform=on,disable-modern=off,disable-legacy=on'
> >> +    kernel_path = None
> >> +    initrd_path = None
> >> +    kernel_params = None
> >> +
> >> +    def set_up_boot(self):
> >> +        path = self.download_boot()
> >> +        self.vm.add_args('-device',
> >> 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
> >> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
> >> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
> >> +        self.vm.add_args('-drive',
> >> +
> >> 'file=%s,if=none,cache=writethrough,id=drv0' % path)
> >> +
> >> +    def setUp(self):
> >> +        super(SMMU, self).setUp(None, 'virtio-net-pci' +
> >> self.IOMMU_ADDON)
> >> +
> >> +    def common_vm_setup(self, custom_kernel=False):
> >> +        self.require_accelerator("kvm")
> >> +        self.vm.add_args("-accel", "kvm")
> >> +        self.vm.add_args("-cpu", "host")
> >> +        self.vm.add_args("-machine", "iommu=smmuv3")
> >> +        self.vm.add_args("-d", "guest_errors")
> >> +        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
> >> +                         'edk2-aarch64-code.fd'))
> >> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> >> +        self.vm.add_args('-object',
> >> +                         'rng-random,id=rng0,filename=/dev/urandom')
> >> +
> >> +        if custom_kernel is False:
> >> +            return
> >> +
> >> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
> >> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
> >> +        self.kernel_path = self.fetch_asset(kernel_url)
> >> +        self.initrd_path = self.fetch_asset(initrd_url)
> >> +
> >> +    def run_and_check(self):
> >> +        if self.kernel_path:
> >> +            self.vm.add_args('-kernel', self.kernel_path,
> >> +                             '-append', self.kernel_params,
> >> +                             '-initrd', self.initrd_path)
> >> +        self.launch_and_wait()
> >> +        self.ssh_command('cat /proc/cmdline')
> >> +        self.ssh_command('dnf -y install numactl-devel')
> >> +
> >> +
> >> +    # 5.3 kernel without RIL #
> >> +
> >> +    def test_smmu_noril(self):
> >> +        """
> >> +        :avocado: tags=smmu_noril
> >> +        :avocado: tags=smmu_noril_tests
> >> +        :avocado: tags=distro_version:31
> >> +        """
> >> +        self.common_vm_setup()
> >> +        self.run_and_check()
> >> +
> >> +    def test_smmu_noril_passthrough(self):
> >> +        """
> >> +        :avocado: tags=smmu_noril_passthrough
> >> +        :avocado: tags=smmu_noril_tests
> >> +        :avocado: tags=distro_version:31
> >> +        """
> >> +        self.common_vm_setup(True)
> >> +        self.kernel_params = self.get_default_kernel_params() + '
> >> iommu.passthrough=on'
> >> +        self.run_and_check()
> >> +
> >> +    def test_smmu_noril_nostrict(self):
> >> +        """
> >> +        :avocado: tags=smmu_noril_nostrict
> >> +        :avocado: tags=smmu_noril_tests
> >> +        :avocado: tags=distro_version:31
> >> +        """
> >> +        self.common_vm_setup(True)
> >> +        self.kernel_params = self.get_default_kernel_params() + '
> >> iommu.strict=0'
> >> +        self.run_and_check()
> >> +
> >> +    # 5.8 kernel featuring range invalidation
> >> +    # >= v5.7 kernel
> >> +
> >> +    def test_smmu_ril(self):
> >> +        """
> >> +        :avocado: tags=smmu_ril
> >> +        :avocado: tags=smmu_ril_tests
> >> +        :avocado: tags=distro_version:33
> >> +        """
> >> +        self.common_vm_setup()
> >> +        self.run_and_check()
> >> +
> >> +    def test_smmu_ril_passthrough(self):
> >> +        """
> >> +        :avocado: tags=smmu_ril_passthrough
> >> +        :avocado: tags=smmu_ril_tests
> >> +        :avocado: tags=distro_version:33
> >> +        """
> >> +        self.common_vm_setup(True)
> >> +        self.kernel_params = self.get_default_kernel_params() + '
> >> iommu.passthrough=on'
> >> +        self.run_and_check()
> >> +
> >> +    def test_smmu_ril_nostrict(self):
> >> +        """
> >> +        :avocado: tags=smmu_ril_nostrict
> >> +        :avocado: tags=smmu_ril_tests
> >> +        :avocado: tags=distro_version:33
> >> +        """
> >> +        self.common_vm_setup(True)
> >> +        self.kernel_params = self.get_default_kernel_params() + '
> >> iommu.strict=0'
> >> +        self.run_and_check()
> >
>

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 11:36 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> Add new tests checking the good behavior of the SMMUv3 protecting
> 2 virtio pci devices (block and net). We check the guest boots and
> we are able to install a package. Different guest configs are tested:
> standard, passthrough an strict=0. This is tested with both fedora 31 and
> 33. The former uses a 5.3 kernel without range invalidation whereas the
> latter uses a 5.8 kernel that features range invalidation.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> v3 -> v4:
> - add tags for machine, distro in the class
> - removed smp and memory overrides
> - set default param value of common_vm_setup to False
>
> v1 -> v2:
> - removed ssh import
> - combined add_command_args() and common_vm_setup()
> - moved tags in class' docstring and added tags=arch:aarch64
> - use self.get_default_kernel_params()
> - added RIL tests with fed33 + introduce new tags
> ---
>  tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 tests/acceptance/smmu.py
>

As stated in the thread, this test does not need to run on CI, so with
the skipIf statement:

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Willian Rampazzo]

Hi Eric,

On Mon, Jul 5, 2021 at 4:55 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> Hi Wainer,
>
> On 7/1/21 1:22 AM, Wainer dos Santos Moschetta wrote:
> > Hi,
> >
> > On 6/29/21 5:17 PM, Eric Auger wrote:
> >> Hi Cleber, all,
> >>
> >> On 6/29/21 4:36 PM, Eric Auger wrote:
> >>> This series adds ARM SMMU and Intel IOMMU functional
> >>> tests using Fedora cloud-init images.
> >>>
> >>> ARM SMMU tests feature guests with and without RIL
> >>> (range invalidation support) using respectively fedora 33
> >>> and 31.  For each, we test the protection of virtio-net-pci
> >>> and virtio-block-pci devices. Also strict=no and passthrough
> >>> modes are tested. So there is a total of 6 tests.
> >>>
> >>> The series applies on top of Cleber's series:
> >>> - [PATCH 0/3] Acceptance Tests: support choosing specific
> >>>
> >>> Note:
> >>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
> >>> test_smmu_noril_nostrict) pass but the log reports:
> >>> "WARN: Test passed but there were warnings during execution."
> >>> This seems due to the lack of hash when fetching the kernel and
> >>> initrd through fetch_asset():
> >>> WARNI| No hash provided. Cannot check the asset file integrity.
> >> I wanted to emphasize that point and wondered how we could fix that
> >> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
> >> Any advice?
> >
> > As Willian mentioned somewhere, to supress the WARN you can pass the
> > kernel and initrd checksums (sha1) to the fetch_asset() method.
> >
> > Below is an draft implementation. It would need to fill out the
> > remaining checksums and adjust the `smmu.py` tests.
> >
> > - Wainer
> >
> > ----
> >
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index 00eb0bfcc8..83637e2654 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -312,6 +312,8 @@ class LinuxDistro:
> >                  {'checksum':
> > 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
> >                  'pxeboot_url':
> > "https://archives.fedoraproject.org/pub/archive/fedora/"
> > "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
> > +                'pxeboot_initrd_chksum':
> > 'dd0340a1b39bd28f88532babd4581c67649ec5b1',
> > +                'pxeboot_vmlinuz_chksum':
> > '5b6f6876e1b5bda314f93893271da0d5777b1f3c',
> where did you get the checksum? I don't see any at the URL? Did you
> generate it yourself?

It is possible to use the hash you generate from the downloaded file.

While I was reviewing this series, I thought it makes more sense to
have Wainer's path applied first and then have your changes. I did
this here, with the addition of myu suggestions in the series:
https://gitlab.com/willianrampazzo/qemu/-/commits/test_eric_auger_v5.

Feel free to pick it and resend a new version.

Wainer, check if you agree with the changes to your patch and ack it.

Regards,

>
> Thanks
>
> Eric
> >                  'kernel_params':
> > "root=UUID=b1438b9b-2cab-4065-a99a-08a96687f73c ro "
> >                                "no_timer_check net.ifnames=0 "
> >                                "console=tty1 console=ttyS0,115200n8"},
> > @@ -371,6 +373,16 @@ def pxeboot_url(self):
> >          """Gets the repository url where pxeboot files can be found"""
> >          return self._info.get('pxeboot_url', None)
> >
> > +    @property
> > +    def pxeboot_initrd_chksum(self):
> > +        """Gets the pxeboot initrd file checksum"""
> > +        return self._info.get('pxeboot_initrd_chksum', None)
> > +
> > +    @property
> > +    def pxeboot_vmlinuz_chksum(self):
> > +        """Gets the pxeboot vmlinuz file checksum"""
> > +        return self._info.get('pxeboot_vmlinuz_chksum', None)
> > +
> >      @property
> >      def checksum(self):
> >          """Gets the cloud-image file checksum"""
> > diff --git a/tests/acceptance/intel_iommu.py
> > b/tests/acceptance/intel_iommu.py
> > index bf8dea6e4f..a2f38ee2e9 100644
> > --- a/tests/acceptance/intel_iommu.py
> > +++ b/tests/acceptance/intel_iommu.py
> > @@ -55,8 +55,10 @@ def common_vm_setup(self, custom_kernel=None):
> >
> >          kernel_url = self.distro.pxeboot_url + 'vmlinuz'
> >          initrd_url = self.distro.pxeboot_url + 'initrd.img'
> > -        self.kernel_path = self.fetch_asset(kernel_url)
> > -        self.initrd_path = self.fetch_asset(initrd_url)
> > +        self.kernel_path = self.fetch_asset(kernel_url,
> > + asset_hash=self.distro.pxeboot_vmlinuz_chksum)
> > +        self.initrd_path = self.fetch_asset(initrd_url,
> > + asset_hash=self.distro.pxeboot_initrd_chksum)
> >
> >      def run_and_check(self):
> >          if self.kernel_path:
> >
> >>
> >> Best Regards
> >>
> >> Eric
> >>> History:
> >>> v3 -> v4:
> >>> - I added Wainer's refactoring of KNOWN_DISTROS
> >>> into a class (last patch) and took into account his comments.
> >>>
> >>> v2 -> v3:
> >>> - Added Intel IOMMU tests were added. Different
> >>> operating modes are tested such as strict, caching mode, pt.
> >>>
> >>> Best Regards
> >>>
> >>> Eric
> >>>
> >>> The series and its dependencies can be found at:
> >>> https://github.com/eauger/qemu/tree/avocado-qemu-v4
> >>>
> >>> Eric Auger (3):
> >>>    Acceptance Tests: Add default kernel params and pxeboot url to the
> >>>      KNOWN_DISTROS collection
> >>>    avocado_qemu: Add SMMUv3 tests
> >>>    avocado_qemu: Add Intel iommu tests
> >>>
> >>> Wainer dos Santos Moschetta (1):
> >>>    avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class
> >>>
> >>>   tests/acceptance/avocado_qemu/__init__.py | 118 +++++++++++++------
> >>>   tests/acceptance/intel_iommu.py           | 115 +++++++++++++++++++
> >>>   tests/acceptance/smmu.py                  | 132
> >>> ++++++++++++++++++++++
> >>>   3 files changed, 332 insertions(+), 33 deletions(-)
> >>>   create mode 100644 tests/acceptance/intel_iommu.py
> >>>   create mode 100644 tests/acceptance/smmu.py
> >>>
> >
>

Re: [PATCH v4 3/4] avocado_qemu: Add Intel iommu tests

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 11:36 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> Add Intel IOMMU functional tests based on fedora 31.
> Different configs are checked:
> - strict
> - caching mode, strict
> - passthrough.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Peter Xu <peterx@redhat.com>
> ---
>  tests/acceptance/intel_iommu.py | 115 ++++++++++++++++++++++++++++++++
>  1 file changed, 115 insertions(+)
>  create mode 100644 tests/acceptance/intel_iommu.py
>
> diff --git a/tests/acceptance/intel_iommu.py b/tests/acceptance/intel_iommu.py
> new file mode 100644
> index 0000000000..0b68d3c572
> --- /dev/null
> +++ b/tests/acceptance/intel_iommu.py
> @@ -0,0 +1,115 @@
> +# INTEL_IOMMU Functional tests
> +#
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Eric Auger <eric.auger@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os

"os" package is not used, you can remove it, unless you add the skipIf
decorator, then you will need it.

> +
> +from avocado_qemu import LinuxTest, BUILD_DIR

BUILD_DIR is not used in this file.

> +from avocado.utils import ssh

The ssh package is not used in this file.

> +
> +class INTEL_IOMMU(LinuxTest):

I suggest you use IntelIOMMU as the class name, so it conforms to
Python class naming.

> +    """
> +    :avocado: tags=arch:x86_64
> +    :avocado: tags=distro:fedora
> +    :avocado: tags=distro_version:31
> +    :avocado: tags=machine:q35
> +    :avocado: tags=accel:kvm
> +    :avocado: tags=intel_iommu
> +    """
> +
> +    IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
> +    kernel_path = None
> +    initrd_path = None
> +    kernel_params = None
> +
> +    def set_up_boot(self):
> +        path = self.download_boot()
> +        self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
> +        self.vm.add_args('-device', 'virtio-gpu-pci' + self.IOMMU_ADDON)
> +        self.vm.add_args('-drive',
> +                         'file=%s,if=none,cache=writethrough,id=drv0' % path)
> +
> +    def setUp(self):
> +        super(INTEL_IOMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON)

If you change the class name, you need to change it here too.

> +
> +    def add_common_args(self):
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object',
> +                         'rng-random,id=rng0,filename=/dev/urandom')
> +
> +    def common_vm_setup(self, custom_kernel=None):
> +        self.require_accelerator("kvm")
> +        self.add_common_args()
> +        self.vm.add_args("-accel", "kvm")
> +
> +        if custom_kernel is None:
> +            return
> +
> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
> +        self.kernel_path = self.fetch_asset(kernel_url)
> +        self.initrd_path = self.fetch_asset(initrd_url)
> +
> +    def run_and_check(self):
> +        if self.kernel_path:
> +            self.vm.add_args('-kernel', self.kernel_path,
> +                             '-append', self.kernel_params,
> +                             '-initrd', self.initrd_path)
> +        self.launch_and_wait()
> +        self.ssh_command('cat /proc/cmdline')
> +        self.ssh_command('dmesg | grep -e DMAR -e IOMMU')
> +        self.ssh_command('find /sys/kernel/iommu_groups/ -type l')
> +        self.ssh_command('dnf -y install numactl-devel')
> +
> +    def test_intel_iommu(self):
> +        """
> +        :avocado: tags=intel_iommu_intremap
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.vm.add_args('-device', 'intel-iommu,intremap=on')
> +        self.vm.add_args('-machine', 'kernel_irqchip=split')
> +
> +        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on'
> +        self.run_and_check()
> +
> +    def test_intel_iommu_strict(self):
> +        """
> +        :avocado: tags=intel_iommu_strict
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.vm.add_args('-device', 'intel-iommu,intremap=on')
> +        self.vm.add_args('-machine', 'kernel_irqchip=split')
> +        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
> +        self.run_and_check()
> +
> +    def test_intel_iommu_strict_cm(self):
> +        """
> +        :avocado: tags=intel_iommu_strict_cm
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.vm.add_args('-device', 'intel-iommu,intremap=on,caching-mode=on')
> +        self.vm.add_args('-machine', 'kernel_irqchip=split')
> +        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on,strict'
> +        self.run_and_check()
> +
> +    def test_intel_iommu_pt(self):
> +        """
> +        :avocado: tags=intel_iommu_pt
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.vm.add_args('-device', 'intel-iommu,intremap=on')
> +        self.vm.add_args('-machine', 'kernel_irqchip=split')
> +        self.kernel_params = self.get_default_kernel_params() + ' quiet intel_iommu=on iommu=pt'
> +        self.run_and_check()
> --
> 2.26.3
>

Re: [PATCH v4 4/4] avocado_qemu: Fix KNOWN_DISTROS map into the LinuxDistro class

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 11:36 AM Eric Auger <eric.auger@redhat.com> wrote:
>
> From: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> As the KNOWN_DISTROS grows, more loosely methods will be created in
> the avocado_qemu/__init__.py file.
>
> Let's refactor the code so that KNOWN_DISTROS and related methods are
> packaged in a class
>
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
>
> [Eric] rebase and add commit message
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 160 +++++++++++-----------
>  tests/acceptance/intel_iommu.py           |  12 +-
>  tests/acceptance/smmu.py                  |  12 +-
>  3 files changed, 94 insertions(+), 90 deletions(-)
>

As suggested in the cover letter, moving the changes from Wainer to
the beginning of the patches makes more sense and reduces the amount
of changes.

The code refactoring from Wainer is here:
https://gitlab.com/willianrampazzo/qemu/-/commits/test_eric_auger_v5

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Philippe Mathieu-Daudé]

On 7/5/21 11:10 PM, Willian Rampazzo wrote:
> Hi Eric,
> 
> On Mon, Jul 5, 2021 at 4:55 AM Eric Auger <eric.auger@redhat.com> wrote:
>>
>> Hi Wainer,
>>
>> On 7/1/21 1:22 AM, Wainer dos Santos Moschetta wrote:
>>> Hi,
>>>
>>> On 6/29/21 5:17 PM, Eric Auger wrote:
>>>> Hi Cleber, all,
>>>>
>>>> On 6/29/21 4:36 PM, Eric Auger wrote:
>>>>> This series adds ARM SMMU and Intel IOMMU functional
>>>>> tests using Fedora cloud-init images.
>>>>>
>>>>> ARM SMMU tests feature guests with and without RIL
>>>>> (range invalidation support) using respectively fedora 33
>>>>> and 31.  For each, we test the protection of virtio-net-pci
>>>>> and virtio-block-pci devices. Also strict=no and passthrough
>>>>> modes are tested. So there is a total of 6 tests.
>>>>>
>>>>> The series applies on top of Cleber's series:
>>>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>>>>
>>>>> Note:
>>>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>>>>> test_smmu_noril_nostrict) pass but the log reports:
>>>>> "WARN: Test passed but there were warnings during execution."
>>>>> This seems due to the lack of hash when fetching the kernel and
>>>>> initrd through fetch_asset():
>>>>> WARNI| No hash provided. Cannot check the asset file integrity.
>>>> I wanted to emphasize that point and wondered how we could fix that
>>>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
>>>> Any advice?
>>>
>>> As Willian mentioned somewhere, to supress the WARN you can pass the
>>> kernel and initrd checksums (sha1) to the fetch_asset() method.
>>>
>>> Below is an draft implementation. It would need to fill out the
>>> remaining checksums and adjust the `smmu.py` tests.
>>>
>>> - Wainer
>>>
>>> ----
>>>
>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py
>>> b/tests/acceptance/avocado_qemu/__init__.py
>>> index 00eb0bfcc8..83637e2654 100644
>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>> @@ -312,6 +312,8 @@ class LinuxDistro:
>>>                  {'checksum':
>>> 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
>>>                  'pxeboot_url':
>>> "https://archives.fedoraproject.org/pub/archive/fedora/"
>>> "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
>>> +                'pxeboot_initrd_chksum':
>>> 'dd0340a1b39bd28f88532babd4581c67649ec5b1',
>>> +                'pxeboot_vmlinuz_chksum':
>>> '5b6f6876e1b5bda314f93893271da0d5777b1f3c',
>> where did you get the checksum? I don't see any at the URL? Did you
>> generate it yourself?
> 
> It is possible to use the hash you generate from the downloaded file.
> 
> While I was reviewing this series, I thought it makes more sense to
> have Wainer's path applied first and then have your changes. I did
> this here, with the addition of myu suggestions in the series:
> https://gitlab.com/willianrampazzo/qemu/-/commits/test_eric_auger_v5.

Off-list review is a bit unhandy (in particular when asked on the list).

Why don't you post your improvements as v5? I don't think Eric will be
offended: this is the opposite, you are helping him to get his patches
merged ;)

> Feel free to pick it and resend a new version.
> 
> Wainer, check if you agree with the changes to your patch and ack it.
> 
> Regards,

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Willian Rampazzo]

On Mon, Jul 5, 2021 at 6:20 PM Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> On 7/5/21 11:10 PM, Willian Rampazzo wrote:
> > Hi Eric,
> >
> > On Mon, Jul 5, 2021 at 4:55 AM Eric Auger <eric.auger@redhat.com> wrote:
> >>
> >> Hi Wainer,
> >>
> >> On 7/1/21 1:22 AM, Wainer dos Santos Moschetta wrote:
> >>> Hi,
> >>>
> >>> On 6/29/21 5:17 PM, Eric Auger wrote:
> >>>> Hi Cleber, all,
> >>>>
> >>>> On 6/29/21 4:36 PM, Eric Auger wrote:
> >>>>> This series adds ARM SMMU and Intel IOMMU functional
> >>>>> tests using Fedora cloud-init images.
> >>>>>
> >>>>> ARM SMMU tests feature guests with and without RIL
> >>>>> (range invalidation support) using respectively fedora 33
> >>>>> and 31.  For each, we test the protection of virtio-net-pci
> >>>>> and virtio-block-pci devices. Also strict=no and passthrough
> >>>>> modes are tested. So there is a total of 6 tests.
> >>>>>
> >>>>> The series applies on top of Cleber's series:
> >>>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
> >>>>>
> >>>>> Note:
> >>>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
> >>>>> test_smmu_noril_nostrict) pass but the log reports:
> >>>>> "WARN: Test passed but there were warnings during execution."
> >>>>> This seems due to the lack of hash when fetching the kernel and
> >>>>> initrd through fetch_asset():
> >>>>> WARNI| No hash provided. Cannot check the asset file integrity.
> >>>> I wanted to emphasize that point and wondered how we could fix that
> >>>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
> >>>> Any advice?
> >>>
> >>> As Willian mentioned somewhere, to supress the WARN you can pass the
> >>> kernel and initrd checksums (sha1) to the fetch_asset() method.
> >>>
> >>> Below is an draft implementation. It would need to fill out the
> >>> remaining checksums and adjust the `smmu.py` tests.
> >>>
> >>> - Wainer
> >>>
> >>> ----
> >>>
> >>> diff --git a/tests/acceptance/avocado_qemu/__init__.py
> >>> b/tests/acceptance/avocado_qemu/__init__.py
> >>> index 00eb0bfcc8..83637e2654 100644
> >>> --- a/tests/acceptance/avocado_qemu/__init__.py
> >>> +++ b/tests/acceptance/avocado_qemu/__init__.py
> >>> @@ -312,6 +312,8 @@ class LinuxDistro:
> >>>                  {'checksum':
> >>> 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
> >>>                  'pxeboot_url':
> >>> "https://archives.fedoraproject.org/pub/archive/fedora/"
> >>> "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
> >>> +                'pxeboot_initrd_chksum':
> >>> 'dd0340a1b39bd28f88532babd4581c67649ec5b1',
> >>> +                'pxeboot_vmlinuz_chksum':
> >>> '5b6f6876e1b5bda314f93893271da0d5777b1f3c',
> >> where did you get the checksum? I don't see any at the URL? Did you
> >> generate it yourself?
> >
> > It is possible to use the hash you generate from the downloaded file.
> >
> > While I was reviewing this series, I thought it makes more sense to
> > have Wainer's path applied first and then have your changes. I did
> > this here, with the addition of myu suggestions in the series:
> > https://gitlab.com/willianrampazzo/qemu/-/commits/test_eric_auger_v5.
>
> Off-list review is a bit unhandy (in particular when asked on the list).
>
> Why don't you post your improvements as v5? I don't think Eric will be
> offended: this is the opposite, you are helping him to get his patches
> merged ;)

Oh, I did review each of his patches in the list and also already made
the changes to speed up the process :)

He mentioned today to me that his series is still depending on one
from Cleber that was not merged yet, so we need to wait for that.

>
> > Feel free to pick it and resend a new version.
> >
> > Wainer, check if you agree with the changes to your patch and ack it.
> >
> > Regards,
>

Re: [PATCH v4 0/4] avocado-qemu: New SMMUv3 and intel IOMMU tests

[Eric Auger]

Hi William, Philippe,

On 7/5/21 11:24 PM, Willian Rampazzo wrote:
> On Mon, Jul 5, 2021 at 6:20 PM Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>> On 7/5/21 11:10 PM, Willian Rampazzo wrote:
>>> Hi Eric,
>>>
>>> On Mon, Jul 5, 2021 at 4:55 AM Eric Auger <eric.auger@redhat.com> wrote:
>>>> Hi Wainer,
>>>>
>>>> On 7/1/21 1:22 AM, Wainer dos Santos Moschetta wrote:
>>>>> Hi,
>>>>>
>>>>> On 6/29/21 5:17 PM, Eric Auger wrote:
>>>>>> Hi Cleber, all,
>>>>>>
>>>>>> On 6/29/21 4:36 PM, Eric Auger wrote:
>>>>>>> This series adds ARM SMMU and Intel IOMMU functional
>>>>>>> tests using Fedora cloud-init images.
>>>>>>>
>>>>>>> ARM SMMU tests feature guests with and without RIL
>>>>>>> (range invalidation support) using respectively fedora 33
>>>>>>> and 31.  For each, we test the protection of virtio-net-pci
>>>>>>> and virtio-block-pci devices. Also strict=no and passthrough
>>>>>>> modes are tested. So there is a total of 6 tests.
>>>>>>>
>>>>>>> The series applies on top of Cleber's series:
>>>>>>> - [PATCH 0/3] Acceptance Tests: support choosing specific
>>>>>>>
>>>>>>> Note:
>>>>>>> - SMMU tests 2, 3, 5, 6 (resp. test_smmu_noril_passthrough and
>>>>>>> test_smmu_noril_nostrict) pass but the log reports:
>>>>>>> "WARN: Test passed but there were warnings during execution."
>>>>>>> This seems due to the lack of hash when fetching the kernel and
>>>>>>> initrd through fetch_asset():
>>>>>>> WARNI| No hash provided. Cannot check the asset file integrity.
>>>>>> I wanted to emphasize that point and wondered how we could fix that
>>>>>> issue. Looks a pity the tests get tagged as WARN due to a lack of sha1.
>>>>>> Any advice?
>>>>> As Willian mentioned somewhere, to supress the WARN you can pass the
>>>>> kernel and initrd checksums (sha1) to the fetch_asset() method.
>>>>>
>>>>> Below is an draft implementation. It would need to fill out the
>>>>> remaining checksums and adjust the `smmu.py` tests.
>>>>>
>>>>> - Wainer
>>>>>
>>>>> ----
>>>>>
>>>>> diff --git a/tests/acceptance/avocado_qemu/__init__.py
>>>>> b/tests/acceptance/avocado_qemu/__init__.py
>>>>> index 00eb0bfcc8..83637e2654 100644
>>>>> --- a/tests/acceptance/avocado_qemu/__init__.py
>>>>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>>>>> @@ -312,6 +312,8 @@ class LinuxDistro:
>>>>>                  {'checksum':
>>>>> 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0',
>>>>>                  'pxeboot_url':
>>>>> "https://archives.fedoraproject.org/pub/archive/fedora/"
>>>>> "linux/releases/31/Everything/x86_64/os/images/pxeboot/",
>>>>> +                'pxeboot_initrd_chksum':
>>>>> 'dd0340a1b39bd28f88532babd4581c67649ec5b1',
>>>>> +                'pxeboot_vmlinuz_chksum':
>>>>> '5b6f6876e1b5bda314f93893271da0d5777b1f3c',
>>>> where did you get the checksum? I don't see any at the URL? Did you
>>>> generate it yourself?
>>> It is possible to use the hash you generate from the downloaded file.
>>>
>>> While I was reviewing this series, I thought it makes more sense to
>>> have Wainer's path applied first and then have your changes. I did
>>> this here, with the addition of myu suggestions in the series:
>>> https://gitlab.com/willianrampazzo/qemu/-/commits/test_eric_auger_v5.
>> Off-list review is a bit unhandy (in particular when asked on the list).
>>
>> Why don't you post your improvements as v5? I don't think Eric will be
>> offended: this is the opposite, you are helping him to get his patches
>> merged ;)
> Oh, I did review each of his patches in the list and also already made
> the changes to speed up the process :)

Yes the review is really helpful. I will have a double check at your
changes and respin quickly.
>
> He mentioned today to me that his series is still depending on one
> from Cleber that was not merged yet, so we need to wait for that.
Yep. The soft freeze is quickly approaching and that would be cool to
see those tests landing upstream.

Thanks

Eric
>
>>> Feel free to pick it and resend a new version.
>>>
>>> Wainer, check if you agree with the changes to your patch and ack it.
>>>
>>> Regards,

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Wainer dos Santos Moschetta]

On 7/5/21 5:00 AM, Eric Auger wrote:
> Hi Wainer,
>
> On 7/1/21 8:13 PM, Wainer dos Santos Moschetta wrote:
>> Hi,
>>
>> On 6/29/21 11:36 AM, Eric Auger wrote:
>>> Add new tests checking the good behavior of the SMMUv3 protecting
>>> 2 virtio pci devices (block and net). We check the guest boots and
>>> we are able to install a package. Different guest configs are tested:
>>> standard, passthrough an strict=0. This is tested with both fedora 31
>>> and
>>> 33. The former uses a 5.3 kernel without range invalidation whereas the
>>> latter uses a 5.8 kernel that features range invalidation.
>>>
>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>
>>> ---
>>>
>>> v3 -> v4:
>>> - add tags for machine, distro in the class
>>> - removed smp and memory overrides
>>> - set default param value of common_vm_setup to False
>>>
>>> v1 -> v2:
>>> - removed ssh import
>>> - combined add_command_args() and common_vm_setup()
>>> - moved tags in class' docstring and added tags=arch:aarch64
>>> - use self.get_default_kernel_params()
>>> - added RIL tests with fed33 + introduce new tags
>>> ---
>>>    tests/acceptance/smmu.py | 132 +++++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 132 insertions(+)
>>>    create mode 100644 tests/acceptance/smmu.py
>> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>>
>> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>>
>> I tested it in a Fedora 32 aarch64 host. The execution output:
>>
>> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
>> JOB ID     : 1625038f5a2ae17c8ba6c503d3df8661ff528942
>> JOB LOG    :
>> /root/avocado/job-results/job-2021-07-01T13.38-1625038/job.log
>>   (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (175.54 s)
>>   (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
>> WARN: Test passed but there were warnings during execution. Check the
>> log for details. (168.39 s)
>>   (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (161.58 s)
>>   (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (150.85 s)
>>   (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (177.56 s)
>>   (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (190.86 s)
>> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
>> | CANCEL 0
>> JOB TIME   : 1026.50 s
>>
>> One thing that caught my attention was the amount of time spent on
>> each test. It spend more than 2 minutes on the package installation
>> (`self.ssh_command('dnf -y install numactl-devel')`) in the guest.
>>
>> Without that operation, it runs way faster:
>>
>> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
>> JOB ID     : 24f22f99169ece37df64d72d2eb373921f378aac
>> JOB LOG    :
>> /root/avocado/job-results/job-2021-07-01T13.28-24f22f9/job.log
>>   (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (39.61 s)
>>   (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
>> WARN: Test passed but there were warnings during execution. Check the
>> log for details. (48.32 s)
>>   (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (48.10 s)
>>   (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (39.22 s)
>>   (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (52.92 s)
>>   (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
>> Test passed but there were warnings during execution. Check the log
>> for details. (50.96 s)
>> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
>> | CANCEL 0
>> JOB TIME   : 280.62 s
>>
>> Install a package seems a good exerciser for disk I/O and networking,
>> but maybe you can use another method for the sake of speed up the tests?
> As discussed earlier with Cleber, I am aware the test duration is long
> but it was useful finding bugs for SMMU with range invalidation. such a
> bug could not be hit with a single boot + ping for instance.
>
> Maybe we should have a mechanism that allows to put some tests out of
> the automatic CI?

Sorry Eric, I missed that discussion. I will review your v5 series very 
soon.

Thanks!

- Wainer

>
> Thanks
>
> Eric
>> - Wainer
>>
>>> diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
>>> new file mode 100644
>>> index 0000000000..c1d4b88e5f
>>> --- /dev/null
>>> +++ b/tests/acceptance/smmu.py
>>> @@ -0,0 +1,132 @@
>>> +# SMMUv3 Functional tests
>>> +#
>>> +# Copyright (c) 2021 Red Hat, Inc.
>>> +#
>>> +# Author:
>>> +#  Eric Auger <eric.auger@redhat.com>
>>> +#
>>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>>> +# later.  See the COPYING file in the top-level directory.
>>> +
>>> +import os
>>> +
>>> +from avocado_qemu import LinuxTest, BUILD_DIR
>>> +
>>> +class SMMU(LinuxTest):
>>> +    """
>>> +    :avocado: tags=accel:kvm
>>> +    :avocado: tags=cpu:host
>>> +    :avocado: tags=arch:aarch64
>>> +    :avocado: tags=machine:virt
>>> +    :avocado: tags=distro:fedora
>>> +    :avocado: tags=smmu
>>> +    """
>>> +
>>> +    IOMMU_ADDON =
>>> ',iommu_platform=on,disable-modern=off,disable-legacy=on'
>>> +    kernel_path = None
>>> +    initrd_path = None
>>> +    kernel_params = None
>>> +
>>> +    def set_up_boot(self):
>>> +        path = self.download_boot()
>>> +        self.vm.add_args('-device',
>>> 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
>>> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
>>> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
>>> +        self.vm.add_args('-drive',
>>> +
>>> 'file=%s,if=none,cache=writethrough,id=drv0' % path)
>>> +
>>> +    def setUp(self):
>>> +        super(SMMU, self).setUp(None, 'virtio-net-pci' +
>>> self.IOMMU_ADDON)
>>> +
>>> +    def common_vm_setup(self, custom_kernel=False):
>>> +        self.require_accelerator("kvm")
>>> +        self.vm.add_args("-accel", "kvm")
>>> +        self.vm.add_args("-cpu", "host")
>>> +        self.vm.add_args("-machine", "iommu=smmuv3")
>>> +        self.vm.add_args("-d", "guest_errors")
>>> +        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
>>> +                         'edk2-aarch64-code.fd'))
>>> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
>>> +        self.vm.add_args('-object',
>>> +                         'rng-random,id=rng0,filename=/dev/urandom')
>>> +
>>> +        if custom_kernel is False:
>>> +            return
>>> +
>>> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
>>> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
>>> +        self.kernel_path = self.fetch_asset(kernel_url)
>>> +        self.initrd_path = self.fetch_asset(initrd_url)
>>> +
>>> +    def run_and_check(self):
>>> +        if self.kernel_path:
>>> +            self.vm.add_args('-kernel', self.kernel_path,
>>> +                             '-append', self.kernel_params,
>>> +                             '-initrd', self.initrd_path)
>>> +        self.launch_and_wait()
>>> +        self.ssh_command('cat /proc/cmdline')
>>> +        self.ssh_command('dnf -y install numactl-devel')
>>> +
>>> +
>>> +    # 5.3 kernel without RIL #
>>> +
>>> +    def test_smmu_noril(self):
>>> +        """
>>> +        :avocado: tags=smmu_noril
>>> +        :avocado: tags=smmu_noril_tests
>>> +        :avocado: tags=distro_version:31
>>> +        """
>>> +        self.common_vm_setup()
>>> +        self.run_and_check()
>>> +
>>> +    def test_smmu_noril_passthrough(self):
>>> +        """
>>> +        :avocado: tags=smmu_noril_passthrough
>>> +        :avocado: tags=smmu_noril_tests
>>> +        :avocado: tags=distro_version:31
>>> +        """
>>> +        self.common_vm_setup(True)
>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>> iommu.passthrough=on'
>>> +        self.run_and_check()
>>> +
>>> +    def test_smmu_noril_nostrict(self):
>>> +        """
>>> +        :avocado: tags=smmu_noril_nostrict
>>> +        :avocado: tags=smmu_noril_tests
>>> +        :avocado: tags=distro_version:31
>>> +        """
>>> +        self.common_vm_setup(True)
>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>> iommu.strict=0'
>>> +        self.run_and_check()
>>> +
>>> +    # 5.8 kernel featuring range invalidation
>>> +    # >= v5.7 kernel
>>> +
>>> +    def test_smmu_ril(self):
>>> +        """
>>> +        :avocado: tags=smmu_ril
>>> +        :avocado: tags=smmu_ril_tests
>>> +        :avocado: tags=distro_version:33
>>> +        """
>>> +        self.common_vm_setup()
>>> +        self.run_and_check()
>>> +
>>> +    def test_smmu_ril_passthrough(self):
>>> +        """
>>> +        :avocado: tags=smmu_ril_passthrough
>>> +        :avocado: tags=smmu_ril_tests
>>> +        :avocado: tags=distro_version:33
>>> +        """
>>> +        self.common_vm_setup(True)
>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>> iommu.passthrough=on'
>>> +        self.run_and_check()
>>> +
>>> +    def test_smmu_ril_nostrict(self):
>>> +        """
>>> +        :avocado: tags=smmu_ril_nostrict
>>> +        :avocado: tags=smmu_ril_tests
>>> +        :avocado: tags=distro_version:33
>>> +        """
>>> +        self.common_vm_setup(True)
>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>> iommu.strict=0'
>>> +        self.run_and_check()

Re: [PATCH v4 2/4] avocado_qemu: Add SMMUv3 tests

[Eric Auger]

Hi Wainer,

On 7/6/21 4:26 PM, Wainer dos Santos Moschetta wrote:
>
> On 7/5/21 5:00 AM, Eric Auger wrote:
>> Hi Wainer,
>>
>> On 7/1/21 8:13 PM, Wainer dos Santos Moschetta wrote:
>>> Hi,
>>>
>>> On 6/29/21 11:36 AM, Eric Auger wrote:
>>>> Add new tests checking the good behavior of the SMMUv3 protecting
>>>> 2 virtio pci devices (block and net). We check the guest boots and
>>>> we are able to install a package. Different guest configs are tested:
>>>> standard, passthrough an strict=0. This is tested with both fedora 31
>>>> and
>>>> 33. The former uses a 5.3 kernel without range invalidation whereas
>>>> the
>>>> latter uses a 5.8 kernel that features range invalidation.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>>
>>>> v3 -> v4:
>>>> - add tags for machine, distro in the class
>>>> - removed smp and memory overrides
>>>> - set default param value of common_vm_setup to False
>>>>
>>>> v1 -> v2:
>>>> - removed ssh import
>>>> - combined add_command_args() and common_vm_setup()
>>>> - moved tags in class' docstring and added tags=arch:aarch64
>>>> - use self.get_default_kernel_params()
>>>> - added RIL tests with fed33 + introduce new tags
>>>> ---
>>>>    tests/acceptance/smmu.py | 132
>>>> +++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 132 insertions(+)
>>>>    create mode 100644 tests/acceptance/smmu.py
>>> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>>>
>>> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>>>
>>> I tested it in a Fedora 32 aarch64 host. The execution output:
>>>
>>> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
>>> JOB ID     : 1625038f5a2ae17c8ba6c503d3df8661ff528942
>>> JOB LOG    :
>>> /root/avocado/job-results/job-2021-07-01T13.38-1625038/job.log
>>>   (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (175.54 s)
>>>   (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
>>> WARN: Test passed but there were warnings during execution. Check the
>>> log for details. (168.39 s)
>>>   (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (161.58 s)
>>>   (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (150.85 s)
>>>   (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (177.56 s)
>>>   (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (190.86 s)
>>> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
>>> | CANCEL 0
>>> JOB TIME   : 1026.50 s
>>>
>>> One thing that caught my attention was the amount of time spent on
>>> each test. It spend more than 2 minutes on the package installation
>>> (`self.ssh_command('dnf -y install numactl-devel')`) in the guest.
>>>
>>> Without that operation, it runs way faster:
>>>
>>> # ./tests/venv/bin/avocado run tests/acceptance/smmu.py
>>> JOB ID     : 24f22f99169ece37df64d72d2eb373921f378aac
>>> JOB LOG    :
>>> /root/avocado/job-results/job-2021-07-01T13.28-24f22f9/job.log
>>>   (1/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril: PASS (39.61 s)
>>>   (2/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
>>> WARN: Test passed but there were warnings during execution. Check the
>>> log for details. (48.32 s)
>>>   (3/6) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (48.10 s)
>>>   (4/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril: PASS (39.22 s)
>>>   (5/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (52.92 s)
>>>   (6/6) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: WARN:
>>> Test passed but there were warnings during execution. Check the log
>>> for details. (50.96 s)
>>> RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 4 | INTERRUPT 0
>>> | CANCEL 0
>>> JOB TIME   : 280.62 s
>>>
>>> Install a package seems a good exerciser for disk I/O and networking,
>>> but maybe you can use another method for the sake of speed up the
>>> tests?
>> As discussed earlier with Cleber, I am aware the test duration is long
>> but it was useful finding bugs for SMMU with range invalidation. such a
>> bug could not be hit with a single boot + ping for instance.
>>
>> Maybe we should have a mechanism that allows to put some tests out of
>> the automatic CI?
>
> Sorry Eric, I missed that discussion. I will review your v5 series
> very soon.

no problem. William helped on this with the

@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

Besides, we still have a dependency on Cleber's
[PATCH 0/3] Acceptance Tests: support choosing specific

Is there anyting controversial with that series?

Thanks

Eric

>
> Thanks!
>
> - Wainer
>
>>
>> Thanks
>>
>> Eric
>>> - Wainer
>>>
>>>> diff --git a/tests/acceptance/smmu.py b/tests/acceptance/smmu.py
>>>> new file mode 100644
>>>> index 0000000000..c1d4b88e5f
>>>> --- /dev/null
>>>> +++ b/tests/acceptance/smmu.py
>>>> @@ -0,0 +1,132 @@
>>>> +# SMMUv3 Functional tests
>>>> +#
>>>> +# Copyright (c) 2021 Red Hat, Inc.
>>>> +#
>>>> +# Author:
>>>> +#  Eric Auger <eric.auger@redhat.com>
>>>> +#
>>>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>>>> +# later.  See the COPYING file in the top-level directory.
>>>> +
>>>> +import os
>>>> +
>>>> +from avocado_qemu import LinuxTest, BUILD_DIR
>>>> +
>>>> +class SMMU(LinuxTest):
>>>> +    """
>>>> +    :avocado: tags=accel:kvm
>>>> +    :avocado: tags=cpu:host
>>>> +    :avocado: tags=arch:aarch64
>>>> +    :avocado: tags=machine:virt
>>>> +    :avocado: tags=distro:fedora
>>>> +    :avocado: tags=smmu
>>>> +    """
>>>> +
>>>> +    IOMMU_ADDON =
>>>> ',iommu_platform=on,disable-modern=off,disable-legacy=on'
>>>> +    kernel_path = None
>>>> +    initrd_path = None
>>>> +    kernel_params = None
>>>> +
>>>> +    def set_up_boot(self):
>>>> +        path = self.download_boot()
>>>> +        self.vm.add_args('-device',
>>>> 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
>>>> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
>>>> +                         'werror=stop,rerror=stop' +
>>>> self.IOMMU_ADDON)
>>>> +        self.vm.add_args('-drive',
>>>> +
>>>> 'file=%s,if=none,cache=writethrough,id=drv0' % path)
>>>> +
>>>> +    def setUp(self):
>>>> +        super(SMMU, self).setUp(None, 'virtio-net-pci' +
>>>> self.IOMMU_ADDON)
>>>> +
>>>> +    def common_vm_setup(self, custom_kernel=False):
>>>> +        self.require_accelerator("kvm")
>>>> +        self.vm.add_args("-accel", "kvm")
>>>> +        self.vm.add_args("-cpu", "host")
>>>> +        self.vm.add_args("-machine", "iommu=smmuv3")
>>>> +        self.vm.add_args("-d", "guest_errors")
>>>> +        self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
>>>> +                         'edk2-aarch64-code.fd'))
>>>> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
>>>> +        self.vm.add_args('-object',
>>>> +                         'rng-random,id=rng0,filename=/dev/urandom')
>>>> +
>>>> +        if custom_kernel is False:
>>>> +            return
>>>> +
>>>> +        kernel_url = self.get_pxeboot_url() + 'vmlinuz'
>>>> +        initrd_url = self.get_pxeboot_url() + 'initrd.img'
>>>> +        self.kernel_path = self.fetch_asset(kernel_url)
>>>> +        self.initrd_path = self.fetch_asset(initrd_url)
>>>> +
>>>> +    def run_and_check(self):
>>>> +        if self.kernel_path:
>>>> +            self.vm.add_args('-kernel', self.kernel_path,
>>>> +                             '-append', self.kernel_params,
>>>> +                             '-initrd', self.initrd_path)
>>>> +        self.launch_and_wait()
>>>> +        self.ssh_command('cat /proc/cmdline')
>>>> +        self.ssh_command('dnf -y install numactl-devel')
>>>> +
>>>> +
>>>> +    # 5.3 kernel without RIL #
>>>> +
>>>> +    def test_smmu_noril(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_noril
>>>> +        :avocado: tags=smmu_noril_tests
>>>> +        :avocado: tags=distro_version:31
>>>> +        """
>>>> +        self.common_vm_setup()
>>>> +        self.run_and_check()
>>>> +
>>>> +    def test_smmu_noril_passthrough(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_noril_passthrough
>>>> +        :avocado: tags=smmu_noril_tests
>>>> +        :avocado: tags=distro_version:31
>>>> +        """
>>>> +        self.common_vm_setup(True)
>>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>>> iommu.passthrough=on'
>>>> +        self.run_and_check()
>>>> +
>>>> +    def test_smmu_noril_nostrict(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_noril_nostrict
>>>> +        :avocado: tags=smmu_noril_tests
>>>> +        :avocado: tags=distro_version:31
>>>> +        """
>>>> +        self.common_vm_setup(True)
>>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>>> iommu.strict=0'
>>>> +        self.run_and_check()
>>>> +
>>>> +    # 5.8 kernel featuring range invalidation
>>>> +    # >= v5.7 kernel
>>>> +
>>>> +    def test_smmu_ril(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_ril
>>>> +        :avocado: tags=smmu_ril_tests
>>>> +        :avocado: tags=distro_version:33
>>>> +        """
>>>> +        self.common_vm_setup()
>>>> +        self.run_and_check()
>>>> +
>>>> +    def test_smmu_ril_passthrough(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_ril_passthrough
>>>> +        :avocado: tags=smmu_ril_tests
>>>> +        :avocado: tags=distro_version:33
>>>> +        """
>>>> +        self.common_vm_setup(True)
>>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>>> iommu.passthrough=on'
>>>> +        self.run_and_check()
>>>> +
>>>> +    def test_smmu_ril_nostrict(self):
>>>> +        """
>>>> +        :avocado: tags=smmu_ril_nostrict
>>>> +        :avocado: tags=smmu_ril_tests
>>>> +        :avocado: tags=distro_version:33
>>>> +        """
>>>> +        self.common_vm_setup(True)
>>>> +        self.kernel_params = self.get_default_kernel_params() + '
>>>> iommu.strict=0'
>>>> +        self.run_and_check()
>