[PULL 0/4] Acceptance testing patches for 2020-10-26

[PULL 0/4] Acceptance testing patches for 2020-10-26

[Philippe Mathieu-Daudé]

The following changes since commit a46e72710566eea0f90f9c673a0f02da0064acce:

  Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20201026' into stag=
ing (2020-10-26 14:50:03 +0000)

are available in the Git repository at:

  https://gitlab.com/philmd/qemu.git tags/acceptance-testing-20201026

for you to fetch changes up to 4c9ac67230d2d8c12bfc0f3447ae353e67b43f7f:

  docs/devel/testing.rst: Update outdated Avocado URLs (2020-10-26 18:20:51 +=
0100)

----------------------------------------------------------------
Acceptance testing patches

- More ARM tests
- Documentation update

CI jobs results:
. https://cirrus-ci.com/build/5754555684093952
. https://gitlab.com/philmd/qemu/-/pipelines/207807648
----------------------------------------------------------------

Philippe Mathieu-Daud=C3=A9 (4):
  tests/boot_linux_console: Boot Trusted Firmware-A on the Raspberry Pi
    3
  tests/acceptance: Add a 'virt_kvm' test using the GICv3
  tests/acceptance: Allow running Orange Pi test using cached artifacts
  docs/devel/testing.rst: Update outdated Avocado URLs

 docs/devel/testing.rst                 |  4 ++--
 tests/acceptance/boot_linux.py         | 17 ++++++++++++++++-
 tests/acceptance/boot_linux_console.py | 24 ++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 3 deletions(-)

--=20
2.26.2

[PULL 1/4] tests/boot_linux_console: Boot Trusted Firmware-A on the Raspberry Pi 3

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This test runs Trusted Firmware-A on the Raspberry Pi 3.
We deliberately stop the boot process when the EDK2 UEFI version
is displayed.

The binary is build on AppVeyor CI using Pete Batard repository [1].
ATF v2.1 binary are used (see [2]).

It is very simple and fast:

  $ avocado --show=app,console run -t atf tests/acceptance
  JOB ID     : 1e748d7c9e9011cf0af3250ddc8ebf2389d6204e
  JOB LOG    : avocado/job-results/job-2020-02-16T18.08-1e748d7/job.log
   (1/1) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
  console: NOTICE:  Booting Trusted Firmware
  console: NOTICE:  BL1: v2.1(release):v2.1
  console: NOTICE:  BL1: Built : 15:26:06, May 13 2019
  console: NOTICE:  rpi3: Detected: Raspberry Pi 3 Model B (1GB, Sony, UK) [0x00a02082]
  console: NOTICE:  BL1: Booting BL2
  console: ERROR:   rpi3_sdhost: timeout status 0x40
  console: NOTICE:  BL2: v2.1(release):v2.1
  console: NOTICE:  BL2: Built : 15:26:01, May 13 2019
  console: NOTICE:  BL1: Booting BL31
  console: NOTICE:  BL31: v2.1(release):v2.1
  console: NOTICE:  BL31: Built : 15:26:04, May 13 2019
  console: =UEFI firmware (version UEFI Firmware v1.15 built at 11:58:44 on Feb 14 2020)
  PASS (1.54 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 1.88 s

[1] https://github.com/pbatard/RPi3#summary
[2] https://github.com/ARM-software/arm-trusted-firmware/blob/v2.1/docs/plat/rpi3.rst

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Luc Michel <luc.michel@greensocs.com>
Message-Id: <20200217103442.30318-1-f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 0118ed59156..415e32ef0d6 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -889,6 +889,28 @@ def test_arm_orangepi_uboot_netbsd9(self):
         # Wait for user-space
         wait_for_console_pattern(self, 'Starting root file system check')
 
+    def test_aarch64_raspi3_atf(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:raspi3
+        :avocado: tags=cpu:cortex-a53
+        :avocado: tags=device:pl011
+        :avocado: tags=atf
+        """
+        zip_url = ('https://github.com/pbatard/RPi3/releases/download/'
+                   'v1.15/RPi3_UEFI_Firmware_v1.15.zip')
+        zip_hash = '74b3bd0de92683cadb14e008a7575e1d0c3cafb9'
+        zip_path = self.fetch_asset(zip_url, asset_hash=zip_hash)
+
+        archive.extract(zip_path, self.workdir)
+        efi_fd = os.path.join(self.workdir, 'RPI_EFI.fd')
+
+        self.vm.set_console(console_index=1)
+        self.vm.add_args('-nodefaults',
+                         '-device', 'loader,file=%s,force-raw=true' % efi_fd)
+        self.vm.launch()
+        self.wait_for_console_pattern('version UEFI Firmware v1.15')
+
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x
-- 
2.26.2

[PULL 2/4] tests/acceptance: Add a 'virt_kvm' test using the GICv3

[Philippe Mathieu-Daudé]

The current 'virt_kvm' test is restricted to GICv2, but can also
work with a GICv3. Duplicate it but add a GICv3 test which can be
tested on some hardware.

Noticed while running:

 $ avocado --show=app run -t machine:virt tests/acceptance/
 ...
 (2/6) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: ERROR: Unexpected empty reply from server (1.82 s)

The job.log content is:

  L0351 DEBUG| Output: 'qemu-system-aarch64: host does not support in-kernel GICv2 emulation\n'

With this patch:

 $ avocado --show=app run -t device:gicv3 tests/acceptance/
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm_gicv3: PASS (55.10 s)

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20200929224857.1225107-1-philmd@redhat.com>
---
 tests/acceptance/boot_linux.py | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
index 0055dc7cee7..c743e231f47 100644
--- a/tests/acceptance/boot_linux.py
+++ b/tests/acceptance/boot_linux.py
@@ -182,10 +182,11 @@ def test_virt_tcg(self):
         self.add_common_args()
         self.launch_and_wait()
 
-    def test_virt_kvm(self):
+    def test_virt_kvm_gicv2(self):
         """
         :avocado: tags=accel:kvm
         :avocado: tags=cpu:host
+        :avocado: tags=device:gicv2
         """
         if not kvm_available(self.arch, self.qemu_bin):
             self.cancel(KVM_NOT_AVAILABLE)
@@ -195,6 +196,20 @@ def test_virt_kvm(self):
         self.add_common_args()
         self.launch_and_wait()
 
+    def test_virt_kvm_gicv3(self):
+        """
+        :avocado: tags=accel:kvm
+        :avocado: tags=cpu:host
+        :avocado: tags=device:gicv3
+        """
+        if not kvm_available(self.arch, self.qemu_bin):
+            self.cancel(KVM_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "kvm")
+        self.vm.add_args("-cpu", "host")
+        self.vm.add_args("-machine", "virt,gic-version=3")
+        self.add_common_args()
+        self.launch_and_wait()
+
 
 class BootLinuxPPC64(BootLinux):
     """
-- 
2.26.2

[PULL 3/4] tests/acceptance: Allow running Orange Pi test using cached artifacts

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Unfortunately the Armbian 19.11.3 image has been removed from the
dl.armbian.com file server. Developers having the artifact cached
can still run the test. Allow them to, until we find a proper
solution to share binaries with the whole community.

This avoids (when file manually added to cache):

  BootLinuxConsole.test_arm_orangepi_bionic_19_11: CANCEL: Missing asset https://dl.armbian.com/orangepipc/archive/Armbian_19.11.3_Orangepipc_bionic_current_5.3.9.7z (1.06 s)

Reported-by: Bin Meng <bmeng.cn@gmail.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Bin Meng <bin.meng@windriver.com>
Tested-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Message-Id: <20201023131808.3198005-3-f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 415e32ef0d6..8f433a67f84 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -780,6 +780,8 @@ def test_arm_orangepi_sd(self):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
+    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
+                'Test artifacts fetched from unreliable dl.armbian.com')
     @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
     @skipUnless(P7ZIP_AVAILABLE, '7z not installed')
     def test_arm_orangepi_bionic(self):
-- 
2.26.2

[PULL 4/4] docs/devel/testing.rst: Update outdated Avocado URLs

[Philippe Mathieu-Daudé]

Avocado documentation referred returns 404 error.
Update the broken links.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201010080741.2932406-1-philmd@redhat.com>
---
 docs/devel/testing.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 0c3e79d31cd..0aa7a13bbad 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -654,7 +654,7 @@ To manually install Avocado and its dependencies, run:
 
 Alternatively, follow the instructions on this link:
 
-  http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
+  https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
 
 Overview
 --------
@@ -837,7 +837,7 @@ Parameter reference
 To understand how Avocado parameters are accessed by tests, and how
 they can be passed to tests, please refer to::
 
-  http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#accessing-test-parameters
+  https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
 
 Parameter values can be easily seen in the log files, and will look
 like the following:
-- 
2.26.2

Re: [PULL 0/4] Acceptance testing patches for 2020-10-26

[Peter Maydell]

On Mon, 26 Oct 2020 at 19:09, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> The following changes since commit a46e72710566eea0f90f9c673a0f02da0064acce:
>
>   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20201026' into stag=
> ing (2020-10-26 14:50:03 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/philmd/qemu.git tags/acceptance-testing-20201026
>
> for you to fetch changes up to 4c9ac67230d2d8c12bfc0f3447ae353e67b43f7f:
>
>   docs/devel/testing.rst: Update outdated Avocado URLs (2020-10-26 18:20:51 +=
> 0100)
>
> ----------------------------------------------------------------
> Acceptance testing patches
>
> - More ARM tests
> - Documentation update
>
> CI jobs results:
> . https://cirrus-ci.com/build/5754555684093952
> . https://gitlab.com/philmd/qemu/-/pipelines/207807648
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.2
for any user-visible changes.

-- PMM