Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output

From: "Alex Bennée" <alex.bennee@linaro.org>
To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
Cc: "Cleber Rosa" <crosa@redhat.com>,
	"Eduardo Habkost" <ehabkost@redhat.com>,
	"Caio Carrara" <ccarrara@redhat.com>,
	"Marc-André Lureau" <marcandre.lureau@redhat.com>,
	qemu-devel@nongnu.org, "Laszlo Ersek" <lersek@redhat.com>,
	"Gerd Hoffmann" <kraxel@redhat.com>
Subject: Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
Date: Thu, 04 Oct 2018 13:58:18 +0100	[thread overview]
Message-ID: <87va6hrhqt.fsf@linaro.org> (raw)
In-Reply-To: <20181003183036.6716-1-philmd@redhat.com>

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Hi,
>
> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
> the pc and virt/aarch64 default machines
>
> Still PoC but can be useful for the Avocado team to test the
> multi-arch targets.

I couldn't get this to work on qemu-test (aarch64):

  12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
  JOB ID     : b6377b8d4196903846c7d57e2b234c523d6c6ba1
  JOB LOG    : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
   (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
   (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
   (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
   (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
   (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
   (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
   (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
   (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
   (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
  RESULTS    : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 142.37 s

The error log: https://transfer.sh/lK71v/avocado-errors.log

>
> Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
> - more Pythonic context managers (Cleber)
> - use wait_for (Cleber)
> - use workdir (Cleber)
> - use Avocado 65.0 "-p" option for aarch64 (Cleber)
> - fixed Q35 incorrect description in cover (Laszlo)
> - use ArmVirtQemu name (Laszlo)
>
> Next:
> - address Laszlo comments about correct QEMU options and flash images
> - use FDDrainer
> - refactor common code in setUp()
> - use new avocado.utils.archive gzip features (Cleber)
> - test x86_64/aarch64 'arch' tags (Cleber)
> - build EDK2 flash images (Laszlo)
> - include variable store flash image (Laszlo)
> - use virtual disk with UEFI shell script (Laszlo)
> - improve OVMF binary selection (releases / snapshots) (Laszlo)
> - check ISO images (Laszlo -> Cleber)
> - add aexpect tests (Cleber?)
>
> Regards,
>
> Phil.
>
> Philippe Mathieu-Daudé (3):
>   acceptance tests: Add SeaBIOS boot and debug console checking test
>   acceptance tests: Add EDK2 OVMF boot and debug console checking test
>   acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
>
>  tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
>  1 file changed, 159 insertions(+)
>  create mode 100644 tests/acceptance/boot_firmware.py

--
Alex Bennée