* [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
@ 2018-10-03 18:30 Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 1/3] acceptance tests: Add SeaBIOS boot and debug console checking test Philippe Mathieu-Daudé
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-03 18:30 UTC (permalink / raw)
To: Cleber Rosa, Eduardo Habkost, Caio Carrara
Cc: Philippe Mathieu-Daudé,
qemu-devel, Marc-André Lureau, Gerd Hoffmann, Laszlo Ersek
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
the pc and virt/aarch64 default machines
Still PoC but can be useful for the Avocado team to test the multi-arch targets.
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- use wait_for (Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)
Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code in setUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)
Regards,
Phil.
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test
acceptance tests: Add EDK2 OVMF boot and debug console checking test
acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
1 file changed, 159 insertions(+)
create mode 100644 tests/acceptance/boot_firmware.py
--
2.17.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [RFC PATCH v2 1/3] acceptance tests: Add SeaBIOS boot and debug console checking test
2018-10-03 18:30 [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Philippe Mathieu-Daudé
@ 2018-10-03 18:30 ` Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 2/3] acceptance tests: Add EDK2 OVMF " Philippe Mathieu-Daudé
` (2 subsequent siblings)
3 siblings, 0 replies; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-03 18:30 UTC (permalink / raw)
To: Cleber Rosa, Eduardo Habkost, Caio Carrara
Cc: Philippe Mathieu-Daudé,
qemu-devel, Marc-André Lureau, Gerd Hoffmann, Laszlo Ersek
This test boots SeaBIOS and check the debug console (I/O port on the ISA bus)
reports enough information on the initialized devices.
Example:
$ avocado run tests/acceptance/boot_firmware.py
JOB ID : 3dac2e738c941747ec01f043092b882a8370a92f
JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.42-3dac2e7/job.log
(1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: PASS (0.27 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 0.56 s
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
tests/acceptance/boot_firmware.py | 74 +++++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)
create mode 100644 tests/acceptance/boot_firmware.py
diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
new file mode 100644
index 0000000000..669e4849f6
--- /dev/null
+++ b/tests/acceptance/boot_firmware.py
@@ -0,0 +1,74 @@
+# coding=utf-8
+#
+# Functional test that boots SeaBIOS and checks the console
+#
+# Copyright (c) 2018 Red Hat, Inc.
+#
+# Author:
+# Philippe Mathieu-Daudé <philmd@redhat.com>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import logging
+
+from avocado_qemu import Test
+from avocado.utils.wait import wait_for
+
+
+def read_console_for_string(console, expected_string, logger, ignore_ansi=True):
+ msg = console.readline()
+ if len(msg) == 0:
+ return False
+ if logger:
+ logger.debug(msg.strip() if ignore_ansi and not '\x1b' in msg else msg)
+ return expected_string in msg
+
+
+class BootFirmware(Test):
+ """
+ Boots a firmware and checks via a console it is operational
+
+ :avocado: enable
+ """
+
+ def test_seabios(self):
+ """
+ Boots SeaBIOS on a default PC machine, checks the debug console
+
+ :avocado: tags=arch:x86_64
+ :avocado: tags=maxtime:5s
+ :avocado: tags=quick
+ """
+ debugcon_path = os.path.join(self.workdir, 'debugconsole.log')
+ serial_logger = logging.getLogger('serial')
+ debugcon_logger = logging.getLogger('debugcon')
+
+ self.vm.set_machine('pc')
+ self.vm.set_console()
+ self.vm.add_args('-nographic',
+ '-net', 'none',
+ '-global', 'isa-debugcon.iobase=0x402',
+ '-debugcon', 'file:%s' % debugcon_path)
+ self.vm.launch()
+ console = self.vm.console_socket.makefile()
+
+ # serial console checks
+ if not wait_for(read_console_for_string, timeout=5, step=0,
+ args=(console, 'No bootable device.', serial_logger)):
+ self.fail("SeaBIOS failed to boot")
+
+ # debug console checks
+ expected = [
+ 'Running on QEMU (i440fx)',
+ 'Turning on vga text mode console',
+ 'Found 1 lpt ports',
+ 'Found 1 serial ports',
+ 'PS2 keyboard initialized',
+ ]
+ with open(debugcon_path) as debugcon:
+ content = debugcon.readlines()
+ for line in content: # TODO use FDDrainer
+ debugcon_logger.debug(line.strip())
+ for exp in expected:
+ self.assertIn(exp + '\n', content)
--
2.17.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [RFC PATCH v2 2/3] acceptance tests: Add EDK2 OVMF boot and debug console checking test
2018-10-03 18:30 [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 1/3] acceptance tests: Add SeaBIOS boot and debug console checking test Philippe Mathieu-Daudé
@ 2018-10-03 18:30 ` Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and " Philippe Mathieu-Daudé
2018-10-04 12:58 ` [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Alex Bennée
3 siblings, 0 replies; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-03 18:30 UTC (permalink / raw)
To: Cleber Rosa, Eduardo Habkost, Caio Carrara
Cc: Philippe Mathieu-Daudé,
qemu-devel, Marc-André Lureau, Laszlo Ersek
This test boots OVMF and check the debug console (I/O port on the ISA bus)
report enough information on the initialized devices.
$ avocado --show=app,debugcon run tests/acceptance/boot_firmware.py
(1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc:
debugcon: SecCoreStartupWithStack(0xFFFCC000, 0x820000)
debugcon: SEC: Normal boot
...
debugcon: [Bds]Booting EFI Internal Shell
PASS (5.27 s)
JOB TIME : 5.60 s
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
tests/acceptance/boot_firmware.py | 49 +++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
index 669e4849f6..2053b1a4b6 100644
--- a/tests/acceptance/boot_firmware.py
+++ b/tests/acceptance/boot_firmware.py
@@ -72,3 +72,52 @@ class BootFirmware(Test):
debugcon_logger.debug(line.strip())
for exp in expected:
self.assertIn(exp + '\n', content)
+
+ def test_ovmf_pc(self):
+ """
+ Boots OVMF on the default PC machine, checks the debug console
+
+ :avocado: tags=arch:x86_64
+ :avocado: tags=maxtime:10s
+ """
+ debugcon_path = os.path.join(self.workdir, 'debugconsole.log')
+ serial_logger = logging.getLogger('serial')
+ debugcon_logger = logging.getLogger('debugcon')
+
+ self.vm.set_machine('pc')
+ self.vm.set_console()
+ self.vm.add_args('-nographic',
+ '-net', 'none',
+ '-global', 'isa-debugcon.iobase=0x402',
+ '-debugcon', 'file:%s' % debugcon_path,
+ '--bios', '/usr/share/OVMF/OVMF_CODE.fd')
+ self.vm.launch()
+ console = self.vm.console_socket.makefile()
+
+ # serial console checks
+ if not wait_for(read_console_for_string, timeout=10, step=0,
+ args=(console, 'EDK II', serial_logger)):
+ self.fail("OVMF failed to boot")
+
+ # debug console checks
+ expected = [
+ 'SEC: Normal boot',
+ 'S3 support was detected on QEMU',
+ 'Platform PEI Firmware Volume Initialization',
+ 'DXE IPL Entry',
+ 'Installing FVB for EMU Variable support',
+ 'SmbiosCreateTable: Initialize 32-bit entry point structure',
+ 'PlatformBootManagerBeforeConsole',
+ 'OnRootBridgesConnected: root bridges have been connected, '
+ 'installing ACPI tables',
+ 'Found LPC Bridge device',
+ 'PlatformBootManagerAfterConsole',
+ 'EfiBootManagerConnectAll',
+ '[Bds]Booting EFI Internal Shell',
+ ]
+ with open(debugcon_path) as debugcon:
+ content = debugcon.readlines()
+ for line in content: # TODO use FDDrainer
+ debugcon_logger.debug(line.strip())
+ for exp in expected:
+ self.assertIn(exp + '\r\n', content)
--
2.17.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
2018-10-03 18:30 [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 1/3] acceptance tests: Add SeaBIOS boot and debug console checking test Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 2/3] acceptance tests: Add EDK2 OVMF " Philippe Mathieu-Daudé
@ 2018-10-03 18:30 ` Philippe Mathieu-Daudé
2018-10-04 15:07 ` Laszlo Ersek
2018-10-04 12:58 ` [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Alex Bennée
3 siblings, 1 reply; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-03 18:30 UTC (permalink / raw)
To: Cleber Rosa, Eduardo Habkost, Caio Carrara
Cc: Philippe Mathieu-Daudé,
qemu-devel, Marc-André Lureau, Laszlo Ersek,
Alex Bennée, Ard Biesheuvel
This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough
information on the initialized devices.
$ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py
JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf
JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log
(1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 5.30 s
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
The '-p' option requires Avocado >= 65.0
---
tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
index 2053b1a4b6..42f0672963 100644
--- a/tests/acceptance/boot_firmware.py
+++ b/tests/acceptance/boot_firmware.py
@@ -121,3 +121,39 @@ class BootFirmware(Test):
debugcon_logger.debug(line.strip())
for exp in expected:
self.assertIn(exp + '\r\n', content)
+
+ def test_ovmf_virt(self):
+ """
+ Boots OVMF on the default virt machine, checks the debug console
+
+ :avocado: tags=arch:aarch64
+ :avocado: tags=maxtime:20s
+ """
+ image_url = ('http://snapshots.linaro.org/components/kernel/'
+ 'leg-virt-tianocore-edk2-upstream/latest/'
+ 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz')
+ image_path_gz = self.fetch_asset(image_url)
+ image_path = os.path.join(self.workdir, 'flash.img')
+
+ # kludge until Avocado support gzip files
+ import gzip, shutil
+ with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img:
+ shutil.copyfileobj(gz, img)
+
+ serial_path = os.path.join(self.workdir, 'serial.log')
+ self.vm.set_machine('virt')
+ self.vm.add_args('-nographic',
+ '-cpu', 'cortex-a57',
+ '-m', '1G', # 1GB min to boot fw?
+ '-drive', 'file=%s,format=raw,if=pflash' % image_path,
+ '-chardev', 'file,path=%s,id=console' % serial_path,
+ '-serial', 'chardev:console')
+ self.vm.launch()
+ serial_logger = logging.getLogger('serial')
+
+ # serial console checks
+ if not wait_for(read_console_for_string, timeout=15, step=0,
+ args=(open(serial_path),
+ 'Start PXE over IPv4',
+ serial_logger)):
+ self.fail("OVMF failed to boot")
--
2.17.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-03 18:30 [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Philippe Mathieu-Daudé
` (2 preceding siblings ...)
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and " Philippe Mathieu-Daudé
@ 2018-10-04 12:58 ` Alex Bennée
2018-10-04 13:04 ` Alex Bennée
2018-10-04 14:40 ` Cleber Rosa
3 siblings, 2 replies; 14+ messages in thread
From: Alex Bennée @ 2018-10-04 12:58 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Cleber Rosa, Eduardo Habkost, Caio Carrara,
Marc-André Lureau, qemu-devel, Laszlo Ersek, Gerd Hoffmann
Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> Hi,
>
> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
> the pc and virt/aarch64 default machines
>
> Still PoC but can be useful for the Avocado team to test the
> multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
(2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
(3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
(4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
(5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
(6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
(8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
(9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
The error log: https://transfer.sh/lK71v/avocado-errors.log
>
> Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
> - more Pythonic context managers (Cleber)
> - use wait_for (Cleber)
> - use workdir (Cleber)
> - use Avocado 65.0 "-p" option for aarch64 (Cleber)
> - fixed Q35 incorrect description in cover (Laszlo)
> - use ArmVirtQemu name (Laszlo)
>
> Next:
> - address Laszlo comments about correct QEMU options and flash images
> - use FDDrainer
> - refactor common code in setUp()
> - use new avocado.utils.archive gzip features (Cleber)
> - test x86_64/aarch64 'arch' tags (Cleber)
> - build EDK2 flash images (Laszlo)
> - include variable store flash image (Laszlo)
> - use virtual disk with UEFI shell script (Laszlo)
> - improve OVMF binary selection (releases / snapshots) (Laszlo)
> - check ISO images (Laszlo -> Cleber)
> - add aexpect tests (Cleber?)
>
> Regards,
>
> Phil.
>
> Philippe Mathieu-Daudé (3):
> acceptance tests: Add SeaBIOS boot and debug console checking test
> acceptance tests: Add EDK2 OVMF boot and debug console checking test
> acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
>
> tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
> 1 file changed, 159 insertions(+)
> create mode 100644 tests/acceptance/boot_firmware.py
--
Alex Bennée
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 12:58 ` [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Alex Bennée
@ 2018-10-04 13:04 ` Alex Bennée
2018-10-04 13:09 ` Peter Maydell
2018-10-04 14:44 ` Philippe Mathieu-Daudé
2018-10-04 14:40 ` Cleber Rosa
1 sibling, 2 replies; 14+ messages in thread
From: Alex Bennée @ 2018-10-04 13:04 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Cleber Rosa, Eduardo Habkost, Caio Carrara,
Marc-André Lureau, qemu-devel, Laszlo Ersek, Gerd Hoffmann
Alex Bennée <alex.bennee@linaro.org> writes:
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>
>> Hi,
>>
>> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
>> the pc and virt/aarch64 default machines
>>
>> Still PoC but can be useful for the Avocado team to test the
>> multi-arch targets.
>
> I couldn't get this to work on qemu-test (aarch64):
>
> 12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
> JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
> JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
> (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
> (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
> (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
> (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
> (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
> (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
> (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
> (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
> (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
> RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME : 142.37 s
>
> The error log: https://transfer.sh/lK71v/avocado-errors.log
It looks like it's trying to launch a pc (rather than virt) machine:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
>
>>
>> Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
>> - more Pythonic context managers (Cleber)
>> - use wait_for (Cleber)
>> - use workdir (Cleber)
>> - use Avocado 65.0 "-p" option for aarch64 (Cleber)
>> - fixed Q35 incorrect description in cover (Laszlo)
>> - use ArmVirtQemu name (Laszlo)
>>
>> Next:
>> - address Laszlo comments about correct QEMU options and flash images
>> - use FDDrainer
>> - refactor common code in setUp()
>> - use new avocado.utils.archive gzip features (Cleber)
>> - test x86_64/aarch64 'arch' tags (Cleber)
>> - build EDK2 flash images (Laszlo)
>> - include variable store flash image (Laszlo)
>> - use virtual disk with UEFI shell script (Laszlo)
>> - improve OVMF binary selection (releases / snapshots) (Laszlo)
>> - check ISO images (Laszlo -> Cleber)
>> - add aexpect tests (Cleber?)
>>
>> Regards,
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (3):
>> acceptance tests: Add SeaBIOS boot and debug console checking test
>> acceptance tests: Add EDK2 OVMF boot and debug console checking test
>> acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
>>
>> tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
>> 1 file changed, 159 insertions(+)
>> create mode 100644 tests/acceptance/boot_firmware.py
--
Alex Bennée
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 13:04 ` Alex Bennée
@ 2018-10-04 13:09 ` Peter Maydell
2018-10-04 14:42 ` Cleber Rosa
2018-10-04 14:44 ` Philippe Mathieu-Daudé
1 sibling, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2018-10-04 13:09 UTC (permalink / raw)
To: Alex Bennée
Cc: Philippe Mathieu-Daudé,
Eduardo Habkost, Cleber Rosa, QEMU Developers, Gerd Hoffmann,
Caio Carrara, Marc-André Lureau, Laszlo Ersek
On 4 October 2018 at 14:04, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>
>>> Hi,
>>>
>>> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
>>> the pc and virt/aarch64 default machines
>>>
>>> Still PoC but can be useful for the Avocado team to test the
>>> multi-arch targets.
>>
>> I couldn't get this to work on qemu-test (aarch64):
>>
>> 12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
>> JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
>> JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
>> (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
>> (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
>> (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
>> (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
>> (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
>> (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
>> (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
>> (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
>> (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
>> RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>> JOB TIME : 142.37 s
>>
>> The error log: https://transfer.sh/lK71v/avocado-errors.log
>
> It looks like it's trying to launch a pc (rather than virt) machine:
>
> 2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
That also suggests a secondary bug somewhere: if QEMU
exits immediately with an error, then the test should
fail immediately, not wait for a 15s timeout before
it notices.
thanks
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 12:58 ` [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Alex Bennée
2018-10-04 13:04 ` Alex Bennée
@ 2018-10-04 14:40 ` Cleber Rosa
2018-10-04 15:29 ` Alex Bennée
1 sibling, 1 reply; 14+ messages in thread
From: Cleber Rosa @ 2018-10-04 14:40 UTC (permalink / raw)
To: Alex Bennée, Philippe Mathieu-Daudé
Cc: Eduardo Habkost, Caio Carrara, Marc-André Lureau,
qemu-devel, Laszlo Ersek, Gerd Hoffmann
On 10/4/18 8:58 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>
>> Hi,
>>
>> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
>> the pc and virt/aarch64 default machines
>>
>> Still PoC but can be useful for the Avocado team to test the
>> multi-arch targets.
>
> I couldn't get this to work on qemu-test (aarch64):
>
> 12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
> JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
> JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
> (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
> (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
> (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
> (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
> (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
> (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
> (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
> (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
> (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
> RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME : 142.37 s
>
> The error log: https://transfer.sh/lK71v/avocado-errors.log
>
The core reason is:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output:
'qemu-system-aarch64: -machine pc: unsupported machine type\nUse
-machine help to list supported machines\n'
I'm minutes away from sending a series that adds support for arch and
machine type defaults.
- Cleber.
>>
>> Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
>> - more Pythonic context managers (Cleber)
>> - use wait_for (Cleber)
>> - use workdir (Cleber)
>> - use Avocado 65.0 "-p" option for aarch64 (Cleber)
>> - fixed Q35 incorrect description in cover (Laszlo)
>> - use ArmVirtQemu name (Laszlo)
>>
>> Next:
>> - address Laszlo comments about correct QEMU options and flash images
>> - use FDDrainer
>> - refactor common code in setUp()
>> - use new avocado.utils.archive gzip features (Cleber)
>> - test x86_64/aarch64 'arch' tags (Cleber)
>> - build EDK2 flash images (Laszlo)
>> - include variable store flash image (Laszlo)
>> - use virtual disk with UEFI shell script (Laszlo)
>> - improve OVMF binary selection (releases / snapshots) (Laszlo)
>> - check ISO images (Laszlo -> Cleber)
>> - add aexpect tests (Cleber?)
>>
>> Regards,
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (3):
>> acceptance tests: Add SeaBIOS boot and debug console checking test
>> acceptance tests: Add EDK2 OVMF boot and debug console checking test
>> acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
>>
>> tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
>> 1 file changed, 159 insertions(+)
>> create mode 100644 tests/acceptance/boot_firmware.py
>
>
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 13:09 ` Peter Maydell
@ 2018-10-04 14:42 ` Cleber Rosa
0 siblings, 0 replies; 14+ messages in thread
From: Cleber Rosa @ 2018-10-04 14:42 UTC (permalink / raw)
To: Peter Maydell, Alex Bennée
Cc: Philippe Mathieu-Daudé,
Eduardo Habkost, QEMU Developers, Gerd Hoffmann, Caio Carrara,
Marc-André Lureau, Laszlo Ersek,
Wainer dos Santos Moschetta
On 10/4/18 9:09 AM, Peter Maydell wrote:
> On 4 October 2018 at 14:04, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>>
>>>> Hi,
>>>>
>>>> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
>>>> the pc and virt/aarch64 default machines
>>>>
>>>> Still PoC but can be useful for the Avocado team to test the
>>>> multi-arch targets.
>>>
>>> I couldn't get this to work on qemu-test (aarch64):
>>>
>>> 12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
>>> JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
>>> JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
>>> (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
>>> (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
>>> (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
>>> (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
>>> (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
>>> (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
>>> (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
>>> (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
>>> (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
>>> RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>> JOB TIME : 142.37 s
>>>
>>> The error log: https://transfer.sh/lK71v/avocado-errors.log
>>
>> It looks like it's trying to launch a pc (rather than virt) machine:
>>
>> 2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
>
> That also suggests a secondary bug somewhere: if QEMU
> exits immediately with an error, then the test should
> fail immediately, not wait for a 15s timeout before
> it notices.
>
That's true, and that's due to a connection timeout to the QMP socket.
It's a known issue that we're tracking here:
https://trello.com/c/O882Vulm/44-qemumachine-even-if-we-expect-qemu-to-exit-with-an-error
And Wainer Moschetta (+cc) is working on it.
Regards,
- Cleber.
> thanks
> -- PMM
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 13:04 ` Alex Bennée
2018-10-04 13:09 ` Peter Maydell
@ 2018-10-04 14:44 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-04 14:44 UTC (permalink / raw)
To: Alex Bennée, Cleber Rosa
Cc: Eduardo Habkost, Caio Carrara, Marc-André Lureau,
qemu-devel, Laszlo Ersek, Gerd Hoffmann
Hi Alex,
On 04/10/2018 15:04, Alex Bennée wrote:
>
> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>
>>> Hi,
>>>
>>> This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
>>> the pc and virt/aarch64 default machines
>>>
>>> Still PoC but can be useful for the Avocado team to test the
>>> multi-arch targets.
>>
>> I couldn't get this to work on qemu-test (aarch64):
Thanks for giving it a try!
Sorry, I only tested this on x86_64 :/
>>
>> 12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
>> JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
>> JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
>> (1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s)
>> (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s)
>> (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s)
>> (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s)
>> (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s)
>> (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
>> (7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s)
>> (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s)
>> (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
>> RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>> JOB TIME : 142.37 s
>>
>> The error log: https://transfer.sh/lK71v/avocado-errors.log
>
> It looks like it's trying to launch a pc (rather than virt) machine:
>
> 2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
Due to some limitation (Cleber is aware of it) you have to use the -p
option as in patch #3 of this series:
$ avocado run --filter-by-tags=arch:x86_64 -p
qemu_bin=x86_64-softmmu/qemu-system-x86_64 tests/acceptance/boot_firmware.py
JOB ID : 5f9078796ef749e0606bbdf9409e91443ab2bad1
JOB LOG :
/home/phil/avocado/job-results/job-2018-10-04T14.30-5f90787/job.log
(1/2) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: PASS
(0.90 s)
(2/2) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc:
FAIL: 'SEC: Normal boot\r\n' not found in [] (9.48 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME : 10.72 s
The error "FAIL: 'SEC: Normal boot\r\n' not found" means the serial test
worked, but not the debugconsole.
You can use verbose Avocado:
$ avocado --show=app,debugcon,serial run --filter-by-tags=arch:x86_64 -p
qemu_bin=x86_64-softmmu/qemu-system-x86_64 tests/acceptance/boot_firmware.py
(2/2) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: \
serial: UEFI Interactive Shell v2.2
serial: EDK II
FAIL: 'SEC: Normal boot\r\n' not found in [] (9.58 s)
I'll look at it.
$ avocado run --filter-by-tags=arch:aarch64 -p
qemu_bin=aarch64-softmmu/qemu-system-aarch64
tests/acceptance/boot_firmware.py
JOB ID : 678d83c0b10d6daadb5fa177318b774281c346b7
JOB LOG :
/home/phil/avocado/job-results/job-2018-10-04T14.29-678d83c/job.log
(1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt:
FAIL: OVMF failed to boot (16.59 s)
RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME : 16.83 s
$ avocado --show=app,serial run --filter-by-tags=arch:aarch64 -p
qemu_bin=aarch64-softmmu/qemu-system-aarch64
tests/acceptance/boot_firmware.py
JOB LOG :
/home/phil/avocado/job-results/job-2018-10-04T14.40-dab2e26/job.log
(1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt:
serial: a
serial: dd-symbol-
serial: file /home/
serial: buildslave/
serial: workspace/l
serial: eg-virt-tia
serial: nocore-edk2
serial: -upstream/e
serial: dk2/Build/A
serial: rmVirtQemu-
serial: AARCH64/DEB
serial: UG_GCC5/AA
serial: RCH64/ArmPl
serial: atformPkg/P
serial: rePeiCore/P
serial: rePeiCoreUn
serial: iCore/DEBUG
serial: /ArmPlatfor
serial: mPrePeiCore
serial: .dll 0x1800
...
serial: Boot00
serial: 04: EFI In
serial: ternal S
serial: h
serial: ell 0x00
serial: 01
serial:
serial: PlatformRe
serial: covery Opti
serial: ons:
FAIL: OVMF failed to boot (16.44 s)
RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME : 16.68 s
So this image boots, this is my ugly way of handling virt console with
avocado which failed :)
QEMU console arg:
'-chardev', 'file,path=%s,id=console' % serial_path,
'-serial', 'chardev:console')
then I'd like to use the FDDrainer class Cleber suggested instead of the
current read_console_for_string().
Thanks,
Phil.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and " Philippe Mathieu-Daudé
@ 2018-10-04 15:07 ` Laszlo Ersek
2018-10-04 15:15 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 14+ messages in thread
From: Laszlo Ersek @ 2018-10-04 15:07 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, Cleber Rosa, Eduardo Habkost, Caio Carrara
Cc: qemu-devel, Marc-André Lureau, Alex Bennée, Ard Biesheuvel
On 10/03/18 20:30, Philippe Mathieu-Daudé wrote:
> This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough
> information on the initialized devices.
>
> $ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py
> JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf
> JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log
> (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s)
> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME : 5.30 s
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> The '-p' option requires Avocado >= 65.0
> ---
> tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
> diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
> index 2053b1a4b6..42f0672963 100644
> --- a/tests/acceptance/boot_firmware.py
> +++ b/tests/acceptance/boot_firmware.py
> @@ -121,3 +121,39 @@ class BootFirmware(Test):
> debugcon_logger.debug(line.strip())
> for exp in expected:
> self.assertIn(exp + '\r\n', content)
> +
> + def test_ovmf_virt(self):
> + """
> + Boots OVMF on the default virt machine, checks the debug console
> +
> + :avocado: tags=arch:aarch64
> + :avocado: tags=maxtime:20s
> + """
> + image_url = ('http://snapshots.linaro.org/components/kernel/'
> + 'leg-virt-tianocore-edk2-upstream/latest/'
> + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz')
> + image_path_gz = self.fetch_asset(image_url)
> + image_path = os.path.join(self.workdir, 'flash.img')
> +
> + # kludge until Avocado support gzip files
> + import gzip, shutil
> + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img:
> + shutil.copyfileobj(gz, img)
> +
> + serial_path = os.path.join(self.workdir, 'serial.log')
> + self.vm.set_machine('virt')
> + self.vm.add_args('-nographic',
> + '-cpu', 'cortex-a57',
> + '-m', '1G', # 1GB min to boot fw?
> + '-drive', 'file=%s,format=raw,if=pflash' % image_path,
> + '-chardev', 'file,path=%s,id=console' % serial_path,
> + '-serial', 'chardev:console')
> + self.vm.launch()
> + serial_logger = logging.getLogger('serial')
> +
> + # serial console checks
> + if not wait_for(read_console_for_string, timeout=15, step=0,
> + args=(open(serial_path),
> + 'Start PXE over IPv4',
> + serial_logger)):
> + self.fail("OVMF failed to boot")
>
I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as
well.
I'd also suggest renaming the method test_ovmf_virt() to
test_armvirtqemu_virt().
If that seems baroque, then I suggest "test_uefi_<machtype>" in both
patches #2 and #3.
(Sorry about my obsession with these names; just trying to capture the
existing "vernacular".)
Thanks
Laszlo
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
2018-10-04 15:07 ` Laszlo Ersek
@ 2018-10-04 15:15 ` Philippe Mathieu-Daudé
2018-10-04 16:19 ` Laszlo Ersek
0 siblings, 1 reply; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2018-10-04 15:15 UTC (permalink / raw)
To: Laszlo Ersek
Cc: Cleber Rosa, Eduardo Habkost, Caio Carrara, qemu-devel,
Marc-André Lureau, Alex Bennée, Ard Biesheuvel
On 04/10/2018 17:07, Laszlo Ersek wrote:
> On 10/03/18 20:30, Philippe Mathieu-Daudé wrote:
>> This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough
>> information on the initialized devices.
>>
>> $ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py
>> JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf
>> JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log
>> (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s)
>> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>> JOB TIME : 5.30 s
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>> The '-p' option requires Avocado >= 65.0
>> ---
>> tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++
>> 1 file changed, 36 insertions(+)
>>
>> diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
>> index 2053b1a4b6..42f0672963 100644
>> --- a/tests/acceptance/boot_firmware.py
>> +++ b/tests/acceptance/boot_firmware.py
>> @@ -121,3 +121,39 @@ class BootFirmware(Test):
>> debugcon_logger.debug(line.strip())
>> for exp in expected:
>> self.assertIn(exp + '\r\n', content)
>> +
>> + def test_ovmf_virt(self):
>> + """
>> + Boots OVMF on the default virt machine, checks the debug console
>> +
>> + :avocado: tags=arch:aarch64
>> + :avocado: tags=maxtime:20s
>> + """
>> + image_url = ('http://snapshots.linaro.org/components/kernel/'
>> + 'leg-virt-tianocore-edk2-upstream/latest/'
>> + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz')
>> + image_path_gz = self.fetch_asset(image_url)
>> + image_path = os.path.join(self.workdir, 'flash.img')
>> +
>> + # kludge until Avocado support gzip files
>> + import gzip, shutil
>> + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img:
>> + shutil.copyfileobj(gz, img)
>> +
>> + serial_path = os.path.join(self.workdir, 'serial.log')
>> + self.vm.set_machine('virt')
>> + self.vm.add_args('-nographic',
>> + '-cpu', 'cortex-a57',
>> + '-m', '1G', # 1GB min to boot fw?
>> + '-drive', 'file=%s,format=raw,if=pflash' % image_path,
>> + '-chardev', 'file,path=%s,id=console' % serial_path,
>> + '-serial', 'chardev:console')
>> + self.vm.launch()
>> + serial_logger = logging.getLogger('serial')
>> +
>> + # serial console checks
>> + if not wait_for(read_console_for_string, timeout=15, step=0,
>> + args=(open(serial_path),
>> + 'Start PXE over IPv4',
>> + serial_logger)):
>> + self.fail("OVMF failed to boot")
>>
>
> I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as
> well.
Oops OK.
>
> I'd also suggest renaming the method test_ovmf_virt() to
> test_armvirtqemu_virt().
>
> If that seems baroque, then I suggest "test_uefi_<machtype>" in both
> patches #2 and #3.
What about this?
- test_bios_seabios_x86_64_pc()
- test_uefi_ovmf_x86_64_pc()
- test_uefi_armvirtqemu_aarch64_virt()
>
> (Sorry about my obsession with these names; just trying to capture the
> existing "vernacular".)
>
> Thanks
> Laszlo
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output
2018-10-04 14:40 ` Cleber Rosa
@ 2018-10-04 15:29 ` Alex Bennée
0 siblings, 0 replies; 14+ messages in thread
From: Alex Bennée @ 2018-10-04 15:29 UTC (permalink / raw)
To: Cleber Rosa
Cc: Philippe Mathieu-Daudé,
Eduardo Habkost, Caio Carrara, Marc-André Lureau,
qemu-devel, Laszlo Ersek, Gerd Hoffmann
Cleber Rosa <crosa@redhat.com> writes:
> On 10/4/18 8:58 AM, Alex Bennée wrote:
>>
<snip>
>>
>> The error log: https://transfer.sh/lK71v/avocado-errors.log
>>
>
> The core reason is:
>
> 2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output:
> 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse
> -machine help to list supported machines\n'
>
> I'm minutes away from sending a series that adds support for arch and
> machine type defaults.
\o/
I see it now, I shall have a look.
>
> - Cleber.
>
>>>
>>> Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
>>> - more Pythonic context managers (Cleber)
>>> - use wait_for (Cleber)
>>> - use workdir (Cleber)
>>> - use Avocado 65.0 "-p" option for aarch64 (Cleber)
>>> - fixed Q35 incorrect description in cover (Laszlo)
>>> - use ArmVirtQemu name (Laszlo)
>>>
>>> Next:
>>> - address Laszlo comments about correct QEMU options and flash images
>>> - use FDDrainer
>>> - refactor common code in setUp()
>>> - use new avocado.utils.archive gzip features (Cleber)
>>> - test x86_64/aarch64 'arch' tags (Cleber)
>>> - build EDK2 flash images (Laszlo)
>>> - include variable store flash image (Laszlo)
>>> - use virtual disk with UEFI shell script (Laszlo)
>>> - improve OVMF binary selection (releases / snapshots) (Laszlo)
>>> - check ISO images (Laszlo -> Cleber)
>>> - add aexpect tests (Cleber?)
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> Philippe Mathieu-Daudé (3):
>>> acceptance tests: Add SeaBIOS boot and debug console checking test
>>> acceptance tests: Add EDK2 OVMF boot and debug console checking test
>>> acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
>>>
>>> tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++
>>> 1 file changed, 159 insertions(+)
>>> create mode 100644 tests/acceptance/boot_firmware.py
>>
>>
>> --
>> Alex Bennée
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
2018-10-04 15:15 ` Philippe Mathieu-Daudé
@ 2018-10-04 16:19 ` Laszlo Ersek
0 siblings, 0 replies; 14+ messages in thread
From: Laszlo Ersek @ 2018-10-04 16:19 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Cleber Rosa, Eduardo Habkost, Caio Carrara, qemu-devel,
Marc-André Lureau, Alex Bennée, Ard Biesheuvel
On 10/04/18 17:15, Philippe Mathieu-Daudé wrote:
> On 04/10/2018 17:07, Laszlo Ersek wrote:
>> On 10/03/18 20:30, Philippe Mathieu-Daudé wrote:
>>> This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough
>>> information on the initialized devices.
>>>
>>> $ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py
>>> JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf
>>> JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log
>>> (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s)
>>> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>> JOB TIME : 5.30 s
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>> The '-p' option requires Avocado >= 65.0
>>> ---
>>> tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++
>>> 1 file changed, 36 insertions(+)
>>>
>>> diff --git a/tests/acceptance/boot_firmware.py b/tests/acceptance/boot_firmware.py
>>> index 2053b1a4b6..42f0672963 100644
>>> --- a/tests/acceptance/boot_firmware.py
>>> +++ b/tests/acceptance/boot_firmware.py
>>> @@ -121,3 +121,39 @@ class BootFirmware(Test):
>>> debugcon_logger.debug(line.strip())
>>> for exp in expected:
>>> self.assertIn(exp + '\r\n', content)
>>> +
>>> + def test_ovmf_virt(self):
>>> + """
>>> + Boots OVMF on the default virt machine, checks the debug console
>>> +
>>> + :avocado: tags=arch:aarch64
>>> + :avocado: tags=maxtime:20s
>>> + """
>>> + image_url = ('http://snapshots.linaro.org/components/kernel/'
>>> + 'leg-virt-tianocore-edk2-upstream/latest/'
>>> + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz')
>>> + image_path_gz = self.fetch_asset(image_url)
>>> + image_path = os.path.join(self.workdir, 'flash.img')
>>> +
>>> + # kludge until Avocado support gzip files
>>> + import gzip, shutil
>>> + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img:
>>> + shutil.copyfileobj(gz, img)
>>> +
>>> + serial_path = os.path.join(self.workdir, 'serial.log')
>>> + self.vm.set_machine('virt')
>>> + self.vm.add_args('-nographic',
>>> + '-cpu', 'cortex-a57',
>>> + '-m', '1G', # 1GB min to boot fw?
>>> + '-drive', 'file=%s,format=raw,if=pflash' % image_path,
>>> + '-chardev', 'file,path=%s,id=console' % serial_path,
>>> + '-serial', 'chardev:console')
>>> + self.vm.launch()
>>> + serial_logger = logging.getLogger('serial')
>>> +
>>> + # serial console checks
>>> + if not wait_for(read_console_for_string, timeout=15, step=0,
>>> + args=(open(serial_path),
>>> + 'Start PXE over IPv4',
>>> + serial_logger)):
>>> + self.fail("OVMF failed to boot")
>>>
>>
>> I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as
>> well.
>
> Oops OK.
>
>>
>> I'd also suggest renaming the method test_ovmf_virt() to
>> test_armvirtqemu_virt().
>>
>> If that seems baroque, then I suggest "test_uefi_<machtype>" in both
>> patches #2 and #3.
>
> What about this?
>
> - test_bios_seabios_x86_64_pc()
> - test_uefi_ovmf_x86_64_pc()
> - test_uefi_armvirtqemu_aarch64_virt()
Those work for me :)
Laszlo
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2018-10-04 16:19 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-03 18:30 [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 1/3] acceptance tests: Add SeaBIOS boot and debug console checking test Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 2/3] acceptance tests: Add EDK2 OVMF " Philippe Mathieu-Daudé
2018-10-03 18:30 ` [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and " Philippe Mathieu-Daudé
2018-10-04 15:07 ` Laszlo Ersek
2018-10-04 15:15 ` Philippe Mathieu-Daudé
2018-10-04 16:19 ` Laszlo Ersek
2018-10-04 12:58 ` [Qemu-devel] [RFC PATCH v2 0/3] acceptance tests: Test firmware checking debug console output Alex Bennée
2018-10-04 13:04 ` Alex Bennée
2018-10-04 13:09 ` Peter Maydell
2018-10-04 14:42 ` Cleber Rosa
2018-10-04 14:44 ` Philippe Mathieu-Daudé
2018-10-04 14:40 ` Cleber Rosa
2018-10-04 15:29 ` Alex Bennée
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.