[Qemu-devel] [PATCH v3 0/3] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Qemu-devel] [PATCH v3 0/3] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Philippe Mathieu-Daudé]

Hi,

I was looking at Thomas' last series [*] where he adds the
NeXTcube machine, thinking about enforcing a new rule "new
machines must have tests". Then I realized the UART is not
emitting characters, so our current sample tests are not
helpful.

Since the framebuffer is working, I gave a try at dumping the
screen content via the HMP 'screendump' command, then parsing
the screenshot with an OCR tool.

The default ROM dump the bootlog to a console. Using the old
good tesseract tool we can recover some useful words to be
sure the guest is sane, its framebuffer is definitively working.

This test takes less than 6s on Travis-CI:
https://travis-ci.org/philmd/qemu/builds/552174983#L1836

   AVOCADO tests/acceptance
 (3/9) /home/travis/build/philmd/qemu/tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer:  PASS (5.69 s)

Since v2: https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg00249.html
- addressed Thomas and Cleber review comments (detailed in each patch)
- add RFC to interract sending VNC keys

Since v1: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg06514.html
- use the English dictionary (Thomas)
- support tesseract v3 and v4 (much better results with v4, but not
  all distros provide it)
- add a test of the framebuffer width/height

Regards,

Phil.

Based-on: 20190709073222.26370-1-huth@tuxfamily.org
[*] "m68k: Add basic support for the NeXTcube machine"
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg02152.html

Philippe Mathieu-Daudé (3):
  tests/acceptance: Add test of NeXTcube framebuffer using OCR
  tests/acceptance: Run commands sending VNC keys
  .travis.yml: Let the avocado job run the NeXTcube tests

 .travis.yml                               |   7 +-
 tests/acceptance/machine_m68k_nextcube.py | 170 ++++++++++++++++++++++
 tests/requirements.txt                    |   1 +
 3 files changed, 177 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py

-- 
2.20.1

[Qemu-devel] [PATCH v3 1/3] tests/acceptance: Add test of NeXTcube framebuffer using OCR

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Add a test of the NeXTcube framebuffer using the Tesseract OCR
engine on a screenshot of the framebuffer device.

The test is very quick:

  $ avocado --show=app,console run tests/acceptance/machine_m68k_nextcube.py
  JOB ID     : 78844a92424cc495bd068c3874d542d1e20f24bc
  JOB LOG    : /home/phil/avocado/job-results/job-2019-08-13T13.16-78844a9/job.log
   (1/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_size: PASS (2.16 s)
   (2/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v3: -
  ue r pun Honl'flx ; 5‘ 55‘
  avg ncaaaaa 25 MHZ, memary jag m
  Backplane slat «a
  Ethernet address a a r a r3 2
  Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  [...]
  Yestlnq the rpu, 5::
  system test raneg Errar egge 51
  Egg: cammand
  Default pggc devlce nut fauna
  NEXY>I
  PASS (2.64 s)
   (3/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v4: SKIP: tesseract v4 OCR tool not available
  RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 5.35 s

Documentation on how to install tesseract:
  https://github.com/tesseract-ocr/tesseract/wiki#installation

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v2:
- test fb sizes
- handle 2 versions of teseract
v3:
- sort imports
- use of ALL_CAPS variables if they are at the module level (PIL_AVAILABLE)
- use find_command from avocado.utils.path
- check tesseract version using regex to avoid exceptions
- do not use '-' in avocado tags
- use use os.path.join()
- add few comments
---
 tests/acceptance/machine_m68k_nextcube.py | 121 ++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
new file mode 100644
index 0000000000..e09cab9f20
--- /dev/null
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -0,0 +1,121 @@
+# Functional test that boots a VM and run OCR on the framebuffer
+#
+# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import re
+import time
+import logging
+import distutils.spawn
+
+from avocado_qemu import Test
+from avocado import skipUnless
+from avocado.utils import process
+from avocado.utils.path import find_command, CmdNotFoundError
+
+PIL_AVAILABLE = True
+try:
+    from PIL import Image
+except ImportError:
+    PIL_AVAILABLE = False
+
+
+def tesseract_available(expected_version):
+    try:
+        find_command('tesseract')
+    except CmdNotFoundError:
+        return False
+    res = process.run('tesseract --version')
+    try:
+        version = res.stdout_text.split()[1]
+    except IndexError:
+        version = res.stderr_text.split()[1]
+    return int(version.split('.')[0]) == expected_version
+
+    match = re.match(r'tesseract\s(\d)', res)
+    if match is None:
+        return False
+    # now this is guaranteed to be a digit
+    return int(match.groups()[0]) == expected_version
+
+
+class NextCubeMachine(Test):
+
+    timeout = 15
+
+    def check_bootrom_framebuffer(self, screenshot_path):
+        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
+                   '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN')
+        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
+        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+
+        self.vm.set_machine('next-cube')
+        self.vm.add_args('-bios', rom_path)
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for display')
+        # TODO: Use avocado.utils.wait.wait_for to catch the
+        #       'displaysurface_create 1120x832' trace-event.
+        time.sleep(2)
+
+        self.vm.command('human-monitor-command',
+                        command_line='screendump %s' % screenshot_path)
+
+    @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
+    def test_bootrom_framebuffer_size(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        width, height = Image.open(screenshot_path).size
+        self.assertEqual(width, 1120)
+        self.assertEqual(height, 832)
+
+    @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
+    def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        console_logger = logging.getLogger('console')
+        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('Backplane', text)
+        self.assertIn('Ethernet address', text)
+
+    # Tesseract 4 adds a new OCR engine based on LSTM neural networks. The
+    # new version is faster and more accurate than version 3. The drawback is
+    # that it is still alpha-level software.
+    @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available')
+    def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        console_logger = logging.getLogger('console')
+        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
+        text = proc.stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('Testing the FPU, SCC', text)
+        self.assertIn('System test failed. Error code 51', text)
+        self.assertIn('Boot command', text)
+        self.assertIn('Next>', text)
-- 
2.20.1

[Qemu-devel] [PATCH v3 2/3] .travis.yml: Let the avocado job run the NeXTcube tests

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v2:
- install tesseract English package (Thomas)
- install PIL
---
 .travis.yml | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index caf0a1f8fa..f6dea2492c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -232,15 +232,20 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
       addons:
         apt:
           packages:
+            - python3-pil
             - python3-pip
             - python3.5-venv
+            - tesseract-ocr
+            - tesseract-ocr-eng
+
+
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.20.1

[Qemu-devel] [RFC PATCH v3 3/3] tests/acceptance: Run commands sending VNC keys

[Philippe Mathieu-Daudé]

Proof of concept to interract with a framebuffer via VNC.

This test send 'mem' and 'help' to the firmware prompt
within the single X terminal opened.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
This test is limited because the '-vnc' option only allow
to bind TCP port. To allow parallel testing, I should try
to bind within a range instead of ':0'.
---
 tests/acceptance/machine_m68k_nextcube.py | 49 +++++++++++++++++++++++
 tests/requirements.txt                    |  1 +
 2 files changed, 50 insertions(+)

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index e09cab9f20..f4f853dfa1 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -13,6 +13,7 @@ import distutils.spawn
 
 from avocado_qemu import Test
 from avocado import skipUnless
+from vncdotool import api as vnc
 from avocado.utils import process
 from avocado.utils.path import find_command, CmdNotFoundError
 
@@ -119,3 +120,51 @@ class NextCubeMachine(Test):
         self.assertIn('System test failed. Error code 51', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)
+
+
+    @staticmethod
+    def vnc_send_string(client, string, eol=True):
+        for key in string:
+            client.keyPress(key)
+            time.sleep(0.02)
+        if eol:
+            client.keyPress('enter')
+            time.sleep(0.02)
+
+    @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
+    def test_bootrom_via_vnc_with_tesseract_v3(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+
+        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
+                   '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN')
+        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
+        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+
+        self.vm.set_machine('next-cube')
+        self.vm.add_args('-bios', rom_path,
+                         '-vnc', ':0') # XXX do not use static TCP port...
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for display')
+        # TODO: Use avocado.utils.wait.wait_for to catch the
+        #       'displaysurface_create 1120x832' trace-event.
+        time.sleep(2)
+
+        with vnc.connect('127.0.0.1') as client:
+            self.vnc_send_string(client, 'help')
+            self.vnc_send_string(client, 'mem')
+            client.captureScreen(screenshot_path)
+            client.disconnect()
+
+        console_logger = logging.getLogger('console')
+        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('address space', text)
+        self.assertIn('System', text)
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 3ae0e29ad7..f7467287a1 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -3,3 +3,4 @@
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 avocado-framework==68.0
 paramiko
+vncdotool
-- 
2.20.1

Re: [Qemu-devel] [RFC PATCH v3 3/3] tests/acceptance: Run commands sending VNC keys

[Thomas Huth]

Am Tue, 13 Aug 2019 15:49:21 +0200
schrieb Philippe Mathieu-Daudé <philmd@redhat.com>:

> Proof of concept to interract with a framebuffer via VNC.
> 
> This test send 'mem' and 'help' to the firmware prompt
> within the single X terminal opened.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> This test is limited because the '-vnc' option only allow
> to bind TCP port. To allow parallel testing, I should try
> to bind within a range instead of ':0'.

Maybe it would be better to use the "sendkey" HMP command here?

 Thomas