[PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[Philippe Mathieu-Daudé]

This series add tests for the MIPS Fuloong 2E machine.

I have been using these tests for over 1 year now.
After recent discussions with Huacai and Mark regarding
how to contribute tests for machines using private blob,
I think it is time to publish this series, to see how
the final part will be discussed, as it tests a blob
which Lemote company doesn't publish the sources. The
binary is know to work and has been used for various
years on real hardware. Its MD5 is also know (I am not
using SHA-1 because it has been published with MD5).

The idea is to be able to share tests without infringing
licenses issues.

Please comment :)

Phil.

Philippe Mathieu-Daudé (6):
  tests/acceptance: Remove unused import
  tests/acceptance: Use .ppm extention for Portable PixMap files
  tests/acceptance: Extract tesseract_available() helper in new
    namespace
  tests/acceptance: Introduce tesseract_ocr() helper
  tests/acceptance: Test Linux on the Fuloong 2E machine
  tests/acceptance: Test PMON on the Fuloong 2E machine

 MAINTAINERS                                |   1 +
 tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
 tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
 tests/acceptance/tesseract_utils.py        |  46 ++++++++++
 4 files changed, 158 insertions(+), 41 deletions(-)
 create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
 create mode 100644 tests/acceptance/tesseract_utils.py

-- 
2.26.2

[PATCH 1/6] tests/acceptance: Remove unused import

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_m68k_nextcube.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index 32cf571f948..9d289f2fab7 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -9,7 +9,6 @@
 import re
 import time
 import logging
-import distutils.spawn
 
 from avocado_qemu import Test
 from avocado import skipUnless
-- 
2.26.2

[PATCH 2/6] tests/acceptance: Use .ppm extention for Portable PixMap files

[Philippe Mathieu-Daudé]

The HMP 'screendump' command generates Portable PixMap files.
Make it obvious by using the .ppm file extention.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_m68k_nextcube.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index 9d289f2fab7..2baba5fdc26 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -69,7 +69,7 @@ def check_bootrom_framebuffer(self, screenshot_path):
 
     @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
     def test_bootrom_framebuffer_size(self):
-        screenshot_path = os.path.join(self.workdir, "dump.png")
+        screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
         width, height = Image.open(screenshot_path).size
@@ -78,7 +78,7 @@ def test_bootrom_framebuffer_size(self):
 
     @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
-        screenshot_path = os.path.join(self.workdir, "dump.png")
+        screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
         console_logger = logging.getLogger('console')
@@ -94,7 +94,7 @@ def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
     # that it is still alpha-level software.
     @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
-        screenshot_path = os.path.join(self.workdir, "dump.png")
+        screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
 
         console_logger = logging.getLogger('console')
-- 
2.26.2

[PATCH 3/6] tests/acceptance: Extract tesseract_available() helper in new namespace

[Philippe Mathieu-Daudé]

We are going to reuse tesseract_available(). Extract it to
a new 'tesseract_utils' namespace.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_m68k_nextcube.py | 25 +++-----------------
 tests/acceptance/tesseract_utils.py       | 28 +++++++++++++++++++++++
 2 files changed, 31 insertions(+), 22 deletions(-)
 create mode 100644 tests/acceptance/tesseract_utils.py

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index 2baba5fdc26..3c7400c43e4 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -1,19 +1,19 @@
 # Functional test that boots a VM and run OCR on the framebuffer
 #
-# Copyright (c) Philippe Mathieu-Daudé <f4bug@amsat.org>
+# Copyright (c) 2019 Philippe Mathieu-Daudé <f4bug@amsat.org>
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
 import os
-import re
 import time
 import logging
 
 from avocado_qemu import Test
 from avocado import skipUnless
 from avocado.utils import process
-from avocado.utils.path import find_command, CmdNotFoundError
+
+from tesseract_utils import tesseract_available
 
 PIL_AVAILABLE = True
 try:
@@ -22,25 +22,6 @@
     PIL_AVAILABLE = False
 
 
-def tesseract_available(expected_version):
-    try:
-        find_command('tesseract')
-    except CmdNotFoundError:
-        return False
-    res = process.run('tesseract --version')
-    try:
-        version = res.stdout_text.split()[1]
-    except IndexError:
-        version = res.stderr_text.split()[1]
-    return int(version.split('.')[0]) == expected_version
-
-    match = re.match(r'tesseract\s(\d)', res)
-    if match is None:
-        return False
-    # now this is guaranteed to be a digit
-    return int(match.groups()[0]) == expected_version
-
-
 class NextCubeMachine(Test):
     """
     :avocado: tags=arch:m68k
diff --git a/tests/acceptance/tesseract_utils.py b/tests/acceptance/tesseract_utils.py
new file mode 100644
index 00000000000..acd6e8c2faa
--- /dev/null
+++ b/tests/acceptance/tesseract_utils.py
@@ -0,0 +1,28 @@
+# ...
+#
+# Copyright (c) 2019 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import re
+
+from avocado.utils.path import find_command, CmdNotFoundError
+
+def tesseract_available(expected_version):
+    try:
+        find_command('tesseract')
+    except CmdNotFoundError:
+        return False
+    res = process.run('tesseract --version')
+    try:
+        version = res.stdout_text.split()[1]
+    except IndexError:
+        version = res.stderr_text.split()[1]
+    return int(version.split('.')[0]) == expected_version
+
+    match = re.match(r'tesseract\s(\d)', res)
+    if match is None:
+        return False
+    # now this is guaranteed to be a digit
+    return int(match.groups()[0]) == expected_version
-- 
2.26.2

[PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Philippe Mathieu-Daudé]

We are going to reuse the tesseract OCR code.
Create a new tesseract_ocr() helper and use it.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
 tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+), 16 deletions(-)

diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index 3c7400c43e4..09e2745cc52 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -7,13 +7,11 @@
 
 import os
 import time
-import logging
 
 from avocado_qemu import Test
 from avocado import skipUnless
-from avocado.utils import process
 
-from tesseract_utils import tesseract_available
+from tesseract_utils import tesseract_available, tesseract_ocr
 
 PIL_AVAILABLE = True
 try:
@@ -61,12 +59,8 @@ def test_bootrom_framebuffer_size(self):
     def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
-
-        console_logger = logging.getLogger('console')
-        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
-        for line in text.split('\n'):
-            if len(line):
-                console_logger.debug(line)
+        lines = tesseract_ocr(screenshot_path, tesseract_version=3)
+        text = '\n'.join(lines)
         self.assertIn('Backplane', text)
         self.assertIn('Ethernet address', text)
 
@@ -77,13 +71,8 @@ def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
     def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
         screenshot_path = os.path.join(self.workdir, "dump.ppm")
         self.check_bootrom_framebuffer(screenshot_path)
-
-        console_logger = logging.getLogger('console')
-        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
-        text = proc.stdout_text
-        for line in text.split('\n'):
-            if len(line):
-                console_logger.debug(line)
+        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
+        text = '\n'.join(lines)
         self.assertIn('Testing the FPU, SCC', text)
         self.assertIn('System test failed. Error code', text)
         self.assertIn('Boot command', text)
diff --git a/tests/acceptance/tesseract_utils.py b/tests/acceptance/tesseract_utils.py
index acd6e8c2faa..72cd9ab7989 100644
--- a/tests/acceptance/tesseract_utils.py
+++ b/tests/acceptance/tesseract_utils.py
@@ -6,7 +6,9 @@
 # later. See the COPYING file in the top-level directory.
 
 import re
+import logging
 
+from avocado.utils import process
 from avocado.utils.path import find_command, CmdNotFoundError
 
 def tesseract_available(expected_version):
@@ -26,3 +28,19 @@ def tesseract_available(expected_version):
         return False
     # now this is guaranteed to be a digit
     return int(match.groups()[0]) == expected_version
+
+
+def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
+    console_logger = logging.getLogger('tesseract')
+    console_logger.debug(image_path)
+    if tesseract_version == 4:
+        tesseract_args += ' --oem 1'
+    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
+                                                       image_path))
+    lines = []
+    for line in proc.stdout_text.split('\n'):
+        sline = line.strip()
+        if len(sline):
+            console_logger.debug(sline)
+            lines += [sline]
+    return lines
-- 
2.26.2

[PATCH 5/6] tests/acceptance: Test Linux on the Fuloong 2E machine

[Philippe Mathieu-Daudé]

Trivial test booting the rescue image provided by Lemote:

  $ AVOCADO_ALLOW_UNTRUSTED_CODE=1 \
    avocado --show=app,console run tests/acceptance/machine_mips_fuloong2e.py
  Fetching asset from tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_linux_kernel_isa_serial
   (1/1) tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_linux_kernel_isa_serial:
  console: Linux version 2.6.27.7lemote (root@debian) (gcc version 4.1.3 20080623 (prerelease) (Debian 4.1.2-23)) #6 Fri Dec 12 00:11:25 CST 2008
  console: busclock=33000000, cpuclock=-2145008360,memsize=256,highmemsize=0
  console: console [early0] enabled
  console: CPU revision is: 00006302 (ICT Loongson-2)
  PASS (0.16 s)
  JOB TIME   : 0.51 s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 MAINTAINERS                                |  1 +
 tests/acceptance/machine_mips_fuloong2e.py | 42 ++++++++++++++++++++++
 2 files changed, 43 insertions(+)
 create mode 100644 tests/acceptance/machine_mips_fuloong2e.py

diff --git a/MAINTAINERS b/MAINTAINERS
index a7f0acf8663..1720a383e6f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1152,6 +1152,7 @@ F: hw/mips/fuloong2e.c
 F: hw/isa/vt82c686.c
 F: hw/pci-host/bonito.c
 F: include/hw/isa/vt82c686.h
+F: tests/acceptance/machine_mips_fuloong2e.py
 
 Loongson-3 virtual platforms
 M: Huacai Chen <chenhc@lemote.com>
diff --git a/tests/acceptance/machine_mips_fuloong2e.py b/tests/acceptance/machine_mips_fuloong2e.py
new file mode 100644
index 00000000000..5fd3da09e7c
--- /dev/null
+++ b/tests/acceptance/machine_mips_fuloong2e.py
@@ -0,0 +1,42 @@
+# Functional tests for the Lemote Fuloong-2E machine.
+#
+# Copyright (c) 2019 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+
+from avocado import skipUnless
+from avocado_qemu import Test
+from avocado_qemu import wait_for_console_pattern
+
+class MipsFuloong2e(Test):
+
+    timeout = 60
+
+    @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+    def test_linux_kernel_isa_serial(self):
+        """
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:fulong2e
+        :avocado: tags=endian:little
+        :avocado: tags=device:bonito64
+        :avocado: tags=device:via686b
+        """
+        # Recovery system for the Yeeloong laptop
+        # (enough to test the fuloong2e southbridge, accessing its ISA bus)
+        # http://dev.lemote.com/files/resource/download/rescue/rescue-yl
+        kernel_url = ('https://github.com/philmd/qemu-testing-blob/'
+                      'raw/f6b2e7e4/mips/yeeloong/mips64el/rescue-yl')
+        kernel_hash = 'ec4d1bd89a8439c41033ca63db60160cc6d6f09a'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        self.vm.set_console()
+        self.vm.add_args('-kernel', kernel_path)
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Linux version 2.6.27.7lemote')
+        cpu_revision = 'CPU revision is: 00006302 (ICT Loongson-2)'
+        wait_for_console_pattern(self, cpu_revision)
-- 
2.26.2

[PATCH 6/6] tests/acceptance: Test PMON on the Fuloong 2E machine

[Philippe Mathieu-Daudé]

Test the PMON firmware. As the firmware is not redistributable,
it has to be downloaded manually first. Then it can be used by
providing its path via the PMON_BIN_PATH environment variable:

  $ PMON_BIN_PATH=~/images/fuloong2e/pmon_2e.bin \
    AVOCADO_ALLOW_UNTRUSTED_CODE=1 \
    avocado --show=app,console run tests/acceptance/machine_mips_fuloong2e.py
  Fetching asset from tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_linux_kernel_isa_serial
   (1/3) tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_pmon_serial_console:
  console: PMON2000 MIPS Initializing. Standby...
  console: ERRORPC=00000000 CONFIG=00030932
  console: PRID=00006302
  console: Init SDRAM Done!
  console: Sizing caches...
  console: Init caches...
  console: godson2 caches found
  console: Init caches done, cfg = 00030932
  console: Copy PMON to execute location...
  console: copy text section done.
  console: Copy PMON to execute location done.
  Uncompressing Bios........................OK,Booting Bios
  PASS (0.25 s)
   (2/3) tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_pmon_framebuffer_console:
  [...]
  Uncompressing Bios........................OK,Booting Bios
  console: FREQ
  console: FREI
  console: DONE
  console: TTYI
  console: TTYD
  console: ENVI
  console: MAPV
  console: Mfg  0, Id 60
  console: STDV
  console: SBDD
  console: PPCIH
  console: PCIS
  console: PCIR
  console: PCIW
  console: NETI
  console: RTCL
  console: PCID
  console: VGAI
  console: Default MODE_ID 2
  console: starting radeon init...
  console: radeon init done
  console: FRBI
  console: cfb_console init,fb=b4000000
  console: Video: Drawing the logo ...
  console: CONSOLE_SIZE 450560HSTI
  PASS (4.10 s)
   (3/3) tests/acceptance/machine_mips_fuloong2e.py:MipsFuloong2e.test_linux_kernel_isa_serial:
  console: Linux version 2.6.27.7lemote (root@debian) (gcc version 4.1.3 20080623 (prerelease) (Debian 4.1.2-23)) #6 Fri Dec 12 00:11:25 CST 2008
  console: busclock=33000000, cpuclock=-2145008360,memsize=256,highmemsize=0
  console: console [early0] enabled
  console: CPU revision is: 00006302 (ICT Loongson-2)
  PASS (0.19 s)
  RESULTS    : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 5.10 s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Cc: Daniel P. Berrange <berrange@redhat.com>
Cc: Alex Bennée <alex.bennee@linaro.org>
---
 tests/acceptance/machine_mips_fuloong2e.py | 59 ++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/tests/acceptance/machine_mips_fuloong2e.py b/tests/acceptance/machine_mips_fuloong2e.py
index 5fd3da09e7c..807e38da74a 100644
--- a/tests/acceptance/machine_mips_fuloong2e.py
+++ b/tests/acceptance/machine_mips_fuloong2e.py
@@ -8,15 +8,74 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import time
 
 from avocado import skipUnless
 from avocado_qemu import Test
 from avocado_qemu import wait_for_console_pattern
 
+from tesseract_utils import tesseract_available, tesseract_ocr
+
 class MipsFuloong2e(Test):
 
     timeout = 60
 
+    @skipUnless(os.getenv('PMON_BIN_PATH'), 'PMON_BIN_PATH not available')
+    def test_pmon_serial_console(self):
+        """
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:fulong2e
+        :avocado: tags=endian:little
+        :avocado: tags=device:bonito64
+        :avocado: tags=device:via686b
+        """
+        pmon_hash = 'c812e1695d7b2320036f3ef494976969' # v1.1.2
+        pmon_path = self.fetch_asset('file://' + os.getenv('PMON_BIN_PATH'),
+                                     asset_hash=pmon_hash, algorithm='md5')
+
+        self.vm.set_console()
+        self.vm.add_args('-bios', pmon_path)
+        self.vm.launch()
+        wait_for_console_pattern(self, 'PMON2000 MIPS Initializing. Standby...')
+        wait_for_console_pattern(self, 'Booting Bios')
+
+    # Tesseract 4 adds a new OCR engine based on LSTM neural networks. The
+    # new version is faster and more accurate than version 3. The drawback is
+    # that it is still alpha-level software.
+    @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available')
+    @skipUnless(os.getenv('PMON_BIN_PATH'), 'PMON_BIN_PATH not available')
+    def test_pmon_framebuffer_console(self):
+        """
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:fulong2e
+        :avocado: tags=endian:little
+        :avocado: tags=device:bonito64
+        :avocado: tags=device:ati-vga
+        """
+        screenshot_path = os.path.join(self.workdir, 'dump.ppm')
+
+        pmon_hash = 'c812e1695d7b2320036f3ef494976969' # v1.1.2
+        pmon_path = self.fetch_asset('file://' + os.getenv('PMON_BIN_PATH'),
+                                     asset_hash=pmon_hash, algorithm='md5')
+
+        self.vm.set_console()
+        self.vm.add_args('-bios', pmon_path,
+                         '-vga', 'std')
+        self.vm.launch()
+
+        wait_for_console_pattern(self, 'Video: Drawing the logo ...')
+        self.log.info('VM launched, waiting for logo on display')
+        time.sleep(2)
+        wait_for_console_pattern(self, 'CONSOLE_SIZE')
+        self.vm.command('human-monitor-command', command_line='stop')
+        self.vm.command('human-monitor-command',
+                        command_line='screendump %s' % screenshot_path)
+
+        lines = tesseract_ocr(screenshot_path, tesseract_version=4,
+                              tesseract_args='--dpi 128')
+        pmon_version = 'PNON2000 for Loongson, Version 1.1.2' # PNON is enough
+        self.assertIn(pmon_version, lines)
+
     @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
     def test_linux_kernel_isa_serial(self):
         """
-- 
2.26.2

Re: [PATCH 2/6] tests/acceptance: Use .ppm extention for Portable PixMap files

[Thomas Huth]

On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
> The HMP 'screendump' command generates Portable PixMap files.
> Make it obvious by using the .ppm file extention.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/machine_m68k_nextcube.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/6] tests/acceptance: Extract tesseract_available() helper in new namespace

[Thomas Huth]

On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
> We are going to reuse tesseract_available(). Extract it to
> a new 'tesseract_utils' namespace.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/machine_m68k_nextcube.py | 25 +++-----------------
>  tests/acceptance/tesseract_utils.py       | 28 +++++++++++++++++++++++
>  2 files changed, 31 insertions(+), 22 deletions(-)
>  create mode 100644 tests/acceptance/tesseract_utils.py

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Thomas Huth]

On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
> We are going to reuse the tesseract OCR code.
> Create a new tesseract_ocr() helper and use it.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
>  tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
>  2 files changed, 23 insertions(+), 16 deletions(-)
> 
> diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
> index 3c7400c43e4..09e2745cc52 100644
> --- a/tests/acceptance/machine_m68k_nextcube.py
> +++ b/tests/acceptance/machine_m68k_nextcube.py
> @@ -7,13 +7,11 @@
>  
>  import os
>  import time
> -import logging
>  
>  from avocado_qemu import Test
>  from avocado import skipUnless
> -from avocado.utils import process
>  
> -from tesseract_utils import tesseract_available
> +from tesseract_utils import tesseract_available, tesseract_ocr
>  
>  PIL_AVAILABLE = True
>  try:
> @@ -61,12 +59,8 @@ def test_bootrom_framebuffer_size(self):
>      def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>          screenshot_path = os.path.join(self.workdir, "dump.ppm")
>          self.check_bootrom_framebuffer(screenshot_path)
> -
> -        console_logger = logging.getLogger('console')
> -        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
> -        for line in text.split('\n'):
> -            if len(line):
> -                console_logger.debug(line)
> +        lines = tesseract_ocr(screenshot_path, tesseract_version=3)
> +        text = '\n'.join(lines)
>          self.assertIn('Backplane', text)
>          self.assertIn('Ethernet address', text)
>  
> @@ -77,13 +71,8 @@ def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>      def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
>          screenshot_path = os.path.join(self.workdir, "dump.ppm")
>          self.check_bootrom_framebuffer(screenshot_path)
> -
> -        console_logger = logging.getLogger('console')
> -        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
> -        text = proc.stdout_text
> -        for line in text.split('\n'):
> -            if len(line):
> -                console_logger.debug(line)
> +        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
> +        text = '\n'.join(lines)
>          self.assertIn('Testing the FPU, SCC', text)
>          self.assertIn('System test failed. Error code', text)
>          self.assertIn('Boot command', text)
> diff --git a/tests/acceptance/tesseract_utils.py b/tests/acceptance/tesseract_utils.py
> index acd6e8c2faa..72cd9ab7989 100644
> --- a/tests/acceptance/tesseract_utils.py
> +++ b/tests/acceptance/tesseract_utils.py
> @@ -6,7 +6,9 @@
>  # later. See the COPYING file in the top-level directory.
>  
>  import re
> +import logging
>  
> +from avocado.utils import process
>  from avocado.utils.path import find_command, CmdNotFoundError
>  
>  def tesseract_available(expected_version):
> @@ -26,3 +28,19 @@ def tesseract_available(expected_version):
>          return False
>      # now this is guaranteed to be a digit
>      return int(match.groups()[0]) == expected_version
> +
> +
> +def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
> +    console_logger = logging.getLogger('tesseract')
> +    console_logger.debug(image_path)
> +    if tesseract_version == 4:
> +        tesseract_args += ' --oem 1'
> +    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
> +                                                       image_path))
> +    lines = []
> +    for line in proc.stdout_text.split('\n'):
> +        sline = line.strip()
> +        if len(sline):
> +            console_logger.debug(sline)
> +            lines += [sline]
> +    return lines

Would it make sense to completely hide the tesseract version handling in
this new tesseract_utils.py file now, so that the tests themselves do not
have to worry about this anymore (i.e. would it be possible to merge
test_bootrom_framebuffer_ocr_with_tesseract_v3 and
test_bootrom_framebuffer_ocr_with_tesseract_v4 into one single test that way?)

 Thomas

Re: [PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Thomas Huth]

On 24/10/2020 08.35, Thomas Huth wrote:
> On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
>> We are going to reuse the tesseract OCR code.
>> Create a new tesseract_ocr() helper and use it.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
>>  tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
>>  2 files changed, 23 insertions(+), 16 deletions(-)
>>
>> diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
>> index 3c7400c43e4..09e2745cc52 100644
>> --- a/tests/acceptance/machine_m68k_nextcube.py
>> +++ b/tests/acceptance/machine_m68k_nextcube.py
>> @@ -7,13 +7,11 @@
>>  
>>  import os
>>  import time
>> -import logging
>>  
>>  from avocado_qemu import Test
>>  from avocado import skipUnless
>> -from avocado.utils import process
>>  
>> -from tesseract_utils import tesseract_available
>> +from tesseract_utils import tesseract_available, tesseract_ocr
>>  
>>  PIL_AVAILABLE = True
>>  try:
>> @@ -61,12 +59,8 @@ def test_bootrom_framebuffer_size(self):
>>      def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>>          screenshot_path = os.path.join(self.workdir, "dump.ppm")
>>          self.check_bootrom_framebuffer(screenshot_path)
>> -
>> -        console_logger = logging.getLogger('console')
>> -        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
>> -        for line in text.split('\n'):
>> -            if len(line):
>> -                console_logger.debug(line)
>> +        lines = tesseract_ocr(screenshot_path, tesseract_version=3)
>> +        text = '\n'.join(lines)
>>          self.assertIn('Backplane', text)
>>          self.assertIn('Ethernet address', text)
>>  
>> @@ -77,13 +71,8 @@ def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>>      def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
>>          screenshot_path = os.path.join(self.workdir, "dump.ppm")
>>          self.check_bootrom_framebuffer(screenshot_path)
>> -
>> -        console_logger = logging.getLogger('console')
>> -        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
>> -        text = proc.stdout_text
>> -        for line in text.split('\n'):
>> -            if len(line):
>> -                console_logger.debug(line)
>> +        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
>> +        text = '\n'.join(lines)
>>          self.assertIn('Testing the FPU, SCC', text)
>>          self.assertIn('System test failed. Error code', text)
>>          self.assertIn('Boot command', text)
>> diff --git a/tests/acceptance/tesseract_utils.py b/tests/acceptance/tesseract_utils.py
>> index acd6e8c2faa..72cd9ab7989 100644
>> --- a/tests/acceptance/tesseract_utils.py
>> +++ b/tests/acceptance/tesseract_utils.py
>> @@ -6,7 +6,9 @@
>>  # later. See the COPYING file in the top-level directory.
>>  
>>  import re
>> +import logging
>>  
>> +from avocado.utils import process
>>  from avocado.utils.path import find_command, CmdNotFoundError
>>  
>>  def tesseract_available(expected_version):
>> @@ -26,3 +28,19 @@ def tesseract_available(expected_version):
>>          return False
>>      # now this is guaranteed to be a digit
>>      return int(match.groups()[0]) == expected_version
>> +
>> +
>> +def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
>> +    console_logger = logging.getLogger('tesseract')
>> +    console_logger.debug(image_path)
>> +    if tesseract_version == 4:
>> +        tesseract_args += ' --oem 1'
>> +    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
>> +                                                       image_path))
>> +    lines = []
>> +    for line in proc.stdout_text.split('\n'):
>> +        sline = line.strip()
>> +        if len(sline):
>> +            console_logger.debug(sline)
>> +            lines += [sline]
>> +    return lines
> 
> Would it make sense to completely hide the tesseract version handling in
> this new tesseract_utils.py file now, so that the tests themselves do not
> have to worry about this anymore (i.e. would it be possible to merge
> test_bootrom_framebuffer_ocr_with_tesseract_v3 and
> test_bootrom_framebuffer_ocr_with_tesseract_v4 into one single test that way?)

If I've got that right, there is also now a proper release 4 of Tesseract,
so maybe we can simply scratch the testing with version 3 now?

 Thomas

Re: [PATCH 1/6] tests/acceptance: Remove unused import

[Thomas Huth]

On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/machine_m68k_nextcube.py | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
> index 32cf571f948..9d289f2fab7 100644
> --- a/tests/acceptance/machine_m68k_nextcube.py
> +++ b/tests/acceptance/machine_m68k_nextcube.py
> @@ -9,7 +9,6 @@
>  import re
>  import time
>  import logging
> -import distutils.spawn
>  
>  from avocado_qemu import Test
>  from avocado import skipUnless
> 

Tested-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Philippe Mathieu-Daudé]

On 10/24/20 8:40 AM, Thomas Huth wrote:
> On 24/10/2020 08.35, Thomas Huth wrote:
>> On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
>>> We are going to reuse the tesseract OCR code.
>>> Create a new tesseract_ocr() helper and use it.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>   tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
>>>   tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
>>>   2 files changed, 23 insertions(+), 16 deletions(-)
>>>
>>> diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
>>> index 3c7400c43e4..09e2745cc52 100644
>>> --- a/tests/acceptance/machine_m68k_nextcube.py
>>> +++ b/tests/acceptance/machine_m68k_nextcube.py
>>> @@ -7,13 +7,11 @@
>>>   
>>>   import os
>>>   import time
>>> -import logging
>>>   
>>>   from avocado_qemu import Test
>>>   from avocado import skipUnless
>>> -from avocado.utils import process
>>>   
>>> -from tesseract_utils import tesseract_available
>>> +from tesseract_utils import tesseract_available, tesseract_ocr
>>>   
>>>   PIL_AVAILABLE = True
>>>   try:
>>> @@ -61,12 +59,8 @@ def test_bootrom_framebuffer_size(self):
>>>       def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>>>           screenshot_path = os.path.join(self.workdir, "dump.ppm")
>>>           self.check_bootrom_framebuffer(screenshot_path)
>>> -
>>> -        console_logger = logging.getLogger('console')
>>> -        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
>>> -        for line in text.split('\n'):
>>> -            if len(line):
>>> -                console_logger.debug(line)
>>> +        lines = tesseract_ocr(screenshot_path, tesseract_version=3)
>>> +        text = '\n'.join(lines)
>>>           self.assertIn('Backplane', text)
>>>           self.assertIn('Ethernet address', text)
>>>   
>>> @@ -77,13 +71,8 @@ def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
>>>       def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
>>>           screenshot_path = os.path.join(self.workdir, "dump.ppm")
>>>           self.check_bootrom_framebuffer(screenshot_path)
>>> -
>>> -        console_logger = logging.getLogger('console')
>>> -        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
>>> -        text = proc.stdout_text
>>> -        for line in text.split('\n'):
>>> -            if len(line):
>>> -                console_logger.debug(line)
>>> +        lines = tesseract_ocr(screenshot_path, tesseract_version=4)
>>> +        text = '\n'.join(lines)
>>>           self.assertIn('Testing the FPU, SCC', text)
>>>           self.assertIn('System test failed. Error code', text)
>>>           self.assertIn('Boot command', text)
>>> diff --git a/tests/acceptance/tesseract_utils.py b/tests/acceptance/tesseract_utils.py
>>> index acd6e8c2faa..72cd9ab7989 100644
>>> --- a/tests/acceptance/tesseract_utils.py
>>> +++ b/tests/acceptance/tesseract_utils.py
>>> @@ -6,7 +6,9 @@
>>>   # later. See the COPYING file in the top-level directory.
>>>   
>>>   import re
>>> +import logging
>>>   
>>> +from avocado.utils import process
>>>   from avocado.utils.path import find_command, CmdNotFoundError
>>>   
>>>   def tesseract_available(expected_version):
>>> @@ -26,3 +28,19 @@ def tesseract_available(expected_version):
>>>           return False
>>>       # now this is guaranteed to be a digit
>>>       return int(match.groups()[0]) == expected_version
>>> +
>>> +
>>> +def tesseract_ocr(image_path, tesseract_args='', tesseract_version=3):
>>> +    console_logger = logging.getLogger('tesseract')
>>> +    console_logger.debug(image_path)
>>> +    if tesseract_version == 4:
>>> +        tesseract_args += ' --oem 1'
>>> +    proc = process.run("tesseract {} {} stdout".format(tesseract_args,
>>> +                                                       image_path))
>>> +    lines = []
>>> +    for line in proc.stdout_text.split('\n'):
>>> +        sline = line.strip()
>>> +        if len(sline):
>>> +            console_logger.debug(sline)
>>> +            lines += [sline]
>>> +    return lines
>>
>> Would it make sense to completely hide the tesseract version handling in
>> this new tesseract_utils.py file now, so that the tests themselves do not
>> have to worry about this anymore

Yes, good idea.

> (i.e. would it be possible to merge
>> test_bootrom_framebuffer_ocr_with_tesseract_v3 and
>> test_bootrom_framebuffer_ocr_with_tesseract_v4 into one single test that way?)
> 
> If I've got that right, there is also now a proper release 4 of Tesseract,
> so maybe we can simply scratch the testing with version 3 now?

Good to know, I'll have a look. Thanks!

> 
>   Thomas
> 

Re: [PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Philippe Mathieu-Daudé]

On 10/24/20 9:37 AM, Philippe Mathieu-Daudé wrote:
> On 10/24/20 8:40 AM, Thomas Huth wrote:
>> On 24/10/2020 08.35, Thomas Huth wrote:
>>> On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
>>>> We are going to reuse the tesseract OCR code.
>>>> Create a new tesseract_ocr() helper and use it.
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>   tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
>>>>   tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
>>>>   2 files changed, 23 insertions(+), 16 deletions(-)
...

>>>
>>> Would it make sense to completely hide the tesseract version handling in
>>> this new tesseract_utils.py file now, so that the tests themselves do 
>>> not
>>> have to worry about this anymore

The problem is the recognized strings differ between versions,
see in tests/acceptance/machine_m68k_nextcube.py:

         lines = tesseract_ocr(screenshot_path, tesseract_version=3)
         text = '\n'.join(lines)
         self.assertIn('Backplane', text)
         self.assertIn('Ethernet address', text)

and:

         lines = tesseract_ocr(screenshot_path, tesseract_version=4)
         text = '\n'.join(lines)
         self.assertIn('Testing the FPU, SCC', text)
         self.assertIn('System test failed. Error code', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)

> 
> Yes, good idea.
> 
>> (i.e. would it be possible to merge
>>> test_bootrom_framebuffer_ocr_with_tesseract_v3 and
>>> test_bootrom_framebuffer_ocr_with_tesseract_v4 into one single test 
>>> that way?)
>>
>> If I've got that right, there is also now a proper release 4 of 
>> Tesseract,
>> so maybe we can simply scratch the testing with version 3 now?
> 
> Good to know, I'll have a look. Thanks!
> 
>>
>>   Thomas
>>
> 

Re: [PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[Mark Cave-Ayland]

On 21/10/2020 11:50, Philippe Mathieu-Daudé wrote:

> This series add tests for the MIPS Fuloong 2E machine.
> 
> I have been using these tests for over 1 year now.
> After recent discussions with Huacai and Mark regarding
> how to contribute tests for machines using private blob,
> I think it is time to publish this series, to see how
> the final part will be discussed, as it tests a blob
> which Lemote company doesn't publish the sources. The
> binary is know to work and has been used for various
> years on real hardware. Its MD5 is also know (I am not
> using SHA-1 because it has been published with MD5).
> 
> The idea is to be able to share tests without infringing
> licenses issues.
> 
> Please comment :)
> 
> Phil.
> 
> Philippe Mathieu-Daudé (6):
>    tests/acceptance: Remove unused import
>    tests/acceptance: Use .ppm extention for Portable PixMap files
>    tests/acceptance: Extract tesseract_available() helper in new
>      namespace
>    tests/acceptance: Introduce tesseract_ocr() helper
>    tests/acceptance: Test Linux on the Fuloong 2E machine
>    tests/acceptance: Test PMON on the Fuloong 2E machine
> 
>   MAINTAINERS                                |   1 +
>   tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
>   tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
>   tests/acceptance/tesseract_utils.py        |  46 ++++++++++
>   4 files changed, 158 insertions(+), 41 deletions(-)
>   create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
>   create mode 100644 tests/acceptance/tesseract_utils.py

In general this looks good: certainly I don't see any reason why the tesseract_ocr 
helper changes shouldn't be merged, since I expect that this is something that will 
be used more in future.

Looking at the fuloong test it seems that it stops fairly early in the kernel boot: 
one of the problems that Zoltan's VIA southbridge patches were trying to solve 
earlier in the year was because of problems with PCI IRQ routing, so it would be nice 
to include a small OS image that can be used to boot to userspace to ensure that such 
problems can be detected in future.


ATB,

Mark.

Re: [PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[Philippe Mathieu-Daudé]

On 10/25/20 12:18 PM, Mark Cave-Ayland wrote:
> On 21/10/2020 11:50, Philippe Mathieu-Daudé wrote:
> 
>> This series add tests for the MIPS Fuloong 2E machine.
>>
>> I have been using these tests for over 1 year now.
>> After recent discussions with Huacai and Mark regarding
>> how to contribute tests for machines using private blob,
>> I think it is time to publish this series, to see how
>> the final part will be discussed, as it tests a blob
>> which Lemote company doesn't publish the sources. The
>> binary is know to work and has been used for various
>> years on real hardware. Its MD5 is also know (I am not
>> using SHA-1 because it has been published with MD5).
>>
>> The idea is to be able to share tests without infringing
>> licenses issues.
>>
>> Please comment :)
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (6):
>>    tests/acceptance: Remove unused import
>>    tests/acceptance: Use .ppm extention for Portable PixMap files
>>    tests/acceptance: Extract tesseract_available() helper in new
>>      namespace
>>    tests/acceptance: Introduce tesseract_ocr() helper
>>    tests/acceptance: Test Linux on the Fuloong 2E machine
>>    tests/acceptance: Test PMON on the Fuloong 2E machine
>>
>>   MAINTAINERS                                |   1 +
>>   tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
>>   tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
>>   tests/acceptance/tesseract_utils.py        |  46 ++++++++++
>>   4 files changed, 158 insertions(+), 41 deletions(-)
>>   create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
>>   create mode 100644 tests/acceptance/tesseract_utils.py
> 
> In general this looks good: certainly I don't see any reason why the 
> tesseract_ocr helper changes shouldn't be merged, since I expect that 
> this is something that will be used more in future.
> 
> Looking at the fuloong test it seems that it stops fairly early in the 
> kernel boot: one of the problems that Zoltan's VIA southbridge patches 
> were trying to solve earlier in the year was because of problems with 
> PCI IRQ routing, so it would be nice to include a small OS image that 
> can be used to boot to userspace to ensure that such problems can be 
> detected in future.

The Linux tests (ab)use of the rescue image prepared for the Yeeloong
machine which we don't model (Loongson 2F CPU, 1GB of DRAM). The kernel
triggers DATA ABORT while accessing >256MB because we do not model
mapping more than 256MB :/ (The Fuloong comes with 512MB extensible to 
1GB). I have a branch adding that but had the idea nobody was interested
in the Fuloong 2E so I didn't insist (I have more interest in embedded
32-bit CPUs). I might try to post it after the release, as it makes this
test complete.

Thanks for giving it a try!

Phil.

Re: [PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[BALATON Zoltan via]

[-- Attachment #1: Type: text/plain, Size: 3559 bytes --]

On Sun, 25 Oct 2020, Philippe Mathieu-Daudé wrote:
> On 10/25/20 12:18 PM, Mark Cave-Ayland wrote:
>> On 21/10/2020 11:50, Philippe Mathieu-Daudé wrote:
>> 
>>> This series add tests for the MIPS Fuloong 2E machine.
>>> 
>>> I have been using these tests for over 1 year now.
>>> After recent discussions with Huacai and Mark regarding
>>> how to contribute tests for machines using private blob,
>>> I think it is time to publish this series, to see how
>>> the final part will be discussed, as it tests a blob
>>> which Lemote company doesn't publish the sources. The
>>> binary is know to work and has been used for various
>>> years on real hardware. Its MD5 is also know (I am not
>>> using SHA-1 because it has been published with MD5).
>>> 
>>> The idea is to be able to share tests without infringing
>>> licenses issues.
>>> 
>>> Please comment :)
>>> 
>>> Phil.
>>> 
>>> Philippe Mathieu-Daudé (6):
>>>    tests/acceptance: Remove unused import
>>>    tests/acceptance: Use .ppm extention for Portable PixMap files
>>>    tests/acceptance: Extract tesseract_available() helper in new
>>>      namespace
>>>    tests/acceptance: Introduce tesseract_ocr() helper
>>>    tests/acceptance: Test Linux on the Fuloong 2E machine
>>>    tests/acceptance: Test PMON on the Fuloong 2E machine
>>> 
>>>   MAINTAINERS                                |   1 +
>>>   tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
>>>   tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
>>>   tests/acceptance/tesseract_utils.py        |  46 ++++++++++
>>>   4 files changed, 158 insertions(+), 41 deletions(-)
>>>   create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
>>>   create mode 100644 tests/acceptance/tesseract_utils.py
>> 
>> In general this looks good: certainly I don't see any reason why the 
>> tesseract_ocr helper changes shouldn't be merged, since I expect that this 
>> is something that will be used more in future.
>> 
>> Looking at the fuloong test it seems that it stops fairly early in the 
>> kernel boot: one of the problems that Zoltan's VIA southbridge patches were 
>> trying to solve earlier in the year was because of problems with PCI IRQ 
>> routing, so it would be nice to include a small OS image that can be used 
>> to boot to userspace to ensure that such problems can be detected in 
>> future.
>
> The Linux tests (ab)use of the rescue image prepared for the Yeeloong
> machine which we don't model (Loongson 2F CPU, 1GB of DRAM). The kernel
> triggers DATA ABORT while accessing >256MB because we do not model
> mapping more than 256MB :/ (The Fuloong comes with 512MB extensible to 1GB). 
> I have a branch adding that but had the idea nobody was interested
> in the Fuloong 2E so I didn't insist (I have more interest in embedded
> 32-bit CPUs). I might try to post it after the release, as it makes this
> test complete.

I've noticed that missing high RAM when cleaning up SPD EEPROMs and addded 
a FIXME comment but did't know MIPS enough to attemt to fix it. Anyway I'm 
also only interested in fuloong2e as a test case for the devices also used 
by Pegasos2 (that I'll need to update and clean up to submit eventually) 
but not beyond that.

I haven't seen these patches before, so I've spotted only now that your 
test case uses -vga std. Shouldn't it work with ati-vga,model=rv100 which 
I think is the default for this machine and that's what the pmon firmware 
expects and has a driver for?

Regards,
BALATON Zoltan

Re: [PATCH 4/6] tests/acceptance: Introduce tesseract_ocr() helper

[Thomas Huth]

On 24/10/2020 19.40, Philippe Mathieu-Daudé wrote:
> On 10/24/20 9:37 AM, Philippe Mathieu-Daudé wrote:
>> On 10/24/20 8:40 AM, Thomas Huth wrote:
>>> On 24/10/2020 08.35, Thomas Huth wrote:
>>>> On 21/10/2020 12.50, Philippe Mathieu-Daudé wrote:
>>>>> We are going to reuse the tesseract OCR code.
>>>>> Create a new tesseract_ocr() helper and use it.
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>> ---
>>>>>   tests/acceptance/machine_m68k_nextcube.py | 21 +++++----------------
>>>>>   tests/acceptance/tesseract_utils.py       | 18 ++++++++++++++++++
>>>>>   2 files changed, 23 insertions(+), 16 deletions(-)
> ...
> 
>>>>
>>>> Would it make sense to completely hide the tesseract version handling in
>>>> this new tesseract_utils.py file now, so that the tests themselves do not
>>>> have to worry about this anymore
> 
> The problem is the recognized strings differ between versions,
> see in tests/acceptance/machine_m68k_nextcube.py:
> 
>         lines = tesseract_ocr(screenshot_path, tesseract_version=3)
>         text = '\n'.join(lines)
>         self.assertIn('Backplane', text)
>         self.assertIn('Ethernet address', text)
> 
> and:
> 
>         lines = tesseract_ocr(screenshot_path, tesseract_version=4)
>         text = '\n'.join(lines)
>         self.assertIn('Testing the FPU, SCC', text)
>         self.assertIn('System test failed. Error code', text)
>         self.assertIn('Boot command', text)
>         self.assertIn('Next>', text)

Ah, right, I forgot about that ... well, one more reason to completely
switch to tesseract v4 now ;-)

 Thomas

Re: [PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[Philippe Mathieu-Daudé]

On 10/21/20 12:50 PM, Philippe Mathieu-Daudé wrote:
> This series add tests for the MIPS Fuloong 2E machine.
> 
> I have been using these tests for over 1 year now.
> After recent discussions with Huacai and Mark regarding
> how to contribute tests for machines using private blob,
> I think it is time to publish this series, to see how
> the final part will be discussed, as it tests a blob
> which Lemote company doesn't publish the sources. The
> binary is know to work and has been used for various
> years on real hardware. Its MD5 is also know (I am not
> using SHA-1 because it has been published with MD5).
> 
> The idea is to be able to share tests without infringing
> licenses issues.
> 
> Please comment :)
> 
> Phil.
> 
> Philippe Mathieu-Daudé (6):
>    tests/acceptance: Remove unused import
>    tests/acceptance: Use .ppm extention for Portable PixMap files
>    tests/acceptance: Extract tesseract_available() helper in new
>      namespace
>    tests/acceptance: Introduce tesseract_ocr() helper
>    tests/acceptance: Test Linux on the Fuloong 2E machine
>    tests/acceptance: Test PMON on the Fuloong 2E machine
> 
>   MAINTAINERS                                |   1 +
>   tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
>   tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
>   tests/acceptance/tesseract_utils.py        |  46 ++++++++++
>   4 files changed, 158 insertions(+), 41 deletions(-)
>   create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
>   create mode 100644 tests/acceptance/tesseract_utils.py

Thanks, patches #1 and #2 applied to my acceptance-testing tree.

Re: [PATCH 0/6] tests/acceptance: Test the Fuloong 2E machine

[Philippe Mathieu-Daudé]

On 10/26/20 9:50 AM, Philippe Mathieu-Daudé wrote:
> On 10/21/20 12:50 PM, Philippe Mathieu-Daudé wrote:
>> Philippe Mathieu-Daudé (6):
>>    tests/acceptance: Remove unused import
>>    tests/acceptance: Use .ppm extention for Portable PixMap files
>>    tests/acceptance: Extract tesseract_available() helper in new
>>      namespace
>>    tests/acceptance: Introduce tesseract_ocr() helper
>>    tests/acceptance: Test Linux on the Fuloong 2E machine
>>    tests/acceptance: Test PMON on the Fuloong 2E machine
>>
>>   MAINTAINERS                                |   1 +
>>   tests/acceptance/machine_m68k_nextcube.py  |  51 ++---------
>>   tests/acceptance/machine_mips_fuloong2e.py | 101 +++++++++++++++++++++
>>   tests/acceptance/tesseract_utils.py        |  46 ++++++++++
>>   4 files changed, 158 insertions(+), 41 deletions(-)
>>   create mode 100644 tests/acceptance/machine_mips_fuloong2e.py
>>   create mode 100644 tests/acceptance/tesseract_utils.py
> 
> Thanks, patches #1 and #2 applied to my acceptance-testing tree.

Dequeuing as Thomas already merged them.