All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/7] runfvp: propagate the exit code correctly
@ 2022-03-31 18:31 Ross Burton
  2022-03-31 18:31 ` [PATCH 2/7] runfvp: check for telnet Ross Burton
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

runfvp could encounter an error but the exit code remained as 0.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index c20455fd..5cab0949 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -221,6 +221,9 @@ async def start_fvp(cli, console_cb):
 
     if await fvp_process.wait() != 0:
         logger.info(f"{cli[0]} quit with code {fvp_process.returncode}")
+        return fvp_process.returncode
+    else:
+        return 0
 
 def runfvp(cli_args):
     args, fvp_args = parse_args(cli_args)
@@ -234,7 +237,7 @@ def runfvp(cli_args):
         expected_terminal = config["console"]
         if not expected_terminal:
             logger.error("--console used but FVP_CONSOLE not set in machine configuration")
-            sys.exit(1)
+            return 1
     else:
         expected_terminal = None
 
@@ -252,9 +255,10 @@ def runfvp(cli_args):
         # When we can assume Py3.7+, this can simply be asyncio.run()
         loop = asyncio.get_event_loop()
         console_cb = expected_terminal and console_started or None
-        loop.run_until_complete(asyncio.gather(start_fvp(cli, console_cb=console_cb)))
+        return loop.run_until_complete(start_fvp(cli, console_cb=console_cb))
     except asyncio.CancelledError:
-        pass
+        # This means telnet exited, which isn't an error
+        return 0
 
 if __name__ == "__main__":
     try:
@@ -264,6 +268,6 @@ if __name__ == "__main__":
         if sys.stdin.isatty():
             signal.signal(signal.SIGTTOU, signal.SIG_IGN)
             os.tcsetpgrp(sys.stdin.fileno(), os.getpgrp())
-        runfvp(sys.argv[1:])
+        sys.exit(runfvp(sys.argv[1:]))
     except KeyboardInterrupt:
         pass
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 2/7] runfvp: check for telnet
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-03-31 18:31 ` [PATCH 3/7] runfvp: strip all suffixes from the image when calculating .fvpconf name Ross Burton
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Check for telnet on startup to avoid mysterious failures later when
telnet isn't available.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index 5cab0949..0ebf873e 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -5,6 +5,7 @@ import collections
 import json
 import os
 import re
+import shutil
 import signal
 import sys
 import subprocess
@@ -42,7 +43,7 @@ class Terminals:
         return config.get("RunFVP", "Terminal", fallback=None)
 
     def preferred_terminal(self) -> str:
-        import shlex, shutil
+        import shlex
 
         preferred = self.configured_terminal()
         if preferred:
@@ -233,6 +234,11 @@ def runfvp(cli_args):
     cli.extend(fvp_args)
     logger.debug(f"Constructed FVP call: {cli}")
 
+    # Check that telnet is present
+    if not bool(shutil.which("telnet")):
+        logger.error("Cannot find telnet, this is needed to connect to the FVP.")
+        return 1
+
     if args.console:
         expected_terminal = config["console"]
         if not expected_terminal:
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 3/7] runfvp: strip all suffixes from the image when calculating .fvpconf name
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
  2022-03-31 18:31 ` [PATCH 2/7] runfvp: check for telnet Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-03-31 18:31 ` [PATCH 4/7] arm/oeqa: add basic runfvp test cases Ross Burton
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Until testimage tells the controller the basename of the image, make sure
to strip all suffixes from the image name to get the base name, not just
one.  Machines such as corstone500 have images called .wic.nopt, so just
stripping one isn't sufficient.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta-arm/lib/oeqa/controllers/fvp.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py
index 7136a003..2913f782 100644
--- a/meta-arm/lib/oeqa/controllers/fvp.py
+++ b/meta-arm/lib/oeqa/controllers/fvp.py
@@ -16,7 +16,9 @@ class OEFVPTarget(oeqa.core.target.ssh.OESSHTarget):
                  **kwargs):
         super().__init__(logger, target_ip, server_ip, timeout, user, port)
         image_dir = pathlib.Path(dir_image)
-        basename = pathlib.Path(rootfs).stem
+        # rootfs may have multiple extensions so we need to strip *all* suffixes
+        basename = pathlib.Path(rootfs)
+        basename = basename.name.replace("".join(basename.suffixes), "")
         self.fvpconf = image_dir / (basename + ".fvpconf")
 
         if not self.fvpconf.exists():
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 4/7] arm/oeqa: add basic runfvp test cases
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
  2022-03-31 18:31 ` [PATCH 2/7] runfvp: check for telnet Ross Burton
  2022-03-31 18:31 ` [PATCH 3/7] runfvp: strip all suffixes from the image when calculating .fvpconf name Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-03-31 18:31 ` [PATCH 5/7] CI: clean up assignments in base.yml Ross Burton
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Add rudimentary but functional test cases for runfvp.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta-arm/lib/oeqa/selftest/cases/runfvp.py    | 51 +++++++++++++++++++
 .../oeqa/selftest/cases/tests/auto-basic.json |  3 ++
 .../oeqa/selftest/cases/tests/auto-basic.sh   | 11 ++++
 .../selftest/cases/tests/auto-parameters.json |  7 +++
 .../lib/oeqa/selftest/cases/tests/mock-fvp.py | 22 ++++++++
 .../selftest/cases/tests/test-parameter.json  |  6 +++
 .../selftest/cases/tests/test-parameters.py   |  1 +
 7 files changed, 101 insertions(+)
 create mode 100644 meta-arm/lib/oeqa/selftest/cases/runfvp.py
 create mode 100644 meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.json
 create mode 100755 meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.sh
 create mode 100644 meta-arm/lib/oeqa/selftest/cases/tests/auto-parameters.json
 create mode 100755 meta-arm/lib/oeqa/selftest/cases/tests/mock-fvp.py
 create mode 100644 meta-arm/lib/oeqa/selftest/cases/tests/test-parameter.json
 create mode 120000 meta-arm/lib/oeqa/selftest/cases/tests/test-parameters.py

diff --git a/meta-arm/lib/oeqa/selftest/cases/runfvp.py b/meta-arm/lib/oeqa/selftest/cases/runfvp.py
new file mode 100644
index 00000000..5b06ca8e
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/runfvp.py
@@ -0,0 +1,51 @@
+import os
+import pathlib
+import subprocess
+
+from oeqa.selftest.case import OESelftestTestCase
+
+runfvp = pathlib.Path(__file__).parents[5] / "scripts" / "runfvp"
+testdir = pathlib.Path(__file__).parent / "tests"
+
+class RunFVPTests(OESelftestTestCase):
+    def setUpLocal(self):
+        self.assertTrue(runfvp.exists())
+
+    def run_fvp(self, *args, should_succeed=True):
+        """
+        Call runfvp passing any arguments. If check is True verify return stdout
+        on exit code 0 or fail the test, otherwise return the CompletedProcess
+        instance.
+        """
+        # Put the test directory in PATH so that any mock FVPs are found first
+        newenv = {"PATH": str(testdir) + ":" + os.environ["PATH"]}
+        cli = [runfvp,] + list(args)
+        print(f"Calling {cli}")
+        ret = subprocess.run(cli, env=newenv, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
+        if should_succeed:
+            self.assertEqual(ret.returncode, 0, f"runfvp exit {ret.returncode}, output: {ret.stdout}")
+            return ret.stdout
+        else:
+            self.assertNotEqual(ret.returncode, 0, f"runfvp exit {ret.returncode}, output: {ret.stdout}")
+            return ret.stdout
+
+    def test_help(self):
+        output = self.run_fvp("--help")
+        self.assertIn("Run images in a FVP", output)
+
+    def test_bad_options(self):
+        self.run_fvp("--this-is-an-invalid-option", should_succeed=False)
+
+    def test_run_auto_tests(self):
+        newenv = {"PATH": str(testdir) + ":" + os.environ["PATH"]}
+
+        cases = list(testdir.glob("auto-*.json"))
+        if not cases:
+            self.fail("No tests found")
+        for case in cases:
+            with self.subTest(case=case.stem):
+                self.run_fvp(case)
+
+    def test_fvp_options(self):
+        # test-parameter sets one argument, add another manually
+        self.run_fvp(testdir / "test-parameter.json", "--", "--parameter", "board.dog=woof")
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.json b/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.json
new file mode 100644
index 00000000..476eb572
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.json
@@ -0,0 +1,3 @@
+{
+    "exe": "auto-basic.sh"
+}
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.sh b/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.sh
new file mode 100755
index 00000000..ea9abac1
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/auto-basic.sh
@@ -0,0 +1,11 @@
+#! /bin/sh
+
+set -e -u
+
+if [ $# = 0 ]; then
+    echo No arguments as expected
+    exit 0
+else
+    echo Unexpected arguments: $*
+    exit 1
+fi
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/auto-parameters.json b/meta-arm/lib/oeqa/selftest/cases/tests/auto-parameters.json
new file mode 100644
index 00000000..0c7d4ef9
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/auto-parameters.json
@@ -0,0 +1,7 @@
+{
+    "exe": "test-parameters.py",
+    "parameters": {
+        "board.cow": "moo",
+        "board.dog": "woof"
+    }
+}
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/mock-fvp.py b/meta-arm/lib/oeqa/selftest/cases/tests/mock-fvp.py
new file mode 100755
index 00000000..2213c9f0
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/mock-fvp.py
@@ -0,0 +1,22 @@
+#! /usr/bin/env python3
+
+import argparse
+import sys
+
+def do_test_parameters(args):
+    if not args.parameter or set(args.parameter) != set(("board.cow=moo", "board.dog=woof")):
+        print(f"Unexpected arguments: {args}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-C", "--parameter", action="append")
+    args = parser.parse_args()
+
+    function = "do_" + parser.prog.replace("-", "_").replace(".py", "")
+    if function in locals():
+        locals()[function](args)
+    else:
+        print(f"Unknown mock mode {parser.prog}")
+        sys.exit(1)
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/test-parameter.json b/meta-arm/lib/oeqa/selftest/cases/tests/test-parameter.json
new file mode 100644
index 00000000..9b565f27
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/test-parameter.json
@@ -0,0 +1,6 @@
+{
+    "exe": "test-parameters.py",
+    "parameters": {
+        "board.cow": "moo"
+    }
+}
diff --git a/meta-arm/lib/oeqa/selftest/cases/tests/test-parameters.py b/meta-arm/lib/oeqa/selftest/cases/tests/test-parameters.py
new file mode 120000
index 00000000..c734eeca
--- /dev/null
+++ b/meta-arm/lib/oeqa/selftest/cases/tests/test-parameters.py
@@ -0,0 +1 @@
+mock-fvp.py
\ No newline at end of file
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 5/7] CI: clean up assignments in base.yml
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
                   ` (2 preceding siblings ...)
  2022-03-31 18:31 ` [PATCH 4/7] arm/oeqa: add basic runfvp test cases Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-03-31 18:31 ` [PATCH 6/7] CI: add basic selftest job Ross Burton
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Split the base local.conf configuration into base (absolutely needed)
and setup (typical configuration). This is needed as oe-selftest needs a
minimal configuration to execute in, for example doesn't inherit
rm_work.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 ci/base.yml | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/ci/base.yml b/ci/base.yml
index 9a59de79..eb5637d4 100644
--- a/ci/base.yml
+++ b/ci/base.yml
@@ -28,15 +28,15 @@ local_conf_header:
   base: |
     BB_SERVER_TIMEOUT = "60"
     CONF_VERSION = "2"
-    PACKAGE_CLASSES = "package_ipk"
+    BB_NUMBER_THREADS = "16"
+    PARALLEL_MAKE = "-j16"
     LICENSE_FLAGS_ACCEPTED += "armcompiler Arm-FVP-EULA"
+  setup: |
+    PACKAGE_CLASSES = "package_ipk"
     PACKAGECONFIG:remove:pn-qemu-system-native = "gtk+ sdl"
     EXTRA_IMAGE_FEATURES:append = " debug-tweaks"
-    BB_NUMBER_THREADS = "16"
-    PARALLEL_MAKE = "-j16"
-    INHERIT += "rm_work"
     PACKAGECONFIG:append:pn-perf = " coresight"
-  noptest: |
+    INHERIT += "rm_work"
     DISTRO_FEATURES:remove = "ptest"
   kvm: |
     QEMU_USE_KVM = ""
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 6/7] CI: add basic selftest job
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
                   ` (3 preceding siblings ...)
  2022-03-31 18:31 ` [PATCH 5/7] CI: clean up assignments in base.yml Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-03-31 18:31 ` [PATCH 7/7] arm/oeqa: add no-op runtime test Ross Burton
  2022-04-01  0:53 ` [PATCH 1/7] runfvp: propagate the exit code correctly Jon Mason
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Currently this just executes the runfvp tests, but we can extend it over
time.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .gitlab-ci.yml  | 8 +++++++-
 ci/selftest.yml | 7 +++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
 create mode 100644 ci/selftest.yml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6d24e0d9..21fcd43f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -25,7 +25,7 @@ stages:
     - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
     # Must do this here, as it's the only way to make sure the toolchain is installed on the same builder
     - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
-    - sudo apt update && sudo apt install telnet -y
+    - sudo apt update && sudo apt install --yes telnet python3-subunit
 
 # Generalised fragment to do a Kas build
 .build:
@@ -201,6 +201,12 @@ tc1:
 toolchains:
   extends: .build
 
+selftest:
+  extends: .setup
+  script:
+    - KASFILES=./ci/qemuarm64.yml:./ci/selftest.yml
+    - kas shell --update --force-checkout $KASFILES -c 'oe-selftest --num-processes 1 --run-tests runfvp'
+
 # Validate layers are Yocto Project Compatible
 check-layers:
   extends: .setup
diff --git a/ci/selftest.yml b/ci/selftest.yml
new file mode 100644
index 00000000..9a587354
--- /dev/null
+++ b/ci/selftest.yml
@@ -0,0 +1,7 @@
+header:
+  version: 11
+
+local_conf_header:
+  setup: |
+    BB_LOGCONFIG = ""
+    SANITY_TESTED_DISTROS = ""
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 7/7] arm/oeqa: add no-op runtime test
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
                   ` (4 preceding siblings ...)
  2022-03-31 18:31 ` [PATCH 6/7] CI: add basic selftest job Ross Burton
@ 2022-03-31 18:31 ` Ross Burton
  2022-04-01  0:53 ` [PATCH 1/7] runfvp: propagate the exit code correctly Jon Mason
  6 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2022-03-31 18:31 UTC (permalink / raw)
  To: meta-arm

Machines which don't have working network stacks in a FVP can still run
a useful testimage, because it will validate that the machine will boot
to a login prompt on the console.

However, we can't set TEST_SUITES to "" because testimage assumes that
was a mistake, so add a no-op test case for use in these situations.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta-arm/lib/oeqa/runtime/cases/noop.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 meta-arm/lib/oeqa/runtime/cases/noop.py

diff --git a/meta-arm/lib/oeqa/runtime/cases/noop.py b/meta-arm/lib/oeqa/runtime/cases/noop.py
new file mode 100644
index 00000000..b5fba7c5
--- /dev/null
+++ b/meta-arm/lib/oeqa/runtime/cases/noop.py
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: MIT
+
+from oeqa.runtime.case import OERuntimeTestCase
+
+class NoopTest(OERuntimeTestCase):
+    """
+    This is a test case which does nothing.  Useful when you want to use
+    testimage to verify that an image boots, but you don't have networking so
+    none of the existing test cases are suitable.
+    """
+    def test_no_op(self):
+        return
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/7] runfvp: propagate the exit code correctly
  2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
                   ` (5 preceding siblings ...)
  2022-03-31 18:31 ` [PATCH 7/7] arm/oeqa: add no-op runtime test Ross Burton
@ 2022-04-01  0:53 ` Jon Mason
  6 siblings, 0 replies; 8+ messages in thread
From: Jon Mason @ 2022-04-01  0:53 UTC (permalink / raw)
  To: meta-arm, Ross Burton

On Thu, 31 Mar 2022 19:31:11 +0100, Ross Burton wrote:
> runfvp could encounter an error but the exit code remained as 0.

Applied, thanks!

[1/7] runfvp: propagate the exit code correctly
      commit: 1a7c48e5f6cfe916e6e7d1b1076f4f9aa3230a19
[2/7] runfvp: check for telnet
      commit: 58f3251598a618cc16fc186102a1783efed660ab
[3/7] runfvp: strip all suffixes from the image when calculating .fvpconf name
      commit: 3a0dbcd37fbfc74b528ab47c78d50080f301f61d
[4/7] arm/oeqa: add basic runfvp test cases
      commit: 2b5cc5da3ea1f8898cb29c60ee6b2fd67bc437a4
[5/7] CI: clean up assignments in base.yml
      commit: ba7803e5138ae0295362b16c93eab5e3ec0cb6bd
[6/7] CI: add basic selftest job
      commit: 5289c05289335d1786d0c70f12a335281f04fd86
[7/7] arm/oeqa: add no-op runtime test
      commit: e4cc4d01cac2d4c1e8b798de42a1644bc869078a

Best regards,
-- 
Jon Mason <jon.mason@arm.com>


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2022-04-01  0:53 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-31 18:31 [PATCH 1/7] runfvp: propagate the exit code correctly Ross Burton
2022-03-31 18:31 ` [PATCH 2/7] runfvp: check for telnet Ross Burton
2022-03-31 18:31 ` [PATCH 3/7] runfvp: strip all suffixes from the image when calculating .fvpconf name Ross Burton
2022-03-31 18:31 ` [PATCH 4/7] arm/oeqa: add basic runfvp test cases Ross Burton
2022-03-31 18:31 ` [PATCH 5/7] CI: clean up assignments in base.yml Ross Burton
2022-03-31 18:31 ` [PATCH 6/7] CI: add basic selftest job Ross Burton
2022-03-31 18:31 ` [PATCH 7/7] arm/oeqa: add no-op runtime test Ross Burton
2022-04-01  0:53 ` [PATCH 1/7] runfvp: propagate the exit code correctly Jon Mason

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.